Greg Hewgill (ghewgill) wrote,
Greg Hewgill
ghewgill

amazon s3 performance

I've been using Amazon S3 for backups. Yesterday I started a really big backup which I expect to be transferring data for hours or possibly days. I noticed something odd in my server activity graph, though:



The green data is outgoing. The baseline is visible on the left at about 2.2 Mbits/s, everything above that is S3 activity. After transferring data continuously for a few hours at full speed (7.4 Mbits/s in this case), it appears as though my transfer rate decreased steadily over time. What caused this? Is this something local on my server? Is this Amazon S3 throttling my transfers?

Before I jump to conclusions, I should note that since approximately the point where the transfer rate started decreasing, duplicity (the backup program I'm using) has been processing a really large single file (8.3 GB). Duplicity is currently using up quite a chunk of my CPU time, so this slowdown could certainly be related to an inefficiency in processing large data files. I guess I'll know more once this file is done.

Update: It looks like my guess was right. After finishing processing that huge file, the transfer rate went back up to somewhere near normal. Looks like purely a local issue.

Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 1 comment