Greg Hewgill (ghewgill) wrote,
Greg Hewgill

amazon s3 performance

I've been using Amazon S3 for backups. Yesterday I started a really big backup which I expect to be transferring data for hours or possibly days. I noticed something odd in my server activity graph, though:

The green data is outgoing. The baseline is visible on the left at about 2.2 Mbits/s, everything above that is S3 activity. After transferring data continuously for a few hours at full speed (7.4 Mbits/s in this case), it appears as though my transfer rate decreased steadily over time. What caused this? Is this something local on my server? Is this Amazon S3 throttling my transfers?

Before I jump to conclusions, I should note that since approximately the point where the transfer rate started decreasing, duplicity (the backup program I'm using) has been processing a really large single file (8.3 GB). Duplicity is currently using up quite a chunk of my CPU time, so this slowdown could certainly be related to an inefficiency in processing large data files. I guess I'll know more once this file is done.

Update: It looks like my guess was right. After finishing processing that huge file, the transfer rate went back up to somewhere near normal. Looks like purely a local issue.


  • islands of the world tour 2009

    Amy and I haven't been back to North America to see family since we got married in 2007. We've been talking for a while about taking a trip back,…

  • changing travel plans

    Things are suddenly moving fast. It looks like Amy and I are going to try to make it to Austin in time for SXSW (that means leaving Canada around…

  • photos from the usa southwest trip

    I've finished culling, sorting, and organizing the photos that I took during our trip. There are 339 photos, 132 of which are in the "60 mi Gallery"…

  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 1 comment