Greg Hewgill (ghewgill) wrote,
Greg Hewgill

amazon s3 performance

I've been using Amazon S3 for backups. Yesterday I started a really big backup which I expect to be transferring data for hours or possibly days. I noticed something odd in my server activity graph, though:

The green data is outgoing. The baseline is visible on the left at about 2.2 Mbits/s, everything above that is S3 activity. After transferring data continuously for a few hours at full speed (7.4 Mbits/s in this case), it appears as though my transfer rate decreased steadily over time. What caused this? Is this something local on my server? Is this Amazon S3 throttling my transfers?

Before I jump to conclusions, I should note that since approximately the point where the transfer rate started decreasing, duplicity (the backup program I'm using) has been processing a really large single file (8.3 GB). Duplicity is currently using up quite a chunk of my CPU time, so this slowdown could certainly be related to an inefficiency in processing large data files. I guess I'll know more once this file is done.

Update: It looks like my guess was right. After finishing processing that huge file, the transfer rate went back up to somewhere near normal. Looks like purely a local issue.


  • 2013 in review

    2013 is the year when everything changed. The biggest event was the birth of our daughter Lily. She was born prematurely in Shanghai while we…

  • 2012 in review

    2012 has been fairly quiet. Maybe it just seems that way because I haven't actually written anything new in this blog since last year's annual…

  • new photo galleries

    I've been busy processing photo galleries from the last year (or two) and putting them online for your perusal. Vancouver 2010 Northland…

  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 1 comment