How to replicate Amazon EBS to S3?

Question

We have a site where users upload files, some of them quite large. We've got multiple EC2 instances and would like to load balance them. Currently, we store the files on an EBS volume for fast access. What's the best way to replicate the files so they can be available on more than one instance?

My thought is that some automatic replication process that uploads the files to S3, and then automatically downloads them to other EC2 instances would be ideal.

EBS snapshots won't work because they replicate the entire volume, and we need to be able to replicate the directories of individual customers on demand.

Answer 1

You could write a shell script that would spawn s3cmd to sync your local filesystem with a S3 bucket whenever a new file is uploaded (or deleted). It would look something like:

s3cmd sync  ./  s3://your-bucket/

Answer 2

Depends on what OS you are running on your EC2 instances:

There isn't really any need to add S3 to the mix unless you want to store them there for some other reason (like backup).

If you are running *nix the classic choice might be to run rsync and just sync between instances.

On Windows you could still use rsync or else SyncToy from Microsoft is a simple free option. Otherwise there are probably hundreds of commercial applications in this space...

If you do want to sync to S3 then I would suggest one of the S3 client apps like CloudBerry or JungleDisk, which both have sync functionality...

If you are running Windows it's also worth considering DFS (Distributed File System) which provides replication and is part of Windows Server...

Answer 3

The best way is to use the Amazon Cloud Front service. All of the replication is managed as part of the AWS. Content is served from several different availability zones, but does not require you to have EBS volumes in those zones.

Amazon CloudFront delivers your static and streaming content using a global network of edge locations. Requests for your objects are automatically routed to the nearest edge location, so content is delivered with the best possible performance.

http://aws.amazon.com/cloudfront/

Answer 4

Two ways:

• Forget EBS, transfer the files to S3 and use S3 as your file-manager than EBS, add cloudfront and use the common-link everywhere.
• Mount S3 bucket on any machines.

1. Amazon CloudFront is a web service for content delivery. It delivers your static and streaming content using a global network of edge locations.

http://aws.amazon.com/cloudfront/

2. You can mount S3 bucket on your linux machine. See below:

s3fs - http://code.google.com/p/s3fs/wiki/InstallationNotes - this did work for me. It uses FUSE file-system + rsync to sync the files in S3. It kepes a copy of all filenames in the local system & make it look like a FILE/FOLDER.

That way you can share the S3 bucket on different machines.