Move files directly from one S3 account to another?

Question

Pretty basic question but I haven't been able to find an answer. Using Transit I can "move" files from one S3 bucket on one AWS account to another S3 bucket on another AWS account, but what it actually does is download the files from the first then upload them to the second.

Is there a way to move files directly from one S3 account to another without downloading them in between?

Answer 1

If you are just looking for a ready made solution there are a few solutions out there that can do this. Bucket Explorer works on Mac and Windows and can copy across accounts as can Cloudberry S3 Explorer and S3 Browser but they are Windows only so may not work for you.

I suspect the AWS console could also do it with the appropriate permissions setup but I haven't tested this.

You can also do it using the AWS API as long as you have given the AWS account you are using write permissions to the destination bucket.

Answer 2

Yes, there is a way. And its pretty simple, though it's hard to find it. 8)

For example, suppose your first account username is [email protected] and second is [email protected].

Open AWS Management Console as acc1. Get to the Amazon S3 bucket properties, and in the "Permissions" tab click "Add more permissions". Then add List and View Permissions for "Authenticated Users".

Next, in AWS IAM (it's accessible from among the console tabs) of acc2 create a user with full access to the S3 bucket (to be more secure, you can set up exact permissions, but I prefer to create a temporary user for the transfer and then delete it).

Then you can use s3cmd (using the credentials of the newly created user in acc2) to do something like:

s3cmd cp s3://acc1_bucket/folder/ s3://acc2_bucket/folder --recursive

All transfer will be done on Amazon's side.

Answer 3

Use the aws cli (I used ubuntu 14 ec2 instance) and just run the following command:

aws s3 sync s3://bucket1 s3://bucket2

You will need to specify the account details for one, and have public write access or public read access to the other.

This will sync the two buckets. You can use the same command again later to sync quickly. Best part is that it doesn't seem t require any bandwidth (e.g. files are not passing through local computer).

Answer 4

boto works well. See this thread. Using boto, you copy objects straight from one bucket to another, rather than downloading them to the local machine and uploading them to another bucket.

Answer 5

On Mac OS X I used the Transmit app from Panic. I opened one window for each S3 account (using the API Keys and secrets). I could then drag from one bucket in one window to another bucket in the other window. No need to download files locally first.

Andrew is correct, Transmit downloads the files locally then uploads the files.

Answer 6

Move S3 files from One account to another account

Let's consider there are two accounts source account and destination account. And two buckets source-bucket and destination bucket. We want to move all files from source-bucket to destination-bucket. We can do it by the following steps:

1. aws configure
   • Configure your destination account using the credential or the IAM role.
2. Create user policy for the destination account user.
3. Give destination user access to the source-bucket by modifying the source-bucket policy and adding destination account user policy into it. By this way, destination user will have the access to source-bucket.
4. aws s3 ls s3://source-bucket/
   • this will check whether the destination account is having access to source-bucket. Just for confirmation do this.
5. aws s3 cp s3://source-bucket s3://destination-bucket --recursive
   • this will copy source-bucket all files to destination-bucket. All files are copied using --recursive flag.
6. aws s3 mv s3://source-bucket s3://destination-bucket --recursive
   • this will move all the files from source-bucket to destination-bucket.

Alternative you can use the sync command - aws s3 sync s3://source-bucket s3://detination-bucket

For Better Explanation follow the link

Answer 7

CrossFTP can copy S3 files straight from one bucket to another without downloading them. It is a GUI S3 client that works on Windows, Mac, and Linux.

Answer 8

You can user Cyberduck (open source)

Answer 9

For newly created files (NOT existing objects), you can take advantage of new functionality from AWS. It is Cross-Region Replication (under "Versioning" for the S3 bucket). You can create a policy that will allow you to replicate new objects to a bucket in a different account.

For existing objects, you will still need to copy your objects using another method - unless AWS introduces native functionality for this in the future.

Answer 10

One can so it with running following :

aws s3 mv (sync for keeping buckets in sync) s3://source-bucket s3://destination-bucket --recursive

1. Attach a bucket policy to the source bucket in Source Account.

2. Attach an AWS Identity and Access Management (IAM) policy to a user or role in Destination Account.

3. Use the IAM user or role in Destination Account to perform the cross-account move.