Python: Syncing between two s3 buckets in different accounts when I can't modify the source bucket

0
votes

This is a similar question to this: Is it possible to copy between AWS accounts using AWS CLI? The difference is, I want to do this in python code, and I can't change the s3 bucket policies in the source bucket (it's owned by a 3rd party). I do have the credentials to both buckets.

How do I run a sync command between these two buckets in python code?

2
pythonamazon-web-servicesamazon-s3

2 Answers

0
votes

To directly copy (eg with CopyObject) objects between Amazon S3 buckets in different accounts, you will need to use a single set of credentials that have:

  • Read permission on the source bucket
  • Write permission on the destination bucket

These credentials can come from either account. However, since you cannot change the Bucket policy on the source bucket to reference credentials from your account, you will need to use the credentials that they have provided to you.

Let's say the scenario is:

  • The source is Bucket-A in Account-A
  • The destination is Bucket-B in Account-B
  • You have IAM credentials from Account-A — let's call it User-A
  • User-A has permission to read from Bucket-A

You will need to :

  • Add a Bucket Policy to Bucket-B that permits User-A to write to the bucket (PutObject)
  • When performing the copy, specify "ACL": "bucket-owner-full-control", which will make the objects owned by the destination account. Without this, the objects will continue to be 'owned' by the Account-A even though it is in a bucket owned by Account-B

Finally, please note that boto3 does not natively provide a sync command. You will be responsible for all the sync logic, copying one object at a time.

0
votes

Do it in Python, like this to call the AWS CLI 

import subprocess

cmd='aws s3 sync s3://mybucket s3://mybucket2' 
push=subprocess.Popen(cmd, shell=True, stdout = subprocess.PIPE)
print push.returncode

Or there abouts. :-) Wherever you run this from, say an EC2 instance, make sure it has the user or role that has valid permissions to access both buckets.