How to access someone else's AWS S3 'bucket' with Boto3 and Username?

2
votes

I'm trying to download AWS S3 content using Python/Boto3.

A third-party is uploading a data, and I need to download it. They provided credentials like this:

  • Username : MYUser
  • aws_access_key_id : SOMEKEY
  • aws_secret_access_key : SOMEOTHERKEY

Using a popular Windows 10 app CyberDuck, my 'Username' is added to the application's path settings, third-party/MYUser/myfolder

Nothing I'm given here is my bucket.

my_bucket = s3.Bucket('third-party/MYUser')
ParamValidationError: Parameter validation failed:
Invalid bucket name 'third-party/MYUser': Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$"

my_bucket = s3.Bucket('third-party')
ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

my_bucket = s3.Bucket('MYStuff')
NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjects operation: The specified bucket does not exist

From what I've read, third-party is the AWS S3 bucket name, but I can't find an explanation for how to access a sub-directory of someone else's bucket.

I'm see Bucket() has some user parameters. I read elsewhere about roles, and access control lists. But I'm not finding a simple example.

How do I access someone else's bucket on AWS S3 given Username?

1
pythonamazon-web-servicesamazon-s3boto3

1 Answers

3
votes

Amazon S3 does not actually have directories. Rather, the Key (filename) of an object contains the full path of the object.

For example, consider this object:

s3://my-bucket/invoices/foo.txt
  • The bucket is my-bucket
  • The Key of the object is invoices/foo.txt

So, you could access the object with:

import boto3

s3_resource = boto3.resource('s3')
object = s3.Object('my-bucket','invoices/foo.txt')

To keep S3 compatible with systems and humans who expect to have folders and directories, it maintains a list of CommonPrefixes, which are effectively the same as directories. They are derived from the names between slashes (/). So, CyberDuck can give users the ability to navigate through directories.

However, the third-party might have only assigned you enough permission to access your own directory, but not the root directory. In this case, you will need to navigate straight to your directory without clicking through the hierarchy.

A good way to use an alternate set of credentials is to store them as a separate profile:

aws configure --profile third-party

You will then be prompted for the credentials.

Then, you can use the credentials like this:

aws s3 ls s3://third-party/MyUser --profile third-party
aws s3 cp s3://third-party/MyUser/folder/foo.txt .

The --profile at the end lets you select which credentials to use.

The boto3 equivalent is:

session = boto3.Session(profile_name='third-party')
s3_resource = session.resource('s3')
object = s3.Object('THEIR-bucket','MYinvoices/foo.txt')

See: Credentials — Boto 3 Documentation