How can I enable an ec2 instance to have private access to an S3 bucket?

Question

First of all i'm aware of these questions:

but the solutions are not working for me.

I created a role "sample_role", attached the AmazonS3FullAccess-policy to it and assigned the role to the ec2-instance.

My bucket-policy is as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::My-Account-ID:role/sample_role"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::my_bucket/*"
        }
    ]
}

On my ec2-instance, listing my buckets works fine, both from the command line (aws s3 ls) and from python script. But when I try to upload a file test.txt to my bucket, I get AccessDenied:

import boto3

s3_client = boto3.client('s3')
s3_resource = boto3.resource('s3')
bucket = s3_resource.Bucket('my_bucket')

with open('test.txt', "rb") as f:
    s3_client.upload_fileobj(f, bucket.name, 'text.txt')

Error message:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

Same happens when i just try to list the objects in my bucket. Command line aws s3api list-objects --my_bucket or python script:

import boto3

s3_resource = boto3.resource('s3')
bucket = s3_resource.Bucket('my_bucket')

for my_bucket_object in bucket.objects.all():
    print(my_bucket_object)

Error message:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

When I turn off "Block all public access" in my bucket settings and enable public access in my access control list, it obviously works. But I need to restrict access to the specified role.

What am I missing? Thanks for your help!

s3:ListBucket is a bucket-level action. It needs a resource of arn:aws:s3:::my_bucket (not arn:aws:s3:::my_bucket/* which pertains to objects). — jarmod
Thanks. So i added my_bucket/* to my resources like: "Resource": [ "arn:aws:s3:::my_bucket", "arn:aws:s3:::my_bucket/*" ], but listing the objects still fails. Also this is not explaning why the upload fails, right? — David Salb
Stepping back, what are you trying to achieve with a policy on the S3 bucket? If you give an IAM role access to the bucket then you don't also need to allow it in the S3 bucket policy (typically you would have no bucket policy). Are you trying to restrict access to that bucket to a specific IAM role? — jarmod
Yes i want to restrict access to the bucket to the IAM role speciefied in my bucket policy like so "Principal": { "AWS": "arn:aws:iam::My-Account-ID:role/sample_role" } — David Salb
There's an article on this topic at aws.amazon.com/blogs/security/… — jarmod

John Rotenstein John Rotenstein · Accepted Answer · 2019-12-22T22:49:14

It appears that your requirement is:

You have an Amazon S3 bucket (my_bucket)
You have an Amazon EC2 instance with an IAM Role attached
You want to allow applications running on that EC2 instance to access my_bucket
You do not want the bucket to be publicly accessible

I will also assume that you are not trying to deny other users access to the bucket if they have already been granted that access. You are purely wanting to Allow access to the EC2 instance, without needing to Deny access to other users/roles that might also have access.

You can do this by adding a policy to the IAM Role that is attached to the EC2 instance:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::my_bucket",
                "arn:aws:s3:::my_bucket/*"
            ]
        }
    ]
}

This grants ALL Amazon S3 permissions to the IAM Role for my_bucket. Note that some commands require permission on the bucket itself, while other commands require permission on the contents (/*) of the bucket.

I should also mention that granting s3:* is probably too generous, because it would allow the applications running on the instance to delete content and even delete the bucket, which is probably not what you wish to grant. If possible, limit the actions to only those that are necessary.

How can I enable an ec2 instance to have private access to an S3 bucket?

3 Answers