IAM user with full S3 access cannot download file

Question

I've created an S3 bucket with my administrator account, and I've created an IAM user (programmatic access only). I've uploaded a dummy txt file (as admin), and I'm using python boto3 to download this file, but I get an Access denied exception.

I have no clear idea why, since it's a very simple setup.

My python code:

import os
import boto3
from dotenv import load_dotenv

# load environment variables
load_dotenv()

boto_session = boto3.session.Session(aws_access_key_id=os.getenv('AWS_ACCESS_ID'), aws_secret_access_key=os.getenv('AWS_ACCESS_KEY'), region_name=os.getenv('AWS_REGION'))
s3 = boto3.client('s3')

with open('test.txt', 'wb') as f:
    s3.download_fileobj('dev-tom', 'readme.txt', f)

The credentials in the environment are correct for the IAM user, I've checked.

Traceback (most recent call last):
  File "/.../lib/python3.7/site-packages/boto3/s3/inject.py", line 678, in download_fileobj
    return future.result()
  File "/.../lib/python3.7/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/.../lib/python3.7/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/.../lib/python3.7/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/.../lib/python3.7/site-packages/s3transfer/download.py", line 343, in _submit
    **transfer_future.meta.call_args.extra_args
  File "/.../lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/.../lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

My bucket has no policy attached, so it's the default.

My IAM user config:

My file details:

The standard environment variables are named AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION. There's no need for you to invent a new set of environment variables. If you use the defaults then you don't even need to fetch them -- boto3 (and all SDKs) will fetch them for you. — jarmod
@JackYu Thanks, but it's no longer necessary.. made a silly programming assumption which was wrong (the bucket is 'public is allowed', that's all) — Tominator
@jarmod I load these from a .env file instead of relying on the ones in my shell environment (since they'll be loaded on a server) — Tominator
Note: if your server is EC2 (or Lambda or ECS) then you shouldn't have any files containing credentials. You should instead use IAM roles to provide credentials to the instance. But I don't know your environment, so this may not be relevant. — jarmod

Tominator Tominator · Accepted Answer · 2020-11-17T14:08:10

Apparently, my issue is not at all with the bucket or its permissions...

This line ignores all previous session things I set up:

s3 = boto3.client('s3')

And just takes whatever access credentials it can find on the environment, which were set to another IAM user that had no full s3 access.

(One of) the correct solutions is to just:

s3 = boto_session.client('s3')

I should learn to program, that is all.

IAM user with full S3 access cannot download file

1 Answers