Can use AWS CLI with credentials file but not with environment variables

Question

I usually use my AWS CLI commands after setting a profile, with the environment variable AWS_PROFILE, with the ~/.aws/credentials file. This works.

What I'm currently trying to do is to set up access via environment variables. To do so, I'm setting those variables in my .bash_profile file - I literally copied the aws_access_key_id and aws_secret_access_key entries from the credentials files and put them in my bash_profile file, under the names of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

The environment variables are being correctly set, and, yet, when I try to access AWS resources (in this case, I'm trying to run a ls S3 command over a bucket, so the region doesn't matter), I get the message

An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records

which is very weird to me, since the keys are exactly the same. To confirm this, I switch to my credential profile with the AWS_PROFILE environment variable, and then the command works normally.

I suspected that, somehow, I was setting the wrong environment variables, or something like that. Then, I read this AWS guide, and ran the command aws configure list, which, in the first case (the case with environment variables only), returned

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************AAAA              env
secret_key     ****************AAAA              env
    region                us-east-1              env    ['AWS_REGION', 'AWS_DEFAULT_REGION']

For the second case (with the profile set), it returned

     Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile              dev-staging           manual    --profile
access_key     ****************AAAA shared-credentials-file
secret_key     ****************AAAA shared-credentials-file
    region                us-east-1              env    ['AWS_REGION', 'AWS_DEFAULT_REGION']

In other words, the environment variables are being correctly set, the AWS CLI acknowledges them, their values are the same as when they are set via the credentials file, and, yet, for some reason, it doesn't work that way.

I thought it could be due to the aws_session_token, which I also tried to set as an environment variable, to no avail.

I need to access AWS resources this way to simulate the environment in which my code will run, and I don't see why this would not work the way I'm intending.

Any ideas on how to solve it are appreciated.

" put them in my bash_profile" - how did you put them? Did you export them from there? — Marcin
Yes, I exported them. They are being correctly set, I can access them from a new terminal session. They even show up on the aws configure list for new sessions, so that part is working... — Lucas Lima
Why do you mean by aws_session_token? This is only for temporary credentials. Why would you have it if you use IAM user credentails, i guess? — Marcin
Well, at that point, all logical explanations had been exhausted, I was only trying random things that I thought could explain what was happening. It was irrelevant anyway, having it there or not. Matter of fact is I still don't understand why I cannot access AWS resources if the credentials are the same, only differing in where they come from. — Lucas Lima
From you description, your setup is correct and in theory should work. Can you run env | grep AWS in console and verify that the credentials were correctly exported, they are not missing a single character, or were not overwritten by some other exports? — Marcin

kk. kk. · Accepted Answer · 2020-09-30T00:06:01

You need to edit your ~/.aws/config file when you would like to refer to the credentials from environment variables instead of credentials file.

With AWS Access Keys in credentials file, you must be having profile setup as OR there is no such source_profile config for any profile:

[default]
source_profile = default

However, when you would like to use the credentials set in your environment variables or bash_profile, change/add this setting to every profile in your config file:

[default]
credential_source = Environment

With this change, it should work with your Environment variables as well.

In case you've multiple profiles in ~/.aws/config file, just replace/add source_profile = <profile-name> with credential_source = Environment

Can use AWS CLI with credentials file but not with environment variables

1 Answers