Boto3 IAM User creation failing with InvalidClientTokenId - The security token included in the request is invalid

Question

When I do this:

session1 = boto3.session.Session(profile_name='my_profile')
iam_client = session1.client('iam')

Given that user='an_existin_iam_user', the below succeeds as I can print the accessKeyId and the accessKeySecret successfully.

responseCreateAccessKey = iam_client.create_access_key(UserName=user)
accessKeyId = responseCreateAccessKey.get('AccessKey').get('AccessKeyId')
accessKeySecret = responseCreateAccessKey.get('AccessKey').get('SecretAccessKey')

Hence, I can use the iam_client to create access credentials for other users (given that the my_profile AWS profile holds creds for an admin IAM user)

Now, with the newly created credentials above, I want to create a new IAM user. The below are the things I have tried to achieve this:

Attempt 1

temp_session = boto3.Session(aws_access_key_id=accessKeyId,aws_secret_access_key=accessKeySecret,region_name='ap-southeast-1')
temp_iam_client = temp_session.client('iam')
responseCreateUser = temp_iam_client.create_user(UserName='my-new-user')

Attempt 2

temp_iam_client = boto3.client('iam',aws_access_key_id=accessKeyId,aws_secret_access_key=accessKeySecret,region_name='ap-southeast-1')
responseCreateUser = temp_iam_client.create_user(UserName='my-new-user')

Attempt 3

temp_session = boto3.session.Session(aws_access_key_id=accessKeyId,aws_secret_access_key=accessKeySecret,region_name='ap-southeast-1')
temp_iam_client = temp_session.client('iam')
responseCreateUser = temp_iam_client.create_user(UserName='my-new-user')

Attempt 4 (as also suggested in one of the answers below)

temp_session = boto3.session.Session(aws_access_key_id=accessKeyId,aws_secret_access_key=accessKeySecret,region_name='ap-southeast-1')
temp_iam_client = temp_session.client('iam')
time.sleep(5)
responseCreateUser = temp_iam_client.create_user(UserName='my-new-user')

Each one of the above fails with the following error

An error occurred (InvalidClientTokenId) when calling the CreateUser operation: The security token included in the request is invalid.

Is there something that I am missing here? All I want to do is create create a new user with the (as shown above) newly generated access credentials (accesskey and access secret). It's ok if I get an AccessDeniedexception. I am expecting that for the users that do not have the right privileges. But the InvalidClientTokenId exception is something I don't understand why am I even getting it.

Also, if it helps, I am trying to do this (creation of access key and secret and then using it to create a new user) in an iterative loop for over a 100 users. (The loop of course handles cases where access keys for the currently being processed user is already maxed out etc.) So say for example, if I try doing the above user creation action, from the ipython console, individually for the user say, existing-test-user, alone, then I get the proper AccessDenied exception. However, when the loop reaches that user name (after iterating over the initial few users for example) then I get the InvalidClientTokenId exception.

On an additional note was going through this the get_credentials() section and I'm just wondering if it has anything to do with credentials caching ?

Llama.new Llama.new · Accepted Answer · 2019-01-17T06:14:02

AWS IAM APIs employ an eventually consistent model, so delays happen during updates to IAM resources (such as creating access keys). That means that even if the function call to create an AWS Access Key succeeds, it does not necessarily mean that the AWS Access Key has completed its creation on the AWS Servers immediately. https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_general.html#troubleshoot_general_eventual-consistency

New Answer

Using backoff, we can set a retry policy for the creation of IAM users using newly created AWS access keys whilst having a timeout:

import boto3
import botocore
import backoff


@backoff.on_exception(backoff.expo, botocore.exceptions.ClientError, max_time=30)
def create_user(access_key_id, access_key_secret, username):
    temp_session = boto3.Session(aws_access_key_id=access_key_id,aws_secret_access_key=access_key_secret,region_name='ap-southeast-1')
    temp_iam_client = temp_session.client('iam')
    temp_iam_client.create_user(UserName=username)


session = boto3.session.Session(profile_name='default')
iam_client = session.client('iam')

response_access_key = iam_client.create_access_key(UserName='user')
access_key_id = response_access_key.get('AccessKey').get('AccessKeyId')
access_key_secret = response_access_key.get('AccessKey').get('SecretAccessKey')
create_user(access_key_id, access_key_secret, 'my-new-user')

Old Answer

Adding a sleep just after creation of the AWS access key is a workaround.

import boto3
import time
session1 = boto3.session.Session(profile_name='my_profile')
iam_client = session1.client('iam')

responseCreateAccessKey = iam_client.create_access_key(UserName=user)
accessKeyId = responseCreateAccessKey.get('AccessKey').get('AccessKeyId')
accessKeySecret = responseCreateAccessKey.get('AccessKey').get('SecretAccessKey')

time.sleep(10)
temp_session = boto3.Session(aws_access_key_id=accessKeyId,aws_secret_access_key=accessKeySecret,region_name='ap-southeast-1')
temp_iam_client = temp_session.client('iam')
responseCreateUser = temp_iam_client.create_user(UserName='my-new-user')

Boto3 IAM User creation failing with InvalidClientTokenId - The security token included in the request is invalid

1 Answers

New Answer

Old Answer