Invalid token while running aws S3 cli command on AWS Lambda function

Question

I have a lambda function attached in which i run the command aws ls s3://bucketname.com, which fails with the error:

botocore.exceptions.ClientError: An error occurred (InvalidToken) when calling the ListObjectsV2 operation: The provided token is malformed or otherwise invalid.

I've been struggling to identify the problem, which looks like a permissions issue? However, the attached IAM role has an inline policy which is essentially allowing everything:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Test0",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::bucketname.com/*",
                "arn:aws:s3:::bucketname.com"
            ]
        }
    ]
}

I've already tried specifying the region in the command (aws --region eu-west-3 s3 ls s3://bucketname.com) as suggested in another SO question, but I still get the same error.

edit: If I add the option --debug, I see:

2019-11-26 11:46:48,292 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '/bucketname.com?list-type=2', 'query_string': {'prefix': '', 'delimiter': '/', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/1.16.291 Python/3.7.5 Linux/4.14.138-99.102.amzn2.x86_64 exec-env/AWS_Lambda_python3.7 botocore/1.13.27'}, 'body': b'', 'url': 'https://s3.eu-west-3.amazonaws.com/bucketname.com?list-type=2&prefix=&delimiter=%2F&encoding-type=url', 'context': {'client_region': 'eu-west-3', 'client_config': <botocore.config.Config object at 0x7fc7e6670e10>, 'has_streaming_input': False, 'auth_type': None, 'encoding_type_auto_set': True, 'signing': {'bucket': 'bucketname.com'}}}
.. removed for conciseness..
2019-11-26 11:46:48,294 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3.eu-west-3.amazonaws.com/bucketname.com?list-type=2&prefix=&delimiter=%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/1.16.291 Python/3.7.5 Linux/4.14.138-99.102.amzn2.x86_64 exec-env/AWS_Lambda_python3.7 botocore/1.13.27', 'X-Amz-Date': b'20191126T114648Z', 'X-Amz-Security-Token': b'REMOVED', 'X-Amz-Content-SHA256': b'REMOVED', 'Authorization': b'AWS4-HMAC-SHA256 Credential=REMOVED/eu-west-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=REMOVED'}>
..removed for conciseness..
2019-11-26 11:46:48,312 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InvalidToken</Code><Message>The provided token is malformed or otherwise invalid.</Message>...

Any ideas of what could be going wrong?

any specific reason you are running aws s3 ls instead of using boto3 s3 client? i suspect that the containers where lambda run don't contain the credentials used by your IAM role, and aws s3 ls will eventually look for the credentials in ~/.aws/credentials but there will be nothing there. The SDK, on the other hand, does check if you're using an IAM role, so it should just work seamlessly. This is almost a blind guess, because CLI commands work just fine in EC2 instances with a role attached, but I am not sure the same behaviour applies to Lambda containers. — Thales Minussi
@ThalesMinussi The CLI tool will check for the same IAM role credentials that boto3 checks for. The CLI tool is actually built using boto3. The Lambda container will have temporary AWS credentials set as environment variables that should in theory work with both the SDK and the CLI tool. — Mark B
FYI, you can use the simulator to check if the IAM Role is really accessible via the API. You'll know if you've written the lambda function incorrectly or given the wrong permissions. policysim.aws.amazon.com — youngminz
Was the IAM role attached to the Lambda function deleted and recreated? If that's the case, you have to reattach the IAM role to the Lambda function. — krishna_mee2004
Technically, it's not a bug. It is detailed here: docs.aws.amazon.com/IAM/latest/UserGuide/…. But I agree that it is an odd issue. — krishna_mee2004

krishna_mee2004 krishna_mee2004 · Accepted Answer · 2019-11-26T19:49:22

If an IAM Role that is attached to a Lambda function is deleted and recreated, the Lambda function will show that the role is attached. However, the Lambda function won't work. The solution in this case is to attach a different IAM role to the Lambda and re-attach the old role back. Refer this document for more details.

There is a note mentioned in the document that states the following:

If your Principal element in a role trust policy contains an ARN that points to a specific IAM role, then that ARN is transformed to the role's unique principal ID when the policy is saved. This helps mitigate the risk of someone escalating their privileges by removing and recreating the role. You don't normally see this ID in the console, because there is also a reverse transformation back to the role's ARN when the trust policy is displayed. However, if you delete the role, then the relationship is broken. The policy no longer applies, even if you recreate the role because the new role has a new principal ID that does not match the ID stored in the trust policy. When this happens, the principal ID shows up in the console because AWS can no longer map it back to a valid ARN. The end result is that if you delete and recreate a role referenced in a trust policy's Principal element, you must edit the role to replace the now incorrect principal ID with the correct ARN. The ARN will once again be transformed into the role's new principal ID when you save the policy.

The key thing to note here is that the role ARN is transformed to a unique principal ID which we don't normally see it (ID) in the console. There is a reverse transformation back to the resource's ARN. When the role is deleted and recreated, the role has a new ID.

Invalid token while running aws S3 cli command on AWS Lambda function

1 Answers