Parse SQS message trigger in AWS Lambda - Python

Question

I have a notification on an S3 bucket upload to place a message in an SQS queue. The SQS queue triggers a lambda function. I am trying to extract the name of the file that was uploaded from the SQS message which triggers the lambda function. My SQS event record looks like this when printed to the CloudWatch logs:

{
"Records": [
    {
        "eventVersion": "2.1",
        "eventSource": "aws:s3",
        "awsRegion": "eu-west-2",
        "eventTime": "2020-04-05T13:55:30.970Z",
        "eventName": "ObjectCreated:Put",
        "userIdentity": {
            "principalId": "A2RFWU4TTDGK95"
        },
        "requestParameters": {
            "sourceIPAddress": "HIDDEN"
        },
        "responseElements": {
            "x-amz-request-id": "024EF2A2E94BD5CA",
            "x-amz-id-2": "P/5p5mDwfIu29SeZcNo3wjJaGAiM4yqBqp4p3gOfLVPeZhf+w5sRjnxsost3BuYub1FVf7tuMFs9KoC98+fwSI9NrT5WbjYq"
        },
        "s3": {
            "s3SchemaVersion": "1.0",
            "configurationId": "ImageUpload",
            "bucket": {
                "name": "HIDDEN",
                "ownerIdentity": {
                    "principalId": "A2RFWU4TTDGK95"
                },
                "arn": "arn:aws:s3:::HIDDEN"
            },
            "object": {
                "key": "activity1.png",
                "size": 41762,
                "eTag": "9e1645a32c2948139a90e75522deb5ab",
                "sequencer": "005E89E354A986B50D"
            }
        }
    }
]
}

Using this code:

import boto3
rek = boto3.client('rekognition')

def test(event, context):
    for record in event['Records']:
       print ("test")
       payload=record["body"]
       fullpayload=str(payload)
       print(fullpayload)

Using ['s3]['object]['key'] to access the filename 'activity1.png' on the payload string gives me this error:

's3': KeyError
Traceback (most recent call last):

How can i access the file name from the lambda function?

Answer 1

What is the output from print(fullpayload)? I would expect payload to be None because there is no attribute named body in the record.

From the example record in your question, you should be doing this:

record['s3']['object']['key']

Answer 2

If this is what I think it is: the S3 Object Create events --> SQS <-- lambda polls, I ran into this also. I was using the s3 put example test and also used my poll from sqs message to make another test. When it came from actually reading the queue not in a test, I had issues.

The output of print(event) is actually json of the entire event like below:

    {   'Records': 
        [   
            {
            'messageId': '61155c1d-aaaa-aaaa-aaaa-aaaaaaaaaaaa',
            'receiptHandle': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaalF5156pb+aSqhRWbEY1XWIAVpingcBgOM8/uv1pIgfVXtfNRwzjtoCcInH6doGo9C38uWG7V48uEzpiAPr6Ao2IkXn5IEQKgxXzgelT5FtW3gpwhsQ3fvsFZdZNkMj2YiBHpdJ9QDgfmjFOWmqEJL+LWHUyksdAHxqVZMFrdaS1Tmno3Xni7DMBg1Ed+HpHkBmAVOWssDfM25lC1RNUivXj8i3iI/gD0yBlCttA4aioAlYNZ0txBrkm8aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaML+jK3JcKXiaslbu+JNZaB7hwevHRNsGIQ2MLuRhX+eHD4BN',
            'body': 
                '{"Records":    
                  [
                    {   
                    "eventVersion":"2.1",
                    "eventSource":"aws:s3",
                    "awsRegion":"us-east-1",
                    "eventTime":"2021-02-24T00:30:07.549Z",
                    "eventName":"ObjectCreated:Put",
                    "userIdentity":{"principalId":"AWS:AROAAAAAAAAAA:Rolehere"},
                    "requestParameters":{"sourceIPAddress":"x.x.x.x"},"responseElements":{"x-amz-request-id":"860A2aaaaaaaB19","x-amz-id-2":"J8epzX+FGaLsliSYSiJaaaaaaaaaaaaaETviVcrVCD/FsQjVLNBJgcv8v/PIh37Y9waaaaaaaaaaaaaaaaoUkoqhlr"},
                    "s3":
                        {"s3SchemaVersion":"1.0",
                        "configurationId":"New arrival",                    
                        "bucket":
                        {"name":"molly-bucketname","ownerIdentity":{"principalId":"A2aaaaaaFMND3"},"arn":"arn:aws:s3:::molly-bucketname"},
                        "object":{"key":"dietcokeofevil.mp3","size":420049,"eTag":"bf153e303affbb6e54feb0a233879d4d","versionId":"B2WJZpLLvpWA4nXP5T5QjVZY09qpnHKa","sequencer":"0060359E131BAA52C0"}
                        }
                    }
                   ]
                 }',
            'attributes': {
                'ApproximateReceiveCount': '1',
                'SentTimestamp': '1614126612305',
                'SenderId': 'AIDAJHaaaaaaaaaaJEBU',
                'ApproximateFirstReceiveTimestamp': '1614126612308'
            },
            'messageAttributes': {},
            'md5OfMessageAttributes': None,
            'md5OfBody': 'c752a7082100075786323ff7e5cdfc26',
            'eventSource': 'aws:sqs',
            'eventSourceARN': 'arn:aws:sqs:us-east-1:#########:queuename',
            'awsRegion': 'us-east-1'
            }
        ]
    }

When an s3 doesn't deliver to lambda, lambda is reading from the queue - it seems like there's a wrapper around the json you actually see in the put examples. If you tried to add the printed event (above) to your test in lambda, it will json error. We need to parse the initial Records json & for the body then json.load the "body"- then parse our s3 info out of it.

import json
import boto3


def lambda_handler(event, context):
    
    #Loops through every file uploaded
    for record in event['Records']:
        #pull the body out & json load it
        jsonmaybe=(record["body"])
        jsonmaybe=json.loads(jsonmaybe)
        
        #now the normal stuff works
        bucket_name = jsonmaybe["Records"][0]["s3"]["bucket"]["name"]
        print(bucket_name)
        key=jsonmaybe["Records"][0]["s3"]["object"]["key"]
        print(key)