AWS Lambda SQS Trigger not firing after queue recreation

Question

I've been using terraform to define some SNS topics and then SQS queues to subscribe to them, these queues then have lambdas that are triggered by SQS and pick up the messages from those queues.

Terraform handles the creation of SNS topics, SQS queues and the policies that allows SNS to write message to the queue. The lambdas are deployed separately via the serverless CLI tool.

All of this seems to work fine when I ran terraform first, and then deployed the lambdas. However, I had some issues with something I'd done in terraform so I decided to pull down the entire infrastructure and have it recreated, which again seemed to work fine.

Looking at the lambdas, which never got removed, they still had the link to the queue displaying in the portal as I'd expect. If I went to the new SQS queue I could see it list the lambda correctly listed in the 'Lambda Triggers' tab.

The problem though was that no messages were being picked up, I could see them all accumulating in the queues. The only way I found to unblock the issue was to go into each lambda, select the SQS queue and toggle them all to 'disabled', and then to 'enabled' again. At this point they all started processing the messages that had been sat in the queues.

Has anyone else experienced this? Is there a better way of getting them to 're-associate' with the queues correctly when they are re-created?

stdunbar stdunbar · Accepted Answer · 2020-06-15T16:44:53

I've gotten bitten by this too. The UI hides just a bit too much here. When you create a subscription between SQS and Lambda there is an event source mapping that is created. With the CLI you can see this:

$ aws lambda list-event-source-mappings --function-name sqs-lambda
{
    "EventSourceMappings": [
        {
            "UUID": "8b182e29-f8b4-4637-b4fa-079923ec0bf9",
            "BatchSize": 10,
            "EventSourceArn": "arn:aws:sqs:us-west-2:123456789:lamba-sqs-queue",
            "FunctionArn": "arn:aws:lambda:us-west-2:123456789:function:sqs-lambda",
            "LastModified": "2020-04-24T15:43:28.192000-06:00",
            "State": "Enabled",
            "StateTransitionReason": "USER_INITIATED"
        }
    ]
}

But if I delete the queue and recreate it, even though both the SQS queue and the Lambda have the same ARN in there the mapping is invalid.

The way I've done this is to first remove the event source mapping:

aws lambda delete-event-source-mapping --uuid <uuid-of-your-mapping>

and then recreate it:

aws lambda create-event-source-mapping --event-source-arn arn:aws:sqs:us-west-2:123456789:lamba-sqs-queue --function-name sqs-lambda

However - be aware that just like it takes a minute or so until you can recreate an SQS queue with the same name it takes a minute or so until you can recreate the event source mapping.

AWS Lambda SQS Trigger not firing after queue recreation

1 Answers