2
votes

My Lambda configuration is as belowLambda cofiguration

Lambda Concurrency is set to 50 And SQS trigger batch size is set to 1

Issue: When my queue is flooded with 200+ messages, some of the sqs triggers are missed and the message from the queue goes to inflight state without even triggering the lambda. This is adding a latency in processing by the timeout value set for lambda as I need to wait for the message to come out of flight for it to be reprocessed.

Any inputs will be highly appreciated.

1

1 Answers

2
votes

SQS is integrated with Lambda through event source mappings.

Thanks to the mappings, the Lambda service is long polling the SQS queue, and invoking your function on your behalf. What's more it automatically removes the messages from the queue if your Lambda successfully processes them.

Since you want to process 200+ messages, and you set concurrency to 50 with batch size of 1, it means that you can process only 50 messages in parallel. The rest will be throttled. When this happens:

If your function is throttled, returns an error, or doesn't respond, the message becomes visible again. All messages in a failed batch return to the queue, so your function code must be able to process the same message multiple times without side effects.

To rectify the issue, the following two immediate actions can be considered:

  • increase concurrency of your function to 200 or more.
  • increase batch size to 10. With the batch size and concurrency of 50, you can process 500 (10 x 50) messages concurrently.

Also since you are heavily throttled, setting up a dead-letter queue can be useful. The DLQ helps captures problematic or missed messages from the queue, so that you can process them later or inspect:

If a message fails to be processed multiple times, Amazon SQS can send it to a dead-letter queue. When your function returns an error, Lambda leaves it in the queue. After the visibility timeout occurs, Lambda receives the message again. To send messages to a second queue after a number of receives, configure a dead-letter queue on your source queue.