AWS Lambda function does not scale out enough to process SQS messages

2
votes

I have a lambda that is subscribed to an SQS queue to process messages. The message volume is very high.

Problem: The queue grows very quickly, and the lambda function does not scale-out to process the messages fast enough. The concurrent lambda executions goes up to only 20 to 25, even though I have a remaining quota of 950 or more un-used lambda executions. Why is it not spinning up more lambda to process my queue faster? Is this configurable?

This is an issue in my application because I am using a standard SQS queue, which provides no ordering guarantee. So, sometimes I see unlucky messages get suck in the queue for hours, whereas some messages are processed in less than one minute. (As an aside, I'm quite shocked that the queue can be processed in such a random order. Even though there is no ordering guarantee, I would not have expected it to be this bad).

2
amazon-web-servicesaws-lambdaamazon-sqs
Are you using a VPC? – Brandon Schabell
No, the lambda is not using VPC – James Wierzba

2 Answers

4
votes

The problem was the memory allocation of the lambda function. I had naively left it as the default of 128MB. Changing this to 2048MB completely resolved the issue. The lambda now has no issues keeping up with high volumes of SQS messages.

1
votes

Regarding SQS, you didn't say what region you are using but SQS does have a FIFO option in come regions.

FIFO queues are available in the US East (N. Virginia), US East (Ohio), US West (Oregon), EU (Ireland), Asia Pacific (Sydney), and Asia Pacific (Tokyo) regions. FIFO queues have all the capabilities of the standard queue.

Regarding Lambda concurrency, It sounds like you are running out of IP addresses in the subnet you're using. This would only apply if you're using a VPC.

If your function connects to VPC based resources, you must make sure your subnets have adequate address capacity to support the ENI scaling requirements of your function. You can estimate the approximate ENI capacity with the following formula:

Concurrent executions * (Memory in GB / 3 GB)

Where:

Concurrent execution – This is the projected concurrency of your workload. Use the information in Understanding Scaling Behavior to determine this value.

Memory in GB – The amount of memory you configured for your Lambda function.

You can set the concurrent execution limit for a function to match the subnet size limits you have.

References

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html

https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html