AWS SQS Lambda Processing n files at once

Question

I have setup an SQS queue where S3 paths are being pushed whenever there is a file upload.

I have also set up a Lambda with an SQS trigger and a batch size of 1.

In my scenario, I have to process n files at a time. Lets say (n = 10).

Say, there are 100 messages in the queue. In my current implementation I'm doing the following steps:

1. Whenever there is a message in the input queue, Lambda will be triggered
2. First I check the active number of concurrent executions I have. If am already running 10 executions, the code will simply return without doing anything. If it is less than 10, it reads one message from the queue and calls for processing.
3. Once the processing is done, the message will be manually deleted from the queue.

With the above mentioned approach, I'm able to process n files at a time. However, Say 100 files lands into S3 at the same time.

It leads to 100 lambda calls. Since we have a condition check in Lambda, the first 10 messages go for processing and the remaining 90 messages go to the in-flight mode.

Now, when some of my processing is done (say 3/10 got over), still the main queue is empty since the messages are still in-flight.

As per my understanding, if processing a file takes x minutes, the visibility timeout of the messages in the queue should be lesser than x (<x) . So that the message would once be available in the queue.

But it also leads to another problem. Say the batch took some more time to complete, message would come back to queue. Lambda would be triggered and once again it goes to the flight mode.

Is there any way, I can control the number of triggers made in lambda. For example: only first 10 messages should be processed however remaining 90 messages should remain visible in the queue. Or is there any other way I can make this design simple ?

I don't want to wait until 10 messages. Even if there are only 5 messages, it should trigger those files. And I don't want to call the Lambda in timely fashion (ex: calling it every 5 minutes).

Answer 1

There is a setting in Lambda called Reserved Concurrency, I'm going to quote from the docs (emphasis mine):

• Reserved concurrency – Reserved concurrency creates a pool of requests that can only be used by its function, and also prevents its function from using unreserved concurrency.

[...]

To ensure that a function can always reach a certain level of concurrency, configure the function with reserved concurrency. When a function has reserved concurrency, no other function can use that concurrency. Reserved concurrency also limits the maximum concurrency for the function, and applies to the function as a whole, including versions and aliases.

For a deeper dive, check out this article from the documentation.

You can use this to limit how many Lambdas can be triggered in parallel - if no Lambda execution contexts are available, SQS invocations will wait.

This is only necessary if you want to limit how many files can be processed in parallel. If there is no actual need to limit this, it won't cost you more to let Lambda scale out for you.

Answer 2

You don't have to limit your concurrent Lambda execution. AWS already handling that for you. Here are the list of maximum concurrent per region from this document: https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html

Burst concurrency quotas

• 3000 – US West (Oregon), US East (N. Virginia), Europe (Ireland)
• 1000 – Asia Pacific (Tokyo), Europe (Frankfurt), US East (Ohio)
• 500 – Other Regions

In this document: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

Scaling and processing

For standard queues, Lambda uses long polling to poll a queue until it becomes active. When messages are available, Lambda reads up to 5 batches and sends them to your function. If messages are still available, Lambda increases the number of processes that are reading batches by up to 60 more instances per minute. The maximum number of batches that can be processed simultaneously by an event source mapping is 1000.

For FIFO queues, Lambda sends messages to your function in the order that it receives them. When you send a message to a FIFO queue, you specify a message group ID. Amazon SQS ensures that messages in the same group are delivered to Lambda in order. Lambda sorts the messages into groups and sends only one batch at a time for a group. If the function returns an error, all retries are attempted on the affected messages before Lambda receives additional messages from the same group.

Your function can scale in concurrency to the number of active message groups. For more information, see SQS FIFO as an event source on the AWS Compute Blog.

You can see that Lambda is handling the scaling up automatically. No need to artificially limit the number of Lambda running to 10.

The idea of Lambda is you want to run as many tasks as possible so that you can achieve parallel execution in the shortest time.