Currently I'm using an SQS queue to trigger the Lambda function. My Lambda function can only handle one SQS record at a time, and I want to deplete the SQS queue as fast as possible
So I set
Delivery Delay: to 0 seconds,
Batch size: to 1
And set Lambda to use unreserved account concurrency 1000
Suppose the case: One SQS message is pushed to Lambda, now it is being processed. Then another message comes to SQS. Will we be sure a new lambda would be created to handle that message?
Michael is right, as the document of AWS states
As the influx of messages to a queue increases, AWS Lambda automatically scales up polling activity until the number of concurrent function executions reaches 1000, the account concurrency limit, or the (optional) function concurrency limit, whichever is lower. Amazon Simple Queue Service supports an initial burst of 5 concurrent function invocations and increases concurrency by 60 concurrent invocations per minute.
So, for example, 1000 messages arrives at the queue at once, only 5 will be processed in the first minute.
65 lambdas will run concurrently in the second minute... so on
Which is very slow and noticeable, especially if your function is long-running
Lambda would execute serially but never exceed the concurrency limit.
Like in your case, your SQS queue may receive 2000 messages at a time. Lambda will concurrently run 1000 functions and remaining 1000 will run only after the completion of previous executions.
So I think to your best solution to consume the queue fast is to write lambda function to handle multiple messages at a time and change batch size to 10.
