AWS Lambda temporal load balancing

Question

I have a AWS Lambda function that I invoke with every 1 minute with >1000 SNS events. This is a problem because my account concurrency is set at 3000, so if I start adding more jobs then eventually I'm going to have >3000 concurrent Lambda instances.

Each job takes around 2-5 seconds to complete which means that within each 1 minute window the concurrency limit will only be threatened within the first 5 seconds and I'll have 0 concurrency for the remaining 55 seconds.

If I set a concurrency limit (e.g. 1000) for the lambda will it handle the first 1000 SNS events and then automatically pick up the remainder once the concurrency frees up? And will I only be charged for the actual runtime rather than time spent waiting for concurrency to reduce?

Otherwise, is there a way that AWS will allow me to spread the load of jobs throughout the 1 minute window so that I can invoke the lambda every ~5 seconds with a subset of the total number of jobs?

Answer 1

If I set a concurrency limit (e.g. 1000) for the lambda will it handle the first 1000 SNS events and then automatically pick up the remainder once the concurrency frees up? And will I only be charged for the actual runtime rather than time spent waiting for concurrency to reduce?

Yes. Setting the concurrency limit definitely comes in handy on your use case and is the way to go. This is one of the reasons why concurrency limit actually exists :)

Unfortunately you can't take advantage of batching with SNS because it always sends one and only event. What you could do is to hook up a SQS queue with your SNS topic and have the Lambda function subscribe to the SQS queue instead, then you can take advantage of batching (max batch size is 10), greatly reducing the amount of concurrent Lambda executions, but still, you'd need to set a concurrency limit to make sure you don't use up all the available concurrency.

Otherwise, is there a way that AWS will allow me to spread the load of jobs throughout the 1 minute window so that I can invoke the lambda every ~5 seconds with a subset of the total number of jobs?

No, but this is unnecessary because of the above.