How to call same AWS Lambda function concurrently?

Question

I have a AWS lambda function which process data and returns calculated results, then results saved to the db by java backend. Each call takes approximately 2-3 seconds. I followed this tutorial and successfully invoked lambda function from java. But my payload is big and I want to split it and call same function multiple times. One way of doing it is using many threads. My question is, is there any better way of doing this? Does aws lambda library has support for it?

Example payload:

{
user_1: data_1,
user_2: data_2,
...
user_300: data_300
}

I simply want to call lambda function for each user and finish processing of all users data less than 5 seconds.

UPDATE: One way is using Asynchronous Programming as stated in the documentation. Another way can be adding an API Gateway and doing async post request.

The link you have added isn't working. You messed up the formatting. I have rolled the edit back. Please add it again. — Sabito 錆兎
There is always more than one way how to do that. I am thinking if in this case you can maybe use two sqs queues - push the packets to one of them and get the results later. The message size limit should be 256k aws.amazon.com/sqs/faqs/…. You can use s3 instead of sending the data in an event. There will be other options... it depends on what do you prefer. Calling lambda in parallel is just fine. It's rather a question of if you prefer to code or to use some service. — petrch
How big is your payload? Have you considered other ways of Lambda accessing the data, such as placing JSON in a S3 bucket and then the Lambda function reading that. I would look at that as opposed to trying to split input data and using multi thread. — smac2020
@Yatin thanks for editing and notifying about broken link. One user data can be ~4 MB , but it is more about processing time. Main feature of Lambda is being scalable and processing in paralel. And since each users data is independent, I thought processing each of them separately will speed up process. — Ilkin

Pablo Gutierrez Pablo Gutierrez · Accepted Answer · 2021-01-12T02:32:17

To me, Lambda should be as lightweight as possible. Doing 4MB of payload, looks like you need a different approach.

S3 bucket will do, but I prefer API gateway with something like How do you create an asynchronous HTTP request in JAVA?, using multiple threads on the producer of that information.

Yes, definitely, I would go for the API gateway approach in which you can still use for smaller payloads as well.

How to call same AWS Lambda function concurrently?

1 Answers