Decrease limit of parallel executions of lambda function?

1
votes

I have a lambda function which takes too long time to complete. It reads records from DynamoDB and for each record executes a HTTP call to a 3rd party service and later saves the result back to DynamoDB. I want to scale this function up, so that it can handle much more records. Redesign in needed because of max lambda duration, which can be exceeded.

Solution seems to be obvious. One lambda reads all records from DynamoDB and instead or making HTTP requests for all, sends messages to SQS. Another lambda listens on the queue and does the job for one portion of data only. Both lambdas are quick then.

Problem: 3rd party API i'm using allows to invoke only 5 requests per second. To meet this requirement i need to make sure that lambdas consuming SQS events does not exceed concurrency limit of the 3rd party API.

Is is possible to throttle AWS Lambda parallel executions?

Solution

Final solution i used is similar to the one proposed by programmersmurf in the comment below.

  1. scheduler Lambda reads all records from DynamoDB and sends messages to SQS. it's set to run at 4am
  2. worker Lambda is scheduled to work at 5am and is triggered only once a day
  3. worker Lambda takes a number of messages from SQS and makes the HTTP requests in parallel
  4. once worker Lambda is complete and number of messages was positive recursive Lambda call is made
  5. if number of received messages is 0 processing terminates

Benefits of recursive worker Lambda invocation: - no manual tuning of trigger vs. amount consumed SQS messages - no pauses between executions of worker Lambdas - as a result of both maximum throughput

Update

Amazon added ability to control level of AWS Lambda concurrency via settings.

Read more:

https://docs.aws.amazon.com/lambda/latest/dg/history.html https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html

1
aws-lambdaamazon-sqs

1 Answers

0
votes

I don't think there is an existing capability to do this however you can try something like below.

  1. Schedule your SQS consumer lambda to run every one minute
  2. Set max number of messages consume to 60 (seconds) * 5 = 300
  3. Use fix thread pool with 5 threads to make calls to 3rd party. Do throttling inside this code if needed

Hope this helps