How to implement a priority queue architucture using lambda and SQS?

2
votes

I am using SQS and lambda to process some specific requests. Each request can contain messages from 1 message up to hundred of thousands messages. Its working fine the only issue is that small requests sometimes have to wait for those large requests that are already in the queue (because all concurrent lambda are taken and I don't want to increase my lambda concurrency). So I'm thinking to have two queues, one for small requests and one for large requests so the small request can be processed faster. but the challenge is how to assign the number of lambda concurrency to each queue. Right now I set the lambda concurrency to 30, but if a large request comes in all the 30 lambda would be busy. Is there any way to tell lambda to use concurrent lambda partially (let's say 20 for large an 10 for small requests) based on the SQS queue that triggers it? or is there any other best practice to implement this kind of requirement?

2
amazon-web-servicesaws-lambdaamazon-sqs
Why do you wish to restrict Lambda concurrency? If there is work to be done, why not get it done sooner? There is no price benefit to delaying the work. – John Rotenstein
the reason is lambda functions write to DynamoDB and if don't limit the concurrency then I would get throttled at the DynamoDB. – Nisman
I wonder if that is better solved with Amazon DynamoDB On-Demand – No Capacity Planning and Pay-Per-Request Pricing | AWS News Blog? – John Rotenstein

2 Answers

1
votes

You can have two copies of your function: one for large requests with 20 reserved concurrency, and second for small requests with 10 reserved concurrency.

Each function triggered by corresponding queue - It is most common approach to take care of priority messages.

However, downside will be that you always reserve 10 concurrency even if priority message queue is empty.

0
votes

Is there any way to tell lambda to use concurrent lambda partially

No, once deployed they will run as configured.

Plus I don't think this should be a Lambda usage problem. You can control your active queue length by having a multi-tiered queue. An overly simplified solution would be

  • Create 2 wait queues, one each for large & short messages.
  • Create one active queue which feeds message into your Lambda consumer.
  • Producers send requests to wait queues.
  • Write logic to move messages from wait queues to active queues. This piece of code should have the logic to distribute messages based on your business requirements.