AWS FIFO Queues with lambda – one concurrent lambda per message group

2
votes

I have set up a FIFO queue and I'd like to partition messages by a message group ID.

As an event trigger, I'm using a lambda function.

Now my question: Is it possible to allow only one concurrent lambda function invocation for every message group?

So, if I have Group A and Group B, and each message group has 20 messages, every message group passes one message at a time (I have set up the batch size of 1) to a lambda function and only passes the next one as soon as the previous message processing has finished.

Is that possible using FIFO queues and lambda or do I need to look into other services to allow that?

Note: I've looking into Kinesis with separate shards already, but because there'll be many message groups, the total cost of the shards would be way too much.

Thank you!

2
amazon-web-servicesaws-lambdaamazon-sqs
I dont believe this is possible; however, SQS is "pay per usage", so can you create a queue per message group ID? - LostJon
Yeah, that idea had crossed my mind as well. What about the concurrency part, thought? Just so I understand FIFO queues properly: do they, out of the box, only trigger a (in my case, lambda), when the previous execution has finished? I know that it's at least not the case with standard queues, and I wouldn't expect that to be the case with FIFO queues, either (although it would be great for my use case). - OhMad
so....in my experience, i normally get a trigger (per putMessage(s) operation). FIFO will provide in order delivery, but if you have multiple lambda's as triggers, you cannot guarantee in order processing (tis the world of concurrent execution). if you REQUIRE order, than you would need to do some sort of psuedo mutex and lock a resource if multiple message access the same. - LostJon
yeah, I think setting up multiple queues should work. I had a misunderstanding about how FIFO queues work. Thanks for your help! - OhMad
@OhMad did this work? Asking as am thinking of the same queue-per-partition strategy - Justin

2 Answers

7
votes

It should work the way you want out of the box.

New for AWS Lambda – SQS FIFO as an event source:

In SQS FIFO queues, using more than one MessageGroupId enables Lambda to scale up and process more items in the queue using a greater concurrency limit. Total concurrency is equal to or less than the number of unique MessageGroupIds in the SQS FIFO queue.

0
votes

There are three rules governing the order of messages leaving a FIFO queue, to help understand the processing behavior:

  1. Return the oldest message where no other message with the same MessageGroupId is in flight.
  2. Return as many messages with the same MessageGroupId as possible.
  3. If a message batch is still not full, go back to the first rule. As a result, it’s possible for a single batch to contain messages from multiple MessageGroupIds.