How to scale a SQS FIFO queue multiple listeners

Question

I have three SQS FIFO queues where each one has a data projection listener daemon in EC2 instance as docker pods (SQL Server, PostgreSQL, Elastic Search, etc.)

All queues have the same settings as below (Dead-Letter Queues to setup later).

Queue Type: FIFO    
Messages Delayed:   0
Content-Based Deduplication:    Enabled
Default Visibility Timeout: 30 seconds
Message Retention Period:   14 days
Maximum Message Size:   256 KB

This is all part of an Event Sourcing architecture I am designing using DynamoDB Stream => Lambda SQS Router => SQS FIFO Queues (due to SNS not supporting FIFO queues as subscribers)

Content-Based Deduplication is enabled to avoid duplicate messages in the queue since an error is always possible in the Lambda Router for any of the queues.

Now, I also have set the MessageGroupId for each message to the AggregateId to group them but don't really understand how that is utilized by the consumer side;

I have only one consumer per SQS queue at the moment but what if I want to scale consumers. Is an application concern to make sure multiple consumers will not process messages from the same MessageGroupId; - which is unacceptable since using FIFO queues is due to order retention of events in the system!

John Rotenstein John Rotenstein · Accepted Answer · 2019-05-03T00:49:18

If a message has been received from a FIFO Amazon SQS queue and it is still invisible ("in-flight"), then SQS will not provide another message with the same MessageGroupId.

Therefore, multiple consumers on the same queue will receive messages with a different MessageGroupId and message order within a given MessageGroupId will be retained.

The important thing here is to use a different MessageGroupId where you wish to retain order, but do not use the same MessageGroupId for every message.

See: AWS SQS FIFO - How to get more than 10 messages at a time?

How to scale a SQS FIFO queue multiple listeners

1 Answers