AWS SQS FIFO - How to get more than 10 messages at a time?

Question

Currently we want to pull down an entire FIFO queue, and process the contents, and if any issues, release messages back into the queue.

The problem is, that currently AWS only gives us 10 messages, and won't give us 10 more (which is the way you get bulk messages in SQS, multiple 10 max message requests) until we delete or release the first 10.

We need to get more than 10 though. Is this not possible? We understand we can set the group_id to a random string, and that allows processing more, but then the order isn't guaranteed, which defeats the purpose of FIFO.

John Rotenstein John Rotenstein · Accepted Answer · 2018-08-03T02:18:59

I managed to reproduce your results -- I could retrieve 10 messages, but then running the same command again would not return another set of messages.

The relevant documentation seems to be:

While messages with a particular MessageGroupId are invisible, no more messages belonging to the same MessageGroupId are returned until the visibility timeout expires. You can still receive messages with another MessageGroupId as long as it is also visible.

I suspect (just a theory!) that this is to preserve the ordering of messages... If a client asked for a set of messages and they are still being processed, there is the chance that the messages might be returned to the queue. Therefore, no further messages are provided until the original messages are deleted or pass their visibility timeout.

This is only a behaviour of FIFO queues.

It seems that you will need to receive and delete all messages to be able to access them all. I would suggest:

Receive one (or more) message.
Process it. If everything worked, delete the message.
If there were problems, push the message to a new queue.
Once the queue is empty, you would need to read from the new queue and send them back to the original queue (which should preserve ordering).

If you frequently require more capabilities that Amazon SQS provides, you could consider using Amazon MQ – Managed message broker service for ActiveMQ. It has many more capabilities (but is accordingly less 'simple').

AWS SQS FIFO - How to get more than 10 messages at a time?

2 Answers