AWS SQS FIFO can't receive all messages

2
votes

I'm learning AWS SQS and I've sent 6 messages to a FIFO queue, with the same GroupId. But when I try to poll for messages, I can only receive 2 of them (Why? I set the MaxNumberOfMessages=10 using boto3 API, but I can only receive 2. How can I receive all of the messages?). enter image description here

(As shown in this picture, I have 5 messages available, but I can only receive 2 messages.)

I tried to delete one of two received messages and poll again. The deleted one is gone, and I received a new message. But in total, it's still 2 messages.

2
amazon-web-servicesamazon-sqs
Why are you using the same Message Group ID? Do you want all messages within that group to be processed in-order? – John Rotenstein
@JohnRotenstein Yes – o_yeah

2 Answers

3
votes

Using an Amazon SQS FIFO queue means that you want to receive messages in order. It will also try to ensure ordering within a Message Group.

This means that, if some messages for a given Message Group ID are currently being processed ("in flight"), no more messages for that Message Group will be provided since an earlier message might be returned to the queue if not fully processed. This could result in messages being processed out-of-order.

From Using the Amazon SQS message group ID - Amazon Simple Queue Service:

To interleave multiple ordered message groups within a single FIFO queue, use message group ID values (for example, session data for multiple users). In this scenario, multiple consumers can process the queue, but the session data of each user is processed in a FIFO manner.

When messages that belong to a particular message group ID are invisible, no other consumer can process messages with the same message group ID.

Therefore, your choices are:

  • Don't uses a FIFO queue, or
  • Use different Message Group IDs, or
  • Be happy with what it is doing because that is desired FIFO behaviour
1
votes

From AWS Docs:

The maximum number of messages to return. Amazon SQS never returns more messages than this value (however, fewer messages might be returned).

Just like doc's write, you can get less messages. You have to call ReceiveMessage multiple times, usually done in a loop. You can also increase WaitTimeSeconds so that the ReceiveMessage does not return immedietly if there are no messages.