Can't see/receive all messages from AWS SQS

2
votes

I have a FIFO SQS queue, and I am sending 4 messages one after another. After sending first 4 messages, I ran a program to receive them, but only 2 messages were returned - even with long polling and the max messages = 10.

I've sent 4 more messages - now I had 8 messages total. Both the AWS SQS UI for receiving messages, and my code for receiving messages - showed only 2 messages, but said that 8 messages were available.

After sending 4 more messages, you can see in the attached screenshot that SQS UI shows 12 messages available, but lists only 2 messages, and I have C# code to receive messages, with long-polling, that also returns only 2 messages.

enter image description here

What am I doing wrong that I can see all available messages?

2
amazon-web-servicesamazon-sqs
You should look into Visibility Timeout, when one consumer is looking at a message, even if said message is still in the queue (and counts as available), it is not visible by other consumers. If you are for instance trying to poll the queue via Web Console and at the same time consuming the queue with your program, all the messages shown in the console won’t be available to your program and viceversa for the duration of the visibility timeout. – Andre.IDK
Did any of the messages share the same message group ID? If so, this would explain what you are seeing. – John Rotenstein
@Andre.IDK - Yeah, good point - I made sure that no programs were reading from the queue when I was looking at it from SQS UI. And it's 5 sec. – Denis Masyukov
@JohnRotenstein - Yes, all messages shared the same message group ID. Is it good or bad? :)) – Denis Masyukov

2 Answers

2
votes

You need to delete the message with the ReceiptHandle to get the other messages. FIFO queues guarantee ordering inside a message group and a message is not delivered until previous messages are processed. You are seeing only some messages but not others because they are the first in the queue for their message groups.

I've encountered it a while back and blogged about how it works.

2
votes

You have select to use a First-In First-Out (FIFO) queue.

To ensure that messages are processed in the correct order, it is not possible to retrieve messages with the same Message Group ID if there are currently messages with that ID being processed. Otherwise, there is a risk that a message might be processed out-of-order.

This is why your request for more messages is not providing back any messages.

If you simply want the messages to be in FIFO order without specific ordering within message groups, it is recommended that you provide a random Message Group ID so that each message can be processed individually.

See: Amazon SQS FIFO (First-In-First-Out) queues - Amazon Simple Queue Service