Amazon SQS - FIFO Queue message request, inconsistent receives

Question

I have a FIFO SQS queue, with visibility time of 30 seconds. The requirement is to read messages as Quickly as possible and clear the queue.

I have code in JAVA in a fashion shown below ( this is just a representation of idea only, not complete code ):

//keep getting messages from FIFO and process them ASAP
while(true)
{
     List<Message> messages = 
     sqsclient.receiveMessage(receiveMessageRequest).getMessages();
    //my logic/code here to process these messages and delete them ASAP
}

In the while loop as soon as the messages are received, they are processed and removed from the queue. But, many times the receiveMessageRequest does not give me messages (returns zero messages).

Also, the messages limitation is only 10 at a time during receive from SQS, which is already an issue, but due to these zero receives, the queues are piling up.

I have no clue why this is happening. The documentation exactly is not clear on this part (or Am I missing in terms of the configuration of the queue?)

Please help!

Note: 1. My FIFO Queue always has messages in this scenario, so there is no case of Queue having zero messages and receive request returning zero

2. The processing and delete times are also Less than the visibility timeout.

Thanks.

Update: I have started running multiple consumers for processing the FIFO queue. Clearly, one consumer is not coping up with the inflow of messages. I shall update in few days how multiple consumers are performing. Thanks

Have you considered using multiple threads/clients to consume messages in parallel? Might you be hitting the limit of 300 transactions per second (TPS) for FIFO queues? Have you considered using Long Polling (Receive Message Wait Time), which will only return a zero-message response after a designated time period (eg 20 seconds)? — John Rotenstein
@John Thanks for replying. Yes. I have tried long polling, similar behavior. It does not always get me messages even if the queue has more than 5000 messages. I am attempting multiple consumers right now. But I wanted to in general understand the behavior — Madhukar R
How many messages are currently showing as in flight? A FIFO queue requires that you handle each message correctly, and if you don't, they'll show up "in flight" and -- by design -- SQS waits for you to act on them or for their visibility timeout to expire. Check this? — Michael - sqlbot
Hi Michael, the In-Flight are almost always zero, that means they are getting processed and deleted. — Madhukar R

user2129038 user2129038 · Accepted Answer · 2017-09-09T03:59:49

You have to first make sure that all messages you received are deleted within VisibilityTimeout. If you are using DeleteMessageBatch for deletion make sure that all 10 messages are deleted.

Also, how did you queue messages when you enqueue them? Order of messages are guaranteed only in a single message group. This also means that if you set the same group id to all messages, you are limited to a single consumer so that order of messages are preserved for sure. Even if use multiple consumers, all messages that belong to a same group becomes invisible to other consumers until visibility timeout expires.

Amazon SQS - FIFO Queue message request, inconsistent receives

1 Answers