fetching all the messages from AWS FIFO SQS

Question

I am trying fetch messages from FIFO sqs queue. Here is the sample code:

import boto3

sqs_client = boto3.resource(
    'sqs',
    #aws_access_key_id=AWS_ACCESS_KEY,
    #aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
    region_name='us-east-2'
    )

queue_name = 'test_queue.fifo'

response = sqs_client.create_queue(
    QueueName=queue_name,
    Attributes={
        'FifoQueue': 'true',
        'ContentBasedDeduplication': 'true'
        }
        )


for i in range(0,50):
    status = response.send_message(MessageBody = 'This is test message #'+str(i), MessageGroupId='586474de88e03')

while True:
    messages = response.receive_messages(MaxNumberOfMessages=10)
    if len(messages)>0:

        for message in messages:
            print message.body

    else:

        print('Queue is now empty')
        break

but what I am getting is only the first 10 messages and then its showing "Queue is now empty" although I can see there are 40 available messages in the queue from AWS console.

So here I want to fetch all the messages from the queue in loop. Any lead would be appreciated. Thanks.

E.J. Brennan E.J. Brennan · Accepted Answer · 2017-09-06T09:19:48

When there is a small number of messages in an SQS queue, especially an extremely small number as in your case, you may not get any messages returned and may need to retry the call:

Short poll is the default behavior where a weighted random set of machines is sampled on a receive-message call. Thus, only the messages on the sampled machines are returned. If the number of messages in the queue is small (fewer than 1,000), you most likely get fewer messages than you requested per receive-message call. If the number of messages in the queue is extremely small, you might not receive any messages in a particular receive-message response. If this happens, repeat the request.

Also, generally speaking, once you receive a set of messages, you process them and then delete the messages that you processed - for testing purpose at least you may want to delete each returned message after each 'print message.body', and before you make the next receive request.

fetching all the messages from AWS FIFO SQS

2 Answers