SQS messages received out of order

6
votes

I'm using boto3 client in python. I'm pushing messages to sqs but receive them in the wrong order. I can see that their sent time is correct.

The queue is created like that:

boto_session = boto3.Session(region_name=..,aws_access_key_id=..,                                      aws_secret_access_key=...)
sqs_client = boto_session.resource('sqs', endpoint_url=endpoint_url)
sqs_client.create_queue(QueueName=...)

The code to push messages:

boto_session = boto3.Session(region_name=..,aws_access_key_id=..,aws_secret_access_key=...)
sqs_client = boto_session.resource('sqs', endpoint_url=endpoint_url)
queue = sqs_client.get_queue_by_name(QueueName=stream_name)

i = 0
while i < 10:
    print 'b ' + str(i)
    queue.send_message(MessageBody=raw_data.push(json.dumps(dict(id=i)))
    sleep(2)
    i += 1

And code for polling messages:

sqs_resource = boto_session.resource('sqs', endpoint_url=endpoint_url)
queue = sqs_resource.get_queue_by_name(QueueName=queue_name)

while True:
   messages = queue.receive_messages(MaxNumberOfMessages=1,VisibilityTimeout=10,WaitTimeSeconds=5)
   for m in messages:
         print  m.data
         queue.delete_messages(
                Entries=[
                    {
                        'Id': m.message_id,
                        'ReceiptHandle': m.receipt_handle
                    }
                ]
            )

I ran the create queue code, then I pushed messages, then ran a process to consume the messages as shown.

I clearly see the messages are randomly ordered.

Is there a solution for this in sqs? or should I replace the queue?

4
amazon-web-servicesamazon-sqs
The behavior is technically valid, but setting MaxNumberOfMessages to 1 and WaitTimeSeconds < 20 is probably exaggerating what you see... because by doing this when the queue is pre-loaded with messages, you're inadvertently setting up an "I need work, so just give me something, anything, quickly" edge case scenario were the first message is likely to be selected more randomly than it would be in normal operations. Set it to 10 and see what happens. Also try starting your consumer separately, and leaving it running while you run the producer and note your observations, with max 1 and max 10. – Michael - sqlbot
I already tried ten and tried changing max time, nothing helps. – user2283268
Try logging out the time at various points in the code, do you get what you expect? You might find there is some sort of network issue, read messages being lost or something. – andrew pate

4 Answers

7
votes

SQS does not guarantee ordering of messages. It's best to write your code in a way that accepts this or switch to a queue system that does do ordering if you absolutely need it.

Documentation is located here, the relevant passage is:

What if I want the messages to be delivered in order? Amazon SQS makes a best effort to preserve order in messages, but due to the distributed nature of the queue, we cannot guarantee that you will receive messages in the exact order you sent them. You typically place sequencing information or timestamps in your messages so that you can reorder them upon receipt.

3
votes

When creating the SQS queue please mark it as the FIFO queue, which will guarantee the correct order of the messages.

Please check the Recommendations for FIFO (First-In-First-Out) Queues on the AWS documentation.

1
votes

If you 100% require a service that retains the order of messages, you could use Amazon Kinesis.

An Amazon Kinesis stream is an ordered sequence of data records. The order of records is retained when data is retrieved, and records can be received by multiple applications. This differs from an Amazon SQS queue, where message order is not guaranteed and, once consumed, messages disappear from the queue.

Consuming messages from Amazon Kinesis is more complex than retrieving messages from an Amazon SQS queue, but it might fit your requirement better.

See: Amazon Kinesis Streams Key Concepts

0
votes

I guess there are four ways to tackle this:

1) Create an independent counting/ordering service so the publishers can mark each event with a count. Consumers then also check with an ACID 'consumer count' (that collectively they update) to ensure the message it received is due to be processed, otherwise it waits and retries or raises an 'out of order' notification.

2) Make the events themselves truely REST, so they hold all the data needed to be consumed properly so ordering now becomes irrelevent.

3) Create a 'feeder' service that reads from your long SQS queue and feeds a much shorter SQS queue only when it is shorter than some given low value. Multiple consumers then feed from the shorter queue. This does not completely ensure ordering with multiple consumers but sets a limit to how far out of order processing can be.

4) Use a large voracious crowd of consumers so that the queue is always really short, This does not completely ensure ordering but sets a limit to how far out of order processing can be.

With regards to (1) it might be possible to break the handling of each event into two seperate phases, a processing phase and a commit phase (where the processing result gets persisted) it is only during this second commit phase that the ordering is enforced, this may allow multiple processing phases to be running concurrently on seperate consumers.