SQS FIFO Queue Only Receiving One Message Using BOTO send_message

Question

I am trying to run a process which sends a series of messages to an SQS FIFO queue for a set of devices we have on hand every n seconds:

while 1:
    for i in range(0, len(device_list)):
        print("Putting message on queue for {}".format(device_list[i]))

        resp = sqs.send_message(
            QueueUrl=url,
            MessageAttributes={
                'device': {'DataType': 'String', 'StringValue': device_list[i]},
                'start': {'DataType': 'Number', 'StringValue': stime},
                'end': {'DataType': 'Number', 'StringValue': etime}
            },
            MessageBody=(
                'SQS message test'
            ),
            MessageGroupId=id
        )

   sleep(n)

Currently testing with just 20 devices. The problem I'm having is that only the first message I send is being received by the FIFO queue (checked the SQS console directly, the MessagesAvailable attribute says 1). I don't have the same problem with a Standard queue (MessagesAvailable attribute says 20), but ordering is important, so a FIFO queue is the way to go for my purposes.

I'm new to SQS, so I'm unsure if there's a problem with my queue configuration (using default settings), if it's just a feature of SQS FIFO queues, or whether I'm using the boto library correctly. Is there something I'm missing or doing incorrectly?

Solaxun Solaxun · Accepted Answer · 2018-07-23T20:39:06

I have no experience with this library, however reading the docs here leads me to believe a possible issue could be the lack of providing a MessageDeduplicationId. Particularly, the following text:

You may provide a MessageDeduplicationId explicitly.

If you aren't able to provide a MessageDeduplicationId and you enable ContentBasedDeduplication for your queue, Amazon SQS uses a SHA-256
hash to generate the MessageDeduplicationId using the body of the
message (but not the attributes of the message).

Without seeing the rest of your code I cannot tell if ContentBasedDeduplication is active. If it is, since it hashes on the body of the message sent and your message body never changes, you may be continually overwriting the previous message in the queue.

SQS FIFO Queue Only Receiving One Message Using BOTO send_message

1 Answers