SQS FIFO guarantees that it will process a message from a queue exactly once. I am wondering what is the core logic behind it. How does it process a message exactly once from the queue in a distributed environment when thousands of transaction are happening. I just wanted to know the basic architecture and design principles. So I think someone who worked one such architecture could really give some insights.
1 Answers
4
votes
According to the documentation, the guarantee is that it won't let the same message be submitted to the queue in a 5 minute window. Message equality is defined by either body hash or an id you pass when submitting.
Unlike standard queues, FIFO queues don't introduce duplicate messages. FIFO queues help you avoid sending duplicates to a queue. If you retry the
SendMessageaction within the 5-minute deduplication interval, Amazon SQS doesn't introduce any duplicates into the queue.
So it sounds to me like they have a central hash table of hashes/ids, they check against it for every new message, and automatically remove hashes/ids older than 5 minutes. You can use Redis with TTL to implement that pretty easily.
I couldn't find any information about the scalability of this but it sounds like it's more expensive to scale than normal queues, judging by the added TPS (transactions per second) limits.
