AWS SQS FIFO queue still requires consumers to be idempotent?

2
votes

Some time ago AWS introduced SQS FIFO queue, which guarantees message ordering and exactly one processing. My question is about the second assumption, shouldn't it be changed to exactly one delivery?

Because even if only one copy of a message can be in transit on the queue, I can think to situations where a message can be processed more than once. One example could be:

  1. Worker1 polls message from the queue. Visibility timeout starts.
  2. Due to high traffic/load, worker1 exceeds visibility window to process the message.
  3. Message returns visible on the queue.
  4. Worker2 polls the same message from the queue.
  5. Worker1 ends processing the message.
  6. Worker2 ends processing the message.

From what I can see, we still need to have idempotent workers even when using FIFO queues. Is my assumption correct or am I missing something?

1
amazon-web-servicesamazon-sqsfifoidempotent
This scenario is not FIFO fault. Your applications is incorrectly designed/implemented as you should ensure sufficient visibility window. – Marcin
Hi, I am not saying this is a FIFO fault. Am i asking if the requirement to have idempotent consumers is still relevant for FIFO queue, to handle these scenarios. – revy
Off course. But then you don't have to create such scenarios. Deduplication interval is 5 min. So your producer can send msgA now, and 6 min later, send same msgA (due to a bug, for instance). FIFO will not protect you from this. – Marcin
Crystal clear, thanks! – revy

1 Answers

1
votes

My question is about the second assumption, shouldn't it be changed to exactly one delivery?

No, because the message can actually be delivered to a consumer multiple times. exactly-once processing is a more appropriate naming convention because acknowledging/deleting the message from the queue is part of "processing" the message.

This is in contrast to standard queues, where a message can appear on the queue multiple times, for example, due to retries in the message producer. The "FIFO delivery logic" section of the docs cover this in more extensive details.

we still need to have idempotent workers even when using FIFO queues

Yes, that's correct. Whether or not you are using a FIFO queue, idempotency is still an important consideration for your application; you can't ignore it just because you use a FIFO queue. This may be misleading based on a different interpretation of what "processing" means. The queue cannot tell whether the work associated with the message is actually performed -- the system only cares if the message is deleted or not because it needs to deliver the message again if it is not deleted.

Also consider it's completely in the realm of possibility for a consumer to acknowledge a message either before or after work is actually performed.

To illustrate both points, take for example a basic queue consumer workflow:

  1. A worker (consumer) picks up a message from the queue. Here, the visibility timeout starts.
  2. The worker performs some task according to the message (say, attempt a bank transaction)
  3. Upon performing the task, the worker then deletes the message, removing it from the queue.

Now, suppose that the worker system experiences sudden failure sometime between steps 1 and 3. The work may (or may not!) have been completed, but the message won't be deleted. Once the visibility timeout expires, the message will be visible to be picked up by another worker.

Therefore, it is necessary for the worker to be able to ascertain the true state of the system. For example, whether or not the transaction related to the message has been performed or not.

There are numerous other considerations in how FIFO queues prevent deduplication that you must consider in the design of your system.