SQS with DLQ - Message processing

Question

I have a doubt about how SQS Source Queue and it´s DLQ handle a message processing.

I know that SQS does not implement a trasaction the same way JMS does, instead it uses a lock with a timeout so it will be available for other subscribers to pull from the queue after this timeout expires.

Having said that how can I implement the behavior below (using more SQS configuration instead of client implementation)?

the subscriber listens for messages in a SQS;
tries to process the message;
If it fails then it retries 3 times, for example;
If my retry attempts fails I must send the message to DLQ and delete it from Source Queue.

stdunbar stdunbar · Accepted Answer · 2017-12-01T20:09:43

This is basically built into SQS for you. In general your configuration will be to have a dead letter queue as the redrive target of your primary queue. Specify 3 as the "Maximum Receives" for the redrive policy. Then, you main code will look something like:

while( true ) {
    ReceiveMessageRequest receiveMessageRequest =
          new ReceiveMessageRequest(eventQueueUrl)
                        .withMaxNumberOfMessages(10)
                        .withWaitTimeSeconds(5);

    List<Message> messages = amazonSQS.receiveMessage(receiveMessageRequest)
                                                              .getMessages();

   for (Message nextMessage: messages) {
       // handle the message.  if the handling fails just continue.

       // if the message was processed correctly, delete it
       amazonSQS.deleteMessage(queueURL,nextMessage.getReceiptHandle() );
   }
}

The key here is that if you don't delete the message from the primary queue it will be available again in, by default, 30 seconds. This can be changed as part of the primary queue's "Default Visibility Timeout". If you do delete it then it will never be seen again. In your failure scenario you'd process the message 3 times and after it fails that many times it will be move to the dead letter queue automatically.

A minor warning about the dead letter queue - the message will not be moved to the dead letter queue until you attempt to read it the fourth time. Your code will never see it but if you try 3 times and expect it to end up in the DLQ after the third read you won't see it. When you try to read it the 4th time AWS realizes that it can't give it to you and moves it for you.

And an aside - I assumed Java given that you mentioned JMS.

SQS with DLQ - Message processing

1 Answers