Can SNS route to one of multiple SQS delay queues?

Question

SQS has delay queues that can add a delay before the message is delivered. However, they have a 120,000 cap on the total number of 'in flight' messages. The documentation recommends falling back to another queue when a client gets an OverLimit error.

Is there any way to automatically fallback to another queue by the client publishing to a single SNS topic connected to several SQS delay queues? By that, I mean that the message would generally be pushed from SNS to one of the SQS queues that has available 'in flight' capacity.

Answer 1

Are you actually worried about exceeding the 120,000 'in flight' messages, or are you possibly confusing that with the maximum queue size? (of which there is none).

There is no limit to the number of messages you can have in a queue, the 120,000 limit has to do with the number of messages where you queue consumers have requested messages to consume, but those messages have not yet been processed/deleted?

This is the defintion of 'in flight' from AWS:

Messages are inflight after they have been received from the queue by a consuming component, but have not yet been deleted from the queue. If you reach the 120,000 limit, you will receive an OverLimit error message from Amazon SQS. To help avoid reaching the limit, you should delete the messages from the queue after they have been processed. You can also increase the number of queues you use to process the messages.

Here is a link to confirm that the queue size itself has no limit:

Q: How big can Amazon SQS queues be?

A: A single queue may contain an unlimited number of messages, and you can create an unlimited number of queues.

http://aws.amazon.com/sqs/faqs/#How_big_can_Amazon_SQS_queues_be

My apologies ahead of time if you are NOT confusing the two issues - if thats the case and you really are talking about exceeding the 120K 'inflight' limit, I'll delete my post.

By the way, found this question/answer, which will confirm for you that just because a message is in the delay queue, they are not 'in flight':

Do Delay Queue messages count as "In Flight" in SQS?

Answer 2

Using SNS wouldn't help in this case, because a copy of the message posted to a SNS topic would be delivered to each SQS subscription.

Some pointers that might help:

• Use Dead Letter Queues to store failures: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/SQSDeadLetterQueue.html

• Use Apache Camel to configure your routing and additional capabilities over SQS: http://camel.apache.org/aws-sqs.html

Also, make sure you understand the difference pointed out in @E.J. Brennan answer.