We're implementing AWS fanout pattern using SNS topic with multiple SQS queue subscribers. I was wondering what would happen if I successfully publish a message on a SNS topic but it fails (for some reason) to forward it to the queue(s). Will SNS retry and if so, is there a way to control this.
I found this page http://docs.aws.amazon.com/sns/latest/dg/DeliveryPolicies.html that talks about configuring retry policies for SNS HTTP/HTTPS endpoints but there's nothing on SQS.
Practically, AWS guarantees the delivery of messages to SQS from SNS. Since the question quotes as below, I want to point out a reason where delivery to SQS can fail.
I was wondering what would happen if I successfully publish a message on a SNS topic but it fails (for some reason) to forward it to the queue(s).
This can happen only if the SQS we are pointing to, has been deleted (or is being deleted) by some process. As taken from SNS Faqs, if the SQS is unavailable, it will retry for a number of tries until the message is discarded.
SQS: If a SQS queue is not available, SNS will retry 10 times immediately, then 100,000 times every 20 seconds for a total of 100,010 attempts over more than 23 days before the message is discarded from SNS.
This was pointed out by @grigori in a comment.
SQS and SNS were a single team back in 2013 and 2014 when I was a developer on that team. As George Lin has pointed out, the SNS backend responsible for delivery does retry thousands of times for SQS queues (exact number was 100,000 times but might have changed). There is an implicit back off policy. In the first attempt itself, there will be several thousand retries in a very tight loop with no delay. If that fails, there is back off policy to gradually attempt to deliver over time. Generally speaking, deliveries to SQS endpoints rarely fail.
AWS guarantees delivery to SQS, so you don't need to worry about it:
Q: Will Amazon SNS guarantee that messages will be delivered to the subscribed end-point?
When a message is published to a topic, Amazon SNS will attempt to deliver notifications to all subscribers registered for that topic. Due to potential Internet issues or Email delivery restrictions, sometimes the notification may not successfully reach an HTTP or Email end-point. In the case of HTTP, an SNS Delivery Policy can be used to control the retry pattern (linear, geometric, exponential backoff), maximum and minimum retry delays, and other parameters. If it is critical that all published messages be successfully processed, developers should have notifications delivered to an SQS queue (in addition to notifications over other transports).
