I'm looking to implement a pub/sub pattern on AWS, using Lambda functions as the publisher/subscriber with SNS in the middle. Each subscriber will have an SQS DLQ for persistence of messages that can't be processed. Is there an advantage to having an SQS queue between the SNS topic and the subscriber as well?
To phrase my question differently perhaps, is it better to have the pattern as Lambda -> SNS -> Lambda -> DLQ or Lambda -> SNS -> SQS -> Lambda -> DLQ?
Adding SQS to your setup allows you to de-couple subscribers from publishes, as SQS persists messages. This means that your publishers can keep publishing messages without requiring subscribers to be available at the same time. In AWS docs it is further explained:
Amazon SQS is a message queue service used by distributed applications to exchange messages through a polling model, and can be used to decouple sending and receiving components—without requiring each component to be concurrently available. Using Amazon SNS and Amazon SQS together, messages can be delivered to applications that require immediate notification of an event, and also persisted in an Amazon SQS queue for other applications to process at a later time.
Lambda -> SNS -> Lambda. However, you then mention "SQS DLQ for persistence". Is this intended to be a separate SQS queue for each recipient? Could you possibly edit your question to clarify your intentions? Thank you. - John Rotenstein