SNS - SQS Fanout with lambda processing

Question

For a certain application I was thinking about a setup using SNS - SQS fanout and subscribing lambdas to the SQS queues. Is it overkill to use the SQS queues in between of SNS and the lambda or should I subscribe the lambda to my SNS topics? I don't want to lose messages which failed during processing but maybe I can solve that with a DLQ bound to the lambda? Is there a specific advantage/difference of using SQS in between?

Answer 1

Generally the reason why someone would want to use a fanout approach is if there are multiple actions that need to be processed simultaneously from the same event.

For example in a booking flow a new booking may publish to an SNS topic that has a queue for notifying the warehouse, a queue for sending the customer a confirmation email and a queue for updating some loyalty points based on there order.

This architecture exists because a single queue will have a consumer process and remove the item from the queue once it has been completed.

If you have a single item you would simply use SQS. SQS supports DLQ for Lambda so you would be able to use this to debug in the event of any items not being successfully processed.

One advantage to SQS over SNS is that with SQS you can group multiple records to be distributed at once (upto 10), whereas with SNS it will only contain a single message. By default you can support upto 1000 simultaneous invocations. If you experience more than this items may become backlogged (if your throughput is enough).

Answer 2

Simply put, SQS is largely used as a decoupling mechanism to minimize spikes and dips coming from a producer(SNS) and hence the consumer can consume the messages/events at a uniform rate.

You need to ask these questions first :

1. Are the consumer lambdas able to scale to the number of messages sent to SNS ?
2. Do you have enough reserved concurrency for the consumer lambda ? (If not set lambda can scale to 1000 (regional concurrency limit) . Although this can be increased via a service limit request to AWS support)
3. Do all the lambdas in the account/region have enough concurrency ? (lambda has account level region level throttling)

If the answer to all the above questions are yes, then you do not need a sqs in betwwen. A dl queue attached to lambda should be just fine.