Invoke AWS Lambda SNS event only after SQS subscription on same topic has been processed

8
votes

I would like to implement an Amazon SNS topic which first delivers messages to a SQS queue that is a subscriber on the topic, and then executes an AWS Lambda function that is also a subscriber on the same topic. The Lambda function can then read messages from the SQS queue and process several of them in parallel (hundreds).

My question is whether there is any way to guarantee that messages sent to the SNS topic would first be delivered to the SQS queue, and only then to the Lambda function?

The purpose of this is to scale to a large number of messages without having to execute the Lambda function separately for every single message.

3
amazon-web-servicesamazon-sqsamazon-snsaws-lambda
Did not get you. Lambda function is subscriber to the topic and also polling the queue that has the same set of messages? Also remember you cannot goto sqs and then ask for a specific message. So how do you intend to design your lambda function to get the same message from queue? – Rohit
In my use case it does not matter which messages the Lambda function will receive. It will process whatever messages are buffered in the SQS queue. E.g. if 100 messages have been queued up before the first Lambda function invocation is called, then it will process all of them. The other 99 function invocations will quit immediately since the queue is already empty. My worry is that if I send only one SNS message, the Lambda invocation occurs before the message gets to SQS, and the queue will look empty to the function. – Kennu
If all you want is a lambda function to run whenever you send a notification out then why not subscribe a lambda function to sns itself. Why worry about sqs at all? – Rohit
I have two worries in the case without SQS: - If the Lambda function fails, some messages will go unprocessed, so I'd like them to stay in SQS - The solution won't scale, if one Lambda function invocation can only process a single message at a time (there's a limit of 100 invocations in parallel). – Kennu
Yes, your assumption about Lambda failure is correct. It's much easier to handle messages stored inside SQS subscribed to SNS Topic. – adamkonrad

3 Answers

1
votes

What you're looking for is currently not possible with one SNS Topic. If you subscribe your Lambda to a SNS Topic that particular Lambda gets executed each time that SNS Topic receives a message, in parallel.

Solution might be to have two SNS Topics and publish messages to the first one and have your SQS subscribe to it. After successful submission of messages to this first topic you could send a message to the second SNS Topic to execute your Lambda to process messages the first SNS Topic stored to SQS.

Another possible solution might be the above, you could just send some periodic message to the second topic to run the subscribed Lambda. This would allow you to scale your Lambda SQS Workers.

4
votes

For this purpose, triggering the lambda could be better and efficient if used from a cloud watch alert. With the cloud watch alert set at a buffer limit on the SQS, that could fire the lambda to start and process the full queue.

0
votes

Subscribing both an SQS queue and a Lambda function to an SNS topic is a good way to have your Lambda function process SNS messages with low latency. I've tested this process just now. With every attempt the lambda function is invoked after the SQS message is inserted. I wouldn't expect this to always be the case, but it fixes the latency problem as best I am willing to measure. It's not guaranteed and you will need a CloudWatch scheduled event to pick up any missed messages.