AWS SQS Required or not

3
votes

I am building one service which would use the data that would come from another source(service). So, I am thinking to use the following pipeline:-

Other service ----> SNS Topic ----> SQS ----> AWS Lambda ----> Dynamo Db

So, what above flow says is Another service will push data to SNS Topic to which an SQS would be a subscriber. Now AWS Lambda will have a trigger on this SQS which would listen to the messages in SQS and push it to Dynamo Db. Although it looks okay to do this. But now I am thinking if I really need SQS or not. Can I avoid using it? Instead of using SQS, AWS Lambda directly has a trigger on SNS. I am just thinking of one case if I don't use AWS SQS. How will it handle the scenario if AWS Dynamo DB fails? I think with only SNS, I would lose some messages during the time, my Dynamo Db is in failed state but if I have SQS, then those messages would be stored in SQS queue.

Please let me know if my understanding is correct.

Thanks a lot for your help.

1
amazon-web-servicesamazon-dynamodbaws-lambdaamazon-sqsamazon-sns
SQS cannot trigger a Lambda because it is a polling architecture. You can, however, trigger a Lambda from SNS as it is event driven like Lambda. See supported event sources for Lambda for more information. – stdunbar
Thanks for your answer. I think there is no direct way to do that from SQS. But there is one custom way I think we could do as mentioned in this blog. cloudonaut.io/… . Other than this, my question is mainly regarding if I use just SNS, no SQS, can there be a problem in case if Dynamo Db fails. – hatellla

1 Answers

6
votes

Couldn't answer as much in the comments so I'll try here.

The architecture you linked to is pretty common. The two biggest downfalls are that you're going to billed for Lambda usage even if there is nothing to do and your data may be delayed by the amount of the polling interval which is a minimum of 1 minute. Neither of these things may matter in your problem though.

SQS could be used as a temporary store for data in the event of a DynamoDB failure. But what exactly are you going to do if it fails? What if SQS fails and loses your messages? What if Lambda fails and never runs your code? DynamoDB is a hosted service just like SQS and Lambda - Amazon is going to work very hard to keep it running just like their other services. Trying to architect around every possible failure scenario will mean you never deliver code. I'd focus on the simplest architecture you can and put some trust in the services you're paying for.