AWS SQS FIFO or AWS DynamoDB for processing large number of messages

0
votes

We have a business scenario to process large number of messages. key factor is order of messages is important.

If we consider to use AWS SQS FIFO, it can process only 300 messages per second. This is not applicable for us, because we expect more than 300 messages per second and batch processing also not possible, since we can send only one message to SQS. Also FIFO can store 20,000 inflate messages. This is also low for us.

Second consideration is to use AWS DynamoDB and use a sorting attribute(like timestamp) to maintain the order of messages.

Will it be suitable to use DynamoDB for our case as we expect huge number of data? Is there any major difference between SQS and DynamoDB in terms of Performance and transactions?

1
amazon-web-servicesamazon-dynamodbamazon-sqsfifoamazon-kinesis
Have you considered Amazon MQ? – Karl Laurentius Roos
What about AWS Kinesis? – Conti
Kinesis also suits our need. – Vasudevan Nair R

1 Answers

1
votes

dynamodb and sqs both scale 'infinitely'. SQS fifo has some limitations as you have pointed out.

if you could get sqs to work for you, it will likely be less expensive, but dynamodb will work as long as you have the ability to pay for the scale you need. Unlimited scaling can be expensive - more so with dynamodb than sqs.