Using Amazon SQS for multiple consumers receiving the same message

3
votes

I have one primary application sending messages to SQS Queue and want 4 consumer applications to consume the same message and process it however they want to

I am not sure what Queuing architecture to use for this purpose.

I see the option of Standard SQS, SQS FIFO, (SQS + SNSTopic) & Kenesis

For the functionality that I want it seems like either (SQS + SNS Topic) or Kenesis would be the way to go.

But I also have a question regarding Standard SQS & SQS FIFO - Is it not possible for all of the consumers to get the same message if I use SQS FIFO or Standard SQS?

I think I am confused between all the options and overwhelmed by all the information available on the Queues but still confused about which architecture to choose

Primary source of information is Amazon docs and https://www.schibsted.pl/blog/choosing-best-aws-messaging-service/

Some of the questions I went through on stackoverflow:

Link_1 This post answers the question of using multiple consumers with the Queue but not sure if it addressing the issue of same messages consumed by multiple consumers

Link_2 This one answers why Kenesis can be used for my scenario

Helpful_Info I used this article just to understand the differences

I would really appreciate some help on this. I am trying to read as much as possible but would definitely appreciate if someone can help me make the right decision

2
amazon-web-servicesmicroservicesmessage-queueamazon-sqs
as far as i understand your use case, yuou want a single publisher and multiple subscriber model, you can not use sqs alone because sqs will remove the message form queue as soon as it will be consumed from one of your subscriber. so you have to use SNS for that and if you want to persist the messages then use SQS also as one of the subscriber to SNS. may be it needs some modification i'm open for discussion just comment below – varnit
Yes, what you said is exactly what I want. But if I understand correctly, arent the messages stored on the SQS for about 4 days by default? I think it can be increased by editing the configuration if I am not wrong. – Nick Div
Also, I didnt understand what you mean by 'to persist the messages you have to use SQS as a subscriber' – Nick Div
sqs message retention time can be expanded to 14 days, and some i said that persistance thing because some people want to keep a backup of all messages so you can keep the backup of all messages by using some schedule job to S3 or some other service if you want, if you don't want to persist it go for sns – varnit
and please take a look at aws kinesis streams as well however it would be an expensive option and the choice depends on use case – varnit

2 Answers

5
votes

This looks like a perfect use case for SNS-SQS fanout notifications - the messages are sent to an SNS "topic", and SNS will deliver it to multiple SQS queues that are "subscribed" to that topic.

Some notes:

  • Each consumer application (that is attached to a queue) will consume at its own rate - this means that it's possible for one or more to "fall behind". In general, that should be ok as long as the consumers are independent - the queue acts as the buffer so no information is lost.
  • If you need them to be in sync, then that won't work - you should just use a single queue, and a process to synchronously poll the queue and deliver the message to each application.
  • You can perform similar logic with Kinesis (it's built to have multiple consumers), but the extra development complexity and cost is typically not worthwhile unless you are dealing with very large message volumes
    • Kinesis bills by data volume (megabytes), while SQS bills by message count - do the math for your use case.

Don't worry about SQS FIFO unless you need the guarantees it provides around ordering. Plain SQS is already roughly ordered, and will suffice for most use cases.

2
votes

According to your use case SNS seems to be a a great choice however if you want to persist the messages you can use SQS with SNS.