Multiple SQS queues vs 1 SNS topic

Question

We were having a debate on the best solution today with neither side coming to an agreement.

We have a product which ingests "messages". Every time we get a new message we need to send this data to 3 services for processing.

Service #1 requires the data to be in a special format. For this, we put the data in SQS which the service reads from.

Service #2 reads message fields: [a, b, c] and we send in a protobuf format.

Service #3 reads message fields: [a, b, c, d, e], also protobuf.

For service #2 and #3 we are sending the data into 2 separate SQS queues.

However, we could send the data in an SNS topic which queue #2 and #3 read off of. For this, we would send service #3's protobuf as it has all fields service #2 needs.

The person who wrote service #2 does not want to do this because he does not want to get extra data which they will just ignore.

The person who wrote service #3 thinks it's a waste of resources for the system to protobuf and send to 2 separate SQS queues instead of 1 SNS topic when service #2 could simply read protobuf #3 and just ignore the unwanted fields.

From an architecture standpoint who is correct?

Answer 1

Use SNS when you want to "notify" other services of an event - especially if there are multiple services that might be interested in the event.

The whole format-based differentiation seems like an artificial construct that makes using SQS necessary. SQS is best for point to point communications.

Posting to 2-3 places from one service opens your service up to atomicity & reliability issues (what happens is the posting service crashes after posting to SQS but before posting to SNS?). Having a single system where the messages are being put makes these issues easier to tackle.

In the end, it comes down to what your system requires. If it were up to me, I would change all the downstream services to get notifications over SNS (with a consistent format for all of them).

Answer 2

The debate you're having is opinionated points. There is a trade-off with both the approaches. In the end, you should strive for what will be better in the long term (say 1 year from now)

Publishing to multiple SQS Queues doesn't really make sense when SNS-SQS is just supposed to do that with high availability. I'll recommend having a single publisher (at the end data set is the same) and same contract on the consumer side.

Let the consumer decide what they want to do with the data (discard the event; discard the fields). If the consumer really doesn't want extra fields, an additional anti-corruption layer can be implemented (this ACL will be inside consumer-domain).

The publisher should publish the data in a format which is in most generic form from their perspective. This also ensures that publisher application is not crossing their domain.

Optimizing resources at the cost of coupling the services will be a really bad approach.