Amazon SQS sub-queues

2
votes

I'm looking at the feasibility of moving our existing message queue (Mosquitto MQTT) to the cloud (Amazon SQS).

Our existing service uses the concept of sub-queues so that users don't receive messages that aren't intended for them. For example:

/MyQueueName/user1/

/MyQueueName/user2/

/MyQueueName/user3/

etc.

An application running on the users computer subscribes to the sub-queue relevant to their own userid. Then they only receive messages relevant to them.

SQS doesn't seem to have the ability to create sub-queues, nor does it have the flexibility to filter on message attributes. I could allow all messages to be sent to all computers, then only acted upon if the content is relevant, but this seems wasteful, especially considering we are talking in the region of 20,000+ sub-queues at the moment.

How would this best be accomplished on SQS? I could programmatically create queues in the form on QueuenameUsername, when there is a message to be delivered from the source, or a user subscribes, then delete the queue when a user un-subscribes. However this doesn't seem like the best way to do things; creating thousands of separate queues, and would require some code to manage the creation and, more specifically, deletion of queues at the correct intervals.

Any insight appreciated.

Thanks, Phil

1
amazon-web-servicesmessage-queueamazon-sqs

1 Answers

1
votes

I don't think SQS will work for this. Yes, you could create the queues, but one thing to keep in mind is that only one consumer will get one message (i.e. when you send to a "topic" queue not everybody that listens to the queue and grabs messages will get it).

One clarifying question: does the order in which you process the messages matter?

The way I would try to approach this if you want to do it in AWS-land is that I would actually use SNS w/ a HTTPS endpoints. You could dynamically subscribe and unsubscribe to the SNS endpoints you create w/ the downside that the endpoint has to be on the public internet. Every time someone publishes a message to the SNS topic all subscribers will get it.