Fair Queuing of Publishers in Azure Service Bus

0
votes

I am looking for a way to manage streams of messages from multiple publishers.

A contrived example is this.

Each of my publishers are submitting jobs that can be broken into smaller components that can be processed in any order. Each component takes several seconds to process.

Publisher A submits a job to a queue that consists of 1000 smaller components. Consumer X subscribes to the queue and starts processing the queue messages and sending the results to a result queue.

Half way through the job, Publisher B submits a job with 1000 components.

I would like Consumer X to now alternate between messages from Publisher A and Publisher B so that Publisher B does not need to wait for the first job to complete before receiving results and/ Publisher A just receives results more slowly now that Publisher B has a job. (Similar to fair queuing in networking)

This should cater for any number of Publishers.

If I then scale up the Consumers, each consumer should behave in the same way. They don't need to coordinate their actions to guarantee fair queuing, just a rough approximation is fine.

Is there a pattern for handling this in Service bus? Is there another azure technology I should be using instead?

1
azureazureservicebusazure-servicebus-queues
The easiest way is to create different queues of a topic with multiple subscriptions and then have multiples consumers – Thomas
Thanks @Thomas, I can't quite see how that helps. Can you explain further? – Sudsy
Servicebus messages are processed as FIFO. If you want to process them in parrallel, easiest option is to create multiple queues – Thomas
Fair enough. The consumers would just need to list all queues and read from each of them. I suppose I could use storage queues for this too. I was hoping to find a solution that I could use with Webjobs/ axure functions where the function would get triggered and scaled by new messages in a particular queue but maybe thats not possible with my use case. – Sudsy
@Thomas Do you know if partition keys would work for my scenario. Each publisher could use a unique partition key. Would a standard queue reader then read the different partition ids in parallel? – Sudsy

1 Answers

1
votes

You could create a Topic to send the smaller component messages to.

Create multiple Subscriptions on that Topic. For example 2, and configure each to filter 50% of the messages. Subscription 1 would contain messages from Publisher A and Subscription 2 would contain messages from Publisher B. (here's how)

Then create a reader for each Subscription, resulting in the ability to process messages from multiple publishers at the same time, using multiple subscribers.