Receiving Azure service bus messages from multiple, dynamically queues with worker role

Question

I'm trying to solve a problem where multiple queues (i.e. one queue per month) is created on Azure Service Bus. The messages in the queue contains information about customer billings. The queues are created like this:

/// <summary>
/// Creates a new queue on Azure Service Bus
/// </summary>
/// <param name="queueName">string</param>
/// <returns>string (name of created queue)</returns>
public string CreateNewQueue(string queueName)
{
    try
    {
        NamespaceManager manager = NamespaceManager.Create();

        if (!manager.QueueExists(queueName))
        {
            manager.CreateQueue(queueName);

            return queueName;
        }

        return string.Empty;
    }
    catch (Exception e)
    {
        Logging.Instance.Fatal(e.ToString());
        return string.Empty;
    }
}

After the queue is created, about 3.000 messages with billing-info (only database ids of the billings, though) are added to the queue.

For processing the queues, I'd like to make use of the Azure Worker Role. I've created an Azure Service project and added a Worker Role with Service Bus Queue worker role to the project.

Now, in the Run() method of the Worker Role, "only" a single queue is used (taken from sample code):

// The name of your queue
const string QueueName = "ProcessingQueue";

I've got a feeling that I'm working against the norm here, by creating a new queue for each billing task :-) However, a new queue is certainly needed since it's critical to track each billing task every month.

Is there any way to couple the Worker Role with the before mentioned dynamically created queues?

Thanks in advance.

Nick Heppleston Nick Heppleston · Accepted Answer · 2015-06-17T08:09:55

I would suggest that you are very much 'working against the norm' here as I can't see how adding a new queue would allow you to track messages from one month to the next. It would make much more sense to have a single queue and log the message (possibly to a DocumentDb instance?) once the message has been successfully sent to the queue and once when it is successfully received and processed from the queue within your Worker Role.

If you absolutely must have separate queues, I can think of two approaches that you could use:

Hold the name of the queue in Service Configuration; at the end of each month, stop the Worker Role, update the Service Config with the name of the 'new month' queue and restart the Worker Role. This has its obvious limitations and is not the right way to go IMHO.
Provide a known name for the queue (such as 'billing-[MONTH]') where [MONTH] is something you can calculate at runtime from DateTime (e.g. 'billing-january', 'billing-august', 'billing-december' etc.); within your Worker Role run-loop, you can then dynamically create a listener against your new queue name and poll the queue.

An Alternative, Suggested Approach

One alternative is to consider looking at Service Bus Topics and Subscriptions. You could have a single 'Billing' Topic, with a Subscription for each month ('January', 'February', 'March' etc.) The Worker Role would listen on each Subscription and only receive messages that were sent in 'February' for example. The message creator would send to a single Topic 'Billing', marking a property on the message 'BillingMonth' as 'January' - this would then be subscribed to by the 'January' Subscription.

For more information about Service Bus Topic's and Subscriptions, see https://azure.microsoft.com/en-gb/documentation/articles/service-bus-dotnet-how-to-use-topics-subscriptions/

Receiving Azure service bus messages from multiple, dynamically queues with worker role

1 Answers