Azure Service Bus Topic/Subcription Quotas and Scaling

1
votes

My application is required to publish messages to thousands of clients with high throughput requirements (hundreds of, maybe 1000+ messages per second) in order to keep local client databases in sync with the server database. Every server database operation would potentially require a message to be published to a location of 20 or less clients. Clients only have database information for their particular location. Thousands of locations exist.

My idea was to use Azure Service Bus topic/subscription model for this. Each client would establish a subscription (and connection) to the database sync topic and receive filtered messages for their location.

However, according to the Azure Service Bus Quota page, I am limited to the following:

  • Number of concurrent connections on a namespace - 1,000
  • Number of concurrent connections on a queue/topic/subscription entity - Capped by the limit of concurrent connections per namespace.
  • Number of subscriptions per topic - 2,000

With these limitations now in mind, my solution does not seem usable. What is the best approach to solving this problem using Azure Service Bus? To get around the subscriptions per topic limit I could just create a topic per location. However, the killer really is the number of concurrent connections on a topic/namespace (I think). All clients would be subscribed and long-polling for sync messages to come down from each topic and that would be thousands. Would I need to create a new namespace and topic per location? This seems problematic since I would have to open a new connection each time I want to publish from the server which is very frequent.

Is Azure Service Bus not the correct solution for this problem?

2
azureazureservicebus
Have you looked into using Azure Queues? A quick check here: msdn.microsoft.com/en-us/library/azure/hh767287.aspx) says that you can create an unlimited number of queues with an unlimited number of concurrent clients. – Brendan Green
I hadn't. Thanks for the recommendation. The major problem I see with Azure Queues is it only supports HTTP which does not allow for long-polling operations from the client receivers. Routing is also a major advantage in Service Bus Topics (deliver once and done), whereas with Azure Queues I would have to deliver the message N number of times for each location (where N is the number of clients at the location). – kspearrin
Yes - unless you look at some kind of location-based logical partitioning? It seems that the core limitation is the concurrent connections per namespace. – Brendan Green

2 Answers

1
votes

Not an expert, but a few points.

Azure service bus does support long-polling: Azure Documentation . "As a solution architect/developer, you should consider using Service Bus queues when: Your solution must be able to receive messages without having to poll the queue. With Service Bus, this can be achieved through the use of the long-polling receive operation using the TCP-based protocols that Service Bus supports."

Also, you are correct: Event hubs are the reverse of your problem.

Lastly, Azure notification hubs is actually the closest analog to your use case (it's designed for mobile push to thousands of endpoints, which is an extremely similar scenario). I am not sure if it's feasible to be used for non-mobile applications, but I have not yet found a reason why not. I am considering switching from topics when I have the time to research and POC notification hubs.