Azure service bus queue design

Question

Each user in my application has gmail account.
the application needs to be in sync with incoming emails.
for each user every 1 minute the application should ask gmail servers if there is something new. 99% of the time nothing is new.

From what I know gmail dosen't provide web-hooks

In order to reduce the load from my servers and specially from the DB I want to use the service bus queue in the following manner.

queue properties:

query method: PEEK_AND_LOCK
lock time : 1 minute
max delivery count: X

flow:

1. queue listener receiving message A from the queue and process it.

2. if nothing is new the listener will not delete the message from the queue

3. the message will be delivered again after lock time (1 minute)

basically instead of sending new message to the queue again and again to be processed we just leave them in the queue and relying on the re-delivery mechanism.

we are expecting many users in the near future (100,000-500,000) which means many messages in the queue in a given moment which needs to be processed each minute.

• lets assume the messages size is very small and less the 5GB all together

I am assuming that the re delivery mechanism is used mainly for error handling and I wonder whether our design is reasonable and the queue is apt for that task? or if there are any other suggestions to achieve our goal.

Thanks

Answer 1

You are trying to use the Service Bus Queue as a scheduler which it not really is. As a result SB Queue will be main bottleneck. With your architecture and expected number of users you will find yourself quickly blocked by limitations of the Service Bus queue. For example you have only max 100 concurrent connections, which means only 100 listeners (assuming long-pooling method).

Another issue might be max delivery count property of SB Queue. Even if you set it to int.MaxValue now, there is no guarantee that Azure Team will not limit it in the future.

Alternative solution might be that you implement your own scheduler worker role (using already existing popular tools, like Quartz.NET for example). Then you can experiment - you can host N jobs (which actually do Gmail api requests) in one worker role (each job runs every X minute(s)), and each job may handle M number of users concurently. Worker role could be easily scaled if number of users increases. Numbers N and M can depend on the worker role configuration and can be determined on practice. If applicable, just to save some resources, X can be made variable, for example, based on the time of the day (maybe you don't need to check emails so often at night). Hope it helps.