How to autoscale Python webjob based on Service Bus Queues in Azure?

Question

When a service bus queue contains any messages, I want my python webjob to scale out so the messages are processed faster.

I have a python webjob which feeds off a Service Bus Queue. The queue is populated each day at midnight and can have between 0 and around 400k messages added to it.

The bottleneck in the current processing is where some data needs to be downloaded, which means that scaling up the webjob won't help as much as parallelising it.

I scaled it up to 10 instances from 1 but that doesn't appear to affect the rate at which messages are consumed from the queue, which suggests that this isn't working the way I expect. When it was on 1 instance it processed ~1.53k in an hour. The hour since scaling out to 10 instances it processed ~1.5k messages (so basically, no difference.)

The code I'm using to interface with the queue is this (if there is a better way of doing this in Python please let me know!):

from azure.servicebus import ServiceBusService, Message, Queue

bus_service = ServiceBusService(
    service_namespace= <namespace>,
    shared_access_key_name='RootManageSharedAccessKey',
    shared_access_key_value=<key>)

while(1):

    msg = bus_service.receive_queue_message(<queue>, peek_lock=False, timeout=1)

    if msg.body is None:

        print("No messages in queue")
        time.sleep(5)
    else:

        number = int(msg.body.decode('utf-8'))
                print(number)

I know in C# there is a QueueTrigger attribute for webjobs but I don't know of anything similar for Python.

I would expect that the more instances running in the app service, the faster messages would be processed, so why isn't that what I see?

You have not shown the part of the code that is time consuming to us. — Klaus D.
It's a function which performs a database call and some other things. While that might also be optimisable, I wouldn't have thought the actual content of the webjob code would make a difference regarding scaling out of the app service? — Loz
I've gone in and done some optimisation/specialisation of the database call, which has sped up the processing but unsure whether that has touched the multiple-instances situation. Will test again. — Loz

Loz Loz · Accepted Answer · 2019-06-14T09:11:23

The bottleneck in the program was the database, which was at maximum. Adding more instances just increased the number of calls on the database and therefore slowed down each instance.

Scaling up the database and optimising the database calls improved performance and also now means that multiple instances can be spun up to further increase throughput.

How to autoscale Python webjob based on Service Bus Queues in Azure?

1 Answers