How to Scale Azure Webjobs

9
votes

I know that webjob scales with webapp i.e. if my webapp is running on 5 instances then webjobs will run on 5 instances as well. I just want to know if there is a way of having multiple instances of the webjob within each instance of the website.

Note: I have max the value of JobHostConfiguration.QueuesConfiguration.BatchSize = 32 already.

Now, my webapp is running on 1 instance. Webjobs is processing 32 concurrently. My goal is to process 3*32 = 96 concurrently.

One possible way to achieve this goal could be deploying the same webjobs 3 times(setting a different name for webJobName property), but i am not sure if this will be a good practice. Please point me the appropriate documentation.

  1. What config.Queues.NewBatchThreshold indicate?

  2. What is the default value for config.Queues.NewBatchThreshold ?

  3. For queue processing what should be the recommend value for config.Queues.NewBatchThreshold ?

1
azureazure-webjobsazure-webjobssdkwebjob

1 Answers

7
votes

Copying Jon's answer from https://github.com/Azure/azure-webjobs-sdk/issues/628:

The NewBatchThreshold is the number of messages to have left to process in the batch before the SDK refreshes the batch.

For example, if your batch size is 20 and NewBatchThreshold is 5, the SDK will grab 20 messages off the queue each time, pass them to your code for you to process them one by one, then when it gets down to the last 5 unprocessed messages (so you've processed 15) it will grab the next 20. See here under "Parallel execution"

By default the NewBatchThreshold is half of BatchSize (see source code) which is why you are seeing a value of 0 for a batch size of 1.

Note that we'd like to make this less confusing, per this issue I just opended.