Is there an automatic way to remove nodes from an Azure Batch pool as their associated tasks completes?

Question

I have an Azure Batch job where each node has a single task to run. Some of the tasks finish quickly, some run for much longer. I am using an auto-pool with around 100 nodes, so I want to give back the unneeded nodes ASAP.

Is there a built-in way to accomplish this pattern of node usage, or should I use auto-scaling where I set the number of dedicated/low-pri nodes to the number of pending tasks? If I do use auto-scaling, will Azure Batch always remove the idle nodes first? I don't want the active nodes to be disturbed. Thanks.

Answer 1

Use the auto scaling rules, this scenario is exactly what they're for. There is an option to only remove the node once it's tasks are complete - see NodeDeallocationOption. Note that the minimum autoscale rule evaluation interval is 5 minutes so you will have a slight delay between adding your tasks and the scaling to kick in.

Check out https://docs.microsoft.com/en-us/azure/batch/batch-automatic-scaling for some of the autoscale formula.