Number of Nodes in Azure Batch Pool when running executable from Azure Data Factory

Question

We have an ADF pipeline that runs 3 other pipelines which run the same executable using the same Azure Batch pool. Most of the time the executable runs successfully. However, there are instances where the pipeline hangs before it starts running the executable.

Note that all 3 pipelines are using the same Azure Batch pool.

• The executable does not have any parallelism, so does the number of nodes make a difference? Should there be at least 3 nodes?

• Is there an issue with the pipelines using the same Azure Batch pool? Should they be using different pools?

Thanks

Mehmet

Answer 1

I might try and attempt cater 2 questions from my Batch hat on: (I hope this gives you idea to put some debugging in place to identify in depth)

• The executable does not have any parallelism, so does the number of nodes make a difference? Should there be at least 3 nodes?

  • No, number of nodes are not restrictive to any parallelism concept in Batch. AFAIK, and I am not sure from where 3 Nodes is mentioned but is it the case that with 3 nodes you have your scenario working?
  • ErrorHandling: As long as the conceptual code is well managed if there is any exception from the BatchException or FailureInfo with in BatchErrors will get pushed for the user. e.g. https://docs.microsoft.com/en-us/azure/batch/batch-task-fail-event or https://docs.microsoft.com/en-us/rest/api/batchservice/batch-status-and-error-codes
  • From the nodes you can always get the service logs: https://docs.microsoft.com/en-us/rest/api/batchservice/computenode/uploadbatchservicelogs
  • What is the node state when your parallel task is hanging?

• Is there an issue with the pipelines using the same Azure Batch pool? Should they be using different pools?

  • No, AFAIK. I would recommend to detail your scenario if there is anything specific you are seeing.

Since its code node is trying to run, I would recommend setting debug point and checking service log to identify user code run behavior.

Thanks,