How to scale Azure VM Cores

Question

I have a Python code that I need to run on 1000 CSVs in parallel computing to do calculations. One CPU core can finish running the code over each CSV in 8 hours.

Thus I am looking for a way to use Azure for this. I would like to create several virtual machines, say 4x D5v2 with 16 cores each to access a Windows Server that runs on a 64 Cores machine.

I tried to create these VMs in the same Cloud Service and I put them into the same Availability Set, which worked fine. When all VMs are running and I access any one of those VMs, I see that the cores on all other VMs are allocated to "Other Roles".

My questions are:

1) Is it possible to create a hypothetical VM out of 4 VMs to use more cores?

2) How can I manually allocate all cores in the Cloud Service to one specific VM?

Answer 1

Your best solution would be to use Azure Batch With Batch you create a job, and it will run on as many CPU's as you specify it can run on.

Taken from the Batch front page

When you are ready to run a job, Batch starts a pool of compute virtual machines for you, installing applications and staging data, running jobs with as many tasks as you have, identifying failures and re-queuing work and scaling down the pool as work completes. You have control over scale to meet deadlines, manage costs, and run at the right scale for your application.

Answer 2

1) Is it possible to create a hypothetical VM out of 4 VMs to use more cores?

No you can not.

2) How can I manually allocate all cores in the Cloud Service to one specific VM?

You can not do this. You need to use a cloud native solution to scale your process over multiple resources.