Dataproc change worker size

0
votes

My jobs (ML jobs) require more than 15GB RAM per worker. How to change the machine type for worker? Currently:

n1-standard-4 (4 vCPU, 15.0 GB memory)

I would prefer to keep my cluster not recreate it.

2
google-cloud-platformgoogle-cloud-dataproc

2 Answers

2
votes

You cannot change the machine type for an existing cluster - this is not something Dataproc supports. What is the reason you want to keep your cluster rather than create a new one?

2
votes

I'm afraid it can be done following these steps:

  1. Go to the VM instances section in the Compute Engine -> Instances
  2. Stop the instance
  3. Change the type machine
  4. Start it

I have just done it and it worked. If a worker is shut down the cluster will continue working, but if you modify the master the cluster may not work after the change, in any case I recommend restart your cluster. Just be careful on doing this because data stored may be removed according to Dataproc documentation.