Distributed Training DeepLabv3+ with Google Cloud ML

Question

I'm a bit new to Google Cloud ML and having difficulties adapting distributed training to a model. I'm trying to train Google's DeepLabv3+ on Google Cloud ML with a custom training set using the Standard_1 Scale Tier and it passes in a TF_CONFIG value that describes the cluster.

It works fine locally, but it of course takes forever because it's just a laptop. According to this I should be able to use the TF_CONFIG variable to set up a distributed training job. In the training task there are some parameters that talk about distributed training but it doesn't seem to match any of the other distributed training documentation I've seen (ie Using tf.train.ClusterSpec).

Can somebody give me a push in the right direction for this? Thanks!

Answer 1

If you are new to Google Cloud ML, I suggest you to follow first this official tutorial in github for Google Cloud ML to observe differences between running in local or using ML Engine.

Then, have a look on the --scale-tier command mentioned in this documentation, which is used to run distributed training. Bare in mind that you could income on some costs performing distributed training operations.