Google Cloud: How to use Cloud ML in Cloud Datalab

Question

I am training models in keras and tensorflow in Google Cloud Datalab. I want to run the training jobs in Cloud ML. Is it possible to do this with a %% bash [...] command?

Let's say my model looks something like this:

X = tf.placeholder(tf.float32, shape=[None, num_inputs])
hidden = fully_connected(X, num_hidden, activation_fn=None)
outputs = fully_connected(hidden, num_outputs, activation_fn=None)
loss = tf.reduce_mean(tf.square(outputs - X))
optimizer = tf.train.AdamOptimizer(learning_rate)
train  = optimizer.minimize( loss)
init = tf.global_variables_initializer()

num_steps = 3000
    with tf.Session() as sess:
    sess.run(init)

    for iteration in range(num_steps):
        sess.run(train,feed_dict={X: scaled_data})

How can I go about training this in a cloud ML job?

Bradley Jiang Bradley Jiang · Accepted Answer · 2018-04-09T16:06:20

You can with the following steps:

Organize your code (https://cloud.google.com/ml-engine/docs/tensorflow/packaging-trainer) so it looks like: ./ ./setup.py ./train/ ./init.py ./task.py

Here setup.py includes your dependencies. task.py is the python module for your trainer so your code should move there. Also here is a CloudML sample: https://github.com/GoogleCloudPlatform/cloudml-samples/tree/master/iris/trainer

In Datalab or Jupyter, there is a %%file command to save files to local disk. This is one way to create and edit your py files.
Run "gcloud ml-engine jobs submit training" command to submit a training request. For example:

gcloud ml-engine jobs submit training my-job-id --runtime-version=1.6 --module-name trainer.task --job-dir=gs://mybucket/mydir --region us-central1 -- --param1 value1 --param2 value2

Google Cloud: How to use Cloud ML in Cloud Datalab

1 Answers