Trainer module not found in Google Cloud ML Engine

Question

I am trying to tune my variational autoencoder's hyperparameters using Google Cloud ML Engine. I set up my package with the structure they recommend in the docs, so that I specify "trainer.task" as my main module name. Below is an image of my directory structure.

image of directory structure

This works on my own machine when I include the following lines:

import sys
sys.path.append("/path/to/project/directory/")

When I run using the below command, I get the error "No module named trainer". Is there a different path I need to specify or something special I need to do for running on Google Cloud ML Engine?

gcloud ml-engine jobs submit training $JOB_NAME --package-path $TRAINER_PACKAGE_PATH --module-name $MAIN_TRAINER_MODULE --job-dir $JOB_DIR --region $REGION --config config.yaml

Answer 1

Do you have a setup.py file? If so you might be hitting this issue

To debug this:

Get the GCS location of the package from the job

gcloud --project=$PROJECT ml-engine jobs describe $JOB_NAME 

This will output something like

jobId: somejob
state: PREPARING
trainingInput:
  jobDir: gs://BUCKET/job
  packageUris:
  - gs://bucket/job/packages/7d2611c7366f266058da5a9e2c93467426c5fdd018491fa33853516d9db533b1/somepackage-0.0.0.tar.gz
  pythonModule: cifar.task
  region: us-central1
trainingOutput: {} 

• Note the values above are for illustrative purposes only and will differ from your output.

Copy the GCS package to your machine

gsutil cp gs://bucket/job/packages/7d2611c7366f266058da5a9e2c93467426c5fdd018491fa33853516d9db533b1/somepackage-0.0.0.tar.gz /tmp

Unpack the .tar.gz and check it has a directory trainer with an __init__.py file and task.py. If not then you probably specified incorrect values for the command line.

If you include the actual command line (i.e. the values for the variables) and the contents of .tar.gz, I can probably provide a better answer.

Answer 2

Jeremy I had a similar problem. I downloaded and unzipped my files but there was no task.py in it.

These are the cmd line arguments I used:

gcloud ml-engine jobs submit training job11 --package-path=./trainer --module-
 name='Keras_On_GoogleCloud.trainer.shallownet_train' --job-dir=gs://zubair-gc-
 bucket/jobs/job11 --region='us-central1' --config=trainer/cloudml-gpu.yaml -- -
 -job_name='zubair-gc-job11'  --dataset='dataset/animals' --model='shallownet_weights1.hdf5'