ML Engine batch prediction job error

Question

I'm encountering a frustrating error when submitting a batch prediction job to GCP ML Engine. The job successfully queues but fails with:

Forbidden: 403 The caller does not have permission (POST https://logging.googleapis.com/v2/entries:write) [while running 'BATCH_PREDICTION/Prediction/ParDo(PredictionDoFn)/Do']

I've successfully trained my model and deployed it all via 'gcloud ml-engine' calls. I can also successfully make an online prediction submission via 'gcloud ml-engine predict'.

I prepared my batch prediction job as instructed here.

DATA_FORMAT="TEXT"
INPUT_PATHS='gs://[BUCKET_NAME]/[INUPT_DIR]/-*'
OUTPUT_PATH='gs://[BUCKET_NAME]/[OUTPUT_DIR]' 
MODEL_NAME='[MODEL_NAME]'
VERSION_NAME='v1'
REGION='europe-west1'
now=$(date +"%Y%m%d_%H%M%S")
JOB_NAME="predict_$now"
MAX_WORKER_COUNT="20"

gcloud ml-engine jobs submit prediction $JOB_NAME \
--model $MODEL_NAME \
--input-paths $INPUT_PATHS \
--output-path $OUTPUT_PATH \
--region $REGION \
--data-format $DATA_FORMAT \
--verbosity debug

I also found a github issue describing a similar problem that recommends including the cloud-logs@google.com group with the owner role in the bucket's ACL. I updated the ACL. When I run

gsutil acl get gs://[BUCKET_NAME]

I see the following as one of the entries.

{
    "email": "cloud-logs@google.com",
    "entity": "group-cloud-logs@google.com",
    "role": "OWNER"
}

However, I still get the 403 error. Any ideas? Thanks.

JJ Geewax JJ Geewax · Accepted Answer · 2017-07-24T09:47:57

It looks like a request being sent to Cloud Logging is being rejected because the account executing the request doesn't have access.

This could be a few things:

You've requested "debug" level verbosity (--verbosity debug) when running the prediction job. Try removing that and see if it works?
You don't have the Stackdriver Logging API enabled. Try turning that on in the API manager
The service account acting on your behalf (I think that's cloud-logs@google.com) doesn't have access to make requests to your project via the Strackdriver logging API. Try adding that specific account e-mail via the IAM console

ML Engine batch prediction job error

1 Answers