Permission issue when deploying a new model version in the training script in GCP AI Platform

Question

I'm trying to automate the model deployment in GCP AI Platform. I.e., after a model is trained successfully, I package and upload the model and source package to GCS, and deploy it as a new version and set as default. I have this at the end of my training script. I need this because the model retrains periodically.

In the training script, the packaging and uploading (calling gsutil with subprocess) worked fine but I ran into permission issues when trying to deploy a new version. I've tried to

call gcloud ai-platform with subprocess
call discovery.build('ml', 'v1').projects().models().versions().create() from googlecloudapis directly
invoke a Cloud Function that does the above

Any way I receive the error

ResponseError: status=[403], code=[Forbidden], message=[Request had insufficient authentication scopes.]

I've added enough permissions to the service account for AI Platform (service-xxxxxxxxx@cloud-ml.google.com.iam.gserviceaccount.com, Google Cloud ML Engine Service Agent), but it didn't work.

It seems inside the training instances a different account is used. discovery.build('ml', 'v1')._http.credentials._service_account returns default instead of an email.

Before I go on to use Cloud Function to monitor the export from the training script, I'd like to ask if I missed anything or if there're any other options?

Thanks.

Have you tried creation of model via gcloud command? cloud.google.com/ai-platform/prediction/docs/… — gogasca

gogasca gogasca · Accepted Answer · 2019-10-10T04:33:04

I look into service account permissions and saw Cloud ML Engine Admin, Developer and Viewer.

Some sample code below:

From here

    Usually, you'll create these credentials with one of the helper
    constructors. To create credentials using a Google service account
    private key JSON file::
        credentials = service_account.Credentials.from_service_account_file(
            'service-account.json')
    Or if you already have the service account file loaded::
        service_account_info = json.load(open('service_account.json'))
        credentials = service_account.Credentials.from_service_account_info(
            service_account_info)
    Both helper methods pass on arguments to the constructor, so you can
    specify additional scopes and a subject if necessary::
        credentials = service_account.Credentials.from_service_account_file(
            'service-account.json',
            scopes=['email'],
            subject='user@example.com')

From here

def get_client(service_account_json):
    """Returns an authorized API client by discovering the Healthcare API and
    creating a service object using the service account credentials JSON."""
    api_scopes = ['https://www.googleapis.com/auth/cloud-platform']

    credentials = service_account.Credentials.from_service_account_file(
        service_account_json)
    scoped_credentials = credentials.with_scopes(api_scopes)

    return discovery.build(
        'ml',
        'v1',        
        credentials=scoped_credentials)

Permission issue when deploying a new model version in the training script in GCP AI Platform

1 Answers