Google AutoML Model Export using CLI/Python API

Question

It has been a few days that GCP AutoML WebUI cannot export models for offline inference (support team says it is a front end component bug that has been fixed, it has been a few days and it not yet).

Said that, I am usually not a fan of GUI to do that, I was hoping I could use CLI or Python google-cloud-automl API to the job. I have to admit Google sucks at documentation, and they keep constantly changing functionalities without giving details and examples. Although all kind of functionalities are working (bucket listing, bucket creation, data upload to bucket, even model training etc.), but model export has been a headache and hard to configure. Before we proceed, please note that GOOGLE_APPLICATION_CREDENTIALS is added to the environment. What I have tried so far:

1. via POST:

# get the token
access_token = subprocess.check_output(
    'gcloud auth application-default print-access-token', shell=True)

data = {
  "outputConfig": {
    "modelFormat": "tf_saved_model", #or "tf-saved-model"
    "gcsDestination": {
      "outputUriPrefix": "gs://bucket/folder"
    }
  }
}


headers = {
    'Authorization': f'Bearer {access_token.decode().rstrip()}',
    'Content-Type': 'application/json; charset=utf-8',
}

url= f"https://automl.googleapis.com/v1/projects/{project_id}/locations/us-central1/models/{model_id}:export"

response = requests.post(url, headers=headers, json=data)

fails with the response.content:

b'{\n  "error": {\n    "code": 400,\n    "message": "List of found errors:\\t1.Field: name; Message: Required field is invalid\\t",\n    "status": "INVALID_ARGUMENT",\n    "details": [\n      {\n        "@type": "type.googleapis.com/google.rpc.BadRequest",\n        "fieldViolations": [\n          {\n            "field": "name",\n            "description": "Required field is invalid"\n          }\n        ]\n      }\n    ]\n  }\n}\n'

2. via google-cloud-automl

location = "us-central1"

client = automl.AutoMlClient()
model_full_id = client.model_path(project=project_id, location=location, model=model_id)

response = client.export_model(????)

There arguments are apparently required (if I run it client.export_model()):

1.Field: name; Message: Location ID was not provided.
2.Field: name; Message: Required field is invalid
3.Field: output_config.gcs_destination.output_uri_prefix; Message: Empty string was given for GCS path or prefix.

I am stuck right in the beginning how to pass arguments. I have tried to configure a metadata for location_to_be_written, and model_format via automl.types.ModelExportOutputConfig(???), but that is a trouble of its own.

I used your code for POST and worked for me. Can you try using the v1 API instead? v1Beta1 is already deprecated, your request might be denied due to the deprecation. cloud.google.com/vision/automl/docs/release-notes#June_04_2020 — Ricco D
Error still persists. I changed to v1 in the url, but POST still gives me same error. I do not think I have authentication issues, since message is not about that, but just to say that I have a service_account oauth2 json in the env! — TwinPenguins
By the way I just revisited the issue tracker that was provided and I saw a screenshot wherein the screen froze. I noticed that the model is not yet deployed. — Ricco D
The model should be deployed first before exporting it to your desired format cloud.google.com/vision/automl/docs/… — Ricco D
Well in past one could simply export without deploying! Deployment is for online prediction, why would one needs to deploy first then be able to export?! Anyways, I even tested this, problem doesn't go away, even after deployment screen freezes whatever format you want to export to!! So annoying — TwinPenguins

Ksign Ksign · Accepted Answer · 2021-03-25T16:44:23

For the post method, You need to use the model-id (e.g. ICN124..., TBL543...)
You can find yours on the UI => AutoML Models tab or by listing your models.
I have managed to export a model by following the example you provided.
For the library, you need to use the full path.
Here is a working example:

from google.cloud import automl

project_id = "YOUR_PROJECT"
model_name="projects/YOUR_PROJECT_NUMBER/locations/us-central1/models/YOUR_MODEL_ID"
gcs_uri = "gs://bucket/folder"

client = automl.AutoMlClient()

gcs_destination = automl.GcsDestination(output_uri_prefix=gcs_uri)
output_config=automl.ModelExportOutputConfig(gcs_destination=gcs_destination, model_format="tflite")

request = automl.ExportModelRequest(name=model_name, output_config=output_config)

response = client.export_model(request=request)

BTW, the issue to export from the UI is now fixed and you should be able to export your models from it.

Google AutoML Model Export using CLI/Python API

2 Answers