Can't install new pipy packages on Google Cloud Composer

Question

We are using Google Cloud Composer with version composer-1.8.0-airflow-1.10.3 for scheduling our pipelines on GCP.

Lately, when we need to add some new Python packages to the environment (whether using Terraform or Cloud Console), we keep getting a strange error message but Cloud Composer is still up and new packages not installed.

The error message that appears on Cloud Composer UI :

Resource not found (resource=europe-west1-xxxxxxxxxxxxxxxxxxxx-composer-backend-sub-3fb66162-3xxxd-4f43-ba47-xxxxxx)., Http error status code: 400 Http error message: BAD REQUEST Additional errors: {"originalResponse":"{\"paths\":[\"/apis\",\"/apis/\",\"/apis/apiextensions.k8s.io\",\"/apis/apiextensions.k8s.io/v1beta1\",\"/healthz\",\"/healthz/etcd\",\"/healthz/log\",\"/healthz/ping\",\"/healthz/poststarthook/crd-informer-synced\",\"/healthz/poststarthook/generic-apiserver-start-informers\",\"/healthz/poststarthook/start-apiextensions-controllers\",\"/healthz/poststarthook/start-apiextensions-informers\",\"/metrics\",\"/openapi/v2\",\"/version\"]}","reason":"The descriptor url 'https://35.187.59.32/swaggerapi/apis/batch/v1' for type provider 'europe-west1-xxxxxxxxxxxxxxxxxxxx-addons-job-typer' could not be fetched."} , Http error status code: 400 Http error message: BAD REQUEST Additional errors: {"ResourceType":"gcp-types/storage-v1:storage.objects.list","ResourceErrorCode":"403","ResourceErrorMessage":{"code":403,"errors":[{"domain":"global","message":"[email protected] does not have storage.objects.list access to composer-unlistable.","reason":"forbidden"}],"message":"[email protected] does not have storage.objects.list access to composer-unlistable.","statusMessage":"Forbidden","requestPath":"https://www.googleapis.com/storage/v1/b/composer-unlistable/o","httpMethod":"GET","suggestion":"Consider granting permissions to [email protected]"}}

When looking to the Stackdriver logging, we can find this error message which is associated with that environment update:

status: { code: 9
message: "FAILED_PRECONDITION"
}

Any idea on what causes this behavior and how to resolve it?

EDIT:

Note that [email protected] is the service account used by Google for internal communication which is not manageable on IAM page (see Service Account Documentation).

Also, we noticed that sometimes adding a new package works correctly. But next time we try to add another package it fails with the same error message as above.

Did you upgrade your airflow version to 1.10.3 soon before you started getting this error? Also, have you manually upgraded the GKE cluster of the environment to version 1.14++? — itroulli
@itroulli No, the error was present with the older versions as well. Yeah, I think we might have upgraded it manually. How could that cause the above error? Thanks — blackbishop

itroulli itroulli · Accepted Answer · 2019-11-26T14:17:32

I was able to reproduce this error by manually upgrading the GKE cluster of the Composer environment (composer-1.8.2-airflow-1.10.3) to 1.14.8-gke.12:

As mentioned in the documentation, Composer environments not using private IP should not upgrade GKE to 1.14+. These versions have deprecated Swagger in favor of OpenAPI. A future Cloud Composer release will create clusters that use OpenAPI and will enable upgrading existing environments to GKE versions 1.14 and beyond.

Currently, OpenAPI is used only for private IP environments and as a result, there is no issue there.

Unfortunately, there is no way to downgrade your GKE cluster, so the only solution would be to delete and recreate your environment.

Can't install new pipy packages on Google Cloud Composer

2 Answers