Cloud Composer airflow webserver issue with KMS

Question

I'm attempting to utilize the KMS library in one of my DAGs which is running the PythonOperator, but I'm encountering an error in the airflow webserver:

details = "Cloud Key Management Service (KMS) API has not been used in project 'TENANT_PROJECT_ID' before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project='TENANT_PROJECT_ID' then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry."

The airflow webserver is unable to import my specific DAG from my host project to the tenant project (which is where the webserver is running). The DAGs runs with no problem as my host project is correctly setup, but not having the opportunity to monitor it in the UI is a huge drawback.

System specifications:

softwareConfig:
  imageVersion: composer-1.8.2-airflow-1.10.3
  pypiPackages:
    google-cloud-kms: ==1.2.1
  pythonVersion: '3'

It would be nice to be able to leverage KMS and the airflow ui, if not then I might have to add my secrets to cloud composer environmental variables (which is not preferred.)

Any known solutions on this?

Have you enabled KMS API for the tenant project? If not you can do by going to this page console.cloud.google.com/apis/dashboard. Can you post your code to check how you are handling keys decryption? — BVSKanth
I've not enabled the KMS API for the tenant project, and I'm not sure how to access the tenant project to enable this as I do not have direct access to this project. Any tips on accessing the tenant project is appreciated! Note that the actually DAG does run on my worker node (hence its able to access KMS on my host project where composer has its worker nodes etc), and my DAG does what it is supposed to do. But when attempting to sync the dagbag to the tenant project running airflow webserver it throws an error stating KMS is not enabled on tenant project. — jpoudroux
You can't enable if you are not able login to dashboard or missing the tenant project from projects list. Contact the project's administrators. — BVSKanth
Seems like the default setup is that airflow webserver is running on appengine on a tenant project managed gcp without end-user access: stackoverflow.com/questions/53762839/… - but I will confirm with our project admin. If this is the case we would have to not use KMS or run the webserver ourself. I'll update the answer in a comment below. Thanks for the help! — jpoudroux
Seems like we do not have access to the tenant project on cloud composer. This is a bit unfortunate as we`re not doing any processing here, but only syncing our dags for the UI (maybe something else is running under the hood however). — jpoudroux

hexacyanide hexacyanide · Accepted Answer · 2019-11-29T23:38:12

The Airflow webserver is a managed component in Cloud Composer, so as other have stated, it runs in a tenant project that you (as the environment owner) do not have access to. There is currently no way to access this project.

If you have a valid use case for enabling extra APIs in the tenant project, I'd recommend submitting product feedback. You can find out how to do that from the product's public documentation (including if you want to submit a feature request to the issue tracker).

Alternatively, if you're willing to experiment, AIP-24 was an Airflow proposal called DAG database persistence that caches DAGs in the Airflow database, as opposed to parsing/importing them in the webserver (which is the reason why you need KMS in this situation). If you're using Composer 1.8.1+, then you can experimentally enable the feature by setting core.store_serialized_dags=True. Note that it's not guaranteed to work for all DAGs, but it may be useful to you here.

Cloud Composer airflow webserver issue with KMS

1 Answers