Cloud Composer missing variables file

Question

I've been trying to import a JSON file of environment variables to a newly created Cloud Composer instance using the airflow CLI but when running the below I get the error: Missing variables file.

gcloud composer environments run ${COMPOSER_NAME} \
    --location=${COMPOSER_LOCATION} \
    variables -- \
    -i ${VARIABLES_JSON}

From looking at the source it seems that this happens when an environment variable file doesn't exist at the expected location. Is this because Cloud Composer sets up its variables in a different location so this CLI won't work? I've noticed that there's a env_var.json file that's created on the instance's GCS bucket, I realise I can overwrite this file but that doesn't seem like best practice.

Josh Laird Josh Laird · Accepted Answer · 2019-01-09T14:40:03

It feels like a hack but I copied over the variables.json to my Composer's GCS bucket data folder and then it worked.

This is due to os.path.exists() checking the container that Airflow is running on. I chose this approach over overwriting env_var.json because I get the variables in Airflow's UI with this method.

Script for anyone interested:

COMPOSER_DATA_FOLDER=/home/airflow/gcs/data

COMPOSER_GCS_BUCKET=$(gcloud composer environments describe ${COMPOSER_NAME} --location ${COMPOSER_LOCATION} | grep 'dagGcsPrefix' | grep -Eo "\S+/")

gsutil cp ${ENV_VARIABLES_JSON_FILE} ${COMPOSER_GCS_BUCKET}data

gcloud composer environments run ${COMPOSER_NAME} \
    --location ${COMPOSER_LOCATION} variables -- \
    -i ${COMPOSER_DATA_FOLDER}/variables.json

Cloud Composer missing variables file

1 Answers