Can't use plugin module with Cloud Composer

Question

I'm trying to use the Cloud Composer to run my workflow. I wanted to use "GoogleCloudStorageToGoogleCloudStorageOperator" operator which is available from Apache Airflow v1.10, but not being supported in current cloud composer (it supports only Apache Airflow v1.9 for now (2019/01/16)).

Following the guidance of the Google's blog post, I added the operator to a cloud composer environment by myself, and it worked well until a few days ago.

However, for now, when I tried to create a new cloud composer env and to deploy the same DAG that worked well previously, I got a following error message on the Airflow Web UI. And DAG is failed.

Broken DAG: [/home/airflow/gcs/dags/xxx.py] Relationships can only be set between Operators; received GoogleCloudStorageToGoogleCloudStorageOperator

I couldn't understand why this error occurred even if I used the same code and followed the same procedure to deploy my DAG to the cloud composer.

I appreciate for those who give me any advice to solve this problem.

FYI, Cloud Composer now supports Airflow v1.10.0. Check the release note here: cloud.google.com/composer/docs/release-notes — Ryan Yuan
Yeah, I saw the release note. But I couldn't find out how to select Airflow v1.10.0. Is there any document to explain how to select Airflow's version? — reddikih
Click into Composer, there is a "Enable Beta Features" checkbox. After you check it, you can select Airflow 1.10.0. — Ryan Yuan
Thanks for your advice. I could run Airflow v1.10.0 on my environment. And my DAG could be executed without above issue. Again, thanks a lot! — reddikih
Possible duplicate of Airflow error importing DAG using plugin - Relationships can only be set between Operators — VictorGGl

Unknown Unknown · Accepted Answer · 2019-02-21T08:31:48

This was due to a bug in Composer 1.4.2 which was already fixed.

Airflow error importing DAG using plugin - Relationships can only be set between Operators

Can't use plugin module with Cloud Composer

2 Answers