Creating Airflow DAGs on GCP Composer

1
votes

I just learnt about GCP Composer and am trying to move the DAGs from my local airflow instance to cloud and had a couple of questions about the transition.

  1. In local instance I used HiveOperator to read data from hive and create tables and write it back into hive. If I had to do this in GCP how would this be possible? Would I have to upload my data to Google Bucket and does the HiveOperator work in GCP?

  2. I have a DAG which uses sensor to check if another DAG is complete, is that possible on Composer?

2
pythongoogle-cloud-platformairflowgoogle-cloud-composer

2 Answers

0
votes

Yes, Cloud Composer is just managed Apache Airflow so you can do that.

Make sure that you use the same version of Airflow that you used locally. Cloud Composer supports Airflow 1.9.0 and 1.10.0 currently.

0
votes

Composer have connection store. See menu Admin--> Connection. Check connection type available.

Sensors are available.