1
votes

In Airflow scheduler, there are things like heartbeat and max_threads.
See How to reduce airflow dag scheduling latency in production?.

If I am using Google Cloud Composer, do I have to worry/set these values?
If not, what are the values that Google Cloud Composer uses?

2

2 Answers

1
votes

You can see the airflow config in the composer instance bucket gs://composer_instance_bucket/airflow.cfg. You can tune this configuration as you wish, keeping in mind that cloud composer has some configurations blocked. Also, if you go in the Airflow UI -> Admin -> Configuration you can see the full configuration.

0
votes

If you'd like more control/visibility of these variables, consider hosted Airflow @ Astronomer https://www.astronomer.io/cloud/ as it runs vanilla Airflow.