Apache Airflow - More number of DAG file in the dags folder deteriorates the performance

Question

We are using Local Executor in Airflow for scheduling jobs. when number of Dags file increased we are facing slowness in the dag run, task inside the Dag picking very slowly to process.

currently we are keep around 30 dag file. each dag will have maximum of 2 to 4 task.

we tried with less number of dag, i could see its running fast.

is there any limitation on number of dag which we can use in airflow?

Please suggest me how to solve this problem.

We are using Airflow 1.10.9.

(only applicable if your DAGs are similar / having same structure) you can look into programmatically generating DAGs; that ways at least the disk I/O will come down since a single (or few) file will be able to generating several hundreds of DAGs — y2k-shubham

cryanbhu cryanbhu · Accepted Answer · 2020-10-13T05:21:44

0

votes

Increase the resources in your Airflow installation, more CPU, more memory.

Apache Airflow - More number of DAG file in the dags folder deteriorates the performance

1 Answers