Airflow: Prioritize the Dags

Question

I'm fetching the category from metadata table and creating dynamic dags for each category using python script. Right now, we have around 15 categories, so each category will have its own dag. My Dag file has 3 tasks, and it is running sequentially.

Using LocalExecutor.All the 15 dags(dag-runs) triggering in parallel. We don't have enough resources(tasks are heavy) to run all the 15 dags in parallel.

Any way to prioritize the dag-runs? 5 dags should run first, then next five should run and so on. Jobs should run based on available resources, others should be in queue.This should be dynamic.

Any best way to fix this? Kindly help.

Sample dag:

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta


default_args = {
    'start_date': datetime(2019, 6, 03),
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}

dag = DAG('test', catchup=False, default_args=default_args, schedule_interval='*/5 * * * *')

t1 = BashOperator(
    task_id='print_start_date',
    bash_command='date',
    dag=dag)

t2 = BashOperator(
    task_id='sleep',
    bash_command='sleep 50s',
    retries=3,
    dag=dag)

t3 = BashOperator(
    task_id='print_end_date',
    bash_command='date',
    dag=dag)

t1 >> t2 >> t3

Meghdeep Ray Meghdeep Ray · Accepted Answer · 2019-06-05T12:16:32

There isn't a good effective way to do this if you are both running on LocalExecutor and if they all run at the same time.

If you were to move to using a CeleryExecutor and have multiple worker machines then you could use the concept of Airflow Queues to create a "priority" queue which serves the DAGs that you indicate to be high priority.

Another options would be using Sub DAGs. Each of the 15 DAGs can be structured as Sub DAGs and run in the order you want. Here is an example of what that could look like:

start ----> Sub Dag 1 --> Sub Dag  6 --> Sub Dag 11
       |--> Sub Dag 2 --> Sub Dag  7 --> Sub Dag 12
       |--> Sub Dag 3 --> Sub Dag  8 --> Sub Dag 13
       |--> Sub Dag 4 --> Sub Dag  9 --> Sub Dag 14
       |--> Sub Dag 5 --> Sub Dag 10 --> Sub Dag 15

Airflow: Prioritize the Dags

1 Answers