Airflow trigger tasks only based on previous runs status

5
votes

Is there a way to trigger the next task based on previous task run states. Scenario as below:

  • Task1 - First task in m DAG
  • Task2 - Run task2 only when task1 has succeeded
  • Task3 - Run task 3 only when task3 has succeeded
  • Task4 - Run task 4 only when task1 has run more than 10 hours(SLA missed)

enter image description here

1
airflowairflow-scheduler

1 Answers

2
votes

You have multiple options here:

  1. Use trigger rules, see trigger-rules on how to use them.
  2. Use on_failure_callback and on_success_callback to define what happens if your task fails/succeeds, see this post or the definitions in the BaseOperator API Reference (see Parameters -> on_failure_callback,on_success_callback).
  3. If you only want the emails to be sent in case of failure or SLA miss and no other task should be executed in that case, define:
    default_args = {'email': ['some_email_adress'],'email_on_failure': True"}, airflow will then send an email with the error/sla miss to the defined emails.