Tasks retrying more than specified retry in Airflow

Question

I have recently upgraded my airflow to 1.10.2. Some tasks in the dag is running fine while some tasks are retrying more than the specified number of retries. One of the task logs shows - Starting attempt 26 of 2. Why is the scheduler scheduling it even after two failure? Anyone facing the similar issue?

Example Dag -

args = {
    'owner': airflow,
    'depends_on_past': False,
    'start_date': datetime(2019, 03, 10, 0, 0, 0),
    'retries':1,
    'retry_delay': timedelta(minutes=2),
    'email': ['[email protected]'],
    'email_on_failure': True,
    'email_on_retry': True
}

dag = DAG(dag_id='dag1',
    default_args=args,
    schedule_interval='0 12 * * *',
    max_active_runs=1)

data_processor1 = BashOperator(
    task_id='data_processor1',
    bash_command="sh processor1.sh {{ ds }} ",
    dag=dag)

data_processor2 = BashOperator(
    task_id='data_processor2',
    bash_command="ssh processor2.sh {{ ds }} ",
    dag=dag)

data_processor1.set_downstream(data_processor2)

Can you post an example of a DAG where such a behaviour is occurring? — dorvak
@dorvak Dag is very simple with just two tasks which are running shell scripts through Bash Operator. But in the logs of the tasks it shows - Dependencies not met for <TaskInstance: dag_id.task_id execution_date [running]>, dependency 'Task Instance Not Already Running' FAILED: Task is already running, it started on execution_date. Dependencies not met for <TaskInstance: dag_id.task_id execution_date [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state which is not a valid state for execution. The task must be cleared in order to be run. — Vipul Pandey
Its giving the above error every time its retrying and the retry attempts are not stopping. — Vipul Pandey
Posting an MCVE is the suggested way to ask questions on stackoverflow. It helps people to reproduce your problem and eventually find a solution. — SergiyKolesnikov
@dorvak I have added the example dag. The dag keeps on retrying in the dag_processor2 task. — Vipul Pandey

Gaurav Shimpi Gaurav Shimpi · Accepted Answer · 2020-02-13T08:27:10

This may be useful,

I tried to generate the same error you are facing in airflow, but I couldn't generate it.

In my Airflow GUI, it shows only single retry then it is marking Task and DAG as failed, which is general airflow behavior, I don't know why and how you're facing this issue.

click here to see image screenshot of my airflow GUI for your DAG

can you please add more details regarding the problem (like logs and all).

Tasks retrying more than specified retry in Airflow

1 Answers