SubDagOperator Invalid arguments were: **kwargs: {'executor': LocalExecutor(parallelism=32)}

Question

We're upgrading to Airflow 2.0 and I have the below task:

with dag:
    cms_ingest = SubDagOperator(
        subdag=cms_s3ingest(
            DAG_NAME, 'cms_s3ingest', default_args['start_date'], dag.schedule_interval),
        task_id='cms_s3ingest',
        # so that subtasks can run in parallel
        executor=LocalExecutor(),
        task_concurrency=4)

but I'm seeing this error:

airflow.exceptions.AirflowException: Invalid arguments were passed to SubDagOperator (task_id: cms_s3ingest). Invalid arguments were: **kwargs: {'executor': LocalExecutor(parallelism=32)}

in my airflow.cfg file I have the value: parallelism = 32

which I believe is what LocalExecutor is using https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/executors/local_executor/index.html#airflow.executors.local_executor.LocalExecutor.

Why is this error populating in Airflow 2.0.2, and how should I fix it?

Elad Elad · Accepted Answer · 2021-05-06T07:13:47

The executor parameter was removed from SubDagOperator when the behavior was changed from backfil to the scheduler logic (PR).

But even before that it wasn't recommended to parallel tasks inside SubDagOperator which is why it was defaulted to SequentialExecutor (you can read about the reasons here)

Since you are using Airflow>2.0.0 note that SubDags are deprecated. You should move to TasksGroups.

SubDagOperator Invalid arguments were: **kwargs: {'executor': LocalExecutor(parallelism=32)}

1 Answers