Airflow PythonOperator task fail - TypeError: The key has to be a string

Question

Airflow newbie here, bear with me.. I don't understand why this simple task is failing:

def getCarJSON():
    dictCars= {'link': '/cars/acura', 'num': '1'}    
    with open('data/dictCars.json', 'w') as fp:
        json.dump(dictCars, fp)

This is simple dict stored on disk as JSON. Why do I get:

Broken DAG: [/home/user/airflow/dags/cars.py] Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/airflow/models/baseoperator.py", line 404, in init validate_key(task_id) File "/usr/local/lib/python3.8/dist-packages/airflow/utils/helpers.py", line 39, in validate_key raise TypeError("The key has to be a string") TypeError: The key has to be a string

I have usual data in DAG file:

# Set default args
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2021, 3, 23),
    'email': ['[email protected]'],
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 0,
    'retry_delay': timedelta(minutes=2)
}

schedule_interval = '30 09 * * *'

# Define DAG: Set ID and assign default args and schedule interval
dag = DAG(
    dag_id = 'get_cars',
    default_args = default_args,
    schedule_interval = schedule_interval
)


# Get cars dict
get_cars_json = PythonOperator(
    task_id=getCarJSON,
    python_callable=getCarJSON,
    dag=dag
)

All I want is to dump data on drive...

you are passing getCarJSON function as task_id instead of name of task (which should be string) — Anand Vidvat
@AnandVidvat oh man, what a retarded error :(. Thank you, maybe you can formulate this into answer so I can accept it. — Harvey

Anand Vidvat Anand Vidvat · Accepted Answer · 2021-03-30T03:04:31

From the broken dag message, it looks like while validating tast_id attribute, this error was raised.

In the following code, you have passed getCarJSON function to task_id, instead of name of the task, which would be of string type hence leading to this issue of TypeError

get_cars_json = PythonOperator(
    task_id='getCarJSON', # Name here was without quotations
    python_callable=getCarJSON,
    dag=dag
)

Airflow PythonOperator task fail - TypeError: The key has to be a string

1 Answers