Cloud Composer scheduler error when adding first dag

Question

I have a DAG running on my local Airflow. I lunched Cloud Composer and wanted to move my DAGs there. When added the first DAG file the scheduler shows this error:

Traceback (most recent call last): File "/usr/local/lib/airflow/airflow/models.py", line 363, in process_file m = imp.load_source(mod_name, filepath) File "/usr/local/lib/python3.6/imp.py", line 172, in load_source module = _load(spec) File "", line 684, in _load File "", line 665, in _load_unlocked File "", line 674, in exec_module File "", line 781, in get_code File "", line 741, in source_to_code File "", line 219, in _call_with_frames_removed File "/home/airflow/gcs/dags/testdag.py", line 95 'start_date': datetime(2018, 12, 05),

This is line 95:

args = {
    'owner': 'Airflow',
    'start_date': datetime(2018, 12, 05),
    'retries': 5,
    'retry_delay': timedelta(minutes=5)
}

Never encountered this error before.

Are there log lines before or after the one you pasted that might be relevant? Since there's not much to go off of in that traceback, I have to ask the obvious question: are datetime and timedelta properly imported in your DAG definition? — Wilson
@Wilson This dag works perfectly in my Airflow installed on EC2 server for 4 months. I wanted to move my dags to cloud composer. I moved the dag and it generated this error. It doesn't indicate a problem with packages. I never saw this error before nor found info on google so I don't know how to debug it. This error doesn't appear on regular Airflow. — Luis

Deep Nirmal Deep Nirmal · Accepted Answer · 2018-12-04T06:29:08

If you want to run the DAG's and do catchup from the historical dates then you give the past dates as start_date

Try giving

from datetime import datetime, timedelta

args = {
    'owner': 'Airflow',
    'provide_context': True,
    'depends_on_past': False,
    'start_date': datetime.combine(datetime.today(),datetime.min.time()),
    'retries': 5,
    'retry_delay': timedelta(minutes=5)
}

Cloud Composer scheduler error when adding first dag

2 Answers