So it looks like my install of apache airflow on a Google Compute Engine instance broke down. Everything was working great and then two days ago all the DAG runs show up stuck in a running state. I am using LocalExecutioner.
When I try to look at the log I get this error:
* Log file isn't local. * Fetching here: http://:8793/log/collector/aa_main_combined_collector/2017-12-15T09:00:00 *** Failed to fetch log file from worker.
I didn't touch a setting anywhere. I looked through all the config files and I scanned the logs and I see this error
[2017-12-16 20:08:42,558] {jobs.py:355} DagFileProcessor0 ERROR - Got an exception! Propagating... Traceback (most recent call last): File "/usr/local/lib/python3.4/dist-packages/airflow/jobs.py", line 347, in helper pickle_dags) File "/usr/local/lib/python3.4/dist-packages/airflow/utils/db.py", line 53, in wrapper result = func(*args, **kwargs) File "/usr/local/lib/python3.4/dist-packages/airflow/jobs.py", line 1584, in process_file self._process_dags(dagbag, dags, ti_keys_to_schedule) File "/usr/local/lib/python3.4/dist-packages/airflow/jobs.py", line 1173, in _process_dags dag_run = self.create_dag_run(dag) File "/usr/local/lib/python3.4/dist-packages/airflow/utils/db.py", line 53, in wrapper result = func(*args, **kwargs) File "/usr/local/lib/python3.4/dist-packages/airflow/jobs.py", line 763, in create_dag_run last_scheduled_run = qry.scalar() File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2843, in scalar ret = self.one() File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2814, in one ret = self.one_or_none() File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2784, in one_or_none ret = list(self) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2855, in iter return self._execute_and_instances(context) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2878, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 945, in execute return meth(self, multiparams, params) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement compiled_sql, distilled_params File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context context) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception util.reraise(*exc_info) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/util/compat.py", line 187, in reraise raise value File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context context) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/default.py", line 470, in do_execute cursor.execute(statement, parameters) File "/usr/local/lib/python3.4/dist-packages/airflow/bin/cli.py", line 69, in sigint_handler sys.exit(0) SystemExit: 0
Any thoughts out there?
