Apache Airflow initdb command fails, due to syntax error

Question

I have created virtualenv for python3 using: virtualenv -p $(which python3) ENV

Then activate the source source /Users/myusername/ENV/bin/activate

Install the apache-airflow:

pip install apache-airflow

then which airflow yields /Users/myusername/ENV/bin/airflow But when I try to initdb using:

airflow initdb

I get below error:

    {db.py:350} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
WARNI [airflow.utils.log.logging_mixin.LoggingMixin] cryptography not found - values will not be stored encrypted.
ERROR [airflow.models.DagBag] Failed to import: /Library/Python/2.7/site-packages/airflow/example_dags/example_http_operator.py
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/airflow/models/__init__.py", line 413, in process_file
    m = imp.load_source(mod_name, filepath)
  File "/Library/Python/2.7/site-packages/airflow/example_dags/example_http_operator.py", line 27, in <module>
    from airflow.operators.http_operator import SimpleHttpOperator
  File "/Library/Python/2.7/site-packages/airflow/operators/http_operator.py", line 21, in <module>
    from airflow.hooks.http_hook import HttpHook
  File "/Library/Python/2.7/site-packages/airflow/hooks/http_hook.py", line 23, in <module>
    import tenacity
  File "/Library/Python/2.7/site-packages/tenacity/__init__.py", line 375, in <module>
    from tenacity.tornadoweb import TornadoRetrying
  File "/Library/Python/2.7/site-packages/tenacity/tornadoweb.py", line 24, in <module>
    from tornado import gen
  File "/Library/Python/2.7/site-packages/tornado-6.0.3-py2.7-macosx-10.14-intel.egg/tornado/gen.py", line 126
    def _value_from_stopiteration(e: Union[StopIteration, "Return"]) -> Any:
                                   ^
SyntaxError: invalid syntax
Done.
(ENV) ---------------------------------------------------------

Seems like example scripts use python 2.7 and it can't recognize the function definition syntax. Does apache-airflow package need to be fixed by the next release or I can do something to fix this? I tried fixing this: Use python2.7 instead of python3 then install airflow on default python 2.7 enabled on mac but this throws other errors like package "six" is not compatible.

do you have a PATH and PYTHON_PATH set up? for example: PATH=${PATH}:${AIRFLOW_HOME}:/usr/local/bin/ PYTHONPATH=${PYTHONPATH}:${AIRFLOW_HOME}:/usr/local/lib/python3.6:/usr/local/lib/python3.6/site-packages — kat
Thanks for point out the path var value, so I changed it now it throws other error: No module named 'werkzeug.wrappers.json'; 'werkzeug.wrappers' is not a package — aks_Nin
Seems like this is a known issue : github.com/puckel/docker-airflow/pull/393 — aks_Nin
@nightgaunt, not using docker airflow, I know the above issues happens for the docker version . but same happens when I run it in python virtual env — aks_Nin

kotartemiy kotartemiy · Accepted Answer · 2019-07-13T12:09:21

You need to turn off the example DAGs to be loaded in config file to solve this problem.

Anyway, it seems weird that airflow uses 2.7 Python when you told that it is installed into Python 3 virtual environment.

Apache Airflow initdb command fails, due to syntax error

1 Answers