I'm new to Apache-Airflow and have Problems setting it up correctly. I'm using Python 3.6.6 in the Linux subsystem on a Windows 10 machine.
leolei@PC-W10E277:~$ cat /etc/issue
Ubuntu 18.04.1 LTS \n \l
leolei@PC-W10E277:~$ cat /proc/version
Linux version 4.4.0-17134-Microsoft ([email protected]) (gcc version 5.4.0 (GCC) ) #345-Microsoft Wed Sep 19 17:47:00 PST 2018
Webserver and scheduler are running without errors and without configuration.
I installed everything I found while researching (see pip freeze and package listing at the bottom).
I'm sure it's easy to fix, but noone else has this exact problem.
This happens when I'm trying to start the worker via "airflow worker" without configuration: It says it lost connection an then after some seconds tries again and loses it and so on.
Thanks in advance for every hint on how I get this to work, I already wasted two work days on this.
Traceback:
leolei@PC-W10E277:~$ airflow worker
[2018-11-29 16:14:09,385] {__init__.py:51} INFO - Using executor SequentialExecutor
-------------- celery@PC-W10E277 v4.1.1 (latentcall)
---- **** -----
--- * *** * -- Linux-4.4.0-17134-Microsoft-x86_64-with-Ubuntu-18.04-bionic 2018-11-29 16:14:09
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: airflow.executors.celery_executor:0x7f5856d0d438
- ** ---------- .> transport: sqla+mysql://airflow:airflow@localhost:3306/airflow
- ** ---------- .> results: mysql://airflow:**@localhost:3306/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> default exchange=default(direct) key=default
[2018-11-29 16:14:10,207: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/home/leolei/.local/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1124, in _do_get
return self._pool.get(wait, self._timeout)
File "/home/leolei/.local/lib/python3.6/site-packages/sqlalchemy/util/queue.py", line 145, in get
raise Empty
sqlalchemy.util.queue.Empty
During handling of the above exception, another exception occurred:
[...]
pip freeze:
leolei@PC-W10E277:~$ pip freeze
adal==1.2.0
alembic==0.8.10
amqp==2.3.2
apache-airflow==1.10.0
asn1crypto==0.24.0
attrs==17.4.0
Automat==0.6.0
Babel==2.6.0
billiard==3.5.0.4
bleach==2.1.2
blinker==1.4
cachetools==3.0.0
celery==4.1.1
certifi==2018.1.18
chardet==3.0.4
click==6.7
cloud-init==18.4
colorama==0.4.0
command-not-found==0.3
configobj==5.0.6
configparser==3.5.0
constantly==15.1.0
croniter==0.3.26
cryptography==2.1.4
defusedxml==0.5.0
dill==0.2.8.2
distro-info==0.18
docutils==0.14
Flask==0.12.4
Flask-Admin==1.4.1
Flask-AppBuilder==1.12.2
Flask-Babel==0.12.2
Flask-Caching==1.3.3
Flask-Login==0.2.11
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.3.2
flask-swagger==0.2.13
Flask-WTF==0.14.2
flower==0.9.2
funcsigs==1.0.0
future==0.16.0
gevent==1.3.7
gitdb2==2.0.5
GitPython==2.1.11
google-auth==1.6.1
greenlet==0.4.15
gunicorn==19.9.0
html5lib==1.0.1
httplib2==0.9.2
hyperlink==17.3.1
idna==2.6
incremental==16.10.1
iso8601==0.1.12
itsdangerous==1.1.0
Jinja2==2.8.1
jsonpatch==1.16
jsonpointer==1.10
jsonschema==2.6.0
kombu==4.2.1
kubernetes==8.0.0
language-selector==0.1
librabbitmq==2.0.0
lockfile==0.12.2
lxml==3.8.0
Mako==1.0.7
Markdown==2.6.11
MarkupSafe==1.0
monotonic==1.5
mysql-connector-python==8.0.13
mysqlclient==1.3.10
numpy==1.15.4
oauthlib==2.0.6
ordereddict==1.1
PAM==0.4.2
pandas==0.23.4
pendulum==1.4.4
protobuf==3.6.1
psutil==4.4.2
pyasn1==0.4.2
pyasn1-modules==0.2.1
Pygments==2.2.0
pygobject==3.26.1
PyJWT==1.5.3
pyOpenSSL==17.5.0
pyserial==3.4
python-apt==1.6.3
python-daemon==2.1.2
python-dateutil==2.7.5
python-debian==0.1.32
python-editor==1.0.3
python-nvd3==0.15.0
python-slugify==1.2.6
python3-openid==3.1.0
pytz==2018.7
pytzdata==2018.7
PyYAML==3.12
redis==3.0.1
requests==2.20.1
requests-oauthlib==1.0.0
requests-unixsocket==0.1.5
rsa==4.0
service-identity==16.0.0
setproctitle==1.1.10
six==1.11.0
smmap2==2.0.5
SQLAlchemy==1.1.18
ssh-import-id==5.7
systemd-python==234
tabulate==0.7.7
tenacity==4.8.0
thrift==0.11.0
tornado==5.1.1
Twisted==17.9.0
tzlocal==1.5.1
ufw==0.35
unattended-upgrades==0.1
unicodecsv==0.14.1
Unidecode==1.0.22
urllib3==1.22
vine==1.1.4
webencodings==0.5.1
websocket-client==0.54.0
Werkzeug==0.14.1
WTForms==2.2.1
zope.deprecation==4.3.0
zope.interface==4.3.2
installed sql packages:
leolei@PC-W10E277:~$ apt list --installed *sql*
Listing... Done
libmysqlclient20/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 amd64 [installed,automatic]
libsqlite3-0/bionic,now 3.22.0-1 amd64 [installed]
mysql-client/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 all [installed]
mysql-client-5.7/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 amd64 [installed,automatic]
mysql-client-core-5.7/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 amd64 [installed,automatic]
mysql-common/bionic,now 5.8+1.0.4 all [installed]
mysql-server/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 all [installed]
mysql-server-5.7/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 amd64 [installed,automatic]
mysql-server-core-5.7/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 amd64 [installed,automatic]
python3-mysqldb/bionic,now 1.3.10-1build1 amd64 [installed]
python3-sqlalchemy/bionic,now 1.1.11+ds1-1ubuntu1 all [installed]
