Airflow upstart script enters stop/waiting state immediately after start/running state

2
votes

I'm trying to run this upstart script from the official repo for airflow: https://github.com/apache/incubator-airflow/blob/master/scripts/upstart/airflow-webserver.conf

start on started networking
stop on (deconfiguring-networking or runlevel [016])

respawn
respawn limit 5 30

setuid airflow
setgid airflow

exec usr/local/bin/airflow webserver

When I run it this is the output:

$ sudo service airflow-webserver start
airflow-webserver start/running, process 7612

$ sudo service airflow-webserver status
airflow-webserver stop/waiting

But just doing exec /usr/local/bin/airflow webserver will run it perfectly without any hiccups, which is puzzling. Does anyone know why this is happening?

* You can get airflow with pip install airflow to test in case you need to see for yourselves.

2
ubuntuserviceupstartairflow
Does airflow still work as intended despite the message? <I have a home server that says that when php5-fpm is started. However, it still works as intended...> – J. Allan
@JefréN. unfortunately not. – m0meni
In that top 'codeblock', does it really say exec usr/local/bin/airflow webserver instead of exec /usr/local/bin/airflow webserver? – J. Allan
@JefréN. yes, but it actually doesn't matter for the upstart script. I've figured out the problem, which is that I neglected that upstart does not get environment variables from /etc/profile, and I needed to set them in the file. – m0meni

2 Answers

2
votes

Upstart runs in a clean environment, meaning that it won't use the variables in /etc/environment. This means you have to set AIRFLOW_HOME and AIRFLOW_CONFIG again.

description "Airflow webserver daemon"

start on started networking
stop on (deconfiguring-networking or runlevel [016])

respawn
respawn limit 5 30

setuid airflow
setgid airflow

# I omitted all of the below because I assumed 
# it would pick it up from the already defined env
env AIRFLOW_CONFIG=/path/to/airflow/airflow.cfg
env AIRFLOW_HOME=/path/to/airflow
export AIRFLOW_CONFIG
export AIRFLOW_HOME

exec usr/local/bin/airflow webserver
0
votes

Here is what I did in Ubuntu 16: (The init scripts are taken from the incubator-airflow git repo)

$ sudo cp incubator-airflow/scripts/systemd/*.service /etc/systemd/system/

In each service file, change user and group lines to the owner of directory, in my case - ubuntu and modify ExecStart:

User=ubuntu

Group=ubuntu

ExecStart=/usr/local/bin/airflow  webserver --pid $AIRFLOW_HOME/webserver.pid

keep in mind that ExecStart line is different in every service line.

Then:

$ sudo cp incubator-airflow/scripts/systemd/*.conf /etc/tmpfiles.d/
$ sudo mkdir /etc/sysconfig
$ sudo cp incubator-airflow/scripts/systemd/airflow /etc/sysconfig/airflow

Add the following lines to /etc/sysconfig/airflow: 

AIRFLOW_CONFIG=YOUR_airflow_home/airflow.cfg

AIRFLOW_HOME=YOUR_airflow_home

Finally:

sudo systemctl daemon-reload
sudo systemctl start airflow-webserver

(Or any other airflow service)

Good Luck!