apache-airflow systemd file using a conda env

Question

I'm trying to run apache-airflow on a Ubuntu 16.04 file, using systemd. I roughly followed this tutorial and installed/setup the following:

• Miniconda 2, 64-bit
• Installed gcc (sudo apt-get install gcc)
• Conda environment, using the yml file of the tutorial

Within the following conda environment:

• export AIRFLOW_HOME="/home/ubuntu/airflow"

When I test Airflow, everything works fine:

airflow webserver --port 8080

But whenever I try to launch airflow using a systemd file, it fails. The systemd file makes use of the conda environment, as far as I understand correctly. My systemd file looks as follows:

[Unit]
Description=Airflow webserver daemon

[Service]
User=ubuntu
Group=ubuntu
Type=simple
ExecStart=/home/ubuntu/miniconda2/envs/airflow-tutorial/bin/airflow webserver --port 8080
Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target

When I start/enable the systemd daemon, status returns the following error:

airflow-webserver.service - Airflow webserver daemon
   Loaded: loaded (/etc/systemd/system/airflow-webserver.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Thu 2018-09-13 08:59:00 UTC; 1s ago
  Process: 18410 ExecStart=/home/ubuntu/miniconda2/envs/airflow-tutorial/bin/airflow webserver --port 8080 (code=exited, status=1/FAILURE)
 Main PID: 18410 (code=exited, status=1/FAILURE)

Sep 13 08:59:00 ip-172-31-46-255 systemd[1]: airflow-webserver.service: Main process exited, code=exited, status=1/FAILURE
Sep 13 08:59:00 ip-172-31-46-255 systemd[1]: airflow-webserver.service: Unit entered failed state.
Sep 13 08:59:00 ip-172-31-46-255 systemd[1]: airflow-webserver.service: Failed with result 'exit-code'.

Help is highly appreciated!

Answer 1

The following is airflow-webserver.service that works for me with a virtual environment:

[Unit]
Description=Airflow webserver daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
PIDFile=/run/airflow/webserver.pid
EnvironmentFile=/etc/default/airflow
User=airflow
Group=airflow
Type=simple
ExecStart=/usr/bin/bash -c 'source /usr/local/airflow/venv/bin/activate ; airflow webserver --pid /run/airflow/webserver.pid'
Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Answer 2

One completetely different, but much easier solution which worked for me:

1. Add new sudo user with sudo rights, for example airflow
2. Install apache-airflow in environment as airflow user
3. Run airflow daemon using: airflow webserver --port 8080 -D

This runs airflow as daemon background process. You can just switch back to other user and continue creating dags, etc.

Answer 3

Answer given by @kaxil kinda worked for me! But with slight modifications.

I have my Airflow installed(Using Anaconda)and is at the location:

#-> whereis airflow
airflow: /root/anaconda3/bin/airflow

Adding the below line in airflow-webserver.service file solved the issue for me.

ExecStart=/usr/bin/bash -c 'source /root/anaconda3/bin/activate ; airflow webserver --pid /run/airflow/webserver.pid'

I got another error after it saying, Error: /run/airflow doesn't exist. Can't create pidfile. , Adding the below under [Service] solved that issue as well

RuntimeDirectory=airflow
RuntimeDirectoryMode=0775

Answer 4

This solution worked for me

• Create directory to keep airflow pid files

  sudo mkdir /run/airflow
  sudo chown -R airflow:airflow /run/airflow/
  

• My Airflow package is installed in Python3 virtual environment , virtual environment is created in this directory /opt/python3_6-virtual_envs/airflow_env/, so this path you may need to be modified according to your python3 virtual environment path

• Webserver systemd file

[Unit]
Description=Airflow Webserver daemon

[Service]
User=airflow
Type=simple
ExecStart= /opt/python3_6-virtual_envs/airflow_env/bin/python  /opt/python3_6-virtual_envs/airflow_env/bin/airflow webserver  -p 8080  --pid /run/airflow/webserver.pid
Environment="PATH=/opt/python3_6-virtual_envs/airflow_env/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/airflow/.local/bin:/home/airflow/bin"

[Install]
WantedBy=multi-user.target

• Scheduler systemd file

[Unit]
Description=Airflow Scheduler daemon

[Service]
User=airflow
Type=simple
ExecStart= /opt/python3_6-virtual_envs/airflow_env/bin/python  /opt/python3_6-virtual_envs/airflow_env/bin/airflow scheduler --pid /run/airflow/scheduler.pid
Environment="PATH=/opt/python3_6-virtual_envs/airflow_env/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/airflow/.local/bin:/home/airflow/bin"

[Install]
WantedBy=multi-user.target