Configuring fs_default connection in Airflow

Question

I would like to configure the fs_default connection of Airflow, basically to make sure paths will always be resolved from the same starting point (the root of my directory/repository).

I set up my airflow home with export AIRFLOW_HOME=./airflow_home. Therefore I have a directory airflow_home at the root of my project.

And then I put this in my airflow.cfg:

[core]

# The folder where your airflow pipelines live, most likely a
# subfolder in a code repository. This path must be absolute.
dags_folder = ./airflow_home/dags

fs_default_conn = ./airflow_home/../

# The folder where airflow should store its log files
# This path must be absolute
base_log_folder = ./airflow_home/logs

I did a airflow resetdb followed by airflow initdb, and then started the webserver (and the scheduler). I went into Admin > Connection to check if my changes were taken into account, but apparently not. The fs_defaultconnection still has this in the Extras field:

{"path": "/"}

Any idea how this should be done?

kaxil kaxil · Accepted Answer · 2020-03-26T16:39:33

You will need to create Connection using Airflow UI / CLI or by an environment variable. It won't work by adding it to airflow.cfg file.

Check https://airflow.apache.org/docs/stable/howto/connection/index.html#creating-a-connection-with-the-ui on how to create a connection.

Configuring fs_default connection in Airflow

1 Answers