Airflow Failed To Import Plugin for a plugin folder with custom name

Question

I have to work on a shared Airflow 1.10 project, so I have cloned the repository and the structure is as follow:

airflow
├── airflow.cfg
├── airflow.db
├── dags
│   ├── dags_here.py
│      
├── dags_conf
│   ├── some settings for dags
│   │   └── settings.py
│   │   ├── __init__.py
│   │   ├── settings.py
│   │   └── todo.txt
|   |__ init.py
|
├── utilities
│   ├── __init__.py
│   └── some_name
│       ├── airflow
│       │   └── hooks.py
            └── sensors.py
            └── plugins.py
│       ├── common
│       │   ├some_other_code_files.py
│       ├── __init__.py
|
├── README.md
├── unittests.cfg
└── venv
    ├── lib64 -> lib
    └── pyvenv.cfg

But when I try to list dags I face this error:

plugins_manager.py:225} ERROR - No module named 'common' Traceback (most recent call last): File "/home/xxx/venv/lib/python3.8/site-packages/airflow/plugins_manager.py", line 218, in m = imp.load_source(namespace, filepath) File "/usr/lib/python3.8/imp.py", line 171, in load_source module = _load(spec) File "", line 702, in _load File "", line 671, in _load_unlocked File "", line 783, in exec_module File "", line 219, in _call_with_frames_removed File "/home/xxx/airflow/utilities/xxx/common/insert_obj.py", line 3, in from common.xxxxx import Class_name ModuleNotFoundError: No module named 'common' [2021-05-30 23:13:05,933] {plugins_manager.py:226} ERROR - Failed to import plugin /home/xxxx/airflow/utilities/xxxxx/common/dag_file.py

I have tried below configs inside may airflow.cfg but always the same error: plugins_folder=/home/xxx/airflow/utilities

Adding a parent directory inside Airflow folder called plugins with an __init__.py file and copying utilities inside that: plugins_folder=/home/xxx/airflow/plugins

could you spread the folder structure a bit along the y axis? The 'dags_conf' and 'utilities' directories are overlapping the 'dags' sub-structure, making it hard to understand. Just moving the whole 'dags' node to the right would help — Nate T
What do you mean by 'when I try to list'? What method / command are you using? — Nate T
There are 3 main folders: Dags/dags_config/utilities. I needed to use sub directories inside the utilities :) — Paniz Asghari

Bruno Campos Bruno Campos · Accepted Answer · 2021-07-09T09:54:14

confirm how the environment variables that Airflow uses or the airflow.cfg settings are configured.

environment variables

AIRFLOW_HOME=<YOUR_PROJ>
AIRFLOW__CORE__DAGS_FOLDER=$AIRFLOW_HOME/dags
AIRFLOW__CORE__PLUGINS_FOLDER=$AIRFLOW_HOME/utilities

airflow.cfg

dags_folder = {AIRFLOW_HOME}/dags
plugins_folder = {AIRFLOW_HOME}/utilities

Also see what is configured in PYTHONPATH:

import sys
from pprint import print

pprint(sys.path)

# [
#  '/ssddisk/project_name/data-pipeline/batch/dags',
#  '/ssddisk/project_name/data-pipeline/batch/config',
#  '/ssddisk/project_name/data-pipeline/batch/utilities',
#  ...
#  ]

See documentation for more details: https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html#creating-a-custom-operator

Airflow Failed To Import Plugin for a plugin folder with custom name

1 Answers