Can't import custom Python macro into Airflow 2

1
votes

I'm upgrading an environment that hosts Airflow from Airflow 1.10.5 and Python 2.7 to Airflow 2.0.1 and Python 3.9.

I'm trying to get macros working again:

/home/ubuntu/airflow/plugins/ben.py contains:

from airflow.plugins_manager import AirflowPlugin

def ret_one():
    return 1

class AirflowPluginsTest(AirflowPlugin):
    name = "ret_one"
    macros = [ret_one]

I took an example DAG from GitHub https://github.com/apache/airflow/blob/master/airflow/example_dags/example_python_operator.py and added from airflow.macros.ben import ret_one but I get the error:

Broken DAG: [/home/ubuntu/airflow/dags/example_python_operator.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/ubuntu/airflow/dags/example_python_operator.py", line 6, in <module>
    from airflow.macros.ben import ret_one
ModuleNotFoundError: No module named 'airflow.macros.ben'

What am I doing wrong here? I've tried various combinations of the import (dropping the ben, etc).

My airflow.cfg contains:

[core]
...
plugins_folder = /home/ubuntu/airflow/plugins

and I restart all Airflow services (via supervisord) between changes. I can see that the ret_one plugin appears in the Airflow UI (Admin -> Plugins).

2
pythonpython-3.xairflow

2 Answers

1
votes

When you call the macro in a template, you must call it as the following:

{{ macros.PLUGIN_NAME.FUNCTION }}

In your case, the PLUGIN_NAME is "ret_one". So your macro call in a template would look like this:

{{ macros.ret_one.ret_one() }}

https://airflow.apache.org/docs/apache-airflow/stable/plugins.html

0
votes

The solution is simply to implement the macro independently of any Airflow libraries, and then import it as you would any class/function on the Python path.