How to submit parameters to a Python program in Airflow?

Question

We have a front-end server that is executing dags directly with the dag API (DagBag(), get_dag() and then dag_run())

Dags run fine, the problem is, that we could not find a way to execute such dags with specific arguments.

The closest solution was to use the Variable API, which uses set() and get() methods, but these variables are global and might have conflicts when working in concurrent operations that might use same variable names.

How could we run a dag and set arguments available to its execution? We are mostly using PythonOperator.

Edit 1: Our program is a Python Django front end server. So, we are speaking with Airflow through another Python program. This means we trigger dags through Python, hence, using DagBag.get_dag() to retrieve information from airflow service. run_dag() does not have a way to pass direct parameters though

Answer 1

If you use trigger_dag_run (through command line or from another dag) to trigger a dag, you can pass any json as payload.

Another option would be to store the argument list on a file, and store that file location as a Variable. The DAG can then pass this file location to python operator and the operator can then handle reading that file and parsing arguments from it.

If both these solutions don't work for your use case, giving more details about your dag and kind of arguments might help.