how to pass parameters to PapermillOperator to run job on airflow?

Question

While running airflow job using PapermillOperator dag execution getting failed.

I am having problem in passing parameters to PapermillOperator.

I opened papermill_operator.py (packages/airflow/operators/papermill_operator.py) and hardcoded one line to specify papameters

def execute(self, context):
        for i in range(len(self.inlets)):
            pm.execute_notebook(self.inlets[i].location, 
                                self.outlets[i].location,
                                parameters = dict(msgs="hello")
                                progress_bar=False, report_mode=True)

Then it's working

while the original code is

def execute(self, context):
        for i in range(len(self.inlets)):
            pm.execute_notebook(self.inlets[i].location, 
                                self.outlets[i].location,
                                parameters=self.inlets[i].parameters,
                                progress_bar=False, report_mode=True)

Tried another solution https://github.com/nteract/papermill/issues/324#issuecomment-472446375 it's working fine

My DAG code is

import airflow

from airflow.models import DAG
from airflow.operators.papermill_operator import PapermillOperator

from datetime import timedelta

args = {
    'owner': 'Airflow',
    'start_date': airflow.utils.dates.days_ago(2),

}

dag = DAG(
    dag_id='9', default_args=args,
    schedule_interval='@once',
    dagrun_timeout=timedelta(minutes=10))

run_this = PapermillOperator(
    task_id="1",
    dag=dag,
    input_nb="/home/exa00112/abc.ipynb",
    output_nb="/home/exa00112/umesh.ipynb",
    parameters = dict("msgs" = "hello")
)

run_this

[2019-09-10 20:36:48,806] {logging_mixin.py:95} INFO - [2019-09-10 > > > 20:36:48,806] {datasets.py:62} INFO - parameters [2019-09-10 20:36:48,806] {init.py:1580} ERROR - Can't compile non > template nodes Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/airflow/models/init.py", line 1441, in _run_raw_task result = task_copy.execute(context=context) File "/usr/local/lib/python3.5/dist-packages/airflow/operators/papermill_operator.py", line 63, in execute parameters=self.inlets[i].parameters, File "/usr/local/lib/python3.5/dist-packages/airflow/lineage/datasets.py", line 66, in getattr return env.from_string(self._data.get(attr)).render(**self.context) File "/home/exa00112/.local/lib/python3.5/site-packages/jinja2/environment.py", line 880, in from_string return cls.from_code(self, self.compile(source), globals, None) File "/home/exa00112/.local/lib/python3.5/site-packages/jinja2/environment.py", line 581, in compile defer_init=defer_init) File "/home/exa00112/.local/lib/python3.5/site-packages/jinja2/environment.py", line 543, in _generate optimized=self.optimized) File "/home/exa00112/.local/lib/python3.5/site-packages/jinja2/compiler.py", line 78, in generate raise TypeError('Can\'t compile non template nodes') TypeError: Can't compile non template nodes [2019-09-10 20:36:48,808] {init.py:1611} INFO - Marking task as FAILED.

playermany2 playermany2 · Accepted Answer · 2019-10-30T22:22:33

Appears to be a known issue for the Papermill Operator passing an invalid parameters data structure to Papermill (passing a string parameters dict when papermill looks for a dict) to

https://issues.apache.org/jira/browse/AIRFLOW-5774

Not sure on when it will be fixed, as it looks like it's a duplicate issue, so hard to track

how to pass parameters to PapermillOperator to run job on airflow?

1 Answers