Create Generic Nifi Flow for re usability purpose

Question

I have a nifi flow which extracts records from RDBMS (SQL Server) table and put these on HDFS in parquet format. This flow will run several times in a day to get delta records from source table. Now I need to replicate this same process for 100+ different source tables. So rather than creating 100+ nifi flows for each separate table, can I create main flow (let's say template) and pass parameter like source extract sql, target file name etc to main flow and repeat these steps for each source table. I am using Nifi Nifi 1.7. I am pretty close to parameterizing the flow itself.

Example- Parameterizing the parameters in the components used in the flow. Example- Filename = ${path}/${filename}

Not sure how to pass values to this flow from command line?

Thanks, Sid

you could create the first step of your process as ListenHTTP, in the body pass the tablename or even json with different parameters... then from command line with curl you can start new instance of your flow. — daggett

Ben Yaakobi Ben Yaakobi · Accepted Answer · 2019-02-17T10:09:04

You can't really pass parameters using a command line.. But you can use a listener(For example: ListenHTTP) and then use the FlowFile's attributes as parameters.

Another option is to use the variable registry to register those parameters, then use it as a NiFi template. Then, you can use NiFi's Rest API to deploy this template many times using different parameters.

Create Generic Nifi Flow for re usability purpose

1 Answers