Multiple flows with nifi

Question

We have multiple (50+) nifi flows that all do basically the same thing: pull some data out of a db, append some columns conver to parquet and upload to hdfs. They differ only in details such as the sql query to run or the location in hdfs that they land.

The question is how to factor these common nifi flows out such that any change made to the common flow automatically applies to all all derived flows. E.g if i want to add an extra step to also publish the data to Kafka I want to make this once and have it automatically apply to all 50 flows.

We’ve tried to get this working with nifi registry, however it seems like an imperfect fit. Essentially the issue is that nifi registry seems to work well for updating a flow in one environment (say wat) and then autmatically updating it in another environment (say prod). It seems less suited for updating multiple flows in the same environment with one specific example bing that it will reset the name of each flow to be the template name every time we redeploy meaning that al flows end up with the same name!

Does anyone know how one is supposed to manage a situation like ours asi guess it must be pretty common.

Answer 1

Apache NiFi has ProcessorGroups. As the name itself suggests, the processor groups are there to group together a set of processors' and their pipeline that does similar task.

So for your case what you can do is, you can refactor the flow by moving the common flow which can be reused with different pipelines to a separate processor group with an input port. Connect the outside flow that depends on this reusable flow by connecting to the input port of the reusable processor group. Depending on your requirement you can create an output port as well in this processor group and connect it with the outside flow.

Attaching a sample:

For the sake of explaining, I have made a mock flow so ignore the Processor types that are used, but rather see the name I had given to those processors.

The following screenshots show that I read from two different sources and individually connect them to two different processors that does the source specific changes to those processors

Then I connect these two flows to the input port of a processor group that has the reusable flow inside. So ultimately the two different flows shown in the above screenshot gets to work with a common reusable flow.

Showing what's inside the reusable flow:

Finally the output port output to outside connects the reusable flow to the outside component Write to somewehere I hope this helps you with refactoring your complex flows. Feel free to get back, if you have any queries.