Is it possible for NiFi to connect to itself with a Remote Process Group?

Question

I'm working on a project that heavily uses Apache NiFi v1.10.0. I'm getting tired of clicking through hundreds of process groups to apply small fixes that are essentially the same.

I've recently discovered Remote Process Groups and I was wondering if there is a way to connect NiFi instance to itself and implement DRY this way? I was thinking of implementing repeating components inside the root component and accessing them with remote access inside other process groups. Is this possible?

Right now I'm getting only SSLHandshakeException / PKIX path building failed

If there are other ways to implement DRY - please tell me.

Answer 1

@Alex. I feel your pain, in a previous role, they had a process group of 100s of flows, and would copy and paste the entire main group, turning into 1000s of flows. All copies with small modifications in random places.

Although I am advocate of programming this way to get a POC operational, I am a huge advocate of evaluating how to make the flows dynamic from the highest level. The process I use to do this is to go through DFDLC and versioning flow until, for example, I have 1 process group that can replace 2 by cleaning up the flow design differences between each other. We consider this part of optimizing the flow to reduce the total number of active processors too.

I highly recommend you do NOT use remote process groups within the same cluster. I also recommend you make common flows on the main canvas, and connect them with input/output ports when you need to move from a deeper process group back up to the main canvas. You will end up with a flow like this:

Answer 2

You can definitely do site-to-site to self, however it will be less performant because now you are taking local flow files and transferring them over a network connection to all the nodes in the cluster, even though some of them will go back to the same node they are on.

You could use NiFi Registry and created versioned flows that contain reusable functionality. Then you make the change once, commit it back to NiFi Registry, and then update the other instances of that versioned flow.

Answer 3

Bryan and Steven both offered good solutions. I will address the PKIX path building error you are encountering -- this indicates that NiFi is attempting to make an HTTPS connection to another service (likely in this case itself), and doesn't know how to verify the presented public certificate. The solution is to reference an SSLContextService configured with a truststore that contains the certificate. The Apache NiFi walkthroughs provide step-by-step instructions for performing these tasks.