Running Nifi cluster on a single host

0
votes

Is it possible to run 3 node Nifi cluster on single host/machine (in my case a virtual one)? And if so what needs to be done for it to work properly?

I've been having a lot of problems trying to set unconflicting ports and make all 3 instances of Nifi run on single host. Most recent problem is that Nifi node is unable to connect to itself.

I know it does not make any sense to run multiple nodes on single host, but it's for testing and development purposes.

Thanks a lot for any advice!

1
testingnetworkingportapache-nifidevelopment-environment
Many of the committers run 2 or 3 node instances on their laptop to test things so it is possible, typically i just go through each nifi.properties and in each one increment all the ports numbers from the previous one, so web port 8080, 8081, 8082, etc - Bryan Bende
I would absolutely not recommend this. If you need multiple nodes within your single machine, you need to use ambari/vagrant and spin up a few virtual nodes... trying to put 3 nifi on a single ip is a giant waste of time as a single node is capable of anything you need to do to test or dev. - steven-matison
Bryan is correct and I think the comment arguing against this doesn't appreciate the entire ecosystem. For using NiFi, a single node is often sufficient in development or test environments, but there are many scenarios where the clustering communication or RPG behavior is what is being tested, and a cluster with multiple nodes running on the same machine certainly makes sense for this case. - Andy

1 Answers

2
votes

There is a walkthrough in the Apache NiFi documentation which goes step-by-step through the process of deploying multiple nodes in a cluster all running on a single physical/virtual machine.

There are also best practices and port configuration values to take into consideration.

Without further details on the specific errors you are encountering, ensure the following:

  1. The port for UI/API, cluster communications, site-to-site, and load-balanced connections are all unique for each node.
  2. Ensure the ZooKeeper (embedded or remote) connections are correct for every node.
  3. If TLS is being used, ensure that all nodes have an identity established and the proper permissions are assigned, and these permissions are replicated to every node.