MiNiFi to NiFi S2S load balancing does not work

0
votes

I've read the question below:

https://community.cloudera.com/t5/Support-Questions/MiNiFi-to-NiFi-S2S-load-balancing-does-not-work/td-p/310664

but in my case that I've implemented, site to site does not balance the incoming flows from MiNiFi. Here is my scenario: I have a NiFi cluster with 4 nodes. in the MiNiFi setup, I've set the remote process group URL to the nifi.remote.input.host that I've already set in the nifi.properties. Although Port 1026 is opened for receiving data in all nodes, when I use tcpdump command, I can see that only the host for the node which I've mentioned above are getting data and other nodes in the cluster do not receive any data.

Nifi version: 1.8.0

MiNiFi version: 0.5.0 java

comment:

I have a 4 node cluster. On the NiFi side, I stopped the node with the """Hostname""" for the nifi.input.remote.host, but on the MiNiFi side, I get the error below: 2021-02-02 10:03:18,105 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector Could not communicate with """Hostname""":1026 to determine which nodes exist in the remote NiFi cluster, due to java.net.ConnectException: Connection refused 2021-02-02 10:03:18,105 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@649b3ed4 Unable to refresh Remote Group's peers due to Unable to communicate with remote NiFi cluster in order to determine which nodes exist in the remote cluster 2021-02-02 10:03:23,155 ERROR [NiFi Site-to-Site Connection Pool Maintenance] o.a.n.r.io.socket.ssl.SSLSocketChannel org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel@4f8aa02f Failed to connect due to {} java.net.ConnectException: Connection refused

I also checked for the heavy load scenario, but when the load for """Hostname""" is high the load balancing does not occur too. I use SSL for communication, S2S With raw transport protocol, and the """Hostname""" is set for the URL in the remote process group on the MiNiFi side.

1
apache-nifiapache-minifi
Have you tried killing the node that receives data in order to see that the other nodes indeed cannot receive messages? - Adam Hunyadi
Thanks, Adam. I added a comment for the question. - Arash
Could you share, your remote process group configuration? - Pdeuxa

1 Answers

0
votes

You have a very, very old versions. Apache NiFi is now in version 1.12.1. There is a newer MiNiFi agent available from Cloudera. I am suspecting an old bug.

When in doubt reboot. Is there a gateway or router or firewall in the way, some networks can cause issues for network traffic.

Also you don't have to use S2S if you are having issues in your environment. I am partial to using ListenHTTP it scales well and more controllable.

https://pierrevillard.com/tag/apache-minifi/

EFM makes this easier to build and deploy. https://www.datainmotion.dev/2019/05/cloudera-edge-management-introduction.html