Trying to run Apache/NIFI on Zookeeper from Confluent

Question

I am trying to run Apache/NIFI on confluent-zookeeper. NIFI ver 1.11.3 installed in /opt/nifi by unpacking tar container, confluent is community edition, ver 5.3. installed using confluent repo https://packages.confluent.io/rpm/5.3.

So NIFI works using integrated zookeper, NIFI works if I download zookeeper separatly from Apache/zookeeper site. Confluent Kafka also works with separate zookeeper and NIFI-integrated. BUT I cannot make it works using zookeeper from confluent.

In logs I see only one warning which is:

WARN Received packet at server of unknown type 15 (org.apache.zookeeper.server.ZooKeeperServer)

My config file for all three zookeepers are the same:

tickTime=2000
dataDir=/var/lib/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
server.1=myhost1:2888:3888
server.2=myhost2:2888:3888
server.3=myhost3:2888:3888
autopurge.snapRetainCount=3
autopurge.purgeInterval=24

I do not think that Confluent really changed smth in their zookeeper. What could be the reason of this error?

Answer 1

As @BryanBende said:

NiFi 1.11.x (In our particular case, 1.11.4) requires ZK 3.5, please confirm the version of ZK used in Confluent platform, IF ITS 3.4 THEN ITS NOT GOING TO WORK – Bryan Bende Mar 27 at 14:35

The typical error you will see in zookeeper log:

Oct 08 17:22:23 some-pro-zk03 zookeeper-server-start[14136]: [2020-10-08 17:22:23,275] INFO Accepted socket connection from /10.10.10.1:53794 (org.apache.zookeeper.server.NIOServerCnxnFactory)

Oct 08 17:22:23 some-pro-zk03 zookeeper-server-start[14136]: [2020-10-08 17:22:23,275] INFO Refusing session request for client /10.10.10.1:53794 as it hasseen zxid 0x400000000 our last zxid is 0x300000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)

Oct 08 17:22:23 some-pro-zk03 zookeeper-server-start[14136]: [2020-10-08 17:22:23,275] INFO Closed socket connection for client /10.159.164.93:53794 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)

The typical error you will see in client log:

2020-10-07 16:00:09,112 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background operation retry gave uporg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLossat org.apache.zookeeper.KeeperException.create(KeeperException.java:102)at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:862)at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:990)at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943)at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)

This takes a full day of work (reading logs!!) to find the error. So, check your zookeeper version:

FOR Zookeeper 3.5+

echo srvr | nc localhost 2181

FOR Zookeeper 3.5<

echo stats | nc localost 2181

Also you can use telnet

FOR Zookeeper 3.5+

telnet localhost 2181
srvr

FOR Zookeeper 3.5<

telnet localhost 2181
stats