Zookeeper java.net.NoRouteToHostException when connecting

Question

Apologies in advance for the data dump here. I have two servers with Zookeeper installed. I want to run them as an ensemble. Here is what's in my zoo.cfg for both.

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zookeeper/data
server.1=solr1:2888:3888
server.2=solr2:2888:3888
clientPort=2181

Both servers have a myid file. Their names are in the /etc/hosts files, so they can definitely see each other. When I start either of them by using zkServer.sh, it says zookeeper is started. However, when I run "zkServer.sh status", I see this message:

Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.

But the zookeeper java process seems to be running and using netstat shows tcp ports 3888 and 2181 as listening. When I try to connect to the server using "sh zkCli.sh -server localhost:2181", I see a bunch of environment info and then a whole bunch of these log entries:

2014-02-03 15:39:00,623 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@966] - Opening socket connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-02-03 15:39:00,646 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@849] - Socket            connection established to localhost.localdomain/127.0.0.1:2181, initiating session
JLine support is enable
2014-02-03 15:39:00,662 [myid:] - INFO  [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@1085] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect

If I try to connect to the other server (sh zkCli.sh -server solr1:2181), I see a bunch of these:

2014-02-03 15:40:42,390 [myid:] - WARN  [main-SendThread(solr1:2181):ClientCnxn$SendThread@1089] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.NoRouteToHostException: No route to host

Zookeeper seems to be running, despite zkServer.sh status saying otherwise. But does anyone have any insight into my connection issues? OS is RHEL 6.4, 64 bit.

That was indeed the problem. I disabled iptables (this is only a dev environment) and that fixed it. I'm using AWS and was using the security groups feature to control open ports. Thanks! — json4639

yunpeng liu yunpeng liu · Accepted Answer · 2016-04-11T06:13:18

aha, I just resolve the same problem now , and the OS is Centos7. I check all the configure files but can't help. finally, I modified the iptable to add such port: 2181,2888,3888 ,then restart the zookeeper server, it works . one of the cmd is like: firewall-cmd --zone=public --add-port=2181/udp --add-port=2181/tcp --permanent then reloat : firewall-cmd --reload of course,you can shutdown the iptables and disable the iptables to load at start the OS to achieve this. other more ,you should check whether there is a "myid" file in your data path:dataDir=/opt/zookeeper/data and its owner should be "username:groupUsername"; May it will help you too.

Zookeeper java.net.NoRouteToHostException when connecting

3 Answers