ZooKeeper start successfully but doesn't work

Question

I am trying to deploy a zookeeper ensemble in fully distributed mode using three nodes. After starting the server no entry comes under jps. On giving "zkServer.sh status" the output is:

    JMX enabled by default
    Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
    Error contacting service. It is probably not running

On giving the command "zkCli.sh -server zks1:2181" is:

Connecting to zks1:2181
2015-05-24 23:02:24,602 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-05-24 23:02:24,605 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=mtech2-OptiPlex-7010
2015-05-24 23:02:24,606 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.7.0_79
2015-05-24 23:02:24,607 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2015-05-24 23:02:24,607 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
2015-05-24 23:02:24,608 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/usr/local/zookeeper/bin/../build/classes:/usr/local/zookeeper/bin/../build/lib/*.jar:/usr/local/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/local/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/local/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/local/zookeeper/bin/../zookeeper-3.4.6.jar:/usr/local/zookeeper/bin/../src/java/lib/*.jar:/usr/local/zookeeper/bin/../conf:
2015-05-24 23:02:24,608 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2015-05-24 23:02:24,608 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2015-05-24 23:02:24,608 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2015-05-24 23:02:24,608 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2015-05-24 23:02:24,608 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2015-05-24 23:02:24,608 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.75
2015-05-24 23:02:24,608 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=hduser
2015-05-24 23:02:24,608 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/home/hduser
2015-05-24 23:02:24,608 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/usr/local/zookeeper
2015-05-24 23:02:24,609 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=zks1:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@634fd55e
Welcome to ZooKeeper!
2015-05-24 23:02:24,741 [myid:] - INFO  [main-SendThread(zks1:2181):ClientCnxn$SendThread@975] - Opening socket connection to server zks1/192.168.4.60:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2015-05-24 23:02:24,755 [myid:] - WARN  [main-SendThread(zks1:2181):ClientCnxn$SendThread@1102] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[zk: zks1:2181(CONNECTING) 0]

The /etc/hosts file is as follows for all the nodes:

    192.168.4.60    zks1
    192.168.4.61    zks2
    192.168.4.66    zks3

The zoo.cfg file is as follows:

    dataDir=/var/zookeeper/                                                                    
    clientPort=2181
    initLimit=5
    syncLimit=2
    server.server1=zks1:2888:3888                                
    server.server2=zks2:2888:3888
    server.server3=zks3:2888:3888

I had created the directory /var/zookeeper on all the nodes and changed the ownership to hduser:hadoop.

The exported environment variables are:

    export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
    export ZK_HOME=/usr/local/zookeeper
    export PATH=$PATH:$ZK_HOME/bin

The myid's in data directory(/var/zookeeper/) are server1, server2 and server3 respectively.

Please point out the problem.

Answer 1

Replacing server.server1 by server.1 and accordingly modifying myid file for each node did the trick.

Answer 2

I encountered same problem,too. In my case problem is about zookeeper locations configuration is not same for each node so zookeeper can not provide Quorum and mentioned nodes can not be part of cluster.

Please be sure server definition for each node is same.

server.server1=zks1:2888:3888                                
server.server2=zks2:2888:3888
server.server3=zks3:2888:3888

Answer 3

In first/top stack trace there is problem with myid value.

"2015-05-24 23:02:24,602 [myid:] - INFO [main:Environment@100] " in above line with myid:* here should be an integer/long .So probably you have not created your myid file inside dataDir folder or you have not entered any long/int value.One more thing myid should be of type file.

In second issue,in server.server1=machine:port1:port2 since server1 is not a long so NumberFormatException will be thrown by org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties() method. so server.1 should be a number,a unique one in ensemble and the same should be in myid file.