ERROR Invalid config, exiting abnormally in Kafka cluster

Question

I have three different nodes that every one has docker with Ubuntu on it. I can run zookeeper server on docker locally (I mean without any configuration for cluster), but I want to make kafka cluster with these three nodes; In fact, I installed docker on each node with loading Ubuntu with on them. I configure "zookeeper.properties" in docker environment for "150.20.11.157" like this:

dataDir=/tmp/zookeeper/data
tickTime=2000
initLimit=10
syncLimit=5
server.1=0.0.0.0:2888:3888
server.2=150.20.11.134:2888:3888
server.3=150.20.11.137:2888:3888
clientPort=2186

For node 150.20.11.134, "zookeeper.properties" file in docker environment is like this:

dataDir=/tmp/zookeeper/data
tickTime=2000
initLimit=10
syncLimit=5
server.1=150.20.11.157:2888:3888
server.2=0.0.0.0:2888:3888
server.3=150.20.11.137:2888:3888
clientPort=2186

For node 150.20.11.137, "zookeeper.properties" file in docker environment is like this:

dataDir=/tmp/zookeeper/data
tickTime=2000
initLimit=10
syncLimit=5
server.1=150.20.11.157:2888:3888
server.2=150.20.11.134:2888:3888
server.3=0.0.0.0:2888:3888
clientPort=2186

Also, I setup "server.properties" like this, for node 150.20.11.157:

broker.id=0
port=9092
listeners = PLAINTEXT://150.20.11.157:9092
log.dirs=/tmp/kafka-logs 
zookeeper.connect=150.20.11.157:2186,150.20.11.134:2186,
150.20.11.137:2186

"server.properties" for node 150.20.11.134 is:

broker.id=1
port=9092
listeners = PLAINTEXT://150.20.11.134:9092
log.dirs=/tmp/kafka-logs 
zookeeper.connect=150.20.11.157:2186,150.20.11.134:2186,
150.20.11.137:2186

"server.properties" for node 150.20.11.137 is:

broker.id=2
port=9092
listeners = PLAINTEXT://150.20.11.137:9092
log.dirs=/tmp/kafka-logs 
zookeeper.connect=150.20.11.157:2186,150.20.11.134:2186,
150.20.11.137:2186

The problem is when I run zookeeper server on docker of each node. I get this error:

[2019-01-16 12:45:54,588] INFO Reading configuration from: ./config/zookeeper.properties 
(org.apache.zookeeper.server.quorum.QuorumPeerConfig)

[2019-01-16 12:45:54,601] INFO Resolved hostname: 172.28.10.137 to address: /172.28.10.137 (org.apache.zookeeper.server.quorum.QuorumPeer)

[2019-01-16 12:45:54,603] INFO Resolved hostname: 0.0.0.0 to address: /0.0.0.0 (org.apache.zookeeper.server.quorum.QuorumPeer)

[2019-01-16 12:45:54,603] INFO Resolved hostname: 172.28.10.157 to address: /172.28.10.157 (org.apache.zookeeper.server.quorum.QuorumPeer)

[2019-01-16 12:45:54,603] INFO Defaulting to majority quorums (org.apache.zookeeper.server.quorum.QuorumPeerConfig)

[2019-01-16 12:45:54,604] ERROR Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)

org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing ./config/zookeeper.properties
    at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:156)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:104)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Caused by: java.lang.IllegalArgumentException: /tmp/zookeeper/data/myid file is missing
    at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:408)
    at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:152)    
... 2 more

Invalid config, exiting abnormally

Would you please tell me how to have a Kafka cluster with three docker that each docker is on one physical node?

Thank you in advance.

Lalit Lalit · Accepted Answer · 2019-01-16T15:07:15

I've run into this too.

I think the clue for this one is in the logs i.e.

Caused by: java.lang.IllegalArgumentException: /tmp/zookeeper/data/myid file is missing at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:408)

The following lines in your zookeeper.properties file provide the details of the zookeeper ensemble:

server.1=0.0.0.0:2888:3888
server.2=150.20.11.134:2888:3888
server.3=150.20.11.137:2888:3888

When one of the zookeeper server starts, it knows which server it is by looking at the myid file in it's own data directory, which in this case would be /tmp/zookeeper/data.

So, all you need to do is just create a file with the name myid in the above mentioned directory of each server and write just x = 1, 2 or 3 (these correspond to server.x in the zookeeper.properties file).

Reference link - Apache Zookeeper.

Hope this helps!

ERROR Invalid config, exiting abnormally in Kafka cluster

1 Answers