java.io.IOException: Connection to kafka VIA port 6667 failed at kafka.utils.NetworkClientBlockingOps$.awaitReady

Question

We have 3 kafka machines and 3 zookper machines

kafka machines are separated from the zookeper machines , and all machines are with OS - redhat 7.x

under /var/log/kafka we can see the following , java.io.IOException that displayed in server.log

               at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2018-10-07 07:42:13,342] WARN [ReplicaFetcherThread-0-1007], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@407ac4e4 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to kafka03.hfge.com:6667 (id: 1007 rack: null) failed
               at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83)
               at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93)
               at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:248)
               at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:238)
               at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
               at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118)
               at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103)
               at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2018-10-07 07:42:13,551] WARN [ReplicaFetcherThread-0-1002], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@2f9f8ecd (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to kafka07.hfge.com:6667 (id: 1002 rack: null) failed
               at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83)
               at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93)
               at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:248)
               at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:238)
               at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
               at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118)
               at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103)
               at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2018-10-07 07:42:15,343] WARN [ReplicaFetcherThread-0-1007], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@4dafd0d9 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to kafka03.hfge.com:6667 (id: 1007 rack: null) failed
               at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83)
               at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93)
               at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:248)
               at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:238)
               at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
               at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118)
               at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103)
               at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2018-10-07 07:42:15,552] WARN [ReplicaFetcherThread-0-1002], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5ce51081 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to kafka07.hfge.com:6667 (id: 1002 rack: null) failed

additional with this , we also have a problem that kafka disconnected from the zookeeper after some time , this means that the broker id deleted from the zookeeper ( from zoo session /brokers/ids )

cd /usr/hdp/current/zookeeper-server/bin

./zkCli.sh

ls /brokers/ids

any idea how to find the cause of

java.io.IOException: Connection to kafka03.hfge.com:6667

I will try , and I will restart the kafka broker , do you want to look on the log after that ? — King David
Yes, please update the new log after you edit the file and restart the broker that is having these warnings — OneCricketeer

Giorgos Myrianthous Giorgos Myrianthous · Accepted Answer · 2018-10-02T05:57:15

This issue occurs when the inter.broker.protocol.version is not updated correctly. To resolve this issue, update the inter.broker.protocol.version and restart the brokers one by one as described in the Documentation.

java.io.IOException: Connection to kafka VIA port 6667 failed at kafka.utils.NetworkClientBlockingOps$.awaitReady

1 Answers