Unable to list Kafka topics

0
votes

I am running ansible to install Kafka on one server. (on Dev env, standalone setup) but my playbook is getting failed for task listing of Kafka topics. below are the logs.

  1. journalctl -fu zookeeper
-- Logs begin at Thu 2020-11-19 01:05:17 IST. --
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119)
Nov 19 22:33:08 <HOSTNAME> zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: main process exited, code=exited, status=1/FAILURE
Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service: control process exited, code=exited status=1
Nov 19 22:33:09 <HOSTNAME> systemd[1]: Unit zookeeper.service entered failed state.
Nov 19 22:33:09 <HOSTNAME> systemd[1]: zookeeper.service failed.

journalctl -fu kafka.service
-- Logs begin at Thu 2020-11-19 01:05:17 IST. --
Nov 19 22:23:30 <HOSTNAME> systemd[1]: Started kafka.service.
Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: main process exited, code=exited, status=1/FAILURE
Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service: control process exited, code=exited status=1
Nov 19 22:23:40 <HOSTNAME> systemd[1]: Unit kafka.service entered failed state.
Nov 19 22:23:40 <HOSTNAME> systemd[1]: kafka.service failed.
Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service.
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process exited, code=exited, status=1/FAILURE
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control process exited, code=exited status=1
Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered failed state.
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed.
  1. Getting same error when run command manually which playbook is trying to run.
/opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh --zookeeper <ip of localhost>:5181 --list
  1. both Kafka and zookeeper are failed and not getting started.
  systemctl status  kafka.service -l
● kafka.service
   Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:17 IST; 5min ago
  Process: 5448 ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-stop.sh (code=exited, status=1/FAILURE)
  Process: 4742 ExecStart=/bin/sh -c /opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-start.sh /opt/kafka/kafka_2.12-2.2.2/config/server.properties > /opt/kafka/kafka_2.12-2.2.2/kafka.log 2>&1 (code=exited, status=1/FAILURE)
 Main PID: 4742 (code=exited, status=1/FAILURE)

Nov 19 22:33:08 <HOSTNAME> systemd[1]: Started kafka.service.
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: main process exited, code=exited, status=1/FAILURE
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service: control process exited, code=exited status=1
Nov 19 22:33:17 <HOSTNAME> systemd[1]: Unit kafka.service entered failed state.
Nov 19 22:33:17 <HOSTNAME> systemd[1]: kafka.service failed.


 systemctl status  zookeeper.service  -l
● zookeeper.service
   Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:09 IST; 5min ago
  Process: 5281 ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-stop.sh (code=exited, status=1/FAILURE)
  Process: 4448 ExecStart=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-start.sh /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties (code=exited, status=1/FAILURE)
 Main PID: 4448 (code=exited, status=1/FAILURE)

Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119)
Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Nov 19 22:33:09 systemd[1]: zookeeper.service: main process exited, code=exited, status=1/FAILURE
Nov 19 22:33:09  systemd[1]: zookeeper.service: control process exited, code=exited status=1
Nov 19 22:33:09  systemd[1]: Unit zookeeper.service entered failed state.
Nov 19 22:33:09  systemd[1]: zookeeper.service failed.
  1. Kafka.log
[2020-11-19 22:33:09,853] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-11-19 22:33:10,584] INFO starting (kafka.server.KafkaServer)
[2020-11-19 22:33:10,585] INFO Connecting to zookeeper on <ip of localhost>:5181 (kafka.server.KafkaServer)
[2020-11-19 22:33:10,640] INFO [ZooKeeperClient] Initializing a new session to <ip of localhost>:5181. (kafka.zookeeper.ZooKeeperClient)
[2020-11-19 22:33:10,646] INFO Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:host.name=<FQDN> (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.version=1.8.0_221 (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.home=/opt/jre1.8.0_221 (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-api-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-basic-auth-extension-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-file-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-json-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-runtime-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-transforms-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/guava-20.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-api-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-locator-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/hk2-utils-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-annotations-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-hk2-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-media-jaxb-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-server-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-examples-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-scala_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-test-utils-2.2.2.jar:/opt/kafka/kafka_2.12-2-2.2.2/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zookeeper-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,646] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:os.version=3.10.0-1062.el7.x86_64 (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:user.name=kafka (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:user.home=/home/kafka (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,647] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,648] INFO Initiating client connection, connectString=<ip of localhost>:5181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@3c419631 (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:10,667] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:10,673] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:10,676] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2020-11-19 22:33:11,796] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:11,797] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:12,897] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:12,898] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:13,999] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:13,999] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:15,101] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:15,101] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:16,202] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:16,202] INFO Socket error occurred: <FQDN>/<ip of localhost>:5181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:16,678] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient)
[2020-11-19 22:33:17,303] INFO Opening socket connection to server <FQDN>/<ip of localhost>:5181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:17,406] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2020-11-19 22:33:17,407] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2020-11-19 22:33:17,409] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
[2020-11-19 22:33:17,412] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
        at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
        at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:96)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
        at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:361)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:385)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:205)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
        at kafka.Kafka$.main(Kafka.scala:75)
        at kafka.Kafka.main(Kafka.scala)
[2020-11-19 22:33:17,415] INFO shutting down (kafka.server.KafkaServer)
[2020-11-19 22:33:17,422] INFO shut down completed (kafka.server.KafkaServer)
[2020-11-19 22:33:17,422] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2020-11-19 22:33:17,426] INFO shutting down (kafka.server.KafkaServer)
  1. /var/log/messages
Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=/opt/kafka/kafka_
2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None argv=None warn=True chdir=None stdin_add_newline=True stdin=None
Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=/opt/kafka/kafka_
2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None argv=None warn=True chdir=None stdin_add_newline=True stdin=None
Nov 19 18:16:07 <HOSTNAME> ansible-command: Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params=/opt/kafka/kafka_
2.12-2.2.2/bin/kafka-topics.sh --zookeeper <FQDN>:5181 --list removes=None argv=None warn=True chdir=None stdin_add_newline=True stdin=None
Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.941+0530#011INFO#011kafka/log.go:53#011client/metadata fetching metadata for [heartbeat] from bro
ker <ip of localhost>:3997
Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011Failed to connect to broker <ip of localhost>:3997: dial tcp
<ip of localhost>:3997: connect: connection refused
Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011client/metadata got error from broker -1 while fetching me
tadata: dial tcp <ip of localhost>:3997: connect: connection refused
Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011kafka message: client/metadata no available broker to send
 metadata request to
Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.942+0530#011INFO#011kafka/log.go:53#011client/brokers resurrecting 1 dead seed brokers
Nov 19 18:16:07 <HOSTNAME> heartbeat: 2020-11-19T18:16:07.942+0530#011INFO#011[publisher]#011pipeline/retry.go:155#011Drop batch
Nov 19 18:16:07 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:07,962] INFO Reading configuration from: /opt/kafka/kafka_2.12-2.2.2/config/zookeep
er.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,009] INFO Resolved hostname: <FQDN> to address: <FQDN>/<ip of localhost> (org.apache.zookeeper.server.quorum.QuorumPeer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,009] ERROR Invalid configuration, only one server specified (ignoring) (org.apac
he.zookeeper.server.quorum.QuorumPeerConfig)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.Datadi
rCleanupManager)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirC
leanupManager)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,011] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain

Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027] INFO Reading configuration from: /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027] INFO Resolved hostname: <FQDN> to address: <FQDN>/<ip of localhost> (org.apache.zookeeper.server.quorum.QuorumPeer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027] ERROR Invalid configuration, only one server specified (ignoring) (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,027] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:host.name=<FQDN> (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:java.version=1.8.0_221 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:java.vendor=Oracle Corporation (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:java.home=/opt/jre1.8.0_221 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,033] INFO Server environment:java.class.path=/opt/kafka/kafka_2.12-2.2.2/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/argparse4j-extension-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/connect-transforms-2.10.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jackson-core-CR2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.inject-2.5.0-b42.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/javax.ws.rs-api-2.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-client-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-common-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jersey-container-servlet-core-2.27.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-client-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/jetty-continuation-9.4.14.v20181114.jar:/opt/kafka/kafka_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-scala_2.12-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/kafka-streams-test-utils-2.2.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/plexus-utils-3.1.0.jar:/opt/kafka/kafka_2.12-1.1.7.2.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zkclient-3.4.13.jar:/opt/kafka/kafka_2.12-2.2.2/bin/../libs/zstd-jni-1.3.8-1.jar (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:os.version=3.10.0-1062.el7.x86_64 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:user.name=kafka (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:user.home=/home/kafka (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,034] INFO Server environment:user.dir=/ (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,043] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,043] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,044] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,049] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,054] INFO binding to port 0.0.0.0/0.0.0.0:5181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: [2020-11-19 18:16:08,131] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: java.io.EOFException
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at java.io.DataInputStream.readInt(DataInputStream.java:392)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:66)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:585)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:604)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:570)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:650)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:219)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:176)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:217)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:284)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:407)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:89)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:55)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:119)
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-start.sh: at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: main process exited, code=exited, status=1/FAILURE
Nov 19 18:16:08 <HOSTNAME> zookeeper-server-stop.sh: No zookeeper server to stop
Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service: control process exited, code=exited status=1
Nov 19 18:16:08 <HOSTNAME> systemd: Unit zookeeper.service entered failed state.
Nov 19 18:16:08 <HOSTNAME> systemd: zookeeper.service failed.
  1. /opt/kafka/kafka_2.12-2.2.2/config/server.properties
broker.id=1
advertised.listeners=PLAINTEXT://hostname of localhost:3997
listeners=PLAINTEXT://hostname of localhost:3997

delete.topic.enable=true
auto.create.topics.enable=True
offsets.topic.replication.factor=1

# The number of threads handling network requests
num.network.threads=3

# The number of threads doing disk I/O
num.io.threads=6

# The send buffer (SO_SNDBUF) used by the socket server
#socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
#socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
#socket.request.max.bytes=104857600

# A comma seperated list of directories under which to store log files
log.dirs=/var/log/kafka

# The default number of log partitions per topic.
num.partitions=3

# Default Replication Factor
default.replication.factor=1

# The minimum age of a log file to be eligible for deletion
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining
# segments don't drop below log.retention.bytes.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

zookeeper.connect= <ip of localhost>:5181
zookeeper.connection.timeout.ms=6000

/opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties

# clientPortBindAddress=localhost
clientPort=5181

dataDir=/var/data/zookeeper
dataLogDir=/var/data/logs/zookeeper
maxClientCnxns=0

# time for inital synchronization
initLimit=10
# how many ticks can pass before timeout
syncLimit=5

server.1=<hostname of localhost>:2888:3888

zookeeper.service

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=kafka
ExecStart=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-start.sh /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties
ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target
1
apache-kafkaapache-zookeeper
What are your zookeeper.properties and server.properties files? Update your question with those two configs. - leandrojmp
Thanks. added config file. Unable to do telnet on port 5181 and its not showing in netstat as service is not running. - pratkul007
Try to remove the line server.1 from your zookeeper config file, you only have one server, so you should not set this config for a standalone operation. - leandrojmp
I commented this line and tried starting zookeeper and Kafka service but its not starting. I have same setup on another server and there playbook ran successfully. able to list the topic there. server.1 entry is present over there as well. only diff might be there I have ran ansible first time and here (where issue exists) it has ran 2,3 times. do I need to delete some old things like indices etc.. ? or can we set logs to verbose mode or anything else to fix the issue ? - pratkul007
You need to make sure that your zookeeper is running before you try to start kafka, forget about kafka until your zookeeper is running without problems. I would recommend that you try to start zookeeper by hand, without ansible, increase the log level to TRACE and see if you can catch what is the problem, your config seems right to me, you will need to troubleshoot more, check configs, permissions, memory available. - leandrojmp

1 Answers

0
votes

all four words commands (stat,ruok) where giving connection refused.

I moved all files (log.1,log.103,log.142,log.16a,log.17d - all where having oct timestamp) present inside /var/data/logs/zookeeper/version-2/ to some other directory and started zookeeper and then kafka and both started and are up and running. I am able to list the topics now. now I am see log.1 file got created. (its binary so can't see inside it)

Can any one know the reason behind it? why it worked after moving these log* files which seems to be little old.

not sure if same has to do after 1 month from now if same issue occurs again.