Error starting multiple brokers in Apache Kafka -

0
votes

I'm noob in Apache Kafka. I'm reading the book Apache Kafka Quick Start Guide and following its steps to configure the environment. So, i startet with success the Confluent Platform (Kafka plus Zookeeper). Next i'm trying initialize multiple brokers, but is ocurring error when broker.id= 2 is started. I made the below steps:

1 - I have created a folder with config/server.properties named
mark-1.properties and mark-2.properties

2 - Next, i have writed in the mark-1.properties and
mark-2.properties:

    broker.id=1
    port=9093
    log.dirs=/tmp/mark-1-logs

    broker.id=2
    port=9094
    log.dirs=/tmp/mark-2-logs

3 - With Confluent already running, i have typed in the command- line:

    ./bin/kafka-server-start ./mark/mark-1.properties &

     and in other command line window

    ./bin/kafka-server-start ./mark/mark-2.properties &

The output error is:

    [2019-06-28 00:35:26,535] ERROR Fatal error during KafkaServer     
    startup. Prepare to shutdown (kafka.server.KafkaServer)
    kafka.common.InconsistentBrokerIdException: Configured broker.id 2     
    doesn't match stored broker.id 1 in meta.properties. If you moved 
    your data, make sure your configured broker.id matches. If you 
    intend to create a new broker, you should remove all data in your 
    data directories (log.dirs).

I have follow all instructions of the book and foruns, but the error persist. What i can do?

I expect the multiple broker running indepently.

All log error:

    [2019-06-28 15:06:51,018] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2019-06-28 15:06:52,717] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2019-06-28 15:06:52,721] INFO starting (kafka.server.KafkaServer)
[2019-06-28 15:06:52,722] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2019-06-28 15:06:52,753] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2019-06-28 15:06:52,768] INFO Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,768] INFO Client environment:host.name=ip-172-31-72-19.ec2.internal (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,768] INFO Client environment:java.version=11.0.3 (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,768] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,768] INFO Client environment:java.home=/usr/lib/jvm/java-11-openjdk-amd64 (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,769] INFO Client environment:java.class.path=/opt/kafka/kafka_2.12-2.3.0/bin/../libs/activation-1.1.1.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/aopalliance-repackaged-2.5.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/connect-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/connect-basic-auth-extension-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/connect-file-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/connect-json-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/connect-runtime-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/connect-transforms-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/guava-20.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/hk2-api-2.5.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/hk2-locator-2.5.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/hk2-utils-2.5.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-annotations-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-core-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-databind-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-dataformat-csv-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-datatype-jdk8-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-jaxrs-base-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-jaxrs-json-provider-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-module-jaxb-annotations-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-module-paranamer-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jackson-module-scala_2.12-2.9.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jakarta.annotation-api-1.3.4.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jakarta.inject-2.5.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-client-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-common-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-container-servlet-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-container-servlet-core-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-hk2-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-media-jaxb-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jersey-server-2.28.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-client-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-continuation-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-http-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-io-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-security-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-server-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-servlet-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-servlets-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jetty-util-9.4.18.v20190429.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/jsr305-3.0.2.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-clients-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-log4j-appender-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-streams-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-streams-examples-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-streams-scala_2.12-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-streams-test-utils-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka-tools-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka_2.12-2.3.0-sources.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/kafka_2.12-2.3.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/log4j-1.2.17.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/lz4-java-1.6.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/maven-artifact-3.6.1.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/paranamer-2.8.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/plexus-utils-3.2.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/reflections-0.9.11.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/rocksdbjni-5.18.3.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/scala-library-2.12.8.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/scala-logging_2.12-3.9.0.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/scala-reflect-2.12.8.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/slf4j-api-1.7.26.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/slf4j-log4j12-1.7.26.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/snappy-java-1.1.7.3.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/spotbugs-annotations-3.1.9.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/validation-api-2.0.1.Final.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/zkclient-0.11.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/zookeeper-3.4.14.jar:/opt/kafka/kafka_2.12-2.3.0/bin/../libs/zstd-jni-1.4.0-1.jar (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,771] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,771] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,772] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,772] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,772] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,772] INFO Client environment:os.version=4.15.0-1041-aws (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,772] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,772] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,773] INFO Client environment:user.dir=/opt/kafka/kafka_2.12-2.3.0 (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,774] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@58e1d9d (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:52,819] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2019-06-28 15:06:52,862] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2019-06-28 15:06:52,876] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2019-06-28 15:06:52,887] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100000ec2640001, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2019-06-28 15:06:52,910] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2019-06-28 15:06:53,309] INFO Cluster ID = LM8y5RcXQJCMJeIJlmi34A (kafka.server.KafkaServer)
[2019-06-28 15:06:53,329] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentBrokerIdException: Configured broker.id 2 doesn't match stored broker.id 1 in meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).
    at kafka.server.KafkaServer.getBrokerIdAndOfflineDirs(KafkaServer.scala:715)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:214)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
    at kafka.Kafka$.main(Kafka.scala:84)
    at kafka.Kafka.main(Kafka.scala)
[2019-06-28 15:06:53,333] INFO shutting down (kafka.server.KafkaServer)
[2019-06-28 15:06:53,350] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2019-06-28 15:06:53,365] INFO EventThread shut down for session: 0x100000ec2640001 (org.apache.zookeeper.ClientCnxn)
[2019-06-28 15:06:53,387] INFO Session: 0x100000ec2640001 closed (org.apache.zookeeper.ZooKeeper)
[2019-06-28 15:06:53,393] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2019-06-28 15:06:53,423] INFO shut down completed (kafka.server.KafkaServer)
[2019-06-28 15:06:53,427] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2019-06-28 15:06:53,479] INFO shutting down (kafka.server.KafkaServer)
3
apache-kafka
delete all data on /tmp/mark* directories, and try again. those dirs store metas that may make the second broker fail by an id mismatch. Another option is modifying those files, but kafka will regenerate it all if you delete+restart - aran
I have just do it, but the error is the same. - Fernando de Sá

3 Answers

0
votes

Did you start with the same./bin/Kafka-server-start file?

If so, you should copy all kafka files to another directory.

Under the same startup directory, it is not possible to specify only different configuration files start kafka.

0
votes

Dears, the error was mine. I was writing lines duplicates with commands about port and log.dir in the new kafka brokers config properties.

Thanks for your attention!

Below a snippet of correct config file:

    # The id of the broker. This must be set to a unique integer for each broker.
broker.id=1

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9093

# Hostname and port the broker will advertise to producers and consumers. If not set, 
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600


############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/tmp/kafka-logs-1
0
votes

I had this error - kafka.common.InconsistentBrokerIdException - when I was trying to run kafka (single zookeeper and single broker) on my local windows machine.

Below change worked for me -

In zookeeper.properties set -

dataDir=C:/Kafka/data/zookeeperlogs

In server.properties set -

log.dirs=C:/Kafka/data/brokerlogs

The parent directory(C:/Kafka/data) was different previously which was making the difference.