I have ended up with the below exception on broker 2 node while implementing SSL for Broker to Zookeeper Authentication
From SSL settings, the cluster is configured with SSL authentication
Broker 1 succesfully connected to the zookeeper but on broker 2, there is an exception org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /brokers/ids
At the same time when I stopped broker 1 and connect broker 2 to the zookeeper, the connection is working. The issue is one Kafka only connecting to the zookeeper, another Kafka not connecting at the same time.
node 1: kafka1 and zookeeper node 2: kafka2
Config files:
zookeeper.properties
clientPort=2181
secureClientPort=2182
authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.trustStore.location=/opt/kafka_2.13-2.5.0/ssl/kafka.zookeeper.truststore.jks
ssl.trustStore.password=123456
ssl.keyStore.location=/opt/kafka_2.13-2.5.0/ssl/kafka.zookeeper.keystore.jks
ssl.keyStore.password=123456
ssl.clientAuth=need
maxClientCnxns=0
admin.enableServer=false
server.1=localhost:2888:3888
KAFKA1 properties(kafka running in same zookeeper machine) server-0.properties
broker.id=0
listeners=SASL_SSL://172.31.70.27:9092
advertised.listeners=SASL_SSL://1172.31.70.27:9092
zookeeper.connect=172.31.70.27:2182
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.client.enable=true
zookeeper.ssl.protocol=TLSv1.2
zookeeper.ssl.truststore.location=/opt/kafka_2.13-2.5.0/ssl/kafka.broker0.truststore.jks
zookeeper.ssl.truststore.password=123456
zookeeper.ssl.keystore.location=/opt/kafka_2.13-2.5.0/ssl/kafka.broker0.keystore.jks
zookeeper.ssl.keystore.password=123456
zookeeper.set.acl=true
ssl.truststore.location=/opt/kafka_2.13-2.5.0/ssl/kafka.broker0.truststore.jks
ssl.truststore.password=Change@Ro0t
ssl.keystore.location=/opt/kafka_2.13-2.5.0/ssl/kafka.broker0.keystore.jks
ssl.keystore.password=Change@Ro0t
ssl.key.password=Change@Ro0t
security.inter.broker.protocol=SASL_SSL
ssl.client.auth=none
ssl.protocol=TLSv1.2
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="broker-admin" password="dev1234";
super.users=User:broker-admin
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
KAFKA2 properties (Kafka running on another machine)
server-1.properties
broker.id=1
listeners=SASL_SSL://172.31.76.221:9093
advertised.listeners=SASL_SSL://1172.31.76.221:9093
zookeeper.connect=172.31.70.27:2182
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.client.enable=true
zookeeper.ssl.protocol=TLSv1.2
zookeeper.ssl.truststore.location=/opt/kafka_2.13-2.5.0/ssl/kafka.broker0.truststore.jks
zookeeper.ssl.truststore.password=123456
zookeeper.ssl.keystore.location=/opt/kafka_2.13-2.5.0/ssl/kafka.broker0.keystore.jks
zookeeper.ssl.keystore.password=123456
zookeeper.set.acl=true
ssl.truststore.location=/opt/kafka_2.13-2.5.0/ssl/kafka.broker1.truststore.jks
ssl.truststore.password=Change@Ro0t
ssl.keystore.location=/opt/kafka_2.13-2.5.0/ssl/kafka.broker1.keystore.jks
ssl.keystore.password=Change@Ro0t
ssl.key.password=Change@Ro0t
security.inter.broker.protocol=SASL_SSL
ssl.client.auth=none
ssl.protocol=TLSv1.2
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="broker-admin" password="dev1234";
super.users=User:broker-admin
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
Can you please help, what could be the reason for the broker to shut down with the below exception?
[2021-05-13 10:44:37,659] INFO Session establishment complete on server ip-172-31-45-2.ec2.internal/172.31.45.2:2182, sessionid = 0x100009398f40002, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2021-05-13 10:44:37,663] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2021-05-13 10:44:37,801] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /brokers/ids
[2021-05-13 10:44:37,804] INFO shutting down (kafka.server.KafkaServer)
[2021-05-13 10:44:37,808] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2021-05-13 10:44:37,814] INFO channel is told closing (org.apache.zookeeper.ClientCnxnSocketNetty)
[2021-05-13 10:44:37,816] INFO EventThread shut down for session: 0x100009398f40002 (org.apache.zookeeper.ClientCnxn)
[2021-05-13 10:44:37,816] INFO Session: 0x100009398f40002 closed (org.apache.zookeeper.ZooKeeper)
[2021-05-13 10:44:37,817] INFO channel is disconnected: [id: 0x5856be9a, L:/172.31.45.2:52900 ! R:ip-172-31-45-2.ec2.internal/172.31.45.2:2182] (org.apache.zookeeper.ClientCnxnSocketNetty)
[2021-05-13 10:44:37,817] INFO channel is told closing (org.apache.zookeeper.ClientCnxnSocketNetty)
[2021-05-13 10:44:37,825] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2021-05-13 10:44:37,828] INFO shut down completed (kafka.server.KafkaServer)
