Kafka in docker-compose: topics not in preferred replica for broker

Question

I am running a testing kafka "cluster" consisting of a single Kafka broker and a single Zookeeper node from docker-compose, using the following (official) docker-compose.yml:

---
    version: '2'
    
    services:
      zookeeper:
        image: confluentinc/cp-zookeeper:6.0.0
        hostname: zookeeper
        container_name: zookeeper
        ports:
          - "2181:2181"
        environment:
          ZOOKEEPER_CLIENT_PORT: 2181
          ZOOKEEPER_TICK_TIME: 2000
    
      broker:
        image: confluentinc/cp-kafka:6.0.0
        hostname: broker
        container_name: broker
        depends_on:
          - zookeeper
        ports:
          - "29092:29092"
        environment:
          KAFKA_BROKER_ID: 101
          KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
          KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
          KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
          KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
          KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

After starting this "cluster" with docker-compose up I successfully create a topic using console command:

docker exec broker kafka-topics --create --topic myTopic --bootstrap-server localhost:29092 --replication-factor 1 --partitions 1

and programmatically create messages with a Golang producer, while listening to them with a console client:

docker exec broker kafka-console-consumer --from-beginning --bootstrap-server localhost:29092 --topic myTopic --timeout-ms 15000

Every once in a while both the console consumer and the Golang programmatic producer hang and stop connecting to kafka. When I look into the logs of Kafka broker to figure out, what's wrong, I see a wall of repeating messages of the following structure about topics being not in preferred replica:

broker       | [2021-05-09 06:08:50,144] INFO [Controller id=101] Processing automatic preferred replica leader election (kafka.controller.KafkaController)
broker       | [2021-05-09 06:08:50,149] TRACE [Controller id=101] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
broker       | [2021-05-09 06:08:50,165] DEBUG [Controller id=101] Topics not in preferred replica for broker 101 HashMap() (kafka.controller.KafkaController)
broker       | [2021-05-09 06:08:50,166] TRACE [Controller id=101] Leader imbalance ratio for broker 101 is 0.0 (kafka.controller.KafkaController)

Could you explain, what's going wrong? How could a wrong replica be assigned, if there's just 1 partition and 1 broker?

Since you're exec-ing into the broker, you'd use kafka:9092 as the connection string, but why do you have TRACE logs enabled? — OneCricketeer
@OneCricketeer Hi, thanks for comments and response. Didn't modify the log level in server.properties. Apparently, that's the default. 9092 and 29092 are both allowed, I believe. — Boris Burkov
You should only use localhost:29092 outside of the Docker network, specially only with code connecting from the Docker host. That's why it's named PLAINTEXT_HOST... More specifically, if your Go code was in its own container, using that address would try to connect to itself, and not the broker — OneCricketeer

OneCricketeer OneCricketeer · Accepted Answer · 2021-05-09T13:00:11

I don't think this small section of logs is your issue

Those logs say there's no topics not in preferred replica set... It's printing an empty Hashmap

Similarly, the imbalance ratio is zero, so the cluster is perfectly balanced

Kafka in docker-compose: topics not in preferred replica for broker

1 Answers