Why is my spring cloud stream config creating multiple kafka consumers

Question

It's not a huge problem but I'm curious where some extra stream consumers are coming from, and if that's a setting I can change.

I've got a very simple spring cloud stream consumer setup against a local Kafka broker. Here's the spring config

spring:
  cloud:
    stream:
      bindings:
        consumer-in-0:
          destination: test-topic
          group: test-group

And the consumer class itself:

@Bean
Consumer<Message<String>> consumer() {
 return message -> System.out.println("Got it: " + message.getPayload());
}

When I run the app though, I can see 3 consumers created in the output. But when I check the consumer-group members in my local broker, it's always just one consumer, and it's always the second consumer created (i.e. with client id test-group-2)

Just for clarity, I'm using Spring Boot version 2.3.4.RELEASE and cloud dependencies version Hoxton.SR10.

And here's the dependencies in the pom:

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-devtools</artifactId>
    <scope>runtime</scope>
    <optional>true</optional>
  </dependency>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-stream</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-stream-binder-kafka</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
  </dependency>
<dependencies>

Why am I getting 3 consumers? Why is the second one the only one that actually listens on the Kafka topic?

I think you mean that another group test-group-2 is created, don't you? And not another topic? — Felipe
There's not multiple topics being created. There's just the one topic test-topic When the spring library is creating the consumers, it creates them with a consumer id that is the group name + what spring adds to uniquely identify the consumers. So it creates 3: test-group-1, test-group-2, test-group-3. There's still only one group, just with 3 consumers. The consumer with consumer id test-group-2 is the only one actually listening on the topic though. — pChip

Gary Russell Gary Russell · Accepted Answer · 2021-04-14T15:19:45

During start up, a temporary consumer is created to get information about the partitions provisioned for the topic.

The second consumer is the real consumer.

If you have the actuator (actually Micrometer) on the classpath the KafkaBinderMetrics creates another consumer so it can calculate the lag. It does not actually consume anything.

Why is my spring cloud stream config creating multiple kafka consumers

1 Answers