I have three consumers all with multiple instances which all are consuming the same topic. I want each consumer to consume the topic once. To do this I have created a consumer group. From what I read Kafka should be smart enough to select one instance of a service to consume the topic but this is not happening and all instances of all consumers are consuming the topic
I thought it might have been something to do with the three consumers all having the same group name so I turned off two of the consumers leaving one consumer with two instances but I still see two records going into the database when the consumer group should only be selecting one instance to insert into the database.
Am I doing something wrong or missing something below?
Application.yml
spring:
cloud:
stream:
bindings:
input-data:
destination: publisheddata.t
group: publisheddata
kafka:
bindings:
input:
consumer:
autoCommitOffset: false
binder:
auto-create-topics: true
kafka:
mode: raw
spring:
cloud:
stream:
kafka:
binder:
brokers: kafka:9092
zk-nodes: kafka:2181
Channels.java
public interface Channels {
String INPUT_DATA = "input-data";
@Input(INPUT_DATA)
SubscribableChannel dataInput();
}
DataHandler.java
@EnableBinding(Channels.class)
@Configuration
public class DataMessageHandler {
@StreamListener(Channels.INPUT_DATA)
public void handle(Message<?> message) {
... handling message ...
}
