spring cloud stream kafka

Question

I've built a producer spring cloud stream app and kafka as binder. Here is the application.yml:

spring:
cloud:
  stream:
    instanceCount : 1
    bindings:
      output:
        destination: topic-sink
        producer:
           partitionSelectorClass: com.partition.CustomPartition
           partitionCount: 1        
...

I have two instances (same app running on a single jvm) as consumers. Here is the application.yml:

spring:  
cloud:
  stream:
    bindings:
      input:
        destination: topic-sink
        group: hdfs-sink
        consumer:
          partitioned: true
...

My understanding of kafka groups is that messages will be consumed only once, for those consumers in same group. Let's say, if the producer app produces messages A, B and there are two consumer apps in the same group, message A will be read by consumer 1 and messages B, C will be read by consumer 2. However, my consumers are consuming same messages. Are my assumptions wrong?

The idea with consumer groups is that all consumers within that group will consume all events from a given topic. However, if there are more consumers than partitions for that topic, then N consumers (where N is #consumers - #partitions) will be doing nothing. — Arek

fjbec fjbec · Accepted Answer · 2016-09-30T16:26:49

I got the solution, thanks Arek. For 1 partition and 1 consumer. I share the solution for producer\consumer in spring cloud stream app. Producer:

spring: cloud: stream: instanceCount : 1 bindings: output: destination: topic-sink producer: partitionSelectorClass: com.partition.CustomPartition partitionCount: 1
Consumer:

spring: cloud: stream: instanceIndex: 0 #between 0 and instanceCount - 1 instanceCount: 1 bindings: input: destination: topic-sink group: hdfs-sink consumer: partitioned: true kafka: binder: autoAddPartitions: true

spring cloud stream kafka

1 Answers