How to make Kafka consumer reads from specific topic partition Spring Boot

Question

I'm kind of new to Kafka and Spring Boot and trying to make my application to read from a specific partition of the topic.

 @KafkaListener(id = "singleLnr", groupId = "${kafka.consumer.group.id}",containerFactory = "singleFactory", topicPartitions = @TopicPartition(topic = "${kafka.topic.singleAttendance}", partitions = {"0"}))
public void consume2(ConsumerRecord attendanceInfo) {
    System.out.println(attendanceInfo);
}

Single factory code

@Bean(name = "singleFactory")
public KafkaListenerContainerFactory singleFactory() {
    ConcurrentKafkaListenerContainerFactory<String, Map<String, String>> factory = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());
    factory.setBatchListener(false);
    factory.setMessageConverter(converter());
    return factory;
}

This are also my consumer factory configuration

 @Bean(name = "consumerFactory")
public ConsumerFactory<String, Map<String, String>> consumerFactory() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaBootstrapAddress);
    props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
    props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, 1000);
    props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 60000);
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.GROUP_ID_CONFIG, kafkaConsumerGroupId);
    return new DefaultKafkaConsumerFactory<>(props);
}

When I'm trying to run the program, it gives me a Error

Offset commit failed on partition single.attendance-0 at offset 308: The coordinator is not aware of this member.

and warning

failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.

How can I make my consumer to read from specific partition? Could you please give at least a hint.

Artem Eduardov Artem Eduardov · Accepted Answer · 2019-10-21T15:09:01

Kafka is assigning consumer for each partition by itself. No need to configure it in @KafkaListener in this implementation.

@KafkaListener(id = "singleLnr", groupId = "${kafka.consumer.group.id}",containerFactory = "singleFactory", topics = "${kafka.topic.singleAttendance}")
    public void consume2(ConsumerRecord attendanceInfo) {
        System.out.println(attendanceInfo);
    }

How to make Kafka consumer reads from specific topic partition Spring Boot

1 Answers