Kafka Reduce Lag for Consumer

3
votes

I am setting up the new Kafka cluster and for testing purpose I created the topic with 1 partition and 3 replicas.

Now, I am firing the messages via producer in parallel say 50K messages per Second. And I have create One Consumer inside a Group and its only able to fetch 30K messages per second.

I can change topic level, partition level, consumer level configurations.

I am monitoring everything via grafana + prometheus.

Any Idea which configuration or something else can help me to consumer more data??

Thanks In Advance

1
apache-kafkakafka-consumer-api
Have You looked at the Kafka metrics? Maybe it's network issue ? – Dominik Wosiński
are you OK with changing the configuration of partitions ? – Kiruparan Balachandran
Yes I can change the partition level configs also. – Akash Sethi

1 Answers

5
votes

A Kafka consumer polls the broker for messages and fetches whatever messages are available for consumption, depending upon the consumer configuration used. In general, it is efficient to transfer as much data is possible in a single poll request if increasing throughput is your aim. But how much data is transferred in a single poll is determined by the size of messages, number of records and some parameters which control how much time to wait for messages to be available.

In general, you can influence throughput using one or more of the following consumer configurations:

fetch.min.bytes

max.partition.fetch.bytes

fetch.max.bytes

max.poll.records

fetch.max.wait.ms