Kafka ConsumerPerformance

Question

My Kafka cluster has the next configuration.

Kafka version cluster (v1.1.0) with 3 brokers
One topic ("fara") with 5 partitions and 3 replicas
In each partition there are 10.000.000 messages. 50.000.000 in total

I am using kafka-consumer-perf.test with is coming with the distro that underneath uses ConsumerPerformance.

My first working configuration is the next:

1 Consumer instance (50.000.000 records by the instance)
5 fetcher threads (1 by partition)
Consumer group = testgroup1

The command I run:

$KAFKA_HOME/bin/kafka-consumer-perf-test.sh \ --new-consumer \ --broker-list $BROKER_LIST \ --messages 50000000 \ --message-size 100 \ --topic fara \ --consumer.config $KAFKA_HOME/config/consumer.ssl.properties \ --num-fetch-threads 5 \ --show-detailed-stats \ --reporting-interval $REPORT_INTERVAL \ --group testgroup1

I am running the command below periodically

./kafka-run-class.sh kafka.admin.ConsumerGroupCommand --bootstrap-server 192.168.10.10:9093 --new-consumer --describe --group testgroup1 --command-config ../config/consumer.properties

          TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
          fara          4          1056241         10000000        8943759         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          3          1075308         10000000        8924692         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          1          1056241         10000000        8943759         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          2          2129355         10000000        7870645         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          0          1075308         10000000        8924692         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1


          fara          4          2302837         10000000        7697163         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          3          2377136         10000000        7622864         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          1          2302837         10000000        7697163         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          2          4641785         10000000        5358215         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          0          2377197         10000000        7622803         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1


          fara          4          5576341         10000000        4423659         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          3          5738479         10000000        4261521         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          1          5576341         10000000        4423659         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          2          10000000        10000000        0               consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          0          5739456         10000000        4260544         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1


          fara          4          8735653         10000000        1264347         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          3          8825872         10000000        1174128         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          1          8735653         10000000        1264347         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          2          10000000        10000000        0               consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1
          fara          0          8827764         10000000        1172236         consumer-1-f9fd5d69-6ba1-4da1-9eb9-29a64a83d1fa /192.168.10.145 consumer-1


          fara          4          10000000        10000000        0               -               -               -
          fara          3          10000000        10000000        0               -               -               -
          fara          1          10000000        10000000        0               -               -               -
          fara          2          10000000        10000000        0               -               -               -
          fara          0          10000000        10000000        0               -               -               -

All the messages are consumed.

The second failing configuration is the next:

5 Consumer instances (10.000.000 records by instance)
1 fetcher threads by consumer (to consume just a partition)
Consumer group = testgroup2 for all the instances

The command I run:

$KAFKA_HOME/bin/kafka-consumer-perf-test.sh \ --new-consumer \ --broker-list $BROKER_LIST \ --messages 10000000 \ --message-size 100 \ --topic fara \ --consumer.config $KAFKA_HOME/config/consumer.ssl.properties \ --num-fetch-threads 1 \ --show-detailed-stats \ --reporting-interval $REPORT_INTERVAL \ --group testgroup2

I am running the command below periodically

./kafka-run-class.sh kafka.admin.ConsumerGroupCommand --bootstrap-server 192.168.10.10:9093 --new-consumer --describe --group testgroup2 --command-config ../config/consumer.properties

         TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
          fara          1          4129909         10000000        5870091         consumer-1-2e20de15-9396-4ebb-a77d-c16ef7c0cb03 /192.168.10.145 consumer-1
          fara          4          4520065         10000000        5479935         consumer-1-ed2e3f63-23e9-489a-a9bc-422e7830fc9c /192.168.10.139 consumer-1
          fara          2          4310725         10000000        5689275         consumer-1-5dbc6ec0-7bc6-4d71-8b17-1b5af261479d /192.168.10.134 consumer-1
          fara          3          4038958         10000000        5961042         consumer-1-646b2ebd-3a36-4e09-a7d4-658225492a03 /192.168.10.145 consumer-1
          fara          0          4524641         10000000        5475359         consumer-1-2a7186c9-c4a0-41d6-ba5e-809964b7b709 /192.168.10.134 consumer-1



          fara          1          6071173         10000000        3928827         consumer-1-2e20de15-9396-4ebb-a77d-c16ef7c0cb03 /192.168.10.145 consumer-1
          fara          4          6689713         10000000        3310287         consumer-1-ed2e3f63-23e9-489a-a9bc-422e7830fc9c /192.168.10.139 consumer-1
          fara          2          6437488         10000000        3562512         consumer-1-5dbc6ec0-7bc6-4d71-8b17-1b5af261479d /192.168.10.134 consumer-1
          fara          3          6194150         10000000        3805850         consumer-1-646b2ebd-3a36-4e09-a7d4-658225492a03 /192.168.10.145 consumer-1
          fara          0          6612859         10000000        3387141         consumer-1-2a7186c9-c4a0-41d6-ba5e-809964b7b709 /192.168.10.134 consumer-1


          fara          1          7489057         10000000        2510943         consumer-1-2e20de15-9396-4ebb-a77d-c16ef7c0cb03 /192.168.10.145 consumer-1
          fara          4          8212273         10000000        1787727         consumer-1-ed2e3f63-23e9-489a-a9bc-422e7830fc9c /192.168.10.139 consumer-1
          fara          2          7883485         10000000        2116515         consumer-1-5dbc6ec0-7bc6-4d71-8b17-1b5af261479d /192.168.10.134 consumer-1
          fara          3          7457277         10000000        2542723         consumer-1-646b2ebd-3a36-4e09-a7d4-658225492a03 /192.168.10.145 consumer-1
          fara          0          8039217         10000000        1960783         consumer-1-2a7186c9-c4a0-41d6-ba5e-809964b7b709 /192.168.10.134 consumer-1


          fara          2          8535701         10000000        1464299         consumer-1-5dbc6ec0-7bc6-4d71-8b17-1b5af261479d /192.168.10.134 consumer-1
          fara          3          8113576         10000000        1886424         consumer-1-5dbc6ec0-7bc6-4d71-8b17-1b5af261479d /192.168.10.134 consumer-1
          fara          0          8580780         10000000        1419220         consumer-1-2a7186c9-c4a0-41d6-ba5e-809964b7b709 /192.168.10.134 consumer-1
          fara          1          7926793         10000000        2073207         consumer-1-2a7186c9-c4a0-41d6-ba5e-809964b7b709 /192.168.10.134 consumer-1
          fara          4          8897425         10000000        1102575         consumer-1-646b2ebd-3a36-4e09-a7d4-658225492a03 /192.168.10.145 consumer-1


          fara          4          9884954         10000000        115046          consumer-1-646b2ebd-3a36-4e09-a7d4-658225492a03 /192.168.10.145 consumer-1
          fara          3          8693017         10000000        1306983         consumer-1-646b2ebd-3a36-4e09-a7d4-658225492a03 /192.168.10.145 consumer-1
          fara          1          8564365         10000000        1435635         consumer-1-646b2ebd-3a36-4e09-a7d4-658225492a03 /192.168.10.145 consumer-1
          fara          2          9197795         10000000        802205          consumer-1-646b2ebd-3a36-4e09-a7d4-658225492a03 /192.168.10.145 consumer-1
          fara          0          9217803         10000000        782197          consumer-1-646b2ebd-3a36-4e09-a7d4-658225492a03 /192.168.10.145 consumer-1


          fara          4          9942050         10000000        57950           -               -               -
          fara          3          8749827         10000000        1250173         -               -               -
          fara          1          8621461         10000000        1378539         -               -               -
          fara          2          9311260         10000000        688740          -               -               -
          fara          0          9274899         10000000        725101          -               -               -

I dont' understand why all the records are not consumed. any help please?

You are using an external tool to validate lag/consumption. What about your consumer processes? Can you confirm that you didn't get all the messages? Can you edit your post so you show the code you're using on the consumer side? — mjuarez
@mjuarez: I add the commands I run in the initial post. I see that some messages are not consumed (in 2 of the 5 consumers) in the statistics that the client is showing me up. — fara
Can you explain "1 fetcher threads by consumer (to consume just a partition)" - you'll have one consumer fetching all 5 partitions in this case; notice that you aren't specifying which partition that's being read from — OneCricketeer
@cricket_007: When the first instance of the consumer is up I might be fetching from all 5 partitions (if it is the only consumer running), but when the other 4 consumers are started, the partitions should be divided, one by consumer, and at the end all the records should be consumed, shouldn't be? — fara
For num-fetch-threads 5, sure, but not for only 1 fetcher thread — OneCricketeer

fara fara · Accepted Answer · 2018-10-28T18:53:22

I have fixed my issues reimplementing the ConsumerPerformance.scala class that is comming in version 1.1.0 in these two cases.

Duplicate records When I run different processes of ConsumerPerformance class at the same time I sometimes see duplicates because of this line (at least this is my impression).

https://github.com/apache/kafka/blob/1.1.0/core/src/main/scala/kafka/tools/ConsumerPerformance.scala#L161

The consumers stop working suddenly The timeout is hardoced to 1000 so if the count of the messages got in the pool takes more than 1 second the consumer stops consuming.

https://github.com/apache/kafka/blob/1.1.0/core/src/main/scala/kafka/tools/ConsumerPerformance.scala#L169

Kafka ConsumerPerformance

1 Answers