How to measure the performance metrics on kafka when pushing/consuming the message to/from the topic

3
votes

Have pushed the message using the producer. It pushed 100000 messages to the topic.

using the command: bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages 100000 --topic perfAtlasTopic get the below producer metrics.

start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec [2015-02-19 00:33:44,133] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties) 2015-02-19 00:33:44:020, 2015-02-19 00:33:46:822, 0, 100, 200, 9.54, 3.4035, 100000, 35688.7937

using the command: bin/kafka-consumer-perf-test.sh --zookeeper localhost:2181 --messages 100000 --topic perfAtlasTopic --threads 10. Get the below consumer metrics:

start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec 2015-02-19 00:53:13:480, 2015-02-19 00:53:53:128, 1048576, 289.0332, 8.3420, 100000, 2886.1695

Getting the above metric before the producer/consumer pushes/consumes the message to/from the topic.

Is there any way to measure the performance metrics while pushing/consuming the message to/from the topic?

Thanks

1
apache-kafka

1 Answers

1
votes

The number shows the metrics of producing/consuming the number of messages you specify (e.g., in your case, 100000). In your case, it shows the start/end time, fetch size, MB data consumed (total 100000 messages), MB/sec, total number of messages consumed (100000 of course) and # messages consumed/sec for producing/consuming 100000 messages. If you want to keep tracking the metrics while you stream the data, try the following: 

for i in `seq 1 1000`; do kafka-producer-perf-test.sh --broker-list localhost:9092 --messages 100000 --topic perfAtlasTopic; done

This will run the test 1000 times. You can change # of times you want to run.