How to monitor Kafka broker using jmxtrans?

Question

Kafka 0.8.1.1 (kafka_2.8.0-0.8.1.1.tgz)

I am using jmxtrans to do JMX monitoring of a Kafka instance (which is running in docker). Unfortunately, kafka metrics are not being returned.

I have tried a few things to debug this and know that kafka is running correctly (I can produce/consume messages successfully) have concluded that jmxtrans does return JMX metrics (for example, java.lang:type=Memory, attribute=HeapMemoryUsage returns correct data) so the general kafka and JMX capability seems to be working. Also, I can access the metrics when I use jconsole -- the metrics seem to be captured with data in all relevant fields.

When I try jmxtrans using the following configuration, unfortunately, I do not get any information back (no data at all in fact). I believe the metrics are supposed to be captured based upon the kafka documentation ("kafka.server:type=BrokerTopicMetrics", attribute="MessagesInPerSec")

The following is the jmxtrans configuration that I used:

{
    "servers" : [ {
        "port" : "9999",
        "host" : "10.0.1.201",
        "queries" : [ {
            "outputWriters" : [ {
                "@class" : "com.googlecode.jmxtrans.model.output.StdOutWriter",
                "settings" : {
                 }
             } ],
             "obj" : "kafka.server:type=BrokerTopicMetrics",
             "attr" : [ "MessagesInPerSec" ]
         } ],
         "numQueryThreads" : 2
     } ]
}

I am not sure why data is not returned. Maybe I setup an invalid jmxtrans configuration or perhaps I am specifying the metric improperly.

Any help is appreciated.

Answer 1

After a lot of experimentation, I have now resolved the question. For completeness, below is how I resolved the problem.

It appears that I specified the "obj" value incorrectly.

The CORRECT obj value (an example) is as follows:

"obj": "\"kafka.server\":type=\"BrokerTopicMetrics\",name \"AllTopicsLogBytesAppendedPerSec\"",
"attr": [ "Count" ]

Note that the "obj" value requires additional quotes. This seems unusual and different than the normal pattern I have seen (no quotes) for other JMX obj values.

JMXTRANS provided valid output after putting the correct (quoted) values in the obj string...

Answer 2

As I could've found out in ./bin/jmxtrans.sh, by default the stdout/log file is /dev/null.

LOG_FILE=${LOG_FILE:-"/dev/null"}

That's why it's important to set the env var to something you can use to see the output:

LOG_FILE=log.txt ./bin/jmxtrans.sh start kafka.json

I'm using the following kafka.json configuration file:

{
  "servers" : [ {
    "port" : "10101",
    "host" : "localhost",
    "queries" : [ {
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.StdOutWriter",
        "settings" : {
        }
      } ],
      "obj" : "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=testowo",
      "attr" : [ "Count" ]
    } ],
    "numQueryThreads" : 2
  } ]
}

When you start jmxtrans it will query the broker with JMX on localhost:10101 about Count attribute for testowo topic. It will print the result out to the file LOG_FILE every 60 secs (you can change it using SECONDS_BETWEEN_RUNS env var), e.g.

LOG_FILE=log.txt SECONDS_BETWEEN_RUNS=5 ./bin/jmxtrans.sh start kafka.json

You may want to use other writers of jmxtrans so the output is not intermingled, e.g.

{
  "servers" : [ {
    "port" : "10101",
    "host" : "localhost",
    "queries" : [ {
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.KeyOutWriter",
        "settings" : {
          "outputFile" : "testowo-counts.txt",
          "maxLogFileSize" : "10MB",
          "maxLogBackupFiles" : 200,
          "delimiter" : "\t",
          "debug" : true
        }
      } ],
      "obj" : "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=testowo",
      "attr" : [ "Count" ]
    } ],
    "numQueryThreads" : 2
  } ]
}

And the last but not least, to set the JMX port to a known value use JMX_PORT env var when starting a Kafka broker using ./bin/kafka-server-start.sh, i.e.

JMX_PORT=10101 ./bin/kafka-server-start.sh config/server.properties