Flink - how to export flink's kafka connector offsets via JMX Reporter?

0
votes

According to here, I enable JMX Exporter's and was able to connect to it to see some metrics: jobmanager.Status, jobmanager.job. 

metrics.reporter.jmx.class: org.apache.flink.metrics.jmx.JMXReporter
metrics.reporter.jmx.port: 8789

However, I did not see metrics of KafkaConnector (commitedOffets, currentOffsets). How do I do this?

I want to get those offset from Flink to calculate

consumer lag = latest kafka partitions offsets - flink partition offsets.

1
apache-flinkflink-streaming

1 Answers

0
votes

I found the answer. JMX port config in flink-conf.yaml should be a range (9250-9260), not 1 port number. Flink needs at least 2 ports to export metrics for JobManager and TaskManager at the same time.

metrics.reporter.jmx.class: org.apache.flink.metrics.jmx.JMXReporter
metrics.reporter.jmx.port: 9250-9260

If only 1 port is specified (8789 as in the question), only metrics from JobManager is exported. Metrics of KafkaConnector belong to TaskManager's metrics hence will not appear.