I have used JMX as a metric reporter to get the Flink metrics, but is there any way to get it as output in the terminal?
I want to plot numRecordsInPerSecond for each operator for performance analysis, how can I do it?
I have seen some example of accumulators but It did not gave me the proper insight how do I do performance analysis of Flink. I will give you an example here
This is the execution plan of my Flink program, I have multiple questions, but I want to ask basic one
how can I measure the latency by each operator and then add it up to compute total latency for a Complex event.
how do I measure output throughput? Currently, I have written some code in select function which counts # of complex events seen and time Flink engine is up. Is this the best way to do it ?
But the basic question remains, which is how can I get the output for system metrics mentioned at Flink metrics via code to be shown in terminal output, as I want to plot graph for performance and the problem with JMX is that it shows me metrics on demand in the sense , I see the values as I click that particular metric in JMX console, which is not perfect fit for analyzing the system.
P.S - I have found one question asked at StackOverflow for computing throughput and latency and the answer was something like this
private static class MyMapper extends RichMapFunction<String, Object> {
private transient Meter meter;
@Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
this.meter = getRuntimeContext()
.getMetricGroup()
.meter("myMeter", new DropwizardMeterWrapper(new com.codahale.metrics.Meter()));
}
@Override
public Object map(String value) throws Exception {
this.meter.markEvent();
return value;
}
}
I have added above class in my code as well but haven't seen any output, and I also wonder how this code will show throughput or latency as we have not mentioned for which operator we want to find latency? For example, I want to find throughput for an operator somewhere in the middle of execution plan rather than at end of plan, will the above code do it for me?
