Flink with StatsD counter metrics comparision

Question

Talking about Counters with respect to StatsD, the way it works is that you keep posting a value of a counter eg. numOfRequests|c:1 whenever app get a request to the StatsD Daemon. The daemon has a flush interval set, when it pushes the aggregate of this counter in that period to an external backend. Additionally it also resets the counter to 0.

Trying to map this to Flink Counters.

1. Flink counters only has inc and dec methods so till the reporting time comes, app can call inc or dec to change the value of a counter.
2. At the time of reporting the latest value of counter is reported to StatsD daemon but the Flink counter value is never reset(Not able to find any code).

So should the flink counter be reported as a gauge value to StatsD. Or Flink does reset the counters?

Answer 1

Flink Counters are basically kind of gauge values. The counters are never reset. So numRecordsIn/numRecordsOut or any other counter metrics kept increasing over the lifetime of a job. If you want to visualise the count over a duration, you need to calculate and send the delta to the external backend yourself in the report method or use the external backend solution capabilities to graph the delta.

We use Datadog and used following to graph the delta over a duration: diff(sum:numRecordsIn{$app_name,$env}.rollup(max))