I have written a java program for Apache Flink. I want to implement some calculations periodically so I used timers to trigger recalculations. For example, every 10s the implementation spends around 1s (soft threshold) to update the state. Because the job should process also regular records, usually in pick times the implementation produces some backpressure and data delays. So I am wondering if you can access any backpressure metrics within the operator to skip the recalculation if the task slot (SubTask) performing recalculation is back pressured? I was searching it in the getRuntimeContext().getMetricGroup() , but I didn't find anything useful.
0
votes
1 Answers
0
votes
One idea would be to have something outside of Flink monitoring the relevant metrics, and have this external process send in an event to trigger a recalculation when it judges that the time is right.
There are several related metrics you might use for this, and isBackPressured is probably the least useful:
isBackPressured- available since Flink 1.10
- based on a point-in-time sample
- returns true or false
idleTimeMsPerSecond- available since Flink 1.11
- in 1.11 and 1.12, idle means not busy, and includes time spent backpressured
- since 1.13, idle is distinct from backPressured
backPressuredTimeMsPerSecond,busyTimeMsPerSecond- new in Flink 1.13
- more accurate than
isBackPressured - backpressured: blocked waiting for an output buffer
- idle: no data to process
- busy: not idle or backpressured
