flink backpressure monitoring

Question

From the beginning of our Flink project.

My cluster have suffered from low back-pressure because of heavy parsing code.

So I put monitoring script on the system which keep asking back-pressure status from the task manager. ( Which run every 20 seconds for getting the highest value and average )

By the way when I turn off the script running, I found increment of back pressure ratio become much less slower than when I using the script.

So is there any efficient way to get the back pressure status without losing performance ?

Answer 1

I don't believe requesting back-pressure every 20 seconds would have a major impact on your workflow's performance.

Separately, if you've got available CPU cycles, then increasing the parallelism of the function(s) doing the parsing would be the next thing to try, to improve throughput.