How to handle backpressure in flink streaming job?

1
votes

I am running a streaming flink job which consumes the streaming data from kafka and do some process over the data in flink map function and write the data to the Azure data lake and the elastic search. For map function I used a parallelism of one because I need to process the incoming data one by one over the list of data I maintain as a global variable. Now when I run the job as the flink starts to get the streaming data from kafka it's backpressure becomes high in the map function. Is there any settings or configurations I could do to avoid the backpressure in flink?

enter image description here

1
apache-flinkflink-streaming
Hello dhinesh and welcome to SO! I would like to advise you some SO documentation that may help you in asking a better qestion. For instance you may be interested about the topics we discuss here, the way to ask a 'good' question and the way to create a minimal, complete and verifiable example. If you find it useful, you can still edit your question. – John-Philip

1 Answers

3
votes

Backpressure on a given operator indicates that the next operator is consuming elements slowly. From your description it would seem that one of the sinks is performing poorly. Consider scaling up the sink, commenting-out a sink for troubleshooting purposes, and/or investigating whether you're hitting an Azure rate limit.