What may cause long barrier alignment durations in Apache Flink jobs?

3
votes

I run my Flink job on YARN and I find a small number of subtasks encounter a long alignment duration.

What may probably cause this problem?

1
apache-flinkflink-streaming

1 Answers

6
votes

For exactly-once semantics, Flink aligns the streams at operators that receive multiple input streams, hence large alignment means the task manager receives some barrier(s) later than the other nodes.

Document about alignment can be found here, and there are ways to monitor checkpointing

To be more specific, the reasons may be:

  1. Data skew. Most of the data has been send to the large alignment duration node(s).
  2. Garbage collection: GC will greatly affect the checkpoint alignment.
  3. Long state access, i.e., take long time to put or get from state. For RocksDB, check whether there are index miss or cache miss problems.
  4. Network buffers problem.
  5. User code bug. For example, endless loop or other problems.