Apache Flink: What's the difference between side outputs and split() in the DataStream API?

Question

Apache Flink has a split API that lets to branch data-streams:

val splited = datastream.split { i => i match {
   case i if ... => Seq("red", "blue")
   case _ => Seq("green")
}}

splited.select("green").flatMap { .... }

It also provides a another approach called Side Output( https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/side_output.html) that lets you do the same thing!

What's the difference between these two way? Do they use from a same lower-level construction? Do they cost the same? When and how we should select one of them?

Fabian Hueske Fabian Hueske · Accepted Answer · 2018-07-20T11:52:57

The split operator is part of the DataStream API since its early days. The side output feature as added later and offers a superset of split's functionality.

split creates multiple streams of the same type, the input type. Side outputs can be of any type, i.e., also different from the input and the main output.

Internally, split adds dedicated operator that just splits the stream. Side outputs are defined within an operator (typically a ProcessFunction or window operator) that apply arbitrary logic and feature multiple outputs. I would not expect this to result in a significant performance difference.

A common use case for side outputs is to filter out invalid (or late) records and pass them unmodified to the side, e.g., to process them later. Such an operator has a regular output with the desired result type and a side output with its input type. This logic would be cumbersome to implement using split.

Apache Flink: What's the difference between side outputs and split() in the DataStream API?

2 Answers