Does Apache Beam support iterative algorithms, just like Apache Flink?

3
votes

Apache Flink supports iterative transformations: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/batch/iterations.html.

Is this also supported using Apache Beam or can we achieve the same thing using Beam concepts?

1
apache-flinkapache-beam

1 Answers

2
votes

I have posted this question on the user mailing list of Apache Beam.

There is an open ticket for this (BEAM-106), but it is not yet started.

There are two solutions:

  • Unroll the loop yourself, which of course limits the number of iterations. Be careful with the number of transformations you create. This can be limited by the runner.
  • Create multiple pipelines and store the intermediate data. This allows for easier debugging, but comes with overhead of storing and loading the data.