Apache Flink vs Twitter Heron?

9
votes

There are a lot of questions comparing Flink vs Spark Streaming, Flink vs Storm and Storm vs Heron.

The origin of this question is from the fact that both Apache Flink and Twitter Heron are true stream processing frameworks (not micro-batch, like Spark Streaming). Storm has been decommissioned by Twitter last year and they're using Heron instead (which is basically Storm reworked).

There are nice presentations by Slim Baltagi on Flink and Flink vs Spark: https://www.youtube.com/watch?v=G77m6Ou_kFA

Nice research by Ilya Ganelin on various streaming frameworks: https://www.youtube.com/watch?v=KkjhyBLupvs

Pretty interesting thoughts on Flink vs Storm: What is/are the main difference(s) between Flink and Storm?

But I haven't seen any comparison of new Storm/Heron vs Apache Flink.

Both of the projects are pretty young, both support using previously written Storm applications and many other things. Flink is more fitting into Hadoop ecosystem, Heron is more into Twitter based ecosystem stack.

Any thoughts?

2
twitterapache-stormapache-flinkflink-streamingheron
Did you have a look Twitter's paper about Heron? It describes the main differences between both systems. Keep in mind though, that Storm made large progress since that paper was published: dl.acm.org/citation.cfm?id=2742788 If you understand the differences between Storm and Heron and know something about Flink, you see how it differs from Flink. – Matthias J. Sax

2 Answers

2
votes

All of the points in the referenced article comparing Apache Flink and Apache Storm also apply to Twitter's Heron. Heron provides exactly the same type of semantics and functionality as Storm. Heron is really best understood simply as a re-implementation of Storm that better fits Twitter's operational requirements.

0
votes

Heron, Stream processing engine developed by twitter and donated to Apache on 26th FEB 2018. As per Twitter, the throughput is 10–14x higher than that of Storm in all experiments, Similarly latency is 5-15x lower than Storm’s latency.

Other then throughput and latency it provides

  • Easy debugging(Every task runs in process-level isolation).
  • Handling spikes and congestion(using backpressure mechanism).
  • Fully backward compatible with Storm which means only pom file changes required.

https://blog.twitter.com/engineering/en_us/a/2015/flying-faster-with-twitter-heron.html https://apache.github.io/incubator-heron/