Flink for embedded stream processing in OSGi

1
votes

I would like to use Apache Flink to process event inside an application. My tests on a standalone JVM worked reasonably well though flink is a really big dependency.

I also tried to get it running in OSGi but gave up for now because of the many dependencies.

So my question is: How small can I make Flink. I currently tried with the maven dependency on flink-streaming-java.

Unfortunately this depends on or embeds (only listing the questionable ones):

  • flink-shaded-hadoop2
  • kryo
  • zookeeper
  • netty
  • jetty
  • apache http client
  • apache http core
  • scala
  • akka
  • jackson

It also looks like several jars embed the same libs again and again. Like some google libs and asm.

So is there some way to get a slimmmer version of flink for local usage that does not depend on so many libs?

1
osgiapache-flink

1 Answers

1
votes

Many of the dependencies are required for Apache Flink's primary use-cases namely, distributed stream and batch processing.

  • Zookeeper for high-availability in case of (process) failures
  • Netty for data network transfer
  • Jetty for monitoring via REST API and web dashboard
  • Akka (and transitively Scala) for coordination of distributed processes

Most of these libraries are tightly coupled with the system and cannot be easily switched off or excluded.

I am sorry, there is no stripped down version for local stream processing.