Spring-Kafka vs. Spring-Cloud-Stream (Kafka)

38
votes

Using Kafka as a messaging system in a microservice architecture what are the benefits of using spring-kafka vs. spring-cloud-stream + spring-cloud-starter-stream-kafka ?

The spring cloud stream framework supports more messaging systems and has therefore a more modular design. But what about the functionality ? Is there a gap between the functionality of spring-kafka and spring-cloud-stream + spring-cloud-starter-stream-kafka ? Which API is better designed?

Looking forward to read about your opinions

2
springspring-integrationspring-cloud-streamspring-kafka
Maybe I haven't made myself clear, but wanting to know about a gap in terms of functionality is something you can't easily extract reading the docs. The foundation and api of the libraries / frameworks may differ but are they offering the same functionality? For example : is the spring-kafka API/functionality richer when using only kafka? – Eike Behrends
Well, each of them are a bit for different purposes. To hard to answer shortly. See their project pages for more info: projects.spring.io/spring-integration, projects.spring.io/spring-kafka, cloud.spring.io/spring-cloud-stream – Artem Bilan
I am aware of the advantages using the concept of binders but I am simply asking myself if there's a tradeoff, since it's build on top of spring-kafka and using it's own API. Every upcoming and new functionality in spring-kafka has somehow be "mapped" in the concept of spring cloud stream. Furthermore the API has to support multiple binder like rabbitmq and that's why the API has to be more abstract / generic. Please correct me if I am wrong but that's why I am asking myself if there is (or always will be) a gap in terms of functionality and if it is better to simply use spring-kafka – Eike Behrends
Well, that’s true. Since Binder API should be as generic as possible for any Binder implementation, there is definitely something missed from the target protocol specifics. If you really need something not implemented in the Kafka Binder, stay with just Spring Kafka and if you need more control over Consumer poll, go down to just raw Apache Kafka. In most streaming scenarios the auto-configuration from the Spring Cloud Stream is really enough – Artem Bilan
Thanks for your time and comments! – Eike Behrends

2 Answers

16
votes

Spring Cloud Stream with kafka binder rely on Spring-kafka. So the former has all functionalities supported by later, but the former will be more heavyweight. Below are some points help you make the choice:

  1. If you might change kafka into another message middle-ware in the future, then Spring Cloud stream should be your choice since it hides implementation details of kafka.
  2. If you want to integrate other message middle with kafka, then you should go for Spring Cloud stream, since its selling point is to make such integration easy.
  3. If you want to enjoy the simplicity and not accept performance overhead, then choose spring-kafka
  4. If you plan to migrate to public cloud service, then use spring cloud stream which is part of spring cloud family.
12
votes

Use Spring Cloud Stream when you are creating a system where one channel is used for input does some processing and sends it to one output channel. In other words it is more of an RPC system to replace say RESTful API calls.

If you plan to do an event sourcing system, use Spring-Kafka where you can publish and subscribe to the same stream. This is something that Spring Cloud Stream does not allow you do do easily as it disallows the following

public interface EventStream {
    String STREAM = "event_stream";

    @Output(EventStream.STREAM)
    MessageChannel publisher();

    @Input(EventStream.STREAM)
    SubscribableChannel stream();
}

A few things that Spring Cloud Stream helps you avoid doing are:

  • setting up the serializers and deserializers