Kafka Streams use case

Question

I am building a simple application which does below in order -

1) Reads messages from a remote IBM MQ(legacy system only works with IBM MQ)

2) Writes these messages to Kafka Topic

3) Reads these messages from the same Kafka Topic and calls a REST API.

4) There could be other consumers reading from this topic in future.

I came to know that Kafka has the new streams API which is supposed to be better than Kafka consumer in terms of speed/simplicity etc. Can someone please let me know if the streams API is a good fit for my use case and at what point in my process i can plug it ?

Answer 1

It is true that Kafka Streams API has a simple way to consume records in comparison to Kafka Consumer API (e.g. you don't need to poll, manage a thread and loop), but it also comes with a cost (e.g. local data store - if you do stateful processing).

I would say that if you need to consume records one by one and call a REST API use the Consumer API, if you need stateful processing, query the topic state, etc. use the Streams API.

For more info take a look to this blog post: https://balamaci.ro/kafka-streams-for-stream-processing/

Answer 2

1) Reads messages from a remote IBM MQ (legacy system only works with IBM MQ)

2) Writes these messages to Kafka Topic

I'd use Kafka's Connect API for (1) and (2).

3) Reads these messages from the same Kafka Topic and calls a REST API.

You can use the Streams API as well as the lower-level Consumer API of Kafka, depending on what you'd prefer.

4) There could be other consumers reading from this topic in future.

This works out-of-the-box -- once data is stored in a Kafka topic according to step 2, many different applications and "consumers" can read this data independently.

Answer 3

Looks like you are not doing any processing/transformation once you consume you message from your IBM MQ or even after your Kafka Topic.

First one -> from IBM Mq to your Kafka Topic is kind of a pipeline and Secondly -> You are just calling the REST API(I assume w/o any processing)

Considering these facts it seems to be a good fit for using Simple consumer.

Let's not use a technology only because it's there :)