I am currently working on a Spring boot application using Spring Kafka Consumer API.
Each message I get on to a topic needs to be converted into a new object type with additional properties coming in from other topics. Currently, these other topics are not yet developed and we are using a mocked version of in-memory data for processing the requests.
For example, a new "shopping order" message arrives, but I am using mocked "Customer" object and mocked "item" object in order to process the order. The plan is to move to use real Customer topic and real Item topic.
Also, currently, the application is only Spring Kafka listeners for getting new order. The listeners invoke a spring bean method which processes the order and creates a new object to be written to another output topic named customer-order by using the same mocks I mentioned above.
We are currently thinking about evolving the architecture for this application. I have been reading up on Kafka streams. The documentation I have read online for streams only take simple examples such as word count, join etc. With my limited knowledge of streams, I don't envision using functionality such as calculating total etc.
I have thought of some options for the architecture...
- I was planning to retain the consumer API i.e. use Spring listeners implementation for receiving new order messages while using streams dependency just to create state stores that will eventually replace the mocked data. The idea is that the mocked data will eventually come from other topics. So in this approach, the "streams" part of Kafka will be used only for creating state store and not for processing incoming records.
- Use purely Kafka consumers API and use API calls to fetch data external to my topic. This is a less preferred option as I don't want to make an external API call for each new order.
- Use Kafka Streams for both reading new incoming orders and also for gathering and storing state. Also, make use of joins and merges so as to process data.
What do you suggest? 1,2, or 3? Is that a good idea to use Streams for this kind of solution? Is there any benefit in moving this implementation to use Kafka streaming? Or am I better of staying with 2.?