How to replicate microservices when consuming same kafka topic?

Question

Consider a "dockerized" application that consumes a single kafka topic and produces on many kafka topics that are consumed by other applications, let's call these ones services.

The application queries the services by producing a message into their unique consumer topic.

The query message is composed by a header and a payload. Among other things, the header contains the topic information where the response should be produced.

Services produce a response and send it to the application consumer topic that completes the request cycle.

Consumers and producers are set at startup and a "ping" message is sent to initialize/authorize topic producers.

All works great with single docker instances.

Now I want to scale up both sides and be able to run several docker instances of the application and the services.

What would be the best approach in terms of topic names and partition replication, considering that the response must always reach the application instance responsible for the request?

Answer 1

I'm not going to talk about partitioning here, as that comes down to your own requirements for ordering and I can't really infer enough information to make any educated guesses on that.

With regards to horizontally scaling your consumers / producers, I can hopefully help.

Simple Approach: You mention that there's a header on your messages, I'd include extra meta data, such as "SENDER". Post all messages to a single topic, but have each service register as a consumer group - using the service name.

If your service gets a message that isn't for it (Checking SENDER, just ACK). Otherwise, handle the message.