Kafka: Can we have consumers subscribing to same topic but have different pipelines inside the topic?

Question

I have 200 Kafka consumers who can do either of these things, 1. They can subscribe to 200 different topics, and will consume messages that are sensitive. 2. All 200 consumers can subscribe to a single topic.

Problem: 1. Is it a good design to create 200 or large number of topics? 2. In second scenario, how we will achieve the implementation where the messages published to the topic should be sent to a particular consumer only based on some parameter.

What we used to do is to have a single topic with a large number of partitions, the producers were forced to use some dedicated partition and consumers were consuming from these dedicated partitions. hence avoiding many topics, though there were some scaling challenges later on, but just giving a thought — Santosh Joshi

shazin shazin · Accepted Answer · 2020-04-09T05:20:27

Is it a good design to create 200 or large number of topics?

Kafka uses replicated files for topics and since it uses offset based message transmission number of topics has no direct effect on performance so it is not a problem.

In second scenario, how we will achieve the implementation where the messages published to the topic should be sent to a particular consumer only based on some parameter.

You can not do this based on a parameter, if you need a message to be delivered to exactly one consumer then you need to group all the consumers in a single consumer group and make the whole group listen to a topic. This way a message will be consumed by only one consumer in that consumer group.

If you need to sequential (ordered) message consumption then you need to create your Kafka topic with only 1 partition.

Kafka: Can we have consumers subscribing to same topic but have different pipelines inside the topic?

1 Answers