Kafka: How can the Producer send only to a certain Consumer/GroupId

Question

Since I started using Kafka, I noticed that Kafka is broadcasting the messages on every consumer or group. How can I get a producer to send the message to just one certain consumer? I'm talking about One Topic System thanks

Answer 1

How Kafka works in terms of consuming messages depends on the number of partitions for the topic and consumer groups. When consumers are part of the same consumer groups listening for the same topic, each consumer will receive messages from one or more partitions but not same messages as other consumers. So for example : if you have a topic with 4 partitions (p1, p2, p3, p4) and 3 consumers (c1, c2 and c3) in the same consumer group then after partitions assignment you could have : c1 receiving from p1 and p2, c2 from p3, c3 from p4. As you can see consumers don't receive the same messages (i.e. c3 doesn't receive messages from p1 and p2 as c1). If you assigne consumers to different consumer groups (so c1 to g1, c2 to g2 and c3 to g3) listening for the same topic (with 4 partitions) they will get all the same messages (c1 from p1, p2, p3, p4 and the same for c2 and c3). So different consumer groups allow to have a publish/subscribe mechanims (broadcasting messages). If you want a producer sends messages to only a specific customer one of the possibilities is to have a topic with number of partitions = number of consumers then not using consumer groups but direct assignment so that a consumer can ask for getting messages from a specific partition (i.e. c1 wants messages only from p1). In this case the producer needs to send a message to that partition but ... it doesn't happen with the default partitioner (which is round robin). You should write your own partitioner based on the destination consumer for sending the message to the right partition (so, if you know that message "m" should go to c1 you should have a partitioner logic which is able to get info from message "m", process it and having p1 as destination which is the c1 listening partition). Another trivial solution could be having one topic - one partition for each consumer. You can do that in different ways but Kafka, in general, isn't done to do that without any work on your side.

Answer 2

1. create "number of partitions=number of consumers"
2. publish messages with key or publish messages to specific partition and consume same.

this way you can achieve what you are looking for

Answer 3

Each consumer opts to pull messages from a given topic. If you don't want a consumer to receive messages from a given topic, then just configure it not to pull from that topic.

If you want to filter a topic to produce a topic with only certain messages on in order to consume only those, you can use Kafka Streams, or KSQL to do this.

Answer 4

• Kafka Producers does not send any message to any Kafka Consumers.
• Kafka Producers publish messages into a particular Topic.
• Every single Kafka Consumer process belongs to a specific groupid and is registered in a particular topic.

Then the consumer can start consuming the data from that particular topic so that if you want a consumer process to get data from only one topic, then register it to just one.

See how in this example a Consumer is being registered in 2 topics:

consumer.subscribe(Arrays.asList("fast-messages", "summary-markers"));

Remember if you have 2 separated Consumers processes consuming the same Topic and they belong to different groups (groupId) then they will bring the same data because they will not share the offset at all.