Implement filering for kafka messages

1
votes

I have started using Kafka recently and evaluating Kafka for few use cases.

If we wanted to provide the capability for filtering messages for consumers (subscribers) based on message content, what is best approach for doing this?

Say a topic named "Trades" is exposed by producer which has different trades details such as market name, creation date, price etc.

Some consumers are interested in trades for a specific markets and others are interested in trades after certain date etc. (content based filtering)

As filtering is not possible on broker side, what is best possible approach for implementing below cases :

  1. If filtering criteria is specific to consumer. Should we use Consumer-Interceptor (though interceptor are suggested for logging purpose as per documentation)?
  2. If filtering criteria (content based filtering) is common among consumers, what should be the approach?

Listen to topic and filter the messages locally and write to new topic (using either interceptor or streams)

2
apache-kafkakafka-consumer-apikafka-producer-api
What have you done so far? – Mahendra Gunawardena
Not sure what you aim for. You cannot do any filtering on the broker side if this would be the goal. Or do you mean to consume a topic, filter it, and write back to a new topic? Can you elaborate your question? – Matthias J. Sax
@MatthiasJ.Sax, I have updated post and provided example. – RGoyal
@MahendraGunawardena, Edited post for more details – RGoyal

2 Answers

1
votes

If I understand you question correctly, you have one topic and different consumer which are interested in specific parts of the topic. At the same time, you do not own those consumer and want to avoid that those consumer just read the whole topic and do the filtering by themselves?

For this, the only way to go it to build a new application, that does read the whole topic, does the filtering (or actually splitting) and write the data back into two (multiple) different topics. The external consumer would consumer from those new topics and only receive the date they are interested in.

Using Kafka Streams for this purpose would be a very good way to go. The DSL should offer everything you need.

As an alternative, you can just write your own application using KafkaConsumer and KafkaProducer to do the filtering/splitting manually in your user code. This would not be much different from using Kafka Streams, as a Kafka Streams application would do the exact same thing internally. However, with Streams your effort to get it done would be way less.

I would not use interceptors for this. Even is this would work, it seems not to be a good software design for you use case.

0
votes

Create your own interceptor class that implements org.apache.kafka.clients.consumer.ConsumerInterceptor and implement your logic in method 'onConsume' before setting 'interceptor.classes' config for the consumer.