Kafka Streams - KTable from topic with retention policy

5
votes

I'm experimenting with kafka streams and I have the following setup:

  • I have an existing kafka topic with a key space that is unbounded (but predictable and well known).
  • My topic has a retention policy (in bytes) to age out old records.
  • I'd like to materialize this topic into a Ktable where I can use the Interactive Queries API to retrieve records by key.

Is there any way to make my KTable "inherit" the retention policy from my topic? So that when records are aged out of the primary topic, they're no longer available in the ktable?

I'm worried about dumping all of the records into the KTable and having the StateStore grow unbounded.

One solution that I can think of is to transform into a windowed stream with hopping windows equal to a TimeToLive for the record, but I'm wondering if there's a better solution in a more native way.

Thanks.

1
apache-kafkaapache-kafka-streams

1 Answers

5
votes

It is unfortunately not support atm. There is a JIRA though: https://issues.apache.org/jira/browse/KAFKA-4212

Another possibility would be to insert tombstone messages (<key,null>) into the input topic. The KTable would pick those up and delete the corresponding key from the store.