How to Delete Record from Kafka global state store?

Question

Global state store during restoration will dump the data from the source topic(Which is consider as change-log topic for global-store).

for delete a record i do something like below

kvStore.put("key-1",null)

how Kafka know's that the record is delete and during restoration it will dump record from source topic (consider source topic has a record with key-1)

In my topology I have

an input topic -> T1
and a process is attached that read data from T1 and construct a key from the record and forward down to topic T2
and topic T2 is a source topic for global state store.

Example:

T1 i got data: {"id":'123', "name":"Mohit", "type":"insert"}
construct a key for record and forward Down to topic with key and value as T2 -> key: 123 and value: {"id":'123', "name":"Mohit"}

After that same key record come as type delete in data. T1 got data: {"id":'123', "name":"Mohit", "type":"insert"}

so i am forward record as like this

this.context.forward(key, null)
key: 123 value:null

Same is updated in state-store

I just want to know that during restoration with this record will be delete means i get null if i go get on store with key 123.

Not 100% sure if I understand the question. However, global stores are read-only and you cannot put/delete data directly (the global processor is only allowed to take the date from the topic as-is to update the store). — Matthias J. Sax

mike mike · Accepted Answer · 2020-05-13T10:54:28

State store changelogs are compacted topics. To delete a message out of a compacted topic, you ’ll need to do a put(key, null) operation. A message with a null value is called a tombstone and it will eventually be deleted by the topic cleaner.

Note, that the message will only be deleted (eventually) in the state store, but not in the input topic.

In the end, the record with key 123 should be removed completely from the state store.

How to Delete Record from Kafka global state store?

1 Answers