Data Enrichment using kafka streams, KStream-GlobalKtable Join

Question

I have a scenario where i create a Kafka KStream by reading data from a kafka topic. KStream records are of key=null, value= {a json object} e.g.

null: { "ID":"1", "name":"XDFER"}
null: { "ID":"1", "name":"TRAFD"}

The real names are stored in a GlobalKTable as:

XDFER : "john"
TRAFD : "albert"

I want to perform data enrichment so that, final result is:

null: { "ID":"1", "name":"john"}
null: { "ID":"1", "name":"albert"}

I started reading about Kafka Stream Applications, in every tutorial/Example, the data enrichment is done comparing key's from KStream and GlobalKTable. I my case I need to compare an item from the value of KStream record with a key in GlobalKTable. Any ideas or examples how this can be achieved.

Answer 1

Input records for the stream with a null key or a null value are ignored and do not trigger the join.

Hence, you need to rekey the stream so that name can be used as key.

stream.selectKey(v-> v.get("name"))

Once rekeyed, you can join stream with GlobalKTable.

You can read the detailed behaviour here :

https://kafka.apache.org/20/documentation/streams/developer-guide/dsl-api.html#kstream-globalktable-join