From Postgres table to KSQL table with updates tracking

Question

My task is transfer data from Postgres table to KSQL table (for future joins with streams). Let's imagine table has three records:

id | name   | description
-------------------------
1  | name1  | description1
2  | name2  | description2
3  | name3  | description3

It is easy to do by means of Kafka JdbcSourceConnector. But there is one little problem - data in table may be changed. Changes must be in KTable too.

According to documentation there is no way to track changes except bulk mode. But bulk mode takes absolutely all rows and inserts them into topic.

I thought to set up bulk mode for connector. Create a KSream for that topic. Create a KTable for that stream...

And here I do not know what to do. How to make sure changes in Postgres table were in KTable too?

OneCricketeer OneCricketeer · Accepted Answer · 2020-03-02T02:33:33

Bulk mode would work, you just define the key of the stream, then new bulk writes will update the KTable of the same key. In other words, you need to ensure the primary keys don't change in your database

Alternatively, Debezium is the CDC version of Kafka Connect.

JDBC source doesn't capture UPDATE queries, as you've stated.

Debezium will produce records that contain previous and new versions of the modified rows

From Postgres table to KSQL table with updates tracking

1 Answers