Debezium connector with TimescaleDB extension

Question

I'm having trouble with detecting changes on Postresql hyper table(TimescaleDB extension).

Setup:

I have Postresql(ver 11.10) installed with TimescaleDB(ver 1.7.1) extension. I have 2 tables I want to monitor them with Debezium(ver 1.3.1) connector installed on Kafka Connect for the purpose CDC(Capture Data Change).

Tables are table1 and table2hyper, but table2hyper is hypertable.

After creating Debezium connector in Kafka Connect I can see 2 topics created(one for each table):

(A) kconnect.public.table1
(B) kconnect.public.table2hyper

When consuming messages with kafka-console-consumer for topic A, I can see the messages after a row update in table1.

But when consuming messages from topic B(table2hyper table changes), nothing is emitted after for example a row update in table2hyper table.

Initialy Debezium connector does a snapshot of rows from table2hyper table and sends them to topic B(I can see the messages in topic B when using kafka-console-consumer), but changes that I do after the initial snapshot are not emitted.

Why am I unable to see subsequent changes(after initial snapshot) from table2hyper?

Connector creation payload:

{
    "name": "alarm-table-connector7",
    "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "tasks.max": "1",
        "database.hostname": "xxx",
        "database.port": "5432",
        "database.user": "xxx",
        "database.password": "xxx",
        "database.dbname": "xxx",
        "database.server.name": "kconnect",
        "database.whitelist": "public.dev_db",
        "table.include.list": "public.table1, public.table2hyper",
        "plugin.name": "pgoutput",
        "tombstones.on.delete":"true",
        "slot.name": "slot3",
        "transforms": "unwrap",
        "transforms.unwrap.type":"io.debezium.transforms.ExtractNewRecordState",
        "transforms.unwrap.drop.tombstones":"false",
        "transforms.unwrap.delete.handling.mode":"rewrite",
        "transforms.unwrap.add.fields":"table,lsn,op"
    }
}

Thx in advance!

Boris Šipoš Boris Šipoš · Accepted Answer · 2021-02-01T23:40:12

After trying for a while, I did not succeed to stream data from hyper table with Debezium connector. I was using version 1.3.1. and upgrade to latest 1.4.1. did not help.

However, I did succeed with Confluent JDBC connector.

As far as my research and testing goes, this is the conclusion and feel free to correct me if necessary:

Debezium works on ordinary tables on INSERT, UPDATE and DELETE events
Confluent connector captures only INSERT events(unless you combine some columns for detecting changes) and works on ordinary and hyper(TimescaleDB) tables.

Debezium connector with TimescaleDB extension

2 Answers