Kafka Connect: streaming changes from Postgres to topics using debezium

Question

I'm pretty new to Kafka and Kafka Connect world. I am trying to implement CDC using Kafka (on MSK), Kafka Connect (using the Debezium connector for PostgreSQL) and an RDS Postgres instance. Kafka Connect runs in a K8 pod in our cluster deployed in AWS.

Before diving into the details of the configuration used, I'll try to summarise the problem:

Once the connector starts, it sends messages to the topic as expected (snahpshot)
Once we make any change to a table (Create, Update, Delete), no messages are sent to the topic. We would expect to see messages about the changes made to the table.

My connector config looks like:

{
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.user": "root",
    "database.dbname": "insights",
    "slot.name": "cdc_organization",
    "tasks.max": "1",
    "column.blacklist": "password, access_key, reset_token",
    "database.server.name": "insights",
    "database.port": "5432",
    "plugin.name": "wal2json_rds_streaming",
    "schema.whitelist": "public",
    "table.whitelist": "public.kafka_connect_cdc_test",
    "key.converter.schemas.enable": "false",
    "database.hostname": "de-test-sre-12373.cbplqnioxomr.eu-west-1.rds.amazonaws.com",
    "database.password": "MYSECRETPWD",
    "value.converter.schemas.enable": "false",
    "name": "source-postgres",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "snapshot.mode": "initial"
}

We have tried different configurations for the plugin.name property: wal2josn, wal2json_streaming and wal2json_rds_streaming.

There's no problem of connection between the connector and the DB as we already saw messages flowing through as soon as the connector starts.

Is there a configuration issue with the connector described above that prevent us to see messages related to new changes appearing in the topic?

Thanks

Not really, the only things we can see from the pod logs is: `` [2021-07-09 09:18:06,632] INFO WorkerSourceTask{id=source-postgres-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) — AlessioG
@IskuskovAlexander we started to think that it might be related to how WAL is set up for our RDS Postgres instance. But the configs look fine so far — AlessioG

broch broch · Accepted Answer · 2021-07-12T08:03:23

Your connector config looks a bit confusing. I'm pretty new to Kafka as well so I don't really know the issue but this is my connector config that works for me.

{
   "name":"<connector_name>",
   "config": {
      "connector.class":"io.debezium.connector.postgresql.PostgresConnector",
      "database.server.name":"<server>",
      "database.port":"5432",
      "database.hostname":"<host>",
      "database.user":"<user>",
      "database.dbname":"<password>",
      "tasks.max":"1",
      "database.history.kafka.boostrap.servers":"localhost:9092",
      "database.history.kafka.topic":"<kafka_topic_name>",
      "plugin.name":"pgoutput",
      "include.schema.changes":"true"
   }
}

If this configuration didn't work aswell, try look up the log console; sometimes the error isn't the last write of the console

Kafka Connect: streaming changes from Postgres to topics using debezium

1 Answers