Debezium kafka connect connector not updating successfully

Question

I am trying to CDC using Debezium and Kafka. I have registered a connector with 3 tables in table.whitelist property and it is working properly. Here is the configuration of running connector.

{"name":"xoom-eds-extactor","config":{"connector.class":"io.debezium.connector.mysql.MySqlConnector","database.user":"debezium","database.server.id":"1","database.hostname":"qa514dtc001.ord.qa001.xoom.com","database.password":"xoom123","database.history.kafka.bootstrap.servers":"eds-kafka1:9092","database.history.kafka.topic":"dbhistory.inventory","name":"xoom-eds-extactor","database.server.name":"MySQL-Database-Docker","database.port":"3306","include.schema.changes":"true","table.whitelist":"xoom_eds_extraction_src_db_mock.postal_code_mock,xoom_eds_extraction_src_db_mock.account_number_mock,xoom_eds_extraction_src_db_mock.credit_card_number_mock,xoom_eds_extraction_src_db_mock.ip_address_details_mock,xoom_eds_extraction_src_db_mock.university_details_mock"},"tasks":[{"connector":"xoom-eds-extactor","task":0}],"type":"source"}

Now I want to update the connector and add two more tables from the same database. I am using the following command and config file to update the connector.

Command :

curl -X PUT -H "Content-Type: application/json" --data @xoom-eds-extactor-conf.json "http://eds-kafka1:8083/connectors/xoom-eds-extactor/config"

Config json :

{
   "name":"xoom-eds-extactor",
   "connector.class":"io.debezium.connector.mysql.MySqlConnector",
   "database.hostname":"qa514dtc001.ord.qa001.xoom.com",
   "database.port":"3306",
   "database.user":"debezium",
   "database.password":"xoom123",
   "database.server.id":"1",
   "database.server.name":"MySQL-Database-Docker",
   "database.history.kafka.bootstrap.servers":"eds-kafka1:9092",
   "database.history.kafka.topic":"dbhistory.inventory",
   "include.schema.changes":"true",
"table.whitelist":"xoom_eds_extraction_src_db_mock.postal_code_mock,xoom_eds_extraction_src_db_mock.account_number_mock,xoom_eds_extraction_src_db_mock.credit_card_number_mock,xoom_eds_extraction_src_db_mock.ip_address_details_mock,xoom_eds_extraction_src_db_mock.university_details_mock"
}

This executes successfully but when I list the Kafka topics only the last 3 topics are present 2 new topics are not added.

MySQL-Database-Docker
MySQL-Database-Docker.xoom_eds_extraction_src_db_mock.account_number_mock
MySQL-Database-Docker.xoom_eds_extraction_src_db_mock.credit_card_number_mock
MySQL-Database-Docker.xoom_eds_extraction_src_db_mock.postal_code_mock
__consumer_offsets
connect-configs
connect-offsets
connect-status
dbhistory.inventory

Can anyone out there help me with this? Thanks in advance.

james.bondu james.bondu · Accepted Answer · 2019-05-30T07:24:49

As far as I know, there was an old pending PR related to this issue and it should have been fixed in 0.9.1 release. But if you are using some old version you can simply follow these steps.

Add the new tables, but don't write to them yet
Shut down the connector
Change the filter configuration and set snapshot mode to "schema_only_recovery"
Delete (or rename) the existing DB history topic
Restart the connector; this will re-create the internal history of the DB schema and then continue streaming from the previously recorded offset
Begin writing to the new tables

At least this is what we do(we are still using 0.8.2).

Source : debezium-google-group.

Debezium kafka connect connector not updating successfully

1 Answers