Debezium MS SQL Server connector issues

0
votes

Kafka newbie here.

Environment setup

OS: Ubuntu 20.04

Kafka is installed locally on the machine within the home folder. Both Kafka and Zookeeper services are running.

I am trying to configure the Kafka connect to connect to database hosted within a SQL Server container that I built with the command below

docker run --name SQL19 -p 1433:1433 -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=Password258*" -e "MSSQL_AGENT_ENABLED=True" -v ~/SqlDockerVol/userdatabase:/userdatabase -v ~/SqlDockerVol/sqlbackups:/sqlbackups -d mcr.microsoft.com/mssql/server:2019-latest

The container is running and can connect successfully.

I enabled the CDC on the database level and on the table level.

EXEC sys.sp_cdc_enable_db
GO
EXEC sys.sp_cdc_enable_table
    @source_schema = N'dbo',
    @source_name   = N'WorkQueue',
    @role_name     = NULL,
    @supports_net_changes = 1

Connector properties file

name= DebeziumSQLServerconnector 
connector.class= io.debezium.connector.sqlserver.SqlServerConnector 
database.hostname=linuxdesktop 
database.port= 1433 
database.user= sa 
database.password= Password258* 
database.dbname= mydb 
database.server.name= linuxdesktop 
table.include.list= dbo.WorkQueue 
database.history.kafka.bootstrap.servers= linuxdesktop:9092 
database.history.kafka.topic= dbhistory.mydb

connect-standalone.properties plugin.path

plugin.path=/home/vinu/kafka-server/connector

I copied all the .JAR files into the plugin.path

I then tried to start the connector with the command below

~/kafka-server/bin/connect-standalone.sh ~/kafka-server/config/connect-standalone.properties  DebeziumSQLServerconnector.properties

But process fails to start and get the following error message

2021-03-15 13:36:36,752] ERROR Failed to create job for DebeziumSQLServerconnector.properties (org.apache.kafka.connect.cli.ConnectStandalone:110) [2021-03-15 13:36:36,753] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:121) java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.debezium.connector.sqlserver.SqlServerConnector , available connectors are: PluginDesc{klass=class io.debezium.connector.sqlserver.SqlServerConnector, name='io.debezium.connector.sqlserver.SqlServerConnector', version='1.4.2.Final', encodedVersion=1.4.2.Final, type=source, typeName='source', location='file:/home/vinu/kafka-server/connector/debezium-connector-sqlserver/'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='2.7.0', encodedVersion=2.7.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='2.7.0', encodedVersion=2.7.0, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='2.7.0', encodedVersion=2.7.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'} at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:115) at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:99) at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:118) Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.debezium.connector.sqlserver.SqlServerConnector , available connectors are: PluginDesc{klass=class io.debezium.connector.sqlserver.SqlServerConnector, name='io.debezium.connector.sqlserver.SqlServerConnector', version='1.4.2.Final', encodedVersion=1.4.2.Final, type=source, typeName='source', location='file:/home/vinu/kafka-server/connector/debezium-connector-sqlserver/'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='2.7.0', encodedVersion=2.7.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='2.7.0', encodedVersion=2.7.0, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='2.7.0', encodedVersion=2.7.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='2.7.0', encodedVersion=2.7.0, type=source, typeName='source', location='classpath'} at org.apache.kafka.connect.runtime.isolation.Plugins.connectorClass(Plugins.java:208) at org.apache.kafka.connect.runtime.isolation.Plugins.newConnector(Plugins.java:180) at org.apache.kafka.connect.runtime.AbstractHerder.getConnector(AbstractHerder.java:572) at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:342) at org.apache.kafka.connect.runtime.AbstractHerder.lambda$validateConnectorConfig$1(AbstractHerder.java:326) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)

What could be the problem?

NOTE All Kafka components are installed locally on the machine. The only container in my setup is SQL Server.

enter image description here enter image description here

2
sql-serverapache-kafkaapache-kafka-connectdebezium
According to the output, your plugin path is /home/vinu/kafka-server/connector/, not /usr/share/java, and properties files don't use colons for key-value pairs.... But output that starts with available connectors are clearly lists the connector you're trying to use... Note, kafka:9092 isn't going to work on the host machine where localhost:1433 is the database connection string. I suggest running Debezium docker container - OneCricketeer
thanks. correct plugin path and changed : to = in the properties file. Also database.history.kafka.bootstrap.servers to localhost:9092 (database.history.kafka.bootstrap.servers:localhost:9092 )but still gets the same error. - Vinu Junkmails

2 Answers

1
votes

The available connectors lists the class, so your plugins are fine.

You have a space issue. Notice the log output comma placement

matches io.debezium.connector.sqlserver.SqlServerConnector ,

Remove trailing whitespace from the whole property file and leading whitespace before values

0
votes

you must not place the connector JARs into /home/vinu/kafka-server/connector but into /home/vinu/kafka-server/connector/debezium-sqlserver-connector or any other name. Connectors are not put to plugin.path but each connector to a separate dir in the plugin.path.