Confluent JDBC connector, where information is kept about last read ID and timestamp

Question

I am running Confluent JDBC connector on my local machine and trying to figure out where it is storing the information about last read ID and timestamp.

As per https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector/#starting-table-capture

It needs to be in offset.storage.topic but I tried to set the configuration but I didn't work.

My connector:

{
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "timestamp.column.name": "updated_at",
    "incrementing.column.name": "id",
    "connection.password": "test123",
    "tasks.max": "1",
    "topics": "dp_status",
    "table.whitelist": "source_test",
    "mode": "timestamp+incrementing",
    "value.converter.schema.registry.url": "http://schema-registry:8081",
    "topic.prefix": "test-postgresql-jdbc-source.",
    "connection.user": "test",
    "name": "postgresql-jdbc-source-gp",
    "connection.url": "jdbc:postgresql://postgresql:5432/test_db",
    "value.converter": "io.confluent.connect.avro.AvroConverter"
}

Are you running Kafka Connect in distributed mode or standalone? — Robin Moffatt

Robin Moffatt Robin Moffatt · Accepted Answer · 2021-01-26T22:28:00

If you run Kafka Connect in standalone mode then information about offsets (so in the case of JDBC source the ID/timestamp) is stored in a local file defined by offset.storage.file.filename in the Kafka Connect worker properties.

Confluent JDBC connector, where information is kept about last read ID and timestamp

1 Answers