Use message key in Kafka connect source connector

5
votes

I'm using the Kafka connect JDBC source connector to read from a view in a database and post it on kafka, it is working fine.

My use case is that a user can create multiple objects and the order of the objects is important in my application. I would like to use the user id as the message key of all the messages I'm posting into the topic to maintain their order.

My question is how can I define a message key in the Kafka connect source connector?

1
apache-kafkaapache-kafka-connect

1 Answers

6
votes

You can use Kafka Connect's SMT (Single Message Transforms) feature by adding following code to connect-file-source configuration file.

transforms=createKey
transforms.createKey.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.createKey.fields=UserId <name of user id column>

More information about SMT here