How to define schema while creating a Kafka connector using REST API

0
votes

I have configured Kafka connect workers to run in cluster and able to get DB data. Also I have stored DB data in Kafka topics in JSON format. Here I used JSON converter for serializing the data

On viewing the DB data in Kafka consumer console I can see that UserCreatedon column value is displayed as integer. The data type of the UserCreatedon column value in DB is int64 (unix epoch time), that’s why timestamp value is displayed as int by Kafka consumer

Is there any way to send schema during connector creation. Because i want UserCreatedon should be displayed in timestamp format instead of int

Sample output

{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"NAME"},{"type":"int64","optional":true,"name":"org.apache.kafka.connect.data.Timestamp","version":1,"field":"UserCreatedON"}],"optional":false},"payload":{"NAME":"UserProvision","UserCreatedon ":1567688965261}}

Kindly requesting your support a lot here.

1
apache-kafkaapache-kafka-connect

1 Answers

0
votes

You have not mentioned what type of connector you are using to bring data to Kafka from DB. Kafka connect supports transformers.

Single Message Transformations (SMTs) are applied to messages as they flow through Connect. SMTs transform inbound messages after a source connector has produced them, but before they are written to Kafka

See here

Specifically for your case you can use TimestampConvertor