Kafka connect: Single topic sink to multiple tables based on schema

1
votes

I am new to Kafka connect. I have a usecase like below :

  1. There is a single shared topic in which I get messages of different entities, say Employee, department (actual table names are different)

  2. Schemas of employee and department are registered in schema registry

Using the Kafka sink connector, is it possible to separate data coming for each entity based on schema and write to respective tables Example, employee data coming to topic should go to employee table and department data to department table

If not, is there any other better way to do this?

1
apache-kafkaapache-kafka-connectconfluent-platform
If two JSON fields have different properties, you can identify based on property and write it to respective table @user - Deadpool
Is it possible to do this between a kafka topic and connect? - Frqa

1 Answers

0
votes

A Connector can only be configured for a single table at a time.

So, other than adding a field like "entityType" into the data itself, and filtering that out in the database layer, I believe the only way to manage this would be Kafka Streams (or KSQL) to filter() out the two possible types of data, then send them to() another topic, which you have Connect running and listening to two topics, which write to their respective tables.