Kafka sink connect - How to create one task per topic(table)

0
votes

We have implemented a kafka sink connector for out product named Ampool ADS which ingestt data from kafka topics to corresponding ampool tables. Topics and Tables are mapped by their names.

I Need to handle individual topic (ingestion from topic ---> table) into a dedicated sink task.

So for example, if my config contains 3 different topics (topic1, topic2, topic3), Sink connector should create 3 different sink tasks, each doing (per table) dedicated ingestion to their respective/mapped table in parallel.

NOTE: Reason behind handling an individual topic into a dedicated sink task is its simple to use RetriableException mechanism if specific table are offline/not-created. Only a individual topic/table records will get replayed after configured time-interval.

Is this possible with kafka connect framework, if so how..?

1
apache-kafkaapache-kafka-connectconfluent-platform
I don't think it is possible to achieve this to tell which task will pick which topic. Only way is create three connectors n assign one topic to each with 1 task. But if the topics increases the connectors will also have to increase, this is not a good design. – Renukaradhya

1 Answers

1
votes

If you set the number of tasks to be equal to the number of partitions (and I think you can do this from the connector code - when creating the configuration), then each task will get exactly one partition.