Kafka connect integration with multiple Message Queues

Question

I'm designing a system where I have to integrate with multiple Message Queues (MQ) based backends. I have one microservice for each backend for processing MQ payloads. I have chosen Kafka as the medium of messaging and considering Kafka-MQ-Connects for MQ integration.

I can think of two approaches to integration.

1. Kafka-MQ-Connect (Source/ Sink) connect per backend + Kafka topic (to/ from) per backend.

Pros. - Can extend to new backends without touching the existing connectors.

Cons. - Too many connectors and topics to maintain.

2. Single Kafka-MQ-Connect (Source/ Sink) + Single Kafka topic (to/ from) for all the backends. Additionally, the Sink connects do dynamic routing to MQs and the microservices will have built-in Message-Filters to filter only relevant messages.

Pros. - Few topics and connectors to maintain.

Cons. - Addition of new MQ backends would require connector changes.

What would be the better approach? Are there any other integration alternatives apart from the above?

Answer 1

Although you haven't provided any further requirements (for example, how frequently are you planning to add new data sources and that traffic do you have), I would pick the first approach. It will be much easier in the future to add/remove new/existing data sources.

And I wouldn't say that it is hard to maintain multiple sink/source connectors and topics. From my experience, it is harder to maintain connectors which are replicating data from multiple topics/sources. For example, if you want to apply SMT (Simple Message Transform) on a particular topic, you won't be able to achieve it if you don't have isolated connectors as SMTs are applied on a connector level. Furthermore, if you configure a single connector for all of your sources and at some point it fails, all of your target systems will encounter downtime.