Outbox pattern with kafka and mongoDB connector

Question

I want to implement Outbox Pattern in our microservices using Mongo-Kafka connector, in my outbox(a MongoDB collection) I store topic data using these fields: kafka_topic, kafka_key, kafka_value, kafka_header_keys, kafka_header_values.

But how should I config mongo-kafka connector to dynamically choose the topic from the kafka_topic field and also its values and headers from other outbox fields? I can not find settings to do this from its config reference.

I'm also thinking about getting a fork from mongo-kafka connector repository and extend it with my custom implementation.

I would greatly appreciate it if you could help me.

Answer 1

Ok, I got my answer, mongo-kafka connector does not support the outbox pattern. for other databases, Debezium has the Outbox event router which does that for us, but it does not support the MongoDB connector until writing this answer.

As a final answer: we forked the mongo-kafka connector and implemented our outbox pattern.

Answer 2

I made a PR for the Outbox Event Router for MongoDB feature and it was just merged yesterday! (2021.11.17). So, I believe this feature would be applied in the near future. It supports same features with the existing Outbox Event Router. But, you would have to use multi-document transaction yet. But, it's planned to be developed in the future without multi-document transaction.