Debezium Embedded Engine with AWS Kinesis - PostgreSQL snapshot load and Transaction metadata stream

Question

I'd like to use Debezium Embedded Engine with AWS Kinesis in order to load initial snapshot of PostgreSQL database and then continuously perform a CDC.

I know, that with Kafka Connect I'll have Transaction metadata topic out of the box in order to check transaction boundaries.

How about the same but with Debezium Embedded Engine and AWS Kinesis ( https://debezium.io/blog/2018/08/30/streaming-mysql-data-changes-into-kinesis/ ) Will I have Kinesis Transaction metadata stream in this case? Also, will Debezium Embedded Engine perform initial snapshot of the existing PostgreSQL data?

UPDATED

I implemented test EmbeddedEngine application with PostgreSQL:

    engine = EmbeddedEngine.create()
            .using(config)
            .using(this.getClass().getClassLoader())
            .using(Clock.SYSTEM)
            .notifying(this::sendRecord)
            .build();

Right now, inside my 'sendRecord(SourceRecord record)' method I can see the correct topics for each database table which participate in transaction, for example:

private void sendRecord(SourceRecord record) {

    String streamName = streamNameMapper(record.topic());
    System.out.println("streamName: " + streamName);

results to the following output:

streamName: kinesis.public.user_states
streamName: kinesis.public.tasks

within the same txId=1510

but I still can't see Transaction metadata stream. How to correctly get Transaction metadata stream with Debezium EmbeddedEngine?

Pavan Kumar Aryasomayajulu Pavan Kumar Aryasomayajulu · Accepted Answer · 2021-02-25T05:20:13

If you are not specific about using just Debezium Embedded Engine then there is an option provided by Debezium itself and it is called Dewbezium Server( Internally I believe it makes use of Debezium Engine). It is a good alternative to making use of Kafka and it supports Kinesis, Google PubSub, Apache Pulsar as of now for CDC.

Here is an article that you can refer to https://xyzcoder.github.io/2021/02/19/cdc-using-debezium-server-mysql-kinesis.html

Debezium Embedded Engine with AWS Kinesis - PostgreSQL snapshot load and Transaction metadata stream

1 Answers