Wanted to know more about the internals of AWS DMS (Data Migration Service)?

Question

When the replication task is created, then when there is any change in source database it is replicated in the target database.

So just wanted to know more about how this service internally works?

• My Understanding DMS has listener service which is used to listen to the triggers/events when they fire on the source, and listener collects these events/triggers and fires these simultaneously on the target, kind of streaming concept has been implemented in on-going replication and at the time of the load batch processing has been implemented.

Can you please clarify me about the internals more or if please share with me any high-level design reference for it.

I need more clarification on this

Answer 1

This doc contains the high level overview of how DMS works:

To perform a database migration, AWS DMS connects to the source data store, reads the source data, and formats the data for consumption by the target data store. It then loads the data into the target data store. Most of this processing happens in memory, though large transactions might require some buffering to disk. Cached transactions and log files are also written to disk. At a high level, when using AWS DMS you do the following:

• Create a replication server.

• Create source and target endpoints that have connection information about your data stores.

• Create one or more tasks to migrate data between the source and target data stores.

There is a lot more details on the page, but this summary is decent at explaining whats happening. There isn't triggers, so much as just reading the data you specify, transforming as needed, then writing to the destination.

Answer 2

How DMS CDC works depends on your source database and will often require configuration of your source database and control parameters in the DMS Task.

This process works by collecting changes to the database logs using the database engine's native API.

Each source engine has specific configuration requirements for exposing this change stream to a given user account. Most engines require some additional configuration to make it possible for the capture process to consume the change data in a meaningful way, without data loss. For example, Oracle requires the addition of supplemental logging, and MySQL requires row-level binary logging (bin logging).

To read ongoing changes from the source database, AWS DMS uses engine-specific API actions to read changes from the source engine’s transaction logs. Following are some examples of how AWS DMS does that:

(Continue reading at: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html)

To find your particular source database, start here select your source, and dive into the detail.

Answer 3

For MsSQL Server:

• Using the sql replication service
• Using Capturing Data Changes

When you activate one of these services, more detailed records will be created in the ldf file.

Works in two types:

1. By preventing the log file from being truncated, it continuously reads the log file and transfers the new operations to the targets.
2. Reads the transaction backups and transfers the new records to the destinations

Check it : https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SQLServer.html