I wanted to upsert the incremental data from sql db to sql warehouse using Azure data factory

Question

I wanted to update and insert (upsert) the incremental data from the azure SQL database to Azure data warehouse using azure data factory

:-> The DB is having the multiple tables.

There are a couple of really good 3rd parties that provide change data capture (CDC) as well as incremental loading capabilities. docs.microsoft.com/en-us/azure/sql-data-warehouse/… — Alberto Morillo

Alberto Morillo Alberto Morillo · Accepted Answer · 2019-03-14T15:43:21

You to define a watermark in your source database. A watermark is a column that has the last updated time stamp or an incrementing key. The delta loading solution loads the changed data between an old watermark and a new watermark. The workflow for this approach is depicted in the following diagram.

For step-by-step instructions, see this article. On this article adjust the sink to be Azure SQL Data Warehouse and you kind of have all you need on this article.

Another useful example can be found on this Microsoft documentation.

I wanted to upsert the incremental data from sql db to sql warehouse using Azure data factory

1 Answers