Copy data activity in Azure Data Factory

Question

I have an adf pipeline that uses copy data activity for copying data from blob storage to table storage. This pipeline runs on a trigger once every day. I'm curious to know how this copy data activity actually works. During pipeline run, does it clear the table and copy data from blob to table or does it just add any updated rows to table after comparison?

For example:

After 1st run, table contains below rows:

Row1 xyz 1

Row2 abc **2**

Now let's say rows in blob are updated as follows:

Row1 xyz 1

Row2 abc **5**

During 2nd run, does it clear table and store the above data or only update Row2 with values abc and "5" instead of "2"

I'm 99.999% sure it just INSERTs to the table. It should not truncate the table first [although you can add a "pre-copy script" to do so]. But it also does not do any "comparison". If you need an "upsert" feature, consider using Data Flow which gives you more options in this regard. — Joel Cochran
Thank you! By 'just INSERTs' do you mean it doesn't remove any data from the table. Let's say I have 100 rows in the table. During copy data activity, I can still see all the 100 rows in the table and I don't have to wait until copy data activity is over in order to see updated 100 rows? — user989988
I'm not trying to be pedantic, but when you say "updated" rows, do you mean NEW rows or are you expecting the existing rows to update? — Joel Cochran

vinay bagannagari vinay bagannagari · Accepted Answer · 2020-05-29T04:11:07

ADF copy activity doesn't update any existing rows in the table. It is an append only process. So, If your table has primary key constraint, the copy activity will fail.

To do an upsert process, you can either use mapping data flow or stored procedure.

Copy data activity in Azure Data Factory

1 Answers