Manage Azure BlobStorage file append with ADF

0
votes

I've azure data factory pipeline which stores data with some operation by calling Azure data flow. Here file name in blob storage should be the pipeline-run-id.

Pipeline copy activity has 'Copy Behavior', I can not find a related option in the sink stream in a data flow?

Now I have a situation where I'm going to call the same azure data flow in the same pipeline execution more than one time. And because of that my file get overwritten in the blob. But I want to append new data to the same file if it exists.

Ex. If pipeline run id '9500d37b-70cc-4dfb-a351-3a0fa2475e32' and data flow call from that pipeline execution 2 times. In that case, 9500d37b-70cc-4dfb-a351-3a0fa2475e32.csv only has data with 2nd azure data flow process.

1
azureblobazure-data-factoryazure-data-flow

1 Answers

0
votes

Data Flow doesn't support copyBehavior. It means that Data Flow doesn't support merge/append files. enter image description here

Every time you call the Data Flow, it will create a new file '9500d37b-70cc-4dfb-a351-3a0fa2475e32.csv' and replace the exist '9500d37b-70cc-4dfb-a351-3a0fa2475e32.csv'.

Hope this helps.