How to handle failures in parallel copy activities in ADF v2?

Question

I'm planning on iterating over a list of connection strings to upsert data from one source to multiple sinks in parallel. So, I'll abstract the copy activity and parameterize the connection string value here. The main pipeline will then iterate over the connection strings and pass on the value to the abstracted copy activity.

Scenario: One of those parallel copy activities fails. How to handle the individual failures?

• Option 1: Abort all the parallel pipelines in order to maintain my sinks in a consistent state. Is this possible to do in Azure data factory?

• Option 2: If I choose to let the other pipelines write data to the sink, will I have to handle the failed slices manually? I'm choosing a tumbling window trigger for the main pipeline here.

Please help me gain clarity over handling it in the best way possible.

Answer 1

A best practice is to have your activities in foreach loop in the separate pipeline, so then you call execute pipeline activity in your foreach. That way each execution will be treated separately and in monitoring view, you will have a clear overview of which one failed.