Azure ADF V2 - Activity on completion

0
votes

I'm working on creating a small project to track ETL logs. I've created a stored procedure with parameters and a custom SQL table to load the ETL logs.

Inside the ADF I have multiple activities. At the end I'm using stored procedure activity with parameters mapped to ADF system variables like pipeline name, error details etc to log in the SQL table.

Issue: whenever there's an error on an activity in middle, the pipeline fails and not touching the stored procedure activity. Like, say I have Copy1, Copy2, Copy3 and at last ETLLog_StoredProcedure. If Copy2 fails, the pipeline run stops there at Copy2 and stored procedure activity is not run.

I have connected all the Copy activities to ETLLog_StoredProcedure using Activity-On-Completion connections. Take a look at the picture below.

Expectation: I need to call the stored procedure activity even if the pipeline fails/succeeds so that I can log the status of the pipeline.

Sample ADF ETLLog

2
sql-serverazureazure-sql-databaseazure-data-factory-2

2 Answers

2
votes

Data factory dependencies are used as an AND condition. This means that the stored procedure will be run once ALL of the 3 activities are "completed" (success or failure). But in your scenario, the second activity is failing and the third one is never running (not even failing) and that's why the Stored Procedure activity is not running.

You can achieve what you are looking for with this, changing the parameters for the stored procedure depending where the failure happened (or to show a success in the last one), for example:

Logging errors in ADF

There are other ways to achieve this, but they require a bit more understanding of ADF variables and functions, this one is the simplest in my opinion.

Hope this helped!!

0
votes

I added the picture for better understanding ADFPicturePipelineRun

To have only one Stored Procedure call in the pipeline you can just add the option "Skipped".

So in general the Activity "Copy data3" has 2 options to full fill the condition to execute Activity "Stored procedure1", Completion OR Skipped. As "Copy data1" and "Copy data2" both completed and "Copy data3" Skipped "Stored procedure1" is executed.