Azure data factory v2 : Customize execute pipeline activity

Question

I am trying to create a master pipeline which will check whether the child pipeline has to be run using the configuration from database. The name of the child pipeline is a parameter to the master pipeline. We would like to choose name of the pipeline for execute pipeline activity using the parameter passed to the project. The ultimate goal is that we don't want to get the data from the source again if we have already retrieved it. If there is any error in the whole ETL process, we would like to avoid running the pipeline which has already completed.

I am trying to avoid creating a custom activity since it requires an Azure Batch account.

Any ideas?

Martin Esteban Zurita Martin Esteban Zurita · Accepted Answer · 2018-04-23T14:14:42

Have you considered calling the pipelines with a powershell script? Install the Azure SDK for powershell from here https://azure.microsoft.com/en-us/downloads/ and you can make pipelines run with it. If you are somehow getting the name of the pipeline to run, I think its the easiest way to handle what you want to do.

This is how you call a pipeline from powershell (with azure sdk installed):

$SubscriptionName = "yourSubscName"
$ResourceGroupName = "your RG name"
$DFName = "your data factory name"
$PipeName = "your pipeline name"
Login-AzureRmAccount
Select-AzureRmSubscription -SubscriptionName $SubscriptionName
Invoke-AzureRmDataFactoryV2Pipeline -DataFactoryName $DFName -ResourceGroupName $ResourceGroupName -PipelineName $PipeName

And that's it! You can run this script from your on-premise pc or with Azure Automation (which is free). For Azure Automation you maybe need to modify it a bit but the basic idea is the same.

Hope this helped!

PS: to run this script you must manually login to azure, but there are ways to automate it that I didn't explain to make this shorter.

Azure data factory v2 : Customize execute pipeline activity

1 Answers