Trigger status update at the time of ADF deployment

Question

I am deploying the ADF ARM template from DEV instance to PROD using Azure DevOps pipelines.

For the deployment below are the steps I followed.

1. Scripts provided by Microsoft to stop the triggers before deployment
2. ARM template deployment
3. Scripts provided by Microsoft to start the triggers and delete the resources not in the JSON file. under the header Sample pre- and post-deployment script in below page https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment

I do have around 20 triggers in which 15 are in running state and 5 are stopped for a while in PROD. In my DEV instance all triggers are in stopped status in order to avoid unnecessary pipeline runs. But at the time of deployment into PROD I need to set status of 15 out of 20 triggers as Started.

To achieve this I have added the runtimeState of triggers in template parameterization and now I could select the trigger status from Override Parameters Option in the ARM Template Deployment task as below.

Now the issue is after PROD deployment, All triggers are in Stopped status only as in DEV instance.

Eventhough the runtimeState is overwritten at the time of deployment, it is not reflecting in the PROD instance after deployment.

After my analysis, I suspect the issue is with start trigger script from Microsoft is referring the ARMTemplateForFactory.json file from DEV where status of all triggers are stopped.

How to resolve this? I have to update the trigger status of each trigger job at the time of deployment from DEV to PROD.

In the script to STOP/START triggers from Microsoft, it is getting list of triggers to be started from Json file which we get from DEV instance through build pipeline. The overwritten pipeline status will not be taken to start the triggers.

One of the solution I found was to start the triggers explicitly after the ARM deployment. For that I found an agent job as shown below.

But the Issue with this job is I am not able to provide name of multiple triggers at a time. I am able to provide name of a single trigger as shown below.

Is there any way to add multiple trigger name in this job?

Answer 1

As you described, the Microsoft provided pre-post deployment script pauses all the active triggers before the deployment, and then restores them to the state specified in the ARM template after the deployment. (They recommend pausing triggers before deployment to avoid issues)

One solution could be to create a custom powershell script and executing that after deploying the ARM template. You can use the Start-AzDataFactoryV2Trigger cmdlet which is also used in the Microsoft provided script.

For example:

$ResourceGroupName = "my-resource-group"
$DataFactoryName = "my-data-factory'
$triggersToStart = @("trigger-1", "trigger-2", "trigger-3")

$triggersToStart | ForEach-Object {
    Start-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_ -Force
}

Answer 2

Please check with the following things:

1. Try the same scripts on your local machine to see if they can work as expected.

2. If the scripts also do not work from your local machine, the issue should be on ADF or the scripts, rather than Azure DevOps. Check if you have any incorrect configuration on ADF or in the scripts.

3. If the scripts can work as expected from your local machine, try to install a self-hosted agent on the local machine to run the pipeline to see if it can work as expected.

Answer 3

For the time being, I have added a new azure powershell task after the ARM template deployment task to start the required triggers.

The ResourceGroupName, DataFactoryName, TriggersToStart details are passed as argument for the script which will start the triggers. So that we could ADD/REMOVE the triggers for enabling during each deployment.

Below is the script i used to fetch the arguments and start the triggers using below script.

param
(
 [parameter(Mandatory = $false)] [String] $ResourceGroupName,
 [parameter(Mandatory = $false)] [String] $DataFactoryName,
 [parameter(Mandatory = $false)] [Array] $triggersToStart
)

 Foreach($trigger_name in $triggersToStart)
 {
  Write-Host "Starting Trigger : " $trigger_name
  Start-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $trigger_name  -Force
 }