To answer your question directly:
No, there is not a separate DEV ADF, The only difference between the Old and New is that you no longer need to manually click publish from your collab branch. The way it works is you now have a build pipleine that is triggered anytime there is an update to your collab branch (via PR), once the build validates and produces the artifact, there is a release pipeline that deploys the ARM template to your DEV Data factory.
Here are screenshots to show:
1st, Add this package.json file to your collaboration branch
{
"scripts":{
"build":"node node_modules/@microsoft/azure-data-factory-utilities/lib/index"
},
"dependencies":{
"@microsoft/azure-data-factory-utilities":"^0.1.5"
}
}
Like I did:
2nd, Create the yaml build pipeline and edit the parameters from script below
# Sample YAML file to validate and export an ARM template into a build artifact
# Requires a package.json file located in the target repository
trigger:
- main #collaboration branch
pool:
vmImage: 'ubuntu-latest'
steps:
# Installs Node and the npm packages saved in your package.json file in the build
- task: NodeTool@0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- task: Npm@1
inputs:
command: 'install'
workingDir: '$(Build.Repository.LocalPath)/<folder-of-the-package.json-file>' #replace with the package.json folder
verbose: true
displayName: 'Install npm package'
# Validates all of the Data Factory resources in the repository. You'll get the same validation errors as when "Validate All" is selected.
# Enter the appropriate subscription and name for the source factory.
- task: Npm@1
inputs:
command: 'custom'
workingDir: '$(Build.Repository.LocalPath)/<folder-of-the-package.json-file>' #replace with the package.json folder
customCommand: 'run build validate $(Build.Repository.LocalPath) /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/testResourceGroup/providers/Microsoft.DataFactory/factories/yourFactoryName'
displayName: 'Validate'
# Validate and then generate the ARM template into the destination folder, which is the same as selecting "Publish" from the UX.
# The ARM template generated isn't published to the live version of the factory. Deployment should be done by using a CI/CD pipeline.
- task: Npm@1
inputs:
command: 'custom'
workingDir: '$(Build.Repository.LocalPath)/<folder-of-the-package.json-file>' #replace with the package.json folder
customCommand: 'run build export $(Build.Repository.LocalPath) /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/testResourceGroup/providers/Microsoft.DataFactory/factories/yourFactoryName "ArmTemplate"'
displayName: 'Validate and Generate ARM template'
# Publish the artifact to be used as a source for a release pipeline.
- task: PublishPipelineArtifact@1
inputs:
targetPath: '$(Build.Repository.LocalPath)/<folder-of-the-package.json-file>/ArmTemplate' #replace with the package.json folder
artifact: 'ArmTemplates'
publishLocation: 'pipeline'
This build should run every time there is a change to your collaboration branch.
3rd, Create the Release Pipeline that will deploy to your DEV ADF using the build artifact (ARM Template and Parameter files)
TASK DETAILS:
Do not forget to set up Continuous deployment trigger on the build for ADF.
And that's all there is to it, no more clicking that pesky Publish button, but instead you do have to deal with manually inputting the override parameters... you win some you lose some....
EDIT:
I found that there was an issue if you have global parameters included in your ARM Template, if they are specified, when you deploy to your DEV ADF, it will disconnect your factory from Azure DevOps Git. To avoid this you can add Global Parameters via the PostDeploy powershell script.
Here are the steps for doing this:
1.
Make sure for Include in ARM Template is unchecked within your Azure Data Factory Global Parameters page:
You need to save a globalParameters json file in your collaboration Branch for each environment of ADF. This file will be used in the Powershell script to ensure the globalParameter exists in your ADF.
2.
Create the Powershell script file and save it in your Collaboration Branch so it can be used in the release via an Az Powershell Task.
param
(
[parameter(Mandatory = $true)] [String] $globalParametersFilePath,
[parameter(Mandatory = $true)] [String] $resourceGroupName,
[parameter(Mandatory = $true)] [String] $dataFactoryName
)
Import-Module Az.DataFactory
$newGlobalParameters = New-Object 'system.collections.generic.dictionary[string,Microsoft.Azure.Management.DataFactory.Models.GlobalParameterSpecification]'
Write-Host "Getting global parameters JSON from: " $globalParametersFilePath
$globalParametersJson = Get-Content $globalParametersFilePath
Write-Host "Parsing JSON..."
$globalParametersObject = [Newtonsoft.Json.Linq.JObject]::Parse($globalParametersJson)
foreach ($gp in $globalParametersObject.GetEnumerator()) {
Write-Host "Adding global parameter:" $gp.Key
$globalParameterValue = $gp.Value.ToObject([Microsoft.Azure.Management.DataFactory.Models.GlobalParameterSpecification])
$newGlobalParameters.Add($gp.Key, $globalParameterValue)
}
$dataFactory = Get-AzDataFactoryV2 -ResourceGroupName $resourceGroupName -Name $dataFactoryName
$dataFactory.GlobalParameters = $newGlobalParameters
Write-Host "Updating" $newGlobalParameters.Count "global parameters."
Set-AzDataFactoryV2 -InputObject $dataFactory -Force
3.
Make sure there is a Az Powershell Task in your Release pipeline to run the PostDeployment powershell script with given parameters.
If all of this is set up correctly then you should have a completely automated CI/CD process built for ADF (and it shouldn't disconnect you from Git)
Let me know if I can be of any other help!
