“Publish” programmatically on Azure Data Factory through powershell or Azure Devops pipeline step

Question

This question won't have any code because I haven't found any possible way so far but not even a straight no, it's not possible.

Azure Data Factory uses adf_publish branch as the official branch on top of the master. ADF publishes the definitions of all the pipelines, triggers, linked services, etc to this adf_publish branch when on the GUI the user clicks on Publish.

I need to do this programmatically. I don't want someone to go on the ADF portal itself and click on the Publish button so that the adf_publish branch is updated correctly. I need to know if there is some powershell code (or step in Azure Devops pipeline) that allows me to do this. Thanks in advance!

UPDATE:

After Kevin's answer below I gave it a try with the suggested powershell script and seems succeeding but unfortunately it doesn't seem to do what it's supposed to do. Whenever I try to "Publish" from the ADF portal I see that it gathers the changes from the collaboration branch (Master) and tries to merge them in the Adf_Publish branch. An example is the pic below listing all the changes that will be merged.

So in theory the powershell script should do the same or that's what I am trying to achieve while instead it seems not doing anything:

Am I missing something?

Answer 1

You could refer to this ticket in Github.

You could install Az.DataFactory and azure.datafactory.tools PowerShell modules, then you could run the Publish-AdfV2FromJson method to publish.

Here is Azure Devops example:

variables:
  ResourceGroupName: 'rg-devops-factory'
  DataFactoryName: 'SQLPlayerDemo'
steps:
- powershell: |
   Install-Module Az.DataFactory -MinimumVersion "1.7.0" -Force
   Install-Module -Name "azure.datafactory.tools" -Force
   Import-Module -Name "azure.datafactory.tools" -Force
  displayName: 'PowerShell Script'
steps:
- task: AzurePowerShell@4
  displayName: 'Azure PowerShell script: InlineScript'
  inputs:
    azureSubscription: 'Subscription'
    ScriptType: InlineScript
    Inline: |
     Publish-AdfV2FromJson -RootFolder "$(System.DefaultWorkingDirectory)/_ArtifactName_/" -ResourceGroupName "$(ResourceGroupName)" -DataFactoryName "$(DataFactoryName)" -Location "$(Location)" -Stage "$(Release.EnvironmentName)"

    FailOnStandardError: true
    azurePowerShellVersion: LatestVersion

For more detailed information, you could refer to this introduction document

On the other hand, you could use the out of the box task: Publish Azure Data Factory from Deploy Azure Data Factory by SQLPlayer extension.

Answer 2

This is definitely possible and quite an expected requirement in real world case. Please go through Continuous integration and delivery in Azure Data Factory. It's too long guide to put it as an answer in short, so you have to read that in details :). Below is just a short quote of the scenario to get you charged:

• A development data factory is created and configured with Azure Repos Git. All developers should have permission to author Data Factory resources like pipelines and datasets.

• A developer creates a feature branch to make a change. They debug their pipeline runs with their most recent changes. For more information on how to debug a pipeline run, see Iterative development and debugging with Azure Data Factory.

• After a developer is satisfied with their changes, they create a pull request from their feature branch to the master or collaboration branch to get their changes reviewed by peers.

• After a pull request is approved and changes are merged in the master branch, the changes get published to the development factory.

• When the team is ready to deploy the changes to a test or UAT (User Acceptance Testing) factory, the team goes to their Azure Pipelines release and deploys the desired version of the development factory to UAT. This deployment takes place as part of an Azure Pipelines task and uses Resource Manager template parameters to apply the appropriate configuration.

• After the changes have been verified in the test factory, deploy to the production factory by using the next task of the pipelines release.

To automate the merge from master branch to adf_publish branch in a CI build which runs on master, you can look at Run Git commands in a script. This merges from feature to master, but you will do the opposite.

Answer 3

@Tarta, there is plenty of things blended up here. I don't want to answer everything here, just let me focus on the tool that you tried to use (azure.datafactory.tools), which means - the first original issue. Looking at the screenshot of log:

There are no objects found by PowerShell module, which likely means you provided the wrong location (folder) to the ADF. Please, read the documentation carefully and make sure you're passing the correct folder while executing Publish-AdfV2FromJson cmdlet.
Also, bear in mind, that the tool deploys ADF from code - master or another collaboration branch you selected. The module does NOT care what is going on at adf_publish branch at all. The changes into that branch are doing by ADF service when you click PUBLISH, but you don't have to worry about it when you choose to deploy from code by azure.datafactory.tools or DevOps extension. Additionally, I hope the page I prepared will help you a little bit to understand the whole concept of deploying ADF from code (not from adf_publish): https://sqlplayer.net/adftools/

Answer 4

Recently, Microsoft has announced that they now support automated Azure Data Factory publish using ADFUtilities NPM (@microsoft/azure-data-factory-utilities) library and Azure DevOps build pipeline.

Read more about it at https://azureops.org/articles/automated-azure-data-factory-publish/

Answer 5

There were improvements made to the Publishing of an Azure Data Factory.

Previously you need to manually "Publish" a data factory from the UI which doesn't go well with the concept of automation.

Now you can use an ADFUtilities NPM package (preferably in your build pipeline), which will validate all the resources and generate the ARM templates which in turn can be used in the release pipeline for your automated deployments. Refer to the below link for the complete details and steps to configure the same.

https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment-improvements