Selective releases in Azure Data Factory and DevOps

0
votes

We have 2 Data factories (dev and prd) with an Azure DevOps Git integration. All setup for CI/CD as mentioned in https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment:

  • Development in different features branches
  • Different features to master branch by pull requests
  • Publish to adf_publish branch
  • Azure pipeline to release from ADF-dev to ADF-prd

An additional requirement we would like to add, is the possibity to perform selective releases to our prd Data Factory. F.e.: New development A was published to our adf_publish branch and the validation of new development A is still in progress. Meanwhile, new request B needs to be released to ADF-prd as soon as possible (not as a hotfix). However, this is currently blocked by development A, which still has to be validated. So we would like to have the possibility to only release request B to our adf-prd and exclude development A from that release (as it's not validated yet).

Unfortunately, this feature is not supported in Data Factory itself (https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment). We are looking for a solution in Azure DevOps, but so far without luck. Does anyone has a solution for this?

Thanks in advance, Kind regards, Jef

1
azureazure-devopsazure-data-factory

1 Answers

0
votes

There is now a way to create an ADF build artifact from a branch and using that artifact to deploy to upper environments, rather than manually clicking Publish in the ADF UX. Using this feature, you could have a breakdown of branches like so:

feature branches (for devlopment)

develop branch (for things not validated yet)

main branch (for things validated or that need to be published to your physical ADF)

using this architecture you could develop a pipeline in feature branch, push it to develop branch where it could have an automated CI build pipeline that generates artifacts, this could either be deployed to a DEV ADF or just used as a validation step, once things are validated in your develop branch, you can do a PR to your main branch which deploys to your DEV or TEST ADF.

In this example, your New Development A would be sitting in the develop branch while your new request B would be pushed directly to your main branch and that would trigger an automated build and deployment to whatever ADF you want...

Please see this StackOverflow Post that I answered on already, it is very thorough.

Azure Data Factory deployments with improved CI/CD