Using task output variables in template parameters

Question

The first stage in my pipeline checks for what services have actually changed. This is in an effort to speed up the pipeline by avoiding rebuilding, retesting, redeploying services if there have been no changes.

This is the changed.yaml for that stage:

parameters:
- name: comparedTo
  default: ''

stages:
- stage: Changed
  displayName: Check for changes in services and configs...
  jobs:
  - job: Changes
    displayName: Checking for changes in services and configs...
    steps:
    - bash: |
        mapfile -t changed < <(git diff  HEAD ${{ parameters.comparedTo }} --name-only | awk -F'/' 'NF!=1{print $1}' | sort -u)
        servicesChanged=()
        configChanged=()
        echo ""
        echo "Total Changed: ${#changed[@]}"
        for i in "${changed[@]}"
        do
          echo $i
          if [[ $i == 'admin' ]]; then
            echo "##vso[task.setvariable variable=adminChanged;isOutput=True]true"
            servicesChanged+=("admin")
          elif [[ $i == 'admin-v2' ]]; then
            echo "##vso[task.setvariable variable=adminV2Changed;isOutput=True]true"
            servicesChanged+=("admin-v2")
          elif [[ $i == 'api' ]]; then
            echo "##vso[task.setvariable variable=apiChanged;isOutput=True]true"
            servicesChanged+=("api")
          elif [[ $i == 'client' ]]; then
            echo "##vso[task.setvariable variable=clientChanged;isOutput=True]true"
            servicesChanged+=("client")
          elif [[ $i == 'k8s' ]]; then
            echo "##vso[task.setvariable variable=k8sChanged;isOutput=True]true"
            configsChanged+=("k8s")
          elif [[ $i == 'pipelines' ]]; then
            echo "##vso[task.setvariable variable=pipelineChanged;isOutput=True]true"
            configsChanged+=("pipelines")
          fi
        done
        echo ""
        echo "Services Changed: ${#servicesChanged[@]}"
        for i in "${servicesChanged[@]}"
        do
          echo $i
        done
        echo ""
        echo "Configs Changed: ${#configsChanged[@]}"
        for i in "${configsChanged[@]}"
        do
          echo $i
        done
        if [[ ${#servicesChanged[@]} > 0 ]]; then
          echo ""
          echo "Any services changed: True"
          echo "##vso[task.setvariable variable=anyServicesChanged;isOutput=true]true"
          echo "##vso[task.setvariable variable=servicesChanged;isOutput=true]${servicesChanged[@]}"
        fi
        if [[ ${#configsChanged[@]} > 0 ]]; then
          echo ""
          echo "Any configs changed: True"
          echo "##vso[task.setvariable variable=anyConfigsChanged;isOutput=true]true"
          echo "##vso[task.setvariable variable=configsChanged;isOutput=true]${configsChanged[@]}"
        fi
        echo ""
      name: detectChanges

As you can see, it creates a number of task output variables:

# This just indicates if the service has changed: true/false
echo "##vso[task.setvariable variable=<service-name>;isOutput=True]true"

# This should be creating a an output variable that is an array of the services that have changed
echo "##vso[task.setvariable variable=servicesChanged;isOutput=true]${servicesChanged[@]}"

So I gave myself two options: just a true/false for each service or iterating (somewhow) through an array of the services that have changed.

Each stage basically has the following form:

# pr.yaml
...
- template: templates/unitTests.yaml
  parameters: 
    services: 
    - api
    - admin
    - admin-v2
    - client
...

parameters:
- name: services
  type: object
  default: []

stages:
- stage: UnitTests
  displayName: Run unit tests on service...
  dependsOn: Changed
  condition: succeeded()
  jobs:
  - job: UnitTests
    condition: or(eq(stageDependencies.Changed.Changes.outputs['detectChanges.anyServicesChanged'], true), eq(variables['Build.Reason'], 'Manual'))
    displayName: Running unit tests...
    steps:
    - ${{ each service in parameters.services }}:
      - bash: |
          echo "Now running ${{ service }} unit tests..."

Here is what I've tried so far and the errors I get:

Adding each service conditionally to services array or adding the array of changed services:

- template: templates/changed.yaml
  parameters:
    comparedTo: origin/production
- template: templates/unitTests.yaml
  dependsOn: Changed
  parameters: 
    services: 
    - ${{ if eq(stageDependencies.Changed.Changes.outputs['detectChanges.apiChanged'], true) }}
      - api
    - ${{ if eq(stageDependencies.Changed.Changes.outputs['detectChanges.adminChanged'], true) }}
      - admin
    - ${{ if eq(stageDependencies.Changed.Changes.outputs['detectChanges.adminV2Changed'], true) }}
      - admin-v2
    - ${{ if eq(stageDependencies.Changed.Changes.outputs['detectChanges.clientChanged'], true) }}
      - client

Or...

- template: templates/changed.yaml
  parameters:
    comparedTo: origin/production
- template: templates/unitTests.yaml
  dependsOn: Changed
  parameters: 
    services: 
    - ${{ if eq(dependencies.Changed.outputs['Changes.detectChanges.apiChanged'], true) }}
      - api
    - ${{ if eq(dependencies.Changed.outputs['Changes.detectChanges.adminChanged'], true) }}
      - admin
    - ${{ if eq(dependencies.Changed.outputs['Changes.detectChanges.adminV2Changed'], true) }}
      - admin-v2
    - ${{ if eq(dependencies.Changed.outputs['Changes.detectChanges.clientChanged'], true) }}
      - client

Or...

- template: templates/changed.yaml
  parameters:
    comparedTo: origin/production
- template: templates/unitTests.yaml
  dependsOn: Changed
  parameters: 
    services: 
    - $[ stageDependencies.Changed.Changes.outputs['detectChanges.servicesChanged'] ] 

This results in:

An error occurred while loading the YAML build pipeline. Object reference not set to an instance of an object.

I know variables: will only take strings and not arrays.

One solution would be to have a variables: for each of the true/false variables and then conditions based on the parameters.services and whether the task output variables is true.

Any suggestions?

---

Ref:

• Task Output Variables: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#set-variables-in-scripts
• Parameters: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops#parameters
• Expressions: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops

Answer 1

The template expression ${{}} is evaluated at compile-time(before the jobs run), which means it cannot access to the variables that are dynamically set at the runtime(after the jobs start). So you cannot use template expression ${{}} in above scenario. See below description from here.

Within a template expression, you have access to the parameters context that contains the values of parameters passed in. Additionally, you have access to the variables context that contains all the variables specified in the YAML file plus many of the predefined variables (noted on each variable in that topic). Importantly, it doesn't have runtime variables such as those stored on the pipeline or given when you start a run. Template expansion happens very early in the run, so those variables aren't available

You can use conditions as a workaround. You need to add multiple tasks to be executed on the conditions. See below example:

- template: templates/changed.yaml
  parameters:
    comparedTo: origin/production
- template: templates/unitTests.yaml
  dependsOn: Changed

---

#unitTests.yaml

stages:
- stage: UnitTests
  displayName: Run unit tests on service...
  dependsOn: Changed
  condition: succeeded()
  jobs:
  - job: UnitTests
    condition: or(eq(stageDependencies.Changed.Changes.outputs['detectChanges.anyServicesChanged'], true), eq(variables['Build.Reason'], 'Manual'))
    displayName: Running unit tests...
    
    variables:
      changedServices: $[stageDependencies.Changed.Changes.outputs['detectChanges.servicesChanged']]
    
    steps:
    - bash: |
         echo "Now running api unit tests..."
      name: Api-unit-test
      conidtion: contains(variables['changedServices'], 'api')
   
    - bash: |
         echo "Now running admin unit tests..."
      name: admin-unit-test
      conidtion: contains(variables['changedServices'], 'admin')
    
    - bash: |
         echo "Now running client unit tests..."
      name: client-unit-test
      conidtion: contains(variables['changedServices'], 'client')

   

Another workaround is to separate your pipeline into two pipelines. The fist pipeline to run the Changed stage. And Then call rest api in a script task to trigger the second pipeline and pass the variables in the request body. See this similar thread.