Azure Pipelines - Parallel steps (YAML)

17
votes

I'm setting up my Azure Devops Pipelines, and have a build that requires some fairly lengthy setup steps to run. These need to run before other tasks, which can be run in parallel.

However, I can only see this being done by specifying jobs, which would require to do these lengthy steps each time. Ie:

jobs:
  - job: Run1
     steps:
       - task: Long running setup task
       - task: Run taskA
  - job: Run2
       - task: Long running setup task
       - task: Run taskB

Is there a way to have this long running task run, and have task A/B depend on that environment without running them sequentially? Ideally it'd be something like:

-job
  steps:
    -task: Long running setup
    -task: Parallel: taskA
    -task: Parallel: taskB

Or have the previous jobs take a container/image snapshot and reuse if that's possible?

2
azure-devopsazure-pipelines
I have the same question. I have a build task and a subsequent validation task. The validation task requires install of json lint. Running the build and json lint install in parallel would be quicker. – Nick.McDermaid
Any update on this one? – Skizo-ozᴉʞS

2 Answers

14
votes

Short answer, you can't.

Tasks within a job cannot run in parallel as they run on the same agent and the environments can't be "snapshotted" by Azure Devops to be re-utilized by other jobs in parallel later. But jobs can run in parallel as they can be scheduled on different agents, so setup will run twice but in parallel. So there is a trade-off between time and resource usage which you will need to decide on based on your requirements.

There is another solution though, based on how much you would like to invest in this:

If your "setup" doesn't change that often and you are willing to host your own agents. Then you could run a separate "setup + agent" build which creates a docker image of your agents, pushes it out to your azure container registry and then deploys this image to your self-hosted agents (Azure Kubernetes Service) cluster. Then Task A and Task B can easily run in parallel with the assumption that the environment they are running in (agent + setup docker image) is always ready. This is exactly my setup.

See : Azure DevOps Docker

0
votes

An update to this - a docker image as suggested by @dparker, while it is probably a better way, was a bit OTT for myself. Instead I used a pipeline artifact to cache some build / setup files. And then each other job was made dependent on this setup job.

This obviously doesn't sound great, but it works fine to get the performance optimizations I was after.

Eg Job1 would include this:

  - task: PublishPipelineArtifact@0
    inputs:
      artifactName: 'Setup-Build'
      targetPath: '$(buildDir)'

And Job2 to X would download this as an artifact:

- task: DownloadPipelineArtifact@1
    displayName: 'Download Setup'
    inputs:
      targetPath: '$(buildDir)'
      artifactName: 'Setup-Build'

Additionally there is the option to use caching, but this didn't quite fit in with my scenario. I'd recommend you make a call between artifacts and caching: https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops