How to Trigger multiple pipelines in a project on azure devops without the other pipeline being queued for completing run on 1st pipeline

Question

I have got two repositories on github which host code ne for front end and other for backend. I have set up 2 yaml CI-CD pipelines on my project in azure devops one which is configured to trigger for change in the Frontend repo and other in Backend repo. When i trigger any change in Frontend repo,the frontend pipeline gets triggered and When i trigger any change in backend repo,the backend pipeline gets triggered.

I have set up multi stage yaml pipelines ex: Build-> deploy->run tests in both the pipelines .When i trigger both pipelines simultaneously, one pipeline gets queued up and waits until the first stage of 1 pipeline gets completed,then the 1st stage of 2nd pipeline is triggered then the 2nd stage of 1st pipeline then 2nd stage of second pipeline. I want to avoid this thing.

 FrontEnd Pipeline-> stage 1    
 BackendEnd Pipeline-> stage 1 
 FrontEnd Pipeline-> stage 2    
 BackendEnd Pipeline-> stage 2 
 FrontEnd Pipeline-> stage 3    
 BackendEnd Pipeline-> stage 3

this takes a lot of time and says that the agent is busy . I want to run these pipelines simultaneously like:

FrontEnd Pipeline-> stage 1 
FrontEnd Pipeline-> stage 2
FrontEnd Pipeline-> stage 3 

getting triggered seperately and

BackendEnd Pipeline-> stage 1
BackendEnd Pipeline-> stage 2
BackendEnd Pipeline-> stage 3

getting triggered seperately at the same time so that both pipelines do execute their own stages. How can this be achieved.

Answer 1

Unfortunately I am pretty sure that this type of feature depends on how much you are willing to pay. In my organization we pay for 2 agents, and that allows for 2 stages to be executed at the same time. The stages could be within a single pipeline or across different pipelines.

Refer to this Microsoft Document for details. I believe this points you in the right direction. https://docs.microsoft.com/en-us/azure/devops/pipelines/licensing/concurrent-jobs?view=azure-devops&tabs=ms-hosted

If you only have one agent, then it will stair-step the completion of 2 pipelines that are triggered simultaneously like you have posted

 FrontEnd Pipeline-> stage 1    
 BackendEnd Pipeline-> stage 1 
 FrontEnd Pipeline-> stage 2    
 BackendEnd Pipeline-> stage 2 
 FrontEnd Pipeline-> stage 3    
 BackendEnd Pipeline-> stage 3

I believe you can trigger a pipeline with a specific agent pool, which may allow you to run in a simultaneous manner like you are wanting, but you would have to have multiple self-hosted pools probably. Otherwise, it will still run in a stair-step fashion between the two pipelines, as it queues the jobs between both pipelines.

I should also mention the agents depend on whether you are free tier or private tier of Azure.

That document should give you some good insight though. Please post what you conclude if you run any tests etc.

Answer 2

A workaround is triggering the BackendEnd pipeline via REST api and through the PowerShell task. You could use the REST API Builds - List to get the detailed build info of FrontEnd Pipeline and check the latest build status:

https://dev.azure.com/{organization}/{project}/_apis/build/builds?definitions={definitions}&api-version=6.0

In the YAML, we could add a powershell task to get the build status, like:

- task: PowerShell@2
  inputs:
   targetType : inline
   script: |
     $url = "https://dev.azure.com/{organization}/{project}/_apis/build/builds?definitions={definitionID}&api-version=6.0"
     $connectionToken="Your PAT Here"
     $base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))

       $buildPipeline= Invoke-RestMethod -Uri $url -Headers @{authorization = "Basic $base64AuthInfo"} -Method Get


       $BuildStatus= $buildPipeline.value.status | Select-Object -first 1


       Write-Host This is Build Status: $BuildStatus

This list all the build status for the specify definitions, then use Select-Object -first 1 to get the latest build status. If the status is completed, then queue the BackendEnd pipeline. If the status is not completed, do not queue the BackendEnd pipeline.