How to trigger an AzureML Pipeline from Azure DevOps?

Question

If we have an AzureML Pipeline published, how can we trigger it from Azure DevOps without using Python Script Step or Azure CLI Step?

The AzureML Steps supported natively in Azure DevOps include Model_Deployment and Model_Profiling.

Is there any step in Azure DevOps which can be used to directly trigger a published Azure Machine Learning Pipeline while maintaining capabilities like using Service Connections and passing environmental variables, Gated Release (Deployment)?

Edit: This process can then be used to run as an agentless job.

Answer 1

I am afraid there is no other steps available in Azure Devops which can directly trigger a published azure ml pipeline. You have to use Python Script Step or Azure CLI Step in azure devops pipeline to trigger azure ml pipeline.

To trigger azure ml pipeline using azure cli task in azure devops pipeline. You can check out below steps.

1, Create an azure pipeline. See example here.

2, Create an azure Resource Manager service connection to connect your Azure subscription to Azure devops. See this thread for an example

3, Add Az cli task in your yaml pipeline. Run below scripts as inline scripts. See document here for more information.

steps:
- task: AzureCLI@2
  displayName: 'Azure CLI '
  inputs:
    azureSubscription: 'azure Resource Manager service connection'
    scriptType: ps
    scriptLocation: inlineScript
    inlineScript: |
     
     #To install the Machine Learning CLI extension
     az extension add -n azure-cli-ml

     az ml run submit-pipeline --pipeline-id "{id}"

Update:

If you want to avoid using build agents. You can run the invoke rest api task in an agentless job. See below steps:

1, Create a Generic service connection in azure devops. See here for creating service connection.

2, Add below url as the Server URL of the generic service connection. See here for more information about below url.

3, Add a agentless job(server job) in your pipeline. Add invoke rest api task in this agentless job. So that, the pipeline will execute the invoke rest api task to trigger the azureml pipeline without using a build agent.

You can also setup an azure logic app in your azure subscription.

You can set the logic app trigger as azure devops events. Or you can set a http request as the trigger events(You can the use invoke rest api task or azure devops web hook to call this http request to trigger this logic app).

And then add a HTTP action with the url as above url screenshot. Please see here for more information.

Answer 2

Assumptions:

1. An AzureML Pipeline is published and the REST endpoint is ready- To be referred to in this answer as <AML_PIPELINE_REST_URI>. And Published Pipeline ID is also ready- To be referred to in this answer as <AML_PIPELINE_ID>
2. You have the Azure Machine Learning Extension installed: Azure Machine Learning Extension

To Invoke the Azure Machine Learning Pipeline we use the Invoke ML Pipeline step available in Azure DevOps. It is available when running an Agentless Job.

To trigger it the workflow is as follows:

1. Create a New Pipeline. Using the Classic Editor, delete the default Agent Job 1 stage.

2. Add an agentless job:

3. Add a task to this Agentless Job:

4. Use AzureML Published Pipeline Task:

5. Use the Service Connection Mapped to the AML Workspace. You can find more on this at the official documentation

6. Choose the Pipeline to trigger using the <AML_PIPELINE_ID>:

7. Give The experiment name and Pipeline Parameters if any:

8. That's it, you can Save and Queue:

Alternatively, you can simply use the following jobs:

- job: Job_2
  displayName: Agentless job
  pool: server
  steps:
  - task: MLPublishedPipelineRestAPITask@0
    displayName: Invoke ML pipeline
    inputs:
      connectedServiceName: <REDACTED-AML-WS-Level-Service_Connection-ID>
      PipelineId: <AML_PIPELINE_ID>
      ExperimentName: experimentname
      PipelineParameters: ''