Accessing pipeline activity status from an Azure Function

1
votes

I have an Azure Function which triggers a Pipeline and I'm able to poll the pipeline status to check when it completes using: Pipeline.Properties.RuntimeInfo.PipelineState

My pipeline uses several parallel Copy activities and I'd like to be able to access the status of these activities incase they fail. The Azure documentation describes how to access the pipeline activities but you can only get at static properties like name and description but not dynamic properties like Status (like you can for the Pipeline via its RuntimeInfo property).

For completeness, I've accessed the activity list using:

IList<Microsoft.Azure.Management.DataFactories.Models.Activity> activityList = plHandle.Pipeline.Properties.Activities;

Is it possible to check individual activity statuses programmatically?

2
azureazure-functionsazure-data-factory

2 Answers

1
votes

Its certainly possible.

I use the ADF PowerShell cmdlets in the Azure module to monitor our data factories.

Maybe do something like the below for what you need with Get-AzureRmDataFactoryActivityWindow command.

Eg:

$ActivityWindows = Get-AzureRmDataFactoryActivityWindow `
    -DataFactoryName $ADFName.DataFactoryName `
    -ResourceGroupName $ResourceGroup `
    | ? {$_.WindowStart -ge $Now} `
    | SELECT ActivityName, ActivityType, WindowState, RunStart, InputDatasets, OutputDatasets `
    | Sort-Object ActivityName

This gives you the activity level details including the status. Being:

  • Ready
  • In Progress
  • Waiting
  • Failed

... I list them because they differ slightly from what you see in the portal blades.

The datasets are also arrays if you have multiple inputs and outputs for particular activities.

More ADF cmdlets available here: https://docs.microsoft.com/en-gb/powershell/module/azurerm.datafactories/?view=azurermps-3.8.0

Hope this helps

0
votes

I've managed to resolve this by accessing the DataSliceRuns (i.e. activities) for the pipeline as follows:

var datasets = client.Datasets.ListAsync(<resourceGroupName>, <DataFactoryName>).Result;

    foreach (var dataset in datasets.Datasets)
    {
        // Check the activity statuses for the pipelines activities.
        var datasliceRunlistResponse = client.DataSliceRuns.List(<resourceGroupName>, <dataFactoryName>,<DataSetName>, new DataSliceRunListParameters()
                                                                        {
                                                                            DataSliceStartTime = PipelineStartTime.ConvertToISO8601DateTimeString()
                                                                        });

        foreach (DataSliceRun run in datasliceRunlistResponse.DataSliceRuns)
        {
            // Do stuff...
        }
    }