Calculate the Total time for pipeline to complete the operation? Not for the individual slices

0
votes

Environment & Details

  • Azure data factory
  • On premise - IaaS Server
  • Azure Data Lake

Scenario

  • I have developed ADF pipeline and deployed to azure data factory. As part of pipeline - i have almost 50 copy activities which copies the data from On premise server and write the data to azure data lake in CSV format.

  • I have one pipeline under which all 50 copy activities are present

  • All the activities start at 12:00 AM UTC time

Problem

  • How do i calculate the Total time for pipeline to complete the operation? Not for the individual slices.
1
azure-data-factoryazure-data-lake
Run it......... – wBob
Pipeline is running @ 12:00 AM UTC. If i look at the individual slices,it is taking 1 or 2 minutes to completes. There are almost 50 activities - and they take 1 or 2 minutes to completes. I want to know, how much the TOTAL Amount of time taken by ADF Pipeline – Mangesh Tambare

1 Answers

0
votes

You can use PowerShell to return this info. For example:

$SubId = Get-AzureSubscription `
    -SubscriptionName $AzureSubscription | SELECT SubscriptionId 

Set-AzureRmContext -SubscriptionId $SubId.SubscriptionId | Out-Null

#Get ADF details
$ADFName = Get-AzureRmDataFactory `
-ResourceGroupName $ResourceGroup | SELECT DataFactoryName

Get-AzureRmDataFactoryActivityWindow `
    -DataFactoryName $ADFName.DataFactoryName `
    -ResourceGroupName $ResourceGroup

This return a duration field.

You could then SUM the results or something. I write them to a SQL DB for monitoring our factories.

For details about the cmdlet here: https://docs.microsoft.com/en-us/powershell/module/azurerm.datafactories/get-azurermdatafactoryactivitywindow?view=azurermps-3.7.0

Hope this helps