Including time elapsed/duration in success notification email from logic apps in data factory

Question

I built out a pipeline in Azure DataFactory to run a Databricks process. Depending on many variables, the databricks process can take 10 hours to run or 18 hours to run. I have an email notification set up through logic apps/Web activity in Data Factory to send me an email when the process is completely finished. However, I can not figure out how I pull in the "duration" of the trigger run into that success email.

This snippet is included in the "Body" section of the Web activity in Data Factory:

{"message":"@{activity('NAME OF ACTIVITY').output.dataWritten}","dataFactoryName":"@{pipeline().DataFactory}","pipelineName":"@{pipeline().Pipeline}","receiver":"@pipeline().parameters.receiver"}

This is the HTTP Request I have in Logic APP:

{
    "properties": {
        "DataFactoryName": {
            "type": "string"
        },
        "EmailTo": {
            "type": "string"
        },
        "ErrorMessage": {
            "type": "string"
        },
        "PipelineName": {
            "type": "string"
        },
        "Subject": {
            "type": "string"
        }
    },
    "type": "object"
}

I expect a success email sent to my email with Your pipeline took xx hrs and xx minutes to run on mm/dd/yyyy

Jay, Thanks for commenting! I appreciate the help. Unfortunately, I got really busy and haven't been able to devote a ton of time to this. I am going to do a deeper dive next week to see if this will be a good solution. Thanks again! — Alex Linke

Jay Gong Jay Gong · Accepted Answer · 2019-10-07T08:31:31

Based on System variables in ADF pipeline, you can't get the duration time of pipeline execution directly. However, you could get the pipeline trigger time.

So,my idea is using azure function activity after the Databricks activity. The main framework is Databricks activity ---> Azure Function Activity ---> Web Activity.

Step1: Process databricks pipleine in a Execute Pipeline Activity.

Step2: After that,execute Azure function activity and pass trigger time into azure function Inside azure function method, calculate duration time and output the string as you want.

Step3: Then pass the params into Web Activity and send email.

Including time elapsed/duration in success notification email from logic apps in data factory

1 Answers