How to Accessing pipeline properties and data set properties from an activity

Question

I have found this documentation, Azure Data Factory - Functions and System Variables, however it does not show how I can access other items. Such as the name of the pipeline so that I can pass it as a property value for example in a SqlServerStoredProcedure, or as part of creating a path, etc.

I would like to be able to add the pipeline name (or other properties of the pipeline) in this "source": "$$Text.Format('{0}/scored/{1:yyyy}/{1:MM}/{1:dd}/ActivityMetrics_NameAndResult_AD_Scored_{1:yyyy}_{1:MM}_{1:dd}.tsv', PipelineName, SliceStart)"

I would like to be able to reference properties of the input/output datasets as well,

"destination":"OTEL_ActivityMetrics_NameAndResult_Scored",

Here I would like to be able to reference the tableName property of the InputDataSet.

Why: We want to log certain actions during our pipeline, we have a series of reports and alerting tools that look at this history and/or lack of history, currently these are hard coded and this will lead to either us having to generate them via a more complex process or for them to be wrong sometimes and not be easy to notice. For example in addition to the example below we also call a rebuild index stored procedure which takes a table name, I could copy the activity from one pipeline to another and forget to change the storedProcedureParameter,

        {
            "type": "SqlServerStoredProcedure",
            "typeProperties": {
                "storedProcedureName": "LogDataMovement",
                "storedProcedureParameters": {
                    "sliceStart": "$$Text.Format('{0:yyyy-MM-dd HH:mm}', SliceStart)",
                    "movementType": "Blob->Sql",
                    "source": "$$Text.Format('{0}/scored/{1:yyyy}/{1:MM}/{1:dd}/ActivityMetrics_NameAndResult_AD_Scored_{1:yyyy}_{1:MM}_{1:dd}.tsv', PipelineName, SliceStart)",
                    "destination":"OTEL_ActivityMetrics_NameAndResult_Scored",
                    "pipeline": "ActivityMetricsNameAndResult_AzureSQL"
                }
            },
            "inputs": [
                {
                    "name": "DS_ASQL_ActivityNameAndResult_AD_Scores"
                }
            ],
            "outputs": [
                {
                    "name": "DS_SQL_DataMovement_NameAndResult_Scored"
                }
            ],
            "scheduler": {
                "frequency": "Day",
                "interval": 1
            },
            "name": "Log Data Movement"
        },

Gaurav Malhotra Gaurav Malhotra · Accepted Answer · 2016-11-04T16:38:28

Today Azure Data Factory System and Function variables are the only things that are accessible inside a pipeline. You cannot access pipeline names and pass it around as variables. We are working to evolve the app model in ADF to support such constructs. As we make more progress and land those updates, we will keep the community updated.

How to Accessing pipeline properties and data set properties from an activity

1 Answers