Passing parameters to copy activity using filter activity

Question

I get a list of files in folder with get metadata activity, then the files are sorted out with the help of filter activity. Now I want to pass the names of these files to copy activity.

Here is the output of filter activity (There is only one file because it is within for each activity.):

{
    "ItemsCount": 2,
    "FilteredItemsCount": 1,
    "Value": [
        {
            "name": "part-00000-622d6021-26bc-4ad5-9e4c-0d80cec7c6b7-c000.csv",
            "type": "File"
        }
    ]
}

How can I pass the name to copy activity on the source side?

For example, I have already tried: @activity('Filter1_copy1').output.value - @activity('Filter1_copy1').output.value[0]

Everything I have tried leads me to error:

ErrorCode=UserErrorInvalidValueInPayload,'Type=Microsoft.DataTransfer.Common.Shared.HybridDelivery
Exception,Message=Failed to convert the value in 'fileName' property to 'System.String' type. 
Please make sure the payload structure and value are correct.,Source=Microsoft.DataTransfer.DataContracts,
''Type=System.InvalidCastException,Message=Object must implement IConvertible.,Source=mscorlib,'

If your ForEach is looping over the Filter's results, use @item().name to get the current iteration's file name. — Joel Cochran

wBob wBob · Accepted Answer · 2021-12-20T16:58:56

The Azure Data Factory (ADF) expression language is not the most intuitive thing in the world, but there is a strong central element which is really just JSONPath, like XPath, the simple language to interrogate complex JSON objects. This uses a dot syntax to access properties ( eg parent.child ) and an array syntax to access arrays Values[0].name.

The Filter activity (like the Get Metadata activity) returns a complex piece of JSON which is the result of the activity, represented as an object.

In your particular example, you have an object with a Values array which can have zero or many items. Remember arrays are zero-based in JPath. Therefore in order to access the first value of this array, we need to use a zero-based position indictor known as ordinal ([0]), and the dot syntax to get the properties, eg name

@activity('Filter1').output.Value[0].name

Remember case is important in JSON and JSONPath, so it is Value (upper-case V) for the array. Amend the ADF activity name as required (eg Filter1_copy instead of Filter1 in my example, but I'd encourage you to look at a naming convention : )

Just for fun, I use jsonpath.com to practice these expressions without having to run and re-run the pipeline every time to see if it's worked. It can get you kind of in the right place. After a bit of practice you'll be able to look at your activity output and work out what the expression should be. A simple example:

I don't think ADF supports the full JSONPath syntax but I just pasted in your sample JSON, and wrote the Values[0].name bit to test.

Passing parameters to copy activity using filter activity

1 Answers