Creating JSON Array in Azure Data Factory with multiple Copy Activities output objects

Question

Is it possible to embed the output of a copy activity in Azure Data Factory within an array that is meant to be iterated over in a subsequent ForEach?

My goal is to create an array with the output of several copy activities and then in a ForEach, access the properties of those copy activities with dot notation (Ex: item().rowsRead). Image shows code details.

Image

Specifically, I have 7 copy activities whose output JSON object (described here) would be stored in an array that I then iterate over. In the ForEach I would be checking the properties on each of the copy activities (rowsRead, rowsCopied, etc.) for validation purposes. https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-monitoring

Do you mean the output of a Copy activity in terms of a Sink or the debugging output? If it’s the first then that is not possible in the way you describe. You would need a separate Lookup activity. It would be better if you try and describe what you want to do more functionally before thinking about it in terms of ADF tasks and I’m sure someone will be able to help you. — wBob
Yes I mean the output of several Copy activities after they've completed with source and sink details as seen here. docs.microsoft.com/en-us/azure/data-factory/… - I'm particularly interested in the rowsRead and rowsCopied properties. — Godwin

Joseph Xu Joseph Xu · Accepted Answer · 2021-01-18T06:46:22

I think we can embed the output of a copy activity in Azure Data Factory within an array. I've created a test to save the output of 2 Copy activities into an array. We need to concat a string type and then convert it to json type. Please see my step2.

We can declare an array type variable named CopyInfo to store the output. The another array type variable named JsonArray is used to see the test result at debug mode.
In Append variable1 activity, I use @json(concat('{"activityName":"Copy1","activityObject":',activity('Copy data1').output,'}')) to save the output of Copy data1 activity and convert it from String type to Json type.
In Append variable2 activity, I use @json(concat('{"activityName":"Copy2","activityObject":',activity('Copy data2').output,'}')) to save the output of Copy data2 activity and convert it from String type to Json type.
Then I assign the value of variable CopyInfo to variable JsonArray
In the end, we can see the json array like :

"name": "JsonArray",
"value": [
    {
        "activityName": "Copy1",
        "activityObject": {
            "dataRead": 643,
            "dataWritten": 643,
            "filesRead": 1,
            "filesWritten": 1,
            ...
    },
    {
        "activityName": "Copy2",
        "activityObject": {
            "dataRead": 643,
            "dataWritten": 643,
            "filesRead": 1,
            "filesWritten": 1,
            ...
            }
    }
]

Creating JSON Array in Azure Data Factory with multiple Copy Activities output objects

1 Answers