Azure Data Factory V2 Dynamic Content

Question

Long story short, I have a data dump that is too large for an azure function. So we are using Data Factory. I have tasked another function to generate an access token for an API and output it as part of a json. I would like to set that token to a variable within the pipeline. So far I have this:

I'm attempting to use the Dynamic Content "language" to set the variable:

@activity('Get_Token').output

I'd like something like pythons:

token = data.get('data', {}).get('access_token', '')

As a secondary question, my next step is to use this token to call an API while iterating over another output, so perhaps this exact step can be added into the ForEach?

@activity('Get_Token').output is correct. I don't get your point about token = data.get('data', {}).get('access_token', ''). Would you please give more details or statemetns? — Jay Gong
I would like to set the variable to a value within the json, not the json in its entirety. — Robert Riley
Output is JSON, so it should something like @activity('Get_Token').output.YourPropertyName or @activity('Get_Token').output.value.YourPropertyName — Joel Cochran

Jason Welch Jason Welch · Accepted Answer · 2020-05-30T23:22:43

Looks like the variable should be @activity('Get token').output.data.access_token as others have indicated but, as you've guessed, there's no need to assign a variable if you only need it within the foreach. You can access any predecessor output from that successor activity. Here's how to use the token while iterating over another output:

Let's say your function also outputs listOfThings as an array within the data key. Then you can set the foreach activity to iterate over @activity('Get token').output.data.listOfThings.
Inside the foreach you will have (let's say) a Copy activity with a REST dataset as the source. Configure the REST linked service with anonymous auth ...
... then you'll find a field called Additional Headers in the REST dataset where you can create a key Authorization with value as above, Basic @activity('Get token').output.data.access_token
The thing that you said you want to iterate over (in the listOfThings JSON array) can be referenced inside the foreach activity with @item() (or, if it's a member of an item in the listOfThings iterable then it would be @item().myMember)

To make #4 explicit for anyone else arriving here:

If listOfThings looks like this, listOfThings: [ "thing1", "thing2", ...]
for example, filenames: ["file1.txt", "file2.txt", ...]
then @item() becomes file1.txt etc.

whereas

If listOfThings looks like this, listOfThings: [ {"key1":"value1", "key2":"value2" ... }, {"key1":"value1", "key2":"value2" ... }, ...]
for example. filenames: [ {"folder":"folder1", "filename":"file1.txt"}, {"folder":"folder2", "filename":"file2.txt"}, ... ]
then @item().filename becomes file1.txt etc.

Azure Data Factory V2 Dynamic Content

1 Answers