Using mapping data flows within for-each activity of azure data-factory pipeline

1
votes

Scenario: I have SQL DB CDC net changes coming into a csv file. In the pipeline for each row in the csv file need to check for updates, inserts and deletes and do some changes in my warehouse DB.

Pipeline:

pipeline

mapping dataflow:

dataflow

Problem:

How to use the item() from foreach in dataflow and fetch that record from csv file and process. I'm not seeing any field/option in source step of dataflow.

2
azure-data-factoryazure-data-factory-2azure-data-flow

2 Answers

0
votes

Firstly, you could add a parameter in Data Flow:

enter image description here

Then out of the Data flow, click the Data Flow, set the data flow parameter with Pipeline expression:

enter image description here

Then you could set the Foreach item() as the dataflow parameter:

enter image description here

Now,you can use the item() from foreach in dataflow and fetch that record from csv file and process.

Hope this helps.

1
votes

No need to use ForEach here. The data flow can read each line from your CDC source file and you can apply the appropriate insert, update, merge, delete operation as policies in the Alter Row transformation.