ADF IF Condition - Count source columns

Question

ADF Copy activity - i have two types of flat files stored in the same directory, i am reading it with a wildcard *.txt

SOURCE type 1 has 18 columns, type 2 has 24 columns. the first 18 columns are identical, type 2 only has an additional 6 cols.

SINK azure sql table with 24 columns

Is it possible to use an IF condition to count the columns, and pass each source file into the correct copy activity - I would then have 2 copy activities (1 for each type of file).

I believe you can use the same copy activity, and will get NULLs for the fields that are missing in the smaller file. Is there a reason you prefer them separate? — d1sh4

Nandan Nandan · Accepted Answer · 2021-08-16T09:46:29

You can use GetMetaData activity to get the structure of file and even count of columns. And then via If activity, split them for 2 different copy activities or else use dynamic mapping based on if condition leveraging only 1 copy activity

https://sqlitybi.com/dynamically-set-copy-activity-mappings-in-azure-data-factory-v2/

ADF IF Condition - Count source columns

1 Answers