How do we create a generic mapping dataflow in datafactory that will dynamically extract data from different tables with different schema?

2
votes

I am trying to create a azure datafactory mapping dataflow that is generic for all tables. I am going to pass table name, the primary column for join purpose and other columns to be used in groupBy and aggregate functions as parameters to the DF. parameters to df

I am unable to refernce this parameter in groupBy Error: DF-AGG-003 - Groupby should reference atleast one column - MapDrifted1 aggregate( ) ~> Aggregate1,[486 619]

Has anyone tried this scenario? Please help if you have some knowledge on this or if it can be handled in u-sql script.

1
azureazure-data-factoryazure-data-factory-2u-sqlazure-data-flow
Can you show what you have done? like with some snapshot of your mapping dataflow. it will be more understandable. – Sohel Reza
i wouldnt do anything with u-sql. it's not even supported in adla2 – Alex Gordon

1 Answers

3
votes

We need to first lookup your parameter string name from your incoming source data to locate the metadata and assign it.

Just add a Derived Column previous to your Aggregate and it will work. Call the column 'groupbycol' in your Derived Column and use this formula: byName($group1).

In your Agg, select 'groupbycol' as your groupby column.