Azure Data Factory - Copy activity mappings

Question

I am trying to copy data from a json file to a database. There are two columns on the database that need to be filled with the same json field. However, if I do this mapping, I get an error when running the activity: An item with the same key has already been added. Is it not possible to do this one to many mapping?

The following thread may be useful: social.msdn.microsoft.com/Forums/azure/en-US/… — Alberto Morillo

Yusheng Yusheng · Accepted Answer · 2018-02-06T07:17:20

Map one column in source dataset schema to multiple columns in destination is not supported yet in ADF.

But there are other ways to achive this, by altering the source dataset schema to let it have a duplicate column. Then you can use these duplicate columns to map to difference destination columns.

Like for SQL dataset, you can use sql query like "select column1 as column1A, column1 as column1B from xxx" to get duplicate columns in source dataset. Then you can map column1A to destCol1, column1B to destCol2.

As for JSON file in this case, you can use jsonPathDefinition to define duplicate columns. Reference https://docs.microsoft.com/en-us/azure/data-factory/supported-file-formats-and-compression-codecs#json-format

For example you have a json with format like

{"a":1,"b":2}

Then you can set the dataset format property like

"format": {
    "type": "JsonFormat",
    "filePattern": "setOfObjects",
    "jsonPathDefinition": {
        "a1": "$.['a']",
        "b": "$.['b']",
        "a2": "$.['a']"
    }
},

It will create 3 columns a1, b, a2 in the source dataset schema with a1 and a2 refering to the same json field a. Then you will be able to map these columns to different columns in destinations.

Thanks

Azure Data Factory - Copy activity mappings

1 Answers