Azure Data Factory Copy Activity error mapping JSON to SQL

Question

I have an Azure Data Factory Copy Activity that is using a REST request to elastic search as the Source and attempting to map the response to a SQL table as the Sink. Everything works fine except when it attempts to map the data field that contains the dynamic JSON. I get the following error:

{ "errorCode": "2200", "message": "ErrorCode=UserErrorUnsupportedHierarchicalComplexValue,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The retrieved type of data JObject with value {\"name\":\"department\"} is not supported yet, please either remove the targeted column or enable skip incompatible row to skip them.,Source=Microsoft.DataTransfer.Common,'", "failureType": "UserError", "target": "CopyContents_Paged", "details": [] }

Here's an example of my mapping configuration:

    "type": "TabularTranslator",
    "mappings": [
        {
            "source": {
                "path": "['_source']['id']"
            },
            "sink": {
                "name": "ContentItemId",
                "type": "String"
            }
        },
        {
            "source": {
                "path": "['_source']['status']"
            },
            "sink": {
                "name": "Status",
                "type": "Int32"
            }
        },
        {
            "source": {
                "path": "['_source']['data']"
            },
            "sink": {
                "name": "Data",
                "type": "String"
            }
        }
    ],
    "collectionReference": "$['hits']['hits']"
}

The JSON in the data object is dynamic so I'm unable to do an explicit mapping for the nested fields within it. That's why I'm trying to just store the entire JSON object under data in a column of a SQL table.

How can I adjust my mapping configuration to allow this to work properly?

kwyss88 kwyss88 · Accepted Answer · 2019-12-10T19:07:12

I posted this question on the MSDN forums and I was told that if you are using a tabular sink you can set this option "mapComplexValuesToString": true and it should allow complex JSON properties to get mapped correctly. This resolved my ADF copy activity issue.

Azure Data Factory Copy Activity error mapping JSON to SQL

2 Answers