Read API data from HTTPS webservice using Azure Data Factory

Question

I am trying to ingest data from Web services API, source is XML format as Azure Data Factory does not support XML format, I am reading it in binary format loading into Azure data lake gen2, Then converting into xml, csv using Azure Databricks.

Problem: I am able to read the tokenid from API but not actual data means using token id i have to get the actual data.Please can anyone help on this. Note: Used HTTP linked service to ingest the data. All connectors are fine and test connections are successful. Below is the error message i can see from ADF.

"errorCode": "2200",
"message": "Failure happened on 'Sink' side. ErrorCode=UserErrorFailedToReadHttpFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The remote server returned an error: (500) Internal Server Error.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (500) Internal Server Error.,Source=System,'",
"failureType": "UserError",
"target": "Copy data1",
"details": []

}

My apologies if anyone do not understand the problem, I can post clearly. Thanks,

Jay Gong Jay Gong · Accepted Answer · 2020-01-02T08:52:25

Based on your description, i think you could use WehHook Activity,not Web Activity.

Using the WehHook activity, you can call an endpoint and pass a callback URL. The pipeline run waits for the callback to be invoked before proceeding to the next activity.

The actual step is passing an additional property “callBackUri”(Ingest Data API) in the body to the url endpoint(Get Token API), and will expect this uri to be invoked before the timeout value specified.

Updates:

Finally,the OP chosen databricks as solution and it works fine.

Read API data from HTTPS webservice using Azure Data Factory

1 Answers