Azure Data Factory Cannot retrieve output

Question

With ADF I created web activity with Post method and got my output. I need to get the output that's in json to be stored in my blob storage. I tried copy data method but it gives me error of "Failure happened on 'Source' side.

ErrorCode=UserErrorHttpStatusCodeIndicatingFailure,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 404 indicates failure. { "statusCode": 404, "message": "Resource not found" },Source=Microsoft.DataTransfer.ClientLibrary,'"

I have added dynamic value as an input in the copy activity but gives me error in debug. I am able to see the desired output in the input(debug mode) but output in not the same. Here is what it shows as output:

{
    "effectiveIntegrationRuntime": "DefaultIntegrationRuntime (West US)",
    "executionDuration": 0,
    "durationInQueue": {
        "integrationRuntimeQueue": 1
    },
    "billingReference": {
        "activityType": "ExternalActivity",
        "billableDuration": [
            {
                "meterType": "AzureIR",
                "duration": 0.016666666666666666,
                "unit": "Hours"
            }
        ]
    }
}

Goal here is to get the output stored in a blob as csv so I can import that file to sql table. How do I get the output stored in blob storage?

Use a Copy Activity instead of a Web Activity with an HTTP dataset as your source and the sink will be your target storage type. — Mark Kromer MSFT
You wont be able to do that since copy activity source side need to comes from dataset etc... not directly input like some other activity, you can use Web Activity mentioned by Mark Kromer, one other way to push the output to copy activity is on source side, choose a dataset that use sql server, choose query mode instead of table mode, and put in `SELECT @{...}', but it hardly accomplish your requirement and hard to maintain. — Phung Duy Phong
@MarkKromerMSFT Thank you for helping, I tried your method but still getting error. Do you have any example for this? This is the error I am getting> ErrorCode=HttpFileFailedToRead,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read data from http server. Check the error from http server：The remote server returned an error: (406) Not Acceptable.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (406) Not Acceptable.,Source=System — prayora kc
@PhungDuyPhong do you have example for this? I get the correct output in the web activity but while i am copying both input and output are not showing and still giving the same output as mentioned above. Thanks — prayora kc
@PhungDuyPhong I tried the alternative solution of adding a sql database and adding the query to get the data but its giving me error on source side or it would run successfully with no data in it. Any suggestions? — prayora kc

prayora kc prayora kc · Accepted Answer · 2021-01-14T16:33:00

I was able to get this to work by simply using copy activity. The reason why it was failing at first was because of the relative URL. On rest data after adding connection, I separated base and relative URL. I added rest data as a source and for sink I just added SQL table. Was able to store the response directly to table.

Azure Data Factory Cannot retrieve output

2 Answers