Calling Webjob from Azure Data Factory pipeline throwing HTTP 409 conflict exception error

Question

I have a OnDemand triggered webjob that I want to trigger through ADF copy activity using HTTP linked service. Here is the linked service:-

{
    "name": "LS_WebJob",
    "properties": {
        "hubName": "yas-cdp-adf_hub",
        "type": "Http",
        "typeProperties": {
            "url": "https://cust-app.scm.azurewebsites.net/api/triggeredwebjobs/ConsoleApplication1/run",
            "authenticationType": "Basic",
            "username": "$custdata-app",
            "password": "**********"
        }
    }
}

Input Dataset

{
    "name": "ZZ_Inp_Webjob",
    "properties": {
        "published": false,
        "type": "Http",
        "linkedServiceName": "LS_WebJob",
        "typeProperties": {
            "requestMethod": "Post",
            "requestBody": "Hey Buddy"
        },
        "availability": {
            "frequency": "Day",
            "interval": 1,
            "style": "StartOfInterval"
        },
        "external": true,
        "policy": {}
    }
}

Output Dataset

{
    "name": "ZZ_Out_WebJob",
    "properties": {
        "published": false,
        "type": "AzureBlob",
        "linkedServiceName": "LS_ABLB",
        "typeProperties": {
            "fileName": "webjob.json",
            "folderPath": "yc-cdp-container/Dummy/temp",
            "format": {
                "type": "TextFormat"
            }
        },
        "availability": {
            "frequency": "Day",
            "interval": 1,
            "style": "StartOfInterval"
        }
    }
}

Pipeline

{
"name": "ZZ-PL-WebJob",
"properties": {
    "description": "This pipeline copies data from an HTTP Marina WiFi Source URL to Azure blob",
    "activities": [
        {
            "type": "Copy",
            "typeProperties": {
                "source": {
                    "type": "HttpSource"
                },
                "sink": {
                    "type": "BlobSink",
                    "writeBatchSize": 0,
                    "writeBatchTimeout": "00:00:00"
                }
            },
            "inputs": [
                {
                    "name": "ZZ_Inp_Webjob"
                }
            ],
            "outputs": [
                {
                    "name": "ZZ_Out_Webjob"
                }
            ],
            "policy": {
                "timeout": "01:00:00",
                "concurrency": 1
            },
            "scheduler": {
                "frequency": "Day",
                "interval": 1,
                "style": "StartOfInterval"
            },
            "name": "WebjobSourceToAzureBlob",
            "description": "Copy from an HTTP source to an Azure blob"
        }
    ],
    "start": "2017-04-10T01:00:00Z",
    "end": "2017-04-10T01:00:00Z",
    "isPaused": false,
    "hubName": "yas-cdp-adf_hub",
    "pipelineMode": "Scheduled"
}
}

My webjob is a simple C# application:-

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("My Team Rocks!");
        }
    }
}

When I am executing the pipeline, the webjob is being successfully triggered. However the pipeline fails with HTTP 409 conflict error.

Copy activity encountered a user error at Source side: ErrorCode=UserErrorFailedToReadHttpFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read data from http source file.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (409) Conflict.,Source=System,'.

Do you also get 409 if you manually call the WebJobs? If not, try to identify differences between the two invocations — David Ebbo
Thanks for the reply @David..as I mentioned, webjob is being successfully triggered be it manual run or through ADF. — Ritesh
Pretty confused here as your title implies that the WebJob fails with a 409. Are you saying that the http call to the WebJob is actually not failing at all, but that only ADF is failing? If so, the question is probably not related to WebJobs. Try isolating. — David Ebbo
Thanks David, I changed the title and hope it makes sense now. Could you please suggest the solution now ? — Ritesh
Still not clear. It's the http request made to invoke the webjob returning the 409 or is it something else that's returning the 409? You wrote above that the webjob is successfully invoked both manually and through ADF, implying a 200 response, and that seems contradictory. — David Ebbo

Carol Carol · Accepted Answer · 2017-10-10T01:32:17

Try adding in the gateway name to the linked service json. Refer to this link How to integrate a WebJob within an Azure Data Factory Pipeline .

Calling Webjob from Azure Data Factory pipeline throwing HTTP 409 conflict exception error

1 Answers