Custom activity failed in Azure Data Factory after running several hours

Question

I was running custom .Net activity in Azure Data Factory using On-demand HDInsight cluster. Activity processes XML files, stored in Azure Blob, and move them to Azure Data Lake Store. It failed after 28 hours of execution with the following error:

"Error in Activity: The request was aborted: The request was canceled.."

There were no log files available for this activity run and above error is not good enough to troubleshoot the problem. How can I troubleshoot this problem?

Looks like the compute requirement of your activity is much more than what your config of the on demand cluster is capable. Is it the first time it happened? Have you defined logging at all necessary check points in your custom code? See if you are able to reproduce the issue with BYOC or Azure Batch.It is quite a pain to work with on demand clusters — Sandesh

Aaron Kim Aaron Kim · Accepted Answer · 2016-04-29T01:56:17

I suggest you check the system log. Even if you didn't have any user log, you shall have a system log which will help you investigate your issue.

You should be able to check system log from detail of failed output dataset(see below)

When you create a linkedServce for ondemand HDInsight, you are supposed to specify "linkedServiceName" in typeProperties

{
    "$schema": "http://datafactories.schema.management.azure.com/schemas/2015-08-01/Microsoft.DataFactory.LinkedService.json",
    "name": "LinkedServiceOnDemand_OnCloud_HDInsight",
    "properties": {
        "type": "HDInsightOnDemand",
        "typeProperties": {
            "clusterSize": 1,
            "timeToLive": "00:10:00",
            "linkedServiceName": "LinkedService_OnCloud_Storage"
        }
    }
}

Logs will be created in that storage. If it hadn't been created, the only option left would be getting a technical support from Microsoft.

Custom activity failed in Azure Data Factory after running several hours

1 Answers