How can I identify issues on an Azure Data Factory pipeline that works in debug mode but not when triggered?

0
votes

On Azure Data Factory I have a parametrized pipeline that uses the Copy Data activity, with the Source being OData and the Sink is an on-prem SQL server. They're being executed with an self-hosted integration runtime.

The pipeline was working successfully until last week or so, being able to dynamically copy data from 32 tables. Now, I'm having issues running the same pipeline for 2 of these tables. They work successfully when executed in debug mode, but not when executed via triggers, even though the parameters are the same. There are no changes to be published on these pipelines, I'm aware that the trigger executes the pipeline version published while debug executes it with unsaved changes.

Issue with table 1

The pipeline completely fails, the error is related to the table not being found in the sink. Error message:

Operation on target Load Data failed: Failure happened on 'Source' side. ErrorCode=UserErrorODataRequestNotSucceeded,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Requesting response from path [redacted] and query failed with status code NotFound and message The resource you are looking for has been removed, had its name changed, or is temporarily unavailable..,Source=Microsoft.DataTransfer.Runtime.ODataConnector,'

Again, when executed via debug, with the same parameters, it works as usual. I've tried removing the parametrization from the copy data activity and creating a separate pipeline for this table, but the result was the same, it's unable to find the table in the sink unless executed in debug.

Issue with table 2

The pipelines successfully executes, however it loads data only for 16 out of 20 columns. All of the column names and types are the same in the source and sink. When executed in debug mode, all 20 columns are populated. The copy data activity does not provide any details about those unpopulated columns, as seen below.

Copy activity details

I would expect a warning or something telling me about incompatible columns, but nothing can be found. This specific issue seems to be happening with only 1 table out of the 32 tables being loaded (at least that I'm aware of).

I didn’t have any of these issues until last week. Why does these issues happens only for some tables and also doesn't happen in debug mode? How can I further troubleshoot these problems?

1
azureazure-data-factoryazure-data-factory-2

1 Answers

1
votes

I've opened a support ticket with Microsoft and figured out the issue - changes to the master branch weren't being published correctly, they were "corrupted". The Data Factory was working with old metadata/code and never updating as it should, hence why it worked in debug mode (current/new metadata) but not with triggers (published metadata/code).

The issue was fixed by recreating the linked services connection with OData and replacing it in the data sets that were using it.