Graceful custom activity timeout in data factory

Question

I'm looking to impose a timeout on custom activities in data factory via the policy.timeout property in the activity json.

However I haven't seen any documentation to suggest how the timeout operates upon Azure batch? I assume that the batch task is forcibly terminated somehow.

But is the task -> custom activity informed so it can tidy up?

The reason I ask is that I could be mid-copying to data lake store and I neither want to let it run indefinitely nor stop it without some sort of clean up (I can't see a way of doing transactions as such using the data lake store SDK).

---

I'm considering putting the timeout within the custom activity, but it would be a shame to have timeouts defined at 2 different levels (I'd probably still want the overall timeout).

Answer 1

I feel your pain.

ADF does simply terminate the activity if its own time out is reached regardless of what state the invoked service is in.

I have the same issue with U-SQL processing calls. It takes a lot of proactive monitoring via PowerShell to ensure data lake or batch jobs have enough compute to complete jobs with natually increasing data volumes before the ADF timeout kill occurs.

I'm not aware of any graceful way for ADF to handle this because it would differ for each activity type.

Time to create another feedback article for Microsoft!