Azure Batch support for Data Lake Store Linked Service

1
votes

I am using a data factory pipeline with a custom activity (configured to run on Azure Batch) that has a data lake store input dataset and output dataset. The data lake store linked service is using service to service auth (service principal) and is working fine while being used in a Copy activity through Copy Wizard. But when used with a custom activity that tries to check if a file is present in the data lake, the activity fails with an error "Authorization is required". Upon using a Azure Blob Store as the input and output datasets, the same custom activity works fine.

Seems like an issue with Azure Batch (Compute node) not able to authorize Data Lake Store. Please help if you have solved the above mentioned problem.

1
azure-data-factoryazure-data-lakeazure-batch
Welcome to Stack Overflow! To give you a great answer, it might help us if you have a glance at How to Ask if you haven't already. It might be also useful if you could provide a minimal reproducible example. – Mat

1 Answers

2
votes

I had this exact same issue about 3 weeks ago. I feel your pain!

This is a Microsoft bug!

After much trial and error and redeployments I raised a support ticket with Microsoft who confirmed that service principal authentication for data lake store currently only works with copy activities. Not with custom activities.

This is the official response I got on Monday 10th April.

The issue happen because of a bug that custom activity’s connector schema doesn’t match the latest published connector schema. Actually, we notice the issue on custom activity and have plan to fix & deploy to prod in next 2 weeks.

Be aware that if you change your linked service back to use a session token etc you'll also need to redeploy your pipelines that contain the custom activities. Otherwise you'll get another error something like the following...

Access is forbidden, please check credentials and try again. Code: 'AuthenticationFailed' Message: 'Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

Hope this helps.