Azure Data Lake Service Principal write w/ Data Factory

0
votes

I have created a Service Principal, and set up the necessary linked services to utilise the credentials and secret key etc in ADF, here is a run down of how this is done:

https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-authenticate-using-active-directory

When i execute my pipeline, and the files are written to the ADL, i can see the top level folder (i am logged in the creator of the ADL service, and am also a contributor on the Resource Group), but i am absolutely unable to drill down any further.

The error i receive basically boils down an ACL error.

Interestingly, i also not at the Execution Location is listed as: East US 2 when using the service principal.

When i manually authenticate the ADL connection in Data Factory (with my own credentials), every works absolutely fine, and the 'execution location' is now listed, correctly, as North Europe.

Has anyone ever seen this?

1
azureazure-data-factoryazure-data-lake

1 Answers

1
votes

Helpful Reading: https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-access-control

The problem that you are running into is like an ACL issue as you mentioned. By just having contributor access, you only have access and permission on the Management Plane and not the Data Plane of the account.

Here is the mental model for thinking about ACLs

  1. If you need to be able to read a file, you need r-x access on that file, and --x permissions on the parent folder all the way up to root.

  2. If you create a new folder, and you create an Default ACL entry for yourself, it will apply to all new files and folders created below it.

To address your issue, please ask a Super User (someone from the Owners group) to give you this access.

Alternatively if you are an owner, you will have RWX access to any files/folder indepedent of any ACLs.

This should solve your problem.