Azure Application is not getting access to newly created files in data lake store

Question

I have written console application to monitor/analyze the files in Azure data lake store. I have created an application in Azure active directory to access the azure resources. I have followed all the steps given here to give access to application on azure data lake store. I have provided access to the parent and all the childern folders/files of data lake store.

Now, I am able to access the files through my code. I am trying to get the modification time and expiration time of a file produced by USQL job in data lake store by using DataLakeStoreFileSystemManagement Client in the code. I am using Microsoft provided .NET API for data lake analytics & data lake store.

I am getting all that information for the files for which i have provided access to. But when the usql job adds a new folder/file in the azure data lake store then i am not getting the modification time and expiration in my code. Instead i am getting exception of Forbidden Error 403.

the Usql jobs creates plenty of folders everyday and i just can't go there and provide access manually to all newly created files and folder. It should inherit the access role for newly created folders/files.

what should I do ? or Is that a bug in Azure data lake store? Please Help.

Michael Rys Michael Rys · Accepted Answer · 2018-01-30T09:23:09

You are not allowed to call REST end points from within user code in U-SQL (reasons are explained here). The DataLakeStoreFileSystem Management client is attempting to (recursively) call into ADL through REST end points and is being blocked by the container boundary protection. So the 403 (Forbidden) is by design.

We are working on adding file properties to our U-SQL APIs as meta properties in one of the upcoming refreshes. Would that help?

Azure Application is not getting access to newly created files in data lake store

2 Answers