Read CSV From Azure Data lake storage Gen 1 in c# .net API

0
votes

We need to read a CSV File of around 2 GB which is stored in Azure Data lake storage Gen1.The purpose is like we have to render the data in Grid format (UI ) with high performance when user request. We are using .Net Core 2.1 (c#) for doing API for the same .

        var creds = new ClientCredential(applicationId, clientSecret);
        var clientCreds = ApplicationTokenProvider.LoginSilentAsync(tenantId, creds).GetAwaiter().GetResult();

        // Create ADLS client object
        AdlsClient client = AdlsClient.CreateClient(adlsAccountFQDN, clientCreds);

        string fileName = "/cchbc/sources/MVP/Data.csv";

        using (var readStream = new StreamReader(client.GetReadStream(fileName)))
        {

            while ((line = readStream.ReadLine()) != null)
            {
                content = content + line;
            }
        }

I have tried the above code but failed with an error GETFILESTATUS failed with HttpStatus:Forbidden RemoteException: AccessControlException GETFILESTATUS failed with error 0x83090aa2 (Forbidden. ACL verification failed. Either the resource does not exist or the user is not authorized to perform the requested operation.)

Any suggestion will be very beneficial .Thanks in advance

1
c#.netazureapi

1 Answers

0
votes

If you want to use a service principal to access files storing in Azure data lake gen 1, we need to configure ACL for the service principal. The ACL has three permissions Read(read the contents of a file) Write(write or append to a file) and Execute(traverse the child items of a folder).

for example I access file /test/test.csv

  1. Configure ACL as below
Opreation    Object        /        test/      test.csv
Read         tets.csv      --X      --X        R--

enter image description here enter image description here

  1. Code
string appId = "service principal appId";
            string appSecret = "service principal appSecret";
            string domain = "service principal domain";

            var serviceSettings = ActiveDirectoryServiceSettings.Azure;
            serviceSettings.TokenAudience = new Uri(@"https://datalake.azure.net/");
            
            var creds = await ApplicationTokenProvider.LoginSilentAsync(domain, appId, appSecret, serviceSettings);

            string accountName = "testadls02";
            AdlsClient client = AdlsClient.CreateClient($"{accountName}.azuredatalakestore.net", creds);
            string fileName = "/test/test.csv";
            string line = null;
            using (var readStream = new StreamReader(client.GetReadStream(fileName)))
            {
                while ((line =  await readStream.ReadLineAsync()) != null) {

                    Console.WriteLine(line);
                }
               
            }

enter image description here

For more details, please refer to here