Unable to access a mounted Azure Data Lake storage using Azure Databricks

1
votes

I'm playing around with the Azure Databricks. Using the documents specified on the Microsoft Learn website, I managed to mount a BLOB storage (ADLS Gen2) to my Databricks.

However, when I try to list the contents of the mounted storage, I get the following error:

ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls.
: GET https://xxxxxxxxxxxxx.dfs.core.windows.net/xxxxxxx?resource=filesystem&maxResults=5000&timeout=90&recursive=false
StatusCode=403
StatusDescription=This request is not authorized to perform this operation using this permission.
ErrorCode=AuthorizationPermissionMismatch

I've checked the permissions, and my ServicePrincipal has been assigned the role 'STORAGE BLOB DATA CONTRIBUTOR' which allows R/W access on my storage container.

Anyone has an idea which part I'm missing to get this working? Help will be much appreciated.

1
azureazure-data-lakeazure-databricksaccess-rights

1 Answers

4
votes

Just found a solution to my own problem.

Root cause was the fact there were no permissions set on the Azure Storage container itself (although in the Azure Portal, everything looked OK).

What I did to resolve this: Download and install Azure Storage Explorer on your local machine

  1. Download and install Azure Storage Explorer on your local machine, using the following url: https://azure.microsoft.com/en-us/features/storage-explorer/
  2. Open the Azure Portal, and launch the Azure Cloud Shell.
  3. Use the following command to retrieve the Object ID of your Application ID (Service Principal). Unfortunately, at the time of this writing, you cannot retrieve this from the portal directly. Command: az ad sp show --id Application ID
  4. Copy the displayed Object ID from the result set.
  5. Right click on your storage container within Azure Storage Explorer, choose 'Manage Access...'
  6. Paste your Object ID, and assign the appropriate rights.
  7. Save and retry your code from the Databricks notebook.

That did the trick for me.