Cannot list Azure Storage Gen 2 files with Databricks

2
votes

I wonder if my databricks code is addressing the correct location and if "contributor" right is enough for accessing storage.

  1. I have Azure Storage Gen 2 with container named staging. (Url in Azure portal is https://datalaketest123.blob.core.windows.net/staging)
  2. I have mounted Azure Storage Gen 2 with Azure Databricks.
  3. I have configured passthrough and assuming that I get access to storage with my AD users. (contributor rights)
  4. i have variable: source = 'abfss://' + in_fileSystemName + '@' + storageAccountName + '.dfs.core.windows.net/'
  5. I tried now to list file system with command: dbutils.fs.ls(source)

I get error:

ET https://datalaketest123.dfs.core.windows.net/staging? 
resource=filesystem&maxResults=500&timeout=90&recursive=false
---------------------------------------------------------------------------
ExecutionError                            Traceback (most recent call last)
<command-1012822525241408> in <module>
 27 # COMMAND ----------
 28 source = 'abfss://' + in_fileSystemName + '@' + storageAccountName + '.dfs.core.windows.net/'
---> 29 dbutils.fs.ls(source)
 30 
 31 # COMMAND ----------

/local_disk0/tmp/1235891082005-0/dbutils.py in f_with_exception_handling(*args, **kwargs)
    312                     exc.__context__ = None
    313                     exc.__cause__ = None
--> 314                     raise exc
    315             return f_with_exception_handling
    316 

ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls.
: GET https://datalaketest123.dfs.core.windows.net/staging? 
resource=filesystem&maxResults=500&timeout=90&recursive=false
StatusCode=403
StatusDescription=This request is not authorized to perform this operation using this permission.
ErrorCode=AuthorizationPermissionMismatch
2
databricksazure-databricks

2 Answers

1
votes

Per official Databricks docs only Contributor is not enough - it should be Storage Blob Data XXX (where XXX is Owner, Contributor, Reader, .... - see docs)

1
votes

When performing the steps in the Assign the application to a role, make sure that your user account has the Storage Blob Data Contributor role assigned to it.

Repro: I have provided owner permission to the service principal and tried to run the “dbutils.fs.ls("mnt/azure/")”, returned same error message as above.

enter image description here

Solution: Now assigned the Storage Blob Data Contributor role to the service principal.

enter image description here

Finally, able to get the output without any error message after assigning the Storage Blob Data Contributor role to the service principal.

enter image description here

Reference: “Tutorial: Azure Data Lake Storage Gen2, Azure Databricks & Spark”.