Databrick connection to ADLS Gen2 text files

Question

I am using databricks to access my ADLS Gen2 container.

dbutils.fs.mount(
  source = "wasbs://<container-name>@<storage-account-name>.blob.core.windows.net",
  mount_point = "/mnt/<mount-name>",
  extra_configs = {"<conf-key>":dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>")})

I am using the following code and changing out what I need to change out of course.

When I run script I get the following error

df = spark.read.text("/mnt/<mount-name>/...")

shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Container <container name> in account <storage account name>.blob.core.windows.net not found, and we can&#39;t create it using anoynomous credentials, and no credentials found for them in the configuration.

I registered databricks in my App registrations and added the name to my ADLS role as Storage Blob Data Contributor.

I'm not sure why my credentials are not allowing me to extract the text files that are in my ADLS account.

Any help is appreciated!

do you really have container with given name? Error explicitly says that container doesn't exist — Alex Ott

UtkarshPal-MT UtkarshPal-MT · Accepted Answer · 2021-08-18T12:27:25

I tried to reproduce the issue and it is working fine for me. I followed this tutorial.

Please find the databrick notebook snippet below:

You can consider the same code sample. Please find below:

configs = {"fs.azure.account.auth.type": "OAuth",
       "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
       "fs.azure.account.oauth2.client.id": "<appId>",
       "fs.azure.account.oauth2.client.secret": "<clientSecret>",
       "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/<tenant>/oauth2/token",
       "fs.azure.createRemoteFileSystemDuringInitialization": "true"}

dbutils.fs.mount(
source = "abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/folder1",
mount_point = "/mnt/flightdata",
extra_configs = configs)

In this code block, replace the appId, clientSecret, tenant, and storage-account-name placeholder values in this code block with the values that you collected while completing the prerequisites. Replace the container-name placeholder value with the name of the container.

Please consider the below points:

Make sure your container and storage account names are correct.
All the prerequisites mentioned in the tutorial page must be done. User has all required permissions.

Databrick connection to ADLS Gen2 text files

1 Answers