How to force refresh secret used to mount ADLS Gen2? Azure Databricks mounts using Azure KeyVault-backed scope -- SP secret update

1
votes

Issue:

  1. Mounted ADLS gen2 container using service principal secret as secret from Azure Key Vault-backed secret scope. All good, can access the data.

  2. Deleted secret from service principal in AAD, added new, updated Azure Key Vault secret (added the new version, disabled the old secret). All was still good, could access the data.

  3. Restarted cluster. Unable to access mount point, error: “AADToken: HTTP connection failed for getting token from AzureAD. Http response: 401 Unauthorized”

  4. Unmount/mount using the same config helped.

Is there a way to refresh the secret used for mount point that I could add to init scripts to avoid this issue? I would rather avoid unmounting/mounting all mount points in init scripts and was hoping that there is something like dbutils.fs.refreshMounts() that would help (refreshMounts didn't help with this particular issue).

I mounted ADLS Gen2 using service principal, oauth2.0, and azure key vault-backed secret scope, following this documentation: https://docs.microsoft.com/en-us/azure/databricks/data/data-sources/azure/azure-datalake-gen2#mount-azure-data-lake-gen2

Also - out of curiosity: does anybody know how long a token to mount to ADLS Gen2 lives? As long as the cluster did not restart, I was able to access my mnt even though the secret was deleted and updated (i.e., secret was updated in AAD and Key Vault; no failures until restarting the cluster - which was more than 12 hours after the update).

1
azuredatabricksazure-databricks

1 Answers

2
votes

This is a known limitation. Whenever you create a mount point using credentials coming from an Azure Key Vault backed secret scope, the credentials will be stored in the mount point and will never be refreshed again.

This is a one-time read activity on mount point creation time. So each time you rotate credentials in Azure Key Vault you need to re-create the mount points to refresh the credentials there.

I would suggest you to provide feedback on the same:

Azure Databricks - Feedback

All of the feedback you share in these forums will be monitored and reviewed by the Microsoft engineering teams responsible for building Azure.