Create Azure databricks notebook from storage account

Question

We have python script stored in Azure storage account in blob. We want to deploy / create this python script (as notebook) in azure databricks cluster so later we can run Azure data factory pipeline and pipeline can execute notebook created/deployed in databricks.

We want to create / deploy this script only one time as and when its available in blob.

I have tried to search over the web but couldn't find proper solution for this.

Is it possible to deploy/create notebook from storage account? if yes, how?

Thank you.

Answer 1

You can import notebook into Databricks using the URL, but I expect that you won't make that notebook public.

Another solution would be to use a combination of azcopy tool with Databricks CLI (workspace sub-command). Something like this:

azcopy cp  "https://[account].blob.core.windows.net/[container]/[path/to/script.py" .
databricks workspace import -l PYTHON script.py '<location_on_databricks>'

You can also do it completely in notebook, combining the dbutils.fs.cp command with Databricks's Workspace REST API, but that's could be more complicated as you need to get personal access token, base64 the notebook, etc.

Answer 2

We can use databricks API 2.0 to import python script in databricks cluster.

Here is the API definition: https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/workspace#--import