How to create a empty folder in Azure Blob from Azure databricks

0
votes

I have scenario where I want to list all the folders inside a directory in Azure Blob. If no folders present create a new folder with certain name.

I am trying to list the folders using dbutils.fs.ls(path).

But the problem with the above command is it fails if the path doesn't exist, which is a valid scenario for me.

If my program runs for the first time the path will not exist and dbutils.fs.ls command will fail. Is there any way I can handle this scenario dynamically from Databricks.

It will also work for me if I can create an empty folder in Azure Blob from Databricks before executing my job.

I have tried running below command from databricks notebook

   %sh mkdir -p /mnt/<mountName>/path/folderName

Here the command runs successfully and even though my container in Azure Blob is mounted it doesn't create the folder. Sorry for such an elongated post. Any help is much appreciated. Thanks in advance

1
azuredatabricksazure-blob-storageazure-databricks
FYI, blob storage(not ADLS Gen2) does not allow to create an empty folder, unless the folder contains at least one file. So you can try to create a folder as well as a file/blob within it. – Ivan Yang
Yes, that is the exact problem. I guess I would have simply handle dbutils.fs.ls command from failing when path doesn't exist using a try and except. There is no way to create the folders if the path or file doesn't exist – Saikat

1 Answers

3
votes

dbutils.fs.mkdirs("/mnt/<mountName>/path/folderName")

I found this was able to create a folder with a mounted blob storage