How to delete all files from folder with Databricks dbutils

4
votes

Can someone let me know how to use the databricks dbutils to delete all files from a folder. I have tried the following but unfortunately, Databricks doesn't support wildcards.

dbutils.fs.rm('adl://azurelake.azuredatalakestore.net/landing/stageone/*')

Thanks

2
databricksazure-databricks
Please provide more information if referring to deleting all files or folder as well - Paul Velthuis

2 Answers

3
votes

According to the documentation, the rm function receives 2 parameters :

rm(dir: String, recurse: boolean = false): boolean -> Removes a file or directory

Where the second parameter is a boolean flag to set the recursitivity, so you just need to set it to true:

dbutils.fs.rm('adl://azurelake.azuredatalakestore.net/landing/stageone/',True)
3
votes

Something like this should work:

val PATH = "adl://azurelake.azuredatalakestore.net/landing/stageone/"
dbutils.fs.ls(PATH)
            .map(_.name)
            .foreach((file: String) => dbutils.fs.rm(PATH + file, true))