Databricks file copy with dbtuils only if file doesn't exist

3
votes

I'm using the following databricks utilites (dbutils) command to copy files from one location to another as shown below:

dbutils.fs.cp('adl://dblake.azuredatalakestore.net/jfolder2/thisfile.csv','adl://cadblake.azuredatalakestore.net/landing/')

However, I want the file to be copied over only if no such file with the same name 'thisfile.csv' exists.

Can someone let me know whether is that possible?

If not, is there any other workaround?

1
databricksazure-databricks

1 Answers

7
votes

dbutils.fs.ls() lists files in the given path.

ls

So you can check if thisfile.csv exists before copying the file:

if "thisfile.csv" not in [file.name for file in dbutils.fs.ls("adl://cadblake.azuredatalakestore.net/landing/")]:
    dbutils.fs.cp("adl://dblake.azuredatalakestore.net/jfolder2/thisfile.csv", "adl://cadblake.azuredatalakestore.net/landing/")