External Properties File in Azure Databricks

1
votes

We have a full fledge Spark Application that is taking a lot off parameter from properties file. Now we want move the application to Azure notebook format .Entire code is working fine and giving expected result with hard coded parameter. But is it possible to use external properties file in Azure Databricks Notebook also ??If we can, then where we need to place properties file??

2
azurepysparkdatabricksazure-databricks

2 Answers

2
votes

You may utilize the Databricks DBFS Filestore, Azure Databricks note books can access user's files from here.

To Upload the properties file you have, you can use 2 options

  1. Using wget,

    import sys
"wget -P /tmp/ http://<your-repo>/<path>/app1.properties"
dbutils.fs.cp("file:/tmp/app1.properties", "dbfs:/FileStore/configs/app1/")

  2. Using dbfs.fs.put, (may be an one-time activity to create this file)

    dbutils.fs.put("FileStore/configs/app1/app1.properties", "prop1=val1\nprop2=val2")

To import the properties file values,

properties = dict(line.strip().split('=') for line in open('/dbfs/FileStore/configs/app1/app1.properties'))

Hope this helps!!

0
votes

There's a possibility of providing/returning arguments with use of Databricks Jobs REST API, more information can be found e.g. here: https://docs.databricks.com/dev-tools/api/latest/examples.html#jobs-api-example