scala code to read parquet file by passing dynamic values using widgets

Question

I have folders structure like process/YYYY/MM/DD i need to write a Scala code to read those files till process/YYYY and i dynamically pass the month and date using widgets.

I pass the mm and dd dynamically using widgets

code val ReadDf = spark.read.format("parquet").option("header","true").load(""mnt/pnt/process/YYYY")

Could you elaborate a bit on your question? What is the problem, what goes wrong? — Rayan Ral
example Path is like : /mnt/pnt/process/yyyy=2020/mm=06/dd=01 where mm is month dd is date. At present i am reading till /mnt/pnt/process/yyyy=2020 and want to pass the month(mm) and date(dd) value dynamically using databricks widgets. hope this helps for understanding my request — Pradyot Mohanty

Carlos David Peña Carlos David Peña · Accepted Answer · 2020-06-12T02:13:08

You can use the following code to get the month from a widget and then create a path to load from:

dbutils.widgets.text("Month", "1")

val widget_month = dbutils.widgets.get("Month").toInt
val path_month = "%02d".format(widget_month)
val pathToReadFrom = s"/mnt/pnt/process/yyyy=2020/mm=${path_month}"

Databricks output shows:

widget_month: Int = 1
path_month: String = 01
pathToReadFrom: String = /mnt/pnt/process/yyyy=2020/mm=01

Now if you want to pass arguments to a notebook through the widgets, you can run it from another notebook using Notebook workflows. This is an example from that link:

dbutils.notebook.run("notebook-name", 60, {"argument": "data", "argument2": "data2", ...})

scala code to read parquet file by passing dynamic values using widgets

1 Answers