Get the latest data from ADLS Gen 2 blob storage to table mounted in Azure DataBricks

0
votes

I have created an unmanaged table in Azure DataBricks using mount path as below:

CREATE TABLE <Table-Name> using org.apache.spark.sql.parquet OPTIONS (path "/mnt/<folder>/<subfolder>/")

Source of mount path is parquet files stored in ADLS Gen2. I see if the underlying data is changed in ADLS Gen 2 blob storage path, it is not reflected in the unmanaged table created in ADB. This ADB table still holds the data which was available in blob storage at time of creation of table Is there any way to get the latest data from blob storage into table in ADB?

1
azure-blob-storageazure-databricksazure-data-lake-gen2

1 Answers

0
votes

There are many who suggest to use ,

   REFRESH TABLE <table-name>

https://docs.databricks.com/data/tables.html#update-a-table

But it never worked for me .

The below think it worked .

yourdataframe.write.mode("overwrite").saveAsTable("test_table")