Convert spark dataframe to Delta table on azure databricks - warning

Question

I am saving my spark dataframe on azure databricks and create delta lake table.

It works fine, however I am getting this warning message while execution.

Question- Why I am still getting this message, even with my table is delta table. What is wrong with my approach, any inputs is greatly appreciated.

Warning Message

This query contains a highly selective filter. To improve the performance of queries, convert the table to Delta and run the OPTIMIZE ZORDER BY command on the table

Code

dfMerged.write\
              .partitionBy("Date")\
              .mode("append")\
              .format("delta")\
              .option("overwriteSchema", "true")\
              .save("/mnt/path..")

spark.sql("CREATE TABLE DeltaUDTable USING DELTA LOCATION '/mnt/path..'")

Some more details

I've mounted azure storage gen 2 to above mount location.
databricks runtime - 6.4 (includes Apache Spark 2.4.5, Scala 2.11)

thanks @thebluephantom for your inputs, that's what I thought so. I will go ahead and close this post. — Idleguys
@thebluephantom Would you like to post that as an answer so you can mark this question as answered? — CHEEKATLAPRADEEP-MSFT

thebluephantom thebluephantom · Accepted Answer · 2020-11-23T11:54:22

0

votes

The warning message is clearly misleading as you already have a Delta option. Ignore it.

Convert spark dataframe to Delta table on azure databricks - warning

1 Answers