Reading Partitioned HIVE table in SPARK SQL

0
votes

I am trying to read the HIVE table has 2 years of data and it is partitioned on day, Reading it through SPARK-SQL takes more than 10 mins even after filter applied for 6 months data.

Any idea how to improve, i have tried with DISTRIBUTE BY clause in Hive QL

1
scalaapache-sparkhadoopapache-spark-sql
Check your query plan - vikrant rana
Check if partition filters are getting applied. - kaysush

1 Answers

0
votes

You can enable partitioning pruning by setting property : 

spark.sql.hive.metastorePartitionPruning=true

This will filter the partitions