Hive not storing Warehouse in HDFS

Question

I have downloaded hive installation on my local system and copied hive-site.xml into Spark conf directory. I tried to create a managed table in Hive context using spark shell.

I have put following property in hive-site.xml (present in spark's conf directory):

<property>
   <name>hive.metastore.warehouse.dir</name>
   <value>/user/hive/warehouse</value>
</property>

Also I have put HADOOP_HOME in spark-env.sh:

export HADOOP_CONF_DIR=/opt/hadoop/conf

As per Hive documentation, the hive warehouse should get stored in HDFS, but the warehouse is getting stored in local drive (/user/hive/warehouse). Please help me out in understanding why Hive is not storing warehouse directory in HDFS.

Could you paste hte value of the property fs.defaultFS found in the file /opt/hadoop/conf/core-site.xml ? — user1314742

Sanskar Suman Sanskar Suman · Accepted Answer · 2019-09-06T14:01:51

Please define your Spark dependencies using 2.0.2

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.2"

You can then use hive.metastore.warehouse.dir or spark.sql.warehouse.dir to set the Spark warehouse and point to HDFS where the other Hive tables live.

Hive not storing Warehouse in HDFS

1 Answers