Unable to run hive by changing hive-site.xml to connect with spark-HiveContext

0
votes

Below is my hive/conf/hive-site.xml:

<configuration>
   <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://127.0.0.1/metastore?createDatabaseIfNotExist=true</value>
      <description>metadata is stored in a MySQL server</description>
   </property>
   <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
      <description>MySQL JDBC driver class</description>
   </property>
   <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>hiveuser</value>
      <description>user name for connecting to mysql server</description>
   </property>
   <property>
      <name>javax.jdo.option.ConnectionPassword</name>\
      <value>hivepassword</value>
      <description>password for connecting to mysql server</description>
   </property>
</configuration>

I want to access Hive existing database and tables using spark-HiveContext. So added below lines to hive/conf/hive-site.xml:

<configuration>
   <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://127.0.0.1/metastore?createDatabaseIfNotExist=true</value>
      <description>metadata is stored in a MySQL server</description>
   </property>
   <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
      <description>MySQL JDBC driver class</description>
   </property>
   <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>hiveuser</value>
      <description>user name for connecting to mysql server</description>
   </property>
   <property>
      <name>javax.jdo.option.ConnectionPassword</name>\
      <value>hivepassword</value>
      <description>password for connecting to mysql server</description>
   </property>
   <property>
      <name>hive.metastore.uris</name>
      <value>thrift://127.0.0.1:9083</value>
   </property>
</configuration>

After editing the hive-site.xml as shown above the hive shell is not working. Please help me in update the hive-site.xml correct way and help me to access hive tables on spark-shell using HiveContext as shown below:

val hc = new org.apache.spark.sql.hive.HiveContext(sc);
hc.setConf("hive.metastore.uris","thrift://127.0.0.1:9083");
val a = hc.sql("show databases");
a.show //should display all my hive databases.

Kindly help me on this issue.

1
hadoopapache-sparkhivehivecontext
Is HiveServer2 running? Mysql running? – OneCricketeer
Mysql is running properly. Could you pls help how to check HiveServer2? – Kumar
so r you getting any error? – Sahil Desai
Nevermind. Spark doesn't require a running Hive Server, only a metastore. Anyway, hc.setConf("hive.metastore.uris" is not necessary. You also shouldn't even be using a HiveContext with Spark2, only a SparkSession – OneCricketeer
@cricket_007 yes we can use without hc.setConf but the thing is when I tried displaying databases spark is showing only default database and not from hive server also no other hive databases are shown on spark. So i tired editing hive-site.xml but facing error as shown above..... I want to access hive tables on spark-shell, is there any other way or i need to configure anything else? – Kumar

1 Answers

0
votes

@Chaithu You need to start your hive metastore using hive --service metastore and then create the sparksession with hivesupport enabled in this way

 val spark= SparkSession
  .builder()
  .master("local")
  .appName("HiveExample").config("hive.metastore.uris","thrift://hadoop-master:9083")
  .enableHiveSupport()
  .getOrCreate()