unable load data into hive using pyspark

Question

unable to write the data into hive using pyspark through jupyter notebook .

giving me below error

Py4JJavaError: An error occurred while calling o99.saveAsTable. : org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;

Note these steps already tried:

copied the hdfs-site.xml , core-site.xml to /conf of hive
removed metasotore_db and created again using below cmd $HIVE_HOME/bin/schematool –initschema –dbtype derby

from pyspark import SparkContext, SparkConf from pyspark.sql import SparkSession, HiveContext SparkContext.setSystemProperty("hive.metastore.uris", "thrift://localhost:10000") sparkSession = SparkSession.builder.appName("example-pyspark-read-and-write").getOrCreate() data = [('First', 1), ('Second', 2), ('Third', 3), ('Fourth', 4), ('Fifth', 5)] df = sparkSession.createDataFrame(data) df.show() # Write into Hive df.write.saveAsTable('example' — vikram bhati
this piece of code i tried and giving me error : org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate — vikram bhati

RacoonOnMoon RacoonOnMoon · Accepted Answer · 2020-03-31T09:11:19

did you use spark-submit for running your script? Also you should add -> ".enableHiveSupport()" like that:

spark = SparkSession.builder \
    .appName("yourapp") \
    .enableHiveSupport() \
    .getOrCreate()

unable load data into hive using pyspark

1 Answers