Spark SQL unable to read HIVE table with org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe serde

0
votes

Hive Table Properties:

| ROW FORMAT SERDE                                   |
|   'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe'  |
| WITH SERDEPROPERTIES (                             |
|   'field.delim'='<~^~>')                           |
| STORED AS INPUTFORMAT                              |
|   'org.apache.hadoop.mapred.TextInputFormat'       |
| OUTPUTFORMAT                                       |
|   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' |
| | TBLPROPERTIES (                                    |
|   'TRANSLATED_TO_EXTERNAL'='TRUE')

/usr/hdp/3.1.5.0-152/spark2/bin/spark-shell --jars /usr/hdp/3.1.5.0-152/hive/lib/hive-contrib-3.1.0.3.1.5.0-152.jar

spark.read.table("db.table").show

20/06/08 03:04:35 ERROR log: error in initSerDe: java.lang.ClassNotFoundException Class org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe not found
java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2501)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:84)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:302)
        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:284)
        at org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:676)
        at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:659)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:371)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:368)
        at scala.Option.map(Option.scala:146)
2
apache-sparkhive
can add the code that you are using? - Mahesh Gupta
Thanks Mahesh for the response. there is no code as such. I have a hive table with "<~^~>" as delimiter created with the above given table properties. I am trying to read it using the command spark.read.table("db.table").show in spark shell. - pss4659

2 Answers

0
votes

It seems that Jar file does not exists on the path .. please pass the jars parameter as shown below. Also check jar on the path below spark-shell --jars /opt/cloudera/parcels/CDH/jars/hive-contrib-3.1.0.3.1.5.0-152.jar

if you are running through hive then set hive.execution.engine=spark; run your select statement ....it will work fine.

Thanks & Regards, Kamleshkumar Gujarathi

0
votes

Also try adding the following setting.

hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat