HIVE Metastore warnings in Spark

3
votes

While starting spark-shell I regularly receive such warning

WARN ObjectStore: Version information not found in 
metastore. hive.metastore.schema.verification is not enabled so 
recording the schema version 1.2.0
WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
java.lang.RuntimeException:

What do they mean and I how do I prevent it. This warning appears on 2.0.2 and 1.6.2 probably on other versions too.

3
apache-sparkhivemetastore
what is version of hive and spark? have you upgraded anything? i mean earlier it was find and now you are facing some issues? - Ram Ghadiyaram
The Spark is 2.0.2 but the same error appear on other version of spark too. - Yakov
Hi, I am facing the same problem, what was the solution? - Harelz

3 Answers

1
votes

By default hive.metastore.schema.verification is set to false. Another config is there to set as false to avoid this error. Set below configuration in your hive-site.xml. (This issue occurs due to the VERSION table in the metastore is empty)

<property>
         <name>hive.metastore.schema.verification</name>
         <value>false</value>
</property>

<property>
         <name>hive.metastore.schema.verification.record.version</name>
         <value>false</value>
</property>
0
votes

I think the version of your hive is too old, try to upgrade hive to 1.2.0 or newer. Another choice is to use spark 1.4 which is compatible with hive 0.13 .

0
votes

I had the same issue in Spark 2.4.7; I did not find a hive-site.xml. It may well be packaged. However, I did enable the requested verification by doing the following imports, and creating a new spark session; on the spark-shell.

  • scala> import org.apache.spark.sql.Row
  • scala> import org.apache.spark.sql.SparkSession
  • scala> val spark =SparkSession.builder.appName("Spark Hive Example").config("spark.sql.warehouse.dir", ".").enableHiveSupport().getOrCreate()