Not able to make Spark as Hive execution engine

1
votes

I am using Hive 3.1.2 and Spark 2.4.5(Hadoop 3 on Ubuntu 18.04) Getting below error while running queries

0: jdbc:hive2://localhost:10000> select count(1) from retail_db.orders; Error: Error while processing statement: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 4aa97085-8a23-4ad8-a8c6-b701d41d7110 (state=42000,code=30041)

Log: 2020-06-19 12:18:28,704 INFO [HiveServer2-Background-Pool: Thread-72] ql.Driver: Concurrency mode is disabled, not creating a lock manager 2020-06-19 12:18:28,731 ERROR [HiveServer2-Background-Pool: Thread-72] operation.Operation: Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 4aa97085-8a23-4ad8-a8c6-b701d41d7110 at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226) at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:329) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 4aa97085-8a23-4ad8-a8c6-b701d41d7110 at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.getHiveException(SparkSessionImpl.java:221) at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:92) at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:115) at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:136) at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:115) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224) ... 11 more Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.generateSparkConf(HiveSparkClientFactory.java:263) at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.(RemoteHiveSparkClient.java:98) at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:76) at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:87)

1
apache-sparkubuntuhive

1 Answers

2
votes

According to this error:
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf

Hive cannot find Spark classes.
If you're using Cloudera, you can add them with this command:

cd /opt/cloudera/parcels/CDH/lib;
cp  spark/jars/*scala-lib* hive/lib/;
cp  spark/jars/*spark-core* hive/lib/;
cp  spark/jars/*spark-netw*  hive/lib/;
cp  spark/jars/chill-java*.jar  hive/lib/;
cp  spark/jars/jackson-module-*  hive/lib/;
cp  spark/jars/jersey-server-*.jar  hive/lib/;
cp  spark/jars/json4s-ast_*.jar  hive/lib/;
cp  spark/jars/kryo-shaded-*.jar  hive/lib/;
cp  spark/jars/minlog-*.jar  hive/lib/;
cp  spark/jars/scala-xml_*.jar  hive/lib/;
cp  spark/jars/spark-launcher_*.jar  hive/lib/;
cp  spark/jars/spark-network-shuffle_*.jar  hive/lib/;
cp  spark/jars/spark-launcher_*.jar  hive/lib/;
cp  spark/jars/spark-unsafe_*.jar  hive/lib/;
cp  spark/jars/xbean-asm7-shaded-*.jar  hive/lib/;

You can find more info here:
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
in Configuring Hive section.