SparkR - ObjectStore: Failed to get database global_temp, returning NoSuchObjectException

1
votes

When trying to connect to Spark cluster using SparkR in RStudio:

if (nchar(Sys.getenv("SPARK_HOME")) < 1) {
  Sys.setenv(SPARK_HOME = "/usr/lib/spark/spark-2.1.1-bin-hadoop2.6")
  .libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))
}

library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib")))

# Starting a sparkR session
sparkR.session(master = "spark://myIpAddress.eu-west-1.compute.internal:7077")

I am getting the following error message:

Spark package found in SPARK_HOME: /usr/lib/spark/spark-2.1.1-bin-hadoop2.6
Launching java with spark-submit command /usr/lib/spark/spark-2.1.1-bin-hadoop2.6/bin/spark-submit   sparkr-shell /tmp/RtmpMWFrt6/backend_port71e6731ea922 
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/05/24 16:17:32 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/24 16:17:37 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Java ref type org.apache.spark.sql.SparkSession id 1

In Spark master I see SparkR application is running, but no sc variable is available. It feels that this error might be related to metastore, but not sure. Does anyone know what prevents my spark session from starting correctly?

Thanks, Michal

1
rapache-sparkhivesparkr
hey micha Im getting the same error. did you solve that? – ℕʘʘḆḽḘ
Unfortunately, no progress in here. Besides - I decided to focus on sparklyr - RStudio plus sparklyr seems to be a better marriage (same community). – mrjoseph

1 Answers

0
votes

1- Removed the linked file using sudo rm -R /etc/spar/conf/hive.xml 2- Again linked the file using sudo ln -s /etc/hive/conf/hive-site.xml /etc/spark/conf/hive-site.xml