Reflection error while creating spark session through Livy

1
votes

I am creating a spark session through LivyClient. The code for creating the session is -

var client: LivyClient = null
  try {
    client = new LivyClientBuilder()
      .setURI(new URI("http://127.0.0.1:4141"))
      .build();
    val pi = client.submit(new FileStream()).get
  } finally {
    client.stop(true)
  }

While creating a session, I am getting the error -

Exception in thread "main" scala.reflect.internal.FatalError: object Predef does not have a member classOf enter image description here

I am using the following versions -

  1. Apache Livy - 0.5.0-incubating
  2. Apache Spark - 2.3.0
  3. Scala - 2.11 (2.11.8)

I have checked the scala-reflect version in all the three and they are same.

I am unable to understand what is causing the below error

1
scalaapache-sparklivy
Welcome to Stack Overflow. Images of errors are highly discouraged.Please edit your question and include the error as a text. – zero323
you should pass Livy Job to client.submit method – morsik
I'm getting the same error. Did you ever find a resolution to this? – user4601931

1 Answers

1
votes

What version of Java are you using? It appears that changes in Java 9 regarding dropping of deprecated features stops Spark with errors referring to reflection. The home page says that the Java must be at least version 8.