Exception while running simple spark task in apache zeppelin

1
votes

I'm getting InvocationTargetException while trying tu run simple spark command in Apache Zeppelin. Below you can see two task have tried to run. Both give me the same exception.

enter image description here Stacktrace from Zeppelin:

DEBUG [2017-02-22 10:14:34,351] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - ERROR [2017-02-22 10:14:34,297] ({pool-2-thread-5} Utils.java[invokeMethod]:40) -
DEBUG [2017-02-22 10:14:34,353] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - java.lang.reflect.InvocationTargetException
DEBUG [2017-02-22 10:14:34,354] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
DEBUG [2017-02-22 10:14:34,356] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
DEBUG [2017-02-22 10:14:34,357] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
DEBUG [2017-02-22 10:14:34,358] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.lang.reflect.Method.invoke(Method.java:498)
DEBUG [2017-02-22 10:14:34,361] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
DEBUG [2017-02-22 10:14:34,361] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
DEBUG [2017-02-22 10:14:34,365] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.createSparkSession(SparkInterpreter.java:350)
DEBUG [2017-02-22 10:14:34,368] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.getSparkSession(SparkInterpreter.java:222)
DEBUG [2017-02-22 10:14:34,368] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:815)
DEBUG [2017-02-22 10:14:34,371] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
DEBUG [2017-02-22 10:14:34,371] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:482)
DEBUG [2017-02-22 10:14:34,375] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
DEBUG [2017-02-22 10:14:34,375] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
DEBUG [2017-02-22 10:14:34,377] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
DEBUG [2017-02-22 10:14:34,378] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
DEBUG [2017-02-22 10:14:34,380] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
DEBUG [2017-02-22 10:14:34,381] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
DEBUG [2017-02-22 10:14:34,381] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
DEBUG [2017-02-22 10:14:34,385] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
DEBUG [2017-02-22 10:14:34,386] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.lang.Thread.run(Thread.java:745)
DEBUG [2017-02-22 10:14:34,387] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - Caused by: java.lang.NullPointerException
DEBUG [2017-02-22 10:14:34,387] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
DEBUG [2017-02-22 10:14:34,389] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
DEBUG [2017-02-22 10:14:34,389] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.hadoop.util.Shell.run(Shell.java:455)
DEBUG [2017-02-22 10:14:34,389] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
DEBUG [2017-02-22 10:14:34,391] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:873)
DEBUG [2017-02-22 10:14:34,392] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853)
DEBUG [2017-02-22 10:14:34,394] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.util.Utils$.fetchFile(Utils.scala:475)
DEBUG [2017-02-22 10:14:34,394] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.SparkContext.addFile(SparkContext.scala:1477)
DEBUG [2017-02-22 10:14:34,395] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.SparkContext.addFile(SparkContext.scala:1425)
DEBUG [2017-02-22 10:14:34,397] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:470)
DEBUG [2017-02-22 10:14:34,398] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:470)
DEBUG [2017-02-22 10:14:34,400] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at scala.collection.immutable.List.foreach(List.scala:381)
DEBUG [2017-02-22 10:14:34,401] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.SparkContext.<init>(SparkContext.scala:470)
DEBUG [2017-02-22 10:14:34,401] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
DEBUG [2017-02-22 10:14:34,403] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
 INFO [2017-02-22 10:14:34,403] ({pool-2-thread-3} SchedulerFactory.java[jobFinished]:137) - Job paragraph_1487752190357_-397170656 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session969841713
DEBUG [2017-02-22 10:14:34,404] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
DEBUG [2017-02-22 10:14:34,407] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at scala.Option.getOrElse(Option.scala:121)
DEBUG [2017-02-22 10:14:34,407] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
DEBUG [2017-02-22 10:14:34,409] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  ... 20 more
DEBUG [2017-02-22 10:14:34,409] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  INFO [2017-02-22 10:14:34,319] ({pool-2-thread-5} SparkInterpreter.java[createSparkSession]:351) - Created Spark session with Hive support
DEBUG [2017-02-22 10:14:34,410] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - ERROR [2017-02-22 10:14:34,319] ({pool-2-thread-5} Job.java[run]:181) - Job failed
DEBUG [2017-02-22 10:14:34,412] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - java.lang.NullPointerException
DEBUG [2017-02-22 10:14:34,412] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:38)
DEBUG [2017-02-22 10:14:34,412] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.Utils.invokeMethod(Utils.java:33)
DEBUG [2017-02-22 10:14:34,414] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext_2(SparkInterpreter.java:380)
DEBUG [2017-02-22 10:14:34,415] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:369)
DEBUG [2017-02-22 10:14:34,415] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:144)
DEBUG [2017-02-22 10:14:34,417] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:817)
DEBUG [2017-02-22 10:14:34,418] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
DEBUG [2017-02-22 10:14:34,420] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:482)
DEBUG [2017-02-22 10:14:34,423] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
DEBUG [2017-02-22 10:14:34,423] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
DEBUG [2017-02-22 10:14:34,425] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
DEBUG [2017-02-22 10:14:34,425] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
DEBUG [2017-02-22 10:14:34,427] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
DEBUG [2017-02-22 10:14:34,428] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
DEBUG [2017-02-22 10:14:34,428] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
DEBUG [2017-02-22 10:14:34,430] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
DEBUG [2017-02-22 10:14:34,431] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  at java.lang.Thread.run(Thread.java:745)
DEBUG [2017-02-22 10:14:34,431] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) -  INFO [2017-02-22 10:14:34,320] ({pool-2-thread-5} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1487754873203 finished by scheduler org.apache.zeppelin.spark.SparkInterpreter1927937575
DEBUG [2017-02-22 10:14:34,433] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - DEBUG [2017-02-22 10:14:34,321] ({pool-1-thread-3} Interpreter.java[getProperty]:165) - key: zeppelin.spark.concurrentSQL, value: false
DEBUG [2017-02-22 10:14:34,433] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - DEBUG [2017-02-22 10:14:34,321] ({pool-1-thread-3} Interpreter.java[getProperty]:165) - key: zeppelin.spark.concurrentSQL, value: false

I'm using Spark "spark-2.1.0-bin-hadoop2.7", hadoop "hadoop-2.7.3" and zeppelin "zeppelin-0.7.0-bin-all".

1
apache-sparkapache-zeppelin

1 Answers

0
votes

You have to build zeppelin for hadoop 2.7

Here is doc for built from source

# make sure that you are using scala 2.11.x, maven 3.1.x+ and JDK 1.7+
# also, you need nodejs 6.9.1+ (will be removed from build requirements in 0.7.1)

$ ./dev/change_scala_version.sh 2.11
$ mvn clean package -DskipTests -Pspark-2.1 -Phadoop-2.7 -Pyarn -Ppyspark -Psparkr -Pr -Pscala-2.11