Mapreduce program with Scala

1
votes

I am trying to run a mapreduce program written in scala. I have included scala-library in the classpath. While running the program my program throws following error.

$hadoop jar ~/HadoopScala.jar com.learning.spark.WordCount /input/wordcountinput.csv /output -libjars ~/lib/org.scala-lang.scala-library_2.12.2.v20170412-161608-VFINAL-21d12e9.jar Exception in thread "main" java.lang.NoClassDefFoundError: scala/Predef$ at com.learning.spark.WordCount$.main(WordCount.scala:52) at com.learning.spark.WordCount.main(WordCount.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:234) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: java.lang.ClassNotFoundException: scala.Predef$ at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

scala.Predef$ is available in scala-library. What could be causing this error.

1
scalahadoopmapreduce
How did you generate the jar file? - OneCricketeer
I think libjars needs to be HDFS files, by the way. - OneCricketeer
Does that provide you with the option to make a "uber jar"? If so, you don't need to pass the libjars - OneCricketeer
Thanks, uber jar solution works. I am still not able to understand why libjars option does not work. - pawinder gupta
Because all map and reduce containers in the cluster need access to that scala library, which you've only specified for your local path on the current machine - OneCricketeer

1 Answers

0
votes

First off, you are not running a mapreduce job but a Spark job.

If you are submitting this spark job to yarn you should user spark-submit.

In any case you would want to build a fat jar ( a.k.a. uber jar).

When you are using SBT, try sbt assembely