i'm having error in running the simple wordcount program

Question

i'm getting error while running the wordcount program..

i tried on python shell

from pyspark import SparkContext

sc = SparkContext("local", "count app")

words = sc.parallelize (
   ["scala", 
   "java", 
   "hadoop", 
   "spark", 
   "akka",
   "spark vs hadoop", 
   "pyspark",
   "pyspark and spark"]
)

counts = words.count()

print("Number of elements in RDD -> %i" % (counts))

Traceback (most recent call last): File "C:/Users/user/AppData/Local/Programs/Python/Python37/pyspark1.py", line 13, in coll = words.collect() File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\pyspark\rdd.py", line 816, in collect sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd()) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\py4j\java_gateway.py", line 1257, in call answer, self.gateway_client, self.target_id, self.name) File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\py4j\protocol.py", line 328, in get_return_value format(target_id, ".", name), value) py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. : java.lang.IllegalArgumentException: Unsupported class file major version 56

at org.apache.xbean.asm6.ClassReader.(ClassReader.java:166)

at org.apache.xbean.asm6.ClassReader.(ClassReader.java:148)

at org.apache.xbean.asm6.ClassReader.(ClassReader.java:136)

at org.apache.xbean.asm6.ClassReader.(ClassReader.java:237)

at org.apache.spark.util.ClosureCleaner$.getClassReader(ClosureCleaner.scala:49)

at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:517)

at org.apache.spark.util.FieldAccessFinder$$anon$3$$anonfun$visitMethodInsn$2.apply(ClosureCleaner.scala:500)

at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)

at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)

at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:134)

at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)

at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)

at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:134)

at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)

at org.apache.spark.util.FieldAccessFinder$$anon$3.visitMethodInsn(ClosureCleaner.scala:500)

at org.apache.xbean.asm6.ClassReader.readCode(ClassReader.java:2175)

at org.apache.xbean.asm6.ClassReader.readMethod(ClassReader.java:1238)

at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:631)

at org.apache.xbean.asm6.ClassReader.accept(ClassReader.java:355)

at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:307)

at org.apache.spark.util.ClosureCleaner$$anonfun$org$apache$spark$util$ClosureCleaner$$clean$14.apply(ClosureCleaner.scala:306)

at scala.collection.immutable.List.foreach(List.scala:392)

at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:306)

at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)

at org.apache.spark.SparkContext.clean(SparkContext.scala:2326)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:2100)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)

at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)

at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)

at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)

at org.apache.spark.rdd.RDD.collect(RDD.scala:944)

at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:166)

at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:567)

at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

at py4j.Gateway.invoke(Gateway.java:282)

at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

at py4j.commands.CallCommand.execute(CallCommand.java:79)

at py4j.GatewayConnection.run(GatewayConnection.java:238)

at java.base/java.lang.Thread.run(Thread.java:835)

Likely duplicate of stackoverflow.com/questions/53583199/… - the exact major version has changed but the core problem of using Java 11 to execute code compiled under a different Java version is probably the same. — Peter DeGlopper

mubai zhang mubai zhang · Accepted Answer · 2019-07-12T19:35:57

0

votes

Spark does not work with Java Version above 9 try downgrading to Java 8

i'm having error in running the simple wordcount program

1 Answers