I'm trying to submit a JAR to my Apache Spark 2.2.1 cluster using Scala 2.11. I included some extra dependencies in my JAR, namely Apache Commons CLI, and packaged it all into a fat JAR. However, I'm getting a NoSuchMethodError when I submit my Spark application. I'm quite sure that it's not due to inconsistency in Scala versions, but rather something weird with dependencies.
The command is simply spark-submit myjar.jar [arguments]
This is the error:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.commons.cli.Options.addRequiredOption(Ljava/lang/String;Ljava/lang/String;ZLjava/lang/ String;)Lorg/apache/commons/cli/Options; at xyz.plenglin.aurum.spark.RunOnSpark$.main(RunOnSpark.scala:46) at xyz.plenglin.aurum.spark.RunOnSpark.main(RunOnSpark.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Running java -jar myjar.jar [arguments] works without any issues. Peeking inside the JAR, I see org.apache.commons.cli.Options where it should be.
