BSONFileInputFormat not found even after adding libs to hadoop folder

0
votes

I was working on the movie recommendations work around using crcmnky's repository. https://github.com/crcsmnky/mongodb-spark-demo I have compiled mongo-hadoop and mongo-java-driver and stored the jars: mongo-hadoop-core-1.3.2-SNAPSHOT and mongo-java-driver-2.13.3.jar in the $HADOOP_HOME/lib folder. After doing all this, I built the project and ran it as per the given instructions on the README file. I get the error:

Exception in thread "main" java.lang.NoClassDefFoundError: com/mongodb/hadoop/BSONFileInputFormat
at com.mongodb.spark.demo.Recommender.main(Recommender.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.mongodb.hadoop.BSONFileInputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

What could have possibly gone wrong? I followed all instructions correctly.

1
javamongodbmavenhadoopbson

1 Answers

2
votes

I had the exact same problem and the son of Zeus took me forever to solve. Try this:

  • Locate your mongo-hadoop-core-1.4.1-SNAPSHOT.jar and mongo-java-driver-2.12.3.jar

  • Add them to the --jars in spark-submit command "before" your --master and the application jar location. This is the crucial step. If you mention the --jars after the two then you will for some insane reason keep getting the BSONFileInputFormat exception. So effectively your spark-submit command would be -

    ./bin/spark-submit --class "com.mongodb.spark.demo.Recommender" --jars /home/killshot/Downloads/mongo-hadoop/core/build/libs/mongo-hadoop-core-1.4.1-SNAPSHOT.jar,/home/killshot/Downloads/mongo-hadoop/work/mongodb-spark-demo/target/lib/mongo-java-driver-2.12.3.jar --master local[4]