Not a valid JAR from testing hadoop mapreduce wordcount

1
votes

New to Hadoop Mapreduce and I am not sure why I am getting this error.

I am currently getting:

Not a valid JAR: /nfsdata/DSCluster/home/user/bli1/wordcount/wc.jar

This is my process:

HDFS file structure:

   /bli1
      /wordcount
         wc.jar
         WordCount.java
         /input
            file01
         /output
            file02
  1. Create on local machine WordCount.java and wc.jar with jar cf wc.jar WordCount.java
  2. Create on local machine file01 and file02
  3. Create /input and /output directory within /wordcount in hdfs

  4. Move files to their directories with hdfs

    • hdfs dfs -moveFromLocal <localsrc> <dst>

  5. run command:

    hadoop jar /user/bli1/wordcount/wc.jar WordCount /user/bli1/wordcount/input /user/bli1/wordcount/output

OUTPUT:

Not a valid JAR: /user/bli1/wordcount/wc.jar

I tried this to make the JAR but I would get an error.

$ hadoop com.sun.tools.javac.Main WordCount.java
Error: Could not find or load main class com.sun.tools.javac.Main

If this process to make the JAR is different than jar cf wc.jar WordCount.java then this might be causing the problem. However I don't know why I am getting the find or load main class error

1
javahadoopjarmapreduce

1 Answers

1
votes

Liondancer, within the Java jar file you must add the compiled *.class files, not the source *.java files.

In addition, once you resolve the problem, I think most probably Hadoop will complaint about the existence of the output directory. I mean, Hadoop will expect to create itself such an output directory, which must not exist.