Hadoop MapReduce wordcount tutorial error: Input path does not exist

Question

I am getting an error when running the basic Hadoop wordcout tutorial in eclipse on my Cloudera VM. The error is:

Input path does not exist: file:/user/cloudera/wordcount/input

However, this directory does exist, I've specifically created it and I'm staring at it in my HDFS file browser.

Screenshot of my command line query and browser.

I have copy pasted the directory from the command you see into my run configuration in eclipse, these are my arguments:

/user/cloudera/wordcount/input /user/cloudera/wordcount/output

And the code itself has not been modified from the example I downloaded, the relevant lines are:

FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

Edit:

hadoop jar /home/cloudera/examples/wordcount.jar WordCount /user/cloudera/wordcount/input /user/cloudera/wordcount/output

This command works in the terminal, but when I execute the java file that I built that jar from (no changes, just export->jar file) with the arguments "/user/cloudera/wordcount/input /user/cloudera/wordcount/output" then I get the above error.

Why won't eclipse run this task properly? It works from the terminal.

Answer 1

Why won't eclipse run this task properly?

I assume you are giving /user/cloudera/wordcount/input as args[0] to Eclipse?

This will create new Path(args[0]), which is the local JVM and filesystem, designated by the URI file://.

When you give the arguments to hadoop jar, it knows that it should read from HDFS, which is specified in an XML file for your Hadoop configurations, and looks for the input at URI hdfs://.

For reference, ls -l / compared to hdfs dfs -ls / are very different outputs. The former doesn't have the specified path.

Sidenote: The default HDFS path is using file://, but Cloudera Quickstart changes this for you.