Cloudera's CDH4 WordCount hadoop tutorial - issues

Question

I am following this tutorial: http://www.cloudera.com/content/cloudera-content/cloudera-docs/HadoopTutorial/CDH4/Hadoop-Tutorial/ht_topic_5_2.html

It says the following:

javac -cp classpath -d wordcount_classes WordCount.java

where classpath is:
CDH4 - /usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/*
CDH3 - /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u4-core.jar

I have downloaded the "cloudera-quickstart-demo-vm-4.2.0-vmware" . Running as user cloudera.

    [cloudera@localhost wordcount]$ javac -cp /usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/* -d wordcount_classes WordCount.java
incorrect classpath: /usr/lib/hadoop/*
incorrect classpath: /usr/lib/hadoop/client-0.20/*
----------
1. ERROR in WordCount.java (at line 8)
        import org.apache.hadoop.fs.Path;
               ^^^^^^^^^^

When checking the cp folder: .

[cloudera@localhost wordcount]$ ls -l /usr/lib/hadoop
total 3500
drwxr-xr-x. 2 root root    4096 Apr 22 14:37 bin
drwxr-xr-x. 2 root root    4096 Apr 22 14:33 client
drwxr-xr-x. 2 root root    4096 Apr 22 14:33 client-0.20
drwxr-xr-x. 2 root root    4096 Apr 22 14:36 cloudera
drwxr-xr-x. 2 root root    4096 Apr 22 14:30 etc
-rw-r--r--. 1 root root   16536 Feb 15 14:24 hadoop-annotations-2.0.0-cdh4.2.0.jar
lrwxrwxrwx. 1 root root      37 Apr 22 14:30 hadoop-annotations.jar -> hadoop-annotations-2.0.0-cdh4.2.0.jar
-rw-r--r--. 1 root root   46855 Feb 15 14:24 hadoop-auth-2.0.0-cdh4.2.0.jar
lrwxrwxrwx. 1 root root      30 Apr 22 14:30 hadoop-auth.jar -> hadoop-auth-2.0.0-cdh4.2.0.jar
-rw-r--r--. 1 root root 2266171 Feb 15 14:24 hadoop-common-2.0.0-cdh4.2.0.jar
-rw-r--r--. 1 root root 1212163 Feb 15 14:24 hadoop-common-2.0.0-cdh4.2.0-tests.jar
lrwxrwxrwx. 1 root root      32 Apr 22 14:30 hadoop-common.jar -> hadoop-common-2.0.0-cdh4.2.0.jar
drwxr-xr-x. 3 root root    4096 Apr 22 14:36 lib
drwxr-xr-x. 2 root root    4096 Apr 22 14:33 libexec
drwxr-xr-x. 2 root root    4096 Apr 22 14:31 sbin

What am I doing wrong? This is directly from the Cloudera Quickstart VM with CDH4 installed. Following the "Hadoop Tutorial" . It even says

**Prerequisites**

Ensure that CDH is installed, configured, and running. The easiest way to get going quickly is to use a CDH4 QuickStart VM

Which is exactly from where I am running this tutorial from - the CDH4 QuickStart VM.

What am I doing wrong?

*update Version information;

[cloudera@localhost cloudera]$ cat cdh_version.properties

# Autogenerated build properties
version=2.0.0-cdh4.2.0
git.hash=8bce4bd28a464e0a92950c50ba01a9deb1d85686
cloudera.hash=8bce4bd28a464e0a92950c50ba01a9deb1d85686
cloudera.base-branch=cdh4-base-2.0.0
cloudera.build-branch=cdh4-2.0.0_4.2.0
cloudera.pkg.version=2.0.0+922
cloudera.pkg.release=1.cdh4.2.0.p0.12
cloudera.cdh.release=cdh4.2.0
cloudera.build.time=2013.02.15-18:39:29GMT

cloudera.pkg.name=hadoop

CLASSPATH ENV:

[cloudera@localhost bin]$ echo $CLASSPATH
:/usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/*

EDIT!! So I think I figured it out. This is a new issue possibly with the Cloudera CD4 VM quickstart VM: from: This Post dated yesterday Another person was having the exact same problem. It seems the javac program does not accept wildcards properly on exported paths. I had to do the following:

export CLASSPATH=/usr/lib/hadoop/client-0.20/\*:/usr/lib/hadoop/\*

Then javac -d [Without a -cp override]

javac -d wordcount_classes/ WordCount.java

Only warnings will appear.

I wonder if Cloudera has to fix their quickstart VM.

make sure the cdh4 libraries are where you think they should be, this error is just saying it can't find the hadoop libs — greedybuddha
How do I know where the libraries are? i did a 'hadoop classpath' command, even tried everything within there. nothing. — Mang G11
I have updated my original post showing all files in /usr/lib/hadoop. — Mang G11

apesa apesa · Accepted Answer · 2013-05-15T16:20:34

You need to have a classpath variable set that includes those directories in /usr/lib/hadoop if you want javac to find them. You can set this env var as follows

$: export CLASSPATH=$CLASSPATH:/usr/lib/hadoop/*:/usr/lib/hadoop/client-0.20/*

javac will now find those libs. If you have any additional complaining regarding classpath variables you can just append them to the above line using a colon (:) as a delimiter

You could include this in a bash script, but it is best practice to set the correct env variables at runtime, then you get exactly what you want. In this case it could be word count or the CDH4 env is setting it, but it is best to just set it yourself.

Cloudera's CDH4 WordCount hadoop tutorial - issues

2 Answers