hadoop classes other than main not found

Question

I'm running hadoop Cloudera CHD4.5 on a VM. From Manhout in Action chapters 9-10, I try to create a custom lucene analyzer. My analyzer is defined in its own class. When I create the JAR the class is present. When I execute via command line, I keep getting java.lang.IllegalStateException: java.lang.ClassNotFoundException: my.org.MyAnalyzer.

HADOOP_CLASSPATH works to pass missing classes to client but my problem is that it seems that myanalyzer class is not passed to the map/reduce JVM. I have not created my own map/reduce classes and do not run a Job from my main class. I'm using the existing mahout classes to achieve my goal. In my code if I use WhitespaceAnalyzer for example instead of my analyzer, I do not have problem.

This is my analyzer class

public final class MyAnalyzer extends Analyzer {

public final StandardAnalyzer stdAnalyzer = new StandardAnalyzer(Version.LUCENE_36);

@Override
 public TokenStream tokenStream(String fieldName, Reader reader) {
    return stdAnalyzer.tokenStream(fieldName, reader);

  }
}

Any ideas?

I tried maven and I have the same result. However, I might not have the right dependency in the pom.xml file.

-libjars is not working as I'm not using GenericOptionsParser method in my code.

user3213526 user3213526 · Accepted Answer · 2014-01-20T16:37:51

It is now working. I used maven-assembly-plugin in my pom.xml and created corresponding job.xml assembly job.

pom.xml

<project    xmlns="http://maven.apache.org/POM/4.0.0" 
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0       http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>

<groupId>my.org</groupId>
<artifactId>maven_NewKmeans</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>

<name>maven_NewKmeans</name>
<url>http://maven.apache.org</url>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <hadoop.version>2.0.0-mr1-cdh4.0.1</hadoop.version>

    <mahout.version>0.7</mahout.version>
    <mahout.groupid>org.apache.mahout</mahout.groupid>
    </properties>

<repositories>
    <repository>
        <id>cloudera-releases</id>
        <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
        <releases>
                <enabled>true</enabled>
        </releases>
        <snapshots>
                <enabled>false</enabled>
        </snapshots>
    </repository>
</repositories>
<build>

<plugins> 

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>2.3.2</version>
        <configuration>
            <source>1.6</source>
            <target>1.6</target>
        </configuration>
    </plugin>

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-resources-plugin</artifactId>
        <version>2.4.3</version>
        <configuration>
            <encoding>UTF-8</encoding>
        </configuration>
    </plugin>

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-assembly-plugin</artifactId>
        <executions>
            <execution>
                <id>job</id>
                <phase>package</phase>
                <goals>
                    <goal>single</goal>
                </goals>
                <configuration>
                    <descriptors>
                             <descriptor>src/main/assembly/job.xml</descriptor>
                    </descriptors>
                </configuration>
            </execution>
            <execution>
                <id>my-jar-with-dependencies</id>
                <phase>package</phase>
                <goals>
                    <goal>single</goal>
                </goals>
                <configuration>
                    <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
            </execution>
        </executions>
    </plugin>

</plugins>

<pluginManagement> 
</pluginManagement>

 </build>

<dependencies>

<dependency>
  <groupId>junit</groupId>
  <artifactId>junit</artifactId>
  <version>3.8.1</version>
  <scope>test</scope>
</dependency>

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-client</artifactId>
  <version>${hadoop.version}</version>
</dependency>

<dependency>
  <groupId>${mahout.groupid}</groupId>
  <artifactId>mahout-core</artifactId>
  <version>${mahout.version}</version>
</dependency>

</dependencies>

</project>

assembly job.xml

<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-   plugin/assembly/1.1.0
http://maven.apache.org/xsd/assembly-1.1.0.xsd">
<id>job</id>
<formats>
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
  <unpack>true</unpack>
  <scope>runtime</scope>
  <outputDirectory>/</outputDirectory>
  <excludes>
    <exclude>org.apache.hadoop:hadoop-core</exclude>
  </excludes>
</dependencySet>
</dependencySets>
<fileSets>
<fileSet>
  <directory>${basedir}/target/classes</directory>
  <outputDirectory>/</outputDirectory>
  <excludes>
    <exclude>*.jar</exclude>
  </excludes>
</fileSet>
<fileSet>
  <directory>${basedir}/target/classes</directory>
  <outputDirectory>/</outputDirectory>
  <includes>
    <include>driver.classes.default.props</include>
  </includes>
</fileSet>

hadoop classes other than main not found

1 Answers