Storm-Kafka : java.lang.NoClassDefFoundError: com/google/common/base/Strings

Question

I am trying to run a storm LocalCluster using storm-core 1.0.2 and kafka-spout 1.0.2.

My pom file looks like below

  <dependencies>
  <!-- Storm Dependency -->
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-core</artifactId>
        <version>1.0.2</version>
        <scope>provided</scope>
    </dependency>

    <!-- Storm Kafka Dependency -->
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-kafka</artifactId>
        <version>1.0.2</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
            </exclusion>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
 .....
 </dependencies>
<build>
    <resources>
        <resource>
            <directory>src/main/resources</directory>
        </resource>
    </resources>
    <plugins>

        <plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>single</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
                <archive>
                    <manifest>
                        <mainClass>com.walmartlabs.midas.storm.Topology</mainClass>
                    </manifest>
                </archive>
            </configuration>
        </plugin>

        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <version>1.2.1</version>
            <executions>
                <execution>
                    <goals>
                        <goal>exec</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <executable>java</executable>
                <includeProjectDependencies>true</includeProjectDependencies>
                <includePluginDependencies>false</includePluginDependencies>
                <classpathScope>compile</classpathScope>
                <mainClass>${storm.topology}</mainClass>
            </configuration>
        </plugin>
    </plugins>
</build>

I use following command to run storm cluster locally.

mvn package
storm  jar target/myproject-0.1-SNAPSHOT-jar-with-dependencies.jar com.company.project.storm.Topology

I get following exception

Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/base/Strings
    at org.apache.storm.kafka.KafkaSpout.declareOutputFields(KafkaSpout.java:184)
    at org.apache.storm.topology.TopologyBuilder.getComponentCommon(TopologyBuilder.java:431)
    at org.apache.storm.topology.TopologyBuilder.createTopology(TopologyBuilder.java:135)
    at com.walmartlabs.midas.storm.Topology.submitLocalTopology(Topology.java:50)
    at com.walmartlabs.midas.storm.Topology.main(Topology.java:75)
Caused by: java.lang.ClassNotFoundException: com.google.common.base.Strings
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 

The google common dependency is that of storm-kafka artifact. And i have set the scope of storm-kafka as default so i believe all its dependency should be build in my jar.

If i replace KafkaSpout with a simple spout that is just reading some file, everything works fine. I believe while packaging the jar i am not packaging all the dependencies of storm-kafka artifact.What is wrong here?

Answer 1

https://github.com/apache/storm/tree/v1.0.2/external/storm-kafka#using-storm-kafka-with-different-versions-of-kafka

Storm 1.0.2 changed the scope of artifact "kafka_[scala_version]" to "provided", which means that you need to add this as dependency. This is for having flexibility of combination on scala version and/or kafka version.

Answer 2

I get this problem,and sovled by adding the dependency like below (ps: guava.jar version is 16.0.1 used by storm1.0.3) .By the way,we can reference the guava version used by storm-starter.

        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>${guava.version}</version>
            <!--<version>16.0.1</version>-->
        </dependency>