how to exclude storm-core jar in clojure project.clj

Question

I am using Leiningen 2.5.3 on Java 1.7.0_79 OpenJDK 64-Bit Server VM. 

I want to exclude storm-core jar using lein uberjar. Below is my project.clj

(defproject kafka2hdfs "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [org.apache.storm/storm-kafka "0.9.5"]
                 [org.apache.storm/storm-hdfs "0.9.5"]
                 [org.apache.kafka/kafka_2.10 "0.8.2.1"]]
  :plugins [[cider/cider-nrepl "0.10.0-SNAPSHOT"]]
  :target-path "target/%s"
  :dev-dependencies [[org.apache.storm/storm-core "0.9.5"]]
  :main kafka2hdfs.core
  :aot [kafka2hdfs.core])

Get error output from lein uberjar, how to fix this?

$ lein uberjar
Compiling kafka2hdfs.core
java.lang.ClassNotFoundException: backtype.storm.StormSubmitter, compiling:(core.clj:1:1)
Exception in thread "main" java.lang.ClassNotFoundException: backtype.storm.StormSubmitter, compiling:(core.clj:1:1)
    at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3463)

My core.clj file

(ns kafka2hdfs.core
  (:import [backtype.storm StormSubmitter LocalCluster spout.SchemeAsMultiScheme]
           [storm.kafka ZkHosts SpoutConfig StringScheme KafkaSpout]
           [org.apache.storm.hdfs.bolt HdfsBolt]
           [org.apache.storm.hdfs.bolt.format DefaultFileNameFormat DelimitedRecordFormat]
           [org.apache.storm.hdfs.bolt.sync CountSyncPolicy]
           [org.apache.storm.hdfs.bolt.rotation TimedRotationPolicy]
           )
  (:use [backtype.storm clojure config]) ;; for (topology ...)
  (:gen-class))

(defn mk-topology []
   ;; ......
  ）

    (topology
     {"kafka-reader" (spout-spec kafka-reader :p 2)}
     {"hdfs-writer" (bolt-spec {"kafka-reader" :shuffle} hdfs-writer :p 1)})
    ))

(defn submit-topology! [name]
  (StormSubmitter/submitTopology
   name
   {TOPOLOGY-DEBUG true
    TOPOLOGY-WORKERS 3}
   (mk-topology)))

(defn -main
  "a simple topology demo read from kafka ans write to hdfs"
  [& args]
  (submit-topology! "kafka2hdfs-topic-bigdata-obd"))

After a few hours, I find if I comment out :aot [kafka2hdfs.core] in project.clj, it works but output warning message:

$ lein do clean, uberjar
Warning: The Main-Class specified does not exist within the jar. It may not be executable as expected. A gen-class directive may be missing in the namespace which contains the main method.

I am new to lein and :aot, why?

Answer 1

I don't really understand understand how excluding the jar would solve the ClassNotFoundException, but you can exclude transitive dependencies in leiningen by

[org.apache.storm/storm-kafka "0.9.5" :exclusions [org.apache.storm/storm-core]]

It looks like the reason for your error is a missing backtype/storm jar. Maybe you need to :import from org.apache.storm instead of backtype.storm?

Answer 2

To exclude jar file from package, need to understand lein project.clj will be used to generate maven pom.xml. use lein pom command to generate pom.xml. In my case, I just need to apply provided scope to storm-core dependency, and use :exclusions to generate dependency exclusion. So my project.clj looks like so

(defproject kafka2hdfs "0.1.0-SNAPSHOT"
  :description "demo to show read from kafka and write to hdfs"
  :url "http://blog.csdn.net/csfreebird"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [org.apache.kafka/kafka_2.10 "0.8.2.1"
                  :exclusions [[org.apache.zookeeper/zookeeper]
                               [log4j/log4j]
                               [slf4j-api/org.slf4j]]
                  ]
                 [org.apache.storm/storm-kafka "0.9.5"]
                 [org.apache.storm/storm-hdfs "0.9.5"]]
  :main kafka2hdfs.core
  :aot [kafka2hdfs.core]
  :profiles {:provided {:dependencies [[org.apache.storm/storm-core "0.9.5"
                                        :exclusions [[org.slf4j/log4j-over-slf4j]
                                                     [org.slf4j/slf4j-api]
                                                     [logback-classic/ch.qos.logback]]
                                        ]]}}
  :plugins [[cider/cider-nrepl "0.10.0-SNAPSHOT"]]
  :target-path "target/%s")

Using lein pom to generate pom.xml

  <dependencies>
    <dependency>
      <groupId>cider</groupId>
      <artifactId>cider-nrepl</artifactId>
      <version>0.10.0-SNAPSHOT</version>
    </dependency>
    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>clojure</artifactId>
      <version>1.5.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka_2.10</artifactId>
      <version>0.8.2.1</version>
      <exclusions>
        <exclusion>
          <groupId>org.apache.zookeeper</groupId>
          <artifactId>zookeeper</artifactId>
        </exclusion>
        <exclusion>
          <groupId>log4j</groupId>
          <artifactId>log4j</artifactId>
        </exclusion>
        <exclusion>
          <groupId>slf4j-api</groupId>
          <artifactId>org.slf4j</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.storm</groupId>
      <artifactId>storm-kafka</artifactId>
      <version>0.9.5</version>
    </dependency>
    <dependency>
      <groupId>org.apache.storm</groupId>
      <artifactId>storm-hdfs</artifactId>
      <version>0.9.5</version>
    </dependency>
    <dependency>
      <groupId>org.apache.storm</groupId>
      <artifactId>storm-core</artifactId>
      <version>0.9.5</version>
      <exclusions>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>log4j-over-slf4j</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-api</artifactId>
        </exclusion>
        <exclusion>
          <groupId>logback-classic</groupId>
          <artifactId>ch.qos.logback</artifactId>
        </exclusion>
      </exclusions>
      <scope>provided</scope>
    </dependency>
  </dependencies>

That's my answer! The class not found exception also resolved.