Building a jar to run on Linux Server

Question

I have a Scala program and I'm using the Intellij IDE. I created a jar of this program and put it on the Linux server. I also made a shell script to execute the program. Whenever I run the shell script, I get this error:

java.lang.ClassNotFoundException: com.tac.cco.associations.HDFStoES.main at java.net.URLClassLoader$1.run(URLClassLoader.java:359) at java.net.URLClassLoader$1.run(URLClassLoader.java:348) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:347) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:278) at org.apache.spark.util.Utils$.classForName(Utils.scala:229) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:695) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Here's also my Shell script

loader.sh

#!/bin/bash

input_path=/user/content/dashboard/t_date=
#output_path=/user/content/t_orc_content
lib=/apps/cco/lib
midt_date=20170626
jar_file=/apps/cco/ReportingData/bin/ReportingData.jar

input=$input_path$t_date
args="--jars $lib/elasticsearch-spark-20_2.11-5.2.1.jar,$lib/elasticsearch-5.2.1.jar,$lib/kafka-clients-0.9.0.2.3.4.51-1.jar,$lib/config-1.3.1.jar,$lib/DistributedLogger-0.0.2.jar,$lib/argparse4j-0.7.0.jar,$lib/spark-sql_2.11-2.1.0.jar,$lib/spark-core_2.11-2.1.0.jar"
opt="--driver-memory 30g --executor-memory 20g --executor-cores 10 --num-executors 6 --master yarn --conf spark.hadoop.yarn.timeline-service.enabled=false"

/usr/local/spark/bin/spark-submit $opt $args --class com.tac.cco.associations.HDFStoES.main $jar_file $t_date
exit

The program's main file is in src/main/scala/com/tac/cco/associations/HDFStoES/main.scala but it seems like it is not recognizing it. The manifest file is under src/main/META-INF/MANIFEST.MF and contains the following:

Manifest-Version: 1.0
Main-Class: com.tac.cco.associations.HDFStoES.main

I don't understand why this error message is popping up rather than the program running.

To create the Jar I did the following: I went into project structure -> artificats -> jar -> from modules with dependencies -> I then selected the ReportingData (name of my project) module. Selected the com.tac.cco.associations.HDFStoES.main as my main class then selected the directory for the mainfest. I then removed all dependencies except the manifest folder. I then created it, went to build artifacts, cleaned it first and then build it.

@lev I went into project structure -> artificats -> jar -> from modules with dependencies -> I then selected the ReportingData (name of my project) module. Selected the com.tac.cco.associations.HDFStoES.main as my main class then selected the directory for the mainfest. I then removed all dependencies except the manifest folder. I then created it, went to build artifacts, cleaned it first and then build it. — user2896120
The class that contains your main method is literally called "main"? That's what you're saying based on the manifest you shared. — D.B.
Ok, as long as the fully qualified name is correct and the class is either contained in the jar or available in the runtime classpath it should work. So there must be some problem with either the jar or your classpath. Also note that naming a class starting with lowercase does not follow Java Naming Conventions — D.B.

Abhimanyu Abhimanyu · Accepted Answer · 2017-11-12T13:52:41

Add this following plugin in your pom.xml

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
  <execution>
    <phase>package</phase>
    <goals>
      <goal>shade</goal>
    </goals>
    <configuration>
      <transformers>
        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
          <mainClass>com.tac.cco.associations.HDFStoES.main</mainClass>
        </transformer>
      </transformers>
    </configuration>
  </execution>
</executions>
</plugin>

Building a jar to run on Linux Server

1 Answers