Error: Could not find or load main class org.apache.flume.node.Application - Install flume on hadoop version 1.2.1

Question

I have built a hadoop cluster which 1 master-slave node and the other is slave. And now, I wanna build a flume to get all log of the cluster on master machine. However, when I try to install flume from tarball and I always get: Error: Could not find or load main class org.apache.flume.node.Application So, please help me to find the answer, or the best way to install flume on my cluster. many thanks!

Answer 1

It is basically because of FLUME_HOME..

Try this command

$ unset FLUME_HOME

Answer 2

I know its been almost a year for this question, but I saw it!

When you set your agnet using sudo bin/flume-ng.... make sure to specify the file where the agent configuration is.

--conf-file flume_Agent.conf -> -f conf/flume_Agent.conf

This did the trick!

Answer 3

look like you run flume-ng in /bin folder flume after build in /flume-ng-dist/target/apache-flume-1.5.0.1-bin/apache-flume-1.5.0.1-bin run flume-ng in this

Answer 4

I suppose you are trying to run flume from cygwin on windows? If that is the case, I had a similar issue. The problem might be with the flume-ng script.

Find the following line in bin/flume-ng:

 $EXEC java $JAVA_OPTS $FLUME_JAVA_OPTS "${arr_java_props[@]}" -cp "$FLUME_CLASSPATH" \
  -Djava.library.path=$FLUME_JAVA_LIBRARY_PATH "$FLUME_APPLICATION_CLASS" $*

and replace it with this

$EXEC java  $JAVA_OPTS $FLUME_JAVA_OPTS "${arr_java_props[@]}" -cp `cygpath -wp "$FLUME_CLASSPATH"` \
  -Djava.library.path=`cygpath -wp $FLUME_JAVA_LIBRARY_PATH` "$FLUME_APPLICATION_CLASS" $*

Notice that the paths have been replaced with the windows directories. Java would not be able to find the library paths from the cygdrive paths and we would have to convert it to the correct windows paths wherever applicable

Answer 5

Maybe you are using the source files, you first should compile the source code and generate the binary code, then inside the binary files directory, you can execute: bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -Dflume.root.logger=DEBUG,console -n agent1. All these information you can follow: https://cwiki.apache.org/confluence/display/FLUME/Getting+Started

Answer 6

I got same issue before, it's simply due to FLUME_CLASSPATH not set

Answer 7

the best way to debug is see the java command being fired and make sure that flume lib is included in the CLASSPATH (-cp),

As in following command its looking for /lib/*, thats where the flume-ng-*.jar are, but its incorrect because there's nothing in /lib, in this line -cp '/staging001/Flume/server/conf://lib/*:/lib/*'. It has to be ${FLUME_HOME}/lib.

usr/lib/jvm/java-1.8.0-ibm-1.8.0.3.20-1jpp.1.el7_2.x86_64/jre/bin/java -Xms100m -Xmx500m $'-Dcom.sun.management.jmxremote\r' \
-Dflume.monitoring.type=http \
-Dflume.monitoring.port=34545 \
-cp '/staging001/Flume/server/conf://lib/*:/lib/*' \
-Djava.library.path= org.apache.flume.node.Application \
-f /staging001/Flume/server/conf/flume.conf -n client

So, if you look at the flume-ng script, There's FLUME_CLASSPATH setup, which if absent it is setup based on FLUME_HOME.

# prepend $FLUME_HOME/lib jars to the specified classpath (if any)
if [ -n "${FLUME_CLASSPATH}" ] ; then
  FLUME_CLASSPATH="${FLUME_HOME}/lib/*:$FLUME_CLASSPATH"
else
  FLUME_CLASSPATH="${FLUME_HOME}/lib/*"
fi

So make sure either of those environments is set. With FLUME_HOME set, (I'm using systemd)

Environment=FLUME_HOME=/staging001/Flume/server/

Here's the working java exec.

/usr/lib/jvm/java-1.8.0-ibm-1.8.0.3.20-1jpp.1.el7_2.x86_64/jre/bin/java -Xms100m -Xmx500m \
$'-Dcom.sun.management.jmxremote\r' \
-Dflume.monitoring.type=http \
-Dflume.monitoring.port=34545 \
-cp '/staging001/Flume/server/conf:/staging001/Flume/server/lib/*:/lib/*' \
-Djava.library.path= org.apache.flume.node.Application \
-f /staging001/Flume/server/conf/flume.conf -n client