Running Java Native libraries on Hadoop

Question

I am trying to run Gurobi optimizer on the local run of hadoop. Gurobi uses JNI. To test how to run it on hadoop I've written the following code.

  public static void main(String[] args) throws Exception {
    try {
      GRBEnv env2 = new GRBEnv();
      env2.set(GRB.IntParam.OutputFlag, 0);
    } catch (Exception e) {
      e.printStackTrace();
    }

    Configuration conf = new Configuration();
    int res = ToolRunner.run(conf, new GurobiTest(), args);
    System.exit(res);
}

Before using, Gurobi requires to set some environmental variables (where LD_LIBRARY_PATH points to Gurobi native library):

export LD_LIBRARY_PATH=/home/username/System/gurobi563/linux64/lib
export GRB_LICENSE_FILE=/home/username/gurobi.lic

When I package my project and run it using jvm

java -cp ./target/GurobiOnHadoop-0.0.1-SNAPSHOT-jar-with-dependencies.jar mpi.de.test.GurobiTest

everything seems to work fine

But when I am trying to run it on hadoop

hadoop jar ./target/GurobiOnHadoop-0.0.1-SNAPSHOT-jar-with-dependencies.jar mpi.de.test.GurobiTest -libjars ./target/GurobiOnHadoop-0.0.1-SNAPSHOT-jar-with-dependencies.jar

I get an error

Exception in thread "main" java.lang.UnsatisfiedLinkError: no GurobiJni56 in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
    at java.lang.Runtime.loadLibrary0(Runtime.java:849)
    at java.lang.System.loadLibrary(System.java:1088)
    at gurobi.GurobiJni.<clinit>(GurobiJni.java:193)
    at gurobi.GRBEnv.<init>(GRBEnv.java:16)
    at gurobi.GRBEnv.<init>(GRBEnv.java:11)
    at mpi.de.test.GurobiTest.main(GurobiTest.java:89)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:197)

I've tried many different things. I've added "-Djava.library.path=/home/username/System/gurobi563/linux64/lib" to the hadoop running command. I've set required environmental variables in hadoop-env.sh. I try to load *.so files from the code level (using e.g. ystem .load("/home/username/System/gurobi563/linux64/lib/libGurobiJni56.so");). I've also tried to send those libraries to the hadoop distributed file system, load files from that place or point enviromental variables to that place.

I am using:

java version "1.7.0_60"

hadoop-0.20.2-cdh3u6

Debian 7.5 wheezy

Its hard to tell what exactly is going on with the information provided. First, if you are specifying a path or filename, you should be using System.loadLibrary not .load. If you use .load, you need to set java.library.path and the argument needs to be a library name without the lib prefix or file extension. Other things to check: Does the hadoop process have +r-x on /home/username/System/gurobi563/linux64/lib? Did you specify JAVA_LIBRARY_PATH and LD_LIBRARY_PATH in the hadoop config? — Alex Barker
JAVA_LIBRARY_PATH and LD_LIBRARY_PATH are set in hadoop-env.sh. Files in /home/username/System/gurobi563/linux64/lib have -rwxrwxrwx permission. System.loadLibrary("libGurobiJni56") returns java.lang.UnsatisfiedLinkError: no libGurobiJni56.so in java.library.path (even when I run java locally with LD_LIBRARY_PATH set). — Anonim33
Either System.loadLibrary("/home/username/System/gurobi563/linux64/lib/libGurobiJni56.so") or System.load("GurobiJni56"); — Alex Barker

Anonim33 Anonim33 · Accepted Answer · 2014-06-26T13:42:20

0

votes

It had turn out that I set JAVA_LIBRARY_PATH in hadoop-env.sh incorrectly.

Running Java Native libraries on Hadoop

1 Answers