Failure to load the native-hadoop library

Question

Without -Djava.library.path=/opt/mapr/hadoop/hadoop-0.20.2/lib/native/Linux-amd64-64/ as an argument to running Java, I get the following error,

2013-11-13 15:23:29,414 WARN    pool-3-thread-3 org.apache.hadoop.util.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2013-11-13 15:23:29,414 INFO    pool-3-thread-3 org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback        Falling back to shell based

What does this mean? Is it slower to use shell based? Should I fix this? Should I care about this warning?

FYI this error (or an UnspecifiedLinkError) can be fixed by either of the following,

See this answer for details setting the java.library.path - rhive.connect() issues with MapR distribution
To the file hadoop-env.sh add the following line

export HADOOP_CLIENT_OPTS=-Djava.library.path=$HADOOP_HOME/lib/native/Linux-amd64-64/

Thanks for the help.

Charles Menguy Charles Menguy · Accepted Answer · 2013-11-14T03:48:07

For the WARN message that you get, I assume you already figured it out, but this just means that you are not using the native libraries for Hadoop which could be slower or just not working properly for gzip.

But for your second message from JniBasedUnixGroupsMappingWithFallback it's a bit more complex. If you look at the source you'll see something like this:

if (NativeCodeLoader.isNativeCodeLoaded()) {
    this.impl = new JniBasedUnixGroupsMapping();
} else {
    LOG.info("Falling back to shell based");
    this.impl = new ShellBasedUnixGroupsMapping();
}

The only difference between ShellBasedUnixGroupsMapping and JniBasedUnixGroupsMapping is that ShellBasedUnixGroupsMapping will just do a bash -c groups via a ProcessBuilder to figure out which groups a user belongs to, while JniBasedUnixGroupsMapping will use JNI to communicate with the libC to get the list of groups.

I haven't run any benchmark, but I don't think there is too much overhead with ProcessBuilder compared to the libC implementation since it's just to get the groups so the impact is minimal. That being said, since it looks like you already figured it out, it definitely doesn't hurt using the native libraries for performance improvements in other areas of Hadoop.

Failure to load the native-hadoop library

1 Answers