Hadoop 2.6.1 Warning: WARN util.NativeCodeLoader

Question

I'm running hadoop 2.6.1 on OS X 10.10.5. I'm getting this warning:

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

I've read that this problem can be caused by running a 32bit native library libhadoop.so.1.0.0 with a 64 bit version of hadoop. I've checked my version of libhadoop.so.1.0.0 and it is 64 bit.

$ find ~/hadoop-2.6.1/ -name libhadoop.so.1.0.0 -ls
136889669     1576 -rwxr-xr-x    1 davidlaxer       staff              806303 Sep 16 14:18 /Users/davidlaxer/hadoop-2.6.1//lib/native/libhadoop.so.1.0.0

$ file /Users/davidlaxer/hadoop-2.6.1//lib/native/libhadoop.so.1.0.0
    /Users/davidlaxer/hadoop-2.6.1//lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

$ env | grep HADOOP
HADOOP_HOME=/Users/davidlaxer/hadoop-2.6.1
HADOOP_COMMON_LIB_NATIVE_DIR=/Users/davidlaxer/hadoop-2.6.1/lib/native
HADOOP_INSTALL=/Users/davidlaxer/hadoop-2.6.1
HADOOP_CONF_DIR=/Users/davidlaxer/hadoop-2.6.1/etc/hadoop
HADOOP_OPTS=-Djava.library.path=/Users/davidlaxer/hadoop-2.6.1/lib

$ hadoop version
Hadoop 2.6.1
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r b4d876d837b830405ccdb6af94742f99d49f9c04
Compiled by jenkins on 2015-09-16T21:07Z
Compiled with protoc 2.5.0
From source with checksum ba9a9397365e3ec2f1b3691b52627f
This command was run using /Users/davidlaxer/hadoop-2.6.1/share/hadoop/common/hadoop-common-2.6.1.jar

$ java -version
java version "1.8.0_05"
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)

$ hadoop checknative -a
15/09/26 11:01:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Native library checking:
hadoop:  false 
zlib:    false 
snappy:  false 
lz4:     false 
bzip2:   false 
openssl: false 
15/09/26 11:01:29 INFO util.ExitUtil: Exiting with status 1


$ sudo port list zlib
Password:
zlib                           @1.2.8          archivers/zlib
$ sudo port list snappy
snappy                         @1.1.1          archivers/snappy
$ sudo port list lz4
lz4                            @r130           archivers/lz4
$ sudo port list bzip2
bzip2                          @1.0.6          archivers/bzip2
$ sudo port list openssl
openssl                        @1.0.2d         devel/openssl

$env | grep CLASS

CLASSPATH=/users/davidlaxer/trunk/core/src/test/java/:/Users/davidlaxer/hadoop-2.6.1-src/hadoop-dist/target/hadoop-dist-2.6.1.jar:/Users/davidlaxer/clojure/target:/Users/davidlaxer/hadoop-2.6.1/lib/native:

˚Any ideas?

Answer 1

I had same problem. Fortunately the problem has been solved. You can see the solution below.

A few times displaying the warning does not let you upload a file to HDFS. So when there is no warning, there is no such problem.

Open the hadoop-env.sh and go to the end of the file and try to add the following lines

export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_ROOT_LOGGER="WARN,DRFA"

Answer 2

I had the same problem and resolved it by adding following xml files in job configuration.

Configuration conf = job.getConfiguration();

//To access HDFS files
conf.addResource(new Path(hadoopHome+"/conf/core-site.xml"));//In my case hadoopHome is /usr/lib/hadoop
conf.addResource(new Path(hadoopHome+"/conf/hdfs-site.xml"));

conf.addResource(new Path(hadoopHome+"/conf/mapred-site.xml")); //To run a mapreduce job

/* Other configurations comes here*/

Answer 3

One reason for the warning can be that there is processor architecture conflict (32 vs 64 bit). Another could be that the native library is not on the java library path (or a library it needs is not on the path).

Executing the following should give you more detailed information:

hadoop checknative -a

Also shouldn't your HADOOP_OPTS actually point to the lib directory instead of lib/native ?

Maybe you should add the following to your log4j.properties file to get more info: log4j.logger.org.apache.hadoop.util.NativeCodeLoader=DEBUG

There are also a few other things you can try:

1)

suda hadoop checknative -a

(to check whether it's a permission issue)

2) Setting the LD_LIBRARY_PATH additionally.

Answer 4

I built hadoop-3.0.0-SNAPSHOT from source with support for native code, adjusted the environment variable HADOOP_COMMON_LIB_NATIVE_DIR:

HADOOP_COMMON_LIB_NATIVE_DIR=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native.

Now, both versions of hadoop (e.g. 2.3.0 and 3.0 SNAPSHOT) get no warning:

HADOOP_HOME=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT
HADOOP_COMMON_LIB_NATIVE_DIR=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native
HADOOP_INSTALL=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT
HADOOP_CONF_DIR=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop
HADOOP_OPTS=-Djava.library.path=/Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native


$ hadoop version
Hadoop 2.3.0
Subversion http://svn.apache.org/repos/asf/hadoop/common -r 1567123
Compiled by jenkins on 2014-02-11T13:40Z
Compiled with protoc 2.5.0
From source with checksum dfe46336fbc6a044bc124392ec06b85
This command was run using /Users/davidlaxer/hadoop-2.3.0/share/hadoop/common/hadoop-common-2.3.0.jar

$ bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Source code repository https://github.com/apache/hadoop.git -r 83e65c5fe84819b6c6da015b269fb4e46a88d105
Compiled by davidlaxer on 2015-09-26T22:46Z
Compiled with protoc 2.5.0
From source with checksum 883bd12016a9bbe21eb0ae4b6beaa
This command was run using /Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar

$ bin/hadoop fs -ls /users/davidlaxer/genomics/reads/HG00103
Found 1 items
drwxr-xr-x   - davidlaxer staff        102 2015-09-26 09:13 /users/davidlaxer/genomics/reads/HG00103/_temporary
David-Laxers-MacBook-Pro:hadoop-3.0.0-SNAPSHOT davidlaxer$ bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Source code repository https://github.com/apache/hadoop.git -r 83e65c5fe84819b6c6da015b269fb4e46a88d105
Compiled by davidlaxer on 2015-09-26T22:46Z
Compiled with protoc 2.5.0
From source with checksum 883bd12016a9bbe21eb0ae4b6beaa
This command was run using /Users/davidlaxer/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar