hadoop error: util.NativeCodeLoader (hdfs dfs -ls does not work!)

Question

I have seen a lot of folks getting problem with hadoop installation. I went through all the related stackoverflow questions, but could not fix the problem.

The problem is :

hdfs dfs -ls 
16/09/27 09:43:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ls: `.': No such file or directory

I am using ubuntu 16.04 and I downloaded hadoop stable version 2.7.2 from Apache mirror:
http://apache.spinellicreations.com/hadoop/common/
I have installed java and ssh already.

which java
java is /usr/bin/java
which javac
javac is /usr/bin/javac
which ssh
ssh is /usr/bin/ssh
echo $JAVA_HOME
/usr/lib/jvm/java-9-openjdk-amd64

Note:

sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                            Priority   Status
------------------------------------------------------------
* 0            /usr/lib/jvm/java-9-openjdk-amd64/bin/java       1091      auto mode
  1            /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java   1081      manual mode
  2            /usr/lib/jvm/java-9-openjdk-amd64/bin/java       1091      manual mode

Press <enter> to keep the current choice[*], or type selection number:

hadoop environment variables in ~/.bashrc

export JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64 export HADOOP_INSTALL=/home/bhishan/hadoop-2.7.2 export PATH=$PATH:$HADOOP_INSTALL/bin export PATH=$PATH:$HADOOP_INSTALL/sbin export HADOOP_MAPRED_HOME=$HADOOP_INSTALL export HADOOP_COMMON_HOME=$HADOOP_INSTALL export HADOOP_HDFS_HOME=$HADOOP_INSTALL export YARN_HOME=$HADOOP_INSTALL export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
export PATH=$PATH:$HADOOP_HOME/bin
Modification of file:
/home/bhishan/hadoop-2.7.2/etc/hadoop/hadoop-env.sh

Added a one line at the end:

export JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64

The link to hadoop-env.sh in the pastebin is here:
http://pastebin.com/a3iPjB04

Then I created some empty directories:

/home/bhishan/hadoop-2.7.2/tmp /home/bhishan/hadoop-2.7.2/etc/hadoop/hadoop_store /home/bhishan/hadoop-2.7.2/etc/hadoop/hadoop_store/hdfs /home/bhishan/hadoop-2.7.2etc/hadoop/hadoop_store/hdfs/datanode /home/bhishan/hadoop-2.7.2/etc/hadoop/hadoop_store/hdfs/namenode
Modifications to the file: /home/bhishan/hadoop-2.7.2/etc/hadoop/hdfs-site.xml

<property>  
  <name>dfs.replication</name>  
  <value>1</value>  
  <description>Default block replication.  
  The actual number of replications can be specified when the file is created.  
  The default is used if replication is not specified in create time.  
  </description>  
 </property>  
 <property>  
   <name>dfs.namenode.name.dir</name>  
  <value>file:/home/bhishan/hadoop-2.7.2/etc/hadoop/hadoop_store/hdfs/namenode</value>

 </property>  
 <property>  
   <name>dfs.datanode.data.dir</name>  
   <value>file:/home/bhishan/hadoop-2.7.2/etc/hadoop/hadoop_store/hdfs/datanode</value>

 </property>

The link in the pastebin is this:
http://pastebin.com/cha7ZBr8

Modifications to the file: /home/bhishan/hadoop-2.7.2/etc/hadoop/core-site.xml

is following:

hadoop.tmp.dir
/home/bhishan/hadoop-2.7.2/tmp A base for other temporary directories.

fs.default.name
hdfs://localhost:54310 The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.

The link to the pastebin for core-site.xml is this: http://pastebin.com/D184DuGB

The Modification to file are given below: /home/bhishan/hadoop-2.7.2/etc/hadoop/mapred-site.xml

mapred.job.tracker
localhost:54311 The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task.

The pastebin link is:
http://pastebin.com/nVxs8nMm

when I type hostname in the terminal it says BP

cat /etc/hosts
127.0.0.1 localhost BP
127.0.1.1 localhost

The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
I have also disabled ipv6

cat /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
hadoop descriptions

hadoop version
Hadoop 2.7.2

which hadoop
hadoop is /home/bhishan/hadoop-2.7.2/bin/hadoop

which hdfs
hdfs is /home/bhishan/hadoop-2.7.2/bin/hdfs
Restarting hadoop

cd /home/bhishan/hadoop-2.7.2/sbin
stop-dfs.sh
stop-yarn.sh

cd /home/bhishan/hadoop-2.7.2/tmp && rm -Rf *
hadoop namenode -format

start-dfs.sh
start-yarn.sh
Now the error comes

hdfs dfs -ls

16/09/26 23:53:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ls: `.': No such file or directory

checking jps

jps
6688 sun.tools.jps.Jps
3909 SecondaryNameNode
3525 NameNode
4327 NodeManager
4184 ResourceManager
3662 DataNode
checknative

hadoop checknative -a
16/09/27 09:28:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Native library checking:
hadoop: false
zlib: false
snappy: false
lz4: false
bzip2: false
openssl: false
16/09/27 09:28:18 INFO util.ExitUtil: Exiting with status 1
Then I installed missing libraries:
a) which hadoop gives Hadoop 2.7.2

b) sudo apt-get install --reinstall zlibc zlib1g zlib1g-dev
From synaptic manager I can see following libraries installed:
zlib1g, zlib1g-dev , zlib1g:i386, zlibc

c) Installed snappy and python-snappy.

d) In Synaptic manager I can see lz4 liblz4-1, liblz4-tool, python-lz4, python3-lz4

e) bzip2 is already installed.

f) openssl is already installed.

All checknative are false and I can not run hdfs dfs -ls I could not find any errors till now. Any help will be appreciated.

Also, I am trying to run hadoop in Single laptop with four cores. The version is 2.7.2, How is version 3.0, If I have to reinstall the hadoop from Scratch, may be I should go with hadoop3. Suggestions will be welcomed.

Related links:
Result of hdfs dfs -ls command
hdfs dfs ls not working after multiple nodes configured
hadoop fs -ls does not work
Namenode not getting started
No Namenode or Datanode or Secondary NameNode to stop
Hadoop 2.6.1 Warning: WARN util.NativeCodeLoader
Hadoop 2.2.0 Setup (Pseudo-Distributed Mode): ERROR// Warn util.NativeCodeLoader: unable to load native-hadoop library
Command "hadoop fs -ls ." does not work

And, also,
hadoop fs -mkdir failed on connection exception
Hadoop cluster setup - java.net.ConnectException: Connection refused
Hadoop (local and host destination do not match) after installing hive

Help will be truly appreciated!

Stephen ODonnell Stephen ODonnell · Accepted Answer · 2016-09-27T15:46:21

From this error:

hdfs dfs -ls 
16/09/27 09:43:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
ls: `.': No such file or directory

Ignore the warning about the native libraries - the command should work fine even with that warning.

When you run hdfs dfs -ls with no path as you have done, it attempts to list the contents of your home directory in HDFS, which is /user/ by default. In this case, I suspect this issue is simply that your user directory does not exist.

Does it work OK if you run:

hadoop fs -ls /

And then do:

hadoop fs -mkdir -p /user/<your_user_name>
hadoop fs -ls

hadoop error: util.NativeCodeLoader (hdfs dfs -ls does not work!)

The following lines are desirable for IPv6 capable hosts

1 Answers