Hadoop: Unable to load native-hadoop library for your platform. using builtin-java classes > where applicable

Question

I am trying to configure hadoop and format namenode using this command:

$ hdfs namenode -format

However, I keep getting this error. How can I fix it?

2017-06-20 12:22:25,792 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ^[[A2017-06-20 12:22:28,825 WARN ipc.Client: Failed to connect to server: localhost/127.0.0.1:9000: try once and fail. java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:681) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:777) at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:408) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1542) at org.apache.hadoop.ipc.Client.call(Client.java:1373) at org.apache.hadoop.ipc.Client.call(Client.java:1337) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115) at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:812) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:398) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:335) at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1638) at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1367) at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1364) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1379) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64) at org.apache.hadoop.fs.Globber.doGlob(Globber.java:269) at org.apache.hadoop.fs.Globber.glob(Globber.java:148) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1960) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:239) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:222) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103) at org.apache.hadoop.fs.shell.Command.run(Command.java:166) at org.apache.hadoop.fs.FsShell.run(FsShell.java:326) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:389) ls: Call From ubuntu/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused \ERROR: JAVA_HOME /opt/jdk1.8.0_91/ does not exist.

Answer 1

2017-06-20 12:22:25,792 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Apache Hadoop can integrate with an optional native library that contains extensions implementing deeper OS integration and performance enhancements for certain features. For more details, refer to the documentation page Native Libraries Guide.

In many cases for client-side usage or developer setups, the native library extensions are optional. However, the log warning can be a nuisance. If you'd like to get it out of the way, there are a few options:

1. Some Apache Hadoop releases include a pre-built native library. You could potentially use this. This bundled library would use whatever OS/architecture executed the Apache Hadoop release build, so there is no guarantee that it will match your own OS/architecture. Do not attempt to mix and match different versions of Hadoop code and native code or you may see unusual linkage errors.
2. Build the native library yourself. This would guarantee the build matches your actual runtime OS/architecture. In addition to the documentation page linked above, there are more specific instructions on how to build in the BUILDING.txt file. Again, it's important to match the version of the Hadoop code and native code.
3. If you use a commercial vendor's distro, then they likely have taken care of guaranteeing the native library is already set up and deployed correctly.

4. If all else fails, you can stifle the warning by adding the following to log4j.properties

   log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR

org.apache.hadoop.fs.FsShell.main(FsShell.java:389) ls: Call From ubuntu/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

This is a different error, unrelated to the native code loader warning. This indicates that the client failed to connect to the NameNode during command execution. This can be caused by lack of network connectivity or simply the NameNode daemon is not running. The linked wiki page in the error message has more details on troubleshooting this.

ERROR: JAVA_HOME /opt/jdk1.8.0_91/ does not exist.

I'm not sure exactly where this message came from, but it's self-explanatory. Either check that you have Java deployed at that path, or change JAVA_HOME to the correct path.