Hadoop Installation on Windows: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

Question

I am getting Unable to load native-hadoop library for your platform... using builtin-java classes where applicable error while trying to install the hadoop version 2.8.3 on my windows 10.

I have set below environment variable in hadoop-env.cmd file: set HADOOP_OPTS=%HADOOP_OPTS% -Djava.library.path=%HADOOP_HOME%/lib/native

also, I have added below configuration

yarn-site.xml

<configuration>
   <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
   </property>
   <property>
        <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>  
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
   </property>
</configuration>

core-site.xml:

<configuration>
   <property>
       <name>fs.defaultFS</name>
       <value>hdfs://localhost:9000</value>
   </property>
</configuration>

hdfs-site.xml:

<configuration>
   <property>
       <name>dfs.replication</name>
       <value>1</value>
   </property>
   <property>
       <name>dfs.namenode.name.dir</name>
       <value>/F:/tools/hadoop-2.8.3/data/namenode</value>
   </property>
   <property>
       <name>dfs.datanode.data.dir</name>
       <value>/F:/tools/hadoop-2.8.3/data/datanode</value>
   </property>
</configuration>

mapred-site.xml:

<configuration>
   <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
   </property>
</configuration>

I have renamed mapred-site.xml.template file to mapred-site.xml

Please let me know if I have done something wrong here

I am stuck in this installation since so long. PLease help.

None of these xml files fix that warning. What is the actual problem you're trying to solve? Hadoop should work without Native libraries, just less optimally (besides, running it on Windows already isn't optimal) — OneCricketeer

esantaremigia esantaremigia · Accepted Answer · 2018-09-08T16:20:02

I was facing the same issue with hadoop 2.9.1 on windows 10. I was able to format namenode but unable to start-dfs.

I build correctly the sources in my development host for windows platform (including winutils.exe and hadoop.dll) using visual studio 2017 community, mvn, etc, following building.txt recommendations, but when I try to run in a clean windows 10 installation I was facing the unable to load native hadoop-library error on startup of the dfs node.

I run winutils.exe and give me an error:

"Missing VCRuntime140.dll"

I installed vc++ 2015 x64 redistributables and solved the problem, namenode ans datanode up. No need to setup env variables apart from HADOOP_HOME, and follow the edits in hadoop-env.com explained in https://wiki.apache.org/hadoop/Hadoop2OnWindows.

Also, that's the results with hadoop checknative command:

C:\APACHE\HADOOP\bin>hadoop checknative
18/09/08 18:23:35 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
18/09/08 18:23:35 WARN zlib.ZlibFactory: Failed to load/initialize native-zlib library
winutils: true C:\APACHE\HADOOP\bin\winutils.exe
Native library checking:
hadoop: true C:\APACHE\HADOOP\bin\hadoop.dll
zlib: false
snappy: false
zstd : false
lz4: true revision:10301
bzip2: false
openssl: false build does not support openssl. winutils: true C:\APACHE\HADOOP\bin\winutils.exe

Hadoop Installation on Windows: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

2 Answers