Yarn Nodemanager not starting on windows binaries

Question

I have compiled the hadoop binaries on windows x64 machine.. I also built the native source in the path hadoop-2.6.0-src.tar\hadoop-2.6.0-src\hadoop-2.6.0-src\hadoop-common-project\hadoop-common\src\main\native

and the corresponding dll and winutils into my bin folder...

The dfs is starting fine... But nodemanager yarn component is not starting.. Its start failing with following error...

15/03/24 17:28:22 FATAL nodemanager.NodeManager: Error starting NodeManager
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
        at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
        at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:557)
        at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:977)
        at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:187)
        at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:174)
        at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:108)
        at org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.testDirs(DirectoryCollection.java:292)
        at org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection.checkDirs(DirectoryCollection.java:231)
        at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.checkDirs(LocalDirsHandlerService.java:355)
        at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.serviceInit(LocalDirsHandlerService.java:160)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService.serviceInit(NodeHealthCheckerService.java:48)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:253)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:462)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509)
15/03/24 17:28:22 INFO impl.MetricsSystemImpl: Stopping NodeManager metrics system...
15/03/24 17:28:22 INFO impl.MetricsSystemImpl: NodeManager metrics system stopped.
15/03/24 17:28:22 INFO impl.MetricsSystemImpl: NodeManager metrics system shutdown complete.
15/03/24 17:28:22 INFO nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at ing5nb014
************************************************************/

aoetalks aoetalks · Accepted Answer · 2015-03-25T01:27:12

There is native code in the hadoop-hdfs module you need to build as well. Easiest way to make sure everything is linked properly is to try building the entire Hadoop Maven project. You need to download the Hadoop source tar and run mvn clean package -Pdist,native-win -DskipTests -Dtar in the root folder of the tar, instead of just building the native modules. Assuming your build completes, in the hadoop-dist/target/hadoop-2.6.0 folder, you will have an entire Hadoop distribution with the native binaries already properly linked.

See this guide this guide I wrote recently on how to build and run Hadoop on Windows.

Yarn Nodemanager not starting on windows binaries

1 Answers