Error Hadoop on Windows via Cygwin: Could Not Locate null\bin\winutils.exe

5
votes

When I set HADOOP_HOME=/cygdrive/c/ecosystem/hadoop-2.5.1 and then attempt to run bin/hadoop fs or bin/hadoop hadoop-streaming.jar from Cygwin, I receive the following error:

    ERROR [main] util.Shell (Shell.java:getWinUtilsPath(373)) - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)
        at org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:432)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:478)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
Exception in thread "main" java.lang.RuntimeException: core-site.xml not found
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2269)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2195)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2112)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:989)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:961)
        at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1299)
        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:319)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:479)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

I can access bin/hadoop version along with classpath and most other commands. I thought that maybe I needed a Windows style path so that it could conform to the \bin\winutils style. So, I set HADOOP_HOME=c:\ecosystem\hadoop-2.5.1 and this error went away. However, now I receive a link error when running in Cygwin. I tested hadoop with HADOOP_HOME=c:\ecosystem\hadoop-2.5.1 via Windows Command, and I can run a MapReduce job without issue.

I want to successfully run hadoop in Cygwin, and I don't want to deal with link errors. If I keep HADOOP_HOME=/cygdrive/c/ecosystem/hadoop-2.5.1, is there any way that I can set hadoop-config or some other file to look for the Windows style path if the Cygwin path fails or is null? Maybe something like this in hadoop-config:

if ["$HADOOP_HOME" = ""]; then
  HADOOP_HOME=$(cygpath -w "$HADOOP_HOME")
fi

Obviously this particular code didn't work in hadoop-config, but I suppose you'll get the general idea. Any help would be appreciated.

4
windowshadoop
Any reasons why you're using cygmin ? Hadoop2 is usable on windows (see wiki.apache.org/hadoop/Hadoop2OnWindows ) – merours
Hadoop, alone, worked fine from windows command. However, I wanted to test an integration with hive, as well as a few database frameworks. Since those frameworks are not really made for Windows, I wanted to leverage cygwin. I ended up just building a linux box, and then constructing my ecosystem there. Worked perfectly. – datasci

4 Answers

7
votes

I solved a similar error with Eclipse when I was trying to execute Hadoop MR program locally in Windows by setting the environment variable HADOOP_HOME through Eclipse: Run->Run Configurations->Environment.

2
votes

Since I was able to successfully operate with Hadoop via the Windows command, it didn't make much sense to waste time trying to figure out how to make it work with cygwin. Instead, for my greater Hadoop ecosystem, I simply built it on linux. It's super easy to use, and is by far the best decision I've made.

0
votes

Try editing hadoop/ hdfs/ yarn shell scripts add -Dhadoop.home.dir= right at the place where JAVA command is invoked.

You might have to add in these shell scripts if ["$HADOOP_HOME" = ""]; then HADOOP_HOME=$(cygpath -w "$HADOOP_HOME") fi

Worst case, you already have src access edit it compile it and set the home variable there compile it and add it to jar.

0
votes

ref. https://issues.apache.org/jira/browse/HADOOP-10133

HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.home.dir="$(cygpath -pw "$HADOOP_PREFIX")""

Based on this, I added the following line to my .zshrc (or .bashrc for you):

export HADOOP_PREFIX=`cygpath -w $HADOOP_HOME`

If you have not set $HADOOP_HOME, you should.