Hadoop Streaming in .NET

Question

I am running hadoop in Pseudo-Distributed mode and using hadoop streaming to do my map-reduce operations. But the problem is I keep getting Streaming Job Failed error message. Following is the log:

stderr logs

java.io.IOException: Cannot run program "input/StdInOut.exe": CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:166)
at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.(ProcessImpl.java:81)
at java.lang.ProcessImpl.start(ProcessImpl.java:30)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 20 more

I know it says java.io.IOException: Cannot run program "input/StdInOut.exe": CreateProcess error=2, The system cannot find the file specified but the file is present both on the local as well as on hdfs on the same path.

Here is the implementation of my own mapper:

static void Main(string[] args)
{
  string s;
  while ((s = Console.ReadLine()) != null) 
  {
    string[] words = s.Split(' ');
    foreach (var word in words) 
    {
      //Setting occurance of each word to 1
      Console.WriteLine(word + "\t" + 1);
    }
  }
}

and I am using this command to execute my job in ssh:

bin/hadoop jar contrib/streaming/hadoop-*-streaming.jar -input input/sample.txt -output output -mapper input/StdInOut.exe -reducer NONE

Any solutions?

And if i use -file input/StdInOut.exe then the error message changes to: java.io.IOException: Cannot run program "C:\tmp\hadoop-AliAmjad\mapred\local\taskTracker\jobcache\job_201010301748_0017\attempt_201010301748_0017_m_000000_0\work\.\StdInOut.exe": CreateProcess error=216, This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information to see whether you need a x86 (32-bit) or x64 (64-bit) version of the program, and then contact the software publisher. I tried with both x86 and x64 versions of my program but no luck... — Ali

Ali Ali · Accepted Answer · 2012-01-24T12:25:51

Error was actually the relative path for the mapper.

Hadoop Streaming in .NET

1 Answers