reducer just wont start in hadoop streaming

Question

I am not sure whats happening, but I wrote a simple mapper and reducer script.

And I am testing it against a small dataset (like few lines long).

For some reason reducer is just not starting.. and mapper is executing again and again?

12/11/20 09:21:18 INFO streaming.StreamJob:  map 0%  reduce 0%
12/11/20 09:22:05 INFO streaming.StreamJob:  map 50%  reduce 0%
12/11/20 09:22:10 INFO streaming.StreamJob:  map 100%  reduce 0%
12/11/20 09:32:05 INFO streaming.StreamJob:  map 50%  reduce 0%
12/11/20 09:32:11 INFO streaming.StreamJob:  map 0%  reduce 0%
12/11/20 09:32:20 INFO streaming.StreamJob:  map 50%  reduce 0%
12/11/20 09:32:31 INFO streaming.StreamJob:  map 100%  reduce 0%
12/11/20 09:42:20 INFO streaming.StreamJob:  map 50%  reduce 0%
12/11/20 09:42:31 INFO streaming.StreamJob:  map 0%  reduce 0%
12/11/20 09:42:32 INFO streaming.StreamJob:  map 50%  reduce 0%
12/11/20 09:42:50 INFO streaming.StreamJob:  map 100%  reduce 0%

Let me know if you want the code also. Any clues of where I am going wrong? Thanks

frazman frazman · Accepted Answer · 2012-11-20T17:01:58

Ok. My bad..

The error was in my shebang, I just had

#!/usr/bin/env

instead of

 #!/usr/bin/env  python

That did the trick :)

Though would be very interested in knowing why did mapper was running again and again if I skipped python from the shebang..

reducer just wont start in hadoop streaming

1 Answers