Hadoop Streaming Never Finishes

Question

I am trying to learn how to use hadoop streaming. I'm trying to run a very simple mapper, and no reducers. When I run the program, it finishes 100% of the map tasks, then does nothing for ten minutes, and then reports that it has finished 0% of all map tasks. I think that means the node manager had to kill off the job, not sure. I have waited up to a half hour in the past and it never finishes.

I am using hadoop 1.2.1. It's documentation says it comes with the hadoop streaming jar, but I could not find it, so I downloaded hadoop-streaming-1.2.1 from the central maven repository. Here is the command line:

[msknapp@localhost data]$ hadoop jar /hadoop/hadoop-streaming-1.2.1.jar -D mapred.reduce.tasks=0 -input /stock -output /company_index -mapper /home/msknapp/workspace/stock/stock.mr/scripts/firstLetterMapper.py -reducer org.apache.hadoop.mapred.lib.IdentityReducer
packageJobJar: [] [/opt/hadoop-1.2.1/hadoop-streaming-1.2.1.jar] /tmp/streamjob7222367580107633928.jar tmpDir=null
13/12/22 07:04:14 WARN snappy.LoadSnappy: Snappy native library is available
13/12/22 07:04:14 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/12/22 07:04:14 INFO snappy.LoadSnappy: Snappy native library loaded
13/12/22 07:04:14 INFO mapred.FileInputFormat: Total input paths to process : 1
13/12/22 07:04:17 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-msknapp/mapred/local]
13/12/22 07:04:17 INFO streaming.StreamJob: Running job: job_201312201826_0009
13/12/22 07:04:17 INFO streaming.StreamJob: To kill this job, run:
13/12/22 07:04:17 INFO streaming.StreamJob: UNDEF/bin/hadoop job  -Dmapred.job.tracker=localhost:9001 -kill job_201312201826_0009
13/12/22 07:04:17 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201312201826_0009
13/12/22 07:04:18 INFO streaming.StreamJob:  map 0%  reduce 0%
13/12/22 07:04:44 INFO streaming.StreamJob:  map 100%  reduce 0%
13/12/22 07:14:44 INFO streaming.StreamJob:  map 0%  reduce 0%
13/12/22 07:15:09 INFO streaming.StreamJob:  map 100%  reduce 0%

The python script I call is very simple. I have python 2.6.6 installed. The script works when I test it:

#!/usr/bin/env
import sys
import string
#import os

def map(instream=sys.stdin,outstream=sys.stdout):
    line = instream.readline()
    output=map_line(line)
    outstream.write(output)

def map_line(line):
    parts=string.split(line,"\t")
    key=parts[0]
    newkey=key[0]
    newvalue=key
    output=newkey+"\t"+newvalue
    return output

map()

the input file is rather short and simple, it has tab delimited lines like "GE\tGeneral Electric", and I am sure they are tabs.

BTW I'm running hadoop 1.2.1 in pseudo distributed mode on CentOS 1.6, on a VMWare virtual machine.

Would somebody please explain to me why this is not working and what I can do to fix it?

Answer 1

I just got this error on a similar(simple) problem. For me, the error was due to the python script dying during execution. Similar to your problem, my script seemed to work just fine for a small subset of the problem, but just wouldn't work on Hadoop for the entire dataset, and that was due to flawed input. So, although it might not be why your script is dying, but you should probably add some sanity checks.


Check if the length of parts is what you expect it to be.

Check if parts is empty.

Also, you can go to the job tracker and see the exact error that caused Hadoop to stop execution. This will not give you the python stacktrace that you might be expecting, but is still helpful. The job tracker can usually be found at

http:// localhost : 50030/jobtracker.jsp

Also, change

#!/usr/bin/env

to

#!/usr/bin/python

This was because the machine running your script does not know what to do with it. It would probably just cause your computer to freeze up as well if you ran it with ./firstLetterMapper.py instead of python firstLetterMapper.py

Answer 2

The hadoop-streaming-x.y.z.jar should be in your $HADOOP_HOME which wasn't defined for me but should have been at /usr/lib/hadoop.

I think the Hadoop Streaming doc is pretty helpful about your issues with a python example.

First, your mapper nodes need a copy of the python file you wrote, so on the command line list it with the file option.
Second, if you're not using a reducer you don't need to define it.

$ hadoop jar /hadoop/hadoop-streaming-1.2.1.jar \
  -D mapred.reduce.tasks=0 \
  -input /stock -output /company_index \
  -mapper firstLetterMapper.py \
  -file /home/msknapp/workspace/stock/stock.mr/scripts/firstLetterMapper.py

Third your shabang would just run env on the file, you should change it to #!/usr/bin/python or #!/usr/bin/env python

That's probably what's causing env to give a non-zero exit value and therefore your mapper, which ran for ~30 seconds, retries with attempt 2, about 10 minutes later.