Amazon MapReduce with my own reducer for streaming

Question

I wrote a simple map and reduce program in python to count the numbers for each sentence, and then group the same number together. i.e suppose sentence 1 has 10 words, sentence 2 has 17 words and sentence 3 has 10 words. The final result will be:

10 \t  2
17 \t  1

The mapper function is:

    import sys
    import re

    pattern = re.compile("[a-zA-Z][a-zA-Z0-9]*")
    for line in sys.stdin:

        word = str(len(line.split()))  # calculate how many words for each line
        count = str(1)
        print "%s\t%s" % (word, count)

The reducer function is:

    import sys


    current_word = None
    current_count = 0
    word = None

    for line in sys.stdin:
        line = line.strip()
        word, count = line.split('\t')
        try:
            count = int(count)
            word = int(word)
        except ValueError:
            continue
        if current_word == word:
            current_count += count
        else:
            if current_word:
                print "%s\t%s" % (current_word, current_count)
            current_count = count
            current_word = word

    if current_word == word:
        print "%s\t%s" %(current_word, current_count)

I tested on my local machine with the first 200 lines of the file : head -n 200 sentences.txt | python mapper.py | sort | python reducer.py The results are correct. Then I used Amazon MapReduce streaming service, it failed at the reducer step. So I changed the print in the mapper function to:

   print "LongValueSum" + word + "\t" + "1"

This fits into the default aggregate in mapreduce streaming service. In this case, I don't need the reducer.py function. I get the final results from the big file sentences.txt. But I don't know why my reducer.py function failed. Thank you!

You may want to check out mrjob: pythonhosted.org/mrjob It's a really convenient way to write MapReduce jobs in Python. Jobs can be developed against a small sample dataset locally, and can then be scaled-up to run against larger datasets using Amazon's Elastic-Mapreduce with a slight tweak to the command-line. — Alex Woolford

ohmygoddess ohmygoddess · Accepted Answer · 2014-10-27T05:12:28

Got it! A "stupid" mistake. When I tested it, I use something like python mapper.py. But for mapreduce, I need make it executable. So just add

    # !/usr/bin/env python

in the beginning.

Amazon MapReduce with my own reducer for streaming

1 Answers