Hadoop streaming with Python: splitting input files manually

Question

I am new to Hadoop and am trying to use its streaming feature with Python written mapper and reducer. The problem is that my original input file will contain sequences of lines which are to be identified by a mapper. If I let Hadoop split the input file, it might do it in the middle of a sequence and, thus, that sequence will not be detected. So, I was thinking about splitting the files manually. This will also break some sequences, therefore, in addition to that I would also provide and alternative split that would create files overlapping the "first" split. That way I will not loose any sequences.

I will be running the following command described in this article:

hduser@ubuntu:/usr/local/hadoop$ bin/hadoop jar contrib/streaming/hadoop-*streaming*.jar \
-file /home/hduser/mapper.py    -mapper /home/hduser/mapper.py \
-file /home/hduser/reducer.py   -reducer /home/hduser/reducer.py \
-input /user/hduser/seq_files/* -output /user/hduser/output_files

I plan to put my input files (i.e. files generated by manual splits) in /user/hduser/seq_files/

The question is:

1. How do I configure Hadoop to take each input file and send it to a mapper as it is?

2. If number of input files is greater than number of nodes, will all the files be mapped? Thanks.

Answer 1

There are a number of issues to consider here.

1. The map part of map/reduce requires that all data you need to map the line resides on the line. Trying to go around this is very bad practice and may be considered a smell that you are doing something wrong.
2. Hadoop only splits input files which are split-able, such as bz2 or uncompressed files. Gzip does not get split for instance.
3. If you are analysing sequences or "things that require a particular order to them to make sense", this is typically done in a reducer, since the data streamed to it is always sorted on the Hadoop sort key (and this is why you map the key out).
4. The reducers will receive a split dataset from the mappers after the dataset has been sorted, to avoid separation of information which all needs to go to the same reducer to be interpreted, use the Hadoop partitioning key

Note that all the links point to the same page, just different chapters. In general, I think reading that page from top to bottom will give you a much firmer notion of how to use Hadoop in a streaming fashion.

Bonus tip: If you want to do map/reduce with python, I can recommend looking at Apache Spark for Python, which runs on Hadoop, but is A whole lot faster It also lets you use the iPython console for developing your map/reduce algorithms, which increases development speed tremendously.