Intermediate files in mapper (Mapreduce)

Question

so I'm new in this Hadoop world and I'm trying to understand how mappers and reducers work, my problem (and question) is:

I have a long mapper that before generating [key,value] pairs needs to generate some intermediate files. For example, in a whole mapper: files A and B as inputs, I need an output file E with some intermediate files that can't be reduced.

file A -> file C
file B -> file D
file C + file D -> file E

Is it possible to archieve this? Do intermediate files stay in the nodes?

Is necessary keep the difference between files C and D?If all the data will finish in E file, you should be emit in Map the processed lines of A and B, and then the Reducer only copy the input to output (File E). In MapReduce, you must forget the files between Map and Reduce, and think in <key,value> pairs. — Tuxman
I need to keep the same piece of file A and file B (that I have solved), cause they'll obtain the correspondent C and D needed to obtain E, the problem is that I need to do the whole thing in one same node to not mix data. Can't I execute a bunch of scripts in one node? Is mapreduce about streaming and do I have to get out and write the process in java to get out of the stdin-stdout world? (sorry if they are dumb questions, I'm just new in this :D). Thanks! — Cristian Perez
You could keep the difference using the key. For example, if you processed file A, the output of Map should be <"A",value>. You could change "A" for any coding. I don't see the need of maintain the file concept in a MapReduce application for this case. — Tuxman
@Tuxman The problem is that the programs that execute the files don't accept stdin, just files, that's because I need C and D as temp files in the nodes, not as key;value, that is my main problem. — Cristian Perez
I don't understand. Do you run a MapReduce app over Hadoop? @CristianPerez — Tuxman

Venkat Venkat · Accepted Answer · 2015-03-02T18:36:17

This cannot be achieved conventionally. But 2 approaches are possible:

From each mapper you could create & write to a file on HDFS. Take care that since based on size of input data/ no. of files there may be many mappers running in parallel so the file name would have to be unique.
Better Approach : Emit a complex key from mapper for reducer. This complex key would have 2 parts : IdentifierOfKey:Key. IdentifierOfKey is nothing but a flag saying this record need to go to file E. Now in reducer you can use multiple outputs to get data into multiple files.

Intermediate files in mapper (Mapreduce)

1 Answers