Running two mapper on a input file and a reducer for simple hadoop mapreduce jobs

Question

In a simple MapReduce program, there is one input file, that after splitting each line, it will be mapped. But now I want to read each line and after split, the line, map it two times by different keys. reading one line and two maps (two times using context.write). I read: Running two mapper and two reducer for simple hadoop mapreduce jobs That answered: So just put 2 files into your input directories so that you can get 2 mappers running. Now I should to put two same files?

I think you can write two separate mapper classes and then use MultipleInputs (available only with mapred) to pass your files to two different mappers. Haven't tried it so can't tell for sure. — philantrovert

AdamSkywalker AdamSkywalker · Accepted Answer · 2017-08-18T11:15:28

There's no point to read line in mapper, split it and pass somewhere. Process it immediately:

void map(K key, Text value, Context ctx) {
    String k1 = getKey1(value);
    String k2 = getKey2(value);
    map1(k1, value);
    map2(k2, value);
}

void map1(...) { ... }
void map2(...) { ... }

Running two mapper on a input file and a reducer for simple hadoop mapreduce jobs

1 Answers