Hadoop HDFS MapReduce output into MongoDb

Question

I want to write Java program which reads input from HDFS, processes it using MapReduce and writes the output into a MongoDb.

Here is the scenario:

I have a Hadoop Cluster which has 3 datanodes.
A java program reads the input from the HDFS, processes it using MapReduce.
Finally, write the result into a MongoDb.

Actually, reading from HDFS and processing it with MapReduce are simple. But I gets stuck about writing the result into a MongoDb. Is there any Java API supported to write the result into MongoDB? Another question is that since it is a Hadoop Cluster, so we don't know which datanode will run the Reducer task and generate the result, is it possible to write the result into a MongoDb which is installed on a specific server?

If I want to write the result into HDFS, the code will be like this:

@Override
public void reduce(Text key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException 
{
    long sum = 0;
    for (LongWritable value : values) 
    {
        sum += value.get();
    }

    context.write(new Text(key), new LongWritable(sum));
}

Now I want to write the result into a MongoDb instead of HDFS, how can I do that?

What do you mean, you get stuck about writing the result into MongoDB? MongoDB has a Java API available (docs.mongodb.org/ecosystem/drivers/java). — WiredPrairie
Why not try it? (And why are you copying data from HDFS into MongoDB?) — WiredPrairie
hey @user2597504, could you tell me how it did work(or how you did solve this issue)? your question is exactly the same that I was curious about. — Hoon

Simplefish Simplefish · Accepted Answer · 2013-12-30T07:03:12

You want «MongoDB Connector for Hadoop». The examples.

It's tempting to just add code in your Reducer that, as a side effect, inserts data into your database. Avoid this temptation. One reason to use a connector as opposed to just inserting data as a side effect of your reducer class is speculative execution: Hadoop can sometimes run two of the exact same reduce tasks in parallel, which can lead to extraneous inserts and duplicate data.

Hadoop HDFS MapReduce output into MongoDb

3 Answers