Why is the result of a reduce function fed back into reduce using mongodb mapreduce

Question

I'm seeing a perplexing behavior using mongo to perform progressive map reduce tasks. The input collection is large set of documents containing:

{_id: , url: 'some url from my swanky site'}

Here's my simple map function:

    map: function() {

        emit(this.url, {count: 1, id: this._id});

    }

And the reduce (with lots of debugging print for logs shown below):

    reduce: function (key, values) {
        var count = 0;
        var lastId = null;
        var first = null;

        if (typeof values[0].id == "undefined") {
            print("bad id");
            printjson(key);
            printjson(values[0]);
            return null;
        } else {
            print ("good id");
            printjson(key);
            printjson(values[0]);
        }

        first = ObjectId(values[0].id).getTimestamp();

        values.forEach(function(v) {

            count += v.count;

            last = ObjectId(v.id).getTimestamp();

            lastId = v.id;

        });

        return {
            count: count, 
            first: first, 
            last: lastId, 
            lastCounted: lastId
        };
    }

Here's how I call mapreduce:

        mrparams.out = {reduce: this.output};
        mrparams.limit = 100;

        mrparams.query = {'_id': {'$gt': mongoId(lastId.toHexString())}};
                    mrparams.finalize = null;


        mrdb.mapReduce(this.map, this.reduce, mrparams, function(d) {
            console.log("Finished mr", d);
            callback();
        });

This is done in a cron type manner so that every time interval, the job is run on limit number of records beginning with the record after the lastId it was run on the time before.

Very basic incremental map reduce stuff...

But, when I run it, I am seeing the return values of the reduce methond being passed back into the reduce method. Here's a snapshot of the logs:

XXXgood id "http://www.nytimes.com/2013/04/23/technology/germany-fines-google-over-data-collection.html" { "count" : 1, "id" : ObjectId("5175a065b25f029a1d0927e6") }

good id "http://www.nytimes.com/2013/04/23/world/middleeast/israel-hagel-iran.html" { "count" : 1, "id" : ObjectId("5175a065d7f115dd41097df6") }

good id "http://www.nytimes.com/interactive/2013/04/22/sports/boston-moment.html" { "count" : 1, "id" : ObjectId("5175a0657c9c963654094d25") }

YYYThu Jun 20 11:42:11 [conn19938] query vox.system.indexes query: { ns: "vox.tmp.mr.pi_analytics_spark_trending_inventories_6667_inc" } nreturned:1 reslen:131 0ms Thu Jun 20 11:42:11 [conn19938] query vox.tmp.mr.pi_analytics_spark_trending_inventories_6667 nreturned:9 reslen:1716 0ms

ZZZbad id "http://www.nytimes.com/2013/04/22/business/comedy-central-to-host-comedy-festival-on-twitter.html" { "count" : 2, "first" : ISODate("2013-04-22T20:41:11Z"), "last" : ObjectId("5175a067b25f029a1d092802"), "lastCounted" : ObjectId("5175a067b25f029a1d092802") }

bad id "http://www.nytimes.com/2013/04/22/business/media/in-boston-cnn-stumbles-in-rush-to-break-news.html" { "count" : 7, "first" : ISODate("2013-04-22T20:41:09Z"), "last" : ObjectId("5175a067d7f115dd41097e3c"), "lastCounted" : ObjectId("5175a067d7f115dd41097e3c") }

XXX - a bunch of records emitted from my map function (containing a value with count and id) YYY - some sort of mongo even that I'm not familiar with ZZZ - after the event, reduce gets called with the output of former reduce jobs...

TLDR, when I run map reduce, the reducing is going fine until a mongo process runs then I start seeing the returned values of previous reduce functions passed into my reduce function.

Any idea why/how this is possible?

Running mongo 2.0.6

Thanks in advance

David Erwin David Erwin · Accepted Answer · 2013-06-20T20:35:31

I figured out the situation. When putting the output of a map reduce job into a collection that already exists, mongo will pass both the newly reduced document and the document that was already in the output collection with the same key back through the reduce function.

This works seamlessly IF you have a consistent format for the value that you emit from map and the value that you return from reduce.

This is not well documented at all, but now that I have figured it out my frustration has transubstantiated into a feeling of smarts. Painful lesson learned. Good times ahead.

Why is the result of a reduce function fed back into reduce using mongodb mapreduce

1 Answers