couchdb map / reduce multiple keys filtering by date

Question

I have a view setup with a map reduce. Right now this code works great:

function(doc) {
    if (doc.type == 'test'){
        if(doc.trash != 1){
                for (var id in doc.items) { 
                emit([id,doc.items[id].name], 1);
            }
        }
    }
}

function(keys,prices){
    return (keys, sum(prices));
}

I get a return and when using the group parameter, it condenses everything just fine.

My issue/question, I want to add a third key.... DATE, so I may only reduce records from certain dates. So for example:

function(doc) {
    if (doc.type == 'test'){
        if(doc.trash != 1){
                for (var id in doc.items) { 
                emit([date,id,doc.items[id].name], 1);
            }
        }
    }
}

My issue is that since date is at the beginning of the array, the reduce groups by date, id etc. I know I use group_level and say just take the first key from the array or the first 2 keys, but that doesn't help either because afaik, group_level goes from left to right in the array. I could put the date on the end of the emit array, but that doesn't help either because I need to have values at the beginning of my startkey and endkey to search on.

Here is an example of the output of data:

{"key":["2012-03-13","356752b8a5f6871f3","Apple"],"value":1},
{"key":["2012-03-20","123752b8a76986857","Pear"],"value":1},
{"key":["2012-04-12","3013531de05871194","Grapefruit"],"value":1},
{"key":["2012-04-12","356752b8a5f6871f3","Apple"],"value":1},

I want APPLE to be added up in one row, here it's adding up apples by date first. I was able to successfully just add up all the apples if I remove DATE as the first key in the array, but then I can't search by date range.

Any ideas on how to accomplish this?

wallacer wallacer · Accepted Answer · 2014-06-23T17:53:00

If I correctly understand what you want to do, then you'd want to put the date as the first element of your array, and use group_level as well as start_key and end_key.

Eg. startkey=[1, "someid"] endkey=[1,"someid",{}] group_level=2

Will get you all items from date 1 (obviously choose your own format here), with id "someid" and any name. It seems funny that you emit id's before names, and without having more information about what you're actually trying to accomplish, it's hard to advise your general data model. If ID is a "type" id meaning that many items share the same ID then this makes sense. If ID is a unique per item ID, then it does not. In that case, you'd want to emit "name" before ID...

Edit 1

As per your comment, to do a range of dates you do this:

startkey=[1] endkey=[5,{}] group_level=2

You will get everything from date 1 to date 5 grouped by id ie. apples, oranges etc. I use this exact technique in a very large scale production application. I actually formatted the dates as an easily human readable integers of the format yyyymmdd, so 20140624 would sort to the top. If I want everything from the start of the month till now grouped by my group ids, I call

startkey=[20140601] endkey=[20140624,{}] group_level=2

It works perfectly and as far as I can tell that's what you're looking to do. I also have a third key layer "detail" which allows me to provide a deeper level of grouping for items that need it. I can then call

startkey=[20140601, "someid"] endkey=[20140624, "someid",{}] group_level=3

To drill to the detail level for a particular id, or just use the previous query with group_level=3 if I want the details for every id. I'm certain you can make this work - I've solved this exact problem in a production application using the techniques described.

Edit 2

If you want to group all apples regardless of date, then you'll need to let apples be the first element in the key. You can then get all apples over all time as a single row in the view result using group_level=1, and Apples over a date range using group_level=2. The difference here is that you'll only be able to do the group_level=2 query on a single item type at a time. If you want the best of both worlds, you unfortunately just need to make 2 views. That's just how key ordering works... If you need fast response times for both types of queries, all item types over a date range, and all of a particular item not grouped by date, I believe 2 views is the only way to achieve that.

Note

Another thing to note is about your reduce function. Wherever possible it is highly recommended that you use the built in reduce functions. They're implemented in erlang and are highly optimized compared to custom javascript reduce functions.

In your case, just replace your reduce function with this

_sum

Easy hey?

If you post more info about your application, data model etc. then I'd be happy to help out more with your database design.

couchdb map / reduce multiple keys filtering by date

1 Answers