I have such problem with couchbase design: Query for view returns 40000 records. I need to add additional filter on them and get "top 100 order by".
I made such filtration/sorting in my application code this means I must fetch 40000 records from Couchbase (and this is time-consuming). Is there a way to execute filter/sort onto couchbase node (without fetching whole 4000 records to appserver)?
My data is touristic tours ie
{
"OT": "tour",
"dd": 20140720,
"city": 1206,
"hotel": 9656,
"stars": 2,
"resort": 23415,
"country": 34,
"price": 24139,
"priceType": 1,
"tickets": "QQYY",
"nights": 5,
"food": 4,
"oper": 18,
"adult": 1,
"ch": 0,
"ch1": 0,
"ch2": 0,
"ch3": 0,
"avail": 1,
"stop": "Q"
}
and i need to select top 10 cheapest tours from London to turkey between 20140622 and 20140710 in 4 or 5 stars hotel...
My view looks like:
function (doc, meta) {
if(meta.type==='json' && doc.OT==='tour'){
emit(["A",doc.country,doc.city,doc.adult,doc.ch,doc.priceType,doc.dd,doc.price]);
emit(["R" + doc.resort,doc.country,doc.city,doc.adult,doc.ch,doc.priceType,doc.dd,doc.price]);
}
}
RESORT+COUNTRY+DEPARTURE_CITY+ADULT_COUNT+CHILD_COUNT+PRICE_TYPE+DEPARTURE_DATE allows me to select ~ 40000 records from 3000000+ (for DEPARTURE_DATE range) but sometimes (according user input) I still need to filter out them by Stars (stars IN (4,5) for example). Also view sorting is "BY Date,Price" this is'nt applicable, for example I got
20140101 110$ <- top but not cheapest
20140101 120$
20140102 100$ <- cheapest but not top
20140102 105$
At other side sometimes I also need to fetch N cheapest tours where DepartureDate BETWEEN x and Y and Price BETWEEN A and Z. All those scenarios require additional filters to filter out from HUGE dataset (even high-selective VIEW as shown above still produce HUGE dataset in my case) and I do not want fetch whole dataset to client (AppServer) for such processing... I realize processing on Couchbase nodes will consume more CPU on them but I prefer to add more Couchbase nodes to cluster... Anyway someone need to do this filtering work, I believe it's much more optimal to do it where data actually placed without additional network overhead...
