dc.js and crossfilter custom valueAccesor

Question

I am attempting to chart average counts by hour and the custom reduce function is almost working here https://jsfiddle.net/dolomite/6eeahs6z/

There is an issue in that some hours have no activity, e.g. there may be three Sundays in the data but only two have activity:

Date, Hour, Count

Sun 02/07/17, 22, 5

Sun 09/07/17, 22, 3

The data contains the date 25/07/17 but has no records for hour 22. The correct average for hour 22 on Sunday should therefore be 2.66 but the current method is producing an average of 4.

So in short I'm trying to work out how to get total counts per hour and then divide by the number of days in the data, whether or not the selected day has a record for each hour.

The current hour dimension and custom reduce is:

hourDim = ndx.dimension(function (d) {
    return d.EventHour;
})

hourAvgGroup = hourDim.group().reduce(
            function (p, v) { // add
                var day = d3.time.day(v.EventDate).getTime();
                p.map.set(day, p.map.has(day) ? p.map.get(day) + 1 : 1);
                //p.avg = average_map(p.map);
                return p;
            },
            function (p, v) { // remove
                var day = d3.time.day(v.EventDate).getTime();
                p.map.set(day, p.map.has(day) ? p.map.get(day) - 1 : 0);
                if(p.map.has(day) && p.map.get(day) == 0) p.map.remove(day);
                //p.avg = average_map(p.map);
                return p;
            },
            function () { // init
                return { map: d3.map() };
            }
        )

The average is computed in the chart valueAccessor as follows:

.valueAccessor(function(d){ return average_map(d.value.map)})

Where

function average_map(m) {
var sum = 0;
m.forEach(function(k, v) {
    sum += v;
});
return m.size() ? sum / m.size() : 0;
}

Do you want count the day in the denominator if there is a record for that hour/day but it's not in the current filter? Or do you want to count the day in the denominator if there is any record with that day (in the current filter?), even if it's a different hour? — Ethan Jewett
Sorry I really struggled to explain this one! I need to count the day in the denominator if there is any record with that day in the current filter. Simplest example would be 1 item in hour 22 on a sunday. there are two sundays in the data, the other sunday doesn't have anything in hour 22, so the average should be 0.5. — Dolomite
From your examples, it sounds like you are trying to average per day of week as well as per hour (i.e. 7 * 24 bins total), but you don't say that in your text. Is that what you mean? — Gordon
Given that there are multiple time-based filters, you'd either have to calculate how many bins there "should be" based on the current filters and the data set, or you'd have to ensure there is a record for every hour of every day. I think it's easier to do the latter. — Gordon
You've just got the forEach function structure a bit wrong. Try myObject.forEach(function(elem) { count += elem.value.map.size(); }) — Ethan Jewett

Dolomite Dolomite · Accepted Answer · 2017-07-12T15:23:54

In case anyone is trying to do similar, I created a dimension to hold all records in the data:

allDim = ndx.dimension(function (d) {
        return typeof Modality === 'string';
})

Then created a group to hold a map of the number of unique days in the data:

 dayCountGroup = allDim.group().reduce(
     function (p, v) { // add
                var day = d3.time.day(v.EventDate).getTime();
                p.map.set(day, p.map.has(day) ? p.map.get(day) + 1 : 1);
                return p;
            },
            function (p, v) { // remove
                var day = d3.time.day(v.EventDate).getTime();
                p.map.set(day, p.map.has(day) ? p.map.get(day) - 1 : 0);
                if(p.map.has(day) && p.map.get(day) == 0) p.map.remove(day);
                return p;
            },
            function () { // init
                return { map: d3.map() };
            }
        )

The hour dimension and group are:

hourDim = ndx.dimension(function (d) {
    return d.EventHour;
})

hourAvgGroup = hourDim.group().reduce(
            function (p, v) { // add
                var day = d3.time.day(v.EventDate).getTime();
                p.map.set(day, p.map.has(day) ? p.map.get(day) + 1 : 1);
                return p;
            },
            function (p, v) { // remove
                var day = d3.time.day(v.EventDate).getTime();
                p.map.set(day, p.map.has(day) ? p.map.get(day) - 1 : 0);
                if(p.map.has(day) && p.map.get(day) == 0) p.map.remove(day);

                return p;
            },
            function () { // init
                return { map: d3.map() };
            }
        )

Then in the value accessor for the barchart i used:

.valueAccessor(function(d){ return sum_map(d.value.map)/size_array_of_maps(dayCountGroup.top(Infinity)) ? sum_map(d.value.map)/size_array_of_maps(dayCountGroup.top(Infinity)) : 0})

Where the two functions used are:

function sum_map(m) {
var sum = 0;
m.forEach(function(k, v) {
    sum += v;
});
return m.size() ? sum : 0;
}

function size_array_of_maps(myObject) {
    var count = 0;

    myObject.forEach(function(key,value) {
            count += key.value.map.size();
    })
   return count;
}

I'm sure there is a lot of redundant code here but the Fiddle seems to be working and I'll tidy it up later :)

https://jsfiddle.net/dolomite/6eeahs6z/126/

dc.js and crossfilter custom valueAccesor

1 Answers