How to apply brushing on a dynamically growing dataset?

Question

I have a dynamically growing timeseries I need to display in a zoomable/panable chart.

Try it out here (in fact: my first jsFiddle ever :) ) :

https://jsfiddle.net/Herkules001/L12k5zwx/29/

I tried to do it the same way as described here: https://dc-js.github.io/dc.js/examples/replacing-data.html

However, each time the chart updates, the zoom and filter are lost on the focus chart. (The brush is preserved on the range chart however.)

How can I add data without resetting the views and losing the zoom?

var chart = dc.lineChart("#test");
var zoom = dc.lineChart("#zoom");

//d3.csv("morley.csv", function(error, experiments) {
var experiments = d3.csvParse(d3.select('pre#data').text());
  experiments.forEach(function(x) {
    x.Speed = +x.Speed;
  });

  var ndx                 = crossfilter(experiments),
      runDimension        = ndx.dimension(function(d) {return +d.Run;}),
      speedSumGroup       = runDimension.group().reduceSum(function(d) {return d.Speed * d.Run / 1000;});

  chart
    .width(768)
    .height(400)
    .x(d3.scaleLinear().domain([6,20]))
    .brushOn(false)
    .yAxisLabel("This is the Y Axis!")
    .dimension(runDimension)
    .group(speedSumGroup)
    .rangeChart(zoom);

  zoom
    .width(768)
    .height(80)
    .x(d3.scaleLinear().domain([6,20]))
    .brushOn(true)
    .yAxisLabel("")
    .dimension(runDimension)
    .group(speedSumGroup);

    zoom.render();
    chart.render();

  var run = 21;
  setInterval(
    () => {

      var chartfilter = chart.filters();
      var zoomfilter = zoom.filters();

      chart.filter(null);
      zoom.filter(null);

      ndx.add([{Expt: 6, Run: run++, Speed: 100 + 5 * run}]);
      chart.x(d3.scaleLinear().domain([6,run]));
      zoom.x(d3.scaleLinear().domain([6,run]));

      chart.filter([chartfilter]);
      zoom.filter([zoomfilter]);

      chart.render();
      zoom.render();
    },
    1000);  

//});

Answer 1

In this case, if you are just adding data, you don't need to do the complicated clearing and restoring of filters demonstrated in the example you cited.

That part is only necessary because crossfilter.remove() originally would remove the data that matched the current filters. An awkward interface, almost never what you want.

If you're only adding data, you don't have to worry about any of that:

setInterval(
() => {
  ndx.add([{Expt: 6, Run: run++, Speed: 5000 + 5 * run}]);
  chart.redraw();
  zoom.redraw();
},
5000);  

Note that you'll get less flicker, and decent animated transitions, by using redraw instead of render. I also added evadeDomainFilter to avoid lines being clipped before the edge of the chart.

Fork of your fiddle

Removing data

If you use the predicate form of crossfilter.remove() you don't have to worry about saving and restoring filters:

  ndx.remove(d => d.Run < run-20);

However, this does expose other bugs in dc.js. Seems like elasticY does not work, similar to what's described in this issue. And you get some weird animations.

Here's a demo with remove enabled.

In the end, dc.js has some pretty neat features, and there is usually a way to get it to do what you want, but it sure is quirky. It's a very complicated domain and in my experience you are going to find some of these quirks in any fully featured charting library.

Update: I fixed the replacing data example, that one is just ndx.remove(() => true) now.

zooming issues

As Joerg pointed out in the comments,

• when the chart is not zoomed, it would be nice to have it also grow to show new data as it arrives
• the X domain was clipped or even reversed if the focus reached outside the original domain of the chart

We can address these issues by adding a preRedraw event handler. That's the ideal place to adjust the domain; for example you can implement elasticX manually if you need to. (As you'll see in a second, we do!)

First, a naive attempt that's easy to understand:

chart.on('preRedraw', () => {
  chart.elasticX(!zoom.filters().length);
});

We can turn elasticX on and off based on whether the range chart has an active filter.

This works and it's nice and simple, but why does the chart get so confused when you try to focus on a domain that wasn't in the original chart?

Welp, it records the original domain (source). So that it can restore to that domain if the focus is cleared, and also to stop you from zooming or panning past the edge of the graph.

But notice from the source link above that we have an escape hatch. It records the original domain when the X scale is set. So, instead of setting elasticX, we can calculate the extent of the data, set the domain of the scale, and tell the chart that the scale is new:

chart.on('preRedraw', () => {
  if(!zoom.filters().length) {
    var xExtent = d3.extent(speedSumGroup.all(), kv => kv.key);
    chart.x(chart.x().domain(xExtent));
  }
});

New fiddle with zooming issues fixed.

There is still one glitch which Joerg points out: if you are moving the brush while data comes in, the brush handles occasionally will occasionally stray from the ends of the brush. In my experience, these kinds of glitches are pretty common in D3 (and dynamic charting in general), because it's difficult to think about data changing during user interaction. It probably could be fixed inside the library (perhaps an interrupted transition?) but I'm not going to get into that here.