D3js: How to interpolate datasets with irregular time intervals before stacking them?

Question

UPDATE: The jsfiddle example illustrates the problem: http://jsfiddle.net/T8SuH/

I am new to both D3js and JavaScript but comfortable with Java. I have a large CSV dataset from a bike-sharing system organized as follows:

event_id, station_id, timestamp, number_of_bikes, number_of_empty_docks

To reduce file size, my data file contains only new events for individual stations. Therefore timestamps of different stations are neither in regular time intervals nor they coincide. For example consecutive entries might look like this:

31044955, 08, 2012-05-01 00:00:00, 9, 6
31044965, 23, 2012-05-01 00:32:00, 2, 6
31044972, 02, 2012-05-01 00:44:00, 10, 5
31044999, 17, 2012-05-01 02:10:10, 12, 6
...

I want to create a stacked area chart in which the x axis is time (24hrs) and the y axis is (stacked) inventories from selected stations. D3 Wiki mentions that data should be interpolated before computing the stack. I consider two ways of doing this: - Interpolate my dataset outside D3 (using Java) and save it as a new CSV file in which all stations have concurrent timestamps. This method would increase my file size more than 6 times (currently it is 25MB) and I fear that D3 would be then slow in processing it. - Interpolate data inside D3/Javascript before doing the stack or plotting the area/path.

While I feel the second way would be faster (in terms of performance), I do not know how to do this and I haven't found any related tutorials/examples. I only guess that I would have to create a range (d3.time.minute.range) and then use it with a map() function to create a new array? Could someone please provide a suggestion of how to interpolate multiple datasets (e.g. for each station) with irregular time intervals before stacking them, inside D3?

Thank you in advance.

Answer 1

I ran into the same problem once before. With a little bit of delicacy you can add in the missing values so that every series has the same set of x-coordinates (timestamps). I added a few lines to you jsfiddle to massage the data before you send it into the stack layout.

To start with I needed the super-set of times:

var times = [];
myData.forEach(function(d) {
  if(times.indexOf(d.date.getTime()) === -1) {
    times.push(d.date.getTime());
  }
});
times.sort();

This results in the array (from your data):

0:00, 4:00, 4:30, 4:45, 8:08, 12:08, 13:08, 16:08, 20:08

Note that at the end I sort it because this will really simplify finding the missing values in the station data series. I intend to do a kind of sorted list merge to fill in no-change data values into the series. This is what your series look like straight from the data (time value pairs):

Initial series
Station_1 0:00 37, 4:30 36, 8:08 18, 12:08 10, 16:08 32, 20:08 35
Station_2 0:00 32, 4:00 29, 8:08 6,  12:08 12, 16:08 29, 20:08 32
Station_3 0:00 36, 4:45 30, 8:08 10, 13:08 13, 16:08 32, 20:08 36
Station_4 0:00 31, 4:00 28, 8:08 12, 12:08 14, 16:08 28, 20:08 31

So for each series, walk through the time array and insert missing values, there may well be a more efficient way to do this:

myNestedData.forEach(function(stationData) {
  stationData.values.sort(function(a,b) {
      return d3.ascending(a.date.getTime(), b.date.getTime());
    });
  var j = 0;
  var lastStationValue = 0;
  for(var i = 0; i < times.length; i++) {
    // If the station series is too short I should not equal any 
    // value from the times array
    stationTime = j <= stationData.values.length ?
        stationData.values[j].date.getTime() : 0;

    if(times[i] !== stationTime) {
      // Missing values need to be spliced in.
      stationData.values.splice(j, 0, { 
          date: new Date(times[i]),
          inventory: lastStationValue,
          station_id: stationData.key
      });
      j++;
    }
    else {
      // We have a value for this time, move along.
      lastStationValue = stationData.values[j].inventory;
      j++;
    }
  }                       
});

And now the series should all line up!

Station_1 0:00 37, 4:00 37, 4:30 36, 4:45 36, 8:08 18, 12:08 10, 13:08 10, 16:08 32, 20:08 35
Station_2 0:00 32, 4:00 29, 4:30 29, 4:45 29, 8:08 6,  12:08 12, 13:08 12, 16:08 29, 20:08 32
Station_3 0:00 36, 4:00 36, 4:30 36, 4:45 30, 8:08 10, 12:08 10, 13:08 13, 16:08 32, 20:08 36
Station_4 0:00 31, 4:00 28, 4:30 28, 4:45 28, 8:08 12, 12:08 14, 13:08 14, 16:08 28, 20:08 31 

You can check out the updated fiddle here