window (aka rolling) join with data.table

3
votes

I am trying to find a way around this request feature : [#2300] Add backwards and firstback to roll=TRUE which was mentioned in this post.

Basically I would like to perform the following "window-join" of X looking up in Y

  1. left join on the first n columns (in the following example {x,y})
  2. AND select values of a last column (t in following example) in Y that falls into the [t-w1,t+w2] interval where t is the last column in X, typically t would be a time column and {w1,w2} some integers (likely w1=w2=something or w1=0)

I built the following example (but feel free to provide another/better one)

library(data.table)
set.seed(123);
X <- data.table(x=c(1,1,1,2,2),y=c(T,T,F,F,F),t=as.POSIXct("08:00:00.000",format="%H:%M:%OS")+sample(0:999,5,TRUE)/1e3)
Y <- copy(X)
set.seed(123)
Y[,`:=`(IDX=.I,t=t+sample(c(-5:5)/1e3,5,T))]
Y <- rbindlist(list(Y, X[5,][,IDX:=6][,t:=t+0.001], X[5,][,IDX:=7][,t:=t+0.002]))

So with (w1,w2) = (.002,.002)

R) X                                 R) Y
   x     y                       t      x     y                       t IDX
1: 1  TRUE 2013-01-25 08:00:00.286   1: 1  TRUE 2013-01-25 08:00:00.284   1
2: 1  TRUE 2013-01-25 08:00:00.788   2: 1  TRUE 2013-01-25 08:00:00.791   2
3: 1 FALSE 2013-01-25 08:00:00.407   3: 1 FALSE 2013-01-25 08:00:00.407   3
4: 2 FALSE 2013-01-25 08:00:00.882   4: 2 FALSE 2013-01-25 08:00:00.886   4
5: 2 FALSE 2013-01-25 08:00:00.940   5: 2 FALSE 2013-01-25 08:00:00.945   5
                                     6: 2 FALSE 2013-01-25 08:00:00.941   6 #by hand
                                     7: 2 FALSE 2013-01-25 08:00:00.942   7 #by hand

The result would be

R) ans
   x     y                       t IDX
1: 1  TRUE 2013-01-25 08:00:00.286   1
2: 1  TRUE 2013-01-25 08:00:00.788  NA
3: 1 FALSE 2013-01-25 08:00:00.407   3
4: 2 FALSE 2013-01-25 08:00:00.882  NA
5: 2 FALSE 2013-01-25 08:00:00.940  6,7

But: IDX here could very well be a list if several rows of Y (which can have more rows than X) matched, one just one, or NA if none matched.

I would be happy with some non-data.table answers too...

2
rjoindata.table

2 Answers

1
votes

Here is a try, not very elegant, without data.table but with plyr. Don't know if it could be useful to you.

Sample data :

X <- data.frame(x=c(1,1,1,2,2),y=c(T,T,F,F,F),t=rep(1,5)+sample(0:999,5,TRUE)/1e3)
Y <- data.frame(x=c(1,1,1,2,2),y=c(T,T,F,F,F),t=rep(1,5)+sample(0:999,5,TRUE)/1e3, IDX=1:5)
w1 <- 0.3
w2 <- 0.3

Which gives :

R> X
  x     y     t
1 1  TRUE 1.880
2 1  TRUE 1.364
3 1 FALSE 1.288
4 2 FALSE 1.170
5 2 FALSE 1.172
R> Y
  x     y     t IDX
1 1  TRUE 1.482   1
2 1  TRUE 1.252   2
3 1 FALSE 1.216   3
4 2 FALSE 1.674   4
5 2 FALSE 1.047   5

Then you can use the following code :

m <- merge(X,Y, by=c("x","y"), all.x=TRUE, all.y=FALSE)
m <- m[m$t.x>m$t.y-w1 & m$t.x<m$t.y+w2,]
m <- ddply(m, c("x","y","t.x"), summarize, IDX=list(IDX))
names(m) <- c("x","y","t","IDX")
merge(X, m, by=c("x","y","t"), all.x=TRUE, all.y=FALSE)

Which gives the following result :

  x     y     t  IDX
1 1 FALSE 1.288    3
2 1  TRUE 1.364 1, 2
3 1  TRUE 1.880   NA
4 2 FALSE 1.170    5
5 2 FALSE 1.172    5
1
votes

Here is a translation of juba's code in data.table (v.1.8.7 r797)

setkey(X,x,y,t); setkey(Y,x,y,t)
m <- merge(X,Y, by=c("x","y"), all.x=TRUE, all.y=FALSE, allow.cartesian=TRUE)
m <- m[t.x>=(t.y-w1) & t.x<=(t.y+w2)]
m <- m[, list(IDX=list(IDX)), by=c("x","y","t.x")];
setnames(m,"t.x","t");
m <- m[X];

I get this (note the NULL, Matthew Dowle might explain why we get this instead of NA)

R) m
   x     y                       t IDX
1: 1 FALSE 2013-01-25 08:00:00.407   3
2: 1  TRUE 2013-01-25 08:00:00.286   1
3: 1  TRUE 2013-01-25 08:00:00.788
4: 2 FALSE 2013-01-25 08:00:00.882
5: 2 FALSE 2013-01-25 08:00:00.940 6,7

I'll wait for an expert to say if it is a data.table-optimal solution to close the post.