Interpolating data in a data.table object inside a function

Question

I'm currently trying to transform a script which performs an interpolation in a data.table object (by group) into a function.

I have data in long format similar to this example:

   UniqueId  Time   Var1      Var2    Var3
1       Id1     0     10   "cat-a"       2
2       Id1     5     12   "cat-a"       4
3       Id1    10     14   "cat-a"       6
4       Id2     0      8   "cat-b"       3
5       Id2    10     16   "cat-b"      10
6       Id3     0      6   "cat-a"       4
7       Id3    10      9   "cat-a"       8
[...]

As you can notice, the number of entries for each UniqueId can differ. What I want to do is to extract all unique values of the "Time" column, and make sure that all UniqueIds have an entry for each one of them, by performing a linear interpolation for numeric variables while for factors or strings, as they do not change over time, I take the first value for each UniqueId and replicate it n times (where n = length(unique(Time)), resulting in something like this:

   UniqueId  Time   Var1      Var2    Var3
1       Id1     0     10   "cat-a"       2
2       Id1     5     12   "cat-a"       4
3       Id1    10     14   "cat-a"       6
4       Id2     0      8   "cat-b"       3
5       Id2     5     12   "cat-b"     6.5
6       Id2    10     16   "cat-b"      10
7       Id3     0      6   "cat-a"       4
8       Id3     5    7.5   "cat-a"       6
9       Id3    10      9   "cat-a"       8
[...]

As the data I'm working on can be rather large (several Go's), I decided to use the data.table package. My current code looks like this:

Time.Common <- unique(MyData[, Time])
Time.Common <- Time.Common[order(Time.Common)]


MyData[, mapply(function(colVals, colName) {
  if(colName == "Time"){ # If we are operating on the time column, we replace its values with those of Time.Common
    return(Time.Common)
  }
  if(is.factor(colVals) | is.character(colVals)){ # If we are working on character or factor columns, we just repeat the value "length(Time.Common)" times
    rep(colVals[1], length(Time.Common))}
  else { # If it's a numeric column, we perform an interpolation
    withCallingHandlers(approx(x      = .SD[,Time],
                               y      = colVals,
                               xout   = Time.Common)$y,
                        warning = function(w){invokeRestart("muffleWarning")})
  }
}, .SD, names(.SD), SIMPLIFY = FALSE), by = eval(expr = "UniqueId")] # The process is repeated for each unique key value

This code produces the expected results. The problem is that when I try to convert into a function, I can't make it work anymore, and as I'm not an expert of data.table, I don't understand where is the issue.

Here is the function:

interpolation <- function(Data, 
                          UniqueKey     = "UniqueId",
                          TimeCol       = "Time",
                          InterMethod   = "linear"){
  # Recovering all unique values of the time column
  Time.Common <- unique(Data[, get(TimeCol, inherits = F)])
  Time.Common <- Time.Common[order(Time.Common)]
  
  # Interpolating
  Data[, mapply(function(colVals, colName) {
    if(colName == TimeCol){ 
      return(Time.Common)
    }
    if(is.factor(colVals) | is.character(colVals)){
      rep(colVals[1], length(Time.Common))}
    else {
      withCallingHandlers(approx(x      = .SD[,eval(parse(text = TimeCol))],
                                 y      = colVals,
                                 xout   = Time.Common,
                                 method = InterMethod)$y,
                          warning = function(w){invokeRestart("muffleWarning")})
    }
  }, .SDcols = .SD, names(.SD), SIMPLIFY = FALSE), by = eval(parse(text = UniqueKey))]
}

It's pretty much the same code, except for the part where I pass column names as variables, where I have to use eval(parse()) or get(). This function does not produce errors when I use it, but does not perform an interpolation neither (my data remains unchanged). Could someone help me on this one and help me find my mistake ?

Thanks in advance !

pseudospin pseudospin · Accepted Answer · 2020-08-05T23:05:57

I'd do the interpolation in data.table like this

MyData[,c(
  list(Time = Time.Common),
  lapply(.SD, function(col) approx(x = Time, y = col, xout = Time.Common)$y)
  ), by = c('UniqueId','Var2'), .SDcols = c('Var1','Var3')]

and maybe use lapply(MyData,typeof) to work out which columns I was going to interpolate (in .SDcols =) and which were in the key (to go in the by =)

Edited to add a generic function to do this, you can use other methods to choose which columns are the keys and which are to be interpolated, but ultimately you can just pass charcter vectors in the by and .SDcols arguments of data.table.

interpolation <- function(mydata, time_col) {
  col_types <- lapply(mydata[,-time_col,with=FALSE],typeof)
  interp_cols <- names(col_types)[col_types %in% c('double', 'integer')]
  key_cols <- names(col_types)[!(col_types %in% c('double', 'integer'))]
  mydata[,c(
    list(Time = Time.Common),
    lapply(.SD, function(col) approx(x = get(time_col), y = col, xout = Time.Common)$y)
  ),by=key_cols, .SDcols = interp_cols]
}
interpolation(MyData, "Time")

Interpolating data in a data.table object inside a function

1 Answers