1
votes

I have a huge data.table dt (almost 1.5 million rows) let say i want to apply a user defined function growth.ls to its rows, where scols (some columns in dt) are the arguments as

growth.ls <- function(values){
  if (any(!is.finite(values)) || any(values <= 0)) return(NA_real_)
  exp(lm(log(values) ~ (seq_along(values)))$coefficients[[2]] - 1) * 100}
dt[, `:=`(var = growth.ls(as.numeric(.SD))), .SDcols = scols, by = 1:nrow(dt)]

this process takes a very long time, I do not know if the problem is the growth.ls, or i am because i am using by: 1:nrow(dt).

1
Yeah, that is not a good way to use a data.frame or data.table -- you are splitting up into a large number of rows, then coercing each row from a data.table to a numeric vector. Try using melt to put your data in long form instead maybe - Frank
you can probably speed things up a lot by using lm.fit; pre-allocating your model matrix (X <- cbind(1,1:ncol(dt))); and possibly even computing the regression slope directly - you can precompute everything except sum(y) and sum(x*y) ... - Ben Bolker
Is it possible to provide an example so i could try it. - Mohamed
not right now. maybe someone else will come along and provide one. - Ben Bolker

1 Answers

1
votes

What about this (using multicores with data.table):

library(parallel)
cl = makeCluster(detectCores())
choose_cols = startsWith(colnames(df),'x')

df[,growth := unlist(parApply(cl, .SD, 1, growth.ls), .SDcols = choose_cols]