Use rollapplyr in parallel computing

Question

I need to do a 36-month rolling regression for each column and get the intercept for each regression. There are about 100 rows and 9000 columns. The original code took 10 hours to run in my laptop. I want to use parallel computing to reduce running time, but it returns error. Below is my code

library(parallel)
library(zoo)

    no_cores <- detectCores()-1
    c1 <- makeCluster(no_cores)

    z <- read.zoo(df, FUN = as.yearmon, format = "%m/%d/%Y")
        getCoef <- function(z, lhs, rhs){
          if(all(is.na(z[,lhs]))) NA
          else coef(lm(paste(lhs, "~", rhs), z))["(Intercept)"] 
        }

        roll <- function(z, lhs, rhs = "A + B + C + D + E") {
          rollapplyr(z, 36, getCoef, by.column = FALSE, coredata = FALSE, lhs = lhs, rhs = rhs)
        }

        ynames <- colnames(df)[2:8785]
        tm1 <- system.time(
          L_rr <- ParLapply(c1, ynames, roll, z = z)
          )

The error that I got is :

Error in checkForRemoteErrors(val) : 
  3 nodes produced errors; first error: could not find function "rollapplyr"

In other words, I guess similar to ParLapply being similar to lapply in a parallel computing setting, there is a function equivalent rollapplyr. I wonder what it is. Thanks.

CPak CPak · Accepted Answer · 2017-09-08T14:46:04

Try this

library(parallel)
library(zoo)

no_cores <- detectCores()-1
c1 <- makeCluster(no_cores)

clusterEvalQ(c1, {
    library(zoo)
    z <- read.zoo(df, FUN = as.yearmon, format = "%m/%d/%Y")
    getCoef <- function(z, lhs, rhs){
                if(all(is.na(z[,lhs]))) NA
                else coef(lm(paste(lhs, "~", rhs), z))["(Intercept)"] 
    }

    roll <- function(z, lhs, rhs = "A + B + C + D + E") {
                rollapplyr(z, 36, getCoef, by.column = FALSE, coredata = FALSE, lhs = lhs, rhs = rhs)
    }

    ynames <- colnames(df)[2:8785]
})

tm1 <- system.time(
            L_rr <- ParLapply(c1, ynames, roll, z = z)
        )
stopCluster(c1)

You need to setup your parallel worker like you would your parent environment. I did this by embedding your functions, libraries, data in clusterEvalQ(cl, {..}. Make sure to kill your cluster after you're done.

Let me know if it doesn't work

Use rollapplyr in parallel computing

1 Answers