How to pass additional parameters (stored in "...") to multiple cores for parLapply, in R?

Question

I'm writing a function f1() that accepts additional parameters "..." to be passed to a function f0() inside of f1(). More precise, f0() is called inside a parallel lapply call, inside of f1. It works fine as long as there is at least one parameter to be passed, but if "..." is empty, I get the error message: " Error in get(name, envir = envir) : argument "..." is missing, with no default"

It works fine if I use lapply, instead of parLapply.

Is there a proper solution? I don't want to define all parameters for f0() explicitly in the definition of f1().

First, I define the function with the two optional parameters.

f0 <- function(a, b, d1 = NULL, d2 = NULL){
   if(is.null(d1)){
      ret <- a * b
   }else{
      ret <- a * b / d1
   }
   if(!is.null(d2)){
      ret <- ret - d2
   }
   ret
}

Next,the functions f1() and f1_par() that do the same, one with sapply and the other with parSapply.

f1 <- function(A, ...){
   # A ... vector of a
   B <- rev(A)

   sapply(seq_along(A), function(i){
      f0(A[i], B[i], ...)
   }) 
}


f1_par <- function(A, ...){
   # A ... vector of a
   B <- rev(A)
   cl <- parallel::makeCluster(2)
   parallel::clusterExport(cl, envir = environment(), c("A", "B", "f0","..."))
   ret <- parallel::parSapply(cl, seq_along(A), function(i){
      f0(A[i], B[i], ...)
   }) 
   parallel::stopCluster(cl)
   ret
}

I get the right results for all of the following six functions calls, except for the last one:

A <- 1:4

# sapply
f1(A, d1 = 2, d2 = 4)
f1(A, d1 = 2)
f1(A)

# parSapply
f1_par(A, d1 = 2, d2 = 4)
f1_par(A, d1 = 2)
f1_par(A) # this one causes the error

HenrikB HenrikB · Accepted Answer · 2019-10-31T20:10:45

I'm pretty sure you cannot export ... that way. Instead, make sure to pass down ... as arguments as in:

f1 <- function(A, ...) {
   # A ... vector of a
   B <- rev(A)

   sapply(seq_along(A), function(i, ...) {
      f0(A[i], B[i], ...)
   }, ...) 
}

Then, do the same with parallel::parSapply().

How to pass additional parameters (stored in "...") to multiple cores for parLapply, in R?

1 Answers