Need help on combine function in a Parallel Simulation study using doMC

Question

I want to ask for some help on writing a combine function for foreach(). Consider the function below:

library(mvtnorm)
library(doMC)

mySimFunc <- function(){
  myNum <- runif(1)
  myVec <- rnorm(10)
  myMat <- rmvnorm(5, rep(0, 3), diag(3))
  myListRslt <- list("myNum" = myNum, "myVec" = myVec, "myMat" = myMat)
return (myListRslt)
}

Now I'd like to run the code above for 1000 times using foreach() %dopar% and in each iteration I'd like to:

return myNum as is
get average of myVec and return it
get colMeans() of myMat and return it.

I'd like foreach() %dopar% to return a final list including:

a vector of length 1000 including 1000 myNum each corresponding to an iteration
a vector of length 1000 including 1000 average of myVec in each iteration
a matrix with 1000 rows where each row includes colMeans of myMat in that iteration

My Ideal solution

My ideal solution is o find a way that foreach() acts exactly like for so that I can simply define:

myNumRslt <- NULL
myVecRslt <- NULL
myMatRslt <- NULL

# and then simply aggregate result of each iteration to the variables above as:
foreach(i = 1:1000) %dopar%{
   rslt <- mySimFunc()
   myNumRslt <- c(myNumRslt, rslt$myNum)
   myVecRslt <- c(myVecRslt, mean(rslt$myVec))
   myMatRslt.tmp <- colMeans(rslt$myMat)
   myMatRslt <- rbind(myMatRslt, myMatRslt.tmp)
}

BUT, unfortunately seems that it's not possible to do that with foreach() so then I think the only solution is to write a combine function that does similar to result aggregation above.

Challenge

1) How could I write a combine function that returns what I explained above?

2) When we do %dopar% (suppose using doMC package), does doMC distribute each iteration to a CPU or it goes further and divide each iteration to further pieces and distribute them?

3) Is there any better (more efficient) way than using doMC and foreach() ? idea's In this question Brian mentioned a brilliant way to deal with lists including numeric values. In my case, I have numeric values as well as vectors and matrices. I don't know how to extend Brian's idea in my case.

Thanks very much for your help.

Hi @BenRollert, I added some clarification to the question. Please read the "ideal solution" in my question (just added to the question). As I explained, my ideal solution is to find a way that foreach() behaves exactly like for so that I would be able to do "result" aggregation in each iteration using "c" and rbind(). So the answer is no; I don't have to use .combine. — Sam
From what I understand, .combine will operate on the list output, i.e. rbind in .combine is the same thing as rbinding the list output. It is entirely possible there is some performance benefit I'm unaware of for edge cases, which is why I'm very interested to see other people's responses. — Ben Rollert

Ben Rollert Ben Rollert · Accepted Answer · 2014-05-05T08:45:42

Edit

Cleaned up, generalizable solution using .combine:

#modify function to include aggregation
mySimFunc2 <- function(){
myNum <- runif(1)
myVec <- mean(rnorm(10))
myMat <- colMeans(rmvnorm(5, rep(0, 3), diag(3)))
myListRslt <- list("myNum" = myNum, "myVec" = myVec, "myMat" = myMat)
return (myListRslt)
}

#.combine function
MyComb1 <- function(...) {
lst=list(...)
vec<-sapply(1:length(lst), function (i) return(lst[[i]][[1]] ))
vecavg<-sapply(1:length(lst),function (i) return(lst[[i]][[2]] ))
colmeans<-t(sapply(1:length(lst), function (i) return(lst[[i]][[3]])))
final<-list(vec,vecavg,colmeans)
names(final)<-c("vec","vecavg","colmeans")
return(final)
}

library(doParallel)
cl <- makeCluster(3) #set cores
registerDoParallel(cl)

foreach(i=1:1000,.export=c("mySimFunc2","MyComb1"),.combine=MyComb1,
.multicombine=TRUE,.maxcombine=1000, .packages=c("mvtnorm"))%dopar%{mySimFunc2()}

You should now have a list output containing the desired three objects, which I've titled respectively as vec, vecavg, and colmeans. Note you must set .maxcombine to the number of iterations if iterations are greater than 100.

As a side note, it does not make sense to parallelize for this example task, although I'm guessing the real task may be more complex.

Need help on combine function in a Parallel Simulation study using doMC

1 Answers