Write results to a file inside foreach loop R

Question

I am using the foreach package with %dopar% to implement a set of simulations. The structure of the simulation is illustrated by this simple version:

library(foreach)
library(doMC)
registerDoMC(4) 

id.list <- c(1:3000)

results <- foreach(i=1:1000,.combine=rbind) %dopar% {

    ## Randomly draw 100 donor pool units and 1 treated unit
    v1 <- sample(id.list, 1, replace = FALSE)  
    v2 <- sample(id.list, 1, replace = FALSE)
    v3 <- sample(id.list, 1, replace = FALSE)
    v4 <- sample(id.list, 1, replace = FALSE)

    c(i,v1,v2,v3,v4)
}

I would like to write the results from this loop to a .csv file or something similar while the loop is still running. For instance, when the loop hits iteration 500, I'd like to write the first 500 rows to .csv.

Alternatively, I would be open to breaking the loop when it hits iteration 500, while extracting the first 500 rows of results.

Is anything like this possible within foreach?

Liz Young Liz Young · Accepted Answer · 2015-07-07T17:56:31

say, instead of 1000 trys, you were going for 10000, I would break it up like this, using part_num to differentiate the blocks of size 500

library(foreach)
library(doMC)
registerDoMC(4) 

id.list <- c(1:3000)

for(part_num in 1:20){
  results <- foreach(i=1:500,.combine=rbind) %dopar% {
    ## Randomly draw 100 donor pool units and 1 treated unit
    v1 <- sample(id.list, 1, replace = FALSE)  
    v2 <- sample(id.list, 1, replace = FALSE)
    v3 <- sample(id.list, 1, replace = FALSE)
    v4 <- sample(id.list, 1, replace = FALSE)
    c(i,v1,v2,v3,v4)
  }

  write.csv(results, file = paste('part', part_num, '.csv', sep = ''))
}

Write results to a file inside foreach loop R

1 Answers