limiting computation time for each subproblem in clusterMap (R parallel package)

Question

I need to run task over a range of input parameters with very different run times. I use parallel::clusterMap() to do this in parallel with dynamic scheduling. Sometimes the computation time is just not feasible for single problems. Is there any way of killing the cluster after some predefined time limit and still retrieve the completed tasks?

If I simply set the timeout parameter the cluster is killed without retrieving the already finished tasks. Minimal example (NOT working!):

f <- function(t) {Sys.sleep(t); return(t)}
t <- c(1, 1, 2, 15, 2, 1, 3, 4, 1, 1, 1,3)
cl <- makeCluster(3, timeout = 5)
as.numeric(clusterMap(cl, f, t, .scheduling = "dynamic"))
stopCluster(cl)

CL. CL. · Accepted Answer · 2015-08-06T12:59:59

I would not kill the workers, instead I would make the workers stop on their own after some time.

Here an example very close to the code you posted: Each worker is active for t seconds, but no longer than 4 seconds. After t or 4 seconds, it stops and returns what has been done so far:

f <- function(t) {
  executionTime <- 0
  while(executionTime < t & executionTime < 4) {
    executionTime <- executionTime + 1
    Sys.sleep(1)
  }
  return(executionTime)
}
t <- c(1, 1, 2, 15, 2, 1, 3, 4, 1, 1, 1,3)


cl <- makeCluster(3)
print(as.numeric(clusterMap(cl, f, t, .scheduling = "dynamic")))
stopCluster(cl)

# [1] 1 1 2 4 2 1 3 4 1 1 1 3

Note the fourth element which is 4, not 15. The worker iterated 4 times / 4 seconds, then stopped and returned 4.

limiting computation time for each subproblem in clusterMap (R parallel package)

1 Answers