Parallelization with Multiple Cores per Worker

问题

Some R packages have functions that can do their work in parallel if multiple cores are available - for example, the rstan package can run multiple MCMC chains in parallel. When I run a number of Stan processes in parallel to each other using, e.g., doSNOW and foreach, I'd like my code to operate in parallel at both levels*. Instead, the Stan processes get farmed out to my workers and seem to run their chains in sequence there, as if once they've been assigned to a core they can't see the machine's other cores and think they're on a single-core machine.

Is there a way to create clusters of 4-core nodes that I can pass to some parallelization package in R, so that I can get the maximum efficiency out of my machine?

*say I have a 36 core machine, and 9 Stan scenarios run with 4 chains each. Ideally, I have 36 processes that I could run all at once. Right now, I get 9 cores used at a time, and it takes 4x as long as I'm hoping it could.

回答1:

Assuming you have a good reason not to create 36 parallel workers directly (e.g. if you have a computing cluster or something like that), then the following should work (using doParallel as an example):

library(doParallel)

# create "outer" workers
outer_workers <- makeCluster(2L)
# register outer_workers
registerDoParallel(outer_workers)

# create "inner" workers
clusterEvalQ(outer_workers, {
    library(doParallel)
    inner_workers <- makeCluster(2L)
    # register inner_workers
    registerDoParallel(inner_workers)

    NULL
})

# assuming you use foreach directly
foreach(i = 1L:2L) %dopar% {
    foreach(j = 1L:2L) %dopar% {
        # code
    }

    NULL
}

# stop inner workers
clusterEvalQ(outer_workers, {
    stopCluster(inner_workers)
    registerDoSEQ()
    NULL
})

stopCluster(outer_workers); registerDoSEQ()

This example creates 4 processes in total, 2 outer and 2 inner. In your example you could have 9 outer and 4 inner processes.

来源：https://stackoverflow.com/questions/49436580/parallelization-with-multiple-cores-per-worker

标签

parallel-processing

parallel-foreach