How to preserve the list of data frame form after using parallel apply

问题

I have the following function my_func which takes parameter stored in a data frame params and take one extra param as another df independently indf

library(tidyverse)

my_func <- function (x=NULL,y=NULL,z=NULL, indf=NULL) {
 out <- (x * y *z )
 out * indf
}


params <- tribble(
  ~x, ~y, ~z,
  5,     1,  1,
  10,     5,  3,
  -3,    10,  5
)

indf <- tribble(
  ~A, ~B, ~C,
  100,     10,  1,
  1000,     300,  3,
  20,    10,  5
)


params %>% 
  pmap(my_func, indf=indf)

It produces the following list of data frames:

#> [[1]]
#>      A    B  C
#> 1  500   50  5
#> 2 5000 1500 15
#> 3  100   50 25
#> 
#> [[2]]
#>        A     B   C
#> 1  15000  1500 150
#> 2 150000 45000 450
#> 3   3000  1500 750
#> 
#> [[3]]
#>         A      B    C
#> 1  -15000  -1500 -150
#> 2 -150000 -45000 -450
#> 3   -3000  -1500 -750

What I want to do then is to run the above function with parallel package. I did it this way:

library(parallel)
params %>% 
  lift(mcmapply, mc.cores = detectCores() - 1)(FUN = my_func, indf=indf)

But it produces the following matrix instead.

     [,1]  [,2] [,3]
[1,]  500  1500 -150
[2,] 5000 45000 -450
[3,]  100  1500 -750

How can I use parallel so that it produces a list of data frames like the initial output?

回答1:

library(parallel)

nc <- max(detectCores() - 1, 1L)

params %>% 
  lift(mcmapply, SIMPLIFY = FALSE, mc.cores = nc)(FUN = my_func, MoreArgs = list(indf = indf))

# [[1]]
#      A    B  C
# 1  500   50  5
# 2 5000 1500 15
# 3  100   50 25
# 
# [[2]]
#        A     B   C
# 1  15000  1500 150
# 2 150000 45000 450
# 3   3000  1500 750
# 
# [[3]]
#         A      B    C
# 1  -15000  -1500 -150
# 2 -150000 -45000 -450
# 3   -3000  -1500 -750

Edit

Here is a "cleaner" option, that should feel more like using pmap:

nc <- max(parallel::detectCores() - 1, 1L)

par_pmap <- function(.l, .f, ..., mc.cores = getOption("mc.cores", 2L)) {
  do.call(
    parallel::mcmapply, 
    c(.l, list(FUN = .f, MoreArgs = list(...), SIMPLIFY = FALSE, mc.cores = mc.cores))
  )
}

library(magrittr)

params %>% 
  par_pmap(my_func, indf = indf, mc.cores = nc)

# [[1]]
#      A    B  C
# 1  500   50  5
# 2 5000 1500 15
# 3  100   50 25
# 
# [[2]]
#        A     B   C
# 1  15000  1500 150
# 2 150000 45000 450
# 3   3000  1500 750
# 
# [[3]]
#         A      B    C
# 1  -15000  -1500 -150
# 2 -150000 -45000 -450
# 3   -3000  -1500 -750

来源：https://stackoverflow.com/questions/47625279/how-to-preserve-the-list-of-data-frame-form-after-using-parallel-apply

标签

parallel-processing

tidyverse