How to stop R from leaving zombie processes behind

问题:

Here is a little reproducible example:

library(doMC) library(doParallel) registerDoMC(4)     timing <- system.time( fitall <- foreach(i=1:1000, .combine = "c") %dopar% {                 print(i)             }) 

I start up R and look at the process table:

> system("ps -efl") F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD 4 S chbr         1     0  5  80   0 - 21399 wait   10:58 ?        00:00:00 /usr/local/lib/R/bin/exec/R --no-save --no-restore 0 S chbr         9     1  0  80   0 -  1113 wait   10:58 ?        00:00:00 sh -c ps -efl 0 R chbr        10     9  0  80   0 -  4294 -      10:58 ?        00:00:00 ps -efl 

If I use the aformentioned simple for loop doMC or doParallel leave a zombie process behind. Output of ps -efl after running the loop:

> system("ps -efl") F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD 4 S chbr         1     0  4  80   0 - 25256 wait   11:00 ?        00:00:00 /usr/local/lib/R/b 1 Z chbr        10     1  0  80   0 -     0 exit   11:00 ?        00:00:00 [R] <defunct> 0 S chbr        12     1  0  80   0 -  1113 wait   11:00 ?        00:00:00 sh -c ps -efl 0 R chbr        13    12  0  80   0 -  4294 -      11:00 ?        00:00:00 ps -efl 

If I repeat the loop without issuing registerDoMC(4) again no additional zombie process gets created. However, if I issue registerDoMC(4) an additional zombie process gets created:

> system("ps -efl") F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD 4 S chbr         1     0  0  80   0 - 25554 wait   11:00 ?        00:00:01 /usr/local/lib/R/b 1 Z chbr        21     1  0  80   0 -     0 exit   11:02 ?        00:00:00 [R] <defunct> 1 Z chbr        22     1  0  80   0 -     0 exit   11:02 ?        00:00:00 [R] <defunct> 0 S chbr        26     1  0  80   0 -  1113 wait   11:03 ?        00:00:00 sh -c ps -efl 0 R chbr        27    26  0  80   0 -  4294 -      11:03 ?        00:00:00 ps -efl 

That's how I figured it could be doMC which is doing something that should not be done. If doMC is causing this is there a way to stop doMC from leaving zombie processes behind? (stopCluster() does not work as no cluster gets created in the first place.)

> sessionInfo() R Under development (unstable) (2014-08-16 r66404) Platform: x86_64-unknown-linux-gnu (64-bit)  locale:  [1] LC_CTYPE=en_IE.UTF-8       LC_NUMERIC=C                [3] LC_TIME=en_IE.UTF-8        LC_COLLATE=en_IE.UTF-8      [5] LC_MONETARY=en_IE.UTF-8    LC_MESSAGES=en_IE.UTF-8     [7] LC_PAPER=en_IE.UTF-8       LC_NAME=C                   [9] LC_ADDRESS=C               LC_TELEPHONE=C             [11] LC_MEASUREMENT=en_IE.UTF-8 LC_IDENTIFICATION=C         attached base packages: [1] parallel  stats     graphics  grDevices utils     datasets  methods   [8] base       other attached packages: [1] doParallel_1.0.8 doMC_1.3.3       iterators_1.0.7  foreach_1.4.2     loaded via a namespace (and not attached): [1] codetools_0.2-8 compiler_3.2.0 

回答1:

This really has nothing to do with foreach or doMC; as Steve Weston has pointed out in answer to other StackOverflow queries, doMC is essentially just a wrapper for mclapply, and you can see zombie processes created with a simple call to mclapply:

library(parallel) mclapply(rep(5,4), rnorm) 

On my system, this leaves two zombie processes:

[richcalaway@richcalaway-pc ~]$ ps -efl | grep defunct 1 Z 1660945517 28701 28624  0 77  0 -     0 exit   12:00 pts/1    00:00:00 [R] <defunct> 1 Z 1660945517 28702 28624  0 78  0 -     0 exit   12:00 pts/1    00:00:00 [R] <defunct> 0 S 1660945517 28704 28308  0 78  0 - 15306 pipe_w 12:00 pts/2    00:00:00 grep defunct 

Under normal circumstances, these zombie processes won't cause any trouble, and they do disappear when the R session ends. You can avoid them by using doParallel and a fork cluster instead of using doMC.

Cheers,

Rich Calaway

Principal Program Manager

Revolution Analytics

文章来源: How to stop R from leaving zombie processes behind