mcapply: all scheduled cores encountered errors in user code

╄→尐↘猪︶ㄣ 提交于 2021-02-09 09:21:06

问题


The following is my code. I am trying get the list of all the files (~20000) that end with .idat and read each file using the function illuminaio::readIDAT.

library(illuminaio)
library(parallel)
library(data.table)

# number of cores to use
ncores = 8

# this gets all the files with .idat extension ~20000 files
files <- list.files(path = './',
                    pattern = "*.idat",
                    full.names = TRUE)

# function to read the idat file and create a data.table of filename, and two more columns
# write out as csv using fwrite
get.chiptype <- function(x)
{
  idat <- readIDAT(x)
  res <- data.table(filename = x, nSNPs = nrow(idat$Quants), Chip = idat$ChipType)
  fwrite(res, file.path = 'output.csv', append = TRUE)
}

# using mclapply call the function get.chiptype on all 20000 files.
# use 8 cores at a time
mclapply(files, FUN = function(x) get.chiptype(x), mc.cores = ncores)

After reading and writing info about 1200 files, I get the following message:

Warning message:
In mclapply(files, FUN = function(x) get.chiptype(x), mc.cores = ncores) :
  all scheduled cores encountered errors in user code

How do I resolve it?


回答1:


Calling mclapply() in some instances requires you to specify a random number generator that allows for multiple streams of random numbers. R version 2.14.0 has an implementation of Pierre L'Ecuyer's multiple pseudo-random number generator.

Try adding the following before the mclapply() call, with a pre-specified value for 'my.seed':

set.seed( my.seed, kind = "L'Ecuyer-CMRG" );


来源:https://stackoverflow.com/questions/39280009/mcapply-all-scheduled-cores-encountered-errors-in-user-code

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!