“foreach” loop : Using all cores in R (especially if we are sending sql queries inside foreach loop)

Question

I intend to use "foreach" to uitlize all the cores in my CPU. The catch is i need to send a sql query inside the loop. The script is working fine with normal 'for' loop, but it is giving following error when i change it to 'foreach'. The error is :

select: Interrupted system call    
select: Interrupted system call    
select: Interrupted system call    
Error in { : task 1 failed - "expired MySQLConnection"

The code i used is :

library(foreach)
library(doMC)
library(RMySQL)
library(multicore)
registerDoMC(cores=6)
m <- dbDriver("MySQL", max.con = 100)
con <- dbConnect(m, user="*****", password = "******", host ="**.**.***",dbname="dbname")
list<-dbListTables(con)
foreach(i = 1:(length(list))%dopar%{
  query<-paste("SELECT * FROM ",list[i]," WHERE `CLOSE` BETWEEN 1 AND 100",sep="")
  t<-dbGetQuery(con,query)
}

Though 'foreach' is working fine in my system for all other purposes, it is giving error only in case of sql queries. Is there a way to send sql queries inside 'foreach' loop?

Answer 1

My suggestion is this: Move the database queries outside the loop, and lock access so you dont do parallel database queries. I think that will speed things up too, as you won't have parallel disk access, while still being able to do parallel processing.

Meaning (pseudo code) db = connect to database threadlock = lock();

parfor { threadlock.lock result = db query (pull all data here, as you cant process while you load without keeping the database locked) thread.unlock process resulting data (which is now just data, and not a sql object). }