pathos

I've a function that takes in a list of images and produces the output, in a list, after applying OCR to the image. I have an another function that controls the input to this function, by using multiprocessing. So, when I have a single list (i.e. no multiprocessing), each image of the list took ~ 1s, but when I increased the lists that had to be processed parallely to 4, each image took an astounding 13s. To understand where the problem really is, I tried to create a minimal working example of the problem. Here I have two functions eat25 and eat100 which open an image name and feed it to the

import pathos.multiprocessing as mp class Model_Output_File(): """ Class to read Model Output files """ def __init__(self, ftype = ''): """ Constructor """ # Create a sqlite database in the analysis directory self.db_name = 'sqlite:///' + constants.anly_dir + os.sep + ftype + '_' + '.db' self.engine = create_engine(self.db_name) self.ftype = ftype def parse_DGN(self, fl): df = pandas.read_csv(...) df.to_sql(self.db_name, self.engine, if_exists='append') def collect_epic_output(self, fls): pool = mp.ProcessingPool(4) if(self.ftype == 'DGN'): pool.map(self.parse_DGN, fls) else: logging.info(

I am trying to parallelize a code using class methods via multiprocessing. The basic structure is the following: # from multiprocessing import Pool from pathos.multiprocessing import ProcessingPool as Pool class myclass(object): def __init__(self): #some code def mymethod(self): #more code return another_instance_of_myclass def myfunc(myinstance,args): #some code test=myinstance.mymethod() #more code return myresult #not an instance,just a number p=Pool() result = p.map(myfunc,listwithdata) After this had failed with the normal multiprocessing, I became aware of the issues with Pickle and