python-multiprocessing

I tried to use cupy in two parts of my program, one of them being parallelized with a pool. I managed to reproduce it with a simple example: import cupy import numpy as np from multiprocessing import pool def f(x): return cupy.asnumpy(2*cupy.array(x)) input = np.array([1,2,3,4]) print(cupy.asnumpy(cupy.array(input))) print(np.array(list(map(f, input)))) p = pool.Pool(4) output = p.map(f, input) p.close() p.join() print(output) The output is the following: [1 2 3 4] [2 4 6 8] Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib/python3.6/threading.py", line 916, in

I have several processes each completing tasks which require a single large numpy array, this is only being read (the threads are searching it for appropriate values). If each process loads the data I receive a memory error. I am therefore trying to minimise the memory usage by using a Manager to share the same array between the processes. However I still receive a memory error. I can load the array once in the main process however the moment I try to make it an attribute of the manager namespace I receive a memory error . I assumed the Managers acted like pointers and allowed seperate

I'm training a neural network with Keras with Tensorflow backend. Data set does not fit in RAM, therefore, I store it in the Mongo database and retrieve batches using subclass of keras.utils.Sequence . Everything works fine, if I run model.fit_generator() with use_multiprocessing=False . When I turn on multiprocessing, I get errors either during spawning of workers or in connection to the data base. If I create a connection in __init__ , I've got an exception whose text says something about errors in pickling lock objects. Sorry, I don't remember exactly. But the training even does not start.

Problem From the multiprocessing.Pool docs : apply_async(func ...) : A variant of the apply() method which returns a result object. ... Reading further ... apply(func[, args[, kwds]]) : Call func with arguments args and keyword arguments kwds. It blocks until the result is ready. Given this blocks, apply_async() is better suited for performing work in parallel. Additionally, func is only executed in one of the workers of the pool. The last bold line suggests only one worker from a pool is used. I find this is only true under certain conditions. Given Here is code that executes Pool.apply_async

I have a relatively simple python multiprocessing script that sets up a pool of workers that append output to a pandas dataframe by way of a custom manager. What I am finding is when I call close()/join() on the pool, not all the tasks submitted by apply_async are being completed. Here's a simplified example that submits 1000 jobs but only half complete causing an assertion error. Have I overlooked something very simple or is this perhaps a bug? from pandas import DataFrame from multiprocessing.managers import BaseManager, Pool class DataFrameResults: def __init__(self): self.results =

I am trying to scrape concurrently with selenium and multiprocessing modules. Below is roughly my approach: create queue with number of webdriver instances equal to number of workers create pool of workers each worker pulls webdriver instance from the queue when function terminates webdriver instance is put back on the queue Here is the code: #!/usr/bin/env python # encoding: utf-8 import time import codecs from selenium import webdriver from selenium.webdriver.common.desired_capabilities import DesiredCapabilities from multiprocessing import Pool from Queue import Queue def download_and_save