joblib

问题 I have the following code, where I try to parallelize: import numpy as np from joblib import Parallel, delayed lst = [[0.0, 1, 2], [3, 4, 5], [6, 7, 8]] arr = np.array(lst) w, v = np.linalg.eigh(arr) def proj_func(i): return np.dot(v[:,i].reshape(-1, 1), v[:,i].reshape(1, -1)) proj = Parallel(n_jobs=-1)(delayed(proj_func)(i) for i in range(len(w))) proj returns a really large list and it's causing memory issues. Is there a way I could work around this? I had thought about returning a

问题 I have the following code, where I try to parallelize: import numpy as np from joblib import Parallel, delayed lst = [[0.0, 1, 2], [3, 4, 5], [6, 7, 8]] arr = np.array(lst) w, v = np.linalg.eigh(arr) def proj_func(i): return np.dot(v[:,i].reshape(-1, 1), v[:,i].reshape(1, -1)) proj = Parallel(n_jobs=-1)(delayed(proj_func)(i) for i in range(len(w))) proj returns a really large list and it's causing memory issues. Is there a way I could work around this? I had thought about returning a

问题 I have a piece of code below where the joblib.Parallel() returns a list. import numpy as np from joblib import Parallel, delayed lst = [[0.0, 1, 2], [3, 4, 5], [6, 7, 8]] arr = np.array(lst) w, v = np.linalg.eigh(arr) def proj_func(i): return np.dot(v[:,i].reshape(-1, 1), v[:,i].reshape(1, -1)) proj = Parallel(n_jobs=-1)(delayed(proj_func)(i) for i in range(len(w))) Instead of a list, how do I return a generator using joblib.Parallel() ? Edit: I have updated the code as suggested by

问题 I'm converting some serial processed python jobs to multiprocessing with dask or joblib. Sadly I need to work on windows. When running from within IPython or from command line invoking the py-file with python everything is running fine. When compiling an executable with cython, it is no longer running fine: Step by step more and more processes (unlimited and bigger than the number of requested processes) get startet and block my system. It somehow feels like Multiprocessing Bomb - but of

问题 I'm running joblib in a Flask application living inside a Docker container together with uWSGI (started with threads enabled) which is started by supervisord. The startup of the webserver shows the following error: unable to load configuration from from multiprocessing.semaphore_tracker import main;main(15) /usr/local/lib/python3.5/dist-packages/sklearn/externals/joblib/_multiprocessing_helpers.py:38: UserWarning: [Errno 32] Broken pipe. joblib will operate in serial mode Any idea how to fix

问题 I have a scikit-learn pipline with kerasRegressor in it: estimators = [ ('standardize', StandardScaler()), ('mlp', KerasRegressor(build_fn=baseline_model, nb_epoch=5, batch_size=1000, verbose=1)) ] pipeline = Pipeline(estimators) After, training the pipline, I am trying to save to disk using joblib... joblib.dump(pipeline, filename , compress=9) But I am getting an error: RuntimeError: maximum recursion depth exceeded How would you save the pipeline to disk? 回答1: I struggled with the same

问题 This is not a real issue, but I'd like to understand: running sklearn from Anaconda distrib on a Win7 4 cores 8 GB system fitting a KMeans model on a 200.000 samples*200 values table. running with n-jobs = -1: (after adding the if __name__ == '__main__': line to my script) I see the script starting 4 processes with 10 threads each. Each process uses about 25% of the CPU (total: 100%). Seems to work as expected running with n-jobs = 1: stays on a single process (not a surprise), with 20

问题 I am using Parallel function from joblib package in Python. I would like to use this function only for handle one of my functions but unfortunately the whole code is running in parallel (apart from other functions). Example: from joblib import Parallel, delayed print ('I do not want this to be printed n times') def do_something(arg): some calculations(arg) Parallel(n_jobs=5)(delayed(do_something)(i) for i in range(0, n)) 回答1: This is a common error to miss a design direction from

问题 I'm running Python 3.5.1 on Windows Server 2013 at work. I have some embarrassingly parallel tasks that seem to work on Python 2.7 with basically the same code, but I am unable to figure out how to get it to run on Python 3.5.1. I'm using Anaconda 2.4.1 The code looks like this... I've stripped it down to basically the minimum. \ ->main.py \apackage\ ->__init__.py ->amodule.py Code for main.py from tpackage import AClass def go(): x = AClass().AFunction() return x if __name__ == '__main__': x

问题 I get the foll. userwarning when trying to cache results using joblib: from tempfile import mkdtemp cachedir = mkdtemp() from joblib import Memory memory = Memory(cachedir=cachedir, verbose=0) @memory.cache def get_nc_var3d(path_nc, var, year): """ Get value from netcdf for variable var for year :param path_nc: :param var: :param year: :return: """ try: hndl_nc = open_or_die(path_nc) val = hndl_nc.variables[var][int(year), :, :] except: val = numpy.nan logger.info('Error in getting var ' +