Exception in thread QueueManagerThread - scikit-learn

问题

When I set n_jobs=-1 I get error and if I set n_jobs equal big value (n_jobs=100), but if set smaller value (e.g. n_jobs=32), it works fine. I've tried reinstall scikit-learn and joblib packages, but to no avail. Also, it (n_jobs=-1) works fine previously, but suddenly go wrong.

from sklearn import datasets
from sklearn.model_selection import cross_validate, StratifiedKFold
from sklearn.linear_model import RidgeClassifier

iris = datasets.load_iris()
iris_X = iris.data
iris_y = iris.target

skf = StratifiedKFold(n_splits=5, random_state=0, shuffle=True)
scoring = {'accuracy': 'accuracy'}

model_ridge = RidgeClassifier(random_state=0)
scores = cross_validate(estimator=model_ridge, 
                            X=iris_X,
                            y=iris_y,
                            scoring=scoring,
                            cv=skf.split(iris_X, iris_y),
                            n_jobs=-1)

Exception in thread QueueManagerThread:

Traceback (most recent call last): File "C:\ProgramData\Anaconda3\envs\py36\lib\threading.py", line 916, in _bootstrap_inner self.run()

File "C:\ProgramData\Anaconda3\envs\py36\lib\threading.py", line 864, in run self._target(*self._args, **self._kwargs)

File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\sklearn\externals\joblib\externals\loky\process_executor.py", line 615, in _queue_management_worker ready = wait(readers + worker_sentinels)

File "C:\ProgramData\Anaconda3\envs\py36\lib\multiprocessing\connection.py", line 859, in wait ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)

File "C:\ProgramData\Anaconda3\envs\py36\lib\multiprocessing\connection.py", line 791, in _exhaustive_wait res = _winapi.WaitForMultipleObjects(L, False, timeout)

ValueError: need at most 63 handles, got a sequence of length 65

回答1:

Sounds like you're running a lot of cores there! Set n_jobs=60 and this should work. See more details here https://github.com/psf/black/issues/564

来源：https://stackoverflow.com/questions/54905588/exception-in-thread-queuemanagerthread-scikit-learn