问题
Is it possible to run, in a qt application, without freezing the gui, let's say a sklearn gird search that use several jobs parallel (n_jobs > 1)? The problem is that joblib that is used for parallelizing sklearn code cannot run multiprocess into a thread.
For example, I'm using Gridsearch to find the best parameters for a svr, which is quite computionnaly intensive.
This question has been asked several times, but no solution found:
pyqt5-run-sklearn-calculations-on-a-separate-qthread, suggest the use of QProcess ?
multiprocessing-backed-parallel-loops-cannot-be-nested-below-threads,the threading.current_thread().name = 'MainThread' workaround does not work after the issue has been fixed
joblib-parallel-uses-only-one-core-if-started-from-qthread, rewrite the task using multiprocessing.Pool(processes=4). This method is not applicable for gridsearch embed njobs.
use sklearn cross validation train, in PyQt button, No aswers...
And any insight why this is purposely not supported (it a feature) ? It seems like it something that would be quite useful ?
回答1:
From my understanding of the issue, the problem resides with the default backend used by joblib, namely loky.
After some digging through the joblib and sklearn documentation, I resolved my issue by switching the joblib backend to threading. Note, the call to register_parallel_backend lies outside the __init__ function.
from sklearn.utils import parallel_backend, register_parallel_backend
from joblib._parallel_backends import ThreadingBackend
class ModelTrainer(QRunnable):
register_parallel_backend('threading', ThreadingBackend, make_default=True)
def __init__(self, **kwargs):
来源：https://stackoverflow.com/questions/53422410/in-a-pyqt5-application-is-it-possible-to-run-sklearn-with-parallel-jobs-without