keras + scikit-learn wrapper, appears to hang when GridSearchCV with n_jobs >1

Question

UPDATE: I have to re-write this question as after some investigation I realise that this is a different problem.

Context: running keras in a gridsearch

Answer 1

I know this is a late answer, but I dealt with this problem too and it really slowed me down not being able to run what is essentially trivially-parallelizable code. The issue is indeed with the tensorflow session. If a session in created in the parent process before GridSearchCV.fit(), it will hang!

The solution for me was to keep all session/graph creation code restricted to the KerasClassifer class and the model creation function i passed to it.

Also what Felipe said about the memory is true, you will want to restrict the memory usage of TF in either the model creation function or a subclass of KerasClassifier.

Related info:

• Session hang issue with python multiprocessing
• Keras + Tensorflow and Multiprocessing in Python