Running Keras model for prediction in multiple threads

Question

similar to this question I was running an asynchronous reinforcement learning algorithm and need to run model prediction in multiple threads to get training data more quickl

Answer 1

It's a bad idea to predict data in multiple threads . You can use greater batch_size in model.predict when you predict data offline and use tensorflow serving when you predict data online.

Answer 2

Keras is not thread safe, for predicting large batch you can use batch_size to set Max limits. If you are deploying to production than the ideal is to convert the model weights tensorflow protobuf and than use tensorflow serving.

You can follow this blog http://machinelearningmechanic.com/keras/2019/06/26/keras-serving-keras-model-quickly-with-tensorflow-serving-and-docker-md.html