Correctly loading Keras model in Django that supports multi-tenancy

Question

I am try to write a REST api in django that uses a Keras model to return a prediction. However the load_model() function takes some time to load the model and I don't want my users to have to wait so long (each time the model is initialized). What would be the correct way to initialize the model so that is is loaded once and the predictions are done using that same model?

On a side note one method that I thought cold be possible was to initialize the model in settings.py as below :

settings.py

json_file=open("model.json","r")
loaded_json=json_file.read()
json_file.close()

model=model_from_json(loaded_json)
model.load_weights("model.h5")
MODEL=model

And then in my views.py I use this variable MODEL as :

views.py

from django.conf import settings
model=settings.MODEL
def index():
    print "Predicting"
    res=model.predict(numpy.stack([test_img]))
    print res

This works great if only one user is active at a time ( model is initialized once and all subsequent predictions are done using that model). However if multiple users are active at a time then it works good for the call that came first but the latter call gives the error

'NoneType' object has no attribute 'shape'
Apply node that caused the error: ConvOp{('imshp', (31, 31, 32)),('kshp', (3, 3)),('nkern', 64),('bsize', None),('dx', 1),('dy', 1),('out_mode', 'valid'),('unroll_batch', None),('unroll_kern', None),('unroll_patch', True),('imshp_logical', (31, 31, 32)),('kshp_logical', (3, 3)),('kshp_logical_top_aligned', True)}(InplaceDimShuffle{0,2,3,1}.0, InplaceDimShuffle{3,2,0,1}.0)
Toposort index: 13
Inputs types: [TensorType(float32, 4D), TensorType(float32, 4D)]
Inputs shapes: [(1L, 31L, 31L, 32L), 'No shapes']
Inputs strides: [(123008L, 124L, 4L, 3844L), 'No strides']
Inputs values: ['not shown', None]
Outputs clients: [[Elemwise{Composite{(i0 * ((i1 + i2) + Abs((i1 + i2))))}}[(0, 1)](TensorConstant{(1L, 1L, 1..1L) of 0.5}, ConvOp{('imshp', (31, 31, 32)),('kshp', (3, 3)),('nkern', 64),('bsize', None),('dx', 1),('dy', 1),('out_mode', 'valid'),('unroll_batch', None),('unroll_kern', None),('unroll_patch', True),('imshp_logical', (31, 31, 32)),('kshp_logical', (3, 3)),('kshp_logical_top_aligned', True)}.0, InplaceDimShuffle{x,0,x,x}.0)]]

How should i load the model properly so that it can be accessed simultaneously?

Thank you for your time.

Answer 1

Look here please https://github.com/keras-team/keras/issues/2397#issuecomment-254919212

eg. in Django settings construct the model...

modelFile = 'path_to_my_model.h5'    
pipe = joblib.load(modelFile.replace('.h5','.pkl'))
model = models.load_model(modelFile)
pipe.steps.append(('nn', model))    
graph = tensorflow.get_default_graph()

and then reuse like this in Django REST method:

import myDjango.settings as sett
# ...

@csrf_exempt
def evaluate(request):
    """
    Do the evaluation.
    """
    if request.method == 'POST':
        data = JSONParser().parse(request)
        i = data['inputs']

        outputs = MyMlClass.PredictArray( sett.graph, sett.pipe , i, 'model.h5' )

        return JsonResponse(outputs, status=201, safe=False)

Works for me very well (VisualStudio Django project, Python 3.6). Construction of the model in REST handler is not recommended and in fact won't work - it will work just in the very first invocation.