How to calculate prediction uncertainty using Keras?

Question

I would like to calculate NN model certainty/confidence (see What my deep model doesn\'t know) - when NN tells me an image represents \"8\", I would like to know how certain

Answer 1

If you want to implement dropout approach to measure uncertainty you should do the following:

1. Implement function which applies dropout also during the test time:

   import keras.backend as K
   f = K.function([model.layers[0].input, K.learning_phase()],
                  [model.layers[-1].output])
   

2. Use this function as uncertainty predictor e.g. in a following manner:

   def predict_with_uncertainty(f, x, n_iter=10):
       result = numpy.zeros((n_iter,) + x.shape)
   
       for iter in range(n_iter):
           result[iter] = f(x, 1)
   
       prediction = result.mean(axis=0)
       uncertainty = result.var(axis=0)
       return prediction, uncertainty
   

Of course you may use any different function to compute uncertainty.