How to calculate prediction uncertainty using Keras?

Question

I would like to calculate NN model certainty/confidence (see What my deep model doesn\'t know) - when NN tells me an image represents \"8\", I would like to know how certain

Answer 1

Made a few changes to the top voted answer. Now it works for me.

It's a way to estimate model uncertainty. For other source of uncertainty, I found https://eng.uber.com/neural-networks-uncertainty-estimation/ helpful.

f = K.function([model.layers[0].input, K.learning_phase()],
               [model.layers[-1].output])


def predict_with_uncertainty(f, x, n_iter=10):
    result = []

    for i in range(n_iter):
        result.append(f([x, 1]))

    result = np.array(result)

    prediction = result.mean(axis=0)
    uncertainty = result.var(axis=0)
    return prediction, uncertainty