If Keras results are not reproducible, what's the best practice for comparing models and choosing hyper parameters?

Question

UPDATE: This question was for Tensorflow 1.x. I upgraded to 2.0 and (at least on the simple code below) the reproducibility issue seems fixed on 2.0. So that solves my p

Answer 1

It's sneaky, but your code does, in fact, lack a step for better reproducibility: resetting the Keras & TensorFlow graphs before each run. Without this, tf.set_random_seed() won't work properly - see correct approach below.

I'd exhaust all the options before tossing the towel on non-reproducibility; currently I'm aware of only one such instance, and it's likely a bug. Nonetheless, it's possible you'll get notably differing results even if you follow through all the steps - in that case, see "If nothing works", but each is clearly not very productive, thus it's best on focusing attaining reproducibility:

Definitive improvements:

• Use reset_seeds(K) below
• Increase numeric precision: K.set_floatx('float64')
• Set PYTHONHASHSEED before the Python kernel starts - e.g. from terminal
• Upgrade to TF 2, which includes some reproducibility bug fixes, but mind performance
• Run CPU on a single thread (painfully slow)
• Do not import from tf.python.keras - see here
• Ensure all imports are consistent (i.e. don't do from keras.layers import ... and from tensorflow.keras.optimizers import ...)
• Use a superior CPU - for example, Google Colab, even if using GPU, is much more robust against numeric imprecision - see this SO

Also see related SO on reproducibility

---

If nothing works:

• Rerun X times w/ exact same hyperparameters & seeds, average results
• K-Fold Cross-Validation w/ exact same hyperparameters & seeds, average results - superior option, but more work involved

---

Correct reset method:

def reset_seeds(reset_graph_with_backend=None):
    if reset_graph_with_backend is not None:
        K = reset_graph_with_backend
        K.clear_session()
        tf.compat.v1.reset_default_graph()
        print("KERAS AND TENSORFLOW GRAPHS RESET")  # optional

    np.random.seed(1)
    random.seed(2)
    tf.compat.v1.set_random_seed(3)
    print("RANDOM SEEDS RESET")  # optional

---

Running TF on single CPU thread: (code for TF1-only)

session_conf = tf.ConfigProto(
      intra_op_parallelism_threads=1,
      inter_op_parallelism_threads=1)
sess = tf.Session(config=session_conf)