Which seeds have to be set where to realize 100% reproducibility of training results in tensorflow?

Question

In a general tensorflow setup like

model = construct_model()
with tf.Session() as sess:
    train_model(sess)

Where construct_model()

Answer 1

The best solution which works as of today with GPU is to install tensorflow-determinism with the following:

pip install tensorflow-determinism

Then include the following code to your code

import tensorflow as tf
import os
os.environ['TF_DETERMINISTIC_OPS'] = '1'

source: https://github.com/NVIDIA/tensorflow-determinism

Answer 2

One possible reason is that when constructing the model, there are some code using numpy.random module. So maybe you can try to set the seed for numpy, too.