Tensorflow Estimator - Periodic Evaluation on Eval Dataset

Question

The tensorflow documentation does not provide any example of how to perform a periodic evaluation of the model on an evaluation set.

Some people suggested the use of an Experiment, which sounds great but unfortunately does not work (depreciation and triggers an error).

Others suggested the use of SummarySaverHook, but I don't see how you can use that with an evaluation set (as opposed to the training set).

A solution would be to do the following

for i in range(number_of_epoch):
    estimator.train(...) // on training set
    estimator.evaluate(...) // on evaluation set

This architecture is explicitly discouraged in this paper (page 4 top right).

Any other idea/implementation?

EDIT:

The error message when running the experiment is the following:

File ".../anaconda2/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 253, in train if (config.environment != run_config.Environment.LOCAL and
AttributeError: 'RunConfig' object has no attribute 'environment'

Tensorflow version 1.3

Answer 1

Only a few parameters/options of Experiment are deprecated (what specific errors are you seeing?). If you create an Estimator that will do periodic checkpoints (using options in RunConfig) and an Experiment using it, you will get evaluation for each checkpoint by default when using train_and_evaluate method.

EDIT: As Maxime pointed out in the comments. He needed to add the following lines to get rid of his error:

os.environ['TF_CONFIG'] = json.dumps({'environment': 'local'})
config = tf.contrib.learn.RunConfig()