How to create ensemble in tensorflow?

Question

I am trying to create an ensemble of many trained models. All models have the same graph and just differ by its weights. I am creating the model graph using tf.get_var

Answer 1

There are several questions regarding this topic and a lot of possible answers/ways to do it. Here, I'd like to show how I came up with the most elegant and cleanest way of making an ensemble of N models, where N is arbitrary. This solution was tested with tf 1.12.0, python 2.7

The following code snippet is what you are looking for (comments below):

import tensorflow as tf
import numpy as np

num_of_ensembles = N
savers = list()
palceholders = list()
inference_ops = list()

for i in xrange(num_of_ensembles):
    with tf.name_scope('model_{}'.format(i)):
        savers.append(tf.train.import_meta_graph('saved_model.ckpt.meta'))

graph = tf.get_default_graph()

for i in xrange(num_of_ensembles):
    placeholders.append(graph.get_operation_by_name('model_{}/input_ph'.format(i)).outputs[0])
    inference_ops.append(graph.get_operation_by_name('model_{}/last_operation_in_the_network'.format(i)).outputs[0])


with tf.Session() as sess:
    for i in xrange(num_of_ensembles):
        savers[i].restore(sess, 'saved_model.ckpt')
        prediction = sess.run(inference_ops[i], feed_dict={placeholders[i]: np.random.rand(your_input.shape)})

So, first thing to do is to import the meta graph of each model. As suggested in the comments above, the key is to create for each model from the ensemble its own scope, in order to add a prefix like model_001/, model_002/ ... to each variable scopes. This will allow you to restore N different models, with their own independent variables.

All this graphs will live in the current default graph. Now, when you load a model you have to extract the inputs, outputs, and operations that you wish to use from the graph into new variables. To do so, you'll need to know the names of those tensors from the old model. You can inspect all saved operations using the command: ops = graph.get_operations(). In the example above the first operation is the placeholder assignment /input_ph while the last operation was named /last_operation_in_the_network (normally, if the author of the network doesn't specify the field name per each layer, you will find something like /dense_3, /conv2d_1 etc.). Note that it must be the exact final operation of your model and, also, you must provide the tensor which is the value .outputs[0] of the operation itself.

Finally, you can run the session with the correct inference operation and placeholder, getting the prediction as numpy array and doing whatever you want (averaging, majority voting, etc.)

Useful links that you may want to check:

• reddit post
• A quick complete tutorial to save and restore Tensorflow models