Average weights in keras models

Question

How to average weights in Keras models, when I train few models with the same architecture with different initialisations?

Now my code looks something like this?

Answer 1

I can't comment on the accepted answer, but to make it work on tensorflow 2.0 with tf.keras I had to make the list in the loop into a numpy array:

new_weights = list()
for weights_list_tuple in zip(*weights): 
    new_weights.append(
        np.array([np.array(w).mean(axis=0) for w in zip(*weights_list_tuple)])
    )

If different input models need to be weighted differently, np.array(w).mean(axis=0) needs to be replaced with np.average(np.array(w),axis=0, weights=relative_weights) where relative_weights is an array with a weight factor for each model.

Answer 2

So let's assume that models is a collection of your models. First - collect all weights:

weights = [model.get_weights() for model in models]

Now - create a new averaged weights:

new_weights = list()

for weights_list_tuple in zip(*weights):
    new_weights.append(
        [numpy.array(weights_).mean(axis=0)\
            for weights_ in zip(*weights_list_tuple)])

And what is left is to set these weights in a new model:

new_model.set_weights(new_weights)

Of course - averaging weights might be a bad idea, but in case you try - you should follow this approach.