LSTM on top of a pre-trained CNN

问题

I have trained a CNN and now I want to load the model and then on top put a LSTM but I'm getting some errors.

'''
Load the output of the CNN
'''
cnn_model = load_model(os.path.join('weights', 'CNN_patch_epoch-20.hdf5'))

last_layer = cnn_model.get_layer('pool5').output    

''' Freeze previous layers '''
for layer in cnn_model.layers:
    layer.trainable = False

x = TimeDistributed(Flatten())(last_layer)
x = LSTM(neurons, dropout=dropout, name='lstm')(x)
out = Dense(n_output, kernel_initializer=weight_init, name='out')(x)

model = Model(inputs=[cnn_model.input], outputs=out)

model.summary() 

I'm not sure where to specify that I want 5 frames (images). So my input would be (None, 5, 224, 224, 3). So my question is where should I specify it?

Thanks

回答1:

You can wrap your cnn_model also in a TimeDistributed wrapper.

frames = Input(shape=(5, 224, 224, 3))
x = TimeDistributed(cnn_model)(frames)
x = TimeDistributed(Flatten())(x)
x = LSTM(neurons, dropout=dropout, name='lstm')(x)
out = Dense(n_output, kernel_initializer=weight_init, name='out')(x)
model = Model(inputs=frames, outputs=out)

来源：https://stackoverflow.com/questions/49535488/lstm-on-top-of-a-pre-trained-cnn