Average layer in multi input deep learning

问题

I am working to create a multi-input Convolutional Neural Network (CNN) model in Keras for Images Classification that takes two images and gives one output which is the class of the two images.

I have two datasets: type1 and type2, and each dataset contains the same classes. The model should take one image from Type1 dataset and one image from Type2 dataset and then classify these images to one class (ClassA or ClassB or------).

I want to create a model that predict the two images and then compute the average of the prediction similar to the following image:

How can I create this model? How can I create the generator in fit_generator?

回答1:

Option 1 - Both sides are the same model, just using different inputs

Suppose you have a model that goes up to "predication", called predModel.
Create two input tensors:

input1 = Input(shape)   
input2 = Input(shape)

Get the outputs for each input:

pred1 = predModel(input1)
pred2 = predModel(input2)

Average the outputs:

output = Average()([pred1,pred2])

Create the final model:

model = Model([input1,input2], output)

Option2 - Both sides are similar models, but use different weights

Basically the same as above, but create the layers individually for each side.

def createCommonPart(inputTensor):
    out = ZeroPadding2D(...)(inputTensor)
    out = Conv2D(...)(out)

    ...
    out = Flatten()(out)
    return Dense(...)(out)

Make the two inputs:

input1 = Input(shape)   
input2 = Input(shape)

Get the two outputs:

pred1 = createCommonPart(input1)
pred2 = createCommonPart(input2)

Average the outputs:

output = Average()([pred1,pred2])

Create the final model:

model = Model([input1,input2], output)

The generator

Anything that yields [xTrain1,xTrain2], y.

You can create one like this:

def generator(files1,files2, batch_size):

    while True: #must be infinite

        for i in range(len(files1)//batch_size)):
            bStart = i*batch_size
            bEnd = bStart+batch_size

            x1 = loadImagesSomehow(files1[bStart:bEnd])
            x2 = loadImagesSomehow(files2[bStart:bEnd])
            y = loadPredictionsSomeHow(forSamples[bStart:bEnd])

            yield [x1,x2], y

You can also implement a keras.utils.Sequence in a similar way.

class gen(Sequence):
    def __init__(self, files1, files2, batchSize):
        self.files1 = files1
        self.files2 = files2
        self.batchSize = batchSize

    def __len__(self):
        return self.len(files1) // self.batchSize

    def __getitem__(self,i):

        bStart = i*self.batchSize
        bEnd = bStart+self.batchSize 

        x1 = loadImagesSomehow(files1[bStart:bEnd])
        x2 = loadImagesSomehow(files2[bStart:bEnd])
        y = loadPredictionsSomeHow(forSamples[bStart:bEnd])

        return [x1,x2], y

来源：https://stackoverflow.com/questions/53545316/average-layer-in-multi-input-deep-learning

标签

tensorflow

machine-learning

keras

neural-network

deep-learning