Creating Neural Network for un-encountered inputs

Question

I am creating a simple Multi-layered feed forward Neural Network using AForge.net NN library. My NN is a 3 Layered Activation Network trained with Supervised Learning approach using BackPropogation Learning algorithm.

Following are my initial settings:

//learning rate
learningRate=0.1;

//momentum value
momentum=0;

//alpha value for bipolar sigmoid activation function
sigmoidAlphaValue=2.0;

//number of inputs to network
inputSize=5;

//number of outputs from network
predictionSize=1;

//iterations
iterations=10000;


// create multi-layer neural network
            ActivationNetwork network = new ActivationNetwork(new BipolarSigmoidFunction

(sigmoidAlphaValue), 5, 5 + 1, 3, 1);

//5 inputs
//6 neurons in input layer
//3 neurons in hidden layer
//1 neuron in output layer

// create teacher
BackPropagationLearning teacher = new BackPropagationLearning(network);

// set learning rate and momentum
teacher.LearningRate = learningRate;
teacher.Momentum = momentum;

Now I have some input series which looks like this, 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20

Using window sliding method (as described here) for input as a time series, my input and

expected output array looks something like this

//Iteration #1
double input[][] = new input[0][5] {1,2,3,4,5};
double output[][] = new output[0][0] {6};

//Iteration #2
double input[][] = new input[1][5] {2,3,4,5,6};
double output[][] = new output[1][0] {7};

//Iteration #3
double input[][] = new input[2][5] {3,4,5,6,7};
double output[][] = new output[2][0] {8};
.
.
.
//Iteration #n
double input[][] = new input[n][5] {15,16,17,18,19};
double output[][] = new output[n][0] {20};

After 10k iterations as such using

teacher.RunEpoch(input, output);

my network is successfully trained for the given training set. So now, if I compute using inputs as 4,5,6,7,8 the network successfully gives 9 as answer fantastic!

However, when the input is provided as 21,22,23,24,25 the NN fails to produce 26!

My Question: How do I train my network to accept such unencountered inputs of such fashion to produce a correct sequence pattern as found in training set during learning?

Answer 1

As @runDOSrun said, your model seems to be overfitting the (training) data. To avoid this problem you could split your set (time series) in 3 parts.

Training Set

The first one could be the training set where you just train your network.

Validation Set

The second one is the validation set, where for each epoch of training, you test the neural network on the validation set and take the error and store this error in a variable and a copy of the neural network (clone). On the next epoch, you have to test the (modified) neural network and if the new error on the validation set is lower than the last you have tested, you store a new "validation neural network". It will provide for you a neural network which generalize better in a set which is not the training set. So you avoid to overfit the training set.

At the end of training you have two neural networks. The training neural network which is the best neural network for training set and the validation neural network, which can provide for you a neural network which generalize better out of training set.

Test Set

This last part, you just test your model in a unseen set and check the errors. The propose for test set is to check the behaviour of the neural network in the unseen test. A real test.

In general, you could slipt the all entire set in 3 equals parts, or for sample

• 60% for training
• 20% for validation
• 20% for test

For sample, look the image bellow:

A sample of pseudo-code of how implement it could be:

int epochs = 1;
double error = 0;
double validationError = 10000;
object validationNetwork;
do
{
    // train your network

    error = getError(trainingSet);

    //validation part...

    var currentValidationError = getError(validationSet);
    if (currentValidationError < validationError)
    {
       validationError = currentValidationError;
       validationNeuralNetwork = neuralNetwork.Clone();
    }

} while (epochs < 2000 && error < 0.001);

Cross-Validation for Time Series

On the other hand, you also could try to do the cross-valdidation for time series. First you split your set in 6 parts (or more) and train neural networks to validate the model like this:

• 1 : training [1], validate [2], test [3]
• 2 : training [2], validate [3], test [4]
• 3 : training [3], validate [4], test [5]
• 4 : training [4], validate [5], test [6]

You could split in more parts if you want.