backpropagation

I am trying to implement a two-layer perceptron with backpropagation to solve the parity problem. The network has 4 binary inputs, 4 hidden units in the first layer and 1 output in the second layer. I am using this for reference, but am having problems with convergence. First, I will note that I am using a sigmoid function for activation, and so the derivative is (from what I understand) the sigmoid(v) * (1 - sigmoid(v)). So, that is used when calculating the delta value. So, basically I set up the network and run for just a few epochs (go through each possible pattern -- in this case, 16

I think I do not understand the multiple-output networks. Althrough i understand how the implementation is made and i succesfully trained one model like this, i don't understand how a multiple-outputs deep learning network is trained. I mean, what is happening inside the network during training? Take for example this network from the keras functional api guide : You can see the two outputs (aux_output and main_output). How is the backpropagation working? My intuition was that the model does two backpropagations, one for each output. Each backpropagation then updates the weight of the layers

For a neural networks library I implemented some activation functions and loss functions and their derivatives. They can be combined arbitrarily and the derivative at the output layers just becomes the product of the loss derivative and the activation derivative. However, I failed to implement the derivative of the Softmax activation function independently from any loss function. Due to the normalization i.e. the denominator in the equation, changing a single input activation changes all output activations and not just one. Here is my Softmax implementation where the derivative fails the

I have a quick question regarding backpropagation. I am looking at the following: http://www4.rgu.ac.uk/files/chapter3%20-%20bp.pdf In this paper, it says to calculate the error of the neuron as Error = Output(i) * (1 - Output(i)) * (Target(i) - Output(i)) I have put the part of the equation that I don't understand in bold. In the paper, it says that the Output(i) * (1 - Output(i)) term is needed because of the sigmoid function - but I still don't understand why this would be nessecary. What would be wrong with using Error = abs(Output(i) - Target(i)) ? Is the error function regardless of the

I posted a question yesterday regarding issues that I was having with my backpropagating neural network for the XOR operator. I did a little more work and realized that it may have to do with not having a bias neuron. My question is, what is the role of the bias neuron in general, and what is its role in a backpropagating neural network that recognizes the XOR operator? Is it possible to create one without a bias neuron? Kiril It's possible to create a neural network without a bias neuron... it would work just fine, but for more information I would recommend you see the answers to this

is it possible to define a TensorFlow graph with more than one input? For instance, I want to give the graph two images and one text, each one is processed by a bunch of layers with a fc layer at the end. Then there is a node that computes a lossy function that takes into account the three representations. The aim is to let the three nets to backpropagate considering the joint representation lossy. Is it possible? any example/tutorial about it? thanks in advance! lejlot This is completely straight forward thing. For "one input" you would have something like: def build_column(x, input_size): w