backpropagation

问题 How to detect source of vanishing gradients in pytorch? By vanishing gradients, I mean then the training loss doesn't go down below some value, even on limited sets of data. I am trying to train some network, and I have the above problem, in which I can't even get the network to over fit, but can't understand the source of the problem. I've spent a long time googling this, and only found ways to prevent over fitting, but nothing about under fitting, or specifically, vanishing gradients. What

问题 How to detect source of vanishing gradients in pytorch? By vanishing gradients, I mean then the training loss doesn't go down below some value, even on limited sets of data. I am trying to train some network, and I have the above problem, in which I can't even get the network to over fit, but can't understand the source of the problem. I've spent a long time googling this, and only found ways to prevent over fitting, but nothing about under fitting, or specifically, vanishing gradients. What

问题 I'm trying to implement a feed-forward backpropagating autoencoder (training with gradient descent) and wanted to verify that I'm calculating the gradient correctly. This tutorial suggests calculating the derivative of each parameter one at a time: grad_i(theta) = (J(theta_i+epsilon) - J(theta_i-epsilon)) / (2*epsilon) . I've written a sample piece of code in Matlab to do just this, but without much luck -- the differences between the gradient calculated from the derivative and the gradient

问题 I'm trying to implement a feed-forward backpropagating autoencoder (training with gradient descent) and wanted to verify that I'm calculating the gradient correctly. This tutorial suggests calculating the derivative of each parameter one at a time: grad_i(theta) = (J(theta_i+epsilon) - J(theta_i-epsilon)) / (2*epsilon) . I've written a sample piece of code in Matlab to do just this, but without much luck -- the differences between the gradient calculated from the derivative and the gradient

问题 I'm optimising my implementation of the back-propagation algorithm to train a neural network. One of the aspects I'm working on is performing the matrix operations on the set of datapoints (input/output vector) as a batch process optimised by the numpy library instead of looping through every datapoint. In my original algorithm I did the following: for datapoint in datapoints: A = ... (created out of datapoint info) B = ... (created out of datapoint info) C = np.dot(A,B.transpose()) _________

问题 I have to design an animated visual representation of a neural network that is functional (i.e. with UI that allows you to tweak values etc). The primary goal with it is to help people visualize how and when the different math operations are performed in a slow-motion, real-time animation. I have the visuals set up along with the UI that allows you to tweak values and change the layout of the neurons, as well as the visualizations for the feed forward stage, but since I don’t actually

问题 I'm trying to implement gradient checking for a simple feedforward neural network with 2 unit input layer, 2 unit hidden layer and 1 unit output layer. What I do is the following: Take each weight w of the network weights between all layers and perform forward propagation using w + EPSILON and then w - EPSILON. Compute the numerical gradient using the results of the two feedforward propagations. What I don't understand is how exactly to perform the backpropagation. Normally, I compare the

问题 I am trying to work out question 26 from this exam paper (the exam is from 2002, not one I'm getting marked on!) This is the exact question: The answer is B. Could someone point out where I'm going wrong? I worked out I1 from the previous question on the paper to be 0.982. The activation function is sigmoid. So should the sum be, for output 1: d1 = f(Ik)[1-f(Ik)](Tk-Zk) From the question: T1 = 0.58 Z1 = 0.83 T1 - Z1 = -0.25 sigmoid(I1) = sigmoid(0.982) = 0.728 1-sigmoid(I1) = 1-0.728 = 0.272

问题 I am trying to work out question 26 from this exam paper (the exam is from 2002, not one I'm getting marked on!) This is the exact question: The answer is B. Could someone point out where I'm going wrong? I worked out I1 from the previous question on the paper to be 0.982. The activation function is sigmoid. So should the sum be, for output 1: d1 = f(Ik)[1-f(Ik)](Tk-Zk) From the question: T1 = 0.58 Z1 = 0.83 T1 - Z1 = -0.25 sigmoid(I1) = sigmoid(0.982) = 0.728 1-sigmoid(I1) = 1-0.728 = 0.272

问题 When I have been attempting to implement this function tf.train.stg(learningRate).minimize(loss) into my code in order to conduct back-propagation. I have been getting multiple errors such The f passed in variableGrads(f) must be a function . How would I implement the function above into the code bellow successfully? and Why does this error even occur? Neural Network: var X = tf.tensor([[1,2,3], [4,5,6], [7,8,9], [10,11,12]]) var Y = tf.tensor([[0,0,0],[0,0,0], [1,1,1]]) var m = X.shape[0]