gradient-descent

I am reading through the documentation of PyTorch and found an example where they write gradients = torch.FloatTensor([0.1, 1.0, 0.0001]) y.backward(gradients) print(x.grad) where x was an initial variable, from which y was constructed (a 3-vector). The question is, what are the 0.1, 1.0 and 0.0001 arguments of the gradients tensor ? The documentation is not very clear on that. The original code I haven't found on PyTorch website anymore. gradients = torch.FloatTensor([0.1, 1.0, 0.0001]) y.backward(gradients) print(x.grad) The problem with the code above there is no function based on what to

I am trying to use caffe to implement triplet loss described in Schroff, Kalenichenko and Philbin "FaceNet: A Unified Embedding for Face Recognition and Clustering", 2015 . I am new to this so how to calculate the gradient in back propagation? Shai I assume you define the loss layer as layer { name: "tripletLoss" type: "TripletLoss" bottom: "anchor" bottom: "positive" bottom: "negative" ... } Now you need to compute a gradient w.r.t each of the "bottom"s. The loss is given by: The gradient w.r.t the "anchor" input ( fa ): The gradient w.r.t the "positive" input ( fp ): The gradient w.r.t the

Why do we need to explicitly zero the gradients in PyTorch? Why can't gradients be zeroed when loss.backward() is called? What scenario is served by keeping the gradients on the graph and asking the user to explicitly zero the gradients? danche We explicitly need to call zero_grad() because, after loss.backward() (when gradients are computed), we need to use optimizer.step() to proceed gradient descent. More specifically, the gradients are not automatically zeroed because these two operations, loss.backward() and optimizer.step() , are separated, and optimizer.step() requires the just computed

I implemented a gradient descent algorithm to minimize a cost function in order to gain a hypothesis for determining whether an image has a good quality. I did that in Octave. The idea is somehow based on the algorithm from the machine learning class by Andrew Ng Therefore I have 880 values "y" that contains values from 0.5 to ~12. And I have 880 values from 50 to 300 in "X" that should predict the image's quality. Sadly the algorithm seems to fail, after some iterations the value for theta is so small, that theta0 and theta1 become "NaN". And my linear regression curve has strange values...

When we train neural networks, we typically use gradient descent, which relies on a continuous, differentiable real-valued cost function. The final cost function might, for example, take the mean squared error. Or put another way, gradient descent implicitly assumes the end goal is regression - to minimize a real-valued error measure. Sometimes what we want a neural network to do is perform classification - given an input, classify it into two or more discrete categories. In this case, the end goal the user cares about is classification accuracy - the percentage of cases classified correctly.

Sometimes I run into a problem: OOM when allocating tensor with shape e.q. OOM when allocating tensor with shape (1024, 100, 160) Where 1024 is my batch size and I don't know what's the rest. If I reduce the batch size or the number of neurons in the model, it runs fine. Is there a generic way to calculate optimal batch size based on model and GPU memory, so the program doesn't crash? EDIT Since my question might seem unclear, let me put it his way: I want the largest batch size possible in terms of my model, which will fit into my GPU memory and won't crash the program. EDIT 2 To whoever