Autograd on a single variable in contructed matrix fails

Question

I want to create a 4x4 matrix to deform my 4 x N tensor-images and optimize this to a loss function. The loss function works but the way I initiali