I’m having troubles with rotation. What I want to do is this:
For a rotation matrix, its transpose is its inverse. So you can just do M.t() * r to move it back to your original frame, where r is a cv::Mat (you might have to convert it to a cv::Mat from a cv::Point2f or whatever, or just write out the matrix multiplication explicitly).
Here's the code to do it explicitly (should be correct, but warning, it's entirely untested):
cv::Point2f p;
p.x = M.at(0, 0) * r.x + M.at(1, 0) * r.y;
p.y = M.at(0, 1) * r.x + M.at(1, 1) * r.y;
// p contains r rotated back to the original frame.