How do I reverse-project 2D points into 3D?

Question

I have 4 2D points in screen-space, and I need to reverse-project them back into 3D space. I know that each of the 4 points is a corner of a 3D-rotated rigid rectangle, and

Answer 1

If you know the shape is a rectangle in a plane, you can greatly further constrain the problem. You certainly cannot figure out "which" plane, so you can choose that it is lying on the plane where z=0 and one of the corners is at x=y=0, and the edges are parallel to the x/y axis.

The points in 3d are therefore {0,0,0},{w,0,0},{w,h,0},and {0,h,0}. I'm pretty certain the absolute size will not be found, so only the ratio w/h is releavant, so this is one unknown.

Relative to this plane the camera must be at some point cx,cy,cz in space, must be pointing in a direction nx,ny,nz (a vector of length one so one of these is redundant), and have a focal_length/image_width factor of w. These numbers turn into a 3x3 projection matrix.

That gives a total of 7 unknowns: w/h, cx, cy, cz, nx, ny, and w.

You have a total of 8 knowns: the 4 x+y pairs.

So this can be solved.

Next step is to use Matlab or Mathmatica.