How do I reverse-project 2D points into 3D?

Question

I have 4 2D points in screen-space, and I need to reverse-project them back into 3D space. I know that each of the 4 points is a corner of a 3D-rotated rigid rectangle, and

Answer 1

D. DeMenthon devised an algorithm to compute the pose of an object (its position and orientation in space) from feature points in a 2D image when knowing the model of the object -- this is your exact problem:

We describe a method for finding the pose of an object from a single image. We assume that we can detect and match in the image four or more noncoplanar feature points of the object, and that we know their relative geometry on the object.

The algorithm is known as Posit and is described in it classical article "Model-Based Object Pose in 25 Lines of Code" (available on its website, section 4).

Direct link to the article: http://www.cfar.umd.edu/~daniel/daniel_papersfordownload/Pose25Lines.pdf OpenCV implementation: http://opencv.willowgarage.com/wiki/Posit

The idea is to repeatedly approximating the perspective projection by a scaled orthographic projection until converging to an accurate pose.