How do I reverse-project 2D points into 3D?

Question

I have 4 2D points in screen-space, and I need to reverse-project them back into 3D space. I know that each of the 4 points is a corner of a 3D-rotated rigid rectangle, and

Answer 1

When you project from 3D to 2D you lose information.

In the simple case of a single point the inverse projection would give you an infinite ray through 3d space.

Stereoscopic reconstruction will typically start with two 2d images and project both back to 3D. Then look for an intersection of the two 3D rays produced.

Projection can take different forms. Orthogonal or perspective. I'm guessing that you are assuming orthogonal projection?

In your case assuming you had the original matrix you would have 4 rays in 3D space. You would then be able to constrain the problem by your 3d rectangle dimensions and attempt to solve.

The solution will not be unique as a rotation around either axis that is parallel to the 2d projection plane will be ambiguous in direction. In other words if the 2d image is perpendicular to the z axis then rotating the 3d rectangle clockwise or anti clockwise around the x axis would produce the same image. Likewise for the y axis.

In the case where the rectangle plane is parallel to the z axis you have even more solutions.

As you don't have the original projection matrix further ambiguity is introduced by an arbitary scaling factor that exists in any projection. You cannot distinguish between a scaling in the projection and a translation in 3d in the direction of the z axis. This is not a problem if you are only interested in the relative positions of the 4 points in 3d space when related to each other and not to the plane of the 2d projection.

In a perspective projection things get harder...