How do I reverse-project 2D points into 3D?

Question

I have 4 2D points in screen-space, and I need to reverse-project them back into 3D space. I know that each of the 4 points is a corner of a 3D-rotated rigid rectangle, and

Answer 1

For my OpenGL engine, the following snip will convert mouse/screen coordinates into 3D world coordinates. Read the commments for an actual description of what is going on.

/*   FUNCTION:        YCamera :: CalculateWorldCoordinates
     ARGUMENTS:       x         mouse x coordinate
                      y         mouse y coordinate
                      vec       where to store coordinates
     RETURN:          n/a
     DESCRIPTION:     Convert mouse coordinates into world coordinates
*/

void YCamera :: CalculateWorldCoordinates(float x, float y, YVector3 *vec)
{
    //  START
    GLint viewport[4];
    GLdouble mvmatrix[16], projmatrix[16];
    
    GLint real_y;
    GLdouble mx, my, mz;

    glGetIntegerv(GL_VIEWPORT, viewport);
    glGetDoublev(GL_MODELVIEW_MATRIX, mvmatrix);
    glGetDoublev(GL_PROJECTION_MATRIX, projmatrix);

    real_y = viewport[3] - (GLint) y - 1;   // viewport[3] is height of window in pixels
    gluUnProject((GLdouble) x, (GLdouble) real_y, 1.0, mvmatrix, projmatrix, viewport, &mx, &my, &mz);

    /*  'mouse' is the point where mouse projection reaches FAR_PLANE.
        World coordinates is intersection of line(camera->mouse) with plane(z=0) (see LaMothe 306)
        
        Equation of line in 3D:
            (x-x0)/a = (y-y0)/b = (z-z0)/c      

        Intersection of line with plane:
            z = 0
            x-x0 = a(z-z0)/c  <=> x = x0+a(0-z0)/c  <=> x = x0 -a*z0/c
            y = y0 - b*z0/c
            
    */
    double lx = fPosition.x - mx;
    double ly = fPosition.y - my;
    double lz = fPosition.z - mz;
    double sum = lx*lx + ly*ly + lz*lz;
    double normal = sqrt(sum);
    double z0_c = fPosition.z / (lz/normal);
    
    vec->x = (float) (fPosition.x - (lx/normal)*z0_c);
    vec->y = (float) (fPosition.y - (ly/normal)*z0_c);
    vec->z = 0.0f;
}