Converting a 2D image point to a 3D world point

Question

I know that in the general case, making this conversion is impossible since depth information is lost going from 3d to 2d.

However, I have a fixed camera and I know

Answer 1

Here is a closed form solution that I hope can help someone. Using the conventions in the image from your comment above, you can use centered-normalized pixel coordinates (usually after distortion correction) u and v, and extrinsic calibration data, like this:

|Tx|   |r11 r21 r31| |-t1|
|Ty| = |r12 r22 r32|.|-t2|
|Tz|   |r13 r23 r33| |-t3|

|dx|   |r11 r21 r31| |u|
|dy| = |r12 r22 r32|.|v|
|dz|   |r13 r23 r33| |1|

With these intermediate values, the coordinates you want are:

X = (-Tz/dz)*dx + Tx
Y = (-Tz/dz)*dy + Ty

Explanation:

The vector [t1, t2, t3]^t is the position of the origin of the world coordinate system (the (0,0) of your calibration pattern) with respect to the camera optical center; by reversing signs and inversing the rotation transformation we obtain vector T = [Tx, Ty, Tz]^t, which is the position of the camera center in the world reference frame.

Similarly, [u, v, 1]^t is the vector in which lies the observed point in the camera reference frame (starting from camera center). By inversing the rotation transformation we obtain vector d = [dx, dy, dz]^t, which represents the same direction in world reference frame.

To inverse the rotation transformation we take advantage of the fact that the inverse of a rotation matrix is its transpose (link).

Now we have a line with direction vector d starting from point T, the intersection of this line with plane Z=0 is given by the second set of equations. Note that it would be similarly easy to find the intersection with the X=0 or Y=0 planes or with any plane parallel to them.

Answer 2

If you have Z=0 for you points in world coordinates (which should be true for planar calibration pattern), instead of inversing rotation transformation, you can calculate homography for your image from camera and calibration pattern.

When you have homography you can select point on image and then get its location in world coordinates using inverse homography. This is true as long as the point in world coordinates is on the same plane as the points used for calculating this homography (in this case Z=0)

This approach to this problem was also discussed below this question on SO: Transforming 2D image coordinates to 3D world coordinates with z = 0

Answer 3

Yes, you can. If you have a transformation matrix that maps a point in the 3d world to the image plane, you can just use the inverse of this transformation matrix to map a image plane point to the 3d world point. If you already know that z = 0 for the 3d world point, this will result in one solution for the point. There will be no need to iteratively choose some random 3d point. I had a similar problem where I had a camera mounted on a vehicle with a known position and camera calibration matrix. I needed to know the real world location of a lane marking captured on the image place of the camera.