To calculate world coordinates from screen coordinates with OpenCV

Question

I have calculated the intrinsic and extrinsic parameters of the camera with OpenCV. Now, I want to calculate world coordinates (x,y,z) from screen coordinates (u,v).

Answer 1

Edit 2 (On the coordinate system used):

This is a convention and I think it depends also on which drivers you use and the kind of data you get back. Check for example that, that and that one.

if you for instance use microsoft sdk: then Z is not the distance to the camera but the "planar" distance to the camera. This might change the appropriate formulas.