OpenCV unproject 2D points to 3D with known depth `Z`

Question

Problem statement

I am trying to reproject 2D points to their original 3D coordinates, assuming I know the distance at which each point is. Following the OpenCV do

Answer 1

Answer to Question 2

I found what the problem was -- The 3D point coordinates matter! I assumed that no matter what 3D coordinate points I choose, the reconstruction would take care of it. However, I noticed something strange: when using a range of 3D points, only a subset of those points were reconstructed correctly. After further investigation, I found out that only the images that are in the field of view of the camera would be properly reconstructed. The field-of-view is the function of the intrinsic parameters (and vice-versa).

For the above codes to work, try setting the parameters as follows (intrinsics are from my camera):

...
const double f_x = 2746.;
const double f_y = 2748.;
const double c_x = 991.;
const double c_y = 619.;
...
const cv::Point3d point_single(10.0, -2.0, 30.0);
...

Also, don't forget that in camera coordinates negative y coordinates is UP :)

Answer to Question 1:

There was a bug where I was trying to access the intrinsics using

...
double f_x = intrinsic.at(0, 0);
double f_y = intrinsic.at(1, 1);
double c_x = intrinsic.at(0, 3);
double c_y = intrinsic.at(1, 3);
...

But intrinsic was a 3x3 matrix.

Moral of the story Write unit tests!!!