I\'m using opencv and openframeworks (ie. opengl) to calculate a camera (world transform and projection matrixes) from an image (and later, several images for triangulation)
I think you should not take the inverse of gWorldToCalibration
ofMatrix4x4 gCalibrationToWorld = gWorldToCalibration.getInverse();
Here I posted code, which is doing more or less what you want OpenCV- to OpenGL COS. It's in C, but should be similar in C++.