Why am i getting a high Reprojection Error while Stereo calibration a RGB and IR camera?

Question

I have written a C++ program to stereo calibrate the camera, initially the Reprojection error was > 5, then deleted few images and was reduced to 2.3, now i have taken ne