OpenCV stereo calibration has low re-projection error but is incorrect

Question

I have been trying to get OpenCV camera calibration working for several months now, but I have been unsuccessful so far. I am at the point where I can get a low re-projection error (below 0.5 pixels), but when I examine the images output by stereoRectify they are clearly not rectified correctly. Corresponding points are not aligned horizontally. The noticeable effects of the rectification are to rotate one image slightly and zoom the other image in. Disparity maps generated by the rectification are noisy and simply not correct (using both StereoSGBM and StereoVar).

I have tried using several different sets of calibration images, ranging from 20 to 80 frames. I have tried various combinations of checkerboard size, distance, and rotation, all to no avail. In each case I get the different results, but all with the type of useless disparity maps described above.

MATLAB stereo calibration produces much better results on the same videos, but MATLAB is unfortunately not available as a long-term solution in my case. I do not understand why OpenCV believes it's finding a good stereo rectification when it so clearly is not. Am I missing something about how to calibrate cameras? Has anyone else encountered such an issue?

Answer 1

Given that it's working in MATLAB I assume your stereo cameras are set up correctly, meaning they are completely fixed. What comes to my mind is: Did you calibrate the cameras individually? So you can use the camera matrix and the distortion coefficients in stereoRectify from there. Also, I noted that findChessboardCorners does not perform well when the image quality is low. But that should result in a high RMS error. In any case, you should not only have a reasonably high number of pictures (I used 100) but also have different angles of the calibration board to the camera. And the distance of the board to the cameras should not be too high so the board covers most of the image. (Since the board needs to be completely visible for both cameras, I used a preview to adjust the position.) That's all not really answering your question but I hope it helps nevertheless.

Answer 2

Are you calibrating using the same images in both MATLAB and OpenCV? Also, are you using the Stereo Camera Calibrator app in MATLAB?

If you are using the exact same calibration images, then the likely explanation for what you are seeing is that the checkerboard detector in MATLAB is more accurate then the one in OpenCV.

If you are using different calibration images, then I would try undistorting the images in OpenCV using the undistort function instead of rectifying them. Chance are that that the undistorted images will look terribly warped and folded upon themselves. That, together with the low reprojection errors, would indicate that you do not get a good estimate of lens distortion. Usually that happens when your checkerboard is mostly in the middle of the image, and you are not getting any points close to the edges. The remedy here would be to get more images with the checkerboard positioned close the edges and corners of the image.