OpenCV Stereo Image Rectification doesn't work properly when the angle between cameras is too high

Question

I want to collect height data about a body sitting at the focus of the two cameras, this is what my stereo set up looks like:

When I calculate the rectified versions of the images using the standard cv2 functions it looks very bad. When I used a similar setup with the cameras in parallel it worked.

I calculated the epilines, and they seem to be right:

However, the resulting rectified images are not (below, the original images are above):

This is the code I used, it was mostly copied from an openCV tutorial

http://www.dmi.unict.it/~furnari/teaching/CV1617/lab1/

The 3d/2d points and the matrices were calculated previously using

cv2.calibrateCamera()

Calibration and Rectification Process

retval, _, _, _, _, R, T, E, F = cv2.stereoCalibrate(imager._3d_points, _2d_points_L,_2d_points_R, mtxL, distL, mtxR, distR, (img.shape[1], img.shape[0])

R1, R2, P1, P2, Q, _, _ = cv2.stereoRectify(mtxL, distL, mtxR, distR, (img.shape[1], img.shape[0]), R, T, )

map1_x, map1_y = cv2.initUndistortRectifyMap(imager.mtxL, imager.distL, R1, P1, (imLeft.shape[1], imLeft.shape[0]), cv2.CV_32FC1)
map2_x, map2_y = cv2.initUndistortRectifyMap(imager.mtxR, imager.distR, R2, P2, (imLeft.shape[1], imLeft.shape[0]), cv2.CV_32FC1)

imgL = cv2.remap(imLeft, map1_x, map1_y , cv2.INTER_CUBIC)
imgR = cv2.remap(imRight, map2_x, map2_y , cv2.INTER_CUBIC)

Is OpenCV simply not able to rectify images with my camera setup or did I do something wrong?

Piotr Siekański Piotr Siekański · Accepted Answer · 2019-07-15T13:12:27

As far as I know, the function stereoRectify assumes that the camera axes are approximately parallel to each other. You can try stereoRectifyUncalibrated, as this function relies on fundamental matrix computation and will handle this case. Make sure you've read the note in the documentation.

Note:

While the algorithm does not need to know the intrinsic parameters of the cameras, it heavily depends on the epipolar geometry. Therefore, if the camera lenses have a significant distortion, it would be better to correct it before computing the fundamental matrix and calling this function. For example, distortion coefficients can be estimated for each head of stereo camera separately by using calibrateCamera(). Then, the images can be corrected using undistort(), or just the point coordinates can be corrected with undistortPoints().

OpenCV Stereo Image Rectification doesn't work properly when the angle between cameras is too high

1 Answers