How can you rectify cropped stereo images in OpenCV?

Question

I have a pair of horizontally aligned stereo cameras which were calibrated using the full size of the images.

I'm rectifying by calling cv2.initUndistortRectifyMap to get the map for each camera, and then cv2.remap

When using the full size images, this looks like:

map1, map2 = cv2.initUndistortRectifyMap(camera_matrix_1, dist_coeffs_1, R1, P1, (w, h), cv2.CV_16SC2)
map3, map4 = cv2.initUndistortRectifyMap(camera_matrix_2, dist_coeffs_2, R2, P2, (w, h), cv2.CV_16SC2)
rectified1 = cv2.remap(img1, map1, map2, cv2.INTER_LINEAR)
rectified2 = cv2.remap(img2, map3, map4, cv2.INTER_LINEAR)

Where the parameters for cv2.initUndistortRectifyMap are the output of cv2.stereoCalibrate and cv2.stereoRectify

For improving processing speed however, the images will be cropped (and potentially binned) when processing, before they can be rectified. Which means that the rectification process can't use the full size of the image.

What needs to be changed to get the code to work with cropping (and binning) of the images before the rectification is done?

binned? That won't affect anything. Rectification doesn't rescale pixel values. As for cropping, rectification I think rescales images so you're going to really have to examine the math to figure out how to compensate for that. I suspect this would involve reading Hartley and Zisserman and probably reading opencv source code. I would recommend 'uncropping' by padding with white, then cropping again after rectification — kevinkayaks

mr NAE mr NAE · Accepted Answer · 2019-01-09T12:11:53

If you want to crop and resize (scale) the original img1 and img2 and use it after calibration, then you need to change camera_matrix_1 and camera_matrix_2. Cropping changes a principal point, the scale changes the focal length. But the result (output) of rectified1 and rectified2 will not change its size:w, h. If you want a smaller output image, you need to change P1 and P2.

z 0 sx
0 z sy * camera_matrix  ->  camera_matrix'
0 0 1

Were z - scale factor, s - shift of cropping image center. Scale and shift could be different for camera_matrix and P. Don't forget to re-create map1, map2... using initUndistortRectifyMap with the new parameters.

How can you rectify cropped stereo images in OpenCV?

1 Answers