Given a stereo disparity map in OpenCV, how do I find true pixel offsets between left and right images (per-pixel)?

Question

I have successfuly created a satisfying disparity map using cv::stereoSGBM class in OpenCV 2.4.5, with 256 different disparity levels. I have divided all the disparity values by 16, as indicated in the documentation, but this final map can't possibly contain the "true" disparities: if I add to a (x, y) pixel's horizontal dimension its computed disparity (x + d), the resulting coordinates (x + d, y) in the other image correspond to a completely different pixel.

I guess the problem stems from the fact that I parameterized stereoSGBM initially for 256 disparity levels, but the actual maximum disparity (in pixels) is way smaller. Since I don't really know the actual maximum disparity, I can't just normalize accordingly the values computed by stereoSGBM.

I need the actual disparity values to compute dense stereo correspondences (necessary for triangulation and 3D reconstruction).

Answer 1

I had the same problem and this is how I fixed it:

Your actual (raw) disparity is what the StereoSGBM class gives as output. You scale it only so that you can see it in an image format.

In order to obtain corresponding pixels of the left and the right images, use the disparity matrix given by the StereoSGBM class as is. That means, you should not be scaling it to view in the image format. Then you'll get the corresponding points properly.

But be careful when you're accessing these values. The data type is _int16. You'll run into exceptions if you use the wrong data type.

This is what you'll do for corresponding points: Suppose I1 is your left image and I2 is your right image, and d is the disparity value at some row i and column j, then, I1(i,j) and I2(i,j-d) will be the corresponding points, assuming horizontal stereo.

Answer 2

I did some tests and the true disparity is given to you when you divide the output of SGBM by 16. Here you see two histograms of the disparity output for -128 to 128 and -128 to 256 disparity: (the displayed disparity is divided by 16)

You see that the middle area between -50 and 128 is equivalent. So the values for num_disparity and min_disparity doesn't affect the scaled output of SGBM.

Note that the disparity is scaled by the baseline and focal length of the used cameras. So if you use several disparity maps from multiple stereo-pair images, every single disparity map is scaled by the baseline of this pair and if different cameras were used by the specific focal lengths.