Stereo Calibration and 3D Reconstruction Issue

2
votes

I am trying to perform stereo camera calibration and 3D reprojection of a calculated disparity image.
For the sake of simplicity I made this example in Matlab but getting the same result in Python and OpenCV.

I use this piece of code for the calibration:

% Detect checkerboards in images
[imagePoints, boardSize, imagesUsed] = detectCheckerboardPoints(imageFileNames_left, imageFileNames_right);

% Generate world coordinates of the checkerboard keypoints
squareSize = 108;  % in units of 'millimeters'
worldPoints = generateCheckerboardPoints(boardSize, squareSize);

% Read one of the images from the first stereo pair
I1 = imread(imageFileNames_left{1});
[mrows, ncols, ~] = size(I1);

% Calibrate the camera
[stereoParams, pairsUsed, estimationErrors] = estimateCameraParameters(imagePoints, worldPoints, ...
    'EstimateSkew', false, 'EstimateTangentialDistortion', true, ...
    'NumRadialDistortionCoefficients', 3, 'WorldUnits', 'millimeters', ...
    'InitialIntrinsicMatrix', [], 'InitialRadialDistortion', [], ...
    'ImageSize', [mrows, ncols]);

The image_FileNames contains the paths to the respective calibration images. I use a checkerboard calibration pattern printed on a rigid panel of size A0.

The resluts of the calibration look very promising:

However calculating the disparity and reprojecting it to 3D gives weird results:

Link to reprojected point cloud image

I used the following code snippet for the projection to 3D:

I1 = imread(strcat(dirs_left{i}, '/', names_left{i}));
I2 = imread(strcat(dirs_right{i}, '/', names_right{i}));

% Rectify using calibration data
[J1, J2] = rectifyStereoImages(I1,I2,stereoParams);

% Calculate Disparity
disparityRange = [0 128];
disparityMap = disparitySGM(rgb2gray(J1),rgb2gray(J2),'DisparityRange', disparityRange);

% Project to 3D
point3D = reconstructScene(disparityMap, stereoParams);

% Convert from millimeters to meters.
point3D = point3D ./ 1000;

% Visualize the 3-D Scene
ptCloud = pointCloud(point3D, 'Color', J1);
h1 = figure; pcshow(ptCloud);

The corresponding disparity image (Calculated using SGBM) looks great:

Link to disparity image

And here is the calibration data: (Attention: OpenCV notation, I transposed all matrices to be compatible with OpenCV):

camera matrix 0:
 - 2362.9276056, 0.0000006, 1034.4700766,
 - 0.0000006, 2366.4078916, 728.4543626,
 - 0.0000006, 0.0000006, 1.0000006,

camera matrix 1:
 - 2366.2683866, 0.0000006, 1030.6057166,
 - 0.0000006, 2366.4804296, 740.0748076,
 - 0.0000006, 0.0000006, 1.0000006,

lens dist 0: ()
-0.201011, 0.094025, -0.000569, 0.000521, 0.252866

lens dist 1:
-0.191647, 0.046607, -0.000569, 0.000521, 0.205665

rotation matrix camera 1:
 - 0.9994606, -0.0068816, 0.0321416,
 - 0.0055786, 0.9991666, 0.0404456,
 - -0.0323936, -0.0402446, 0.9986656,

translation vector camera 1:
 - -485.0037626, -48.0975216, 48.2236646,

I get a similar point cloud using OpenCV instead of Matlab for the re-projection, the above calibration info.

Also I found this question, where they seem to get a similar output after the projection. But in my case all the reprojection errors are rather small (0.26pixels in average) and the disparity range is set by default to 0-128, which is the maximum, and the disparity map looks great.

Any ideas?

1
matlabstereo-3dcalibration3d-reconstructionstereoscopy
Did you try to zoom into the point cloud? Currently it is on the order of kilometers, but the data should be visible on the order of tens of meters. The rest might be just noise – fdermishin
The baseline as defined by the translation vector in mm (The cameras are around 485mm apart) . So in my understanding, this is also the metric that will be used for the projection, which is why in this line: ``` % Convert from millimeters to meters. point3D = point3D ./ 1000; ``` I convert from mm to m, which should render the point cloud in matter of meters, Where do you see the order of kilometers? Thanks for the reply :) – schurist
Thank you for clarification. I thought that the scale was in order of kilometers because the max value on the vertical axis of reprojected point cloud image is 2000, and according to the code, you are displaying the point cloud after conversion to meters. So I assumed that the most distant points were 2000 meters apart – fdermishin

1 Answers

1
votes

The predicted camera paramters match the physical properties of the camera lens used. This led me to limiting the projection space, basically as fdermishin suggested. Removing all points outside a 45 m distance gave reasonable results!

I used the following piece of code to filter the points:

roi = [-50 50 -10 40 0 45]; % filter x, y, and z
indices = findPointsInROI(ptCloud,roi);
ptCloud_clean = select(ptCloud,indices);