Does OpenCV's undistortPoints also rectify them?

Question

I'm trying to determine the distance between two object by using OpenCV 2.3.1 with two cameras but can't calculate the object's objectPoints (OCV 2.3.1, MSVC++, Windows 7). I think this is because the image points aren't being rectified before their disparity is calculated.

I. WHAT I DO FIRST

Step 1. Calibrate each camera by itself

int numSquares = numCornersHor * numCornersVer;
Size board_sz = Size(numCornersHor, numCornersVer);

Mat cameraMatrix = Mat(3, 3, CV_32FC1);
Mat distCoeffs;

vector<Mat> rvecs, tvecs;

cameraMatrix.ptr<float>(0)[0] = 1;
cameraMatrix.ptr<float>(1)[1] = 1;

calibrateCamera(object_points, 
       image_points, 
       image.size(), 
       cameraMatrix, distCoeffs, 
       rvecs, tvecs);

Step 2. Calibrate the cameras together

int numCornersHor = 4;
int numCornersVer = 3;
const float squareSize = 1.75;

Size imageSize = Size(numCornersHor, numCornersVer);
int numSquares = numCornersHor * numCornersVer;

for(int i = 0; i < pairs; i++ )
{
for( int j = 0; j < imageSize.height; j++ )
{
    for( int k = 0; k < imageSize.width; k++ )
    {
                    objectPoints[i].push_back(Point3f(j*squareSize, k*squareSize, 0));
    }
}
}

Mat R, T, E, F;

rms = stereoCalibrate(  objectPoints, 
                          imagePoints[0],   imagePoints[1],
        cameraMatrix[0],    distCoeffs[0],
        cameraMatrix[1],    distCoeffs[1],
        imageSize, 
        R,  T,  E,  F,
        TermCriteria(CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 100, 1e-5),
        CV_CALIB_FIX_ASPECT_RATIO +
        CV_CALIB_ZERO_TANGENT_DIST +
        CV_CALIB_SAME_FOCAL_LENGTH +
        CV_CALIB_RATIONAL_MODEL +
        CV_CALIB_FIX_K3 +   CV_CALIB_FIX_K4 +   CV_CALIB_FIX_K5
        );

Step 3. Create the rectification data

  stereoRectify(
        cameraMatrix[0],    cameraMatrix[1], 
        distCoeffs[0],  distCoeffs[1],
                  imageSize, 
        R, T, 
        RC1, RC2,  //RC1: Rotation matrix Camera 1
        PC1, PC2, 
        Q,
            CALIB_ZERO_DISPARITY, 
        1,  
        imageSize);

II. WHAT I BELIEVE

Goal: I'm trying undistort and rectify the image points of one object in the image from camera 1 and the image from camera 2 (I do this process twice: once while the clay pigeon's on the launcher and once one frame before the clay pigeon disintegrates)

Method: I believe that I don't need to use initUndistortRectifyMap then Remap but can instead just use undistortPoints. I think undistortPoints undistorts the points of interest and rectifies them.

III. WHAT I DO SECOND

You can ignore this if my beliefs aren't correct.

undistortPoints(launcherC1, launcherC1Undistorted, cameraMatrixC1, distCoeffsC1, R1, P1);   
undistortPoints(launcherC2, launcherC2Undistorted, cameraMatrixC2, distCoeffsC2, R2, P2);   

undistortPoints(clayPigeonC1, clayPigeonC1Undistorted, cameraMatrix1, distCoeffs1, R1, P1);
undistortPoints(clayPigeonC2, clayPigeonC2Undistorted, cameraMatrix2, distCoeffs2, R2, P2);

The input and output arrays for undistortPoints (launcherC1, launcherC1Undistorted, ... clayPigeonC2, clayPigeonC2Undistorted) are vectors of Point2f objects.

IV. DISCREPANCY BETWEEN BELIEF AND REALITY

After all undistortPoints functions are run,

launcherC1Undisorted.y does not equal launcherC2Undistorted.y
clayPigeonC1Undistorted.y does not equal clayPigeonC2Undistorted.y.

They are up to 30% different.

V. QUESTIONS

Q1 In addition to undistorting them does undistortPoints also rectify points?
Q1.1_yes. Are the values of y supposed to be equal after rectification?
Q1.1.1_yes Can you tell from the code I've included what I'm doing wrong so that they don't?
Q1_no If undistortPoints doesn't rectify the points then how do I rectify them?

This is a very lengthy question. If you would like to maximise the number of people that manage to read through to the end, I would suggest editing it to the bare minimum - ask "In addition to undistorting them does undistortPoints also rectify points?", with perhaps a minimal further description. Also, can you confirm that you have read the manual? — Chris

ancajic ancajic · Accepted Answer · 2015-10-23T15:01:50

undistortPoints cannot rectify points because you never give it the required parameter for rectification - and that is the camera's pose relation to the other camera.
When you do rectification the proper way, stereoCalibrate + initUndistortRectifyMap + remap, then yes - the y values of corresponding points will be identical.

Does OpenCV's undistortPoints also rectify them?

1 Answers