Camera pose estimation is inverted

Question

I am working on a Structure from Motion framework for generating 3D models of moving objects with fixed cameras. To do this, I am following this pipeline:

Obtain the fundamental matrix F through keypoints. As I am still in development and I can not (yet) access the final objects I will model, I am doing this with a small object and manually annotated keypoint pairs between two images: These are the correspondences used to calculate F. Computing the error of F by x'Fx where x' are the points from the right image and x the points of the left image (in pixel coordinates) gives an error of 0,1196.
Compute the essential matrix E using the intrinsic matrices and the fundamental: E = Kleft'*F*Kright where Kleft' is the inverse intrinsic matrix of the left camera. With the SVD decomposition I create a new E, which has only two singular values equal to 1.
Decompose E to get R and t. To do so, I have used a self made custom version of OpenCV recoverPose() function, that allows for two different camera matrices.
Obtain Pleft as a diagonal matrix and Pright as the construction of R and t.

Here are the significant parts of the code:

F = findFundamentalMat( alignedLeft.points, alignedRight.points, mask, RANSAC);
E = cameraMatrixLeftInv*F*cameraMatrixRight;
SVD::compute(E, w, u, vt);

Mat diag = Mat::zeros(Size(3,3),6);
diag.at<double>(0,0) = 1;
diag.at<double>(1,1) = 1;       
E = u*diag*vt;
int good = recoverPoseFromTwoCameras(E,alignedLeft.points,alignedRight.points,intrinsicsLeft.K,intrinsicsRight.K,R,t,mask);
Pleft = Matx34f::eye();
Pright = Matx34f(R.at<double>(0,0), R.at<double>(0,1), R.at<double>(0,2), t.at<double>(0),
                 R.at<double>(1,0), R.at<double>(1,1), R.at<double>(1,2), t.at<double>(1),
                 R.at<double>(2,0), R.at<double>(2,1), R.at<double>(2,2), t.at<double>(2));

Then I use viz to visualize the camera poses:

viz::Viz3d myWindow("Results");
viz::WCameraPosition cameraLeft(imgLeft->getIntrinsics().K,imgLeft->getImage());
viz::WCameraPosition cameraRight(imgRight->getIntrinsics().K,imgRight->getImage());

Which I place in the viewer using Pleft and Pright:

myWindow.showWidget("cameraLeft",cameraLeft,Affine3d(Pleft(Range::all(),Range(0,3)),Pleft.col(3)));
myWindow.showWidget("cameraRight",cameraRight,Affine3d(Pright(Range::all(),Range(0,3)),Pright.col(3)));

However, if I do that, the result is inverted. I can't embed more than one link because of low reputation, but the camera1 is where the camera 2 should be and viceversa.

But if I apply the matrices like this:

myWindow.showWidget("cameraLeft",cameraLeft,Affine3d(Pright(Range::all(),Range(0,3)),Pleft.col(3)));
myWindow.showWidget("cameraRight",cameraRight,Affine3d(Pleft(Range::all(),Range(0,3)),Pleft.col(3)));

The result is correct.

What am I missing?

JunWangCas JunWangCas · Accepted Answer · 2016-12-22T10:58:35

Hope you have worked out! If you have not, you'd better upload more data. For example, the features, the F/E you computed, the camera intrinsic. Then we can test your code and try to find your bug, otherwise, there are many possible for this.

Camera pose estimation is inverted

1 Answers