OpenCV: Camera Pose Estimation

Question

I try to match two overlapping images captured with a camera. To do this, I'd like to use OpenCV. I already extracted the features with the SurfFeatureDetector. Now I try to to compute the rotation and translation vector between the two images.

As far as I know, I should use cvFindExtrinsicCameraParams2(). Unfortunately, this method require objectPoints as an argument. These objectPoints are the world coordinates of the extracted features. These are not known in the current context.

Can anybody give me a hint how to solve this problem?

Answer 1

The problem of simultaneously computing relative pose between two images and the unknown 3d world coordinates has been treated here:

Berthold K. P. Horn. Relative orientation revisited. Berthold K. P. Horn. Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 545 Technology ...

EDIT: here is a link to the paper: http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.64.4700

Please see my answer to a related question where I propose a solution to this problem:

OpenCV extrinsic camera from feature points

EDIT: You may want to take a look at bundle adjustments too,

http://en.wikipedia.org/wiki/Bundle_adjustment

That assumes an initial estimate is available.

EDIT: I found some code resources you might want to take a look at:

Resource I:

http://www.maths.lth.se/vision/downloads/

Two View Geometry Estimation with Outliers

C++ code for finding the relative orientation of two calibrated cameras in presence of outliers. The obtained solution is optimal in the sense that the number of inliers is maximized.

Resource II:

http://lear.inrialpes.fr/people/triggs/src/ Relative orientation from 5 points: a somewhat more polished C routine implementing the minimal solution for relative orientation of two calibrated cameras from unknown 3D points. 5 points are required and there can be as many as 10 feasible solutions (but 2-5 is more common). Also requires a few CLAPACK routines for linear algebra. There's also a short technical report on this (included with the source).

Resource III:

http://www9.in.tum.de/praktika/ppbv.WS02/doc/html/reference/cpp/toc_tools_stereo.html vector_to_rel_pose Compute the relative orientation between two cameras given image point correspondences and known camera parameters and reconstruct 3D space points.

Answer 2

There is a theoretical solution, however, the OpenCV implementation of camera pose estimation lacks the needed tools.

The theoretical approach: Step 1: extract the homography (the matrix describing the geometrical transform between images). use findHomography() Step 2. Decompose the result matrix into rotations and translations. Use cv::solvePnP();

Problem: findHomography() returns a 3x3 matrix, corresponding to a projection from a plane to another. solvePnP() needs a 3x4 matrix, representing the 3D rotation/translation of the objects. I think that with some approximations, you can modify the solvePnP to give you some results, but it requires a lot of math and a very good understanding of 3D geometry.

Read more about at http://en.wikipedia.org/wiki/Transformation_matrix