Reconstruction 3d on plane from 2d projection with camera calibration parametrs of plane

Question

Trying to reconstruct 3D-coordinates of point on plane from 2D-Pixel-Coordinates in a Camera Picture, using a extrinsic and intrinsic camera parameters from "Camera Calibration Toolbox for Matlab".

intrinsic parameters: Focal Length: fc = [ 1017.21523 1012.54901 ] Principal point: cc = [ 319.50000 239.50000 ]

extrinsic parameters: Translation vector: Tc_ext = [ 4.409693 -74.116018 393.057934 ] Rotation matrix: Rc_ext = [ -0.083632 0.991715 -0.097501 0.832136 0.015674 -0.554350 -0.548230 -0.127495 -0.826553 ]

Can anybody help how to get 3d-coordinates of point on plane from 2d from camera image?

Answer 1

There are 4 cases to consider, in order of increasing complexity. The basic goal is to locate in space the plane of interest. After you have it, you can just intersect it with the ray that back-projects your pixel and thus obtain the solution.

1. The plane of interest contains the checkerboard target AND the image is one of those you used to calibrate the camera. Then the extrinsic parameters [R | T] for that image returned by the calibration routine contain the answer, since the target is the XY plane of the returned world-to-camera coordinate transform at that image, and the world origin is one of the corners (you can see which corner it is by projecting point (0,0,0) into image coordinates. The XY plane (in camera coordinates) is the plane spanned by the first two columns of the rotation matrix R, and its origin is at point T.
2. The plane of interest contains the checkerboard target, but its image is NOT one of those used to calibrate. You could just add it to the set, re-calibrate and goto 1., but a faster alternative is to (a) extract the corners as you do for calibration, (b) compute the homography H from their image positions to their "true" position on the physical target at Z=0; (c) decompose H into K * [R | T] using the known K from the calibration (you use an algorithm called "RQ decomposition" for this purpose, look it up). Then goto 1.
3. The calibration target is not in the image, but you can identify in the image at least 4 points on that plane, such that no 3 of them are collinear and they are at known positions with respect to each other. For example, you see a rectangle of known sides in the image. Then, just like in point 2. you can compute the homography between those physical points and their images, and extract the [R|T] transform knowing the camera matrix. Then goto 1.
4. None of the above: sorry, you are stuck.