How to map points from left camera to right camera using Rotation(R) and translation(t) between two cameras obtained from stereoCalibrate()?

Question

I have R|t between two cameras which is estimated using stereoCalibrate() function from Opencv. From stereoCalibrate() function, we are getting R1 t1 and R2 t2 for each cameras respectively. We also getting between camera R t(R t between both cameras). Further, we also getting 2 intrinsic matrices K1 and K2, one for each of the camera.

I tried to map points from one camera to another camera using estimated R|t (between two cameras). However, I failed to map, even the points which I used for estimating R|t. I tried to map using depth data also but i failed. Any idea how to map the points from one camera to another?.

I tried Pose estimation of 2nd camera of a calibrated stereo rig, given 1st camera pose but didn't get success.

Answer 1

The "mapping" you seek requires knowledge of the 3D geometry of the scene. This can be inferred from a depth map, i.e. an image associated to a camera, whose pixel values equal the distance from the camera of the scene object seen through each pixel. The depth map itself can be computed from a stereo algorithm.

In some special cases the mapping can be computed without knowledge of the scene geometry. These include:

• The camera displacement is a pure rotation (or, more generally, the translation between the cameras is very small compared to the distance of the scene objects from the cameras). In this case the image mapping is a homography.
• The scene lies in a plane. In this case also the image mapping is a homography.