Mapping pixel from one image to the other image in stereo vision

Question

Right now I have two cameras and finish the stereo calibration step for grabbing intrinsic and extrinsic parameters. I extract few feature points in left camera and wondering any way to map them on the right image. Anyone can help please?

I tried to compute them by rotation and translation matrix of each camera like this.

With SolvePnP function in openCV, I got R1 T1 for left camera and R2 T2 for right camera.

To the same world point [X,Y,Z], I can get

[x1,y1,z1] = R1[X,Y,Z]+T1

and

[x2,y2,z2] = R2[X,Y,Z]+T2 

which the camera coordinate in their own coordinate system.

When I try to map [x1,y1,z1] and [x2,y2,z2] by

R2*inv(R1)[x1,x2,x3]+(T2-R2*inv(R1)*T1) = [x2,y2,z2]

I got this result.

the image on left is result of mapping chessboard corner from left image by method above; image on right is corners of right image computing by findChessBoardCorner:

Answer 1

Rotation and translation matrices maps 3D coordinates in world coordinate system to cameras local 3D coordinates In order to find 2D-2D transformation i suggest you to use homography:

https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html?highlight=findhomography#findhomography

Answer 2

Results From function "findHomography"

Computing points mapping relation by "findHomography" only works well on the plane of the chessboard (which i got the relation from corners on the board). However when i compute the foreground/background points by H, results showed above.

The problem should be "matrix H is work on 2-D coordinate system(or object on a plane like chessboard above, that`s why it woks if i reflect a chessboard point to the other image), object of fore/background their coordinate is on a 3-D coordinate system which mapping on image, that means the matrix failed working(actually i think its not failed working, it mapped the point on the extend position of chessboard)"

To solve the mentioned problem, idea i think is "reproject image point to a 3d coordinate system(perhaps with help of ToF camera or other way), call 'solvePnP' function to get pose relation R,T between world point on left and right camera. pick any point on left image, reproject back on 3D coordinate system, and back-project to right image by R,T"