Head Pose Estimation with 2 cameras

Question

I am working on head pose estimation project using 2 cameras. For one camera system works and returns rotation matrix and translation vector of a head with respect to each camera coordinate system. I have rendered object in OpenGL scene which is rotated and translated to represent head movements. To display computed rotation matrix and translation vector I simply use the following OpenGL commands.

glMatrixMode(GL_MODELVIEW);
glLoadMatrixd(pose_matrix);

where pose matrix is OpenGL ModelView matrix constructed from rotation matrix and translation vector of a head.

Now I am trying to do this for 2 calibrated cameras. When the first camera lost the track of a face but the second one estimates head pose I display the rotation and translation with respect to second and visa versa. I want to display one OpenGl object and move it for both cases. For that I need to transfer pose matrices into the common coordinate frame.

I know the relative geometry of 2 cameras with respect to each other. I assume one of the cameras is the world coordinate frame and I transfer the head pose matrix of the second camera to the frame of first camera by multiplying pose matrix and calibration matrix of second camera with respect to first camera. When I load this multiplied matrix into the OpenGL ModelView matrix I get wrong results. When first camera captures face the object is moving right but for the second camera object is translated and rotated and is not in the same place as for the case of first camera.

What could be the problem? Maybe OpenGL displaying part is wrong or?

Answer 1

Safe default assumption: OpenGL is right, your code is wrong.

Without seeing your code, I can suggest to printf your matrices and double-check the math with them in Matlab or Octave.

A common mistake is to forget that by default OpenGL PRE-multiplies ROW-vectors by the modelvie matrix (indeed all matrices). That is, it multiplies as v_row * M, with the matrix stored in col-major order, whereas you may be thinking within the common mathematical convention of treating vectors as COLUMN ones, and POST-multiplying them as M * v_col (with M stored row-major).

If you prefer working with the latter convention(recommended), look up up the GL_ARB_transpose_matrix extension.