OpenGL ES Screen Projection Matrix and Camera Projection Matrix

Question

I have a projection matrix derived from the camera calibration in an augmented reality app and as long as the screen aspect ratio matches the camera image aspect ratio everything is fine. When the camera image doesn't match the screen edge-for-edge you'll get distortion in the tracking.

The problem scenarios:

• 1280x720 video on an iPad
• 640x480 video on an iPhone 5S.

The working scenarios:

• 640x480 video on an iPad
• 1280x720 video on an iPhone 5S.

Goal: I want to handle this screen / camera aspect ratio mismatch in a general way.

This problem exists because the view has normalized device coordinates in the aspect ratio of the screen (4:3 for iPad), whereas the projection matrix has an aspect ratio of the camera image (16:9 for 720p). The background image needs to match up to the projection matrix or the illusion of augmented reality fails so if I want to toggle between 'fit' and 'fill' I'll need to change projection matrix to match the image size.

Note: I'm looking to deal with this problem without an OpenGL specific solution. So I'm looking for a more general mathematical answer that involves manipulating the projection matrix.

Answer 1

However, I realize I need to map the camera's projection matrix to the screen projection matrix or perhaps scale it.

Aspect ratio really just determines what is on screen (i.e. amount of peripheral vision you get) - it shouldn't fundamentally change the projection (assuming you keep the near and far clips the same).

First thing to note is that there isn't really a safe mapping (i.e. not every coordinate in the screen view has an equivalent in the camera view, and visa-versa).

To avoid distortion you really have two options - clip to the narrower of the two options (discard information) or scale down with letter box and keep the original aspect ratio (discard screen-area) .

Both feel like it should be possible with a simple scale factor applied to the final clip-space transform (assuming you clip equally on both sides of the frustum you should just be able to multiply up the clip-space coordinate of the axis you want to clip (scale > 1), or multiple down (scale < 1) if you want letterbox). To make letterboxing work you'll need to to use glViewport or glScissor to stop primitives outside of the letterbox being drawn.

Answer 2

Avoid dealing with the projection matrix

The easy way, although not as intellectually satisfying as a mathematical solution, is to place your scene in a view the size of the camera image. Then place that view within a containing view. Then you resize the entire view to match the screen or fill the screen size.