iPhone picking/unproject when in landscape view (projection matrix is rotated)

Question

I am trying to get 'picking' working in a 3D scene, where the view is rotated such that the iPhone is being held in a landscape mode. I'm using OpenGL ES 2.0 (so all shaders, no fixed-function pipeline).

I'm performing the unproject from within the rendering code and immediately drawing the resulting ray using GL_LINES (ray only gets calculated the 1st time that I touch the screen, so afterwards I can move the camera around to observe the resulting line from various angles).

My unproject code/call is fine (lots of examples of gluUnproject online). My matrix-inversion code is fine (even compared with excel for a few matrices). However, the resulting ray is off by at least 5-15 degrees from where I actually 'clicked' (in the Simulator it really is a click, so I'm expecting a lot more precision from the unproject).

My view is rotated to landscape (after I create the perspective-projection matrix, I rotate it around the Z by -90 degrees; the aspect ratio remains at a portrait one). I believe that the problem with the math being off lies here.

Does anyone have any experience doing picking/unprojection with specifically a landscape view?

Answer 1

Is it possible you simply have the field of view off? Assuming you've stuck to something a lot like the traditional pipeline, if you were inverting your modelview matrix then using generic unproject code (ie, code that assumes a 90 degree field of view in both directions to fill eye space) then that would explain it.

A quick diagnostic test is to see how far off it is for different touches. Touches nearer the centre of projection should be closer to the correct answer.

On a screen with square pixels like the iPhone, the aspect ratio is just the proportion of the horizontal field compared to the vertical. So if you want to be unscientific about it, find the field of view you're using, say f, and try multiplying your results by 90/f or f/90. If that doesn't work, try also throwing a factor of 480/320 or 320/480 in there.

A better solution is to follow your code through and figure out what your actual horizontal and vertical fields of view are. And multiply your results by that over 90.