I think you're referring to gimbal lock? You're right that each rotation modifies the axes around which subsequent local rotations will occur. In your case that affects the yaw because the OpenGL matrix stack works so that each thing you add to it occurs conceptually before whatever is already on the stack (ie, it's post multiplication in matrix terms).
Your solution, however, won't solve the problem even if implemented correctly. What you're trying to do is get the global y axis in local coordinate space so that you can rotate around the global y even after you've rotated around the global z, shifting the local axes. But that just buys you much the same problems as if you'd stuck with global axes throughout and applied the rotations in the other order. The second rotation will now interfere with the first rather than vice versa.
Another way to convince yourself that what you're doing is wrong is to look at how much information you have. You're trying to describe the orientation of an object with two numbers. Two numbers isn't enough to describe any rotation whatsoever, so there's obviously some other rule in there to convert two numbers into a complete orientation. Whatever you do to modify that rule, you're going to end up limiting the orientations you can reach. But with an aeroplane you really want to be able to reach any orientation, so that's a fundamental contradiction.
The confusion comes because, if you have a suitable way of storing orientation, it's completely valid to work forward from that by saying 'what is the orientation if I modify that by rotating around local y by 5, then around local z by 10?', etc. The problem is trying to aggregate all those transformations into a single pair of rotations. It isn't possible.
The easiest solution if you're already generally up on OpenGL tends to be to store the orientation as a complete matrix. You accumulate pitch and yaw rotations by applying them as they occur to that matrix. You pass that matrix to OpenGL via glMultMatrix
to perform your drawing.
It's not an optimal solution but a quick fix test solution would be to use glLoadMatrix
and glGet
to apply transformations by loading your matrix to and then retrieving it from the OpenGL matrix stack, separately from your drawing. It's not really what the stack is for so you'll probably get some performance problems and over time rounding errors will cause odd behaviour but you can fix those once you're persuaded by the approach. The OpenGL man pages give the formulas for all transformation matrices and you should look up matrix normalisation (you'll probably be using an orthonormal matrix whether you realise it or not, which should help with Google) to deal with cumulative rounding.
EDIT: with respect to the code you've posted while I was rambling, quaternions are another valid way of representing orientation and another thing that you can apply incremental updates to safely. They're also compact very easy to protect from rounding errors. However I think your problem may be that you aren't using quaternions as the storage for orientation, merely as an intermediate container. So adding them to the chain doesn't fix any of your problems.
EDIT2: a further bit of hand-waving explanation to push the idea that directly storing pitch and yaw isn't good enough: imagine that, from the point of view of the pilot, you apply a yaw of 90 degrees, then a pitch of 30 degrees, then a yaw of -90 degrees. Then you end up exactly as if you'd applied a roll of 30 degrees. But if you're just storing pitch and yaw then you've no way of storing roll. Furthermore, if you just add up the total yaw and total pitch you end up thinking you've applied a pitch of 30 degrees rather than a roll. So it doesn't matter what order you apply pitch and yaw, or whether you use global or local axes, you get the wrong result.