I'm writing a game in Java using OpenGL (the LWJGL binding, to be specific). Each entity, including the camera, has a quaternion that represents it's rotation. I've figured out how to apply the quaternion to the current OpenGL matrix and everything rotates just fine. The issue I'm having is getting the camera to rotate with the mouse.
Right now, every frame, the game grabs the amount that the mouse has moved on one axis, then it applies that amount onto the quaternion for the camera's rotation. Here is the code that rotates the quaternion, I'll post it since I think it's where the problem lies (although I'm always wrong about this sort of stuff):
public void rotateX(float amount){
Quaternion rot = new Quaternion(1.0f, 0.0f, 0.0f, (float)Math.toRadians(amount));
Quaternion.mul(rot, rotation, rotation);
rotation.normalise();
}
This method is supposed to rotate the quaternion around the X axis. 'rotation' is the quaternion representing the entity's rotation. 'amount' is the amount that I want to rotate the quaternion (aka the amount that the mouse was moved). 'rot' is a normalized vector along the X axis with a w value of the amount converted to radians (I guess the goal here is to give it an angle- say, 10 degrees- and have it rotate the quaternion along the given axis by that angle). Using Quaternion.mul takes the new quaternion, multiplies it by the rotation quaternion, and then stores the result as the rotation quaternion. I don't know if the normalization is necessary, since 'rot' is normal and 'rotation' should already by normalized.
The rotateY and rotateZ methods do the same thing, except for changing the vector for 'rot' (0.0, 1.0, 0.0 for y and 0.0, 0.0, 1.0 for z).
The code appears to work fine when the game starts and the camera is looking down the negative Z axis. You can spin all the way around on the Y axis OR all the way around the X axis. But as soon as you try to rotate the camera while not looking down the Z axis, everything gets really screwy (I can't even describe it, it rotates very oddly).
My end goal here is to have something to use for controlling a ship in a space with no up vector. So when you move the mouse on the Y axis, no matter what angle the ship is at, it changes the pitch of the ship (rotation along the X axis). Similarly, when you move the mouse on the X axis, it changes the yaw (rotation along the Y axis). I might be going about this the wrong way and I probably just need a push (or shove) in the right direction.
If you need more details on anything (how my rendering is done, any other maths that I'm trying to do) just ask and I'll put it up. I understood everything when I was using euler angles (which apparently are a big no-no for 3D application development... wish somebody would have told me that before I sunk a lot of time into getting them to work) but as soon as I switched over to quaternions, I got in over my head really fast. I've spent the past few months just playing with this code and reading about quaternions trying to get it to work, but I haven't really gotten anywhere at all :'(
Very, very frustrating... starting to regret trying to make something in 3D >_<