OpenGL Matrix Camera controls, local rotation not functioning properly

Question

So I'm trying to figure out how to mannually create a camera class that creates a local frame for camera transformations. I've created a player object based on OpenGL SuperBible's GLFrame class.

I got keyboard keys mapped to the MoveUp, MoveRight and MoveForward functions and the horizontal and vertical mouse movements are mapped to the xRot variable and rotateLocalY function. This is done to create a FPS style camera.

The problem however is in the RotateLocalY. Translation works fine and so does the vertical mouse movement but the horizontal movement scales all my objects down or up in a weird way. Besides the scaling, the rotation also seems to restrict itself to 180 degrees and rotates around the world origin (0.0) instead of my player's local position.

I figured that the scaling had something to do with normalizing vectors but the GLframe class (which I used for reference) never normalized any vectors and that class works just fine. Normalizing most of my vectors only solved the scaling and all the other problems were still there so I'm figuring one piece of code is causing all these problems?

I can't seem to figure out where the problem lies, I'll post all the appropriate code here and a screenshot to show the scaling.

Player object

Player::Player()
{
    location[0] = 0.0f; location[1] = 0.0f; location[2] = 0.0f;
    up[0] = 0.0f; up[1] = 1.0f; up[2] = 0.0f;
    forward[0] = 0.0f; forward[1] = 0.0f; forward[2] = -1.0f;
}

// Does all the camera transformation. Should be called before scene rendering!
void Player::ApplyTransform()
{
    M3DMatrix44f cameraMatrix;
    this->getTransformationMatrix(cameraMatrix);

    glRotatef(xAngle, 1.0f, 0.0f, 0.0f);
    glMultMatrixf(cameraMatrix);
}

void Player::MoveForward(GLfloat delta)
{
    location[0] += forward[0] * delta;
    location[1] += forward[1] * delta;
    location[2] += forward[2] * delta;
}

void Player::MoveUp(GLfloat delta)
{
    location[0] += up[0] * delta;
    location[1] += up[1] * delta;
    location[2] += up[2] * delta;
}

void Player::MoveRight(GLfloat delta)
{
    // Get X axis vector first via cross product
    M3DVector3f xAxis;
    m3dCrossProduct(xAxis, up, forward);

    location[0] += xAxis[0] * delta;
    location[1] += xAxis[1] * delta;
    location[2] += xAxis[2] * delta;
}

void Player::RotateLocalY(GLfloat angle)
{
    // Calculate a rotation matrix first
    M3DMatrix44f rotationMatrix;
    // Rotate around the up vector
    m3dRotationMatrix44(rotationMatrix, angle, up[0], up[1], up[2]); // Use up vector to get correct rotations even with multiple rotations used.

    // Get new forward vector out of the rotation matrix
    M3DVector3f newForward;
    newForward[0] = rotationMatrix[0] * forward[0] + rotationMatrix[4] * forward[1] + rotationMatrix[8] * forward[2]; 
    newForward[1] = rotationMatrix[1] * forward[1] + rotationMatrix[5] * forward[1] + rotationMatrix[9] * forward[2];
    newForward[2] = rotationMatrix[2] * forward[2] + rotationMatrix[6] * forward[1] + rotationMatrix[10] * forward[2];

    m3dCopyVector3(forward, newForward);
}

void Player::getTransformationMatrix(M3DMatrix44f matrix)
{
    // Get Z axis (Z axis is reversed with camera transformations)
    M3DVector3f zAxis;
    zAxis[0] = -forward[0];
    zAxis[1] = -forward[1];
    zAxis[2] = -forward[2];

    // Get X axis
    M3DVector3f xAxis;
    m3dCrossProduct(xAxis, up, zAxis);

    // Fill in X column in transformation matrix
    m3dSetMatrixColumn44(matrix, xAxis, 0); // first column
    matrix[3] = 0.0f; // Set 4th value to 0

    // Fill in the Y column
    m3dSetMatrixColumn44(matrix, up, 1); // 2nd column
    matrix[7] = 0.0f;

    // Fill in the Z column
    m3dSetMatrixColumn44(matrix, zAxis, 2); // 3rd column
    matrix[11] = 0.0f;

    // Do the translation
    M3DVector3f negativeLocation; // Required for camera transform (right handed OpenGL system. Looking down negative Z axis)
    negativeLocation[0] = -location[0];
    negativeLocation[1] = -location[1];
    negativeLocation[2] = -location[2];
    m3dSetMatrixColumn44(matrix, negativeLocation, 3); // 4th column
    matrix[15] = 1.0f;
}

Player object header

class Player
{
public:
    //////////////////////////////////////
    // Variables
    M3DVector3f location;
    M3DVector3f up;
    M3DVector3f forward;
    GLfloat xAngle; // Used for FPS divided X angle rotation (can't combine yaw and pitch since we'll also get a Roll which we don't want for FPS)
    /////////////////////////////////////
    // Functions
    Player();
    void ApplyTransform();
    void MoveForward(GLfloat delta);
    void MoveUp(GLfloat delta);
    void MoveRight(GLfloat delta);
    void RotateLocalY(GLfloat angle); // Only need rotation on local axis for FPS camera style. Then a translation on world X axis. (done in apply transform)

private:
    void getTransformationMatrix(M3DMatrix44f matrix);
};

Applying transformations

// Clear screen
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glLoadIdentity();

// Apply camera transforms
player.ApplyTransform();


// Set up lights
...

// Use shaders
...

// Render the scene
RenderScene();

// Do post rendering operations
glutSwapBuffers();

and mouse

float mouseSensitivity = 500.0f;

float horizontal = (width / 2) - mouseX;
float vertical = (height / 2) - mouseY;

horizontal /= mouseSensitivity;
vertical /= (mouseSensitivity / 25);

player.xAngle += -vertical;
player.RotateLocalY(horizontal);

glutWarpPointer((width / 2), (height / 2));

Answer 1

Honestly I think you are taking a way to complicated approach to your problem. There are many ways to create a camera. My favorite is using a R3-Vector and a Quaternion, but you could also work with a R3-Vector and two floats (pitch and yaw).

The setup with two angles is simple:

glLoadIdentity();
glTranslatef(-pos[0], -pos[1], -pos[2]);
glRotatef(-yaw, 0.0f, 0.0f, 1.0f);
glRotatef(-pitch, 0.0f, 1.0f, 0.0f);

The tricky part now is moving the camera. You must do something along the lines of:

flaot ds = speed * dt;
position += tranform_y(pich, tranform_z(yaw, Vector3(ds, 0, 0)));

How to do the transforms, I would have to look that up, but you could to it by using a rotation matrix

Rotation is trivial, just add or subtract from the pitch and yaw values.

I like using a quaternion for the orientation because it is general and thus you have a camera (any entity that is) that independent of any movement scheme. In this case you have a camera that looks like so:

class Camera
{
public:
   // lots of stuff omitted

   void setup();

   void move_local(Vector3f value);

   void rotate(float dy, float dz);

private:
    mx::Vector3f position;
    mx::Quaternionf orientation;
};

Then the setup code uses shamelessly gluLookAt; you could make a transformation matrix out of it, but I never got it to work right.

void Camera::setup()
{
    // projection related stuff

    mx::Vector3f eye     = position;
    mx::Vector3f forward = mx::transform(orientation, mx::Vector3f(1, 0, 0));
    mx::Vector3f center  = eye + forward;
    mx::Vector3f up      = mx::transform(orientation, mx::Vector3f(0, 0, 1));
    gluLookAt(eye(0), eye(1), eye(2), center(0), center(1), center(2), up(0), up(1), up(2));
}

Moving the camera in local frame is also simple:

void Camera::move_local(Vector3f value) 
{
    position += mx::transform(orientation, value);
}

The rotation is also straight forward.

void Camera::rotate(float dy, float dz)
{
    mx::Quaternionf o = orientation;
    o = mx::axis_angle_to_quaternion(horizontal, mx::Vector3f(0, 0, 1)) * o;
    o = o * mx::axis_angle_to_quaternion(vertical, mx::Vector3f(0, 1, 0));   
    orientation = o;
}

(Shameless plug):

If you are asking what math library I use, it is mathex. I wrote it...