Manual mouse ray picking ray calculation in java rotation problem

Question

I am having trouble finishing my mouse ray picking algorithm for a game I am creating using java (and opengl/lwjgl). I implemented ray tracing using inspiration from here and here. I followed a lwjgl book for the implementation of the game. The following code is the mouse ray picking algorithm:

// Calculate the normalized device coordinates
Vector2d mouseCoords = mouseInput.getPositionVector();
mouseCoords.mul((float)2 / window.width(), (float)2 / window.height());           
Vector4f mouseNDS = new Vector4f((float)mouseCoords.x, (float)mouseCoords.y, -1f, 1f);     

// Calculate the ray in cam space 
Vector4f eyeRay = mouseNDS.mul(renderer.getTransformations().getProjectionMatrix().invert());
eyeRay = new Vector4f(eyeRay.x, eyeRay.y, eyeRay.z, 0f);

// Calculate the ray in world space
Vector4f worldRay4f = eyeRay.mul(renderer.getTransformations().getViewMatrix(camera).invert());
Vector3f worldRay = new Vector3f(worldRay4f.x, worldRay4f.y, -worldRay4f.z).normalize();

Eventually, I want to intersect it with the ground plane (y = 0). This is my code for this:

float lambda = camera.getPos().dot(new Vector3f(0, 1f , 0)) / worldRay.dot(new Vector3f(0, 1f, 0));      

Vector3f ray_scaled = worldRay.mul(lambda);
Vector3f ground_vec = new Vector3f();
camera.getPos().sub(ray_scaled, ground_vec);

This code works only if the camera is pointing in the negative y direction (i.e. directly down), and keeps working when I apply a position change. However, if I rotate the camera, the code breaks, i.e. the ground vector does not end under the mouse position anymore.

I suspect something is wrong with the inversion of the viewMatrix of the camera. The viewMatrix is calculated by this code:

public Matrix4f getViewMatrix(Camera camera) {
        Vector3f cameraPos = camera.getPos();
        Vector3f rotation = camera.getRot();

        viewMatrix.identity();
        viewMatrix.rotate((float)Math.toRadians(rotation.x), new Vector3f(1, 0, 0))
                  .rotate((float)Math.toRadians(rotation.y), new Vector3f(0, 1, 0))
                  .rotate((float)Math.toRadians(rotation.z), new Vector3f(0, 0, 1));
        viewMatrix.translate(-cameraPos.x, -cameraPos.y, -cameraPos.z);
        return viewMatrix;
}

Can anyone point me in the right direction?

Rabbid76 Rabbid76 · Accepted Answer · 2020-01-02T17:12:19

There is an issue when you set the world ray (worldRay). The z component has not to be inverted:

~~Vector3f worldRay = new Vector3f(worldRay4f.x, worldRay4f.y, -worldRay4f.z).normalize();~~

Vector3f worldRay = new Vector3f(worldRay4f.x, worldRay4f.y, worldRay4f.z).normalize();

If the camera points in the negative y direction, then the x and z component of worldRay4f is 0.0 and the error is not noticeable.

The origin of the window coordinates is at the top left. In normalized device space the top left is (-1, 1) and the bottom left is (-1, -1). So the y coordinate has to be flipped when transforming the mouse coordinate to normalized device space:

Vector2d mouseCoords = mouseInput.getPositionVector();
Vector4f mouseNDS = new Vector4f(
    (float)mouseCoords.x * 2f / window.width() - 1f,
    1f - (float)mouseCoords.y * 2f / window.height(),
    -1f, 1f);

Manual mouse ray picking ray calculation in java rotation problem

1 Answers