OpenGL Unprojecting mouse cursor to find relative position to a certain object?

Question

I have a player object that's moving around the screen. The camera is a fixed camera, looking down on the player (like in Diablo).

Now I want the player object to rotate towards the mouse cursor. The player is not always on the center of the screen (for this case I already have a solution). In order to do this, I think I need to project the mouse cursor to the same height (y-axis) that my player is on (y-axis is "up" in my game) and then check compare player position with cursor position on the same height in world space.

So far, my unprojecting method looks like this:

private bool Unproject(float winX, float winY, float winZ, out Vector3 position)
{
    position = Vector3.Zero;
    Matrix4 transformMatrix = Matrix4.Invert(World.CurrentWindow.GetViewMatrix() * World.CurrentWindow.GetProjectionMatrix());

    Vector4 inVector = new Vector4(
        (winX - World.CurrentWindow.X) / World.CurrentWindow.Width * 2f - 1f, 
        (winY - World.CurrentWindow.Y) / World.CurrentWindow.Height * 2f - 1f, 
        2f * winZ - 1f, 
        1f
        );

    Matrix4 inMatrix = new Matrix4(inVector.X, 0, 0, 0, inVector.Y, 0, 0, 0, inVector.Z, 0, 0, 0, inVector.W, 0, 0, 0);
    Matrix4 resultMtx = transformMatrix * inMatrix;
    float[] resultVector = new float[] { resultMtx[0, 0], resultMtx[1, 0], resultMtx[2, 0], resultMtx[3, 0] };
    if (resultVector[3] == 0)
    {
        return false;
    }
    resultVector[3] = 1f / resultVector[3];
    position = new Vector3(resultVector[0] * resultVector[3], resultVector[1] * resultVector[3], resultVector[2] * resultVector[3]);

    return true;
}

Now I unproject the mouse cursor once for the near plane (winZ = 0) and the far plane (winZ = 1).

protected Vector3 GetMouseRay(MouseState s)
{
    Vector3 mouseposNear = new Vector3();
    Vector3 mouseposFar = new Vector3();
    bool near = Unproject(s.X, s.Y, 0f, out mouseposNear);
    bool far  = Unproject(s.X, s.Y, 1f, out mouseposFar);
    Vector3 finalRay = mouseposFar - mouseposNear;
    return finalRay;
}

My problem is:

How do I know if the values are correct. The values in the "finalRay" Vector are quite small - always. I would have thought that i would get much bigger z-values because my near plane (perspective projection) is 0.5f and my far plane is 1000f.

And how can I find out if the mouse cursor is left/right (-x, +x) or behind/in front of (-z, +z) the player? (I know the player's position)

Where is my error?

Rabbid76 Rabbid76 · Accepted Answer · 2018-01-17T17:32:33

And how can I find out if the mouse cursor is left/right (-x, +x) or behind/in front of (-z, +z) the player?

Do it in the opposite direction. Project the postion of the player to the screen. So you can easily compare the position of the player with the mouse position:

// positon of the player in world coordinates
Vector3 p_ws ....; 

// positon of the player in view space
Vector4 p_view = World.CurrentWindow.GetViewMatrix() * Vector4(p_ws.X, p_ws.Y, p_ws.Z, 1.0);

// postion of the player in clip space (Homogeneous coordinates)
Vector4 p_clip = World.CurrentWindow.GetProjectionMatrix() * p_view;

// positon of the player in normailzed device coordinates (perspective divide)
Vec3 posNDC = new Vector3(p_clip.X / p_clip.W, p_clip.Y / p_clip.W, p_clip.Z / p_clip.W);  

// screen position in pixel
float p_screenX = (posNDC.X * 0.5 + 0.5) * World.CurrentWindow.Width;
float p_screenY = (posNDC.Y * 0.5 + 0.5) * World.CurrentWindow.Height;

OpenGL Unprojecting mouse cursor to find relative position to a certain object?

1 Answers