OpenGL C++ mouse ray picking glm:unproject

4
votes

I'm currently working on a C++ game engine and I want to build mouse interaction in the application. I've done this previously by ray picking, but back then I used a fixed mouse position and now I want to do it without. I read that you can use the glm::unProject function to do this, but mine just doesn't work. The coördinates this function gives aren't right. What am I doing wrong?

rscore_projection_matrix = glm::perspective(45.0f, (float)(windowWidth)/(float)(windowHeight), 0.1f, 1000.0f);
rscore_view_matrix = glm::lookAt(glm::vec3(lengthdir_x(16, rscam_direction)+rscam_x, rscam_z, lengthdir_y(16, rscam_direction)+rscam_y), glm::vec3(rscam_x, 0, rscam_y), glm::vec3(0,1,0));
rscore_model_matrix = glm::mat4(1.0f);
glm::vec3 screenPos = glm::vec3(rscore_mouse_x, rscore_mouse_y, 0.1f);
glm::vec4 viewport = glm::vec4(0.0f, 0.0f, windowWidth, windowHeight);
glm::vec3 worldPos = glm::unProject(screenPos, rscore_model_matrix, rscore_projection_matrix, viewport);

I use the vec3 worldPos positions to draw the object.

2
c++openglglm-math
What does " just doesn't work" mean? YOu should really provide more source code. Are the matrices the same you are using for rendering? Is screenPos actually using the correct OpenGL convents (origin at bottom left of the viewport)? - derhass
@derhass Yes, the upper 3 variables are also the ones I use to render the scene. I've also tryied to mirror the vertical mouse in the screenPos code, but this also doesn't work right. I draw a cube that should represent the mouse cursor in 3d space, but this cube is just somewhere in the room (at the origin I think) and moves a very little bit when I move the mouse cursor. I also don't understand why the view matrix isn't involved in the unProject script. - The RuneSnake
The view matrix must be involved. GLM's "model" parameter name might be a bit misleading here. YOu must put the modelView matrix there. - derhass

2 Answers

14
votes

Not sure if this would help you, but i implmented ray-picking (calculating ray's direction) this way:

glm::vec3 CFreeCamera::CreateRay() {
    // these positions must be in range [-1, 1] (!!!), not [0, width] and [0, height]
    float mouseX = getMousePositionX() / (getWindowWidth()  * 0.5f) - 1.0f;
    float mouseY = getMousePositionY() / (getWindowHeight() * 0.5f) - 1.0f;

    glm::mat4 proj = glm::perspective(FoV, AspectRatio, Near, Far);
    glm::mat4 view = glm::lookAt(glm::vec3(0.0f), CameraDirection, CameraUpVector);

    glm::mat4 invVP = glm::inverse(proj * view);
    glm::vec4 screenPos = glm::vec4(mouseX, -mouseY, 1.0f, 1.0f);
    glm::vec4 worldPos = invVP * screenPos;

    glm::vec3 dir = glm::normalize(glm::vec3(worldPos));

    return dir;
}

// Values you might be interested:
glm::vec3 cameraPosition; // some camera position, this is supplied by you
glm::vec3 rayDirection = CFreeCamera::CreateRay();
glm::vec3 rayStartPositon = cameraPosition;
glm::vec3 rayEndPosition = rayStartPosition + rayDirection * someDistance;

Explanation:

When you multiply vertice's position with View and Projection matrix, then you get pixel position. If you multiply pixel position with inversion of View and Projection matrix, then you get world's position.

Though calculating inverse matrix is expensive, i'm not sure how glm::unProject works, it may do the same thing.

This only gives you world-oriented direction of ray (and you should already have camera's position). This code doesn't do 'collision' with objects.

Rest of te code of camera's class is here.

More info can be found - in example - here.

0
votes

Below you can see how gluUnproject works. This highlights the fact that you forgot to use view matrix and instead used just model matrix.

int glhUnProjectf(float winx, float winy, float winz,
    float* modelview, float* projection, int* viewport, float* objectCoordinate)
{
    // Transformation matrices
    float m[16], A[16];
    float in[4], out[4];
    // Calculation for inverting a matrix, compute projection x modelview
    // and store in A[16]
    MultiplyMatrices4by4OpenGL_FLOAT(A, projection, modelview);
    // Now compute the inverse of matrix A
    if(glhInvertMatrixf2(A, m)==0)
       return 0;
    // Transformation of normalized coordinates between -1 and 1
    in[0]=(winx-(float) viewport[0])/(float) viewport[2]*2.0-1.0;
    in[1]=(winy-(float) viewport[1])/(float) viewport[3]*2.0-1.0;
    in[2]=2.0* winz-1.0;
    in[3]=1.0;
    // Objects coordinates
    MultiplyMatrixByVector4by4OpenGL_FLOAT(out, m, in);
    if(out[3]==0.0)
       return 0;
    out[3]=1.0/out[3];
    objectCoordinate[0]=out[0]*out[3];
    objectCoordinate[1]=out[1]*out[3];
    objectCoordinate[2]=out[2]*out[3];
    return 1;
}

The code is taken from here.