glm::unProject doesn't work and incorrect ray position on screen

Question

I try to implement a ray-picking method on GLM function glm::unProject. My problems with wrong ray position on screen and wrong perspective. Below some simplified code...

My LookAt function:

glm::mat4 ViewMatrix()
{
    Front.x = cos(glm::radians(Yaw)) * cos(glm::radians(Pitch));
    Front.y = -sin(glm::radians(Pitch));
    Front.z = sin(glm::radians(Yaw)) * cos(glm::radians(Pitch));
    Right = glm::normalize(glm::cross(Front, WorldUp));
    Up = glm::normalize(glm::cross(Right, Front));
    glm::vec3 PositionPan = RightPan + UpPan;
    Position += PositionPan;

    return glm::lookAt((Position + radius * (Front)), Position , Up); 
}

You'll notice my function include wrong parameter, but I want to make free camera (something like 3ds max and etc.) with targeted on 0,0,0 with possible panning(PositionPan for Position) and zooming (radius). I spent so much time to work it correct and I don't know any other way to make it another way. I have a suspicion these function is cause of wrong work the Ray function with GLM. If you noticed something wrong let me know please.

My Transform functions:

glm::mat4 view = ViewMatrix();
glm::mat4 projection = glm::perspective(1.0f, 
(float)screenWidth / (float)screenHeight, 0.1f, 100.0f);

My Ray function:

void Ray(posx, posy, glm::mat4 view, glm::mat4 projection )
{
     x = (posx / screenWidth - 0.5f) * 2.0f; 
     y = (posy / screenHeight - 0.5f) * -2.0f; 

     worldSpaceNear = glm::unProject(glm::vec3(x, y, 0.0), 
     view, projection,          
     glm::vec4(0.0, 0.0, screenWidth, screenHeight));

     worldSpaceFar = glm::unProject(glm::vec3(x, y, 1.0), 
     view, projection,   
     glm::vec4(0.0, 0.0, screenWidth, screenHeight));
}

Next I pass worldSpaceNear/Far parameters in VAO and produce in geometric shader a long parallelepiped starts from Near to Far plane. Transformation from screen space to clip space is correct, but on debugging the ray starts from left-lower corner and immediately dissapears. Position of mouse on screen influences not significantly on the ray. BUT! if I change the Ray function to classic method:

void mRay(glm::mat4 view, glm::mat4 projection
{
    x = (posx / screenWidth - 0.5f) * 2.0f; 
    y = (posy / screenHeight - 0.5f) * -2.0f;

    glm::vec4 localNear{ x,  y, 0.0, 1.0 };
    glm::vec4 localFar{ x, y, 1.0, 1.0 };

    auto wsn = glm::inverse(projection * view) * localNear;
    worldSpaceNear = glm::vec3(wsn.x, wsn.y, wsn.z);

    auto wsf = glm::inverse(projection * view) * localFar;
    worldSpaceFar = glm::vec3(wsf.x, wsf.y, wsf.z);

}

everything is almost OK. The Ray to follow up a mouse but it has a incorrect perspective. The ray must disperses in accordance with perspective, but in my case it aspire to the center of Far plane. How can I fix it, or how can I solve it on GLM?

UPDATE FOR DERHASS

After mRay() I send worldSpaceNear/Far to render loop. I khow ussually works with direction, but I did so:

 renderLine(glm::vec3 worldSpaceNear, glm::vec3 worldSpaceFar, 
 glm::mat4 view, glm::mat4 projection, glm::vec3 colorOverlay, 
 Shader shader)
 {
 ...
 //Create VAO, VBO
 ...
 GLfloat vertices[] ={
    worldSpaceNear.x - 0.01, worldSpaceNear.y + 0.01, worldSpaceNear.z,
    worldSpaceFar.x - 0.01,  worldSpaceFar.y + 0.01,  worldSpaceFar.z,
    worldSpaceNear.x - 0.01, worldSpaceNear.y - 0.01, worldSpaceNear.z...
 // Long box from near to far
...
glDrawArrays(GL_LINE_LOOP, 0, 8);
}

In the vertex shader just:

gl_Position = projection * view * model * vec4(position, 1.0f);

Now I think may be I forgot to remove rotate component from view matrix for line, or vice versa to add rotation from camera.

derhass derhass · Accepted Answer · 2017-02-15T20:00:46

glm::unProject unprojects a point from window sapce to eye space (or whatever matrices you put into this). However, the point you are unprojecting is

 x = (posx / screenWidth - 0.5f) * 2.0f; 
 y = (posy / screenHeight - 0.5f) * -2.0f;

which is not window space. You actually transform it into normalized device coordinates here. The unproject function will do that transform already internally (that's why you need to pass the four viewport parameters). This also explains why the ray isn't moving much, because moving the mouse across the whole screen would just change it by two pixels when these values get interpreted as window space.

The only thing you have to do yourself is the mirroring in y, as GL's window space origin is at the bottom left:

 x = posx;
 y = screenHeight - 1.0f - posy;

glm::unProject doesn't work and incorrect ray position on screen

1 Answers