I am using OpenGL 3.2 + GLM for my matrix math. Below is my code:
GLint viewport[4];
GLfloat winZ;
glGetIntegerv(GL_VIEWPORT,viewport);
glBindFramebuffer(GL_FRAMEBUFFER,fbo);
glReadPixels(x,y,1,1,GL_DEPTH_COMPONENT,GL_FLOAT,&winZ);
glBindFramebuffer(GL_FRAMEBUFFER,0);
glm::vec3 screen=glm::vec3(x,viewport[3]-y,winZ);
glm::mat4 view=glm::lookAt(glm::vec3(camera.x,camera.y,camera.z),glm::vec3(0,0,0),glm::vec3(0,1,0));
glm::mat4 projection=glm::perspective(90.0f,(float)viewport[2]/(float)viewport[3],10.0f,500.0f);
glm::vec3 pos=glm::unProject(screen,view,projection,glm::vec4(0,0,viewport[2],viewport[3]));
return {pos.x,pos.y,pos.z};
winZ is correct. When I click off of the model, it is 1.0 (zfar) and when I click on the model it looks similar to what I would expect it to be. However, the returned value pos.z is not what I expect. When the mouse is over the background: if it's toward the center, pos.z gets smaller, and when it's at the screen edges pos.z is zfar. Can anyone explain this?
The actual issue I am having is that the pos.x and pos.y don't return what I expect ether. My model is 4.88 units wide, yet unProject is saying that it is around 20 units wide? (if i add the values of hovering on the left and right sides).
Oh, and the screen center is 0,0 as expected.
EDIT: I also should note that the returned pos.z is dependent on the position of the mouse from the center of the screen. It does change to a much higher number when it's on top of the model. If the Y-pos of the cursor is above screen center, pos.z is positive. If Y-pos is below, pos.z is negative. Seems reaaallly weird.
