Screen space to world space

Question

I'm making a drawing program in C++ and can't figure out how to convert from screen space to world space

I have access to the projection and view matrices as well as the screen size ect

This is the code I'm using but it doesn't work

There are no errors but it doesn't output world space coodinates at all, it just slightly changes all of the pixel coodinates

I was following an answer I found here OpenGL Math - Projecting Screen space to World space coords

I'm only using an orthographic camera right now

glm::mat4 proj = cameraComp.Camera.GetProjection();
glm::mat4 view = glm::translate(glm::mat4(1.0f), transformComp.Position)
       * glm::rotate(glm::mat4(1.0f), glm::radians(transformComp.Rotation.z), glm::vec3(0, 0, 1));

glm::mat4 inversed = glm::inverse(proj * view);

glm::vec4 worldPos = { newMousePos, 0, 1 };
worldPos *= inversed;

Answer 1

The view matrix transforms form world space to view space. Therefore the inverse view matrix transforms from view space to world space.
The projection matrix transforms form view space to clip space and the inverse projection matrix from clip space to view space. Clip space is a Homogeneous coordinate system. In order to transform from the clip space to normalized device space, you must do a Perspective divide. Normalized device space is Cartesian coordinates system. Hency, if you want to transform from normalized device space to view space, you need to transform through the inverse projection matrix and divide the x, y, and z components through the w component.
See also OpenGL - Mouse coordinates to Space coordinates

Since you are using GLM, I recommend using glm::unProject for the transformation.

Answer 2

It not GLM but it more common style for any type of math libraries

// make cursor coordinates from -1 to +1
float pt_x = (point.x / screenSize.x) * 2.f - 1.f;
float pt_y = -(point.y / screenSize.y) * 2.f + 1.f;

//                       z value from 0.f to 1.f for d3d
vec4 origin = math::mul(vec4(pt_x, pt_y, -1.f, 1.f), VPinv);
origin.w = 1.0f / origin.w;
origin.x *= origin.w;
origin.y *= origin.w;
origin.z *= origin.w;

You need to calculate ViewProjectInvert matrix only once when your camera has been modified to improve performance.

void camera::update(){
    ....
    m_viewProjectionMatrix = m_projectionMatrix * m_viewMatrix;
    m_viewProjectionInvertMatrix = m_viewProjectionMatrix;
    m_viewProjectionInvertMatrix.invert();
}