Finding world position of element in screen space

Question

I'm attempting to find the world position of an object that exists in screen space. For example in the below screenshot the pistol and hands are rendered without being translated into world space (using only projectionMatrix * modelMatrix), while the other objects in the scene exist in world space (translated via projectionMatrix * viewMatrix * modelMatrix):

I have added a "muzzle" bone to the end of the armature for this pistol, and I am attempting to convert the translation of the muzzle bone into world space so that I can cast a ray starting at this position.

I have this kind of working, but something isn't quite right. For example, note the closest green sphere in the above screenshot. That is the position which I come up with in world space for my muzzle bone. What I am expecting is below:

To obtain this position I am multiplying the position of the muzzle bone by the inverse of the projection and view matrixes of the world camera as follows:

glm::mat4 invMat = glm::inverse(worldStage.getCamera().value().getProjectionMatrix() * 
worldStage.getCamera().value().getViewMatrix());

glm::vec4 tmp = this->weapons[this->currentWeapon].actor->getTransform().getMatrix() * 
glm::vec4(this->weapons[this->currentWeapon].muzzleBone->translation, 1.0f);

glm::vec4 muzzleWorldTranslation = invMat * tmp;

muzzleWorldTranslation /= muzzleWorldTranslation.w;

It feels like I'm pretty close and there's just something that I'm missing or messed up slightly. If anyone is able to point out what that might be, I would be very grateful!

That's should be right if you are not using model matrix, assuming P is a projection matrix and V is the view matrix, the point of what you are doing that, inv (P * V) = inv (V) * inv (P), gives you the identity P * V * inv (V) * inv (P) = I, you need to follow the same concept using the model matrix. i.e. inv(PVM), where M is the model matrix. — 4.Pi.n
I attempted to include the model matrix in invMat, however this yielded worse results. I did notice however through some more debugging that the muzzleWorldTranslation value I am coming up with appears to be mirrored, almost like they are rotated 180 around the up axis (Y) from what they should be. — whitwhoa
Your code assumes that this->weapons[this->currentWeapon].actor->getTransform().getMatrix() containts the transformation to clip space, which doesn't make any sense whatsoever.Why would you mix in the projection matrix at this point? — derhass
this->weapons[this->currentWeapon].muzzleBone->translation is relative to the transform of the weapon actor which is why I'm multiplying it by this->weapons[this->currentWeapon].actor->getTransform().getMatrix(). Should I not be multiplying these matrices by the inverted projectionView matrix? — whitwhoa
In any sane scenario, you should not. These transforms should not include the projection matrix at all, so you should not need to unproject anything. It is hard to tell what transforms you escactly included, but in the simplest of all cases, all you would need to do is applying the inverse view transform, since your model is drawn in view space directly. — derhass

derhass derhass · Accepted Answer · 2021-02-16T22:23:32

If you render an object with standard model M, view V and P matrices, you get the P*V*M as the overall transformation. If you draw a model without the view transform, you're placing the object directly in view space. However, conceptually, that will be the same as first placing your object in world space (by applying a modified model matrix M'), and rendering that with the full P*V*M' pipline.

Since V transforms from world to view space, inverse(V) will transform from view space to world space, and therefore M' = inverse(V) * M will get you P*V*M' = P*V*inverse(V)*M = P*M, and M' is exactly the transformation you are looking for. Note that the projection matrix has nothing to do with that at all, and you are not unprojecting any positions back from screen space as the question title suggests.

Finding world position of element in screen space

1 Answers