Can't get ray picking to work in my 3D scene when the camera moves

Question

I have my own 3D engine implementation around OpenGL (C++) and it has worked fine for everything these last years.

But today I stumbled upon a problem. I have this scene with spheres (planets around a sun, orbit rings and things like that, very very simple) and I want to ray pick them with the mouse.

As long as the camera/view matrix is identity, picking works. When the camera is rotated and then moved, the picking goes completely haywire. I have been searching for a solution for a while now so now I'm asking you guys.

This is the code (summarized for this question):

mat4f mProj = createPerspective(PI / 4.0f, float(res.x) / float(res.y), 0.1f, 100.0f);
mat4f mCamera = createTranslation(-1.5f, 3, -34.0f) * createRotationZ(20.0f * PI / 180.0f) * createRotationX(20.0f * PI / 180.0f);
... render scene, using shaders that transform vertices with gl_Position = mProj * mCamera * aPosition;

mat4f mUnproject = (mProj * mCamera).getInverse();
vec2f mouseClip(2.0f * float(coord.x) / float(res.x) - 1.0f, 1.0f - 2.0f * float(coord.y) / float(res.y));
vec3f rayOrigin = (mUnproject * vec4f(mouseClip, 0, 1)).xyz();
vec3f rayDir = (mUnproject * vec4f(mouseClip, -1, 1)).xyz();

// In a loop over all planets:
    mat4f mObject = createRotationY(planet.angle) * createTranslation(planet.distance, 0, 0);
    vec3f planetPos = mObject.transformCoord(vec3f(0, 0, 0));
    float R = planet.distance;

    float a = rayDir.dot(rayDir);
    float b = 2 * rayDir.x * (rayOrigin.x - planetPos.x) + 2 * rayDir.y * (rayOrigin.y - planetPos.y) + 2 * rayDir.z * (rayOrigin.z - planetPos.z);
    float c = planetPos.dot(planetPos) + rayOrigin.dot(rayOrigin) -2 * planetPos.dot(rayOrigin) - R * R;
    float d = b * b - 4 * a * c;
    if (d >= 0)
        HIT!

So when I use identity for mCamera, everything works fine, even when I use only rotation for mCamera, it works fine. It is when I start using the translation that it goes completely wrong. Anyone knows where I am going wrong?

Aren't you missing a perspective divide? After an inverse perspective projection the w-coordinate will not be 1 anymore. — BDL
Funny, I just came to that conclusion too before coming back here. I fixed it, it is still not working, but I'm now looking into it again because the debugged numbers seem to be a lot more correct now :) — scippie
Fixed it! The origin and direction of the ray were wrong too. I now used the position out of the inverted camera matrix and calculated the direction the same way but subtracted the camera position from it and then normalized. — scippie
Not calling it a duplicate. But just leaving this here: 3D Ray Picking use Mouse Coordinates when Mouse isn't locked — vallentin

scippie scippie · Accepted Answer · 2017-03-31T11:34:02

BDL's answer was spot on and put me back in the right direction. Indeed, when transforming coordinates myself, I forgot to do the perspective-divide. After writing so much shader code where the gpu does this for you, you forget about these things.

It is logical that this only gave issues when the camera moved and not when it was at (0, 0, 0) as then, the translation part of the transformation matrices stayed 0 and the w-factor of the coordinates were unaffected.

I immediately wrote transformCoord and transformNormal implementations in my matrix classes to prevent this error from happening again.

Also, the ray origin and direction were incorrect, although I don't really understand why yet. I now take the origin from my camera matrix (inverted of course) and calculate the direction the same way but now subtract the camera position from it to make it a direction vector. I normalize it, although I don't think it is really necessary in this case, but normalizing it will make its numbers look more readable when debugging anyway.

This works:

vec2f mouseClip(2.0f * float(coord.x) / float(res.x) - 1.0f, 1.0f - 2.0f * float(coord.y) / float(res.y));

mat4f mUnproject = (mProj * mCamera).getInverse();
mat4f mInvCamera = mCamera.getInverse();
vec3f rayOrigin(mInvCamera.m[12], mInvCamera.m[13], mInvCamera.m[14]);
vec3f rayDir = (mUnproject.transformCoord(vec3f(mouseClip, 1)) - rayOrigin).normalized();

... per planet
    vec3f planetPos = mObject.transformCoord(vec3f(0, 0, 0));
    float a = rayDir.dot(rayDir);
  float b = 2 * rayDir.x * (rayOrigin.x - planetPos.x) + 2 * rayDir.y * (rayOrigin.y - planetPos.y) + 2 * rayDir.z * (rayOrigin.z - planetPos.z);
  float c = planetPos.dot(planetPos) + rayOrigin.dot(rayOrigin) -2 * planetPos.dot(rayOrigin) - 0.4f * 0.4f;
  float d = b * b - 4 * a * c;
  if (d >= 0)
    ... HIT!

Can't get ray picking to work in my 3D scene when the camera moves

1 Answers