OpenGL - Correct Surface Normals after Projection

Question

I am currently working on a little toy program with OpenGL which shows a scene in clip-space view, i.e. it draws a cube to visualize the canonical view volume and inside the cube, the projectively transformed model is drawn. To show a code snippet for the model drawing:

glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glScalef(1.0f, 1.0f, -1.0f);
glMultMatrixd(projectionMat);
glMultMatrixd(modelviewMat);
glEnable(GL_LIGHTING);
draw_model();
glDisable(GL_LIGHTING);

So, naturally, the drawn model is "distorted" (which is the desired behaviour). However, the lighting is wrong, as the surface normals are also transformed by the projection matrix and, thus, are not orthogonal to their surfaces after transform. What I am trying to accomplish is lighting that is "correct" in the sense that the surfaces of the distorted models have correct normals.

The question is - how can I do that? I was playing with the usual transposed-inverse-matrix rule for normals, but as far as I understand, that's what OGL does with its normals by default. I think I would have to recalculate the surface normals AFTER the surfaces are transformed with the modelview matrix, but how to do that? Or is there another way?

Actually, "double projection" is exactly what I am trying to do, yes :) As you describe in your tutorial, the point is to see the "weird" shape of objects in normalized device coordinates. However, I don't really find/understand what you are doing to the surface normals to get "correct" lighting in NDC space. In my example, would it be sufficient to "remove" the projective part from the inverse-transpose-modelview matrix so that normals are only transformed by the "true" modelview matrix? I am very much an OGL newb, so please bear with me :D — user1845810
Did you read my answer? Also, the tutorial doesn't do lighting in NDC space. The space of the rendered positions doesn't have to be the space you do lighting in. — Nicol Bolas
@NicolBolas The link to your tutorial now points to some weird ad page and I also just had a popup where it wanted me to install some Chrome addon. Better remove/update the link so that nobody clicks it and falls for it. Here's the tutorial on web.archive.org: web.archive.org/web/20150225192609/http://www.arcsynthesis.org/… — Ela782

datenwolf datenwolf · Accepted Answer · 2012-11-22T18:53:16

glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glScalef(1.0f, 1.0f, -1.0f);
glMultMatrixd(projectionMat);

The projection matrix goes into glMatrixMode(GL_PROJECTION);. Transforming the normals happens with the inverse transpose of the modelview. If there's a projection component in the modelview it messes up your normal transformation.

The correct code would be

glMatrixMode(GL_PROJECTION);
glLoadMatrixd(projectionMat);

glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glScalef(1.0f, 1.0f, -1.0f);
glMultMatrixd(modelviewMat);
glEnable(GL_LIGHTING);
draw_model();
glDisable(GL_LIGHTING);

OpenGL - Correct Surface Normals after Projection

2 Answers