The correct way to calculate normal matrix?

Question

I'm trying to implement a simple ray tracing application, and to keep things uniform and clean I've decided to use transformation matrices like in OpenGL. I have a model matrix for every node in the scene.Everything seems to work fine (ray intersection, models transformations) except for the normals. From what I read one can use glm::inverseTranspose(modelViewMatrix) to get the normal matrix which should keep the normals perpendicular to the faces. (All the calculations are made in the world space so my viewMatrix is an identity, so I calculate the normal matrix as glm::inverseTranspose(modelMatrix)) However, I get strange results: when I calculate a normal in the world space as newNormal = normalMatrix * glm::vec4(normnal, 0.0f) I get rubbish in its w coordinate. What am I doing wrong?

Reto Koradi Reto Koradi · Accepted Answer · 2014-12-23T08:10:35

From looking at the source code (the documentation was not very helpful), my interpretation of the mat4x4 version of glm::inverseTranspose() is that it calculates the inverse transpose of a full 4x4 matrix. Which is somewhat logical, but not really what you need to get a normal transformation matrix. It might be possible to derive the desired result from it, but it seems much more complicated than necessary.

When working with 4x4 matrices to represent linear/affine transformations in 3D space, you only care about the top-left 3x3 part and the additional 3 elements that represent the translation. Since we're transforming vectors in the case of normals, the translation part does not apply. You can reduce the original 4x4 matrix to a 3x3 matrix by simply dropping the remaining elements.

Then, using glm::inverseTranspose() on the 3x3 matrix should give you exactly what you need. It will also avoid having to use the somewhat awkward trick of adding 0.0f as the 4th element of the vector.

I haven't used GLM, but based on the doc/headers, the calculation should look like this:

mat3x3 modelMatrix3(modelViewMatrix);
mat3x3 normalMatrix = = glm::inverseTranspose(modelMatrix3);
newNormal = normalMatrix * normal;

If you don't want to rely on a library, calculating the normal matrix yourself is actually quite easy. If the original matrix is written as consisting of row vectors:

    [ r0 ]
M = [ r1 ]
    [ r2 ]

The normal matrix can be calculated by replacing each row by the cross product of the other two rows:

    [ r1 x r2 ]
N = [ r2 x r0 ]
    [ r0 x r1 ]

This neglects a constant factor (which is 1 divided by the determinant). But often times you'll have to re-normalize the transformed normals anyway, so the constant factor does not matter.

One more consideration: In most real world use cases, you don't have to worry about calculating a specific normal transformation. Most of the time, you can simply use the 3x3 part of the original matrix. This is valid as long as the transformation was build only from rotations, translations, and combinations of them. In that case, the 3x3 part is just the rotation, and the inverse transpose of the matrix is the matrix itself.

This is also consistent with the last calculation above, because the cross product of two row vectors is always equal to the 3rd row if a matrix is orthonormal, which is true for rotation matrices.

Uniform scaling also doesn't add much difficulty, since it just multiplies the resulting vector by a constant. As long as you normalize the result, uniform scaling does not matter.

Reasonably common transformations where you do indeed need to use a specifically calculated normal matrix for correctness include:

Non-uniform scaling. This is the type of transformation that can for example turn a sphere into an ellipsoid.
Shear transforms. Less common, but they do show up for example in scientific applications where non-orthogonal coordinate systems are used (e.g. crystallography).

The correct way to calculate normal matrix?

1 Answers