Normal mapping: TBN matrix different result in vertex shader compared to fragment shader

Question

I'm working on a normal mapping implementation for a tutorial and for teaching purposes I'd like to pass a TBN matrix to the fragment shader (from the vertex shader) so I can transform normal vectors in tangent space to world-space for lighting calculations. The normal mapping is applied to a 2D plane with its normal pointing in the positive z direction.

However, when I calculate the TBN matrix in the vertex shader of a flat plane (so all tangents/bitangents are the same for all vertices) the displayed normals are completely off. While if I pass the tangent/bitangent and normal vectors to the fragment shader and construct the TBN there, it works just fine as the image below shows (with displayed normals):

This is where it gets weird. Because the plane is flat, the T,B and N vectors are the same for all its vertices thus the TBN matrix should also be the same for each fragment (as fragment interpolation doesn't change anything). The TBN matrix in the vertex shader should be exactly the same as the TBN matrix in the fragment shader but the visual outputs say otherwise.

The source code of both the vertex and fragment shader are below:

Vertex:

#version 330 core
layout (location = 0) in vec3 position;
layout (location = 1) in vec3 normal;
layout (location = 2) in vec2 texCoords;
layout (location = 3) in vec3 tangent;
layout (location = 4) in vec3 bitangent;

out VS_OUT {
    vec3 FragPos;
    vec3 Normal;
    vec2 TexCoords;
    vec3 Tangent;
    vec3 Bitangent;
    mat3 TBN;
} vs_out;

uniform mat4 projection;
uniform mat4 view;
uniform mat4 model;

void main()
{
    gl_Position = projection * view * model * vec4(position, 1.0f);
    vs_out.FragPos = vec3(model * vec4(position, 1.0));   
    vs_out.TexCoords = texCoords;

    mat3 normalMatrix = transpose(inverse(mat3(model)));
    vs_out.Normal = normalize(normalMatrix * normal);

    vec3 T = normalize(normalMatrix * tangent);
    vec3 B = normalize(normalMatrix * bitangent);
    vec3 N = normalize(normalMatrix * normal);
    vs_out.TBN = mat3(T, B, N);

    vs_out.Tangent = T;
    vs_out.Bitangent = B;
}

Fragment

#version 330 core
out vec4 FragColor;

in VS_OUT {
    vec3 FragPos;
    vec3 Normal;
    vec2 TexCoords;
    vec3 Tangent;
    vec3 Bitangent;
    mat3 TBN;
} fs_in;

uniform sampler2D diffuseMap;
uniform sampler2D normalMap;

uniform vec3 lightPos;
uniform vec3 viewPos;

uniform bool normalMapping;

void main()
{           
    vec3 normal = fs_in.Normal;
    mat3 tbn;
    if(normalMapping)
    {
        // Obtain normal from normal map in range [0,1]
        normal = texture(normalMap, fs_in.TexCoords).rgb;
        // Transform normal vector to range [-1,1]
        normal = normalize(normal * 2.0 - 1.0);   
        // Then transform normal in tangent space to world-space via TBN matrix
        tbn = mat3(fs_in.Tangent, fs_in.Bitangent, fs_in.Normal); // TBN calculated in fragment shader
        // normal = normalize(tbn * normal); // This works!
        normal = normalize(fs_in.TBN * normal); // This gives incorrect results
    }

    // Get diffuse color
    vec3 color = texture(diffuseMap, fs_in.TexCoords).rgb;
    // Ambient
    vec3 ambient = 0.1 * color;
    // Diffuse
    vec3 lightDir = normalize(lightPos - fs_in.FragPos);
    float diff = max(dot(lightDir, normal), 0.0);
    vec3 diffuse = diff * color;
    // Specular
    vec3 viewDir = normalize(viewPos - fs_in.FragPos);
    vec3 reflectDir = reflect(-lightDir, normal);
    vec3 halfwayDir = normalize(lightDir + viewDir);  
    float spec = pow(max(dot(normal, halfwayDir), 0.0), 32.0);

    vec3 specular = vec3(0.2) * spec; // assuming bright white light color
    FragColor = vec4(ambient + diffuse + specular, 1.0f);
    FragColor = vec4(normal, 1.0); // display normals for debugging
}

Both TBN matrices are clearly different. Below I compiled an image of different fragment shader outputs:

You can see that the T,B and N vectors are correct and so is the fragment shader's tbn matrix, but the TBN matrix from the vertex shader fs_in.TBN gives completely bogus values.

I am completely clueless as to why it doesn't work. I know I can simply pass the Tangent and Bitangent vector to the fragment shader, calculate it there and be done with it but I'm quite curious as to the exact reason why this doesn't work?

Answer 1

Some general remarks:

1) You should normalize fs_in.Tangent, fs_in.Bitangent, and fs_in.Normal in the fragment shader, since it is not guaranteed that they are after the varying interpolation. They should be normalized, because they are the basis vectors of a coordinate system.

2) You don't need to pass all three, tangent, bitangent, and normal, since one of them can be calculated using the cross product: bitangent = cross(tangent, normal). This point also speaks in favor of passing (two) vectors instead of the whole matrix.

To your question, why fs_in.TBN does not look like tbn in the fragment shader:

The images you provide look very strange, indeed. Looks like the matrix' vectors are somehow mixed up. Which output do you get, displaying the transposed matrix transpose(fs_in.TBN)?

Also make sure that the matrix' columns are accessed correctly like described in [1]! (It looks like they are, but please double check, you never know.)

[1] Geeks3D; GLSL 4×4 Matrix Fields; http://www.geeks3d.com/20141114/glsl-4x4-matrix-mat4-fields/