OpenGL shadow mapping weirdness

Question

I have been playing around with OpenGL and shaders and got myself into shadow mapping.

Trying to follow tutorials on the Internet (ogldev and learnopengl), got some unexpected results.

The issue is best described with few screenshots (I have added a static quad with depth framebuffer for debugging):

Somehow I managed to get shadows to be rendered on a ground quad once, with a static light (this commit). But the shadow pattern is, again, incorrect. I strongly suspect model transformation matrix calculaitons on this:

The way I render the scene is quite straightforward:

1. create the pipelines:

• for mapping the shadows (filling the depth frame buffer)
• for rendering the scene using the depth frame buffer
• (extra) debugging one, rendering depth frame buffer to a static quad on a screen

2. fill the depth frame buffer: using the shadow mapping pipeline, render the scene from the light point, using orthographic projection
3. render the shaded scene: using the rendering pipeline and depth frame buffer bind as the first texture, render the scene from a camera point, using perspective projection

Seems like the algorithm in all those tutorials on shadow mapping out there. Yet, instead of a mouray effect (like in all of the tutorials), I get no shadow on the bottom plane whatsoever and weird artifacts (incorrect shadow mapping) on the 3D (chicken) model.

Interestingly enough, if I do not render (for both the shadow mapping and final rendering pass) the chicken model, the plane is lit with the same weird pattern:

I also had to remove any normal transformations from the fragment shader and disable face culling to make the ground plane lit. With front-face culling the plane does not appear in the shadow map (depth buffer).

I assume the following might be causing this issue:

• wrong depth frame buffer setup (data format or texture parameters)
• flipped depth frame buffer texture
• wrong shadow calculations in rendering shaders
• wrong light matrices (view & projection) setup
• wrong matrix calculations in the rendering shaders (given the model transformation matrices for both chicken model and the quad contain both rotation and scaling)

Unfortunately, I ran out of ideas even on how to assess the above assumptions.

Looking for any help on the matter (also feel free to criticize any of my approaches, including C++, CMake, OpenGL and computer graphics).

The full solution source code is available on GitHub, but for convenience I have placed the heavily cut source code below.

shadow-mapping.vert:

#version 410

layout (location = 0) in vec3 vertexPosition;

out gl_PerVertex
{
    vec4 gl_Position;
};

uniform mat4 lightSpaceMatrix;
uniform mat4 modelTransformation;

void main()
{
    gl_Position = lightSpaceMatrix * modelTransformation * vec4(vertexPosition, 1.0);
}

shadow-mapping.frag:

#version 410

layout (location = 0) out float fragmentDepth;

void main()
{
  fragmentDepth = gl_FragCoord.z;
}

shadow-rendering.vert:

#version 410

layout (location = 0) in vec3 vertexPosition;
layout (location = 1) in vec3 vertexNormal;
layout (location = 2) in vec2 vertexTextureCoord;

out VS_OUT
{
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
    vec4 fragmentPositionInLightSpace;
} vsOut;

out gl_PerVertex {
    vec4 gl_Position;
};

uniform mat4 projection;
uniform mat4 view;
uniform mat4 model;
uniform mat4 lightSpaceMatrix;

void main()
{
    vsOut.fragmentPosition = vec3(model * vec4(vertexPosition, 1.0));
    vsOut.normal = transpose(inverse(mat3(model))) * vertexNormal;
    vsOut.textureCoord = vertexTextureCoord;
    vsOut.fragmentPositionInLightSpace = lightSpaceMatrix * model * vec4(vertexPosition, 1.0);

    gl_Position = projection * view * model * vec4(vertexPosition, 1.0);
}

shadow-rendering.frag:

#version 410

layout (location = 0) out vec4 fragmentColor;

in VS_OUT {
    vec3 fragmentPosition;
    vec3 normal;
    vec2 textureCoord;
    vec4 fragmentPositionInLightSpace;
} fsIn;

uniform sampler2D shadowMap;
uniform sampler2D diffuseTexture;

uniform vec3 lightPosition;
uniform vec3 lightColor;
uniform vec3 cameraPosition;

float shadowCalculation()
{
    vec2 shadowMapCoord = fsIn.fragmentPositionInLightSpace.xy * 0.5 + 0.5;
    float occluderDepth = texture(shadowMap, shadowMapCoord).r;

    float thisDepth = fsIn.fragmentPositionInLightSpace.z * 0.5 + 0.5;

    return occluderDepth < thisDepth ? 1.0 : 0.0;
}

void main()
{
    vec3 color = texture(diffuseTexture, fsIn.textureCoord).rgb;
    vec3 normal = normalize(fsIn.normal);

    // ambient
    vec3 ambient = 0.3 * color;

    // diffuse
    vec3 lightDirection = normalize(lightPosition - fsIn.fragmentPosition);
    float diff = max(dot(lightDirection, normal), 0.0);
    vec3 diffuse = diff * lightColor;

    // specular
    vec3 viewDirection = normalize(cameraPosition - fsIn.fragmentPosition);
    vec3 halfwayDirection = normalize(lightDirection + viewDirection);
    float spec = pow(max(dot(normal, halfwayDirection), 0.0), 64.0);
    vec3 specular = spec * lightColor;

    // calculate shadow
    float shadow = shadowCalculation();

    vec3 lighting = ((shadow * (diffuse + specular)) + ambient) * color;

    fragmentColor = vec4(lighting, 1.0);
}

main.cpp, setting up shaders and frame buffer:

// loading the shadow mapping shaders
auto shadowMappingVertexProgram = ...;
auto shadowMappingFragmentProgram = ...;

auto shadowMappingLightSpaceUniform = shadowMappingVertexProgram->getUniform<glm::mat4>("lightSpaceMatrix");
auto shadowMappingModelTransformationUniform = shadowMappingVertexProgram->getUniform<glm::mat4>("modelTransformation");

auto shadowMappingPipeline = std::make_unique<globjects::ProgramPipeline>();
shadowMappingPipeline->useStages(shadowMappingVertexProgram.get(), gl::GL_VERTEX_SHADER_BIT);
shadowMappingPipeline->useStages(shadowMappingFragmentProgram.get(), gl::GL_FRAGMENT_SHADER_BIT);

// (omitted) loading the depth frame buffer debugging shaders and creating a pipeline here

// loading the rendering shaders
auto shadowRenderingVertexProgram = ...;
auto shadowRenderingFragmentProgram = ...;

auto shadowRenderingModelTransformationUniform = shadowRenderingVertexProgram->getUniform<glm::mat4>("model");
auto shadowRenderingViewTransformationUniform = shadowRenderingVertexProgram->getUniform<glm::mat4>("view");
auto shadowRenderingProjectionTransformationUniform = shadowRenderingVertexProgram->getUniform<glm::mat4>("projection");
auto shadowRenderingLightSpaceMatrixUniform = shadowRenderingVertexProgram->getUniform<glm::mat4>("lightSpaceMatrix");

auto shadowRenderingLightPositionUniform = shadowRenderingFragmentProgram->getUniform<glm::vec3>("lightPosition");
auto shadowRenderingLightColorUniform = shadowRenderingFragmentProgram->getUniform<glm::vec3>("lightColor");
auto shadowRenderingCameraPositionUniform = shadowRenderingFragmentProgram->getUniform<glm::vec3>("cameraPosition");

auto shadowRenderingPipeline = std::make_unique<globjects::ProgramPipeline>();
shadowRenderingPipeline->useStages(shadowRenderingVertexProgram.get(), gl::GL_VERTEX_SHADER_BIT);
shadowRenderingPipeline->useStages(shadowRenderingFragmentProgram.get(), gl::GL_FRAGMENT_SHADER_BIT);

// loading the chicken model
auto chickenModel = Model::fromAiNode(chickenScene, chickenScene->mRootNode, { "media" });

// INFO: this transformation is hard-coded specifically for Chicken.3ds model
chickenModel->setTransformation(glm::rotate(glm::scale(glm::mat4(1.0f), glm::vec3(0.01f)), glm::radians(-90.0f), glm::vec3(1.0f, 0, 0)));

// loading the quad model
auto quadModel = Model::fromAiNode(quadScene, quadScene->mRootNode);

// INFO: this transformation is hard-coded specifically for quad.obj model
quadModel->setTransformation(glm::rotate(glm::scale(glm::translate(glm::mat4(1.0f), glm::vec3(-5, 0, 5)), glm::vec3(10.0f, 0, 10.0f)), glm::radians(-90.0f), glm::vec3(1.0f, 0, 0)));

// loading the floor texture
sf::Image textureImage = ...;

auto defaultTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));
defaultTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<GLint>(GL_LINEAR));
defaultTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<GLint>(GL_LINEAR));
defaultTexture->image2D(0, static_cast<gl::GLenum>(GL_RGBA8), glm::vec2(textureImage.getSize().x, textureImage.getSize().y), 0, static_cast<gl::GLenum>(GL_RGBA), static_cast<gl::GLenum>(GL_UNSIGNED_BYTE), reinterpret_cast<const gl::GLvoid*>(textureImage.getPixelsPtr()));

// initializing the depth frame buffer
auto shadowMapTexture = std::make_unique<globjects::Texture>(static_cast<gl::GLenum>(GL_TEXTURE_2D));
shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MIN_FILTER), static_cast<gl::GLenum>(GL_LINEAR));
shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_MAG_FILTER), static_cast<gl::GLenum>(GL_LINEAR));
shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_S), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));
shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_WRAP_T), static_cast<gl::GLenum>(GL_CLAMP_TO_BORDER));
shadowMapTexture->setParameter(static_cast<gl::GLenum>(GL_TEXTURE_BORDER_COLOR), glm::vec4(1.0f, 1.0f, 1.0f, 1.0f));
shadowMapTexture->image2D(0, static_cast<gl::GLenum>(GL_DEPTH_COMPONENT), glm::vec2(window.getSize().x, window.getSize().y), 0, static_cast<gl::GLenum>(GL_DEPTH_COMPONENT), static_cast<gl::GLenum>(GL_FLOAT), nullptr);

auto framebuffer = std::make_unique<globjects::Framebuffer>();
framebuffer->attachTexture(static_cast<gl::GLenum>(GL_DEPTH_ATTACHMENT), shadowMapTexture.get());

main.cpp, rendering (main loop):

// (omitted) event handling, camera updates go here

glm::mat4 cameraProjection = glm::perspective(glm::radians(fov), (float) window.getSize().x / (float) window.getSize().y, 0.1f, 100.0f);
glm::mat4 cameraView = glm::lookAt(cameraPos, cameraPos + cameraForward, cameraUp);

// moving light together with the camera, for debugging purposes
glm::vec3 lightPosition = cameraPos;

// light settings
const float nearPlane = 1.0f;
const float farPlane = 10.0f;
glm::mat4 lightProjection = glm::ortho(-5.0f, 5.0f, -5.0f, 5.0f, nearPlane, farPlane);
glm::mat4 lightView = glm::lookAt(lightPosition, glm::vec3(0.0f, 0.0f, 0.0f), glm::vec3(0.0f, 1.0f, 0.0f));
glm::mat4 lightSpaceMatrix = lightProjection * lightView;

::glViewport(0, 0, static_cast<GLsizei>(window.getSize().x), static_cast<GLsizei>(window.getSize().y));

// first render pass - shadow mapping

framebuffer->bind();

::glClearColor(1.0f, 1.0f, 1.0f, 1.0f);
::glClear(GL_DEPTH_BUFFER_BIT);
framebuffer->clearBuffer(static_cast<gl::GLenum>(GL_DEPTH), 0, glm::vec4(1.0f));

glEnable(GL_DEPTH_TEST);

// cull front faces to prevent peter panning the generated shadow map
glCullFace(GL_FRONT);

shadowMappingPipeline->use();

shadowMappingLightSpaceUniform->set(lightSpaceMatrix);
shadowMappingModelTransformationUniform->set(chickenModel->getTransformation());
chickenModel->draw();

shadowMappingModelTransformationUniform->set(quadModel->getTransformation());
quadModel->draw();

framebuffer->unbind();

shadowMappingPipeline->release();

glCullFace(GL_BACK);

// second pass - switch to normal shader and render picture with depth information to the viewport

glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

shadowRenderingPipeline->use();

shadowRenderingLightPositionUniform->set(lightPosition);
shadowRenderingLightColorUniform->set(glm::vec3(1.0, 1.0, 1.0));
shadowRenderingCameraPositionUniform->set(cameraPos);

shadowRenderingProjectionTransformationUniform->set(cameraProjection);
shadowRenderingViewTransformationUniform->set(cameraView);
shadowRenderingLightSpaceMatrixUniform->set(lightSpaceMatrix);

// draw chicken

shadowMapTexture->bind();

shadowRenderingModelTransformationUniform->set(chickenModel->getTransformation());
chickenModel->draw();

shadowRenderingModelTransformationUniform->set(quadModel->getTransformation());
defaultTexture->bind();
quadModel->draw();
defaultTexture->unbind();

shadowMapTexture->unbind();

shadowRenderingPipeline->release();

// (omitted) render the debugging quad with depth (shadow) map

window.display();

Answer 1

As shameful as it might be, the issue was with the wrong texture being bound.

The globjects library that I use to have few nice(-r) abstractions over OpenGL actually does not provide a smart logic around texture binding (as I blindly assumed). So using just Texture::bind() and Texture::unbind() won't automagically keep track of how many textures have been bound and increment an index.

E.g. it does not behave (roughly) like this:

static int boundTextureIndex = -1;

void Texture::bind() {
  glBindTexture(this->textureType, this->textureId);
  glActivateTexture(GL_TEXTURE0 + (++boundTextureIndex));
}

void Texture::unbind() {
  --boundTextureIndex;
}

So after changing the texture->bind() to texture->bindActive(0) followed by shaderProgram->setUniform("texture", 0), I finally got to the mouray effect and correct shadow mapping:

Full change is in this commit.