DirectX11: SSAO breaks when tilting the camera up/down

Question

I've been struggling with getting SSAO to work with my DirectX11/C++ graphics engine for weeks now and I just can't think of any more mistakes I could have possibly made in this code.

I'm following this OpenGL tutorial and basically have the exact same implementation using DirectX with HLSL (shader model 5) shaders. I have two framebuffers, one for position data, one for normals, both get transformed to view space and exported in the first shader pass:

Geometry pass vertex shader:

struct VStoPS {
    float4 pos_ : SV_Position;
    float4 posView_ : POSITION1;
    float4 normalView_ : NORMAL1;
};

/********************** constant buffers ***********************/
cbuffer cbCamera_ {
    float4x4 matView_;
    float4x4 matProjection_;
};

cbuffer cbTransformations_ {
    float4x4 matModel_;
    float4x4 matNormalView_;
};

/*************************** main ******************************/
VStoPS vs_main(float3 pos : POSITION, float3 normal0 : NORMAL0, float2 texCoord0 : TEXCOORD0) {
    VStoPS output = (VStoPS) 0;

    output.posView_ = mul(matView_, mul(matModel_, float4(pos, 1.0)));
    output.normalView_ = normalize(mul(matNormalView_, float4(normal0, 0.0)));

    float4x4 viewProj = mul(matProjection_, matView_);
    float4x4 mvp = mul(viewProj, matModel_);
    output.pos_ = mul(mvp, float4(pos, 1.0));

    return output;
}

Geometry pass pixel shader:

/************************** structs ****************************/
struct VStoPS {
    float4 pos_ : SV_Position;
    float4 posView_ : POSITION1;
    float4 normalView_ : NORMAL1;
};

struct PS_Output {
    float4 positionView;
    float4 normalView;
};

/*************************** main ******************************/
PS_Output ps_main(VStoPS input) : SV_Target
{
    PS_Output output = (PS_Output)0;
    output.positionView = input.posView_;
    output.normalView = input.normalView_;
    return output;
}

I calculate the normal view matrix like this:

mat4 normalView = (viewMatrix * modelMatrix).getTransposed().getInverse();

I built the sample kernel and its random rotations like this (the random function returns a float between 0.0 and 1.0):

// Build the main kernel with random samples
for (int i = 0; i < D3D_SSAO_SAMPLE_COUNT; i++)
{
    // Sample kernel is a hemisphere along the positive z axis
    vec3 sample(
        random() * 2.0f - 1.0f,
        random() * 2.0f - 1.0f,
        random()
    );

    // Put more samples closer to the origin of the hemisphere for better results
    float scale = lerp(0.1f, 1.0f, pow(static_cast<float>(i) / static_cast<float>(D3D_SSAO_SAMPLE_COUNT), 2));
    ssaoKernel_[i] = sample.getNormalized() * scale;
}

// Build random kernel rotations to reduce banding
for (int i = 0; i < D3D_SSAO_ROTATIONS_COUNT; i++)
{
    vec3 rotation(
        random() * 2.0f - 1.0f,
        random() * 2.0f - 1.0f,
        0.0f
    );

    ssaoKernelRotations_[i] = rotation.getNormalized();
}

And then I render the SSAO pass. The vertex shader just renders a fullscreen quad, the pixel shader does the actual SSAO work like this:

/************************** structs ****************************/
struct VStoPS {
    float4 pos_ : SV_Position;
    float2 texCoord0_ : TEXCOORD0;
};

/********************** constant buffers ***********************/
cbuffer cbSSAO_ {
    float3 samples_[32];
    float3 rotations_[9];
};

cbuffer cbGBufferCamera_ {
    float4x4 matCameraView_;
    float4x4 matCameraProjection_;
};

cbuffer cbScreenInfo_ {
    int screenWidth_;
    int screenHeight_;
};

/********************** shader resources ***********************/
SamplerState sampler_;

Texture2D<float4> gPositionViewFramebuffer_;
Texture2D<float4> gNormalViewFramebuffer_;

/*************************** main ******************************/
float4 ps_main(VStoPS input) : SV_Target {
    const int kernelSize = 32;

    // Get the proper rotation vector for the current fragment
    const float w = (float) screenWidth_;
    const float h = (float) screenHeight_;
    const float2 noiseScale = float2(w / 3.0, h / 3.0);
    const float2 scaledCoordinates = input.texCoord0_ * noiseScale;
    const uint rotationIndex = (scaledCoordinates.x % 3) * (scaledCoordinates.y % 3);
    const float3 kernelRotationVector = normalize(rotations_[rotationIndex]);

    // Sample fragment position and normal from textures
    const float3 fragPos = gPositionViewFramebuffer_.Sample(sampler_, input.texCoord0_).xyz;
    const float3 normal = normalize(gNormalViewFramebuffer_.Sample(sampler_, input.texCoord0_).xyz);

    // Build a transformation matrix from tangent space to view space
    const float3 tangent = normalize(kernelRotationVector - normal * dot(kernelRotationVector, normal));
    const float3 bitangent = cross(normal, tangent);
    const float3x3 TBN = transpose(float3x3(tangent, bitangent, normal));

    // Calculate occlusion
    float occlusion = 0.0;
    const float radius = 0.5;

    for (int i = 0; i < kernelSize; i++)
    {
        // Transform the sample
        float3 currentSample = mul(TBN, samples_[i]);
        currentSample = fragPos + (currentSample * radius);

        // Get the respective depth value from the gBuffer at the same projected location
        float4 offset = float4(currentSample, 1.0);
        offset = mul(matCameraProjection_, offset);
        float2 coords = ((offset.xy / offset.w) + float2(1.0, 1.0)) / 2.0;

        float sampleDepth = gPositionViewFramebuffer_.Sample(sampler_, coords.xy).z;

        // Increase occlusion if the sample is actually occluded
        float rangeCheck = smoothstep(0.0, 1.0, radius / abs(fragPos.z - sampleDepth));
        occlusion += (sampleDepth <= currentSample.z ? 1.0 : 0.0) * rangeCheck;
    }

    occlusion = 1.0 - (occlusion / ((float) kernelSize));
    return float4(occlusion, occlusion, occlusion, 1.0);
}

After that, I apply a blur filter to fix the noise created by the random rotation vectors. The results don't look too bad when looking straight forward:

Sure, there are some artifacts, it's not super clean, but workable. There is actually a cube hovering in front of the camera, but since there's no ambient occlusion around that, it doesn't even pop out. But once I tilt the camera upwards, this happens:

Basically, the whole scene gets squashed along the y axis and the cube gets mirrored along the x axis. I've been debugging this for hours and can't figure out what's wrong. Here's a list of the possible issues I ruled out (mostly using NSight):

• The sample kernel and rotation vectors are properly created and uploaded to the constant buffers; they are 100% correct
• The position and normal data seems to get transformed properly, though I'm not 100% sure what the framebuffers containing the data should look like
• There are no warnings in the renderer or the shaders themselves, no truncation of data or similar
• I tried transposing the different matrices since DirectX uses a row-major layout unlike OpenGL, but that didn't seem to change anything about the problem
• The matrices are getting uploaded to the constant buffers properly, here's an example of the view- and projection matrices when I tilt the camera upwards:

I'm really at a loss here. Any help or tips on what I could try out to fix this is greatly appreciated. I'll be happy to provide further information on the code if necessary.

Answer 1

Since I couldn't find out what the issue was, I implemented this solution instead, which works fine and is already written with DirectX in mind. I'm marking this as solved.