MSAA and vertex interpolation cause out-of-range values

Question

I am using GLSL with a vertex shader and a fragment shader.

The vertex shader outputs a highp float in the range of [0,1]

When it arrives in the fragment shader, I see values (at triangle edges) that exceed 1.1 no less!

This issue goes away if I...

Either Disable MSAA
Or disable the interpolation by using the GLSL interpolation qualifier flat.

How can a clamped 0-to-1 high precision float arrive in fragment shader as a value that is substantially larger than 1, if MSAA is enabled?

vertex shader code:

out highp float lightcontrib2;
...
lightcontrib2 = clamp( irrad, 0.0, 1.0 );

fragment shader code:

in highp float lightcontrib2;
...
if (lightcontrib2>1.1) { fragColor = vec4(1,0,1,1); return; }

And sure enough, with MSAA 4x, this is the image generated by OpenGL. (Observe the magneta coloured pixels in the centre of the window.)

I've ruled out Not-A-Number values.

GL_VERSION: 3.2.0 NVIDIA 450.51.06

Nicol Bolas Nicol Bolas · Accepted Answer · 2020-09-15T01:38:12

How can a clamped 0-to-1 high precision float arrive in fragment shader as a value that is substantially larger than 1, if MSAA is enabled?

Multisampling at its core is a variation of supersampling: of taking multiple samples from a pixel-sized area of a primitive. Different locations within the space of that pixel-sized area are sampled to produce the resulting value.

When you're at the edge of a primitive however, some of the locations in that pixel-sized area are outside of the area that the primitive actually covers. In supersampling that's fine; you just don't use those samples.

However, multisampling is different. In multisampling, the depth samples are distinct from the fragment shader generated samples. That is, the system might execute the FS only once, but take 4 depth samples and test them against 4 samples in the depth buffer. Any samples that pass the depth test get their color values from the single FS invocation that was executed. If some of those 4 depth samples are outside of the primitive's area, that's fine; they don't count.

However, by divorcing the FS invocation values from the depth sampling, we now encounter an issue: exactly where did that single FS invocation execute within the pixel area?

And that's where we encounter the problem. If the FS invocation executes on a location that's outside of the area of the primitive, that normally gets tossed away. But if any depth samples are within the area of the primitive, then those depth samples still need to get color data. And the whole point of MSAA is to not execute the FS for each sample, so they may get their color data from an FS invocation executed on a different location.

Ideally, it would be from an FS invocation executed on a location within the primitive's area. But hardware can't guarantee that. Well, it can't guarantee it by default, at any rate. Not every algorithm has issues if an FS location happens to fall slightly outside of the primitive's area.

But some algorithms do have issues. This is why we have the centroid qualifier for fragment shader inputs. It ensures that a particular interpolated value will be generated within the area of the primitive.

As you might have guessed, this isn't the default because it's slower than non-centroid interpolation. So use it only when you need it.

MSAA and vertex interpolation cause out-of-range values

1 Answers