Fragment shader output interferes with conditional statement

Question

Context: I'm doing all of the following using OpenGLES 2 on iOS 11

While implementing different blend modes used to blend two textures together I came across a weird issue that I managed to reduce to the following:

I'm trying to blend the following two textures together, only using the fragment shader and not the OpenGL blend functions or equations. GL_BLEND is disabled.

Bottom - dst:

Top - src:

(The bottom image is the same as the top image but rotated and blended onto an opaque white background using "normal" (as in Photoshop 'normal') blending)

In order to do the blending I use the

#extension GL_EXT_shader_framebuffer_fetch

extension, so that in my fragment shader I can write:

void main()
{
    highp vec4 dstColor = gl_LastFragData[0];
    highp vec4 srcColor = texture2D(textureUnit, textureCoordinateInterpolated);

    gl_FragColor = blend(srcColor, dstColor);
}

The blend doesn't perform any blending itself. It only chooses the correct blend function to apply based on a uniform blendMode integer value. In this case the first texture gets drawn with an already tested normal blending function and then the second texture gets drawn on top with the following blendTest function:

Now here's where the problem comes in:

highp vec4 blendTest(highp vec4 srcColor, highp vec4 dstColor) {

    highp float threshold = 0.7; // arbitrary
    highp float g = 0.0;

    if (dstColor.r > threshold && srcColor.r > threshold) {
        g = 1.0;
    }

    //return vec4(0.6, g, 0.0, 1.0); // no yellow lines (Case 1)
    return vec4(0.8, g, 0.0, 1.0); // shows yellow lines (Case 2)
}

This is the output I would expect (made in Photoshop):

So red everywhere and green/yellow in the areas where both textures contain an amount of red that is larger than the arbitrary threshold.

However, the results I get are for some reason dependent on the output value I choose for the red component (0.6 or 0.8) and none of these outputs matches the expected one. Here's what I see (The grey border is just the background):

Case 1:

Case 2:

So to summarize: If I return a red value that is larger than the threshold, e.g

return vec4(0.8, g, 0.0, 1.0);

I see vertical yellow lines, whereas if the red component is less than the threshold there will be no yellow/green in the result whatsoever.

Question:

Why does the output of my fragment shader determine whether or not the conditional statement is executed and even then, why do I end up with green vertical lines instead of green boxes (which indicates that the dstColor is not being read properly)?

Does it have to do with the extension that I'm using?

I also want to point out that the textures are both being loaded in and bound properly. I can see them just fine if I just return the individual texture info without blending or even with a normal blending function that I've implemented everything works as expected.

This still suggest that your issue is in your code... Are you swapping your used program? The first texture should not be drawn with this fragment shader if I understand correctly. And it seems this issue might be possible exactly because of that. If your first draw call sets fragments to either less then or greater then threshold then your second call will simply either draw nothing (in green) which is example 1, and will draw all vertical lines or in the other case draw full vertical lines... — Matic Oblak
@MaticOblak I actually simplified the code here compared to the full shader that I have in order to focus on this particular issue, but you're right, I think I took out a little too much for this demonstration. In my actual shader instead of calling blendTest directly I call a blend function which does nothing more than determine the type of blending to apply using a uniform integer that gets passed in. The first texture gets drawn using a normal blending function (which is tested and it works) and then the second one gets drawn using the blendTest function. — Keiwan
It happens a lot that pipeline/shaders we build for apps get these bugs which usually tend to be one of those minimal mistakes you have trouble tracking down. I would try to iterate the whole thing if possible... 1 Draw a single texture. 2. draw the other texture 3. overwrite one texture with another 4. Add extension and redraw the frame-buffer. 5. Add step shaders for red and green... I believe you are implying that (4) is the issue. — Matic Oblak
@MaticOblak By 3 do you mean draw the bottom texture first and then draw the second one so that it replaces all of the previous framebuffer content? If so, then that works fine as well. I can also perform perfectly accurate normal blending and darken blending which would imply that the extension should be working as well. The only way I could think of how the shader output affects the conditional statement is if the output value gets read in again for some reason. But even then that wouldn't fully explain the output that I'm seeing. — Keiwan
Yes, that is what I meant by (3). I agree that the only way is that the value is being read again or being read because it has been written to twice. As I said, if you by mistake use the same shader for the first draw call then the second will produce the results you are seeing... Another way would be for you to use different methods in this shader depending on texture coordinates.. Like y<1/5 discard else y<2/5 overwrite else y<3/5 blend... else use what you do now — Matic Oblak

Keiwan Keiwan · Accepted Answer · 2018-02-27T16:50:31

I found out what the problem was (and I realize that it's not something anyone could have known from just reading the question):

There is an additional fully transparent texture being drawn between the two textures you can see above, which I had forgotten about.

Instead of accounting for that and just returning the dstColor in case the srcColor alpha is 0, the transparent texture's color information (which is (0.0, 0.0, 0.0, 0.0)) was being used when blending, therefore altering the framebuffer content.

Both the transparent texture and the final texture were drawn with the blendTest function, so the output of the first function call was then being read in when blending the final texture.

Fragment shader output interferes with conditional statement

1 Answers