OpenGL transparency in texture when render with stencil buffer

Question

The question has been updated thanks to the comments.

Screenshot of how textures overlap

To draw 2 points with brush texture using the stencil buffer to avoid textures transparency overlap, the following code is used:

    glEnable(GL_STENCIL_TEST.gluint)

    glClear(GL_STENCIL_BUFFER_BIT.gluint | GL_DEPTH_BUFFER_BIT.gluint)

    glStencilOp(GL_KEEP.gluint, GL_KEEP.gluint, GL_REPLACE.gluint)

    glStencilFunc(GL_ALWAYS.gluint, 1, 1)
    glStencilMask(1)

    glDrawArrays(GL_POINTS.gluint, 0, 1)

    glStencilFunc(GL_NOTEQUAL.gluint, 1, 1)
    glStencilMask(1)

    glDrawArrays(GL_POINTS.gluint, 1, 1)

    glDisable(GL_STENCIL_TEST.gluint)

And stencil buffer works, however, each point fill a full rectangle in the stencil buffer, but a texture image has transparency. So maybe texture used in the wrong way?

The texture is loaded like this

    glGenTextures(1, &gl_id)
    glBindTexture(GL_TEXTURE_2D.gluint, gl_id)
    glTexParameteri(GL_TEXTURE_2D.gluint, GL_TEXTURE_MIN_FILTER.gluint, GL_LINEAR)
    glTexImage2D(GL_TEXTURE_2D.gluint, 0, GL_RGBA, gl_width.int32, gl_height.int32, 0, GL_RGBA.gluint, GL_UNSIGNED_BYTE.gluint, gl_data)

Blending set as

    glEnable(GL_BLEND.gluint)
    glBlendFunc(GL_ONE.gluint, GL_ONE_MINUS_SRC_ALPHA.gluint)

Could you please advice where to look in order to fill 1s in stencil buffer by exactly not transparent area of brush image?

Try glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA) glBlendEquation(GL_MAX) — Rabbid76
@Rabbid76 thank you for the comment! With such blend, it does not draw at all. I've added initGL code that is used. Please check it out if you would have a chance. — smartwolf
have you considered using stencil masking? what you do is discard if stencil has been written to, and write to stencil and color otherwise. to clear the stencil so that it can be used by other brushes, render the exact same geometries and reset the stencil bit. apply write mask on color — Andreas
@Andreas thank you for your suggestion! Could you please take a look at updated questions with how I use a stencil? — smartwolf
looking closer it does not look right at all... your glStencilOp says "keep stencil value if fail, replace with reference if pass". first glStencilFunc should be glStencilFunc(GL_NOTEQUAL, 1, -1) - "pass if stencil value is not (reference & mask), reference is 1, mask is all ones". this will together with glStencilOp "replace where stencil value is not 1 with 1". all glStencilMask can be removed since the default (all ones) suffices. second draw can in your code be replaced with glClear, or redrawing with glStencilFunc(GL_ALWAYS, 0, 0) - "always pass, replace with zero". — Andreas

Rabbid76 Rabbid76 · Accepted Answer · 2018-07-07T20:01:37

I recommend to discard the transparent parts of the texture in the fragment shader. A fragment can be completely skipped in the fragment shader by the discard keyword.
See Fragment Shader - Special operations.

Use a small threshold and discard a fragment, if the alpha channel of the texture color is below the threshold:

vec4  texture_color = .....;
float threshold = 0.01;
if ( texture_color.a < threshold )
    discard;

An other possibility would be to use an Alpha test. This would be only available in OpenGL compatibility profile, but not in core profile or OpenGL ES.

See Khronos OpenGL-Refpages glAlphaFunc:

The alpha test discards fragments depending on the outcome of a comparison between an incoming fragment's alpha value and a constant reference value.

With the following alpha test, the fragments whos alpha channel is below the threshold are discarded:

float threshold = 0.01;
glAlphaFunc(GL_GEQUAL, threshold);
glEnable(GL_ALPHA_TEST)

OpenGL transparency in texture when render with stencil buffer

1 Answers