Making composite of two alpha textures in a glsl shader

Question

I wish to combine two textures in a shader exactly the same as you would expect from a photoshop like program. That is, one texture ontop of another as if layered. Both textures will have alpha, and the result output should also have the correct combined alpha.

So the goal is;

However despite googleing for the formula, I am not able to get the correct colors out. Using the referance here: (https://microsoft.github.io/Win2D/html/PremultipliedAlpha.htm) for example produces;

(note difference in grey 50% overlay)

The shader code is set in libgdx to use this as the blending;

  context.setBlending(true,GL20.GL_SRC_ALPHA ,GL20.GL_ONE_MINUS_SRC_ALPHA);

I am fairly sure the mistake is in my fragment shaders blending (see code in image above), because if I pass the input texture (A) straight to gl_FragColor, then I get a pixel-perfect color match of A, including its alpha.
eg;

gl_FragColor = backdiffuse;

works fine.

=====

edit

Following advice, I have tried using a premulitiplied alpha instead.

context.setBlending(true,GL20.GL_ONE, GL20.GL_ONE_MINUS_SRC_ALPHA);

However, this produces wrong effects even when the shader is just set to output A

backdiffuse = texture2D(u_backSample2D, vTexCoord);     
backdiffuse=backdiffuse*backdiffuse.a; //convert to premultiply     
gl_FragColor =  backdiffuse;

Am I setting libgdxs blending wrong? This is clearly too bright.

Solution (thanks to accepted answer);

 backdiffuse.rgb *= backdiffuse.a; //premultiply source 
 v_backBackColor.rgb *= v_backBackColor.a; //premultiply source 

finalCol.rgb = backdiffuse.rgb + (v_backBackColor.rgb * (1 - backdiffuse.a));       
finalCol.a = backdiffuse.a + (v_backBackColor.a * (1.0 - backdiffuse.a));

Tenfour04 Tenfour04 · Accepted Answer · 2017-06-03T05:33:20

If you want to use the premultiplied alpha equation, you must also draw the output to screen with the appropriate blend equation: GL_ONE, GL_ONE_MINUS_SRC_ALPHA.

And the destination alpha must also be multiplied by one minus source alpha. So:

finalCol.a = backDiffuse.a + v_backBackColor * (1.0 - backDiffuse.a);

If you want to use the more typical blend equation you are currently using, the equations in the shader would be more complicated. You can find them if you look up alpha compositing on Wikipedia. However, premultiplied alpha is a preferred format since it avoids dark or light fringing on antialiased edges of sprites.

Making composite of two alpha textures in a glsl shader

1 Answers