OpenGL - mask with multiple textures

Question

I have implemented masking in OpenGL according to the following concept:

The mask is composed of black and white colors.
A foreground texture should only be visible in the white parts of the mask.
A background texture should only be visible in the black parts of the mask.

I can make the white part or the black part work as supposed by using glBlendFunc(), but not the two at the same time, because the foreground layer not only blends onto the mask, but also onto the background layer.

Is there anyone who knows how to accomplish this in the best way? I have been searching the net and read something about fragment shaders. Is this the way to go?

I know there's a way of doing this, I'm pretty sure it's a simple 'mask' function, but it's been ages since I last used OpenGL. Have you seen: nehe.gamedev.net/data/lessons/lesson.asp?lesson=20 — Chris Pfohl
Thanks for the fast reply. Yes, my implementation is based on that tutorial and works for one mask with one texture, but when I use two textures for one mask as described above, the result is not what I expected: the foreground image colors also blend onto the background layer instead of the mask alone. Should I post a screenshot to illustrate my problem? — red
It's a very basic (simple) task for a fragment shader. Though, I can't point you to a proper Fixed-Function solution. — kvark

Stefan Monov Stefan Monov · Accepted Answer · 2011-02-24T00:21:43

This should work:

glEnable(GL_BLEND);
// Use a simple blendfunc for drawing the background
glBlendFunc(GL_ONE, GL_ZERO);
// Draw entire background without masking
drawQuad(backgroundTexture);
// Next, we want a blendfunc that doesn't change the color of any pixels,
// but rather replaces the framebuffer alpha values with values based
// on the whiteness of the mask. In other words, if a pixel is white in the mask,
// then the corresponding framebuffer pixel's alpha will be set to 1.
glBlendFuncSeparate(GL_ZERO, GL_ONE, GL_SRC_COLOR, GL_ZERO);
// Now "draw" the mask (again, this doesn't produce a visible result, it just
// changes the alpha values in the framebuffer)
drawQuad(maskTexture);
// Finally, we want a blendfunc that makes the foreground visible only in
// areas with high alpha.
glBlendFunc(GL_DST_ALPHA, GL_ONE_MINUS_DST_ALPHA);
drawQuad(foregroundTexture);

This is fairly tricky, so tell me if anything is unclear.

Don't forget to request an alpha buffer when creating the GL context. Otherwise it's possible to get a context without an alpha buffer.

Edit: Here, I made an illustration.

Edit: Since writing this answer, I've learned that there are better ways to do this:

If you're limited to OpenGL's fixed-function pipeline, use texture environments
If you can use shaders, use a fragment shader.

The way described in this answer works and is not particularly worse in performance than these 2 better options, but is less elegant and less flexible.

OpenGL - mask with multiple textures

2 Answers