How Stencil Buffer Operates in OpenGL

Question

I'm trying to achieve the following effect in OpenGL as depicted in the given link http://commons.wikimedia.org/wiki/File:OpenGL_Tutorial_Stencil.png, code snippet is also below, this is the official openGL wikipedia tutorial.

Although it has code and comment out explanations I still find it hard to understand the logic of stencil buffer operation in openGL. I kindly remind that before starting to stenciling I read most of the instructions in the red book, so I know and I'm familiar with, how glStencilFunc, glStencilOp and glStencilMask etc.. works, but there are still some obscure and unclarified things in my mind related to the stenciling.

Here is my walkthrough on how I interpret the code :

With glStencilFunc(GL_NEVER, 1, 0xFF); glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP); I believe that stencil test will always pass and failing the stencil test will result in 1s filled in stencil buffer due to the instructions set in glStencilOp
If the previous code blocks (#1) sets the all pixels in stencil buffer what we achieve with glStencilMask(0xFF); procedure call. Do we guarantee that all the 1s are written into stencil buffer successfully or what actually?
We draw the circle then switch on the masks but this time glStencilMask(0x00), here is one of the confusion, I think this deals with preventing stencil buffer values to be overriden by new values but code comments says that this simply fills the stencil buffer with 0s.
Finally, if you we don't change any value (due to glStencilMask(0x00) ) in stencil buffer what is the sense of having the

// fill 0s glStencilFunc(GL_EQUAL, 0, 0xFF); /* (nothing to draw) */ // fill 1s glStencilFunc(GL_EQUAL, 1, 0xFF);

Main program body :

 void onDisplay() {       
  glClearColor(0.1, 0.1, 0.1, 1.0);
  glClear(GL_COLOR_BUFFER_BIT); 

  glClear(GL_DEPTH_BUFFER_BIT);
  glEnable(GL_STENCIL_TEST);
  glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
  glDepthMask(GL_FALSE);
  glStencilFunc(GL_NEVER, 1, 0xFF);
  glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);  // draw 1s on test fail (always)

  // draw stencil pattern
  glStencilMask(0xFF);
  glClear(GL_STENCIL_BUFFER_BIT);  // needs mask=0xFF
  draw_circle();

  glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
  glDepthMask(GL_TRUE);
  glStencilMask(0x00);
  // fill 0s
  glStencilFunc(GL_EQUAL, 0, 0xFF);
  /* (nothing to draw) */
  // fill 1s
  glStencilFunc(GL_EQUAL, 1, 0xFF);

  draw_scene();

  glDisable(GL_STENCIL_TEST);

  glutSwapBuffers();
}

SDLeffler SDLeffler · Accepted Answer · 2014-03-21T05:40:12

glStencilMask(0xFF) is analogous to the glColorMask(true, true, true, true) or glDepthMask(true) functions - 0xFF is a mask that is bitwise ANDed with any stencil data before it is written to the stencil buffer. Therefore, via glStencilMask(0xFF), we are ensuring that any value that OpenGL attempts to write to the stencil buffer is successfully written.

glStencilMask(0x00): when 0x00 is bitwise ANDed with any byte, the output will always be 0. So, any time bytes are attempted to be written to the stencil buffer, the byte written in will be 0. Essentially, at this point, it simply ensures that the stencil buffer does NOT have any new data written to it.

As per my interpretation, the section with the /* (nothing to draw) */ comment is where you would draw materials intended to be outside the stencil tested circle. 0 is the default value in the stencil buffer; therefore, since the area in the circle is marked 1, pixels that are outside the circle will pass, and pixels inside will be discarded. It's simply a placeholder.

I believe that when the comments say // fill 0s, what they mean is that "anything drawn here will fill the part of the stencil buffer that is set to 0. Considering the call to glStencilFunc with GL_EQUAL and 0 as the arguments, this is what will happen. I also am sure that when the comments say // fill 1s they mean the same but with the part of the stencil buffer set to 1 (i.e. the area where the circle has been drawn to the stencil buffer.)

How Stencil Buffer Operates in OpenGL

1 Answers