Black out everything outside a polygon

Question

I have a map, for simplicity let's say it's just a single texture. On top of this map, I have a polygon which indicates the route that the user must follow.

What I want is to draw everything outside the polygon black. Or, of course to only draw things inside the polygon.

To explain it better I made a picture. The blue lines define the polygon, with each corner being a point in the polygon. The red with yellow lines is the part I want to black out of the picture, and only leave the red with the purple lines. The polygon starts from A and ends at B.

enter image description here

Would it not be better to just paint whats meant to be black black? And perhaps a solid color for the part that needs to be rendered a solid color. Now i am blind and I'm holding you responsible. — Madmenyo
@MennoGouw The red square in the picture is my map's ground(for simplicity lets say its a big solid texture),the blue polygon is not the same in every level of the game and sometimes it gets created by random ,so I cant just predraw a different map ground for every level(if that's what you are saying). — SteveL
I'm saying to tone down the contrast, i need my sunglasses to see anything :D. — Madmenyo
@MennoGouw ,Haha I thought you where talking about how to solve the problem.... Yea I know the picture is lame.The program that i use to paint starts with a red canvas and blue and yellow are the first colors int the list,I am really lazy. — SteveL

Reto Koradi Reto Koradi · Accepted Answer · 2014-08-23T15:48:36

The main challenge here is that you need to draw a non-convex polygon, which is not directly supported by OpenGL. One approach for drawing it is to break it down into triangles. Depending on how much you know about the shape of the polygon, and how constraint it is, this might be fairly easy. For a general non-convex polygon, it's slightly painful. But there are algorithms you can find if you search for keywords like "polygon triangulation".

OpenGL has another mechanism that works great for these kinds of use cases: stencil buffers. You can find an explanation of this approach in the Red Book under Drawing Filled, Concave Polygons Using the Stencil Buffer. The main idea is that you can draw a triangle fan with an arbitrary origin and your polygon vertices. The pixels that are inside the polygon will then be drawn an odd number of times, while the pixels outside the polygons are drawn an even number of times. The stencil buffer is used to track the odd/even count of how many times each pixel is drawn.

To outline the main steps:

While setting up your context and drawing surface, make sure that you request a configuration with a stencil buffer.
During drawing, clear the stencil buffer along with the color buffer, and enable the stencil test.
```
glClear(GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
glEnable(GL_STENCIL_TEST);
```
Set up state for the render pass that counts if pixels are rendered an odd/even number of times. Note that this must only write to the stencil buffer, so color writes are disabled. The key part is the GL_INVERT for the stencil op, which flips the stencil value each time a pixel is rendered, giving us the odd/even count.
```
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilFunc(GL_ALWAYS, 0, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_INVERT);
glStencilMask(1);
```
Render a triangle fan with an arbitrary point, e.g. (0.0, 0.0), as the first vertex, and the polygon corners as the remaining vertices. The polygon must be closed, so the first and last polygon corner must be the same. If p1, p2, ... , pN are your polygon corners, the sequence of vertices for the GL_TRIANGLE_FAN draw call is:
```
(0.0f, 0.0f), p1, p2, ... , pN, p1
```
You can use a trivial shader for this pass since the color value is not even written.
Enable color writes again, and set up the stencil test attributes to render only pixels that were rendered an odd number of times in the previous pass.
```
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
```
Draw your entire content. Only the part within the polygon outline will be rendered, the rest is eliminated by the stencil test.

Black out everything outside a polygon

1 Answers