How to colour vertices as a grid (like wireframe mode) using shaders?

Question

I've created a plane with six vertices per square that form a terrain. I colour each vertex using the terrain height value in the pixel shader.

I'm looking for a way to colour pixels between vertexes black, while keeping everything else the same to create a grid effect. The same effect you get from wireframe mode, except for the diagonal line, and the transparent part should be the normal colour.

My terrain, and how it looks in wireframe mode:

How would one go about doing this in pixel shader, or otherwise?

Answer 1

See "Solid Wireframe" - NVIDIA paper from a few years ago.

The idea is basically this: include a geometry shader that generates barycentric coordinates as a varying for each vertex. In your fragment / pixel shader, check the value of the bary components. If they are below a certain threshold, you color the pixel however you'd like your wireframe to be colored. Otherwise, light it as you normally would.

Given a face with vertices A,B,C, you'd generate barycentric values of:

A: 1,0,0

B: 0,1,0

C: 0,0,1

In your fragment shader, see if any component of the bary for that fragment is less than, say, 0.1. If so, it means that it's close to one of the edges of the face. (Which component is below the threshold will also tell you which edge, if you want to get fancy.)

I'll see if I can find a link and update here in a few.

Note that the paper is also ~10 years old. There are ways to get bary coordinates without the geometry shader these days in some situations, and I'm sure there are other workarounds. (Geometry shaders have their place, but are not the greatest friend of performance.)

Also, while geom shaders come with a perf hit, they're significantly faster than a second pass to draw a wireframe. Drawing in wireframe mode in GL (or DX) carries a significant performance penalty because you're asking the rasterizer to simulate Bresenham's line algorithm. That's not how rasterizers work, and it is freaking slow.

This approach also solves any z-fighting issues that you may encounter with two passes.

If your mesh were a single triangle, you could skip the geometry shader and just pack the needed values into a vertex buffer. But, since vertices are shared between faces in any model other than a single triangle, things get a little complicated.

Or, for fun: do this as a post processing step. Look for high ddx()/ddy() (or dFdx()/dFdy(), depending on your API) values in your fragment shader. That also lets you make some interesting effects.

Answer 2

Given that you have a vertex buffer containing all the vertices of your grid, make an index buffer that utilizes the vertex buffer but instead of making groups of 3 for triangles, use pairs of 2 for line segments. This will be a Line List and should contain all the pairs that make up the squares of the grid. You could generate this list automatically in your program.

Rough algorithm for rendering:

1. Render your terrain as normal
2. Switch your primitive topology to Line List
3. Assign the new index buffer
4. Disable Depth Culling (or add a small height value to each point in the vertex shader so the grid appears above the terrain)
5. Render the Line List

This should produce the effect you are looking for of the terrain drawn and shaded with a square grid on top of it. You will need to put a switch (via a constant buffer) in your pixel shader that tells it when it is rendering the grid so it can draw the grid black instead of using the height values.