How does tessellation know what vertices belong together in a face, when patch size > 4?

Question

When not using tessellation shaders, you can pass a primitive type (GL_TRIANGLES, GL_TRIANGLE_STRIP, etc.) to let OpenGL know how the vertex stream is representing geometry faces.

Using tessellation shaders, GL_PATCHES replaces these primitive type enums. This makes sense in my head when processing patches of size 3 or 4, and setting the corresponding layout in the TES to triangles or quads.

But if I have a patch of size 16 (some tutorials do this), how does the TPG know what 3 or 4 vertices form what faces? I've read several times that the order in your vertex buffer does not matter when GL_PATCHES is used, but surely there must be a point where a specific set of 3 vertices is considered a triangle (to pass to e.g. the geometry shader). How is this decided?

Answer 1

Tessellation doesn't actually use your patch data. The tessellation unit, the part of the pipeline that actually generates triangles (or lines), operates on an abstract patch. If your TES specifies that the abstract patch is a quad, then the tessellator will perform tessellation on a unit quad.

How many patch vertices you have is completely irrelevant to this process.

The TES is a bit like a vertex shader; it gets called once per vertex. But not per vertex in the GL_PATCHES primitive. It's once per-vertex of the tessellated primitive. So, if you tessellate a quad, and your outer levels are (4, 4, 4, 4) and the inner levels are (4, 4), that will generate 25 vertices. The TES will be called that many times (at least; it can be called multiple times for the same vertex), and it will be told where within the abstract patch that particular vertex is.

The job of the TES is to decide how to use the abstract patch coordinate and the real patch vertex data to generate the actual per-vertex data to be used by the rendering pipeline. For example, if you're creating a Bezier patch, you would have 16 patch vertices. The TES's job would be to take the abstract patch coordinate and use bicubic Bezier interpolation to interpolate the 16 positions on the patch to that particular location in the abstract patch. That becomes the output position. Normals, texture coordinates, and other data can be computed similarly.

Answer 2

The short answer is: The TPG (Tessellation Primitive Generator) doesn't know/care at all about the patch size.

The TPG only cares about the specified input layout of the tessellation evaluation shader (triangle, quad, isoline) and the tessellation levels set by the control shader. It does not have access to the patch itself but generates the tessellation coordinates only based on the type and level used in this tessellation.

The user is than in the tessellation evaluation shader responsible for establishing the relation between the per control point parameters (passed from the tessellation control shader, donated by in/out, somehow similar to varyings) and the gl_TessCoord (coming from the TPG).

Note, that the amount of per control point parameters and the amount of tessellation coordinates is not necessarily the same.

For further reading, the following articles might help: Primitive Processing in Open GL, especially Figure 8.1