Silhouette detection (geometry shader) for edges that connect only one triangle

Question

I want to draw a mesh silhouette using geometry shader(line_strip).

The problem occurs when the mesh has edges with only one triangle(like the edge of a cloth). Enclosed(all edges connect 2 triangles) meshes work.

I built adjacency index buffer using RESTART_INDEX where neighbor vertex didn't exist, during rendering I tried to used:

glEnable(GL_PRIMITIVE_RESTART);
glPrimitiveRestartIndex(RESTART_INDEX); // RESTART_INDEX = ushort(-1)

The result:

As you can see head, feet, hands are OK, other parts of the model not so much.

// silhouette.gs.glsl
void main(void) {
  vec3 e1 = gs_in[2].vPosition - gs_in[0].vPosition;   // 1 ---- 2 ----- 3
  vec3 e2 = gs_in[4].vPosition - gs_in[0].vPosition;   // \     / \     /
  vec3 e3 = gs_in[1].vPosition - gs_in[0].vPosition;   //  \   e1  \   /
  vec3 e4 = gs_in[3].vPosition - gs_in[2].vPosition;   //   \ /     \ /
  vec3 e5 = gs_in[4].vPosition - gs_in[2].vPosition;   //    0 -e2-- 4
  vec3 e6 = gs_in[5].vPosition - gs_in[0].vPosition;   //     \     /
  //                                                   //      \   /
  vec3 vN = cross(e1, e2);                             //       \ /
  vec3 vL = u_vLightPos - gs_in[0].vPosition;          //        5

How does gs manage vertices when it encounters a primitive restart index? Example: For some triangles 1,3 or 5 shouldn't have any vertex.

Answer 1

If a triangle edge is not part of any other triangle then it is still adjacent to the back face of its own triangle. For example: if there is no other triangle attached to e1 in your diagram, you can use the third vertex of the triangle (4 in this case, as e1 consists of 0 and 2) in place of 1. This will not require any additional check in the geometry shader.

Answer 2

I removed:

glEnable(GL_PRIMITIVE_RESTART);
glPrimitiveRestartIndex(RESTART_INDEX);

during draw pass and this is the result:

which closer to the desired result.

Answer 3

Another possible solution to this problem could be tackled while constructing the adjacency index buffer.

// for every mesh
    // for every triangle in the mesh // first pass - edges & neighbors
        ...

    // for every triangle in the mesh // second pass - build indices
        const triangle_t* triangle = &tcache[ti]; // unique index triangles
        for(i = 0; i < 3; ++i)  // triangle = 3 edges
        {
          edge_t edge(triangle->index[i], triangle->index[(i+1) % 3]); // get edge

          neighbors_t neighbors = oEdgeMap[edge]; // get edge neighbors

          ushort tj = getOther(neighbors, ti); // get the opposite triangle index 
          ushort index = RESTART_INDEX; // = ushort(-1)

          if(tj != (ushort)(-1))
          {
            const triangle_t& opposite = tcache[tj]; // opposite triangle
            index = getOppositeIndex(opposite, edge) // opposite vertex from other triangle
          }
          else
          {
            index = triangle->index[i];
          }

          indices[ii+i*2+0] = triangle->index[i];
          indices[ii+i*2+1] = index;
        }

Instead of using a RESTART_INDEX as mentioned in the first post, use the same vertex. Edge triangles will have a neighbor triangle with a 0 length edge(built from the same vertices). I think this needs to be checked during gs.

Does the geometry shader fire fore triangle with incomplete adjacency information?