GLSL triangles_adjacency for Silhouette Edges

In OpenGL and GLSL, the layout(triangles_adjacency) input layout qualifier in a geometry shader grants access to the vertices of an active triangle alongside the outer vertices of its three adjacent neighboring triangles. This complete topological context is essential for detecting geometric silhouette edges in real time. Because a silhouette edge is defined by the boundary between a front-facing surface and a back-facing surface, a geometry shader requires access to adjacent face normals—an operation made possible only by the extra vertex data provided by adjacency primitives.

The Mechanism of Silhouette Detection

A geometric silhouette occurs along an edge shared by two triangles where:

  1. One triangle faces the viewer (front-facing).
  2. The adjacent triangle faces away from the viewer (back-facing).

A standard layout(triangles) geometry shader receives only three vertices representing a single, isolated triangle. While the shader can compute whether this triangle faces the camera using its vertex positions and the camera's view vector, it has no topological awareness of adjacent geometry. It cannot determine if a neighboring triangle is facing toward or away from the camera, making edge classification impossible without multi-pass techniques or precomputed data buffers.

Primitive Layout and Vertex Ordering

When configuring the geometry shader input with layout(triangles_adjacency) in;, the shader receives six vertices per primitive (gl_in[6]).

The standard OpenGL winding order defines these vertices as follows:

  • \(V_0\), \(V_2\), \(V_4\): The primary triangle.
  • \(V_1\): The adjacent vertex sharing the edge between \(V_0\) and \(V_2\).
  • \(V_3\): The adjacent vertex sharing the edge between \(V_2\) and \(V_4\).
  • \(V_5\): The adjacent vertex sharing the edge between \(V_4\) and \(V_0\).

Using these six vertices, the geometry shader can construct the normal of the primary triangle and the normals of the three neighboring triangles:

  • Primary Triangle: Formed by \((V_0, V_2, V_4)\).
  • Adjacent Triangle 1: Formed by \((V_0, V_1, V_2)\).
  • Adjacent Triangle 2: Formed by \((V_2, V_3, V_4)\).
  • Adjacent Triangle 3: Formed by \((V_4, V_5, V_0)\).

Silhouette Edge Computation

Within the geometry shader, silhouette classification is evaluated for each of the three edges of the primary triangle:

  1. Calculate the View Vectors: Transform the vertices into view space or world space, and compute vectors pointing from the vertices to the camera position.
  2. Calculate the Primary Face Facing: Compute the normal of \((V_0, V_2, V_4)\) using the cross product: \[\vec{N}_{main} = (V_2 - V_0) \times (V_4 - V_0)\] Compute the dot product of \(\vec{N}_{main}\) and the view vector to classify the primary face as front-facing (\(> 0\)) or back-facing (\(\le 0\)).
  3. Evaluate Shared Edges: For a specific edge (for example, the edge between \(V_0\) and \(V_2\)), compute the normal of the adjacent triangle using \((V_0, V_1, V_2)\): \[\vec{N}_{adj} = (V_1 - V_0) \times (V_2 - V_0)\] Compute the dot product of \(\vec{N}_{adj}\) and the view vector.
  4. Identify the Silhouette: If the primary triangle is front-facing and the adjacent triangle is back-facing (or vice versa), the shared edge is a silhouette edge.

Rendering Extrusions and Outlines

Once a silhouette edge is identified, the geometry shader can emit the edge as a line primitive or expand it into a screen-aligned quad (a "fin") using triangle strips. This technique is widely used in non-photorealistic rendering (NPR), cartoon/cel-shading outlines, and shadow volume generation, where shadow silhouettes must be extruded away from light sources to construct volume boundaries.