Oblique Clipping Projection Matrix for Water in GLSL

Rendering planar water reflections requires rendering the scene from a mirrored camera position into an off-screen framebuffer. Without proper clipping, geometry situated beneath the water plane is erroneously rendered into the reflection texture, causing artifacts. Modifying the standard projection matrix using oblique near-plane clipping shifts and tilts the near clipping plane so that it coincides precisely with the water surface in view space, avoiding the performance penalties of user-defined clipping planes (gl_ClipDistance) or fragment discards.

Deriving the View-Space Clip Plane

Before modifying the projection matrix, the world-space water plane must be transformed into camera (view) space. A plane is represented as a 4D vector \(\mathbf{C} = (n_x, n_y, n_z, d)\), where \(\mathbf{n}\) is the normal pointing above the water surface, and \(d\) is the distance from the origin.

To transform the plane into view space:

\[\mathbf{C}_{\text{view}} = (\mathbf{V}^{-1})^T \cdot \mathbf{C}_{\text{world}}\]

Where \(\mathbf{V}\) is the view matrix of the reflected camera. The resulting plane equation must be normalized so that its normal vector has unit length:

\[\mathbf{C} = \frac{\mathbf{C}_{\text{view}}}{\|\mathbf{C}_{\text{view}}.xyz\|}\]

Calculating the Frustum Corner

The standard perspective projection matrix maps the viewing frustum into Normalized Device Coordinates (NDC) ranging from \([-1, 1]\) (OpenGL standard). To alter the near plane without distorting the far plane or clipping the sides of the frustum, find the corner of the frustum that is farthest from the clip plane.

This point \(\mathbf{q}\) in view space is derived by evaluating the signs of the clipping plane normal components:

\[q_x = \frac{\operatorname{sgn}(C_x) + P_{02}}{P_{00}}\] \[q_y = \frac{\operatorname{sgn}(C_y) + P_{12}}{P_{11}}\] \[q_z = -1.0\] \[q_w = \frac{1.0 + P_{22}}{P_{23}}\]

Here, \(P_{ij}\) refers to row \(i\), column \(j\) of the standard projection matrix \(\mathbf{P}\).

Scaling the Clip Plane

The view-space clipping plane vector \(\mathbf{C}\) is scaled by an adjustment factor so that the dot product of the scaled plane and the corner point \(\mathbf{q}\) equals \(2\) (the depth range of NDC space in OpenGL):

\[\mathbf{C}' = \mathbf{C} \cdot \left( \frac{2.0}{\mathbf{C} \cdot \mathbf{q}} \right)\]

Replacing the Matrix Row

For an OpenGL projection matrix where NDC \(z \in [-1, 1]\), the third row of the projection matrix directly controls the near and far plane mapping. Replace the third row (index 2 in zero-based indexing) with the modified plane equation:

\[\mathbf{P}' = \begin{bmatrix} P_{00} & P_{01} & P_{02} & P_{03} \\ P_{10} & P_{11} & P_{12} & P_{13} \\ C'_x & C'_y & C'_z + 1.0 & C'_w \\ P_{30} & P_{31} & P_{32} & P_{33} \end{bmatrix}\]

For systems using a \([0, 1]\) clip volume (such as Vulkan, Direct3D, or glClipControl(GL_LOWER_LEFT, GL_ZERO_TO_ONE)), the scale factor is \(\frac{1.0}{\mathbf{C} \cdot \mathbf{q}}\) and the third-row replacement does not add \(1.0\) to \(C'_z\):

\[\mathbf{P}'_{\text{zero\_to\_one}} = \begin{bmatrix} P_{00} & P_{01} & P_{02} & P_{03} \\ P_{10} & P_{11} & P_{12} & P_{13} \\ C'_x & C'_y & C'_z & C'_w \\ P_{30} & P_{31} & P_{32} & P_{33} \end{bmatrix}\]

Implementation in GLSL

This matrix transformation is typically precalculated on the CPU, but it can be evaluated directly in a GLSL compute shader or vertex stage when dynamic adjustments are needed:

mat4 createObliqueNearPlaneMatrix(mat4 proj, vec4 clipPlaneViewSpace) {
    // Determine the corner point opposite the clipping plane
    vec4 q;
    q.x = (sign(clipPlaneViewSpace.x) + proj[2][0]) / proj[0][0];
    q.y = (sign(clipPlaneViewSpace.y) + proj[2][1]) / proj[1][1];
    q.z = -1.0;
    q.w = (1.0 + proj[2][2]) / proj[3][2];

    // Calculate the scaled plane vector
    vec4 c = clipPlaneViewSpace * (2.0 / dot(clipPlaneViewSpace, q));

    // Replace the third row (column-major indexing: proj[col][row])
    mat4 obliqueProj = proj;
    obliqueProj[0][2] = c.x;
    obliqueProj[1][2] = c.y;
    obliqueProj[2][2] = c.z + 1.0; // OpenGL [-1, 1] range
    obliqueProj[3][2] = c.w;

    return obliqueProj;
}

Depth Precision Trade-offs

Tilting the near plane alters how depth values are distributed across the viewport. As the angle between the original near plane and the arbitrary water plane increases, depth buffer precision degrades toward the far plane. To minimize depth fighting (Z-fighting) artifacts during the reflection pass:

  1. Keep the camera's original near plane distance as high as possible.
  2. Ensure the far plane distance is tightly bounded to the visible scene.
  3. Use a 24-bit or 32-bit floating-point depth buffer (GL_DEPTH_COMPONENT32F) for reflection framebuffers.