mediump Precision Artifacts in GLSL ES Coordinates

The mediump precision qualifier in OpenGL ES Shading Language (GLSL ES) balances performance and memory bandwidth on mobile GPUs, but its restricted bit-depth can severely degrade coordinate calculations. This article explores how the reduced numerical range and mantissa precision of mediump cause rendering artifacts—including texture swimming, spatial jitter, banding, and moiré effects—during vertex, texture, and procedural coordinate transformations, as well as practical techniques to mitigate these visual errors.

The Limits of mediump Precision

In GLSL ES, the mediump qualifier enforces a minimum representation that typically maps to IEEE 754 half-precision (16-bit floating-point, or FP16). Under this specification:

  • Mantissa: 10 bits of explicit precision (roughly 3 to 4 decimal digits of precision).
  • Exponent: 5 bits, giving an operational range of approximately \(\pm 65,504\) (minimum absolute value of \(\approx 2^{-14} \approx 6.1 \times 10^{-5}\)).
  • Relative Resolution: Differences smaller than roughly \(0.000976\) (\(2^{-10}\)) cannot be distinguished when values approach \(1.0\).

When calculating spatial positions or texture coordinates, these limitations produce significant rounding and truncation errors, especially when inputs vary over wide ranges.

Common Rendering Artifacts in Coordinate Calculations

1. Texture Swimming and Jitter

When continuous texture coordinates (UVs) are computed using mediump, the 10-bit mantissa restricts the number of unique sample points across a surface. As coordinates grow beyond \([0.0, 1.0]\)—such as in tiled terrains, repeating background patterns, or scrolling textures—the step size between representable numbers widens:

  • At a UV coordinate of \(1.0\), the minimum delta between steps is approximately \(0.00098\).
  • At a UV coordinate of \(64.0\), the step size expands to approximately \(0.0625\).

This quantization forces texture lookups to snap abruptly from one texel to another during camera motion. The visual result is "texture swimming," where the surface appears to crawl, vibrate, or jitter inconsistently.

2. Geometric Shaking in World-Space Transforms

Using mediump for position vectors in the vertex shader introduces truncation during world-space or view-space matrix transformations. If an object is positioned far from the coordinate origin, the higher bits consume the available mantissa, leaving minimal precision for fine spatial movements.

Vertices snap to a coarse virtual grid. When the camera pans or rotates, vertices pop between discrete grid locations, creating noticeable vertex shaking and geometric deformation.

3. Stepping and Banding in Procedural Generation

In procedural shaders (such as screen-space raymarching, Signed Distance Functions (SDFs), or noise-based shaders), coordinates drive mathematical evaluations over continuous domains.

In a raymarcher, for example, position vectors accumulate step lengths along the camera ray. Low-precision accumulation causes the ray to undershoot or overshoot geometric boundaries irregularly. This produces concentric ring artifacts, jagged edges, or stepped banding along smoothly curved surfaces.

4. Moiré Patterns and Aliasing

When coordinate precision drops, the derivative functions (dFdx, dFdy) used by the GPU to select mipmap levels receive discontinuous inputs. The irregular derivative calculations cause the texture sampler to fluctuate wildly between adjacent mipmap levels. This causes high-frequency moiré patterns and visual noise in areas where smooth filtering should occur.

Mitigation Strategies

  • Use highp for Position and Coordinate Pipelines: Reserve highp (32-bit float) for all position inputs, interpolated varying coordinates (such as UVs), and projection matrices.
  • Employ Camera-Relative Coordinates: Subtract the camera position from world-space objects on the CPU before passing matrices to the GPU. Keeping coordinate values small preserves the mantissa's fractional precision.
  • Wrap UVs Early: For repeating textures, wrap texture coordinates within the \([0.0, 1.0]\) interval using fract() before passing them to texture sampling functions, preventing coordinate values from inflating.
  • Confine mediump to Colors and Normals: Restrict mediump usage to normalized vectors, color blending, light calculations, and other mathematical operations bounded strictly within \([-1.0, 1.0]\) or \([0.0, 1.0]\).