GLSL Derivatives for Non-Square Anisotropic Textures

Sampling non-square textures with anisotropic filtering in GLSL requires transforming normalized screen-space derivatives into unnormalized texel space to prevent distortion. When texture dimensions deviate from a 1:1 aspect ratio, naive hardware derivatives lead to asymmetric footprint calculation and incorrect Level of Detail (LOD) selection. This article explains the derivative scaling formulas needed to calculate the sampling footprint accurately and pass scaled gradients to functions like textureGrad.

Normalized Coordinates vs. Texel Space

Standard GLSL texture coordinates \((u, v)\) reside in the normalized range \([0.0, 1.0]\). The standard implicit derivatives computed via hardware finite differences—dFdx(uv) and dFdy(uv)—represent the rate of change of normalized texture coordinates relative to screen pixels:

\[\mathbf{J}_{norm} = \begin{bmatrix} \frac{\partial u}{\partial x} & \frac{\partial u}{\partial y} \\ \frac{\partial v}{\partial x} & \frac{\partial v}{\partial y} \end{bmatrix}\]

For a square texture where width \(W\) equals height \(H\), the ratio of physical texels to normalized units is uniform along both axes. For a non-square texture of dimensions \((W, H)\), one unit of \(u\) spans \(W\) texels, while one unit of \(v\) spans \(H\) texels.

To determine the true pixel footprint on the texture surface, the derivatives must be scaled into texel space using the texture resolution vector \(\mathbf{S} = (W, H)\):

\[\frac{\partial u_{texel}}{\partial x} = W \cdot \frac{\partial u}{\partial x}, \quad \frac{\partial v_{texel}}{\partial x} = H \cdot \frac{\partial v}{\partial x}\]

\[\frac{\partial u_{texel}}{\partial y} = W \cdot \frac{\partial u}{\partial y}, \quad \frac{\partial v_{texel}}{\partial y} = H \cdot \frac{\partial v}{\partial y}\]

In vector notation:

\[\mathbf{d}_{x, texel} = \mathbf{dFdx}(uv) \odot \mathbf{S}\] \[\mathbf{d}_{y, texel} = \mathbf{dFdy}(uv) \odot \mathbf{S}\]

where \(\odot\) denotes component-wise multiplication.

Footprint Vectors and Anisotropic Ellipse Derivation

Anisotropic filtering constructs an ellipse in texel space using the major and minor axes derived from \(\mathbf{d}_{x, texel}\) and \(\mathbf{d}_{y, texel}\). The squared lengths of the differential vectors define the extent of the footprint:

\[l_x^2 = \left(W \frac{\partial u}{\partial x}\right)^2 + \left(H \frac{\partial v}{\partial x}\right)^2\]

\[l_y^2 = \left(W \frac{\partial u}{\partial y}\right)^2 + \left(H \frac{\partial v}{\partial y}\right)^2\]

The lengths of the major axis \(L_{max}\) and minor axis \(L_{min}\) of the projection ellipse determine the degree of anisotropy:

\[L_{max} = \sqrt{\max(l_x^2, l_y^2)}, \quad L_{min} = \sqrt{\min(l_x^2, l_y^2)}\]

The anisotropy ratio \(\rho\) is:

\[\rho = \frac{L_{max}}{L_{min}}\]

Hardware clamping limits \(\rho\) to the maximum supported anisotropy value \(N_{max}\) (typically 8.0 or 16.0):

\[\rho_{clamped} = \min(\rho, N_{max})\]

The base mipmap level \(\lambda\) is then calculated based on the minor axis rather than the major axis, preserving high-frequency details along the direction of highest stretch:

\[\lambda = \log_2(L_{min}) = \log_2\left(\frac{L_{max}}{\rho_{clamped}}\right)\]

Scaling Derivatives for textureGrad

When manually computing derivatives to feed into GLSL's textureGrad(sampler2D, vec2 P, vec2 dPdx, vec2 dPdy), the input vectors dPdx and dPdy must remain in normalized texture coordinate space.

If UV manipulation or geometric distortion adjustments occur in an isotropic metric space, you must rescale back to normalized space before sampling:

\[\mathbf{dPdx} = \mathbf{d}_{x, metric} \odot \left(\frac{1}{W}, \frac{1}{H}\right)\] \[\mathbf{dPdy} = \mathbf{d}_{y, metric} \odot \left(\frac{1}{W}, \frac{1}{H}\right)\]

If an aspect ratio correction factor \(A = \frac{W}{H}\) is applied to coordinates prior to sampling to prevent stretching, the derivative scaling must account for \(A\):

\[\mathbf{dPdx} = \begin{pmatrix} \frac{\partial u}{\partial x} \\ \frac{\partial v}{\partial x} \cdot \frac{1}{A} \end{pmatrix}, \quad \mathbf{dPdy} = \begin{pmatrix} \frac{\partial u}{\partial y} \\ \frac{\partial v}{\partial y} \cdot \frac{1}{A} \end{pmatrix}\]

GLSL Implementation Example

// Uniform inputs
uniform sampler2D u_Texture;
in vec2 v_TexCoord;

void main() {
    // Obtain texture dimensions
    vec2 texSize = vec2(textureSize(u_Texture, 0));

    // Base screen-space derivatives (normalized space)
    vec2 dx = dFdx(v_TexCoord);
    vec2 dy = dFdy(v_TexCoord);

    // Transform derivatives to texel space
    vec2 dx_texels = dx * texSize;
    vec2 dy_texels = dy * texSize;

    // Calculate footprint magnitudes
    float lx2 = dot(dx_texels, dx_texels);
    float ly2 = dot(dy_texels, dy_texels);

    // Determine anisotropic ratio
    float maxAxis = sqrt(max(lx2, ly2));
    float minAxis = sqrt(min(lx2, ly2));
    float maxAniso = 16.0; // Clamped to hardware target
    float anisoRatio = min(maxAxis / max(minAxis, 0.0001), maxAniso);

    // Sample using the explicitly verified gradients in normalized space
    vec4 color = textureGrad(u_Texture, v_TexCoord, dx, dy);

    // Output
    gl_FragColor = color;
}

Applying the dimension vector \(\mathbf{S} = (W, H)\) to map from normalized space to texel space ensures that anisotropic sample distributions match physical pixel distortion across non-square surfaces without introducing blurring along the narrower texture axis.