Prevent Texture Bleeding in GLSL Texture Atlases

Texture bleeding in a GLSL texture atlas occurs when texture filtering (such as bilinear filtering or mipmapping) samples pixels outside the designated sub-texture region, pulling in unwanted colors from adjacent sub-textures or empty space. The standard and most effective padding algorithm to eliminate this artifact is edge extrusion (also referred to as border replication or dilation). This article explains how edge extrusion functions, how to size gutters for mipmapping, and how shader-level UV clamping can complement asset padding.

The Root Cause: Texture Filtering Artifacts

When a shader samples a texture using GL_LINEAR, the GPU interpolates between four neighboring texels. If a sample falls near the boundary of a sub-texture, the interpolation window extends beyond the sub-texture's UV coordinates, pulling colors from neighboring assets.

When mipmapping is enabled, the issue multiplies. Each successive mip level halves the resolution of the atlas, blending adjacent 2x2 texel blocks into a single texel. Without adequate margins, distinct sub-textures merge directly into each other at lower mip levels.

The Solution: Edge Extrusion (Border Duplication)

The edge extrusion algorithm prevents filtering artifacts by adding a transparent border (gutter) around each sub-texture and replicating the boundary pixels outward into this gutter.

Instead of padding with transparent black (rgba(0, 0, 0, 0)) or solid color, the algorithm operates as follows:

  1. Allocate Gutters: Reserve a transparent spacing border around each packed sub-texture inside the atlas.
  2. Replicate 1D Edges: Extend the outermost row of pixels upward and downward, and extend the outermost columns leftward and rightward into the gutter.
  3. Replicate Corners: Fill the corner diagonal spaces of the gutter by duplicating the outermost corner pixel of the sub-texture.

By filling the gutter with the exact color and alpha of the sub-texture's borders, any bilinear interpolation that steps outside the boundary samples the identical color, resulting in zero visible discoloration or seams.

Sizing Gutters for Mipmaps

For standard bilinear filtering without mipmaps, an extruded border of 1 pixel is mathematically sufficient.

For mipmapped atlases, the gutter must be large enough so that sub-textures do not bleed into one another at the smallest generated mip level. The required extrusion size is calculated as:

\[\text{Padding} = 2^{N}\]

where \(N\) is the number of mipmap levels generated. For example, preserving clear boundaries across 4 mip levels requires at least 16 pixels of extruded border between neighboring sub-textures (8 pixels of padding per side).

GLSL In-Shader Boundary Clamping

When modifying the texture atlas directly is not possible, texture bleeding can be constrained entirely within the GLSL fragment shader by clamping the interpolated UV coordinates to the inner half-texel boundary of the sub-texture:

// Calculate half-texel offset
vec2 halfTexel = 0.5 / u_atlasSize;

// Clamp UV to sub-texture bounding box
vec2 safeMin = u_subTextureMin + halfTexel;
vec2 safeMax = u_subTextureMax - halfTexel;
vec2 clampedUV = clamp(v_uv, safeMin, safeMax);

// Sample using explicit derivatives to prevent mipmap distortion at borders
vec4 color = textureGrad(u_atlas, clampedUV, dFdx(v_uv), dFdy(v_uv));

Using textureGrad with the original v_uv derivatives prevents the GPU from computing abnormal derivatives at the clamped boundary, which would otherwise force the sampler to choose an excessively blurry mip level.