Polygonal Bokeh Simulation Using Separable GLSL Passes

Simulating polygonal diaphragm bokeh—such as the hexagonal or octagonal highlights produced by physical camera apertures—is typically computationally prohibitive because non-circular kernels are not natively separable along Cartesian axes. In real-time rendering, this limitation is bypassed by decomposing the 2D polygonal point spread function (PSF) into a combination of directional 1D filter passes or by using complex-valued separable approximations. Executed across multiple render passes in GLSL, these separable techniques reduce the convolution complexity from \(O(R^2)\) to \(O(R)\) per pixel, achieving authentic geometric bokeh at interactive frame rates.

The Non-Separability Challenge

A standard circular blur or Gaussian blur can be factored into horizontal and vertical 1D convolutions: \(K(x, y) = f(x) \cdot g(y)\). Polygonal apertures cannot be decomposed in this Cartesian manner because their boundary conditions depend on linear combinations of \(x\) and \(y\).

Performing a brute-force 2D convolution requires \(N^2\) texture samples per pixel for a kernel of diameter \(N\). To make this practical for real-time graphics pipelines, developers use two primary separable approximations:

Directional Line Convolution (Minkowski Sum Approach): Convolving multiple 1D line segments oriented along the aperture's edge directions.
Complex-Valued Separable Decomposition: Approximating a polygonal mask through a linear combination of separable 1D filters operating in the complex domain.

Method 1: Multi-Axis Directional Convolution

Any convex polygon can be approximated by convolving 1D box filters along axes parallel to its sides. For example, a regular hexagon can be simulated by successive 1D blurs along three directions separated by 60 degrees (0°, 60°, and 120°).

Pass Structure

The rendering pipeline allocates intermediate ping-pong framebuffers:

Pass 1: Sample along direction vector \(\vec{v}_1 = (1, 0)\) across the Circle of Confusion (CoC) radius.
Pass 2: Sample the output of Pass 1 along direction vector \(\vec{v}_2 = (\cos(60^\circ), \sin(60^\circ))\).
Pass 3: Sample the output of Pass 2 along direction vector \(\vec{v}_3 = (\cos(120^\circ), \sin(120^\circ))\).

GLSL Directional Kernel Fragment Shader

Each pass uses a 1D loop oriented along the uniform step vector:

#version 450 core

in vec2 vTexCoord;
out vec4 fragColor;

uniform sampler2D uSourceTex;
uniform sampler2D uCoCTex;
uniform vec2 uDirection; // Normalized screen-space direction
uniform float uMaxCoC;
uniform int uSampleCount;

void main() {
    float coc = texture(uCoCTex, vTexCoord).r * uMaxCoC;
    if (coc <= 0.5) {
        fragColor = texture(uSourceTex, vTexCoord);
        return;
    }

    vec4 accum = vec4(0.0);
    float totalWeight = 0.0;
    vec2 stepVec = (uDirection * coc) / float(uSampleCount);

    for (int i = -uSampleCount; i <= uSampleCount; ++i) {
        vec2 sampleUV = vTexCoord + stepVec * float(i);
        vec4 color = texture(uSourceTex, sampleUV);
        
        // Weighting to maintain energy conservation
        float weight = 1.0; 
        accum += color * weight;
        totalWeight += weight;
    }

    fragColor = accum / totalWeight;
}

While successive convolution of box filters produces a smooth B-spline-like shape rather than a hard-edged polygon, shaping the 1D weights with inverse-filtering or sharpening stages preserves the crisp polygonal edges characteristic of photographic lenses.

Method 2: Complex-Valued Separable Approximation

A mathematically precise approach involves decomposing the 2D polygonal kernel matrix into a small sum of rank-1 outer products using complex numbers. A 2D polygonal kernel \(K\) can be represented as:

\[K \approx \text{Re}\left( \sum_{i=1}^{M} u_i v_i^T \right)\]

where \(u_i\) and \(v_i\) are complex-valued 1D vectors, and \(M\) is typically 2 or 3.

Implementation with RGBA Channels

Modern graphics hardware does not natively support complex numbers, but a single four-channel (RGBA) floating-point texture can store two complex components per pixel (Real and Imaginary parts for two separate bases):

Horizontal Pass: Samples the real input image and multiplies it by a complex 1D kernel \(u(x) \in \mathbb{C}^2\). The output stores \((Re_1, Im_1, Re_2, Im_2)\) into an RGBA16F or RGBA32F buffer.
Vertical Pass: Reads the complex intermediate values, multiplies them by the complex 1D vertical kernel \(v(y) \in \mathbb{C}^2\) via complex multiplication, and takes the real component of the result.

GLSL Complex Multiplication Pass

The vertical accumulation step implements complex arithmetic across the channels:

#version 450 core

in vec2 vTexCoord;
out vec4 fragColor;

uniform sampler2D uComplexIntermediateTex;
uniform vec2 uPixelSize;
uniform int uRadius;

// Complex kernel weights per sample (real, imag)
uniform vec2 uKernelWeightsX[64];
uniform vec2 uKernelWeightsY[64];

vec2 complexMul(vec2 a, vec2 b) {
    return vec2(a.x * b.x - a.y * b.y, a.x * b.y + a.y * b.x);
}

void main() {
    vec2 finalComponent1 = vec2(0.0);
    vec2 finalComponent2 = vec2(0.0);

    for (int y = -uRadius; y <= uRadius; ++y) {
        vec2 uv = vTexCoord + vec2(0.0, float(y)) * uPixelSize;
        vec4 sampledComplex = texture(uComplexIntermediateTex, uv);

        vec2 k1 = uKernelWeightsX[y + uRadius];
        vec2 k2 = uKernelWeightsY[y + uRadius];

        // Complex multiply-accumulate for both components
        finalComponent1 += complexMul(sampledComplex.xy, k1);
        finalComponent2 += complexMul(sampledComplex.zw, k2);
    }

    // Combine the real parts of both approximations
    float result = finalComponent1.x + finalComponent2.x;
    fragColor = vec4(vec3(result), 1.0);
}

Handling Depth of Field and Intensity Bleeding

Applying polygonal convolution naively leads to optical artifacts such as foreground color leaking into sharp backgrounds. Robust GLSL implementations solve this with:

Coc-Driven Sampling: Using depth differences to clamp or discard samples where background pixels attempt to blur over in-focus foreground pixels.
HDR Thresholding: Pre-weighting pixel intensities using an exponential curve or luminance mask prior to convolution. Polygonal aperture corners are only perceptually visible in high-intensity highlights; isolating and emphasizing high-dynamic-range points produces distinct polygonal highlights without over-blurring midtones.