Implementing TAA with Motion Vectors in DirectX

Temporal Anti-Aliasing (TAA) is an image reconstruction technique that eliminates edge jaggies and temporal shimmering by accumulating samples across multiple frames. In DirectX, this process relies on sub-pixel camera jittering, screen-space motion vectors (velocity buffers) to reproject previous samples into the current frame, and neighborhood color clamping to prevent ghosting artifacts. This article outlines the architectural pipeline and HLSL implementation details required to integrate motion-vector-based TAA into a Direct3D 11 or Direct3D 12 rendering engine.

1. Sub-Pixel Camera Jittering

TAA requires varying the sample locations within each pixel across successive frames. This is achieved by offsetting the projection matrix using a low-discrepancy sequence, typically an 8- or 16-phase Halton(2, 3) sequence.

In DirectX, modify the projection matrix's translation components:

// Generate jitter offsets in normalized device coordinates (NDC)
float jitterX = (HaltonSequence(frameIndex, 2) - 0.5f) * (2.0f / renderWidth);
float jitterY = (HaltonSequence(frameIndex, 3) - 0.5f) * (2.0f / renderHeight);

// Apply to projection matrix (DirectX row-major notation)
XMMATRIX jitteredProjection = originalProjection;
jitteredProjection.r[2] = XMVectorSetZ(jitteredProjection.r[2], jitteredProjection.r[2].m128_f32[2] + jitterX);
jitteredProjection.r[2] = XMVectorSetW(jitteredProjection.r[2], jitteredProjection.r[2].m128_f32[3] + jitterY);

The jittered projection matrix is used for primary rasterization, while the unjittered projection matrix must be preserved to generate accurate motion vectors.

2. Generating the Velocity Buffer

The velocity buffer encodes screen-space displacement from the previous frame to the current frame. This can be rendered during a dedicated G-Buffer pass or reconstructed from depth. For dynamic meshes, motion vectors must be generated explicitly in a vertex/pixel shader pass to account for object-level transformations.

DirectX render target configuration:

  • Format: DXGI_FORMAT_R16G16_FLOAT (2D screen-space displacement)
  • Vertex Inputs: Current model-view-projection matrix and previous model-view-projection matrix.

HLSL Vertex Shader:

struct VS_INPUT
{
    float3 position : POSITION;
};

struct PS_INPUT
{
    float4 currentPos   : SV_POSITION;
    float4 currentNDC   : TEXCOORD0;
    float4 previousNDC  : TEXCOORD1;
};

cbuffer TransformBuffer : register(b0)
{
    float4x4 g_CurrentWorldViewProjUnjittered;
    float4x4 g_PrevWorldViewProjUnjittered;
};

PS_INPUT VSMain(VS_INPUT input)
{
    PS_INPUT output;
    output.currentPos = mul(float4(input.position, 1.0f), g_CurrentWorldViewProjUnjittered);
    output.currentNDC = output.currentPos;
    output.previousNDC = mul(float4(input.position, 1.0f), g_PrevWorldViewProjUnjittered);
    return output;
}

HLSL Pixel Shader:

float2 PSMain(PS_INPUT input) : SV_TARGET
{
    // Convert to NDC (-1 to 1)
    float2 currentCoords = input.currentNDC.xy / input.currentNDC.w;
    float2 previousCoords = input.previousNDC.xy / input.previousNDC.w;

    // Convert from NDC to UV space [0, 1] (flip Y for DirectX)
    currentCoords = currentCoords * float2(0.5f, -0.5f) + 0.5f;
    previousCoords = previousCoords * float2(0.5f, -0.5f) + 0.5f;

    // Return UV delta
    return currentCoords - previousCoords;
}

3. History Sampling and Velocity Dilation

Before sampling the history texture, sample the velocity buffer using a \(3\times3\) pixel neighborhood to find the velocity of the pixel closest to the camera (the minimum depth). This process, known as velocity dilation, prevents foreground objects from smearing over static backgrounds.

float2 GetDilatedVelocity(Texture2D depthTex, Texture2D velocityTex, SamplerState pointSampler, float2 uv, float2 texelSize)
{
    float minDepth = 1.0f; // DirectX reverse-Z uses 0.0f as furthest
    float2 bestUV = uv;

    [unroll]
    for (int y = -1; y <= 1; ++y)
    {
        [unroll]
        for (int x = -1; x <= 1; ++x)
        {
            float2 offsetUV = uv + float2(x, y) * texelSize;
            float depth = depthTex.SampleLevel(pointSampler, offsetUV, 0).r;
            if (depth < minDepth)
            {
                minDepth = depth;
                bestUV = offsetUV;
            }
        }
    }
    return velocityTex.SampleLevel(pointSampler, bestUV, 0).xy;
}

4. History Rectification (Neighborhood Clamping)

When dynamic objects move or occlusions occur, the reprojected pixel in the history buffer may no longer match the current pixel, causing "ghosting." To fix this, clamp or clip the history sample to the bounding box of the current pixel’s \(3\times3\) neighborhood.

Transforming color to a perceptually uniform color space such as YCoCg produces better results than standard RGB:

float3 RGBToYCoCg(float3 c)
{
    return float3(
         c.r * 0.25f + c.g * 0.5f + c.b * 0.25f,
         c.r * 0.5f               - c.b * 0.5f,
        -c.r * 0.25f + c.g * 0.5f - c.b * 0.25f
    );
}

float3 YCoCgToRGB(float3 c)
{
    return float3(
        c.x + c.y - c.z,
        c.x + c.z,
        c.x - c.y - c.z
    );
}

Sample the \(3\times3\) neighborhood in the current frame to compute an Axis-Aligned Bounding Box (AABB):

float3 aabbMin = 9999.0f;
float3 aabbMax = -9999.0f;

[unroll]
for (int y = -1; y <= 1; ++y)
{
    [unroll]
    for (int x = -1; x <= 1; ++x)
    {
        float3 sampleColor = RGBToYCoCg(CurrentFrameTex.SampleLevel(LinearSampler, uv + float2(x, y) * texelSize, 0).rgb);
        aabbMin = min(aabbMin, sampleColor);
        aabbMax = max(aabbMax, sampleColor);
    }
}

Clamp the reprojected history color to this bounding box:

float2 historyUV = uv - dilatedVelocity;
float3 historyColor = RGBToYCoCg(HistoryTex.SampleLevel(LinearSampler, historyUV, 0).rgb);

// Clamp history to current frame's local neighborhood
historyColor = clamp(historyColor, aabbMin, aabbMax);

5. Temporal Accumulation (Resolve Pass)

The final resolve pass runs as a full-screen pixel shader or compute shader. Blend the current jittered frame with the clamped history color using an exponential moving average.

// Discard history if reprojected UV is out-of-bounds
bool offScreen = any(historyUV < 0.0f) || any(historyUV > 1.0f);

float alpha = 0.05f; // Weight for the current frame (e.g., 5% current, 95% history)
if (offScreen)
{
    alpha = 1.0f;
}

float3 currentColor = RGBToYCoCg(CurrentFrameTex.SampleLevel(LinearSampler, uv, 0).rgb);
float3 resolvedYCoCg = lerp(historyColor, currentColor, alpha);

float3 finalColor = YCoCgToRGB(resolvedYCoCg);

6. Resource Ping-Ponging in DirectX

To sustain the temporal feedback loop in Direct3D:

  1. Maintain two textures formatted as DXGI_FORMAT_R11G11B10_FLOAT or DXGI_FORMAT_R16G16B16A16_FLOAT: HistoryTextureA and HistoryTextureB.
  2. On frame \(N\), bind HistoryTextureA as a Shader Resource View (SRV) and render the resolved TAA output to HistoryTextureB as a Render Target View (RTV).
  3. For frame \(N+1\), swap the pointers so HistoryTextureB serves as the SRV input and HistoryTextureA serves as the RTV output.