Multi-Resolution Texture Streaming for glTF

This article outlines the architecture required to stream multi-resolution textures for high-fidelity glTF assets, mitigating memory bottlenecks and high initial load times. It details the end-to-end pipeline, spanning progressive asset authoring with modern container formats, client-side Sparse Virtual Texturing (SVT), GPU-driven visibility feedback, and asynchronous network fetching. By implementing this decoupled architecture, real-time 3D applications can render massive textures progressively with minimal GPU memory consumption.

The Texture Streaming Challenge in glTF

Standard glTF 2.0 assets typically bundle monolithic texture images (JPEG, PNG, or .ktx2 via KHR_texture_basisu). For high-fidelity models using 4K or 8K textures, downloading and uploading full texture payloads to GPU memory before rendering causes significant latency, memory pressure, and frame drops. Solving this requires streaming only the specific mip levels or spatial regions (tiles) needed for the viewer's current camera distance and frustum.

Core Architectural Layers

Enabling multi-resolution texture streaming requires a four-part system: asset preparation, streaming infrastructure, GPU cache management, and visibility determination.

1. Texture Packaging and Formatting

The asset pipeline must structure textures as tiled mip-pyramids rather than contiguous image blocks.

  • KTX 2.0 Containers: The KTX 2.0 file format organizes image data into explicit mip levels. When combined with Basis Universal compression (UASTC for high fidelity or ETC1S for maximum compression), individual mips can be transcoded directly to GPU-native formats (BCn, ASTC).
  • Tiled Pyramids: Textures are subdivided into fixed-size spatial tiles (e.g., 128×128 or 256×256 pixels) across all mip levels. This enables byte-range retrieval of partial textures rather than entire mip maps.
  • glTF Manifest Extensions: glTF assets reference external streaming manifests (or custom vendor extensions like MSFT_texture_dds adapted for streaming, or custom SVT metadata) containing the byte offsets, tile dimensions, and mip hierarchies.

2. GPU Feedback and Visibility Analysis

The rendering engine must dynamically detect which texture resolutions are actively visible on screen.

  • Feedback Buffer Pass: The engine renders the scene to a low-resolution render target (e.g., 1/8th or 1/16th of screen resolution). Instead of computing full lighting, the fragment shader writes the texture ID, UV coordinates, and screen-space derivative (mip level) required for that fragment.
  • Readback and Aggregation: A compute shader (or CPU readback) parses this feedback buffer to produce a minimal, deduplicated list of missing tiles required for the active frame.

3. Client-Side Cache Management

The runtime manages a fixed-size texture cache on the GPU, decoupling scene fidelity from physical VRAM limits.

  • Physical Tile Pool: A large pre-allocated 2D texture array on the GPU acts as a cache for active tiles.
  • Indirection Table (Page Table): A two-dimensional index texture maps the original texture coordinate space to the physical slot in the tile cache. When a fragment shader samples the material, it uses the indirection table to locate the valid tile.
  • Fallback Resolution: If a high-resolution tile is not yet resident in memory, the indirection table points to a coarser parent mip level that is already loaded, preventing visual holes during streaming.

4. Asynchronous Network Streaming

A lightweight transport layer fetches missing assets over the network without blocking the main render loop.

  • HTTP Range Requests: The client uses standard HTTP/2 or HTTP/3 range requests against the hosted KTX2 or tiled texture files on a CDN to request individual byte regions for missing tiles.
  • Priority Queue: Requests are prioritized based on proximity to the camera, object importance, and screen coverage.
  • Asynchronous Upload: Downloaded tiles are decoded on worker threads and uploaded to the GPU tile pool using asynchronous staging buffers before updating the indirection table.

Operational Flow

  1. The glTF loader parses the asset, loading only the lowest-resolution fallback mips and geometry directly into memory.
  2. The scene renders a downscaled feedback pass to identify missing high-resolution tiles within the current view.
  3. The cache manager queues fetch requests for absent tiles via HTTP Range requests.
  4. Downloaded data streams directly into the physical tile pool on the GPU.
  5. The indirection table updates, seamlessly transitioning the displayed surface from low-resolution fallback to high-resolution detail.