DirectX 12 Cross-Adapter Transfers via Copy Queues

This article provides an overview of how DirectX 12 manages cross-adapter resource transfers between linked GPUs using dedicated copy command queues. In multi-GPU configurations, transferring data such as rendered frames, shadow maps, or compute buffers requires coordinated memory placement, synchronization, and explicit hardware execution. By leveraging asynchronous copy queues and cross-adapter resources, DirectX 12 enables efficient data movement across PCIe or high-speed interconnects without stalling primary 3D and compute engines.

Linked Adapter Architecture in DirectX 12

In DirectX 12, linked GPUs are managed under a single ID3D12Device using the Linked Display Adapter (LDA) configuration. Each physical GPU represents an individual node indexed by a bitmask (NodeMask).

While both GPUs share the same device context, each GPU maintains its own distinct physical memory pools, command queues, and hardware engines. Cross-adapter transfers refer to moving data from the local VRAM of one GPU node to the local VRAM or shared system memory accessible by another GPU node.

The Role of Asynchronous Copy Queues

Modern graphics hardware includes dedicated DMA (Direct Memory Access) copy engines that operate independently of the graphics and compute pipelines. DirectX 12 exposes these engines via D3D12_COMMAND_LIST_TYPE_COPY.

Using a copy queue for cross-adapter transfers provides critical advantages:

  • Asynchronous Execution: Copy operations execute concurrently alongside rendering workloads on the 3D queue, preventing pipeline bubbles.
  • Direct Bus Utilization: The DMA engine directly drives the PCIe or interconnect bus transfers, freeing the graphics pipeline from managing memory transfers.
  • Minimized Latency Impact: Synchronization points can be isolated to the transfer boundaries, allowing the source GPU to proceed with subsequent frames while the transfer takes place.

Resource Requirements for Cross-Adapter Sharing

Standard texture resources use vendor-specific, proprietary tiling configurations that are incompatible across different physical GPUs or memory controllers. To move data between nodes, resources must be created with explicit cross-adapter flags:

  1. Cross-Adapter Flag: The resource must specify D3D12_RESOURCE_FLAG_ALLOW_CROSS_ADAPTER.
  2. Standardized Layout: Textures must use linear row-major layout (D3D12_TEXTURE_LAYOUT_ROW_MAJOR) rather than vendor-specific swizzling.
  3. Heap Properties: Cross-adapter resources are allocated within a shared cross-adapter heap or placed in system memory (D3D12_HEAP_TYPE_DEFAULT with cross-adapter capabilities or D3D12_HEAP_TYPE_CUSTOM).

Because row-major layouts are inefficient for real-time sampling and rendering, applications typically do not render directly into cross-adapter resources. Instead, rendering occurs in a native, swizzled local texture, which is then copied into the intermediate cross-adapter buffer.

The Transfer Process

A standard cross-adapter transfer between Node 0 (Source) and Node 1 (Destination) follows a structured sequence:

  1. Local Render: Node 0 renders data into a high-performance, locally tiled resource on its direct 3D queue.
  2. Local-to-Shared Copy: On Node 0's copy queue, a command resolves or copies the tiled texture into the linearly formatted cross-adapter intermediate resource.
  3. Cross-Adapter Fence Signal: Node 0 signals a cross-adapter fence on its copy queue upon completion of the write operation.
  4. Queue Wait: Node 1’s copy queue waits for the cross-adapter fence to reach the signaled value before reading from the shared resource.
  5. Shared-to-Local Copy: Node 1 copies the data from the cross-adapter intermediate resource into its own local, tiled VRAM for subsequent use in shaders or display presentation.

Cross-Adapter Synchronization

Data integrity is maintained using cross-adapter fences (ID3D12Fence created with D3D12_FENCE_FLAG_SHARED and D3D12_FENCE_FLAG_SHARED_CROSS_ADAPTER).

Unlike standard fences that coordinate operations within a single GPU, cross-adapter fences are visible across PCIe boundaries. When Node 0 issues a Signal on its copy queue, the fence value updates globally across the linked device, unblocking Node 1's Wait command without requiring host CPU intervention.

To eliminate latency bottlenecks, applications implement multi-buffering (typically double or triple buffering) of the intermediate cross-adapter resources, allowing Node 0 to write to buffer N+1 while Node 1 reads from buffer N.