How DirectML Maps ONNX Models to Hardware Primitives

This article explains how Microsoft DirectML bridges the gap between high-level Open Neural Network Exchange (ONNX) models and physical silicon. By functioning as a specialized execution provider within the ONNX Runtime, DirectML translates abstract neural network graphs into hardware-agnostic intermediate representations, applies graph-level optimizations, and ultimately compiles them into low-level DirectX 12 compute commands and vendor-specific driver optimizations across diverse GPUs and NPUs.

The ONNX Runtime Execution Provider Layer

The mapping pipeline begins when an ONNX model is loaded into the ONNX Runtime. The runtime parses the model's directed acyclic graph (DAG), which consists of standardized ONNX operators such as Conv, MatMul, and Softmax.

Rather than executing these operators directly on the CPU, the ONNX Runtime queries its registered Execution Providers (EPs). The DirectML Execution Provider identifies the subgraphs containing operators it supports and partitions them away from unsupported operations (which fall back to the default CPU execution provider). For each supported node, the DirectML EP extracts metadata, including tensor shapes, data types (such as FP32, FP16, or INT8), padding, strides, and dimension semantics.

Graph Lowering to DirectML Operator Descriptors

Once the subgraphs are partitioned, DirectML lowers each ONNX operator into its own internal representation using the DirectML API. Each ONNX node maps to a corresponding DirectML operator description structure (prefixed as DML_OPERATOR_DESC), such as DML_CONVOLUTION_OPERATOR_DESC or DML_GEMM_OPERATOR_DESC.

During this stage, DirectML standardizes tensor layouts. ONNX predominantly uses NCHW or NHWC memory formats. DirectML normalizes these input and output descriptions into native 4D or 5D strided tensor representations (DML_BUFFER_TENSOR_DESC), ensuring that memory strides, memory alignment requirements, and broadcast flags match what underlying hardware drivers expect.

Operator Fusion and Internal Graph Optimization

DirectML does not necessarily map high-level operators to hardware on a strict one-to-one basis. Before hardware scheduling, the DirectML internal graph compiler analyzes the operator chain to perform critical performance optimizations:

  • Operator Fusion: DirectML identifies sequences that can be computed in a single pass. A common pattern is fusing an activation function (like ReLU or GELU) and bias addition directly into a preceding Conv or Gemm operation. This reduces round-trip memory bandwidth to VRAM.
  • Layout and Transform Folding: Transpose, Reshape, or Squeeze operations are frequently folded into the tensor stride metadata of adjacent compute nodes, eliminating redundant copy passes.
  • Resource Reuse and Memory Planning: DirectML computes the precise memory footprint and lifetime of intermediate activation tensors, assigning temporary allocations to shared scratch buffers to minimize overall VRAM consumption.

Compilation to Direct3D 12 Compute Pipelines

DirectML is built directly on top of the Direct3D 12 (D3D12) API. Once the graph is optimized, DirectML translates the intermediate representations into native Direct3D 12 primitives through two distinct paths:

  1. Standard Compute Shaders (HLSL): For broad compatibility across all DirectX 12-capable hardware, DirectML maintains a library of highly optimized compute shaders written in High-Level Shader Language (HLSL). It compiles these shaders into Direct3D 12 Pipeline State Objects (PSOs) tailored to the exact tensor shapes and precision types defined by the model.
  2. Driver Meta-Commands: For maximum performance, modern GPU and NPU drivers (from vendors like Intel, NVIDIA, AMD, and Qualcomm) implement Direct3D 12 "Meta-Commands." Meta-commands allow the DirectML operator to bypass generic HLSL shaders and call proprietary, micro-architectural microcode baked into the hardware driver. This enables DirectML to directly leverage specialized silicon features, such as NVIDIA Tensor Cores, Intel XMX engines, AMD Matrix Cores, or dedicated NPU vector units.

Hardware Dispatch and Execution

At runtime, the compiled PSOs or meta-commands are bound to a DirectML Command Recorder. The command recorder serializes the operations onto a standard Direct3D 12 Command List.

When the application calls for inference evaluation, the command list is closed and submitted to the hardware's direct compute queue via the Direct3D 12 command queue. The GPU or NPU scheduler receives these dispatches as native GPU tasks, reading input buffers, executing hardware-accelerated matrix math across available compute units, and writing results to the designated output buffers without intermediate virtualization overhead.