Run FP16 and INT8 Models with DirectML in DirectX

Running low-precision inference—specifically FP16 (half-precision float) and INT8 (8-bit integer)—via DirectML allows DirectX developers to significantly boost inference throughput and reduce memory bandwidth usage on modern GPUs and NPUs. This guide details how to query hardware capabilities, configure low-precision tensors, utilize quantized DirectML operators, and execute the compiled operator graph using standard DirectX 12 command lists.

1. Check Hardware and Feature Support

Before allocating resources or compiling operators, verify that both the underlying DirectX 12 device and the DirectML device support native low-precision types.

To query native DirectML support for FP16 and INT8:

DML_FEATURE_QUERY_TENSOR_DATA_TYPE_SUPPORT fp16Query = { DML_TENSOR_DATA_TYPE_FLOAT16 };
DML_FEATURE_DATA_TENSOR_DATA_TYPE_SUPPORT fp16Support = {};
dmlDevice->CheckFeatureSupport(
    DML_FEATURE_TENSOR_DATA_TYPE_SUPPORT,
    sizeof(fp16Query),
    &fp16Query,
    sizeof(fp16Support),
    &fp16Support
);

if (!fp16Support.IsSupported) {
    // Fall back to FP32 or handle lack of native FP16 support
}

Repeat this check using DML_TENSOR_DATA_TYPE_INT8 and DML_TENSOR_DATA_TYPE_UINT8 to confirm support for quantized integer workloads.

2. Define Low-Precision Tensor Descriptions

DirectML represents model inputs, weights, and activations using DML_BUFFER_TENSOR_DESC. Specify the appropriate precision directly in the tensor definition:

  • For FP16: Set DataType to DML_TENSOR_DATA_TYPE_FLOAT16. Ensure your buffer sizes in bytes reflect that each element is 2 bytes rather than 4 bytes.
  • For INT8: Set DataType to DML_TENSOR_DATA_TYPE_INT8 or DML_TENSOR_DATA_TYPE_UINT8. Each element occupies 1 byte.
UINT dimensions[] = { 1, 64, 128, 128 };
UINT strides[]    = { 64 * 128 * 128, 128 * 128, 128, 1 };

DML_BUFFER_TENSOR_DESC bufferDesc = {};
bufferDesc.DataType = DML_TENSOR_DATA_TYPE_FLOAT16;
bufferDesc.Flags = DML_TENSOR_FLAG_NONE;
bufferDesc.DimensionCount = ARRAYSIZE(dimensions);
bufferDesc.Sizes = dimensions;
bufferDesc.Strides = strides;
bufferDesc.TotalTensorSizeInBytes = DMLCalcBufferTensorSize(
    bufferDesc.DataType,
    bufferDesc.DimensionCount,
    bufferDesc.Sizes,
    bufferDesc.Strides
);

DML_TENSOR_DESC tensorDesc = {};
tensorDesc.Type = DML_TENSOR_TYPE_BUFFER;
tensorDesc.Desc = &bufferDesc;

3. Implement Quantization and Operators

There are two primary ways to run low-precision graphs in DirectML:

Native FP16 Graphs

Construct standard DirectML operators (e.g., DML_OPERATOR_CONVOLUTION, DML_OPERATOR_GEMM) while supplying FP16 tensor descriptions for inputs, filters, and outputs. You can also supply the DML_EXECUTION_FLAG_ALLOW_HALF_PRECISION_COMPUTATION flag to IDMLDevice::CompileOperator when executing mixed-precision graphs where intermediate FP32 operations can safely run in FP16 to maximize ALU throughput.

Quantized INT8 Workloads

For INT8 inference, use dedicated quantized operators such as:

  • DML_OPERATOR_QUANTIZED_LINEAR_MATRIX_MULTIPLY
  • DML_OPERATOR_ELEMENT_WISE_DEQUANTIZE_LINEAR
  • DML_OPERATOR_ELEMENT_WISE_QUANTIZE_LINEAR

These operators accept integer input tensors alongside scale (floating-point) and zero-point tensors to accurately map 8-bit values back to continuous ranges.

4. Allocate DirectX 12 Buffers and Bind Resources

DirectML does not allocate device memory directly; it binds to standard ID3D12Resource buffers.

  1. Create Committed Resources: Allocate default heap buffers (D3D12_HEAP_TYPE_DEFAULT) using ID3D12Device::CreateCommittedResource sized via bufferDesc.TotalTensorSizeInBytes.
  2. Upload Weights: Convert your pre-trained weights to FP16 or quantized INT8 offline, then upload them via an upload heap to your default heap buffers.
  3. Create Binding Table: Instantiate an IDMLBindingTable matching the operator's binding requirements.
  4. Bind Resources: Wrap the D3D12 buffers in DML_BUFFER_BINDING structures and attach them to the binding table for inputs, outputs, and any required persistent or temporary resources.

5. Record and Execute on the Command List

Once configured, record the execution on a DirectX 12 command list:

IDMLCommandRecorder* recorder = ...;
ID3D12GraphicsCommandList* commandList = ...;
IDMLCompiledOperator* compiledOp = ...;
IDMLBindingTable* bindingTable = ...;

recorder->RecordDispatch(commandList, compiledOp, bindingTable);

Insert standard DirectML-to-Direct3D UAV barriers (D3D12_RESOURCE_BARRIER_TYPE_UAV) on the command list between dependent inference steps or before reading the outputs within graphics or compute passes.