Tracing DirectX Crashes with Breadcrumbs

Diagnosing GPU hangs and Device Removed errors (DXGI_ERROR_DEVICE_REMOVED) in DirectX requires visibility into what the hardware was processing right before the failure. Breadcrumb markers are lightweight checkpoints written to GPU-visible memory immediately before and after command execution. By reading these values post-crash, developers can identify the precise draw call, dispatch, or state transition that caused the driver timeout or page fault.

Device Removed Extended Data (DRED) Auto-Breadcrumbs

The primary native mechanism for breadcrumb tracing in modern DirectX 12 is Microsoft’s Device Removed Extended Data (DRED). DRED provides automated breadcrumb tracking without requiring manual insertion of write commands around every render pass.

When enabled via the ID3D12DeviceRemovedExtendedDataSettings interface prior to device creation, DRED injects internal markers before and after every draw, dispatch, and copy command.

CComPtr<ID3D12DeviceRemovedExtendedDataSettings> pDredSettings;
if (SUCCEEDED(D3D12GetDebugInterface(IID_PPV_ARGS(&pDredSettings))))
{
    pDredSettings->SetAutoBreadcrumbsEnablement(D3D12_DRED_ENABLEMENT_FORCED_ON);
    pDredSettings->SetBreadcrumbContextEnablement(D3D12_DRED_ENABLEMENT_FORCED_ON);
}

Upon a crash, querying ID3D12Device5::GetDeviceRemovedReason and accessing ID3D12DeviceRemovedExtendedData::GetAutoBreadcrumbsOutput exposes an execution history list (D3D12_AUTO_BREADCRUMB_NODE). This list tracks:

  • The command list where the crash occurred.
  • The total number of executed breadcrumbs versus scheduled breadcrumbs.
  • The specific command (e.g., DRAWINDEXEDINSTANCED, DISPATCH) where the pipeline stalled.

Custom Markers via WriteBufferImmediate

For developers who require custom instrumentation or support for older DirectX versions, manual breadcrumb systems rely on writing marker values directly to GPU-accessible memory.

In DirectX 12, developers use ID3D12GraphicsCommandList2::WriteBufferImmediate. This function allows the GPU to write a 32-bit immediate value to a specified virtual address on the GPU timeline without stalling the pipeline:

D3D12_WRITEBUFFERIMMEDIATE_PARAMETER param = { gpuVirtualAddress, markerValue };
D3D12_WRITEBUFFERIMMEDIATE_MODE mode = D3D12_WRITEBUFFERIMMEDIATE_MODE_MARKER_IN;

pCommandList2->WriteBufferImmediate(1, &param, &mode);
pCommandList->DrawIndexedInstanced(...);

Using MARKER_IN registers the value when the command begins execution on the front-end, while MARKER_OUT writes when execution completes.

To safely retrieve this data after a crash:

  1. Allocate memory in a persistent upload or readback heap configured with appropriate cache properties.
  2. Maintain a ring buffer of marker IDs mapped to CPU-side call stacks or debug strings.
  3. Upon catching a device removal error, read back the last committed marker from CPU-mapped memory to locate the faulting call.

PIX Events and Context Strings

Numeric IDs can be difficult to map back to specific game systems. Developers can enrich breadcrumbs with context strings using the WinPixEventRuntime library.

Functions such as PIXSetMarker and PIXBeginEvent / PIXEndEvent place human-readable metadata into the command stream:

PIXBeginEvent(pCommandList, PIX_COLOR_DEFAULT, "PostProcessing::Bloom");
PIXSetMarker(pCommandList, PIX_COLOR_DEFAULT, "BlurPass_Horizontal");

When DRED’s breadcrumb context enablement is active, these PIX markers are included in the DRED crash log (D3D12_DRED_BREADCRUMB_CONTEXT). This associates raw numeric indices directly with developer-defined pass names, resource names, or object identifiers.

NVIDIA Nsight Aftermath SDK

For NVIDIA hardware, the Nsight Aftermath SDK provides dedicated crash analysis that augments standard DirectX mechanisms. Aftermath inserts proprietary, low-overhead GPU markers and captures a comprehensive GPU state dump upon failure.

Developers integrate Aftermath by initializing the SDK, setting event markers on command lists via GFSDK_Aftermath_SetEventMarker, and configuring a callback to write a crash dump (.nv-gpudmp) to disk. This dump file can be opened directly in NVIDIA Nsight Graphics to view the shader source line, draw command, and pipeline state active at the moment of the crash.