Why Do Supercomputers Use Dragonfly or Torus Networks?

Supercomputers rely on high-radix Dragonfly and multidimensional Torus network topologies instead of standard Ethernet trees because they must move massive amounts of data between tens of thousands of processors with minimal latency and maximum bisection bandwidth. Standard tree networks suffer from severe bottlenecks at their upper tiers, high cabling costs, and excessive power consumption when scaled to exascale sizes. By contrast, Dragonfly and Torus topologies distribute traffic evenly across direct peer-to-peer links or low-diameter hierarchical groups, keeping communication fast, predictable, and cost-effective for tightly coupled scientific simulations.

The Bottleneck of Standard Ethernet Trees

In a traditional data center, servers are often connected in a hierarchical tree or fat-tree topology. At the bottom, compute nodes connect to top-of-rack switches, which connect to aggregation switches, which in turn connect to core switches at the root of the tree.

While this works well for independent web requests or cloud workloads, it breaks down in classical high-performance computing (HPC). Supercomputer applications—such as fluid dynamics, climate modeling, and molecular dynamics—require frequent all-to-all communication and global synchronization barriers. Every node constantly exchanges boundary data with many other nodes.

In a standard tree, traffic moving between different racks must travel up to the core switches and back down. Unless the network is built with full bisection bandwidth at every level, the upper tiers become severely oversubscribed. Achieving full bisection bandwidth in a fat-tree for \(N\) nodes requires \(O(N \log N)\) switches and cables, which becomes prohibitively expensive and physically unwieldy when \(N\) reaches tens of thousands of nodes.

Multidimensional Torus Topologies

A multidimensional Torus topology connects compute nodes directly to their nearest neighbors in a \(k\)-dimensional grid (such as 3D, 5D, or 6D), with wrap-around links connecting the edges of the grid back to the opposite side. Each node acts as both a compute element and a router, eliminating the need for large external core switches.

Torus networks excel at workloads with regular, nearest-neighbor communication patterns, such as finite-element meshes or stencil computations. Because nodes are wired directly to their neighbors, local traffic never traverses a central switch.

Key advantages of a Torus include:

• Predictable Latency: The maximum number of hops between any two nodes (the network diameter) in a \(k\)-ary \(n\)-cube scales as \(O(n \cdot k)\), where \(k\) is the number of nodes along each dimension.
• Cost Efficiency: There are no expensive, power-hungry core switches. Every link is short and uniform.
• Fault Tolerance: Multiple independent paths exist between any two nodes, allowing traffic to route around failed links.

However, as supercomputers grow larger, the diameter of a Torus increases, meaning global all-to-all communication must traverse many intermediate hops, which adds latency and consumes bandwidth on intermediate nodes.

High-Radix Dragonfly Topologies

To overcome the hop-count limitations of Torus networks while avoiding the cost of fat-trees, modern supercomputers frequently use the high-radix Dragonfly topology.

A Dragonfly network is built using high-radix routers—switches with a large number of ports (e.g., 64 ports or more). Routers are organized into densely connected local groups (often an all-to-all electrical connection within a cabinet), and each group is connected to every other group via at least one direct global optical link.

This design achieves a remarkably low network diameter. Any two nodes in the entire supercomputer can communicate in at most three router hops:

1. One hop within the source group.
2. One global hop across an optical cable to the destination group.
3. One hop within the destination group.

Because global optical cables are the most expensive part of a supercomputer interconnect, Dragonfly minimizes their count by concentrating traffic onto fewer, longer-distance links while using cheap, short electrical cables inside each group. To prevent congestion on those global links, Dragonfly networks use adaptive routing (such as Valiant's algorithm) to dynamically spread traffic across indirect paths when direct global links are busy.

Summary Comparison

Ultimately, classical supercomputers abandon standard Ethernet trees because they cannot afford the bisection bottlenecks or the \(O(N \log N)\) switching overhead at extreme scale. Torus networks solve this for localized physics simulations through cheap, direct neighbor links, while high-radix Dragonfly networks provide low-diameter, all-to-all connectivity for diverse, large-scale parallel workloads.