High-speed interconnects are the links and protocols that move data between processors, accelerators, memory, storage and servers fast enough that none of them sit idle waiting. In an HPC or AI cluster, the interconnect often decides whether the money spent on GPUs turns into real throughput. This guide explains how the main options work, how an InfiniBand network is laid out, and what to settle before you buy hardware.
TL;DR
- High-speed interconnects give you high bandwidth and low latency between CPUs, GPUs, servers and storage, so that data movement does not become the bottleneck.
- InfiniBand is a switched fabric of point-to-point links with RDMA, lossless flow control and direct application access to the network. Its transfer rates run from 10 to 400Gb/s today, with 800Gb/s per port specified for XDR.
- Fat-tree is the most widely used topology for InfiniBand clusters. Design it non-blocking when you need the best performance at scale, and accept oversubscription only where the workload tolerates it.
- Settle topology, cabling, firmware and fabric management before the hardware ships, not after it lands in the rack.
Table of Contents
- What high-speed interconnects are and why clusters depend on them
- How InfiniBand architecture connects servers, accelerators and storage
- InfiniBand topologies: why fat-tree dominates
- Bandwidth, latency and oversubscription: what the numbers mean
- InfiniBand or Ethernet for HPC and AI
- What to check before deployment
- Our perspective
- Plan your interconnect with Dormedia
- FAQ
- Sources
What high-speed interconnects are and why clusters depend on them
An interconnect is the path data takes between components. IEEE Toronto describes high-speed interconnects as physical links and protocols that move data rapidly between chips, devices or systems, often at hundreds of gigabytes per second with low latency. They connect CPUs to accelerators, GPUs to GPUs, and servers to storage.
The reason this matters for AI and HPC is simple. Training a large model or running a tightly coupled simulation means thousands of processing elements exchange intermediate results constantly. If the network cannot keep up, expensive accelerators wait. The more nodes you add, the more the network, rather than the chips, sets the pace.
Interconnects exist at several scales, and it helps to keep them apart:
- On the board or package: links between a CPU and its memory, or between chiplets.
- Within a server: links between CPUs, GPUs and local storage.
- Between servers: the cluster fabric, which is where InfiniBand and high-speed Ethernet compete and where most design decisions for a buyer sit.
The rest of this guide focuses on that third level, the fabric that ties nodes into one system.
How InfiniBand architecture connects servers, accelerators and storage
InfiniBand is an open industry-standard specification. IBM's networking design guide notes it is independent of the host operating system and hardware platform, and lists low latency, high bandwidth, quality of service, CPU offloading, RDMA and lossless link-level flow control among its features. The InfiniBand Trade Association adds that one connection can carry clustering, communications, storage and management traffic with minimal processing overhead.
The building blocks
An InfiniBand fabric is a switched network of serial point-to-point links. Servers attach through Host Channel Adapters (HCAs), and the HCAs connect to InfiniBand switches. Storage can attach to the same fabric, so an application can reach other applications and storage over one network.
Why applications see less overhead
In a conventional network, data passes through the operating system's TCP/IP stack. NVIDIA's introduction to InfiniBand explains that InfiniBand starts from an application-centric view: the application gets direct access to a messaging service, bypassing the OS-owned stack.
The mechanism is the Queue Pair (QP). Each QP is the endpoint of a channel and consists of a send queue and a receive queue. Applications use SEND/RECEIVE operations, plus RDMA READ and WRITE, which let one machine read or write another machine's memory without involving the remote operating system in the data path. The IBM guide describes the same model, with channels created between otherwise separate virtual address spaces.
In practice this is why InfiniBand suits MPI jobs and distributed training: small messages cross the fabric quickly, and the CPUs are free to compute instead of shuffling packets.
InfiniBand topologies: why fat-tree dominates
You can wire a fabric in several ways, but one pattern appears in most cluster designs. NVIDIA's cluster design guidance identifies fat-tree as the most widely used InfiniBand topology in HPC clusters. When it is configured non-blocking, it typically gives the best performance at large scale.
In a fat-tree:
- Servers connect to leaf switches on the lowest level.
- Spine switches on the upper levels connect the leaves to each other.
- Any server can reach any other through a leaf, a spine and a leaf again.
The IBM guide describes the same layered structure. A university HPC facility explains why operators keep choosing it: ULHPC's documentation cites versatility, high bisection bandwidth and well-understood routing, and says each of its production clusters uses fat-tree.
Non-blocking versus blocking
A non-blocking fat-tree has as much uplink capacity toward the spine as downlink capacity toward the servers, so every node can send at full rate at the same time. NVIDIA's guidance says a non-blocking cluster must be balanced, with the same number of links from each upper-level switch to every lower-level switch. A blocking design, also called oversubscribed, uses fewer uplinks than downlinks. It costs less in switches and cables and is acceptable when the workload does not saturate the network at all nodes simultaneously.
A quick way to think about the choice:
| Design | What it means | Best fit |
|---|---|---|
| Non-blocking (1:1) | Uplink capacity equals downlink capacity | Large-scale AI training, tightly coupled simulation, workloads where every node communicates at once |
| Blocking (oversubscribed) | Fewer uplinks than downlinks | Mixed or throughput workloads with modest cross-node traffic, where cost matters more than peak all-to-all speed |
A routing rule that avoids deadlock
NVIDIA also warns against routes that go up, back down and then up again through the tree. Those paths can create credit loops and traffic deadlocks. This is one reason fabric design is more than adding up port counts: the routing and the topology have to be planned together.
Bandwidth, latency and oversubscription: what the numbers mean
Three properties decide how an interconnect behaves under load.
Bandwidth is how much data a link can move per second. The InfiniBand Trade Association reports data rates from 10 to 400Gb/s across the technology's generations. Its XDR specification announcement describes twice the bandwidth of the previous generation, with 200Gb/s per lane and 800Gb/s per port, and 1.6Tb/s switch-to-switch connections.
Latency is the delay for a message to arrive. The IBTA reports measured end-to-end delays of 600ns. For jobs that exchange many small messages, such as MPI collectives, latency matters as much as raw bandwidth.
Oversubscription is the ratio of server-facing capacity to uplink capacity. At 1:1 the fabric is non-blocking. At 3:1, three servers' worth of traffic shares one uplink's capacity when everyone talks at once. The cost saving is real, and so is the performance risk if the workload turns out to be communication-heavy.
There is also congestion, which appears when traffic converges on the same links. Lossless link-level flow control keeps packets from being dropped under pressure, but a poorly balanced fabric still loses performance to queuing. Balanced links and sound routing are the first defense.
A practical sizing sequence:
- Profile the workload: how much data does each node exchange per step, and how often?
- Decide the number of nodes now and the number you expect to add over the next two to three years.
- Pick the oversubscription ratio the workload tolerates, and 1:1 if you are unsure and the budget allows.
- Choose switch radix and generation so the leaf-spine layout can grow without a rebuild.
- Check that the cable lengths and types fit the physical room.
InfiniBand or Ethernet for HPC and AI
Both are used in modern clusters, and the right answer depends on the workload, the existing staff skills and the budget. The research available points in a clear direction on pure HPC performance. ULHPC states that Ethernet's underlying protocol has inherent limits that prevent the low latency expected in real HPC environments, and considers InfiniBand the better option when high bandwidth and low latency are needed.
NVIDIA's design article puts numbers on the gap for cluster efficiency: compared with InfiniBand, 1GbE cuts cluster performance by 50% and 10GbE by 30%, while InfiniBand yields about 90% system efficiency. Those figures describe older Ethernet speeds, so treat them as an illustration of how much the network can cost a cluster, not as a comparison with the latest Ethernet generations.
Adoption data supports the picture at the top end. The IBTA notes that InfiniBand connects 63 of the top 100 systems on the June 2023 TOP500 list, with 200 InfiniBand-connected systems overall.
When choosing, ask:
- Is the workload tightly coupled (large training runs, MPI simulation) or mostly independent jobs?
- Does the cluster need to grow past a few hundred nodes?
- Will your team operate a dedicated fabric, with its own subnet management?
- Does the facility already have an Ethernet fabric you must integrate with?
Tightly coupled work at scale leans to InfiniBand. Loosely coupled work and mixed enterprise environments may be served by a well-designed Ethernet fabric. A consultant who has built both can tell you which fits your situation, which is more useful than a blanket rule.
What to check before deployment
Most interconnect problems are found during bring-up, and most could have been prevented in planning. NVIDIA's InfiniBand cluster bring-up procedure is a good outline of the order of work: choose the desired topology, create and save a point-to-point (PTP) file, generate a topology file, check component firmware and software versions, validate cabling, install the Unified Fabric Manager (UFM) and confirm the topology with it.
Use it as a checklist.
| Area | What to confirm |
|---|---|
| Topology | Fat-tree levels, ratio of uplinks to downlinks, balanced links to every switch, growth path |
| Hardware | HCAs and switches from the same generation, port speeds that match the cables |
| Cabling | Correct cable type and length, no mismatched connectors, bend radius respected |
| Firmware and software | Versions of every component recorded and consistent, drivers matched to the operating system |
| Management | Subnet manager and fabric management planned, with monitoring and alert thresholds |
| Verification | Cluster topology compared against the design file before production jobs run |
Two details are worth planning early. First, fabric management software has its own prerequisites. NVIDIA's UFM documentation describes required driver stacks and server sizing, so the management node needs to be specified alongside the compute nodes. Second, physical cabling is where schedules slip: the wrong cable length or an unsupported transceiver stops an entire row of racks.
Once the fabric powers on, the subnet manager discovers the links. Operators then check link health with diagnostic tools and watch error counters over time so that a marginal cable is replaced before it degrades a long job.
Procurement is part of the design
A fabric design is only as good as the parts that arrive. Lead times for switches, optics and cables vary, and a missing component can hold a whole cluster. That is why sourcing, shipping and import compliance belong in the same plan as the topology. For multi-country deployments, tariffs and customs treatment of networking equipment also affect the delivered cost.
Our perspective
We see the same pattern across HPC and AI projects: teams spend months selecting accelerators and then treat the network as an afterthought. The interconnect is where a cluster's design either holds together or comes apart at scale.
Our view is that the fabric should be designed with the workload, the budget and the delivery plan together. Topology, oversubscription ratio, cabling and fabric management are one decision, and they have to be matched to the servers, the data center space and the timeline in which the parts can actually arrive.
Since 2011 we have worked in infrastructure, and since 2017 deeply in HPC and AI. Through our network of vendors and logistics partners we aim to bring design, sourcing, global shipping, tax and tariff handling and ongoing support under one project, so that the network you designed is the network you receive and can run.
Dormedia
Plan your interconnect with Dormedia
If you are designing a cluster, expanding one or sourcing hardware across borders, we can help with the whole path: design, component sourcing, data center solutions, global shipping, tax and tariff solutions, project management, strategic advisory and ongoing support. Our services cover customized server and component configurations built around your workload. See the full range on the Dormedia homepage or browse available hardware inventory.
For a closer look at how a consulting engagement is scoped, read our guide to high performance computing consulting.
FAQ
What is the difference between bandwidth and latency in an interconnect?
Bandwidth is how much data a link carries per second, and latency is how long a single message takes to arrive. Large data transfers depend mostly on bandwidth, while many small messages, as in MPI jobs, depend heavily on latency. Most HPC and AI clusters need both.
Why is fat-tree the usual topology for InfiniBand clusters?
It offers high bisection bandwidth, well-understood routing and flexibility across cluster sizes, according to the ULHPC documentation. Configured as non-blocking, it typically gives the best performance at large scale. It can also be built oversubscribed where cost matters more.
Do I need a non-blocking network?
If many nodes communicate at the same moment, as in large training runs or tightly coupled simulations, a non-blocking design avoids contention. If your jobs are mostly independent and move little data between nodes, a modestly oversubscribed fabric can cut cost without hurting results. Profile the workload first.
What does RDMA do?
RDMA lets one server read or write the memory of another through the network adapter, without the remote operating system handling each transfer. This lowers latency and frees CPU cycles, and it is a core part of the InfiniBand messaging model.
What should I settle before ordering switches and cables?
Fix the topology and oversubscription ratio, the port speed, the cable types and lengths, the firmware baseline and the fabric management plan. Then confirm delivery lead times for every part so the build is not held up by a single missing component.
Sources
- High-Speed Interconnects: The AI Backbone, IEEE Toronto
- Networking Design for HPC and AI on IBM Power Systems
- InfiniBand: a low-latency, high-bandwidth interconnect, IBTA
- Introduction to InfiniBand, NVIDIA
- Designing an HPC Cluster with InfiniBand Solutions, NVIDIA
- Fast InfiniBand Interconnect, ULHPC Technical Documentation
- IBTA Unveils XDR InfiniBand Specification
- InfiniBand Cluster Bring-up Procedure, NVIDIA