Networking · InfiniBand

InfiniBand: the network built for clusters.

InfiniBand moves data between servers with very low latency and almost no work for the CPU. Here’s how it works, where it shines and what it costs.

Talk to an engineer See the pros and cons

What it is

A network designed for parallel computing.

InfiniBand is a high-speed network standard built for clusters, where many servers work on one job and constantly exchange data. Ethernet grew out of office networks; InfiniBand was designed from the start for low latency, high bandwidth and lossless delivery.

Its key idea is remote direct memory access (RDMA). The network adapter in one server writes data straight into another server’s memory, without involving either CPU or the operating system’s network stack. The processors and GPUs stay busy computing, and messages arrive quickly.

Current NVIDIA switches run at 400 Gb/s per port (NDR) and 800 Gb/s per port (XDR), and newer generations can run at a lower speed to connect older adapters.

How it works

Three ideas behind the fabric.

RDMA

Data skips the CPU.

Adapters read and write application memory directly over the network. The operating system stays out of the data path, so large transfers take very little CPU time.

Subnet manager

One manager sets up the fabric.

A subnet manager discovers every switch and adapter, assigns addresses and calculates routes. It runs on a switch or a server, usually with a standby ready to take over.

In-network computing

The switches do some of the math.

NVIDIA SHARP runs collective operations, such as adding up results from every node, inside the switches. Less data crosses the network, which speeds up MPI and AI training.

Advantages

Where InfiniBand shines.

The more often your nodes talk to each other, the more these matter.

Latency

Messages arrive fast.

RDMA and hardware transport keep latency very low. That matters most for tightly coupled MPI codes that trade small messages thousands of times a second.

Bandwidth

Up to 800 Gb/s per port.

NDR runs at 400 Gb/s and XDR at 800 Gb/s. A single 1U NDR switch has 64 ports and 51.2 Tb/s of throughput.

Lossless

No dropped packets.

Credit-based flow control means a sender transmits only when the receiver has room. The fabric stays lossless under load without special tuning.

Congestion

Built to handle busy traffic.

Adaptive routing, congestion control, quality of service and self-healing links are built into the switches, so throughput holds up when the whole cluster is busy.

GPUs and MPI

Supported everywhere it counts.

GPUDirect RDMA moves data between GPUs in different servers without a stop in system memory, and every major MPI library supports InfiniBand.

Scale

Grows from one switch to thousands of nodes.

Switches connect in fat-tree and other topologies, and the subnet manager handles routing automatically as the fabric grows.

Drawbacks

What it costs you.

None of these rule InfiniBand out, but they belong in the plan.

Price

It costs more than Ethernet.

Adapters, switches and especially optical cables cost more than comparable Ethernet. For small clusters running independent jobs, the difference may not pay off.

Supplier

One main vendor.

NVIDIA, which bought Mellanox, is effectively the only InfiniBand supplier. Pricing, lead times and roadmaps depend on one company.

Operations

A second network to run.

User access, management and your campus network still run on Ethernet, so the cluster has two networks to cable and maintain. InfiniBand tools are also new to many network teams.

InfiniBand or RoCE?

Why we usually recommend InfiniBand.

RoCE (RDMA over Converged Ethernet) brings RDMA to Ethernet, and it can work well. But Ethernet drops packets when it’s congested, so RoCE depends on priority flow control and congestion notification being configured correctly on every switch and adapter. When that setup drifts, performance suffers in ways that are hard to trace.

InfiniBand is lossless without that tuning, and its management is built in. For clusters where nodes work together on one job, it’s the more predictable choice, and it’s the one we recommend.

We design and deploy InfiniBand fabrics of every generation, from EDR and HDR through today’s NDR and XDR.

Sources: switch speeds and port counts from NVIDIA’s QM97xx NDR switch user manual and Q3200-RA/Q3400-RA XDR switch user manual; SHARP, self-healing and congestion control from the QM9700 product introduction.

Planning an InfiniBand fabric? Let’s design it together.

Talk to an engineer See our HPC clusters