InfiniBand Architecture First Principles: Physical Link Rates, Credit-Based Link Flow Control, and Subnet Manager Fabric Orchestration

InfiniBand architecture represents the foundational principles of physical link rates, credit-based link flow control, and subnet manager fabric orchestration in supercomputing. Unlike Ethernet, which retrofits for RDMA with reactive PFC pause frames and DCQCN congestion tuning, InfiniBand is a purpose-built network designed for high-performance computing and large-scale AI.

The layered architecture consists of Upper Layer Protocols (ULP) including MPI and NCCL, a Transport Layer handling Reliable Connection (RC) and Unreliable Datagram (UD) with Hardware Checksum & Retransmit (BTH), a Network Layer for intra-subnet 16-bit Local Identifier (LID) routing and inter-subnet GID routing, a Link Layer providing Credit-Based Hardware Flow Control, Virtual Lanes (VL), and Nanosecond Cut-Through Switching, and a Physical Layer encompassing SerDes, Serial Links, PAM4 High-Frequency Modulation, and OSFP & QSFP-DD Transceivers.

Physical link speeds have evolved from Single Data Rate (SDR) at 2.5 Gbps to the 2026 flagship XDR at 800 Gbps, increasing aggregate bidirectional network throughput to 1600 Gbps per host. The Link Layer's credit-based hardware flow control model grants buffer credits to the receiver, allowing the sender to transmit only when the receiver has available buffer space, thereby preventing packet drops and eliminating deadlock hazards.

Cut-Through switching in InfiniBand ensures sub-100 nanosecond forwarding, a stark contrast to traditional Ethernet's Store-and-Forward mode, which introduces microseconds of latency by buffering entire packets and verifying frame CRC before forwarding. InfiniBand's packet header includes a Local Route Header (LRH) with a single 16-bit destination LID, enabling rapid forwarding decisions.