InfiniBand Networking: Exploring Features, Components, and Benefits
Oct 18, 20231 min read
InfiniBand is an open standard designed for high-performance connectivity between CPU/GPU servers, storage systems, and other devices. This article explores the fundamental aspects of InfiniBand networking, including its core features, structural components, and advantages over traditional Ethernet solutions.
What is InfiniBand?
InfiniBand is a high-speed, open industrial standard for interconnecting servers, storage devices, and computing systems. It utilizes point-to-point bidirectional links to enable seamless communication across multiple processors and servers. With its ultra-low latency and exceptional bandwidth, InfiniBand is the preferred choice for high-performance computing (HPC) environments.
How does InfiniBand work?
InfiniBand operates as a high-speed, channel-based fabric that enables efficient communication between interconnected nodes. A typical InfiniBand network consists of processor nodes—including servers, storage systems, and other computing devices—along with network infrastructure components such as switches, routers, cables, and connectors.
Each processor node within an InfiniBand network is equipped with a host channel adapter, while peripheral devices feature target channel adapters. InfiniBand follows a channel-based communication model, ensuring that all transmissions are initiated or received through these adapters. These adapters also play a key role in enforcing security protocols and supporting Quality of Service (QoS) levels to optimize data flow.
InfiniBand networks transmit data in the form of packets, which are structured into messages. This enables InfiniBand to excel in high-performance applications such as deep learning, big data analytics, and scientific computing, meeting the demands for ultra-low latency and high throughput.
Structural Components of an InfiniBand Network
An InfiniBand network consists of the following elements:
Host Channel Adapter (HCA)
The HCA, also known as a network card, serves as an end node connected to the InfiniBand network. It facilitates transport layer functions and supports the verbs interface, which provides a programming interface for InfiniBand devices.
InfiniBand to Ethernet Gateway/Bridge
This device allows for the conversion of InfiniBand and Ethernet messages, enabling communication between the InfiniBand network and Ethernet network when necessary.
InfiniBand Switch
An InfiniBand switch forwards messages between InfiniBand networks, enabling efficient data transmission across connected devices.
Subnet Manager (SM)
The Subnet Manager is responsible for managing the InfiniBand subnet. It can run on hosts, switches, or be deployed alongside Unified Fabric Manager (UFM) for comprehensive management.
InfiniBand Router
An InfiniBand router facilitates the transmission of messages between different InfiniBand subnets.
Core Features of InfiniBand
Subnet Manager (SM): The Subnet Manager program ensures centralized routing management, enabling plug-and-play functionality across all nodes in the network. Each subnet requires a master SM, with other SMs operating in standby mode.
GPU Direct: GPU Direct allows direct data transfers between GPUs, reducing latency and enhancing performance, particularly in GPU-based computing. NVIDIA® GPUs also support compute task offloading.
Low Latency: InfiniBand achieves extremely low latency through hardware offloading and acceleration mechanisms. Cut-through forwarding mode in InfiniBand switches reduces transmission latencies to as low as 130ns. RDMA technology further reduces end-to-end transport latency.
Network Scalability: InfiniBand enables the interconnection of multiple subnets using InfiniBand routers, facilitating easy scalability to accommodate over 48,000 nodes.
Fault-Tolerant Stable Network: InfiniBand networks ensure rapid traffic recovery, thanks to the subnet manager's routing algorithm and efficient flow reordering. This results in quick traffic restoration.
Self-Healing Network: NVIDIA® InfiniBand switches feature a hardware-based self-healing mechanism, enabling fast recovery in just one millisecond.
Adaptive Routing: Adaptive routing balances traffic distribution across switch ports. NVIDIA® InfiniBand switches incorporate this feature in their hardware and manage it through the Adaptive Routing Manager.
SHARP (Scalable Hierarchical Aggregation and Reduction Protocol): SHARP, based on NVIDIA InfiniBand switch hardware and central management packets, optimizes collective communication and reduces data transfers between nodes during MPI-based applications like machine learning.
Advantages of InfiniBand vs. Ethernet
InfiniBand offers several advantages compared to traditional Ethernet networking solutions, making it the ideal choice for high-performance applications requiring fast data transfer and minimal latency. The benefits of InfiniBand include:
High Bandwidth and Low Latency: InfiniBand delivers significantly higher bandwidth and lower latency than Ethernet, ensuring optimal performance for real-time applications and large-scale data transfers.
RDMA Support: InfiniBand supports Remote Direct Memory Access (RDMA), enabling direct data transfer between node memories. This reduces CPU overhead and improves transfer efficiency.
Scalability: InfiniBand Fabric provides seamless scalability, supporting a large number of nodes and high-density server layouts. The infrastructure can be effortlessly expanded with additional InfiniBand switches and cables to increase network scale and bandwidth capacity.
High Reliability: InfiniBand networks incorporate redundant designs and fault isolation mechanisms, ensuring superior reliability and uninterrupted network connectivity even in the event of hardware failures.
FS InfiniBand Solution
FS provides a comprehensive range of NVIDIA® InfiniBand products, each undergoing rigorous performance and scenario testing. FS InfiniBand transceivers undergo 100% target device testing to ensure seamless compatibility. For more testing details, visit the FS Testing Center.
Type | ||
Transceivers and Cables | 1.6T XDR OSFP DAC | |
800G NDR OSFP Transceivers 800G NDR OSFP DAC | ||
400G NDR OSFP Transceivers 400G NDR QSFP112 Transceivers 400G NDR OSFP DAC 400G NDR OSFP AOC | ||
200G HDR OSFP56 Transceivers 200G HDR QSFP56 DAC 200G HDR OSFP56 ACC 200G HDR OSFP56 AOC | ||
NICs | ConnectX-6 ConnectX-7 | |
Switches | Quantum-2 InfiniBand Switches Quantum InfiniBand Switches | |
FS offers high-performance InfiniBand network solutions, specifically designed for various devices and systems in data center and high-performance computing (HPC) environments. It ensures high-speed, low-latency, and scalable data transfer capabilities.
The topology diagram below visually represents product connectivity within an InfiniBand network. This illustration helps users understand how FS InfiniBand network solution optimizes data transfer efficiency, reduces latency, and enhances system performance.
By partnering with FS and implementing a stable InfiniBand network, you can unlock new opportunities, accelerate business growth, and enhance the overall user experience! Discover the ideal solution with FS Solution Services!