Unlocking HPC: Why InfiniBand is the Preferred Choice for RDMA
Updated at May 22nd 20241 min read
When it comes to high-performance computing (HPC), speed and efficiency are paramount. Remote Direct Memory Access (RDMA) is a technology that enables memory in one computer to be accessed by another computer without involving the operating system or CPU, which drastically reduces latency and improves throughput. While there are several RDMA technologies, InfiniBand has established itself as the leader in this space. This article highlights the reasons why InfiniBand stands out as the technology of choice for RDMA applications.
SDN Principles: Enhanced Network Efficiency and Simplicity
One of InfiniBand's standout features is its alignment with the principles of Software-Defined Networking (SDN). Managed by a subnet manager acting as the SDN controller, InfiniBand eliminates the need for routing protocols traditionally found in Ethernet networks, including RoCE. The entire network's forwarding tables are computed and distributed by a centralized subnet manager. Additionally, the subnet manager handles crucial configuration aspects within the InfiniBand subnet, such as partitioning and Quality of Service (QoS). Unlike RoCE, InfiniBand networks do not rely on broadcast mechanisms like ARP for forwarding table learning, effectively eliminating broadcast storms and unnecessary bandwidth consumption.
In contrast, traditional Ethernet, including RoCE, does support SDN controller-based networking. However, network vendors have shifted away from the earlier OpenFlow-based flow table forwarding concept to avoid becoming mere "white-box" manufacturers. Instead, they have embraced solutions based on netconf, VXLAN, and EVPN. While SDN controllers have evolved into advanced "network management systems" focused on deploying control policies, forwarding still heavily relies on device-based learning, such as MAC table learning, ARP tables, and routing tables. This divergence has resulted in RoCE losing the efficiency and simplicity advantages found in InfiniBand.
Congestion Avoidance Mechanism: Achieved Native Lossless Network
InfiniBand networks leverage a credit-based mechanism that effectively prevents buffer overflow and packet loss issues. This mechanism ensures that packet transmission is initiated only when the receiver has sufficient credits to accept the corresponding number of messages.
The credit-based mechanism functions as follows: Each link in the InfiniBand network has a predetermined buffer for storing packets to be transmitted. Before sending data, the sender checks the receiver's available credits, representing the current buffer size. Based on this credit value, the sender determines whether to initiate packet transmission. If the receiver has insufficient credits, the sender waits until the receiver releases enough buffer space and reports new available credits.
Once the receiver finishes forwarding, it releases the utilized buffer and continuously reports the current available buffer size to the sender. This dynamic adjustment enables the sender to fine-tune packet transmission based on the receiver's buffer status. Such link-level flow control ensures that the sender does not overwhelm the receiver with excessive data, effectively preventing network buffer overflow and packet loss.
In contrast, RoCE employs a "post-congestion" management mechanism. It does not negotiate resources with the receiver before sending packets but directly forwards them without prior coordination. Only when the receiver's switch experiences port buffer congestion (or imminent congestion) does it send congestion management messages using Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) protocols to reduce or pause packet transmission on the opposing switch and network card. While this "post-congestion" approach can partially alleviate congestion impact, it falls short of completely preventing packet loss and maintaining network stability.
Direct Forwarding Mode: Enabled Networks to Achieve Lower Latency
Ethernet networks, like those utilizing RoCE, generally use a store-and-forward method where switches buffer the incoming data packet, check its destination and integrity, and then send it on. This can lead to delays, especially under heavy packet traffic loads.
Conversely, switches with Cut-through technology scan only the packet's header to identify the port for delivery, beginning the forwarding process at once. This slashes the packet's dwell time in the switch and minimizes latency.
InfiniBand switches employ such Cut-through forwarding, streamlining the process for layer 2 packets. They quickly pin down the routing path using a 16-bit LID from the subnet manager, dropping latency below 100 nanoseconds. Ethernet switches, conversely, rely on MAC table lookups and store-and-forward techniques that take longer owing to additional tasks like handling IP and MPLS. While some Ethernet switches might use Cut-through, their delays can still be over 200 nanoseconds.
Professional Provider of Networking Solutions-FS
FS offers comprehensive solutions with its selection of InfiniBand optical modules and high-speed cables. Below are the original IB products available from FS.
Categories | Ports | Speeds | PCIe Interface | Features |
32 Ports / 40 Ports | 40 x HDR 200G QSFP56 / 32 x NDR 800G OSFP | / | Managed / Unmanaged | |
Dual / Single | 100G QSFP56 200G QSFP56 400G QSFP112 400G OSFP | PCIe 4.0 x 16 PCIe 5.0 x 16 | ConnectX®-6 VPI onnectX®-7 ConnectX®-7 VPI | |
/ | 800G NDR / 400G NDR / 200GHDR / 100G EDR / 56G FDR / 40G FDR | / | ≦50m distance ≦100m | |
/ | 800G NDR / 400G NDR / 200GHDR / 100G EDR / 56G FDR / 40G FDR | / | ≦50m distance ≦40km |
Quick Shipping: Our ample inventory guarantees speedy delivery. Orders are reliably dispatched ASAP, accelerating your project’s timeline by minimizing delays.
Test Support: FS provides professional performance test, reliability test, scenario test,and compatibility test. If you want to find out more about testing, please go to Test Support Center.
Reliable Support and Delivery: We have a track record of stable product functioning that has earned customer trust through multiple successful partnerships. Our agile technical support offers peace of mind with continual after-sales assistance.
Rest assured, our consistent stock levels cater to your immediate needs, eliminating concerns about quality or availability.
Conclusion
InfiniBand's native lossless networking and superior flow control make it the preferred choice for RDMA deployments. By leveraging these unique advantages, businesses can optimize their networking infrastructure and enhance overall performance.