N8610-32D Switch Datasheet

18-12-2025 - N8610-32D Switch Datasheet Product Overview A key enabler in the ecosystem of ultra-high performance applications including hyper-scale cloud computing and AI/ML clusters is 400 Gigabit Ethernet. Faster, higher capacity CPUs, specialist processors, Smart NICs, GPUs, DPUs and flash storage enable the construction of larger clusters which require high bandwidth, low latency, large radix networks to achieve optimal performance. N8610-32D switch delivers 32 400G ports in a 1U system with an overall throughput of 12.8 Tbps in a single chip platform. Deterministic, low latency, line rate performance, proven layer 2 and layer 3 features, and advanced traffic awareness, congestion handling and instrumentation provides the ideal foundation for ultra-high performance applications with scale to match the largest clusters’ requirements. N8610-32D switch supports QSFP-DD interfaces compatible with industry standard optics and cables allowing for ease of migration to 400G. All ports allow a choice of speeds including 400GbE, 100GbE operation, or breakout into 2x 200GbE, 4x 100GbE, up to 128 interfaces. Figure 1 shows the FS N8610-32D Switch N8610-32D.png Figure 1. The N8610-32D is a high-speed, high-density, spine-and-leaf switch featuring: 32 400GbE (QSFP-DD) or 100GbE (QSFP28) downlink/uplink ports in 1U form factor Broadcom BCM56993 Tomahawk 4 with 128GB (SSD) storage Up to 12.8 Tbps (unidirectional) L2 and L3 performance Advanced PicOS® features, such as RoCEv2, PFC, ECN, DLB, EVPN-VXLAN* and MLAG Using breakout cables, each of the 32 400GbE QSFP-DD ports can be broken into 2x 200GbE or 4x 100GbE ports, increasing the total number of supported 200GbE ports per switch to 64 and 100GbE ports per switch to 128. PicOS® The high-performance N8610-32D switch runs PicOS, a powerful and robust network operating system that supports all FS PicOS® network switches. Key PicOS features that enhance the functionality and capabilities of the N8610-32D include: Commit, Review, and Rollback: Prevents network configuration errors and enables rapid recovery to a stable state in case of anomalies, ensuring configuration accuracy and business continuity. Virtual ASIC Technology: Implements a hardware abstraction layer, allowing support for multiple hardware platforms and chipsets with minimal modifications. This vendor-agnostic solution enables rapid iteration and updates. Modular Design: Allows independent component operation and updates, enhancing system flexibility and stability. This architecture enables seamless integration of new features and simplifies maintenance and troubleshooting. Linux Debian Architecture: One of the most innovative open network operating systems in the industry, featuring built-in automation tools for easy implementation, management, customization and scalability. Automation and Programmability: PicOS® offers a rich set of standardized programmable interfaces and automation tools, including Ansible, OpenFlow, and NETCONF, enabling automated network configuration and improved operational efficiency. Features and Benefits Built-in Broadcom Tomahawk 4 Chip: Provides high-speed data transfer, low latency and 12.8 Tbps throughput for superior stability and performance. RoCEv2 (RDMA over Converged Ethernet version 2): RoCEv2 allows Remote Direct Memory Access (RDMA) over Ethernet. Adopting RoCEv2 networking capabilities offers additional benefits such as lower latency and lossless communication, enabled by congestion management mechanisms like Priority-based Flow Control (PFC) and Explicit Congestion Notification (ECN). DLB (Dynamic Load Balancing): DLB considers member bandwidth utilization along with packet content for member selection, enabling better link utilization based on real-time link loads. At the same time, it avoids hash collision drops seen in SLB by ensuring that links occupied by elephant flows are not selected for mice flows, thus delivering more balanced and reliable traffic distribution. Unified Operating System and Management Platform: Unified PicOS® and AmpCon-DC management platform, automate the entire network lifecycle to simplify design and deployment. Free Virtual Machine (VM): PicOS-V is a Virtual Machine designed to help customers become familiar with the network functionalities and performance of PicOS®, without the need to wait for switching hardware. AmpCon-DC Management Platform The FS AmpCon-DC management platform ensures fast, accurate, and consistent delivery of the changes needed for data center services. It also leverages built-in assurance and analytics features to quickly resolve Day-2 operational issues. Fabric Management: AmpCon-DC management platform provides full Day 0 through Day 2+ lifecycle management capabilities for IP/EVPN fabrics with closed-loop assurance in the data center Telemetry for Real-time Network Monitoring: Optimizes network performance through continuous data insights. Topology Auto-discover for Visual Management: Enhances efficiency in network management and operations. Overlay/Underlay-based Auto Configuration*: Centralized configuration is automatically issued to increase configuration efficiency (reduce configuration errors and time to familiarize with commands). Lossless Network Automation*: The overall network can be monitored and optimized, which improves business efficiency and the operation and maintenance efficiency of network administrators. Lossless Network O&M Monitoring: If a link failure occurs in the network, the chip can achieve sub-millisecond convergence, minimizing the impact on user services. Notice：*Expected to be available in Q4 2025 N8610-32D Switch Specifications Tables 1 through 4 show the FS N8610-32D switch hardware specifications. Table 1. Interface Options P/N N8610-32D Console port 1 x RJ-45 port 1 x Micro USB console port Management port 1 x RJ-45 port USB port 1 100GbE QSFP28 128 (with breakout cable) 200GbE QSFP56 64 (with breakout cable) 400GbE QSFP-DD 32 Table 2. Power Supplies and Fans P/N N8610-32D Power supply 2x Hot-swappable AC Power Supplies (1+1 Redundancy) Fan number 6x Hot-swappable Fans (5+1 Redundancy) Airflow Front-to-Back Max. power consumption Max. power draw: 1370W Input-voltage range and frequency AC Input: Rated Voltage Range: 100~127VAC, 50~60Hz Rated Input Current Range: 12A Max. Rated Voltage Range: 200~240VAC, 50~60Hz Rated Input Current Range: 8A Max. Table 3. Performance Specifications P/N N8610-32D Switching capacity 12.8/25.6 Tbps (uni/bidirectional) Forwarding rate 5,079 Mpps Switch chip Broadcom BCM56993 Tomahawk 4 CPU Intel® Xeon® Processor D-1713NTE 4-Core 2.2 GHz SDRAM 32GB DDR4 SO-DIMM with ECC Flash memory 128GB m.2 SATA SSD with PLP support Jumbo Frame 9K Packet buffer 56.83MB MAC addresses 131K Table 4. Product Specifications P/N N8610-32D Environmental Operating temperature 0°C to 45°C (32°F to 113°F) Storage temperature -40°C to 70°C (-40°F to 158°F) Operating humidity 5% to 95% (Non-condensing) Physical specifications Weight 26.12 lbs (11.85 kg) Dimensions (H x W x D) 1.71"x 17.26"x 23.23" (43.5x 438.4x 590mm) Rack units (RU) 1 RU Electrical Voltage (auto ranging) 100V~127V AC, 200V~240VAC Frequency 50-60Hz Current 100~127 VAC, 50~60 Hz, 12A Max. 200~240 VAC, 50~60 Hz, 8A Max. Software Features Supported Table 5 lists the software spotlights for the FS N8610-32D switch. Table 5. Software Spotlights Functionality Description Layer 2 Features Support MLAG (Multi-chassis Link Aggregation) Support EVPN-VXLAN* Support STP (IEEE 802.1D) Support Rapid Spanning Tree Protocol (RSTP) (IEEE 802.1w), MSTP (IEEE 802.1s) Support VLAN—IEEE 802.1Q VLAN trunking Support Link Aggregation and Link Aggregation Control Protocol (LACP) (IEEE 802.3ad) Support IEEE 802.1AB Link Layer Discovery Protocol (LLDP) Support Bridge protocol data unit (BPDU) protect Layer 3 Features Support static routing Support OSPFv2, OSPFv3 Support Virtual Router Redundancy Protocol (VRRP) Support IPv6 Support BGP4/BGP4+ Support DHCP snooping/Relay Support RIP, RIPng Support Equal-Cost Multipath Routing (ECMP) Support PBR (Policy-Based Routing) Multicast Support Internet Group Management Protocol (IGMP) v1/v2/v3 Support IGMP v1/v2 snooping Support Protocol Independent Multicast PIM-SM and PIM-SSM QoS Support IEEE 802.1p-based CoS Support TOS or DSCP-based CoS Support ACL classification, metering, and remarking Support SP, WRR, WFQ scheduling Support Tail drop Support WRED congestion control High Availability Support Bidirectional Forwarding Detection (BFD) Support GR for RIP/OSPF/BGP Support Device Link Detection Protocol (DLDP) Support Rapid Ethernet Uplink Protection (REUP) Visibility and Analytics Support AmpCon-DC Management Platform for full Day 0 to Day 2+ capabilities Support Test Automation with PicOS-V Support Switched Port Analyzer (SPAN) Support Remote SPAN (RSPAN) Support Encapsulated Remote SPAN (ERSPAN) Support sFlow Support Telemetry Support gRPC protocol Support SNMP v1/v2c/v3 Support Ansible Support Python Support Command line interface (CLI) Support Netconf configuration and automation tools Support Openflow, CrossFlow Lossless Network Features Support RoCE/RoCEv2 (RDMA over Converged Ethernet) Support PFC (Priority Flow Control) Support PFC deadlock prevention Support PFC deadlock detection Support fully shared packet buffer Support RoCE EasyDeploy Support ECN (Explicit Congestion Notification) Support Easy ECN Support Dynamic Load Balancing (DLB) Standards Compliance Table 6 lists the standards compliance for the FS N8610-32D switch. Table 6. Standards Compliance Category Description IEEE Standard IEEE 802.1 IEEE 802.1AB IEEE 802.1ad IEEE 802.1ax IEEE 802.1D IEEE 802.1p IEEE 802.1Q IEEE 802.1Qbb IEEE 802.1w IEEE 802.3x Supported RFCs RFC 768 UDP RFC 791 IP RFC 792 ICMP RFC 793 TCP RFC 826 ARP RFC 854 Telnet client and server RFC 894 IP over Ethernet RFC 1058 RIP RFC 1112 IP Multicast Host Extensions RFC 1142 OSI IS-IS Intra-domain Routing Protocol RFC 1492 TACACS RFC 1519 Classless Interdomain Routing (CIDR) RFC 1534 DHCP-BOOTP Interoperation RFC 1745 BGP4/IDRP for IP—OSPF Interaction RFC 1771 BGP-4 RFC 1812 Requirements for IP Version 4 Routers RFC 1997 BGP Communities Attribute RFC 2080 RIP for ipv6 RFC 2131 DHCP RFC 2132 DHCP Options & BOOTP Extensions RFC 2138 RADIUS Authentication RFC 2139 RADIUS Accounting RFC 2154 OSPF with Digital Signatures (Password, MD-5) RFC 2236 IGMP v2 RFC 2328 OSPF v2 RFC 2338 VRRP RFC 2370 OSPF Opaque LSA Option RFC 2385 Protection of BGP Sessions via the TCP MD5 Signature Option RFC 2453 RIP v2 RFC 3031 MPLS Architecture RFC 3032 MPLS Label Stack Encoding RFC 3034 Label Switching over Frame Relay RFC 3036 LDP Specification RFC 3037 LDP RFC 3046 DHCP Relay Agent Info Option RFC 3101 NSSA Option RFC 3215 LDP State Machine RFC 3376 IGMP v3 RFC 3446 Anycast RP Mechanism (PIM+MSDP) RFC 3569 SSM Overview RFC 3618 MSDP RFC 4541 IGMP/MLD Snooping Guidelines RFC 4601 PIM-SM(Recised) RFC 4607 IP Source-Specific Multicast RFC 5036 LDP Specification (Updated) RFC 5443 LDP-IGP Synchronization RFC 5561 BGP-Signaled IP/VPNs RFC 5880 BFD Base Protocol RFC 5881 BFD for IPv4/IPv6 RFC 5882 BFD Generic Application RFC 5883 BFD for Multihop Paths RFC 6720 Early IANA Code Point Allocation RFC 7348 VXLAN RFC 7552 GMPLS Packet-Optical Integration RFC 8365 EVPN-VXLAN Warranty, Service and Support FS N8610-32D switch has a 5-year limited warranty against defects in materials or workmanship. For more information for FS Returns & Refunds policy, visit https://www.fs.com/policies/warranty.html or https://www.fs.com/policies/day_return_policy.html FS provides a personal account manager, free professional technical support, and 24/7 live customer service to each customer. Professional Lab: Test each product with the latest and advanced networking equipment. Free Technical Support: Provide free & tailored solutions and services for your businesses. 80% Same-day Shipping: Immediate shipping for in-stock items. Fast Response: Direct and immediate assistance from an expert. For more information, visit https://www.fs.com/service/fs_support.html Ordering Information Table 7 provides the ordering information for N8610-32D switch and AmpCon-DC Management Platform Table 7. Ordering Information Product Description Switch Hardware N8610-32D N8610-32D, 32-Port Ethernet HPC/AI Data Center Switch, 32 x 400G QSFP56-DD, PicOS®, Broadcom Tomahawk 4 Chip AmpCon-DC Management Platform LIS-AMPCON-DC-FPSW-Foundation-1Y AmpCon-DC Management Platform for PicOS® Data Center Switches with 1 Years Service Bundle, Support Remote Deployment and Automate Network Management (Per Device) LIS-AMPCON-DC-FPSW-Foundation-3Y AmpCon-DC Management Platform for PicOS® Data Center Switches with 3 Years Service Bundle, Support Remote Deployment and Automate Network Management (Per Device) LIS-AMPCON-DC-FPSW-Foundation-5Y AmpCon-DC Management Platform for PicOS® Data Center Switches with 5 Years Service Bundle, Support Remote Deployment and Automate Network Management (Per Device)

Lossless Network Configuration

19-11-2025 - Lossless Network Configuration 1. Lossless Network Introduction 1.1 Overview Lossless networks are crucial for the implementation of RDMA over Converged Ethernet (RoCE), a network protocol that enables high-throughput and low-latency data communication between nodes in a network. RoCE leverages the advantages of Remote Direct Memory Access (RDMA) technology over standard Ethernet networks, making it highly suitable for applications requiring rapid data transfer and minimal latency, such as high-performance computing, distributed storage, AI deep learning model training and big data analytics. 1.2 Advantages Low Latency: RoCE reduces the latency of data transfers by bypassing the CPU and the operating system in the data path, allowing direct memory-to-memory data transfers. High Throughput: By enabling direct memory access, RoCE can achieve higher data transfer rates compared to traditional Ethernet communications. Efficient CPU Usage: Because RoCE offloads data transfer tasks from the CPU, it frees up CPU cycles for other processing tasks, enhancing overall system performance. Ethernet Compatibility: RoCE leverages the existing Ethernet infrastructure, making it cost-effective to deploy in data centers without requiring specialized networking equipment. 1.3 Versions of RoCE There are two main versions of RoCE: RoCEv1: This version operates at Layer 2 of the OSI model, meaning it is not routable beyond the local Ethernet network. It requires all devices to be in the same Ethernet broadcast domain. RoCEv2: This version operates at Layer 3, making it routable over IP networks. RoCEv2 packets can traverse multiple subnets, allowing for greater scalability and flexibility in network design. 1.4 Supported Platforms To achieve lossless network communication, high throughput, and low latency data exchange between nodes, features like PFC Watchdog, Explicit Congestion Notification (ECN), and Dynamic Load Balancing are only supported on Trident3-X5, Trident3-X7, Trident4-X9, Trident4-X11, Tomahawk2, Tomahawk3, Tomahawk4, Tomahawk5 platforms. 1.5 Switch Configuration To effectively leverage RoCE in your network, configuring your switches properly is crucial. Here’s the guide to configure lossless network on network switches, focusing on key features: PFC, PFC Watchdog, ECN and Dynamic Load Balancing (DLB). 1. Interface Enable with PFC on the required priority. 2. Global Configuration Configure ECN via WRED Enable WRED. Set the maximum and minimum thresholds. Set drop probability. Enable ECN. Configure PFC Watchdog Enable PFC on the interface before enabling PFC watchdog. Enable PFC watchdog. Configure the time interval of PFC deadlock detection. Configure the restore time and restore action when PFC deadlock occurs. Configure Dynamic Load Balancing 2. Application Scenarios Lossless networks are utilized across various industries and applications due to its ability to provide low-latency, high-throughput communication. Here are some key use cases of lossless networks: 2.1 Application Scenarios 1: High Performance Computing (HPC) Lossless networks are widely used in High Performance Computing (HPC) due to its ability to provide low-latency and high-throughput communication. These attributes are critical for HPC environments, which involve complex computational tasks requiring efficient data transfer between numerous computing nodes. Parallel Computing: Inter-node Communication: Lossless network is used for fast communication between nodes in a cluster, essential for parallel computing tasks where multiple nodes work on different parts of a problem simultaneously. MPI Acceleration: Lossless network enhances the performance of MPI (Message Passing Interface) applications by reducing communication overhead. Distributed Databases: Efficient Data Replication: Lossless network facilitates high-speed data replication between database nodes, ensuring data consistency and high availability. Query Processing: Faster data movement between nodes improves the performance of distributed query processing. Data Analytics and Machine Learning: Large Dataset Handling: Lossless network enables efficient handling of large datasets, which is crucial for big data analytics and machine learning applications. Model Training: Accelerates the training of machine learning models by speeding up data transfer between compute nodes and storage systems. Scientific Simulations: Real-time Data Sharing: Scientific simulations often require real-time data sharing between nodes, which lossless network supports through its low-latency and high-throughput capabilities. Collaborative Research: Facilitates collaborative research by enabling seamless data exchange and communication between geographically distributed research centers. 2.2 Application Scenarios 2: Distributed Storage Lossless network is increasingly being applied in distributed storage systems due to its ability to provide low-latency and high-throughput data transfers. These features are crucial for distributed storage environments, which require efficient and reliable data movement between storage nodes. Distributed File Systems: Fast Data Access: Lossless network enhances the performance of distributed file systems like HDFS (Hadoop Distributed File System) by enabling fast data access and transfers between nodes. Efficient Data Replication: Ensures that data replication between storage nodes is performed quickly and reliably, maintaining data consistency and availability. Object Storage: High-Performance Object Storage: Lossless network can be used to improve the performance of object storage systems like Ceph, which require efficient handling of large objects across distributed nodes. Reduced Latency: Low-latency data transfers ensure quick access to stored objects, enhancing the overall user experience and system performance. Block Storage: Enhanced Block Storage Performance: Lossless network improves the performance of block storage solutions by enabling low-latency access to storage blocks, which is crucial for applications requiring fast I/O operations. Efficient Volume Management: Facilitates efficient volume management and data migration between storage devices. Software-Defined Storage (SDS): Improved SDS Efficiency: Lossless network can enhance the efficiency of software-defined storage systems by enabling high-speed, low-latency communication between storage nodes and controllers. Scalable Storage Solutions: Supports the scalability needs of SDS, allowing for seamless expansion and management of storage resources. 2.3 Application Scenarios 3: Artificial Intelligence (AI) Lossless network is highly beneficial in AI environments, particularly for deep learning and large-scale AI workloads. The need for high-speed, low-latency communication between numerous GPUs or compute nodes makes lossless network an ideal choice for AI applications. Deep Learning Model Training: Distributed Training: Lossless network facilitates efficient communication between multiple GPUs or nodes during distributed training, reducing training time. Data Parallelism: Enhances data parallelism by allowing seamless data exchange between nodes, ensuring that each node has the required data for training. Inference Serving: Low-latency Inference: Lossless network’s low-latency capabilities are critical for real-time inference serving, enabling quick responses in AI-driven applications. Scalable Inference: Supports scaling inference workloads across multiple nodes or GPUs, ensuring that large-scale inference tasks are handled efficiently. AI Data Processing Pipelines: High-throughput Data Transfers: Lossless network can handle the high-throughput data transfers required in AI data processing pipelines, such as ETL (Extract, Transform, Load) operations. Streamlined Data Movement: Ensures efficient data movement between storage and compute nodes, enhancing the performance of data preprocessing steps in AI workflows. Big Data Analytics: Accelerated Analytics: By providing high-speed data transfers, lossless network accelerates the analytics processes that feed into AI models, improving the overall pipeline efficiency. Integration with Hadoop and Spark: Enhances the performance of big data frameworks like Hadoop and Spark, which are often used in conjunction with AI workloads. 3. Key Features of Lossless Network Lossless network is a powerful networking protocol designed to enable high-performance, low-latency data transfers over Ethernet networks. By leveraging RDMA technology, lossless network offers significant benefits for various high-demand applications, making it a crucial component in modern data centers, high-performance computing environments, and cloud services. Its ability to utilize existing Ethernet infrastructure makes it a cost-effective solution for enhancing network performance. Lossless network switches are designed to support the unique requirements of RDMA, providing low-latency and high-throughput communication over Ethernet networks. Here are the key technologies and features that are crucial for lossless network switches: 3.1 Priority Flow Control (PFC) Priority Flow Control (PFC) is a network protocol designed to manage congestion on Ethernet networks by allowing for the independent pausing of traffic based on priority levels. It is part of the IEEE 802.1Qbb standard and is particularly useful in data center environments where different types of traffic need to be handled with varying degrees of urgency. PFC is a critical technology for managing congestion in Ethernet networks, particularly in data center environments. By enabling traffic differentiation and providing lossless transport for high-priority traffic, PFC helps maintain performance and reliability for critical applications. 3.2 PFC Watchdog PFC Watchdog is a critical component in Ethernet networks that use Priority Flow Control to manage congestion. By detecting and mitigating PFC deadlocks, it ensures that the network remains stable and performs optimally. This mechanism is particularly important in data centers and other environments where maintaining high performance and reliability is crucial. 3.3 Explicit Congestion Notification (ECN) Explicit Congestion Notification (ECN) is an effective mechanism for managing network congestion by marking packets instead of dropping them. It is an extension of the IP and TCP protocols, enhancing the way congestion is managed by marking packets instead of discarding them. It helps improve network performance, reduce packet loss, and maintain high throughput, making it especially valuable in data centers and for real-time applications. 3.4 Dynamic Load Balance Dynamic Load Balancing is a network management technique used to distribute network traffic across multiple paths or resources dynamically. Unlike static load balancing, which assigns fixed routes or resources to handle traffic, dynamic load balancing adjusts traffic distribution in real-time based on current network conditions. It is a vital network management technique that optimizes performance, scalability, and reliability by dynamically distributing traffic across multiple paths or resources. By adapting to changing network conditions in real-time, dynamic load balancing ensures efficient resource utilization and enhances overall network performance and resilience. In the following sections, we will explain the details of what they are, how they work, and how to configure. 4. Configuring Priority-based Flow Control (PFC) 4.1 Enabling PFC Function 4.1.1 Overview Priority-based Flow Control (PFC) is a type of flow control mechanism. The advantage of PFC over traditional flow control mechanisms is that PFC provides flow control based on per-code-point (priority). In other words, PFC offers a more granular form of flow control. This means that if traffic from one particular priority suffers from congestion, only that traffic is paused until the congestion clears, while traffic for other priorities continues unhindered. On each physical port, there are 8 (0 to 7) Class of Service (CoS) queues. If congestion is detected on the egress physical port, the ingress port will send a PAUSE frame to the transmitting node to pause transmission until the receiving node is ready to accept packets again. PFC applies only to packets entering a port. PFC has a higher priority than traditional flow control. For example, if both flow control and PFC are configured on the same port, PFC will take precedence over traditional flow control. PFC uses the IEEE 802.1p CoS values in the IEEE 802.1Q VLAN tag to generate the flow control frame with the corresponding priority on the ingress physical port when the egress physical port suffers congestion. This indicates that the ingress port requires CoS classifier configuration. 1. Key Features of PFC Lossless Transmission: PFC ensures lossless transmission for high-priority traffic classes by pausing traffic when congestion is detected, preventing packet loss. Per-Priority Flow Control: Unlike traditional Ethernet flow control, which applies to all traffic on a link, PFC operates on individual traffic priorities, allowing selective flow control. Congestion Management: By preventing packet loss for specific traffic classes, PFC helps manage congestion and improve overall network performance. 2. Working Mechanism of PFC PFC operates using the following key components and steps: Traffic Classification: Network traffic is classified into different priorities based on the IEEE 802.1p standard, which defines eight priority levels (0-7). Each priority level corresponds to a specific type of traffic, such as storage, voice, or best-effort data. Priority Mapping: The switch maps traffic to these priority levels using VLAN tags (802.1Q) or Differentiated Services Code Point (DSCP) values. Each priority level is associated with a separate queue within the switch. Congestion Detection: When a switch detects congestion in one of its priority queues (e.g., due to high traffic volume or insufficient buffer space), it generates a PFC frame for that specific priority level. PFC Frame Transmission: The PFC frame, also known as a PAUSE frame, is sent to the upstream device (e.g., another switch or a network interface card in a server). This frame instructs the upstream device to pause the transmission of traffic for the specified priority level. Traffic Pause: Upon receiving the PFC frame, the upstream device temporarily stops sending traffic of the specified priority to the congested switch. Other traffic classes (priorities) continue to flow without interruption, ensuring that only the affected traffic is paused. Congestion Alleviation: The switch continues to process and forward the paused traffic until the congestion is alleviated. Once the buffer space is available, the switch sends another PFC frame to the upstream device to resume the paused traffic flow. Resume Transmission: The upstream device resumes sending traffic for the previously paused priority, restoring normal traffic flow. 4.1.2 Advantages of PFC Improved Performance: By preventing packet loss for high-priority traffic, PFC ensures that critical applications, such as storage and real-time communications, perform reliably. Enhanced Network Efficiency: PFC allows network devices to manage congestion more effectively, reducing the likelihood of network bottlenecks and improving overall network efficiency. Coexistence of Traffic Types: PFC enables different types of traffic to coexist on the same network infrastructure without interfering with each other, supporting the convergence of storage, data, and voice traffic. 4.1.3 Restrictions and Guidelines When you configure PFC, follow these restrictions and guidelines: It is essential to ensure that the PFC functionality is enabled on all ports through which the packets flow. To prevent packet loss due to congestion during transmission, please configure the same PFC settings on all ports through which the packets flow. 4.1.4 Procedure Step 1 Configure PFC Profile. By default, PFC is enabled. PFC is disabled when drop value is set to true and enabled when drop value is set to false. The default value of drop is false. set class-of-service pfc-profile code-point drop Step 2 Apply PFC profile to the interface. set class-of-service interface pfc-profile Step 3 Commit the configuration. commit 4.1.5 Show PFC Frame Statistics on Port After complete the configuration, the command run show class-of-service interface can be used to show the class of service statistics information on specified interface. 4.1.6 Configuration Example The following commands complete the configurations: Configure PFC profile pfc1. Apply PFC profile to the port ge-1/1/1. admin@PICOS# set class-of-service pfc-profile pfc1 admin@PICOS#set class-of-service interface ge-1/1/1 pfc-profile pfc1 Show the class of service statistics information on specified interface. The class 0~7 in PFC frame corresponds to the following "802.1P" item. The value of ”RxPFC“ item will be incremented by 1 if ge-1/1/1 receives a PFC frame. The value of ”TxPFC“ item will be incremented by 1 if ge-1/1/1 sends out a PFC frame. admin@PICOS# run show class-of-service interface ge-1/1/1 Interface : ge-1/1/1 802.1P Priority Flow Control RxPFC TxPFC ----------- --------------------- --------------- --------------- 0 false 0 500 1 false 0 0 2 false 0 71 3 false 0 0 4 false 0 0 5 false 0 0 6 false 0 102 7 false 0 0 trust mode : ieee-802.1 Default ieee-802.1 : 0 Default dscp : 0 Default inet-precedence : 0 Local-priority Queue-Schedule Code-points -------------- --------------------------- ------------------------- 0 SP,0kbps 1 SP,0kbps 2 SP,0kbps 3 SP,0kbps 4 SP,0kbps 5 SP,0kbps 6 SP,0kbps 7 SP,0kbps 4.2 Configuring PFC Buffer 4.2.1 Overview The buffer is configured to implement traffic control and PFC watchdog, which is based on interface and priority queue. The storage space of each interface is divided into different buffers independently and a certain action will be executed after the number of accumulated packets reaches the buffer threshold (the unit is cell). Guaranteed: the dedicated buffer available for a specified queue. This buffer space cannot be used by other queues. Shared: the public buffer available for all queues. The shared buffer will be occupied when the guaranteed buffer reaches the threshold. Headroom: the maximum number of cell resources available for packets with a specific 802.1p priority. When the occupied shared buffer reaches the threshold, the packets will be saved in the headroom buffer. 1. Threshold and Execution Action After the PFC function is enabled, the default storage space is available for a specified queue of an interface. To flexibly control the PFC function and make good use of the storage space, you can configure the thresholds for different buffer types and the corresponding action will be executed. The detailed information is shown below. Figure 1. Communication Between Upstream and Downstream Switches image.png Figure 2. Buffer Threshold and Execution Action of an Interface image.png Initially, the packets of a specified queue on an interface are saved in the guaranteed buffer. When the occupied space is higher than the guaranteed buffer threshold, the shared buffer will be occupied. When the occupied space is higher than the shared buffer threshold, the ingress interface and egress interface of the downstream switch will execute different actions. a) For the ingress interface, it will generate and send Pause frames to the egress interface of the upstream switch to stop sending packets. The Pause frames will stop being generated when the occupied space is reduced to a certain level (the shared threshold minus the offset value). When the occupied space is higher than the global threshold, the packets will be saved in the headroom buffer. b) For the egress interface, the packets will be directly saved in the headroom buffer. When the occupied space is higher than the headroom buffer threshold, the packets will be dropped. 4.2.2 Restrictions and Guidelines When you configure PFC buffer, follow these restrictions and guidelines: PFC should be enabled on the interface before configuring PFC buffer. Currently, the PFC buffer is only supported on Trident3-X5, Trident3-X7, Trident4-X9, Trident4-X11, Tomahawk2, Tomahawk3, Tomahawk4, Tomahawk5 platforms. The values of cell are different on different platforms, as shown below. Platform Cell value Trident3-X5 256bytes Trident3-X7 256bytes Trident4-X9 318bytes Trident4-X11 254bytes Tomahawk2 208bytes Tomahawk3 254bytes Tomahawk4 254bytes Tomahawk5 254bytes 4.2.3 Procedure Step 1 Enable PFC on the interface before configuring PFC buffer. set class-of-service pfc-profile [code-point drop ] set class-of-service interface pfc-profile Step 2 Set the upper threshold of guaranteed buffer for a PFC queue on the ingress interface. When the guaranteed buffer threshold is reached, the packets will be saved in the shared service pool. set interface gigabit-ethernet ethernet-switching-options buffer ingress-queue guaranteed Step 3 Set the upper threshold of shared buffer for a PFC queue on the ingress interface. When the occupied buffer space exceeds the specified threshold, the Pause frame will be generated and sent to the egress interface. You can specify the threshold in the static (a fixed value) or dynamic (percentage) way. Only one way can be configured, or the error prompt will appear. set interface gigabit-ethernet ethernet-switching-options buffer ingress-queue shared-ratio set interface gigabit-ethernet ethernet-switching-options buffer ingress-queue threshold Step 4 Set the offset value of shared buffer for a PFC queue on the ingress interface. The Pause frames will stop being generated when the occupied space is reduced to a certain level (the upper threshold minus the offset value). set interface gigabit-ethernet ethernet-switching-options buffer ingress-queue reset-offset Step 5 Set the global threshold of shared buffer for a PFC queue on all ingress interfaces. When the global threshold is reached, the packets will be saved in the headroom buffer. set interface ethernet-switching-options buffer service-pool threshold Step 6 Set the upper threshold of headroom buffer for a PFC queue on the ingress interface. When the headroom buffer threshold is reached, the interface will drop received packets. set interface gigabit-ethernet ethernet-switching-options buffer ingress-queue headroom Step 7 Set the upper threshold of shared buffer for a PFC queue on the egress interface of the downstream switch. When the occupied buffer space exceeds the specified threshold, the packets will be saved in the headroom buffer. You can specify the threshold in the static (a fixed value) or dynamic (percentage) way. Only one way can be configured, or the error prompt will appear. set interface gigabit-ethernet ethernet-switching-options buffer egress-queue shared-ratio set interface gigabit-ethernet ethernet-switching-options buffer egress-queue threshold Step 8 Commit the configuration. commit Step 9 (Optional) View the PFC buffer information of an ingress interface or an egress interface. run show interface gigabit-ethernet ingress-buffer run show interface gigabit-ethernet egress-buffer 4.3 Configuring PFC Watchdog Priority Flow Control (PFC) is a mechanism used in data center networks to ensure lossless transmission for high-priority traffic by pausing traffic when congestion is detected. While PFC helps in managing traffic congestion, it can potentially lead to a situation known as a PFC deadlock. To address this issue, network devices employ a PFC watchdog mechanism to detect and mitigate PFC deadlocks. 4.3.1 Understanding PFC Deadlock A PFC deadlock occurs when multiple devices in a network are continuously sending PFC pause frames to each other, leading to a situation where traffic is indefinitely paused, causing a complete halt in data transmission. This deadlock can severely impact network performance and application availability. 4.3.2 PFC Watchdog Mechanism The PFC watchdog is designed to detect and resolve PFC deadlocks. It monitors the duration of PFC pause frames and takes corrective actions if a potential deadlock is detected. 1. PFC Deadlock Detection and Recovery By configuring the PFC deadlock detection function, the device can periodically check if it is in a PFC deadlock state. When the device detects a PFC deadlock, it will automatically resolve the deadlock within the recovery period. The system will resume sending traffic for the corresponding priority queue, or it can configure to discard the traffic for the corresponding priority queue. After the recovery period, the normal PFC flow control mechanism will be restored. If a deadlock is detected again during the next detection cycle, a new cycle of deadlock recovery procedures will be initiated. 2. PFC Deadlock Control Process If the above deadlock recovery procedures are ineffective and PFC deadlocks continue to occur, users can configure the system to forcibly enter the deadlock control process after a certain number of deadlocks within a specified period. For example, if PFC deadlocks are triggered a certain number of times within a set period, indicating a high risk of frequent deadlocks in the network, the system will enter the deadlock control process. At this point, the device will automatically disable the PFC function to ensure normal packet forwarding and to clear the deadlock state. After the PFC deadlock state is resolved, users have to restore the PFC deadlock detection function manually. Restoring the PFC deadlock detection function reactivates the PFC feature. 4.3.3 Key Components and Functions of PFC Watchdog 1. Monitoring PFC Pause Frames: The PFC watchdog continuously monitors the network for PFC pause frames. It keeps track of the duration for which these frames are active for each priority class on each port. The following command can be used to enable PFC watchdog functionality: set class-of-service interface pfc-watchdog code-point enable 2. Timeout Threshold: A configurable timeout threshold is set for PFC pause frames. If a pause frame for a specific priority exceeds this threshold, it indicates a potential deadlock situation. The detect timer can be configured by the following commands: set class-of-service pfc-watchdog granularity <10 | 100> set class-of-service pfc-watchdog code-point detect-interval 3. Deadlock Detection: When the duration of a PFC pause frame surpasses the timeout threshold, the PFC watchdog triggers a deadlock detection process. This process identifies the ports and priority classes involved in the deadlock. 4. Restore Actions: Once a potential deadlock is detected, the PFC watchdog takes restore actions to break the deadlock. These actions typically include: Forward: During the PFC deadlock recovery process, the received PFC PAUSE frame will be ignored, and the internal scheduler will resume forwarding the traffic. Drop: Drops received data packets. After a predefined period, the PFC watchdog re-enables PFC for the affected priority class. This period allows the network to stabilize and clear any residual congestion that could cause another deadlock. The restore action can be configured by the following commands: set class-of-service pfc-watchdog restore-action set class-of-service pfc-watchdog code-point restore-interval 5. Set the Recovery Mode for a Deadlocked Port The following command can be used to configure the restore mode, the default is automatic recovery. set class-of-service interface pfc-watchdog restore-mode The two different restore modes Manual and Auto represent different deadlock detection processes and different PFC deadlock recovery methods: Auto When PFC watchdog functionality is enabled and the restore mode for a port is set to Auto recovery, the PFC watchdog continuously monitors the PFC activity on the port. If deadlocks repeatedly occur and the count exceeds the configured threshold within the specified time period, the system determines that the port is in an unstable state and will automatically disable the PFC feature to prevent further network disruption. Once PFC is disabled, the port will no longer use PFC to manage congestion until it is manually reset by using the following command: run clear class-of-service interface pfc-watchdog auto Manual When PFC watchdog functionality is enabled and the restore mode for a port is set to Manual recovery, the PFC watchdog continuously monitors the PFC activity on the port, once a PFC deadlock occurs, the PFC function on that port will be automatically disabled. Once PFC is disabled, the port will no longer use PFC to manage congestion until it is manually reset by using the following command: run clear class-of-service interface pfc-watchdog manual This ensures that persistent deadlocks do not degrade network performance. 6. Set the Maximum Number of PFC Deadlocks within a Specified Period When the recovery mode for a port is set to automatic recovery, and the number of PFC deadlocks reaches the upper limit within the specified period, the PFC function on that port will be disabled. In this case, users need to do step 7 to re-enable PFC function. set class-of-service pfc-watchdog threshold period set class-of-service pfc-watchdog threshold count 7. Re-enabling PFC and PFC Watchdog • When a port experiences a deadlock and the recovery mode is set to manual, this command needs to be run to re-enable the PFC function: run clear class-of-service interface pfc-watchdog manual • When the deadlock limit is configured and the port deadlock reaches the upper limit, the reset command needs to be executed to re-enable the PFC function: run clear class-of-service interface pfc-watchdog auto 4.3.4 Restrictions and Guidelines When you configure PFC watchdog, follow these restrictions and guidelines: PFC should be enabled on the interface before enabling PFC watchdog. PFC watchdog is only supported on Trident3-X5, Trident3-X7, Trident4-X9, Trident4-X11, Tomahawk2, Tomahawk3, Tomahawk4, Tomahawk5 platforms. 4.3.5 Configuring PFC Watchdog 1. Procedure Step 1 Enable PFC on the interface before enabling PFC watchdog. set class-of-service pfc-profile [code-point drop ] set class-of-service interface pfc-profile Step 2 Enable PFC watchdog. By default, PFC watchdog is disabled. set class-of-service interface pfc-watchdog code-point enable Step 3 (Optional) Configure the time interval of PFC deadlock detection. The default detection timer is 1.5 seconds. The value of detection time = granularity x detect-interval. set class-of-service pfc-watchdog granularity <10 | 100> set class-of-service pfc-watchdog code-point detect-interval (On Trident3 platforms) set class-of-service interface pfc-watchdog code-point detect-interval (On Tomahawk3 platforms) Step 4 (Optional) Configure the restore time and restore action when PFC deadlock occurs. The default restore action is forward. The restore time = granularity x restore-interval. set class-of-service pfc-watchdog restore-action set class-of-service pfc-watchdog code-point restore-interval (On Trident3 platforms) set class-of-service interface pfc-watchdog code-point restore-interval (On Tomahawk3 platforms) Step 5 (Optional) Set the restore mode for a deadlocked port. If this command is not configured, the default is automatic recovery. set class-of-service interface pfc-watchdog restore-mode Step 6 (Optional) Set the maximum number of PFC deadlocks within a specified period. When the recovery mode for a port is set to automatic recovery, and the number of PFC deadlocks reaches the upper limit within the specified period, the PFC function on that port will be disabled. In this case, users need to do step 8 to re-enable PFC function. set class-of-service pfc-watchdog threshold period set class-of-service pfc-watchdog threshold count Step 7 Commit the configuration. commit Step 8 (Optional) Re-enable PFC and PFC watchdog. When a port experiences a deadlock and the recovery mode is set to manual, this command needs to be run to re-enable the PFC function: run clear class-of-service interface pfc-watchdog manual When the deadlock limit is configured and the port deadlock reaches the upper limit, the reset command needs to be executed to re-enable the PFC function: run clear class-of-service interface pfc-watchdog auto 4.3.6 Verifying the Configuration After the configuration, use command run show pfc-watchdog config to view the configuration information about PFC watchdog. admin@PICOS# run show pfc-watchdog config PORT ACTION QUEUE DETECTION TIME RESTORATION TIME ---------- ----------- ------------ ---------------- ------------------ te-1/1/25 drop 5 150 150 6 150 150 7 120 110 Use command run show pfc-watchdog stats to view the statistics information about PFC watchdog, including the number of PFC pause storms that have been detected and restored, as well as the number of packets that have been dropped, on the PFC queues on an interface. admin@PICOS# run show pfc-watchdog stats QUEUE STATUS STORM DETECTED/RESTORED TX OK/DROP TX LAST OK/DROP ------------ ----------- ------------------------- ---------------- ----------------- te-1/1/25:5 stormed 9/8 82072626556/0 32053822365/0 te-1/1/25:6 stormed 9/8 31504345475/0 32053822365/0 te-1/1/25:7 operational 0/0 0/0 0/0 In the show result, STATUS: The status of PFC watchdog. The value could be operational or stormed. operational: Currently under detection, no deadlock found. stormed: Currently in a deadlock state. STORM DETECTED: Queue deadlock counter. STORM RESTORED: Queue restore counter. TX DROP and TX LAST DROP: Number of Tx packets dropped due to PFC deadlock. TX OK and TX LAST OK: Number of Tx packets transmitted during deadlock (Forward action). 4.4 Configuring PFC Deadlock Prevention 4.4.1 Overview Consider a data center running RoCE (RDMA over Converged Ethernet) traffic for high-performance computing workloads. These workloads require low-latency, lossless traffic flow, which PFC is used to enforce. However, as network congestion builds up, PFC pause frames are triggered, potentially leading to a deadlock if multiple paths become blocked. By employing a PFC deadlock prevention solution, the network can identify RoCE flows that are prone to triggering deadlocks. The solution adjusts the queue priorities so that other critical flows are not blocked, and it reduces the load on congested paths. This prevents the generation of circular wait conditions and ensures the smooth operation of high-priority traffic, ensuring that business-critical applications continue to function smoothly without being interrupted by PFC-induced deadlocks. How PFC Deadlock Prevention Works in Practice 1. Monitoring and Analytics Figure 1. PFC Hook Flows image.png Figure 1 shows a CLOS network, it is a highly scalable and high-performance switching network commonly used in modern data centers. It is a multi-stage network topology, typically with multiple leaf and spine switches, which is designed to handle massive amounts of data with minimal latency and is often used to interconnect thousands of servers. Usually, PFC is deployed to manage flow control and avoid packet loss. PFC Uplink Port Group As shown in Figure 1, interfaces Te-1/1/1 and Te-1/1/2 are the uplinks connecting Leaf2 to the spines. They are added to the PFC uplink port group so that the system treats them collectively, allowing the device to manage them as a single entity when assessing traffic flow and PFC behavior. High-Risk Hook Flow As shown in Figure 1, when a leaf device detects that the same business flow (i.e., a specific set of traffic identified by its characteristics, such as source/destination IP, port, etc.) is traversing multiple interfaces within the PFC uplink port group, it marks this flow as a high-risk hook flow. When a high-risk hook flow generates congestion across multiple interfaces (uplinks), PFC pause frames may be issued by the leaf to its upstream spine switches. If both interfaces in the uplink group send pause frames, and the upstream spine switches are also congested, it can result in a circular wait scenario (deadlock). The switches are effectively waiting for each other to release the paused traffic, leading to a network stall. The Deadlock Prevention solution proactively monitors the data center network for high-risk hook flow that may lead to the generation of PFC pause frames. 2. Dynamic Queue Management After the device receives the packet, it modifies the DSCP value and the corresponding dot1p priority of the packet, so that the packet is forwarded in the new dot1p priority queue using the new DSCP value. The PFC deadlock prevention function in CLOS networks works by creating PFC uplink port groups that combine uplink interfaces on leaf devices together. The system detects high-risk flows that traverse these grouped uplinks and identifies them as potential deadlock triggers (high-risk hook flows). By preemptively modifying queue priorities and managing these flows, the system prevents deadlocks from occurring, ensuring the stability and efficiency of data center networks. 4.4.2 Restrictions and Guidelines When you configure PFC deadlock prevention, follow these restrictions and guidelines: PFC Deadlock Prevention is only supported on Trident3-X5, Trident3-X7, Trident4-X9, Trident4-X11, Tomahawk2, Tomahawk3, Tomahawk4, Tomahawk5 platforms. To ensure proper functioning, it is important that if any Equal-Cost Multi-Path (ECMP) output interfaces exist within the PFC uplink port group, all of these ECMP interfaces must be included in the group. Failing to do so may result in incorrect queue switching for Layer 3 traffic on the PFC uplink port group interfaces, leading to unexpected modifications of the DSCP (Differentiated Services Code Point) values. This could impair traffic handling and potentially lead to inefficiencies or incorrect prioritization. Each device supports only one PFC uplink port group. 4.4.3 Configuring PFC Deadlock Prevention 1. Procedure Step 1 Create a PFC uplink port group. set class-of-service interface pfc-uplink-group Step 2 Modify the queue priority of hook flow packets that match the PFC uplink port group and the original DSCP value. set class-of-service pfc-uplink-group original-dscp to-code-point Step 3 Modify the queue priority of hook flow packets that match the PFC uplink port group and the original DSCP value. If this command is not configured, it means the DSCP value carried by the packets will not be adjusted. NOTE: When configuring on the Trident3-X5 and Trident3-X7 platforms, the configurations in step 2 and step 3 both need to be configured and submitted in the same commit. set class-of-service pfc-uplink-group original-dscp dscp Step 4 Commit the configuration. commit 2. Configuration Example The following commands complete the configurations: Create a PFC uplink port group group1. Modify the queue priority to 4 and DSCP value to 48 of the hook flow packets that match the PFC uplink port group group1 and the original DSCP value 32. admin@PICOS# set class-of-service interface te-1/1/1 pfc-uplink-group group1 admin@PICOS# set class-of-service interface te-1/1/2 pfc-uplink-group group1 admin@PICOS# set class-of-service pfc-uplink-group group1 original-dscp 32 to-code-point 4 admin@PICOS# set class-of-service pfc-uplink-group group1 original-dscp 32 dscp 48 admin@PICOS# commit 5. Configuring Explicit Congestion Notification (ECN) 5.1 Overview Explicit Congestion Notification (ECN) is a network protocol feature that allows end-to-end notification of network congestion without dropping packets. It is an extension of the IP and TCP protocols, enhancing the way congestion is managed by marking packets instead of discarding them. 5.1.1 How ECN Works ECN works by marking packets instead of dropping them when network devices detect congestion. This approach allows the sender to reduce its transmission rate proactively, improving overall network performance and reducing packet loss. 5.1.2 Key Concepts ECN-Capable Transport (ECT): Indicates that the endpoints are ECN-aware and can handle ECN markings. Congestion Experienced (CE): Indicates that the network is experiencing congestion. TCP ECN Echo (ECE): A TCP flag used to notify the sender that the receiver has received a packet marked with CE. Congestion Window Reduced (CWR): A TCP flag used by the sender to indicate that it has reduced its congestion window in response to receiving an ECE flag. 5.1.3 ECN Field in the IP Header ECN uses two bits in the IP header, known as the ECN field, to signal congestion information: 00: Not ECN-Capable Transport (Non-ECT) 01: ECN Capable Transport (ECT(1)) 10: ECN Capable Transport (ECT(0)) 11: Congestion Experienced (CE) 5.1.4 ECN Operation Negotiation: During the TCP three-way handshake, both endpoints negotiate the use of ECN. If both support ECN, they set the appropriate flags in their SYN packets. Packet Marking: When an ECN-capable router detects congestion, it marks packets instead of dropping them. It sets the ECN field in the IP header to CE (11). Receiver Notification: The receiver detects the CE marking and sets the ECE flag in the TCP header of its acknowledgment (ACK) packets. Sender Adjustment: Upon receiving an ACK with the ECE flag, the sender reduces its transmission rate and sets the CWR flag in subsequent packets to indicate that it has responded to the congestion notification. Therefore, it is necessary to reasonably set the ECN thresholds so that the buffer space between the ECN thresholds and PFC thresholds can accommodate the traffic sent during the time after the ECN congestion marking and before the source end slows down. This helps to avoid triggering PFC flow control as much as possible. 5.1.5 Advantages of ECN Reduced Packet Loss: By marking packets instead of dropping them, ECN helps in avoiding packet loss, which is particularly beneficial for applications sensitive to packet drops, such as real-time video or voice. Improved Network Efficiency: ECN allows for more efficient use of network resources by preventing congestion before it becomes severe enough to cause packet drops. Better Performance: By avoiding packet loss, ECN can lead to better overall performance for TCP connections, resulting in higher throughput and lower latency. Smooth Traffic Flow: ECN provides a mechanism for more graceful handling of congestion, leading to smoother traffic flow and improved end-user experience. 5.2 Restrictions and Guidelines When you configure ECN, follow these restrictions and guidelines: To use ECN, both the network devices (such as routers and switches) and the endpoints (such as servers and clients) must support ECN. 5.3 Configuring ECN via WRED On PicOS® switches, ECN is used in conjunction with Weighted Random Early Detection (WRED) to provide early signals of congestion to avoid packet loss and improve network performance. WRED is an active queue management (AQM) mechanism that selectively drops packets based on the average queue length, helping to manage congestion before the queue becomes full. When combined with ECN, WRED can mark packets instead of dropping them when congestion is detected. 5.3.1 Procedure Step 1 Enable WRED. set interface gigabit-ethernet wred queue enable Step 2 Set the maximum and minimum thresholds. set interface gigabit-ethernet wred queue max_thresh set interface gigabit-ethernet wred queue min_thresh Step 3 Set drop probability. set interface gigabit-ethernet wred queue drop_probability Step 4 Enable ECN. set interface gigabit-ethernet wred queue ecn_thresh Step 5 Commit the configuration. commit Key Configuration Parameters Min_Thresh: The minimum average queue size at which WRED starts to mark packets. Max_Thresh: The maximum average queue size at which WRED starts to drop packets with the configured Drop_Probability. Drop_Probability: The probability that a packet will be marked with ECN when the average queue size is between the min and max thresholds. 5.3.2 Verify Configuration Use the command run show interface gigabit-ethernet wred to check the settings of ECN and WRED . 5.3.3 Monitor and Adjust Monitor the network to ensure that ECN marking and WRED are effectively managing congestion. Adjust the thresholds and probabilities as needed to optimize performance. 5.4 Configuration Example The following commands complete the configurations: Enable WRED on queue 0 of interface ge-1/1/3; Set the maximum threshold to 400 and the minimum threshold to 200 on queue 0 of interface ge-1/1/3; Set the drop probability to 50% on queue 0 of interface ge-1/1/3; Enable ECN (Explicit Congestion Notification) on queue 0 of interface ge-1/1/3. admin@PICOS# set interface gigabit-ethernet ge-1/1/3 wred queue 0 enable true admin@PICOS# set interface gigabit-ethernet ge-1/1/3 wred queue 0 max_thresh 400 admin@PICOS# set interface gigabit-ethernet ge-1/1/3 wred queue 0 min_thresh 200 admin@PICOS# set interface gigabit-ethernet ge-1/1/3 wred queue 0 drop_probability 50 admin@PICOS# set interface gigabit-ethernet ge-1/1/3 wred queue 0 ecn_thresh 1 admin@PICOS# commit Show the WRED information of the specified interface. admin@PICOS# run show interface gigabit-ethernet ge-1/1/3 wred Queue Num Min Thresh Max Thresh Drop Probability ECN Thresh Status ---------- ---------- ---------- ---------------- ----------- ---------- 0 200 400 50% Enabled Enabled 1 0 0 0% Disabled Disabled 2 0 0 0% Disabled Disabled 3 0 0 0% Disabled Disabled 4 0 0 0% Disabled Disabled 5 0 0 0% Disabled Disabled 6 0 0 0% Disabled Disabled 7 0 0 0% Disabled Disabled 5.5 Test Recommended Configuration Configuring ECN thresholds properly is crucial for ensuring optimal network performance, which helps in managing congestion proactively, leading to improved network performance, reduced packet loss, and lower latency.Testing recommended configurations involves setting specific parameters for ECN to manage congestion effectively and then evaluating the performance outcomes. In the Lab, we use network simulation tools to create a controlled environment for testing. Simulate various traffic patterns and congestion scenarios to test how the network handles congestion. The table below provides recommended configuration values to prevent packet loss and avoid triggering the PFC threshold. Test Method Switch Port MTU Switch Egress Port Link Bandwidth ECN Min Threshold (Bytes) ECN Max Threshold (Bytes) Probability image.png Three Ingress Port to One Egress Port 9000 99.75G 80000 8000000 80 image.png Three Ingress Port to One Egress Port 4096 97.68G 1000000 30000000 50 image.png Three Ingress Port to One Egress Port 9000 99.75G 1000000 32000000 50 Users have to continuously monitor network performance and ECN marking rates. Make dynamic adjustments to the ECN thresholds as needed to respond to changing network conditions. 6. Configuring Easy ECN 6.1 Overview The Easy ECN feature streamlines the traditionally complex configuration of Explicit Congestion Notification (ECN) by eliminating the need for manual setup of interface queues, WRED (Weighted Random Early Detection) policies, thresholds, and drop probabilities. In standard ECN configurations, users need to configure multiple parameters, such as: Enabling ECN on specific interface queues, ensuring that the network is aware of which traffic classes should use ECN. Setting WRED thresholds, which determine at what point packets will be marked or dropped based on queue depth. Configuring the maximum packet drop probability, dictating how likely a packet is to be dropped when congestion is high. Easy ECN simplifies all of these by allowing users to configure ECN with just a single command, focusing on the network’s traffic optimization strategy. By using the command set class-of-service easy-ecn mode , network administrators can efficiently configure ECN with a focus on either throughput or latency globally for all the interface queues, without needing to manually set all the individual parameters for each interface queue. This simplified approach not only saves time but also ensures more consistent and effective congestion management. 1. Throughput-first Mode: This mode prioritizes maximizing the amount of data transferred across the network, making it ideal for use cases such as bulk data transfers or content delivery systems. It dynamically adjusts traffic flow to ensure that congestion management does not overly restrict throughput, while still using ECN to prevent excessive congestion. In this mode, the threshold for marking packets is set relatively high. This allows for larger buffers to accumulate before marking occurs, reducing the chances of packet drops and enabling more traffic to be handled. 2. Latency-first Mode: This mode prioritizes minimizing latency, ensuring that packets experience the least possible delay. This is particularly important for real-time applications such as VoIP, video conferencing, or online gaming, where even small delays can negatively impact performance. In this mode, the ECN marking threshold is set lower, meaning packets will be marked sooner when congestion starts to build up. This results in less buffer buildup, thus reducing queuing delays and minimizing latency. Table 1 below shows ECN threshold and marking probability values for throughput-first and latency-first modes on Trident3-X5, Trident3-X7, Trident4-X9, Trident4-X11, Tomahawk2, Tomahawk3, Tomahawk4, Tomahawk5 platforms. Table 1. ECN Threshold and Marking Probability: Throughput-First vs. Latency-First Easy ECN Mode ECN Threshold and Marking Probability Tomahawk3 Trident3-X5 Trident3-X7 Tomahawk2 Trident4-X9 Trident4-X11 Tomahawk4 Tomahawk5 throughput-first min_thresh (Bytes) 50800 25600 41600 31800 50800 max_thresh (Bytes) 254000 128000 208000 159000 254000 drop_probability 25% 25% 25% 25% 25% latency-first min_thresh (Bytes) 5080 2560 4160 3180 5080 max_thresh (Bytes) 25400 12800 20800 15900 25400 drop_probability 25% 25% 25% 25% 25% To simplify the configuration, users can enable Easy ECN for global congestion management. However, if the global configuration does not fully address the specific needs of queue-level congestion management, standard ECN can be fine-tuned for individual interface queues. NOTE: When both global configuration and user-specified interface queue settings are applied, the settings for the user-specified interface queue will take effect. 6.2 Configuration Notes and Constraints When configuring Easy ECN, consider the following points: Easy ECN is only supported on Trident3-X5, Trident3-X7, Trident4-X9, Trident4-X11, Tomahawk2, Tomahawk3, Tomahawk4, Tomahawk5 platforms. Throughput-First mode prioritizes high data transfer rates. It is ideal for environments where bulk data transmission is critical. Latency-First mode focuses on minimizing delays in packet transmission, making it suitable for real-time applications (e.g., VoIP, video streaming). Easy ECN provides a simplified, global configuration for congestion management. If more granular control is required, standard ECN can be configured on specific interface queues. In such cases, interface-specific ECN configurations override global settings. 6.3 Configuring Easy ECN To simplify the configuration, users can enable Easy ECN for global congestion management. However, if the global configuration does not fully address the specific needs of queue-level congestion management, standard ECN can be fine-tuned for individual interface queues. The following command can be used to enable easy ECN globally. set class-of-service easy-ecn mode Example Configure easy ECN mode to latency-first. admin@PICOS# set class-of-service easy-ecn mode latency-first admin@PICOS# commit Configure easy ECN mode to throughput-first. admin@PICOS# set class-of-service easy-ecn mode throughput-first admin@PICOS# commit 7. PFC and ECN Statistical Reporting through gRPC In modern network systems, efficient congestion management and data flow control are crucial to maintain high-performance levels, especially in data centers and cloud environments. PFC (Priority Flow Control) and ECN (Explicit Congestion Notification) are two complementary mechanisms that help manage traffic and congestion, while gRPC (Google Remote Procedure Call) facilitates real-time communication and reporting between network components. PFC and ECN work together to handle congestion in a network. PFC ensures that critical traffic continues to flow by pausing lower-priority traffic, while ECN marks packets to indicate congestion. This cooperation minimizes packet loss and optimizes network performance. As PFC and ECN manage traffic, they generate valuable statistics such as the number of paused frames (PFC) and the number of ECN-marked packets. These statistics are essential for understanding network health and congestion levels. gRPC is used to gather and transmit these statistics to a central server. Because gRPC is designed for high-performance, low-latency communication, it is well-suited to handle the real-time transmission of PFC and ECN data. The statistical data can be analyzed to adjust network policies, improve performance, and prevent future congestion. 7.1 PFC and ECN Statistics PFC and ECN, in conjunction with gRPC, can provide PFC pause frame counts, PFC deadlock monitoring and ECN-marked packet counts for statistical queries. The statistics that PFC supports reporting include: Number of PFC pause frames sent Number of PFC pause frames received Number of PFC deadlock monitoring instances Number of PFC deadlock recovery instances Rate of PFC pause frames received Rate of PFC pause frames sent The statistics that ECN supports reporting include: Number of ECN marked packets Rate of ECN marked packets You can view the PFC and ECN statistics through run show commands. For details, see PFC Commands and run show class-of-service ecn statistics. 7.2 Configuration Procedure Configure PFC on your network devices. Refer to Configuring Priority-based Flow Control (PFC) for details about how to configure PFC. Enable ECN to help manage congestion. Refer to Configuring Explicit Congestion Notification (ECN) for details about how to enable ECN. Deploy gRPC servers and clients to handle the statistical query requests. Refer to Configuring gNMI-gRPC Based Telemetry Technology for details about how to deploy gRPC. This setup ensures you have real-time monitoring and statistical analysis capabilities for PFC and ECN, which can significantly aid in preventing network performance issues due to congestion and deadlocks. 8. Configuring Dynamic Load Balancing 8.1 Overview Traditional static load balancing does not consider the utilization of each member link within the load-balancing group, leading to uneven load distribution among the member links. This issue becomes more pronounced with the appearance of large data flows, exacerbating congestion on the selected member link and potentially causing packet loss. By enabling dynamic load balancing, the traffic of Equal-Cost Multi-Path (ECMP) routing can be distributed across different member links through dynamic load balancing, maximizing load balancing among the member links. 8.2 Implementation Principle PicOS® supports three modes of Dynamic load balancing: Normal Mode, Optimal Mode and Assigned Mode. 8.2.1 Normal Mode There is link transmission delay between two directly connected devices. If the time interval between two data packets to be sent is greater than the maximum link transmission delay of the member links in the load-balancing group, these two packets can be forwarded using different member links without causing out-of-order delivery at the receiving end. Normal mode dynamic load balancing is based on this principle. In Normal mode, the device determines the time interval between the packet to be forwarded and the previous packet in its flow. If the interval is greater than the maximum link transmission delay of the member links, the packet to be forwarded is considered the first packet of a new flowset. If the interval is less than the maximum link transmission delay, the packet is considered part of the same flowset as the previous packet. The device forwards the packets based on the flowset, selecting the member link with the lighter load for forwarding. Packets within the same flowset are forwarded using the same member link. 8.2.2 Optimal Mode The Optimal mode of dynamic load balancing operates on a per-packet distribution system, where each packet is forwarded independently based on the real-time load conditions of the available links. This mechanism allows the device to select the least congested or lightly loaded link at any given moment, distributing the packets dynamically to optimize network performance. However, this method introduces a potential issue when packets belonging to the same flow are forwarded through different links. If the time interval between two consecutively sent packets is shorter than the maximum transmission delay across the links, the packets may arrive out of order at the receiving end. This happens because different links may have varying transmission delays, causing later-sent packets to arrive before earlier ones. For this reason, Optimal mode can lead to out-of-order packet delivery, particularly in scenarios where flows are split across multiple paths. Out-of-order packets can disrupt certain types of network traffic, especially those that rely on sequence consistency, such as real-time voice or video communication. To mitigate this, it is crucial that the receiving device or terminal has the capability to reorder packets properly. Devices must be equipped with reassembly buffers or out-of-order packet handling mechanisms to ensure the integrity of the flow, allowing the receiving system to reorder the packets correctly before processing them. Optimal mode, while improving load distribution and avoiding bottlenecks, requires robust support on the receiving end to handle these potential sequencing issues effectively. 8.2.3 Assigned Mode In Assigned mode dynamic load balancing, the system ensures that each packet within a specific flow follows a consistent path by using the same forwarding link as the previous packet in that flow. This approach minimizes the risk of out-of-order packet delivery by keeping the traffic for the flow on the same route. However, when a flow starts and the first packet is sent, the system cannot rely on a previous packet’s path. In this case, a static load balancing mechanism comes into play, where a hash-based algorithm determines the member link to forward the packet. The hash function typically considers factors such as source and destination IP addresses or port numbers to ensure an even distribution of traffic across the available links while maintaining flow consistency. Once the path is set for the first packet, subsequent packets in the same flow will continue to follow that route until the flow is completed. 8.3 Restrictions and Guidelines When you configure Dynamic Load Balancing, follow these restrictions and guidelines: Dynamic Load Balancing for ECMP is only supported on Trident3-X5, Trident3-X7, Trident4-X9, Trident4-X11, Tomahawk2, Tomahawk3, Tomahawk4, Tomahawk5 platforms. When the ECMP hash-mapping mode configuration (set/delete) changes, a system restart is required for the configuration to take effect. ECMP hash-mapping modes (round-robin-load-balancing, randomized-load-balancing, symmetric, resilient-load-balancing, dlb-normal, dlb-optimal and dlb-assigned) are mutually exclusive. To switch between modes, you must first delete the configured mode before setting up the new one. Then, restart the system for the configuration to take effect. 8.4 Configuring Dynamic Load Balancing 8.4.1 Procedure Step 1 Enable one of the three modes of Dynamic load balancing for ECMP: Normal mode, Optimal mode and Assigned mode. set interface ecmp hash-mapping dlb-normal [flowset-time ] set interface ecmp hash-mapping dlb-optimal set interface ecmp hash-mapping dlb-assigned By default, Dynamic Load Balancing for ECMP is disabled. Step 2 Enable IP routing function when using dynamic load balancing. set ip routing enable true Step 3 Commit the configuration. commit Step 4 Under CLI operational mode (prompted with “>“), run the following command to restart the system for the configuration to take effect. request system reboot 8.4.2 Configuration Example The following commands enable normal mode of Dynamic Load Balancing for ECMP. admin@PICOS# set interface ecmp hash-mapping dlb-normal admin@PICOS# set ip routing enable true admin@PICOS# commit admin@PICOS# exit admin@PICOS> request system reboot The following commands change Dynamic Load Balancing normal mode to optimal mode for ECMP. admin@PICOS# delete interface ecmp hash-mapping dlb-normal admin@PICOS# commit admin@PICOS# set interface ecmp hash-mapping dlb-optimal admin@PICOS# commit admin@PICOS# exit admin@PICOS> request system reboot 8.4.3 Verify the Configuration Use the command run show route ipv4 to view the ECMP routes. In the following show result, we can see that the destination address 10.20.51.0/24 is reached via three different next hops: 192.168.0.1, 192.168.1.1, and 192.168.2.1, forming ECMP routes. admin@PICOS# run show route ipv4 Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, T - Table, D - SHARP, F - PBR, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure K * 0.0.0.0/0 [255/8192] unreachable (blackhole), 23:42:36 K>* 0.0.0.0/0 [0/2] via 10.10.51.1, eth0, 23:42:36 C>* 10.10.51.0/24 is directly connected, eth0, 23:42:36 S>* 10.20.51.0/24 [1/0] via 192.168.0.1, vlan10, weight 1, 00:09:06 * via 192.168.1.1, vlan20, weight 1, 00:09:06 * via 192.168.2.1, vlan30, weight 1, 00:09:06 C>* 192.168.0.0/24 is directly connected, vlan10, 00:09:09 C>* 192.168.1.0/24 is directly connected, vlan20, 00:09:09 C>* 192.168.2.0/24 is directly connected, vlan30, 00:09:09 Use the command run show interface gigabit-ethernet to view the traffic distribution for each interface. Here is an example: admin@PICOS# run show interface gigabit-ethernet ge-1/1/5 Physical interface: ge-1/1/5, Enabled, error-discard False, Physical link is Up Interface index: 5, Mac Learning Enabled Port mode: trunk Description: Link-level type: Ethernet, MTU: 1518, Speed: 2.5Gb/s, Duplex: Full Source filtering: Disabled, Flow control: Disabled Auto-negotiation: Enabled, Advertised speed modes: 10M,100M,1G,2.5G Interface flags: SNMP-Traps Internal: 0x0 Interface rate limit ingress: unlimited, egress: unlimited Interface burst limit ingress: unlimited, egress: unlimited Precision Time Protocol mode: none Current address: 1c:72:1d:c9:1b:e1, Hardware address: 1c:72:1d:c9:1b:e1 Traffic statistics: 5 sec input rate 5440 bits/sec, 1 packets/sec 5 sec output rate 0 bits/sec, 0 packets/sec Input Packets............................12235 Output Packets...........................135 Input Octets.............................3943964 Output Octets............................14660 9. Configuring RoCE EasyDeploy 9.1 Overview RoCE EasyDeploy is a feature designed to simplify the deployment and configuration of RoCE (RDMA over Converged Ethernet) on switches, enabling seamless integration with servers for optimized network performance. This feature allows users to easily select and switch between lossless and lossy modes, ensuring the best performance for different network environments. Key Features and Application Advantages Rapid Deployment with Minimal Configuration: RoCE EasyDeploy requires minimal configuration steps to enable quick deployment of RoCE across all or specific interfaces, reducing the complexity of setting up RoCE. With just a few commands, users can enable lossless or lossy mode without manually fine-tuning multiple QoS, PFC, and ECN parameters. Seamless Mode Switching for Different Workloads: Supports both Lossless Mode (PFC & ECN-enabled, for zero packet loss, ideal for critical, latency-sensitive applications) and Lossy Mode (ECN-based, for environments where packet loss is acceptable for reduced latency) to meet varying network demands. Users can switch between modes dynamically, optimizing traffic handling based on application needs. Optimized Default Parameters: By default, RoCE EasyDeploy applies recommended QoS settings, WRED (Weighted Random Early Detection) and buffer configurations based on the switch model, eliminating the need for manual tuning while ensuring optimal performance. Granular Interface-Level Control: Users can apply RoCE settings globally to all interfaces or selectively configure specific interfaces, providing greater flexibility in network management. Enhanced Network Stability and Reliability: Ensures consistent RoCE performance across network devices, minimizing configuration errors and reducing troubleshooting time. Supports post-deployment adjustments for fine-tuning PFC, ECN, and QoS settings without disrupting active traffic. 9.2 Restrictions and Guidelines When you configure RoCE EasyDeploy, follow these restrictions and guidelines: RoCE EasyDeploy is supported on Trident3-X5, Trident3-X7, Trident4-X9, Trident4-X11, Tomahawk2, Tomahawk3, Tomahawk4, Tomahawk5 platforms. On the switch, ECN and PFC queue number setting should align with the server-side queue number to ensure balanced traffic scheduling and prevent uneven load distribution. Post-deployment fine-tuning of PFC, ECN, and QoS settings is supported and should be dynamically adjusted based on real-time network conditions and traffic patterns to ensure optimal performance and stability. Monitor RoCE statistics to ensure configurations are applied correctly, and adjust QoS and buffer parameters if needed. 9.3 Server Configuration Ensure that the network card is deployed in RoCEv2 mode and configure QoS for RoCE (trust mode as PCP (priority code point) or DSCP), enabling ECN and PFC. Below is an example using a Mellanox network card: Set RDMA CM Work Mode: [root@server ~]# cma_roce_mode -d mlx5_0 -p 1 -m Set NIC QoS Priority Type to DSCP: [root@server ~]# mlnx_qos -i enp1s0f0 --trust=dscp Enable PFC on Queue 3: [root@server ~]# mlnx_qos -i enp1s0f0 -f 0,0,0,1,0,0,0,0 Enable DCQCN on Queue 3: [root@server ~]# echo 1 > /sys/class/net/enp1s0f0/ecn/roce_np/enable/3 Set CNP DSCP: [root@server ~]# echo 48 > /sys/class/net/enp1s0f0/ecn/roce_np/cnp_dscp 9.4 Switch Configuration 9.4.1 Selecting RoCE Mode The switch supports RoCE EasyDeploy deployment with the server and allows switching between lossless and lossy modes. 1. Lossless Mode In lossless mode, the following features and configurations are enabled by default: Enable both PFC and ECN functionality. ECN enables WRED strategy. Enable QoS policy with the following default settings: Egress forwarding-class default: WRR scheduling (for other traffic forwarding), weight 16. forwarding-class roce: WRR scheduling (for RoCE traffic forwarding), weight 16. forwarding-class cnp: SP scheduling (for forwarding CNP packets). Ingress Default trust mode dscp. DSCP maps to local priority. 2. Lossy Mode In lossy mode, the following features and configurations are enabled by default: Enable ECN only. ECN enables WRED strategy. Enable QoS policy with the following default settings: forwarding-class default: WRR scheduling (for other traffic forwarding). forwarding-class roce: WRR scheduling (for RoCE traffic forwarding). forwarding-class cnp: SP scheduling (for forwarding CNP packets). Configuration Command set class-of-service roce mode 9.4.2 Applying Mode to Physical Interfaces and Queue To ensure optimal RoCE performance, the mode should be applied correctly to the physical interfaces: Apply RoCE mode to all physical interfaces or specific interfaces. If the queue is not configured, queue 3 is enabled by default. PFC default buffer ingress queue parameters (threshold/guaranteed/headroom/reset-offset) is configured. WRED default parameters (max-thresh/min-thresh/drop-probability) adapt to different switch chips automatically. Configuration Command set class-of-service roce apply {all | interface } set class-of-service roce queue NOTE: The command set class-of-service roce apply {all | interface } does not allow simultaneous configuration of both all and per-interface settings. To configure one, you must first remove the other. 9.4.3 Adjusting WRED and Buffer Parameters After enabling RoCE mode, users can still modify the PFC, ECN, and QoS configurations using the following commands. 1. ECN Configuration set interface gigabit-ethernet wred queue enable set interface gigabit-ethernet wred queue max_thresh set interface gigabit-ethernet wred queue min_thresh set interface gigabit-ethernet wred queue drop_probability set interface gigabit-ethernet wred queue ecn_thresh 2. Buffer Configuration set interface gigabit-ethernet ethernet-switching-options buffer ingress-queue guaranteed set interface gigabit-ethernet ethernet-switching-options buffer ingress-queue threshold set interface gigabit-ethernet ethernet-switching-options buffer ingress-queue reset-offset set interface gigabit-ethernet ethernet-switching-options buffer ingress-queue headroom 3. QoS Parameter Configuration set class-of-service forwarding-class local-priority set class-of-service scheduler mode set class-of-service classifier in-bond trust-mode set class-of-service classifier in-bond forwarding-class set class-of-service scheduler-profile out-bond forwarding-class scheduler set class-of-service interface classifier set class-of-service interface scheduler-profile 9.4.4 Checking the Default RoCE Configuration After configuring RoCE EasyDeploy, you can use the command run show class-of-service roce to check the default RoCE configuration settings. admin@PICOS# run show class-of-service roce status applied mode lossless congestion-control congestion-mode ECN enabled-queue 3 max-threshold 1500000 bytes min-threshold 150000 bytes probability 100 pfc pfc-priority 3 rx-enabled enabled tx-enabled enabled trust trust-mode dscp RoCE PCP/DSCP->LP mapping configurations =========================================== local-priority dscp ------------- ------------------- 0 0,1,2,3,4,5,6,7 1 8,9,10,11,12,13,14,15 2 16,17,18,19,20,21,22,23 3 24,25,26,27,28,29,30,31 4 32,33,34,35,36,37,38,39 5 40,41,42,43,44,45,46,47 6 48,49,50,51,52,53,54,55 7 56,57,58,59,60,61,62,63 RoCE LP->FC mapping and ETS configurations ============================================= local-priority forwarding-class scheduler-weight -------------- ---------------- ---------------- 0 default WRR-8 1 default WRR-8 2 default WRR-8 3 roce WRR-8 4 default WRR-8 5 default WRR-8 6 cnp SP 7 default WRR-8 9.4.5 Checking RoCE Statistics The command run show class-of-service roce statistics interface can be used to display RoCE interface statistics. The command run clear class-of-service roce statistics interface can be used to clear RoCE statistics. 9.5 Configuration Example The following commands complete the configurations: Configure RoCE mode to lossless. Apply RoCE mode to all the interfaces. admin@PICOS# set class-of-service roce mode lossless admin@PICOS# set class-of-service roce apply all admin@PICOS# commit Verify RoCE configuration. Use the following command to check the default RoCE configuration settings. admin@PICOS# run show class-of-service roce status applied mode lossless congestion-control congestion-mode ECN enabled-queue 3 max-threshold 1500000 bytes min-threshold 150000 bytes probability 100 pfc pfc-priority 3 rx-enabled enabled tx-enabled enabled trust trust-mode dscp RoCE PCP/DSCP->LP mapping configurations =========================================== local-priority dscp ------------ ------------------- 0 0,1,2,3,4,5,6,7 1 8,9,10,11,12,13,14,15 2 16,17,18,19,20,21,22,23 3 24,25,26,27,28,29,30,31 4 32,33,34,35,36,37,38,39 5 40,41,42,43,44,45,46,47 6 48,49,50,51,52,53,54,55 7 56,57,58,59,60,61,62,63 RoCE LP->FC mapping and ETS configurations ============================================= local-priority forwarding-class scheduler-weight -------------- ---------------- ---------------- 0 default WRR-8 1 default WRR-8 2 default WRR-8 3 roce WRR-8 4 default WRR-8 5 default WRR-8 6 cnp SP 7 default WRR-8 (Optional) Adjust WRED and buffer parameters for queue 3 of interface te-1/1/1. After enabling RoCE mode, adjust ECN, PFC, and QoS parameters if needed. admin@PICOS# set interface gigabit-ethernet te-1/1/1 wred queue 3 enable true admin@PICOS# set interface gigabit-ethernet te-1/1/1 wred queue 3 max_thresh 8000 admin@PICOS# set interface gigabit-ethernet te-1/1/1 wred queue 3 min_thresh 4000 admin@PICOS# set interface gigabit-ethernet te-1/1/1 wred queue 3 drop_probability 5 admin@PICOS# set interface gigabit-ethernet te-1/1/1 wred queue 3 ecn_thresh 3000 admin@PICOS# set interface gigabit-ethernet te-1/1/1 ethernet-switching-options buffer ingress-queue 3 guaranteed 2000 admin@PICOS# set interface gigabit-ethernet te-1/1/1 ethernet-switching-options buffer ingress-queue 3 threshold 6000 admin@PICOS# set interface gigabit-ethernet te-1/1/1 ethernet-switching-options buffer ingress-queue 3 reset-offset 1000 admin@PICOS# set interface gigabit-ethernet te-1/1/1 ethernet-switching-options buffer ingress-queue 3 headroom 1500 admin@PICOS# commit 10. Typical Configuration Example of Lossless Network 10.1 Networking Requirements In Figure 1, in the following data center network PoD, two Server Clusters are connected to the network through devices Leaf 1, Leaf 2, Spine 1 and Spine 2. A lossless network is required to apply in the network devices to prevent packet loss during data transmission, ensuring high reliability and performance, especially for applications that are sensitive to latency and data integrity. Setting up a lossless network involves careful planning, consistent configuration across all devices, and continuous monitoring and management. Follow the configuration roadmap below to ensure all components are correctly configured: Configure PFC on all relevant ports to prevent packet loss by pausing traffic during congestion. Employ a PFC watchdog mechanism to detect and mitigate PFC deadlocks. Enable Explicit Congestion Notification (ECN) on all switches and endpoints to signal congestion without dropping packets. Configure ECN thresholds to detect and mark packets early before severe congestion occurs. Enable dynamic load balancing, the traffic of Equal-Cost Multi-Path (ECMP) routing can be distributed across different member links through dynamic load balancing. Figure 1. Typical Configuration Example of Lossless Network image.png 10.2 Procedure The following configuration steps are for Leaf 1. The configurations for Leaf 2, Spine 1 and Spine 2 are similar to Leaf 1 and will not be repeated. Step 1 On Leaf 1, configure PFC on all relevant ports to prevent packet loss by pausing traffic during congestion. The following commands complete the configurations: Configure PFC profile pfc1. Apply PFC profile to the port te-1/1/1. admin@PICOS# set class-of-service pfc-profile pfc1 admin@PICOS#set class-of-service interface te-1/1/1 pfc-profile pfc1 admin@PICOS# commit Show the class of service statistics information on specified interface. The class 0~7 in PFC frame corresponds to the following "802.1P" item. The value of ”RxPFC“ item will be incremented by 1 if te-1/1/1 receives a PFC frame. The value of ”TxPFC“ item will be incremented by 1 if te-1/1/1 sends out a PFC frame. admin@PICOS# run show class-of-service interface te-1/1/1 Interface : te-1/1/1 802.1P Priority Flow Control RxPFC TxPFC ----------- --------------------- --------------- --------------- 0 true 0 500 1 true 0 0 2 true 0 71 3 true 0 0 4 true 0 0 5 true 0 0 6 true 0 102 7 true 0 0 trust mode : ieee-802.1 Default ieee-802.1 : 0 Default dscp : 0 Default inet-precedence : 0 Local-priority Queue-Schedule Code-points -------------- --------------------------- ------------------------- 0 SP,0kbps 1 SP,0kbps 2 SP,0kbps 3 SP,0kbps 4 SP,0kbps 5 SP,0kbps 6 SP,0kbps 7 SP,0kbps Step 2 (Optional) Configure the PFC buffer. After the PFC function is enabled, the default storage space is available for priority queues. You can flexibly make good use of the storage space through configuring buffer thresholds based on certain queues as needed. The following commands complete the configurations: Set the upper threshold of guaranteed buffer for PFC queue 3 on the ingress interface te-1/1/1 as 24000 cells. Set the static threshold of shared buffer for PFC queue 1 on the ingress interface te-1/1/1 as 10000 cells. Set the offset value of shared buffer for PFC queue 1 on the ingress interface te-1/1/1 as 3000 cells. admin@PICOS# set interface gigabit-ethernet te-1/1/1 ethernet-switching-options buffer ingress-queue 3 guaranteed 24000 admin@PICOS# set interface gigabit-ethernet te-1/1/1 ethernet-switching-options buffer ingress-queue 1 threshold 10000 admin@PICOS# set interface gigabit-ethernet te-1/1/1 ethernet-switching-options buffer ingress-queue 1 reset-offset 3000 admin@PICOS# commit Step 3 Configure PFC watchdog. The following commands complete the configurations: Enable PFC watchdog on queue5 of interface te-1/1/1. Configure the time interval of PFC deadlock detection to 10 x 100ms, where 100ms is the default value of PFC deadlock detection timer granularity. Configure the restore time to 10 x 100ms when PFC deadlock occurs, where 100ms is the default value of PFC deadlock restore timer granularity. admin@PICOS# set class-of-service interface te-1/1/1 pfc-watchdog code-point 5 enable true admin@PICOS# set class-of-service pfc-watchdog code-point 5 detect-interval 10 admin@PICOS# set class-of-service pfc-watchdog code-point 5 restore-interval 10 admin@PICOS# commit After the configuration, use command run show pfc-watchdog config to view the configuration information about PFC watchdog. admin@PICOS# run show pfc-watchdog config PORT ACTION QUEUE DETECTION TIME RESTORATION TIME ---------- ----------- ------------ ---------------- ------------------ te-1/1/1 forward 5 1000 1000 Use command run show pfc-watchdog stats to view the statistics information about PFC watchdog, including the number of PFC pause storms that have been detected and restored, as well as the number of packets that have been dropped, on the PFC queues on an interface. admin@PICOS# run show pfc-watchdog stats QUEUE STATUS STORM DETECTED/RESTORED TX OK/DROP TX LAST OK/DROP ------------ ----------- ------------------------- ---------------- ----------------- te-1/1/1:5 stormed 9/8 82072626556/0 32053822365/0 Step 4 Configure ECN. Users have to continuously monitor network performance and ECN marking rates. Make dynamic adjustments to the ECN thresholds as needed to respond to changing network conditions. The following commands complete the configurations: Enable WRED on queue 0 of interface te-1/1/1; Set the maximum threshold to 400 and the minimum threshold to 200 on queue 0 of interface te-1/1/1; Set the drop probability to 50% on queue 0 of interface te-1/1/1; Enable ECN (Explicit Congestion Notification) on queue 0 of interface te-1/1/1. admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 enable true admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 max_thresh 400 admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 min_thresh 200 admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 drop_probability 50 admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 ecn_thresh 1 admin@PICOS# commit Show the WRED information of the specified interface. admin@PICOS# run show interface gigabit-ethernet te-1/1/1 wred Queue Num Min Thresh Max Thresh Drop Probability ECN Thresh Status ---------- ---------- ---------- ---------------- ----------- ---------- 0 200 400 50% Enabled Enabled 1 0 0 0% Disabled Disabled 2 0 0 0% Disabled Disabled 3 0 0 0% Disabled Disabled 4 0 0 0% Disabled Disabled 5 0 0 0% Disabled Disabled 6 0 0 0% Disabled Disabled 7 0 0 0% Disabled Disabled Step 5 Enable normal mode of Dynamic Load Balancing for ECMP. By enabling dynamic load balancing, the traffic of Equal-Cost Multi-Path (ECMP) routing can be distributed across different member links through dynamic load balancing, maximizing load balancing among the member links. admin@PICOS# set interface ecmp hash-mapping dlb-normal admin@PICOS# commit Step 6 Enable the IP routing function. admin@PICOS# set ip routing enable true admin@PICOS# commit

PicOS® Troubleshooting Guide

04-11-2025 - PicOS® Troubleshooting Guide 1. L2/L3 Troubleshooting Guide This guide describes how to identify and resolve common problems related to the PicOS® software used on supported switches. 1.1 Monitoring and Debugging L2/L3 protocols 1.1.1 Find and Configure the Log File By default, the syslog local-file is ram. The log file name is "message" which is in the directory "/tmp/log" admin@XorPlus$cd /tmp/log admin@XorPlus$ls lastlog lighttpd messages wtmp You can use "tail -f /tmp/log/messages" to display the log messages. You can set the syslog local-file location to disk. The log file name is "messages" which is in the directory of "/var/log" admin@XorPlus# set system syslog local-file disk admin@XorPlus# commit Commit OK. Save done. admin@XorPlus# admin@XorPlus$cd /var/log/ admin@XorPlus$ls apt dmesg fsck last_death lastlog messages news ntpstats wtmp admin@XorPlus$ You can use "tail -f /var/log/messages" to show the log messages. 1.1.2 Enable Important Debugs Enable debug interface： ##Global Interface traceoptions. admin@XorPlus# set interface traceoptions flag Possible completions: <[Enter]> Execute this command all Configure all tracing config Configure configuration tracing ethernet-switching-options Configure ethernet-switching-options tracing neighbor-event Configure neighbor event tracing packets Configure received or sent packets event tracing port-security Configure port security tracing raw-packet Configure receive raw packet tracing route-event Configure route event tracing static-ethernet-switching Configure static-ethernet-switching tracing timer Configure timer tracing admin@XorPlus# set interface traceoptions flag config <[Enter]> Execute this command disable Disable configuration tracing admin@XorPlus# set interface traceoptions flag config disable false admin@XorPlus# commit Commit OK. Save done. admin@XorPlus# admin@XorPlus# set interface traceoptions line-card ? Possible completions: <[Enter]> Execute this command statistic Configure line card statistic module trace trace-level Configure line card trace level trace-type Configure line card trace type admin@XorPlus# set interface traceoptions line-card trace-level all disable false admin@XorPlus# commit Commit OK. Enable debug of protocals: admin@Xorplus# set protocols Possible completions: <[Enter]> Execute this command arp Configure ARP bgp Configure BGP inter-domain routing dhcp Dynamic Host Configuration Protocol dot1x 802.1x protocol igmp Configure the IGMP protocol igmp-snooping Configure the igmp snooping lacp Link Aggregation Control Protocol lldp Link Layer Discovery Protocol 802.1AB mlag Configure MLAG neighbour Configure Neighbour Discovery Protocol netconf Configure NETCONF ospf Configure the OSPF protocol ovsdb Enable OVSDB pim PIM protocol sflow Configure sflow snmp Simple network management protocol configuration spanning-tree Configure Spanning Tree static Configure static routes udld Unidirectional Link Detection Protocol vrrp Configure VRRP admin@Xorplus# set protocols bgp traceoption updates in admin@Xorplus# commit Commit OK Save Done! Enable debug of LLDP: ## LLDP global traceoptions. admin@Xorplus# set protocols lldp traceoptions flag Possible completions: <[Enter]> Execute this command all Configure all events and packets tracing configuration Configure configuration tracing message-in Configure received message tracing message-out Configure send message tracing state-change Configure LLDP state change tracing admin@Xorplus# set protocols lldp traceoptions flag message-in disable false admin@XorPlus# commit Commit OK. Enable debug of LACP: ## LACP global traceoptions. admin@Xorplus# set protocols lacp traceoptions flag Possible completions: <[Enter]> Execute this command all Configure all events and packets tracing configuration Configure configuration tracing fallback Configure FALLBACK tracing message-in Configure received message tracing message-out Configure send message tracing mlag Configure MLAG tracing state-change Configure LACP state change tracing admin@Xorplus# set protocols lacp traceoptions flag message-in disable false admin@XorPlus# commit Commit OK. ##LACP per interface traceoptions. admin@Xorplus# set protocols lacp traceoptions interface ge-1/1/1 flag Possible completions: <[Enter]> Execute this command all Configure all events and packets tracing configuration Configure configuration tracing message-in Configure received message tracing message-out Configure send message tracing state-change Configure LACP state change tracing admin@Xorplus# set protocols lacp traceoptions interface ge-1/1/1 flag configuration disable false admin@XorPlus# commit Commit OK. Enable debug of UDLD: ## UDLD global traceoptions. admin@Xorplus# set protocols udld traceoptions Possible completions: <[Enter]> Execute this command all Configure all events and packets tracing configuration Configure configuration tracing event Configure event tracing packet Configure the sending/receiving packets tracing raw-packet Configure UDLD raw packet tracing state-change Configure state change tracing timer Configure UDLD timer tracing admin@Xorplus# set protocols udld traceoptions event disable false admin@XorPlus# commit Commit OK. Enable debug of BGP: admin@XorPlus# set protocols bgp traceoption ? admin@XorPlus# set protocols bgp traceoption Possible completions: <[Enter]> Execute this command bestpath BGP bestpath evpn EVPN keepalives BGP IPv4 neighbor to debug neighbor-events BGP Neighbor Events updates BGP updates zebra BGP zebra messages admin@XorPlus# set protocols bgp traceoption updates in admin@XorPlus# commit Commit OK. Save done. admin@XorPlus# Enable debug of ospf: admin@XorPlus# set protocols ospf traceoption ? Possible completions: <[Enter]> Execute this command ism Configure tracing of OSPF interface state machine lsa Configure tracing of OSPF link state advertisement nsm Configure tracing of OSPF neighbor state machine packet Configure tracing of OSPF packets zebra Configure tracing of zebra information admin@XorPlus# set protocols ospf traceoption packet all detail admin@XorPlus# commit Enable debug of stp: admin@XorPlus# set protocols spanning-tree traceoptions interface ge-1/1/1 ? Possible completions: <[Enter]> Execute this command all Configure all tracing operations bridge-detection-machine Configure bridge detection state machine tracing configuration Configure configuration tracing events Configure events tracing message-in Configure receive message tracing message-out Configure send message tracing mlag Configure mlag tracing port-information-machine Configure port information state machine tracing port-migration-machine Configure port migration state machine tracing port-receive-machine Configure port receive state machine tracing port-role-selection-machine Configure port role selection state machine tracing port-role-transition-machine Configure port role transition state machine tracing port-state-transition-machine Configure port state transition state machine tracing port-transmit-machine Configure port transmit state machine tracing state-machine-variables Configure state machine variables tracing timers Configure timers tracing topology-change-machine Configure topology change state machine tracing admin@XorPlus# set protocols spanning-tree traceoptions interface ge-1/1/1 all disable false admin@XorPlus# commit Commit OK. Save done. admin@XorPlus# Enable debug of igmp: admin@XorPlus# set protocols igmp traceoption ? Possible completions: <[Enter]> Execute this command events IGMP protocol events packets IGMP protocol packets trace IGMP internal daemon activity admin@XorPlus# set protocols igmp traceoption events admin@XorPlus# commit Commit OK. Save done. admin@XorPlus# 1.1.3 Find the Core Dump File When the device crashes, it will create a core file which can be found in a directory called pica/core. admin@R2:/pica/core$ pwd /pica/core 1.1.4 Find the last_death file for Troubleshooting You can view the last_death file after the device crashes. It will record the last log message and is located in /var/log directory. admin@R2:/pica/core$ cd /var/log/ admin@R2:/var/log$ ls btmp faillog fsck lastlog messages report_diag.log dmesg frr last_death lighttpd private wtmp 1.2 Routing and Forwarding Table 1.2.1 Check the Software and Hardware Route Tables To display the hardware host route table, use the show route forward-host ipv4 all command in L2/L3 operation mode. admin@Switch> show route forward-host ipv4 all Address HWaddress Port --------------- ----------------- --------- 10.10.3.2 48:6E:73:02:03:DA ge-1/1/48 Total host count:1 To display the hardware route table, use the show route forward-route ipv4 all command in L2/L3 operation mode. admin@Switch> show route forward-route ipv4 all Destination NextHopMac Port --------------- ----------------- --------- 10.10.3.0/24 48:6E:73:02:04:64 connected 101.101.101.0/24 48:6E:73:02:03:DA ge-1/1/48 102.102.102.0/24 48:6E:73:02:03:DA ge-1/1/48 Total route count:4 To display the software route table, use the show route ipv4 command in L2/L3 operation mode. admin@Switch> show route ipv4 If PicOS® is running in OVS mode, check the software and hardware flow tables. 1.3 Using Pipe (|) Filter Functions 1.3.1 Pipe (|) Filter Functions This topic describes the pipe (|) filter functions supported in the PicOS® L2/L3 CLI (command-line interface). The PicOS® L2/L3 mode has a growing number of CLI commands that users can use to troubleshoot common problems. These commands usually generate a lot of output. The use of pipe (|) filter functions increases readability of command output, making troubleshooting more effective. The following filter functions are available with the PicOS® L2/L3: Function Description compare Compare configuration changes with a prior version count Count occurrences display Display additional configuration information except Show only the lines of output that do not contain a pattern find Show output starting from the first occurrence of a pattern match Show only the lines of output that contain a pattern no-more Disable pagination of command output 1.3.2 Comparing Configurations The compare filter compares the current committed configuration with a previously committed configuration. admin@XorPlus# show | compare rollback nn nn is the index into the list of previously committed configurations, also known as the rollback number. The range of values for nn is 01-48. For example: admin@XorPlus# show | compare rollback 03 1.3.3 Counting the Number of Output Lines To count the number of lines in the output of a command, enter count after the pipe symbol (|). The following example uses count with the show command in configuration mode to display the number of non-default configuration lines: admin@XorPlus# show | count Count: 11 lines 1.3.4 Displaying Output that Matches a Pattern To display only the lines of output that match a pattern, enter match after the pipe symbol (|). The following example displays the status of only TbE (terabit Ethernet) interfaces: admin@XorPlus> show interface brief | match te- te-1/1/1 Enabled Down Disabled Full Auto te-1/1/2 Enabled Down Disabled Full Auto te-1/1/3 Enabled Down Disabled Full Auto te-1/1/4 Enabled Down Disabled Full Auto te-1/1/5 Enabled Down Disabled Full Auto te-1/1/6 Enabled Down Disabled Full Auto te-1/1/7 Enabled Down Disabled Full Auto te-1/1/8 Enabled Down Disabled Full Auto te-1/1/9 Enabled Down Disabled Full Auto te-1/1/10 Enabled Down Disabled Full Auto 1.3.5 Omitting Output that Matches a Pattern To omit lines from the output of a command that make up a pattern, enter except after the pipe symbol (|). The following example uses except with the show interface brief command in the operation mode to list the interfaces that are not down: admin@XorPlus> show interface brief | except Down 1.3.6 Preventing Output from Being Paginated By default, if the output of a command is longer than the length of terminal screen, user will see the --More-- message to display the remaining output. Press the space bar to display the remaining output. User can disable pagination by entering no-more after the pipe symbol (|). The following example displays the output of show command, executed in PicOS® L2/L3 configuration mode, all at once: admin@XorPlus# show | no-more This feature is useful, for example, when user wants to copy the entire output of a command and paste it into an e-mail to be sent to technical support. 1.4 Using the show tech-support Command 1.4.1 Show Tech-Support Command When contacting Pica8 for technical support, issue the command show tech-support because it captures the complete status of a PicOS® switch. It is recommended to send the output of show tech-support command along with the system log. The following samples describe how to obtain the output. Log in to the switch and enter the cli command at the Linux shell to reach the PicOS® L2/L3 operation mode. admin@Leaf-1$cli Synchronizing configuration...OK. Pica8 PicOS Version 2.6 Welcome to PicOS L2/L3 on Leaf-1 admin@Leaf-1> Enter the show tech-support command. admin@Leaf-1> show tech_support Start...... Item 1: Display version finished! Item 2: Display interface finished! Item 3: Display pica configuration finished! Item 4: Display system config files finished! Item 5: Display system process finished! Item 6: Display fdb table finished! Item 7: Display fdb entries finished! Item 8: Display ospf neighbors finished! Item 9: Display ospf interfaces finished! Item 10: Display kernel route table finished! Item 11: Display kernel ipv4 neigh table finished! Item 12: Display kernel ipv6 neigh table finished! Item 13: Display kernel neigh vrf finished! Item 14: Display hard-route table finished! Item 15: Display system hard-route for host finished! Item 16: Dispaly system spanning tree interfaces finished! Item 17: Dispaly spanning tree bridge finished! Item 18: Display vlans table finished! Item 19: Display vlan-interfaces finished! Item 20: Display core-dump finished! Item 21: Display system uptime finished! Item 22: Display arp table! Item 23: Display neighbor table! Item 24: Display routes table! Item 25: Display ipv4 routes in hardware table! Item 26: Display ipv6 routes in hardware table! Item 27: Display ipv4 hosts in hardware table! Item 28: Display ipv6 hosts in hardware table! Item 29: Display copp statistics! Item 30: Display mlag domain! Item 31: Display mlag link! Item 32: Display mlag config consistency! Item 33: Display mlag statistic! Item 34: Display license! Item 35: Display set! Item 36: Get error event from log! Item 37: Display frr configuration finished! Process BCM commands, total count=47 The information has been stored in /tmp/Leaf-1-201507050614-techSupport.log, please forward to support@pica8.com The last line of the output of show tech-support command provides the name and location of the file to which the output was saved. In the above example, the name of the file is Leaf-1-201507050614-techSupport.log that has been saved to the /tmp directory. You can transfer the file, generated by show tech-support command, from the switch to your computer over SCP (Secure Copy Protocol). There is a nice free Windows utility called WinSCP, available for download at https://winscp.net/eng/download.php, which you can use to copy the file from the switch to your computer over SCP. 2. PicOS® OVS Troubleshooting This section details basic procedures to troubleshoot PicOS® switches in OVS (Open vSwitch) mode. 2.1 Verifying PicOS® Mode Verify if PicOS® is actually running in OVS (Open vSwitch) mode, as described in Checking PicOS® Mode. When PicOS® is running in the OVS mode, two processes should be running: ovsdb-server and ovs-vswitchd. admin@XorPlus$ps -ef | grep ovs root 1356 1 0 Jan26 ? 00:00:10 /ovs/sbin/ovsdb-server /ovs/ovs-vswitchd.conf.db --pidfile --remote=punix:/ovs/var/run/openvswitch/db.sock root 1358 1 0 Jan26 ? 00:19:07 /ovs/sbin/ovs-vswitchd --enable-shared-lcmgr In CrossFlow mode, the router stack must have been initialized in addition to having ovsdb-server and ovs-vswitchd processes running. admin@XorPlus$ps -ef | grep pica root 12430 1 0 Jan07 ? 00:05:49 pica_cardmgr root 12432 1 0 Jan07 ? 01:03:19 pica_sif root 12439 1 0 Jan07 ? 00:08:45 pica_lacp root 12441 1 19 Jan07 ? 4-10:50:14 pica_lcmgr root 12447 1 0 Jan07 ? 00:09:58 pica_login root 13218 1 0 Jan07 ? 00:20:47 pica_mstp root 13236 1 0 Jan07 ? 01:25:30 /pica/bin/xorp_rtrmgr -d -L local0.info -P /var/run/xorp_rtrmgr.pid 2.2 Verifying Bridge Configuration For the bridge and ports to forward frames in hardware, the datapath_type configured for each entity must be set to pica8. admin@PicOS-OVS$ovs-vsctl show ac9e5b1e-4234-4158-9214-5660b9343779 Bridge east Controller "tcp:172.16.0.142:6653" is_connected: true fail_mode: standalone Port "ae1" tag: 1 Interface "ae1" type: "pica8_lag" options: {lacp-mode=active, lacp-system-priority="32768", lacp-time=slow, lag_type=lacp, link_speed=auto, members="te-1/1/2"} Port "te-1/1/2" tag: 1 Interface "te-1/1/2" type: "pica8" options: {flow_ctl=none, link_speed=auto} Port "te-1/1/1" tag: 1 Interface "te-1/1/1" type: "pica8" options: {flow_ctl=none, link_speed=auto} admin@PicOS-OVS$ovs-ofctl show east OFPT_FEATURES_REPLY (OF1.4) (xid=0x2): dpid:1deb0ae61be44040 n_tables:254, n_buffers:256 capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS OFPST_PORT_DESC reply (OF1.4) (xid=0x4): 1(te-1/1/1): addr:ff:ff:ff:ff:ff:00 config: 0 state: LINK_UP current: 1GB-FD COPPER advertised: 1GB-FD 10GB-FD FIBER supported: 10MB-FD 100MB-FD 1GB-FD 10GB-FD FIBER AUTO_NEG speed: 1000 Mbps now, 10000 Mbps max 2(te-1/1/2): addr:ff:ff:ff:ff:ff:00 config: 0 state: LINK_DOWN current: 1GB-FD COPPER advertised: 1GB-FD 10GB-FD FIBER supported: 10MB-FD 100MB-FD 1GB-FD 10GB-FD FIBER AUTO_NEG speed: 1000 Mbps now, 10000 Mbps max 1025(ae1): addr:ff:ff:ff:ff:ff:00 config: 0 state: LINK_UP current: 1GB-FD COPPER advertised: 1GB-FD 10GB-FD FIBER supported: 10MB-FD 100MB-FD 1GB-FD 10GB-FD FIBER AUTO_NEG speed: 1000 Mbps now, 10000 Mbps max LOCAL(east): addr:0a:e6:1b:e4:40:40 config: 0 state: LINK_UP current: 10MB-FD COPPER supported: 10MB-FD COPPER speed: 10 Mbps now, 10 Mbps max OFPT_GET_CONFIG_REPLY (OF1.4) (xid=0x6): frags=normal miss_send_len=0 admin@PicOS-OVS$ Once the ports are configured and verified, flows can be managed in OVS. 2.3 Checking Flow Discrepancies Check ovs-vswitchd flow discrepancies between the control plane and hardware: admin@PicOS-OVS$ovs-ofctl dump-tables br0 | grep -v active=0: 0: active=4, lookup=n/a, matched=n/a admin@PicOS-OVS$ovs-ofctl dump-flows br0 OFPST_FLOW reply (OF1.4) (xid=0x2): cookie=0x0, duration=1449.903s, table=0, n_packets=n/a, n_bytes=0, in_port=1,dl_src=00:00:3d:a6:c8:f2 actions=output:2 cookie=0x0, duration=1444.537s, table=0, n_packets=n/a, n_bytes=0, in_port=1,dl_src=00:00:3d:a6:c9:14 actions=output:1 cookie=0x0, duration=71723.842s, table=0, n_packets=n/a, n_bytes=0, mpls,in_port=1,dl_vlan=1,mpls_label=10 actions=output:3 cookie=0x0, duration=74839.581s, table=0, n_packets=n/a, n_bytes=923443200, in_port=1 actions=output:2 Display hardware flows as shown below: admin@PicOS-OVS$ovs-appctl pica/dump-flows #24 normal permanent priority=32769,in_port=1,dl_src=00:00:3d:a6:c8:f2, actions:2 #23 normal permanent priority=32769,in_port=1,dl_src=00:00:3d:a6:c9:14, actions:1 #22 normal permanent priority=32769,mpls,in_port=1,dl_vlan=1,mpls_label=10, actions:3 #21 normal permanent priority=32769,in_port=1, actions:2 #20 normal permanent priority=0, actions:drop Total 5 flows in HW. 2.4 Displaying OVSDB Display the full OVSDB (Open vSwitch Database) as shown below: admin@Leaf1$ovsdb-client dump Bridge table _uuid controller datapath_id datapath_type external_ids fail_mode flood_vlans flow_tables ipfix lldp_enable mirrors name netflow other_config ports protocols sflow status stp_enable ------------------------------------ -------------------------------------- ------------------ ------------- ------------ --------- ----------- ----------- ----- ----------- ------- -------- ------- ------------ ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------- ----- ------ ---------- c880536a-b614-41bf-9870-2d0bdab3664f [bedb4af7-2125-4346-8c89-bf61bd21f63b] "4c3e486e730203da" "pica8" {} [] [] {} [] false [] "ECODE3" [] {} [31605950-d9be-40b2-9ccb-bc4fd09991f0, 61ac5778-554f-4553-83ae-3bbc19ccf715, 62b35f47-e8ca-4496-8b37-f9bfbb7e80b0, 6dee5c6a-e9b8-41f7-87ef-b9379637a7c4, 99ac75b7-9fa1-4583-85f7-66d3145e7fa4] ["OpenFlow13"] [] {} false 2.5 Debug Packet-In Messages To debug the protocol messages between the switch and the controller, use the ovs-ofctl snoop command in the OVS mode. The following commands debug the protocol messages exchanged between the br0 bridge and the controller: admin@Switch$ovs-ofctl snoop br0 3. PicOS® System Troubleshooting User can troubleshoot by checking system logs and PicOS® works mode. Reset the Switch to Factory Default Automating Ping to Multiple Hosts Troubleshooting Switch Crashes CPU/Memory Rate Limit High CPU Utilization Backup Partition for PicOS® SSH Server Preparation Linux_configure.py script Provision.py script How to Disable Weak SSH Cipher/ MAC Algorithms in PicOS® Check log using two methods as follows: NOTE: Users should not put their other files in the /tmp directory, because the space size limit of the /tmp directory is 50M, once exceeded, it will lead to unpredictable system errors. System logs are stored in two locations: /var/log/messages This directory is stored in Flash. and /tmp/log/messages This directory is stored in RAM. Switches use flash memory that has a limited number of lifetime write operations. Hence, it is important that logs are not written continuously to the flash memory. This would dramatically impact the lifetime of the flash memory. This is why most of the log information is written by default on the /tmp directory. admin@XorPlus$df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 6202636 900184 4987368 16% / /dev/root 6202636 900184 4987368 16% / tmpfs 207348 28 207320 1% /run tmpfs 5120 0 5120 0% /run/lock tmpfs 414680 0 414680 0% /run/shm tmpfs 51200 36 51164 1% /tmp The "/tmp" directory is mounted on "tmpfs" which is a filesystem mounted in RAM. The tmp logs are moved to /var/log when dramatic events occur like system crash or system reboot. Checking PicOS® works mode as follows: In L2/L3 Mode (Or XORP), only the XORP system is running. admin@PicOS-OVS$ps aux | grep ovs | grep -v grep admin@PicOS-OVS$ admin@XorPlus$ps aux | grep xorp | grep -v grep root 16383 0.0 1.2 18100 6596 ? S Jan29 5:26 xorp_policy root 16385 0.3 2.5 34980 13380 ? Ss Jan29 99:20 /pica/bin/xorp_rtrmgr -d -L local0.info -P /var/run/xorp_rtrmgr.pid In OVS Mode, only the OVS daemon is running. admin@PicOS-OVS$ps aux | grep xorp | grep -v grep admin@PicOS-OVS$ admin@PicOS-OVS$ps aux | grep ovs | grep -v grep root 1984 0.0 0.1 6696 2524 ? S Nov13 0:10 ovsdb-server /ovs/ovs-vswitchd.conf.db --pidfile --remote=ptcp:6640:10.10.51.166 --remote=punix:/ovs/var/run/openvswitch/db.sock root 1989 25.6 1.5 113256 32392 ? Sl Nov13 1393:50 ovs-vswitchd --pidfile=ovs-vswitchd.pid --overwrite-pidfile 3.1 Reset the Switch to Factory Default Occasionally, it could be useful to reset the equipment to factory default (to erase all configurations or tools on the equipment). This can be done using the Upgrade command and an image of PicOS, for details about the usage of Upgrade command, please see Upgrading PICOS from Version 4.0.0 or Later Using Upgrade Command. Here is an example: admin@XorPlus$sudo upgrade picos-2.4-P3295-13912.tar.gz factory-default 3.2 Automating Ping to Multiple Hosts PicOS® switches support ping, which may be used to test connectivity to remote IP addresses. Users often want to test connectivity to all subnets in their network. This can be accomplished manually by pinging IP addresses in all subnets one-by-one, but that method is error-prone and tedious. This section describes how to write a simple re-usable script to ping a number of IP addresses at once. This script is especially useful when troubleshooting connectivity in a network, and user needs to ping a number of IP addresses again and again for verification. User can create the script once and use it again and again from the PicOS® L2/L3 operation mode. This requires a text editor to create the script and save it as a file on user's PicOS® switch. PicOS® includes the vi text editor, which can be run from the Linux shell on user's PicOS® switch. We choose to call our script pingAll.sh though user may choose any other name. The .sh file extension is not mandatory, though we recommend using it to make it obvious to anyone that the file is a shell script. admin@Leaf-1$vi pingAll.sh Inside the vi editor, press i to be able to insert text. Paste the following lines of text (after modifying them for user's network): ip[0]='192.168.42.2' ip[1]='192.168.42.4' ip[2]='192.168.42.5' ip[3]='192.168.42.9' ip[4]='192.168.42.20' ip[5]='192.168.42.40' ip[6]='192.168.42.60' ip[7]='192.168.42.100' ip[8]='192.168.42.110' ip[9]='192.168.42.120' ip[10]='192.168.42.130' ip[11]='192.168.42.240' ip[12]='192.168.42.22' for ((i=0; i <=12; i++)) do ping -c 3 ${ip[$i]} done Press Esc and then enter :wq to save the file and exit the vi editor. Some information about the script follows. The ip[] array has thirteen elements (ip[0] – ip[12]) and each element holds an IP address. User can change both the IP addresses and the number of array elements. The script will send three ping requests to each IP address in the ip[] array, one by one. If user is familiar with shell scripting or programming in C-like languages, the script should be self-descriptive. Even if user is an absolute beginner to programming and scripting, user should be able to modify and use the script after some research. List the contents of user's home directory. admin@Leaf-1$ls pingAll.sh Make the new file pingAll.sh executable. admin@Leaf-1$chmod +x pingAll.sh Enter the PicOS® L2/L3 operation mode. admin@Leaf-1$cli Synchronizing configuration...OK. Pica8 PicOS Version 2.6 Welcome to PicOS L2/L3 on Leaf-1 admin@Leaf-1> Run the script from PicOS® L2/L3 operation mode. admin@Leaf-1> bash /home/admin/pingAll.sh PING 192.168.42.2 (192.168.42.2) 56(84) bytes of data. 64 bytes from 192.168.42.2: icmp_req=1 ttl=64 time=4.66 ms 64 bytes from 192.168.42.2: icmp_req=2 ttl=64 time=0.848 ms 64 bytes from 192.168.42.2: icmp_req=3 ttl=64 time=0.910 ms --- 192.168.42.2 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.848/2.142/4.669/1.787 ms PING 192.168.42.4 (192.168.42.4) 56(84) bytes of data. 64 bytes from 192.168.42.4: icmp_req=1 ttl=64 time=8.27 ms 64 bytes from 192.168.42.4: icmp_req=2 ttl=64 time=1.98 ms 64 bytes from 192.168.42.4: icmp_req=3 ttl=64 time=2.94 ms --- 192.168.42.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 1.986/4.401/8.273/2.765 ms PING 192.168.42.5 (192.168.42.5) 56(84) bytes of data. 64 bytes from 192.168.42.5: icmp_req=1 ttl=64 time=6.59 ms 64 bytes from 192.168.42.5: icmp_req=2 ttl=64 time=3.22 ms 64 bytes from 192.168.42.5: icmp_req=3 ttl=64 time=1.81 ms --- 192.168.42.5 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 1.812/3.876/6.594/2.006 ms 3.3 Troubleshooting Switch Crashes The PicOS® switch may restart after detecting an unrecoverable error. This situation is usually referred to as a system crash. When the switch crashes, it will create a core file that user can use to figure out what went wrong. The core file is stored in the directory /pica/core. Use the file list command in PicOS® L2/L3 operation mode to display the contents of the directory. admin@LEAF-A> file list /pica/core total 0 The output above shows that there is no file in the /pica/core directory. The switch we used never crashed and did not create any core file. The PicOS® writes the last log messages to the /var/log/last_death file after a system crash. admin@LEAF-A> file show /var/log/last_death | count Count: 405 lines admin@LEAF-A> file show /var/log/last_death | match lcmgr Jun 23 2015 04:08:56 XorPlus local0.info : [PICA_MONITOR]Process pica_lcmgr, running, PID 2823 Jun 23 2015 04:08:56 XorPlus local0.info : [PICA_MONITOR]Monitor for process pica_lcmgr started Jun 23 2015 04:44:32 XorPlus local0.err : [LCMGR]Someone set counter interval to ZERO!! Jun 23 2015 04:44:34 XorPlus local0.err : [RTRMGR]XRL Death: class lcmgr01 instance lcmgr01-5eeea3b7d435a0b7277ba879a582fff6@127.0.0.1 time:Thu Jan 1 00:43:34 1970 death module:lcmgr01 3.4 CPU/Memory Rate Limit From PicOS® 2.6, we added the CPU rate limit for processes of PicOS®. Including pica_lcmgr, pica_sif, and ovs-vswitchd. Summary: The default CPU usage is 40% if not provided, and default memory size is 150 MB. The warning message will be printed if the memory size is bigger than the default value. The CPU limitation is based on all CPU's on the system. If the system CPU is P2020 dual cores, 40% CPU limitation is equal to 80% single CPU. Running CPU/memory rate limit tools manually as follows: sudo /pica/bin/system/tools/pica_monitor -v -c 40 -m 150 pica_lcmgr Checking CPU/memory rate limit tools as follows: admin@XorPlus$ps -aux | grep pica_monitor warning: bad ps syntax, perhaps a bogus '-'? See http://gitorious.org/procps/procps/blobs/master/Documentation/FAQ root 3420 0.6 0.7 38944 3896 ? S 3.5.3 Step 3 Check the core dump in the /pica/core directory. 3.5.4 Step 4 To display the virtual interfaces configured on the switch, use the ifconfig command at the Linux shell: admin@Switch$ifconfig eth0 Link encap:Ethernet HWaddr 48:6e:73:02:04:63 inet addr:192.168.42.110 Bcast:192.168.42.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2379952 errors:0 dropped:0 overruns:0 frame:0 TX packets:1060135 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:354374731 (337.9 MiB) TX bytes:152816006 (145.7 MiB) Base address:0x2000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:98303973 errors:0 dropped:0 overruns:0 frame:0 TX packets:98303973 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:860429061 (820.5 MiB) TX bytes:860429061 (820.5 MiB) vlan.3 Link encap:Ethernet HWaddr 48:6e:73:02:04:64 inet addr:10.10.3.1 Bcast:10.10.3.255 Mask:255.255.255.0 inet6 addr: fe80::4a6e:73ff:302:464/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:36973 errors:0 dropped:0 overruns:0 frame:0 TX packets:36446 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:2927602 (2.7 MiB) TX bytes:2743024 (2.6 MiB) 3.5.5 Step 5 To display packets on a specific virtual interface, use the tcpdump command at the Linux shell: admin@Switch$sudo tcpdump -i vlan.3 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vlan.3, link-type EN10MB (Ethernet), capture size 65535 bytes To debug the protocol messages between the switch and the controller, use the ovs-ofctl snoop command in the OVS mode. The following commands debug the protocol messages exchanged between the br0 bridge and the controller: admin@Switch$ovs-ofctl snoop br0 3.5.6 Common Causes In the CrossFlow mode, both L2/L3 and OVS processes are running. The switch has to process both OVS protocol messages and the L2/L3 packets like BPDUs, OSPF packets, and BGP packets. The switch is likely to have a higher CPU utilization in the CrossFlow mode compared with the L2/L3 or OVS modes. Normally, the CPU-bound packets are less than 1000 pps (packets per second), and the CPU utilization is not high. However, the eth0 management interface has no rate limiting configured. Therefore, an attacker can send a large number of packets to the management interface, making the switch slow and even unusable for legitimate traffic. 3.5.7 Possible Fixes User can deploy the following fixes for high CPU utilization: Add a default drop flow for table-miss packets, to prevent these packet from causing high CPU utilization. Remove some flows with actions: Controller, LOCAL Make sure that the controller is not sending exessive OpenFlow messages to the switch. Configure the management interface eth0 at a low speed like 10 Mbps, using the ethtool -s eth0 speed 10 command. Reload the switch 3.6 Backup Partition for PicOS® Backup partition for PicOS®: PowerPc Platform: We use backup partitions for PicOS® to upgrade the system and recover PicOS®. Usually users need to reserve about 400 MB for partition 2(eg:sda2). The rest of the SD card belongs to partition 1(eg：sda1). If the size of the SD card is 2 GB, partition 1 should be 1.6GB (1600M) and partition 2 is 400M. Command (m for help): p Disk /dev/sda: 8004 MB, 8004304896 bytes 247 heads, 62 sectors/track, 1020 cylinders, total 15633408 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sda1 62 12603421 6301680 83 Linux ------the primary partition for PicOS /dev/sda2 12603422 15620279 1508429 83 Linux ------the backup partition for PicOS X86 platform: There are two partitioning ways used with ONIE, GPT and MBR. With GPT partitioning, the sda1/2MB, is allocated to GRUB as BOOT PARTITION. The second partition is used by ONIE itself. The 3rd or others are free, and can be used by NOS. In this mode, the 3rd partition is allocated to PicOS® GRUB, for the grub bootup config files. 4th and 5th are for PicOS® and PicOS®-BACKUP. When the user runs uninstall from ONIE, all partitions except 1st and 2nd are reserved, all NOS are wiped out. With MBR partitioning mode (which is not recommended), the GRUB boot codes are saved before MBR sector and first partition, the first partition is used by ONIE itself. PicOS® begins from the 2nd partition for PicOS®-GRUB, PicOS® and PicOS®-BACKUP partitions. eg:(With MBR) /dev/sda1: LABEL="ONIE-BOOT" UUID="08ae2c6a-6f14-498f-8e13-d0e7c0a567c1" /dev/sda3: LABEL="PicOS" UUID="b2735e76-8594-41b9-87e7-d25113dc22f7" ------the primary partition for PicOS /dev/sda2: LABEL="PICOS-GRUB" UUID="ca79674b-70fc-4540-b9ef-c98c3afadce3" /dev/sda4: LABEL="PICOS-BAK" UUID="92028225-403a-44d4-a40e-25e26d46373b" ------the backup partition for PicOS eg:(with GPT) Disk /dev/sda: 15649200 sectors, 7.5 GiB Logical sector size: 512 bytes Disk identifier (GUID): 1687245E-B39A-48E5-860B-D7967A67FBE8 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 15649166 Partitions will be aligned on 1-sector boundaries Total free space is 8547665 sectors (4.1 GiB) Number Start (sector) End (sector) Size Code Name 1 2048 6143 2.0 MiB EF02 GRUB-BOOT 2 6144 268287 128.0 MiB 3000 ONIE-BOOT 3 268288 1244140 476.5 MiB 0700 PICOS-GRUB 4 1244141 5150390 1.9 GiB 0700 PicOS -----the primary partition for PicOS 5 5150391 7103515 953.7 MiB 0700 PICOS-BAK ------the backup partition for PicOS 3.7 SSH Server Preparation Add the PKI files、two scripts and the PicOS® image on the ssh server. 1: The directory of ssl-private-key (Our openssl connection is not ready, so you only need to create these key files on the server) root@dev-1:/ssl#ls cacert.pem sc-cert.pem sc-privkey.pem 3.8 Linux_configure.py script This script usually starts automatically at the end of the configuration interactive shell. This script can set hostname, create accounts and update the time via ntp. You can modify or add this script to define the hostname and accounts and passwords. root@dev-1:/pica8# vim linux_configure.py _hostname = "HostName-Test" _accounts = {"lily":"1R.O.4HRDfvEY", "tom":"7hCft0situjJQ"} NOTE: The password of the user should be created by password generator. 3.9 Provision.py script This script usually starts automatically at the end of configuration interactive shell. It is used to download PKI files、PicOS® image and linux_configure.py, and then for updating the image and running the linxu_configure.py. You should modify the scipt to define the directory of the files. root@dev-1:/pica8# vim provision.py _server_paths = { "pki_sw_pri_key":"/ssl/sc-privkey.pem", "pki_sw_ca":"/ssl/sc-cert.pem", "pki_ctl_ca":"/ssl/cacert.pem", "ovs_upgrade_deb":"/pica8/pica-ovs-2.5-P3290-17741.deb", "linux_configure_script":"/pica8/linux_configure.py" } 3.10 How to Disable Weak SSH Cipher/ MAC Algorithms in PicOS® 3.10.1 Requirement Some of the security scans may show below Server-to-Client or Client-To-server encryption algorithms as vulnerable: arcfour arcfour128 arcfour256 Below are some of the Message Authentication Code (MAC) algorithms: hmac-md5 hmac-md5-96 hmac-sha1-96 NOTE: PicOS® 3.1.0 and the later version use OpenSSH(?) version is 6.7p1 and following are default Ciphers: chacha20-poly1305@openssh.com, aes128-ctr,aes192-ctr,aes256-ctr, aes128-gcm@openssh.com,aes256-gcm@openssh.com 3.10.2 Description Verify weak cipher and MAC algorithms are currently used by the SSH running in PicOS® switch. Perform following three steps: First check the cipher and MAC algorithms currently supported in the PicOS® SSH protocol. Check the version of SSH: root@Xorplus:/etc/ssh# ssh -v OpenSSH_6.0p1 Debian-4+deb7u2, OpenSSL 1.0.1e 11 Feb 2013 Check what cipher and MAC algorithms are currently supported. From another Linux Server run the following to list the cipher and MAC algorithms supported by PicOS®, using the following command: nmap --script ssh2-enum-algos -sV -p 22 Example output: root@AutomationServer1 html]# nmap --script ssh2-enum-algos -sV -p 22 172.16.0.191 Starting Nmap 6.40 ( http://nmap.org ) at 2019-03-14 14:13 PDT Nmap scan report for 172.16.0.191 Host is up (0.00079s latency). PORT STATE SERVICE VERSION 22/tcp open ssh OpenSSH 6.0p1 (protocol 2.0) | ssh2-enum-algos: | kex_algorithms (7) | ecdh-sha2-nistp256 | ecdh-sha2-nistp384 | ecdh-sha2-nistp521 | diffie-hellman-group-exchange-sha256 | diffie-hellman-group-exchange-sha1 | diffie-hellman-group14-sha1 | diffie-hellman-group1-sha1 | server_host_key_algorithms (3) | ssh-rsa | ssh-dss | ecdsa-sha2-nistp256 | encryption_algorithms (13) | aes128-ctr | aes192-ctr | aes256-ctr | arcfour256 | arcfour128 | aes128-cbc | 3des-cbc | blowfish-cbc | cast128-cbc | aes192-cbc | aes256-cbc | arcfour | rijndael-cbc@lysator.liu.se | mac_algorithms (11) | hmac-md5 | hmac-sha1 | umac-64@openssh.com | hmac-sha2-256 | hmac-sha2-256-96 | hmac-sha2-512 | hmac-sha2-512-96 | hmac-ripemd160 | hmac-ripemd160@openssh.com | hmac-sha1-96 | hmac-md5-96 From the above output decide which cipher or MAC algorithm you want to disable. For example say you want to disable arcfour cipher algorithm. 3.10.3 Solution Disable weak Cipher and MAC algorithms used by the SSH running in PicOS® switch by performing the following three steps: Disable the weak Cipher and MAC algorithms used by the SSH running in PicOS® switch as follows: You could disable the Ciphers using the command below: # vi /etc/ssh/sshd_config Press key 'i' to insert and copy the lines below to the end of the file (put only the cipher and MAC algorithms that needs to supported, and not include the weaker cipher and Mac algorithms). Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc Macs hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512 Save the file. On the PicOS® switch restart SSH with the following Linux command: /etc/init.d/ssh restart Verify whether weak Cipher and MAC algorithms are now not used by the SSH running in PicOS® switch: From another Linux Server run the following to list the cipher and MAC algorithms supported by PicOS®, using the following command: nmap --script ssh2-enum-algos -sV -p 22 You will see arcfour cipher algorithm is not used by SSH from the following output. This would show the only the allowed cipher and MAC algorithms now. Example output: root@AutomationServer1 html]# nmap --script ssh2-enum-algos -sV -p 22 172.16.0.191 Starting Nmap 6.40 ( http://nmap.org ) at 2019-03-14 14:35 PDT Nmap scan report for 172.16.0.191 Host is up (0.00055s latency). PORT STATE SERVICE VERSION 22/tcp open ssh OpenSSH 6.0p1 (protocol 2.0) | ssh2-enum-algos: | kex_algorithms (7) | ecdh-sha2-nistp256 | ecdh-sha2-nistp384 | ecdh-sha2-nistp521 | diffie-hellman-group-exchange-sha256 | diffie-hellman-group-exchange-sha1 | diffie-hellman-group14-sha1 | diffie-hellman-group1-sha1 | server_host_key_algorithms (3) | ssh-rsa | ssh-dss | ecdsa-sha2-nistp256 | encryption_algorithms (8) | aes128-ctr | aes192-ctr | aes256-ctr | aes128-cbc | 3des-cbc | blowfish-cbc | aes192-cbc | aes256-cbc | mac_algorithms (4) | hmac-sha1 | umac-64@openssh.com | hmac-sha2-256 | hmac-sha2-512 4. Technical Support Execute the diagnostic command show tech_support to send information to Pica8 Technical Support and receive a diagnostic report back. 4.1 Executing the Diagnostic Command admin@XorPlus> show tech_support Start...... Item 1: Display version finished! Item 2: Display interface finished! Item 3: Display pica configuration finished! Item 4: Display system config files finished! Item 5: Display system process finished! Item 6: Display fdb table finished! Item 7: Display fdb entries finished! Item 8: Display ospf neighbors finished! Item 9: Display ospf interfaces finished! Item 10: Display kernel route table finished! Item 11: Display kernel ipv4 neigh table finished! Item 12: Display kernel ipv6 neigh table finished! Item 13: Display kernel neigh vrf finished! Item 14: Display hard-route table finished! Item 15: Display system hard-route for host finished! Item 16: Dispaly system spanning tree interfaces finished! Item 17: Dispaly spanning tree bridge finished! Item 18: Display vlans table finished! Item 19: Display vlan-interfaces finished! Item 20: Display core-dump finished! Item 21: Display system uptime finished! Item 22: Display arp table! Item 23: Display neighbor table! Item 24: Display routes table! Item 25: Display ipv4 routes in hardware table! Item 26: Display ipv6 routes in hardware table! Item 27: Display ipv4 hosts in hardware table! Item 28: Display ipv6 hosts in hardware table! Item 29: Display copp statistics! Item 30: Display mlag domain! Item 31: Display mlag link! Item 32: Display mlag config consistency! Item 33: Display mlag statistic! Item 34: Display license! Item 35: Display set! Item 36: Get error event from log! Item 37: Display frr configuration finished! Process BCM commands, total count=47 The information has been stored in /tmp/XorPlus-201307052220-techSupport.log, please forward to support@pica8.com admin@XorPlus> 5. General PicOS® FAQ We have summarized the general PicOS® FAQ here, please download it from the following link: General_PICOS_FAQ.docx 6. Traceoptions Configuration Commands set interface traceoptions flag config disable true set interface traceoptions flag ethernet-switching-options disable true set protocols mlag traceoptions all disable false set interface traceoptions flag neighbor-event disable true set interface traceoptions flag packets disable true set interface traceoptions flag route-event disable true set interface traceoptions flag static-ethernet-switching disable true set interface traceoptions line-card statistic disable true set interface traceoptions line-card trace-level all disable true set interface traceoptions line-card trace-level api debug disable true set interface traceoptions line-card trace-level api error disable true set interface traceoptions line-card trace-level api information disable true set interface traceoptions line-card trace-level api warning disable true set interface traceoptions line-card trace-level sdk debug disable true set interface traceoptions line-card trace-level sdk error disable true set interface traceoptions line-card trace-level sdk information disable true set interface traceoptions line-card trace-level sdk warning disable true set interface traceoptions line-card trace-level xrl debug disable true set interface traceoptions line-card trace-level xrl error disable true set interface traceoptions line-card trace-level xrl information disable true set interface traceoptions line-card trace-level xrl warning disable true set interface traceoptions line-card trace-type all disable true set interface traceoptions line-card trace-type configuration disable true set interface traceoptions line-card trace-type link-change disable true set interface traceoptions line-card trace-type mac-update disable true set interface traceoptions line-card trace-type packet disable true set interface traceoptions line-card trace-type packet-receive disable true set interface traceoptions line-card trace-type packet-transmit disable true set interface traceoptions line-card trace-type statistics disable true 7. Displaying the Debugging Message User can configure the debugging message in a current window. 7.1 Syslog Monitor On admin@XorPlus> syslog monitor on Nov 21 2000 22:27:39 XorPlus local0.warn : [SIF]Interface ge-1/1/3, changed state to up Nov 21 2000 22:27:41 XorPlus local0.warn : root logined the switch Nov 21 2000 22:41:18 XorPlus local0.info xinetd[1102]: START: telnet pid=7650 from=10.10.50.16 Nov 21 2000 22:41:23 XorPlus authpriv.debug login[7651]: pam_unix(login:account): account admin has password changed in future Nov 21 2000 22:41:26 XorPlus local0.warn : admin logined the switch Nov 21 2000 22:55:58 XorPlus local0.info xinetd[1102]: START: telnet pid=8039 from=10.10.51.16 Nov 21 2000 22:56:01 XorPlus authpriv.debug login[8040]: pam_unix(login:account): account root has password changed in future Nov 21 2000 23:31:13 XorPlus local0.info xinetd[1102]: START: telnet pid=9028 from=10.10.50.16 Nov 21 2000 23:31:16 XorPlus authpriv.debug login[9029]: pam_unix(login:account): account admin has password changed in future Nov 21 2000 23:31:21 XorPlus local0.warn : admin logined the switch admin@XorPlus>

PicOS® Software Installation and Upgrade Guide

04-11-2025 - PicOS® Software Installation and Upgrade Guide 1. ONIE Version and BIOS/U-Boot Information of Verified Platforms The ONIE and BIOS/U-Boot Version information of platforms verified in the lab are listed below. The users can find the ONIE version information in the onie-syseeprom command output. Platform BIOS/U-Boot Version ONIE Version AS4610_54P none ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 159 TLV Name Code Len Value -------------------- ---- --- ----- Product Name 0x21 15 4610-54P-O-AC-F Part Number 0x22 13 FP1ZZ5654001A Serial Number 0x23 12 EC1731000333 Base MAC Address 0x24 6 A8:2B:B5:70:43:40 Manufacture Date 0x25 19 08/22/2017 19:30:27 Label Revision 0x27 3 R01 Platform Name 0x28 23 arm-accton_as4610_54-r0 ONIE Version 0x29 13 2016.05.00.04 MAC Addresses 0x2A 2 55 Manufacturer 0x2B 6 Accton Country Code 0x2C 2 TW Vendor Name 0x2D 8 Edgecore Diag Version 0x2E 5 001.9 CRC-32 0xFE 4 0xDABC2397 Checksum is valid. AS4610_54T none ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 159 TLV Name Code Len Value -------------------- ---- --- ----- Product Name 0x21 15 4610-54T-O-AC-F Part Number 0x22 13 F0PEC4654000Z Serial Number 0x23 12 EC1741001625 Base MAC Address 0x24 6 A8:2B:B5:CD:6C:C0 Manufacture Date 0x25 19 10/30/2017 12:56:49 Label Revision 0x27 3 R01 Platform Name 0x28 23 arm-accton_as4610_54-r0 ONIE Version 0x29 13 2016.05.00.04 MAC Addresses 0x2A 2 55 Manufacturer 0x2B 6 Accton Country Code 0x2C 2 TW Vendor Name 0x2D 8 Edgecore Diag Version 0x2E 5 001.9 CRC-32 0xFE 4 0xF40F7512 Checksum is valid. AS4610_54T_B none ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 159 TLV Name Code Len Value -------------------- ---- --- ----- Product Name 0x21 15 4610-54T-O-AC-B Part Number 0x22 13 F0PEC4654003Z Serial Number 0x23 12 EC1631000053 Base MAC Address 0x24 6 C4:39:3A:FF:2D:C0 Manufacture Date 0x25 19 08/05/2016 11:45:43 Label Revision 0x27 3 R0A MAC Addresses 0x2A 2 55 Manufacturer 0x2B 6 Accton Country Code 0x2C 2 TW Vendor Name 0x2D 8 Edgecore Diag Version 0x2E 5 001.7 Platform Name 0x28 23 arm-accton_as4610_54-r0 ONIE Version 0x29 13 2018.02.00.03 CRC-32 0xFE 4 0x9DC28EDF Checksum is valid. AS4610_30P none ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 160 TLV Name Code Len Value -------------------- ---- --- ----- Product Name 0x21 15 4610-30P-O-AC-F Part Number 0x22 13 F0PEC4630402Z Serial Number 0x23 12 EC1815000436 Base MAC Address 0x24 6 3C:2C:99:89:89:00 Manufacture Date 0x25 19 04/15/2018 23:45:48 Label Revision 0x27 4 R01A Platform Name 0x28 23 arm-accton_as4610_30-r0 ONIE Version 0x29 13 2016.05.00.04 MAC Addresses 0x2A 2 31 Manufacturer 0x2B 6 Accton Country Code 0x2C 2 TW Vendor Name 0x2D 8 Edgecore Diag Version 0x2E 5 001.9 CRC-32 0xFE 4 0xCD54AF53 Checksum is valid. AS4610_30T none ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 160 TLV Name Code Len Value -------------------- ---- --- ----- Product Name 0x21 15 4610-30T-O-AC-F Part Number 0x22 13 F0PEC4630001Z Serial Number 0x23 12 EC1806001291 Base MAC Address 0x24 6 3C:2C:99:41:47:E0 Manufacture Date 0x25 19 02/12/2018 23:08:49 Label Revision 0x27 4 R01A Platform Name 0x28 23 arm-accton_as4610_30-r0 ONIE Version 0x29 13 2016.05.00.04 MAC Addresses 0x2A 2 31 Manufacturer 0x2B 6 Accton Country Code 0x2C 2 TW Vendor Name 0x2D 8 Edgecore Diag Version 0x2E 5 001.9 CRC-32 0xFE 4 0x4FF5BAD3 Checksum is valid. S4048-ON Version 2.16.1242. Copyright (C) 2013 American Megatrends, Inc. BIOS Date: 02/22/2017 02:29:52 Ver: 0ACBZ018 ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 149 TLV Name Code Len Value -------------------- ---- --- ----- Product Name 0x21 7 S4048ON Part Number 0x22 6 099TJK Serial Number 0x23 20 CN099TJK282985640054 Base MAC Address 0x24 6 34:17:EB:FA:90:C4 Manufacture Date 0x25 19 06/08/2015 20:36:30 Label Revision 0x27 3 A00 MAC Addresses 0x2A 2 256 Manufacturer 0x2B 5 28298 Country Code 0x2C 2 CN Service Tag 0x2F 7 FX4PX42 Vendor Extension 0xFD 6 0x36 0x37 0x34 0x2D 0x46 0x46 Platform Name 0x28 26 x86_64-dell_s4000_c2338-r0 Loader Version 0x29 8 3.21.1.1 CRC-32 0xFE 4 0x7EB3C763 Checksum is valid. S4128F-ON Version 2.17.1245. Copyright (C) 2017 American Megatrends, Inc. BIOS Date: 04/26/2017 04:20:58 Ver: 0ACBZ028 ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 180 TLV Name Code Len Value -------------------- ---- --- ----- Product Name 0x21 9 S4128F-ON Part Number 0x22 6 02NK09 Serial Number 0x23 20 CN02NK092829886J0109 Base MAC Address 0x24 6 E4:F0:04:DF:67:16 Manufacture Date 0x25 19 06/19/2018 11:32:52 Device Version 0x26 1 1 Label Revision 0x27 3 A02 Platform Name 0x28 30 x86_64-dellemc_s4128f_c2338-r0 ONIE Version 0x29 10 3.33.1.1-4 MAC Addresses 0x2A 2 128 Manufacturer 0x2B 5 28298 Country Code 0x2C 2 CN Vendor Name 0x2D 8 Dell EMC Diag Version 0x2E 10 3.33.3.0-1 Service Tag 0x2F 7 HPPKXC2 Vendor Extension 0xFD 4 0x00 0x00 0x02 0xA2 CRC-32 0xFE 4 0x1A25266A Checksum is valid. S4148T-ON Version 2.17.1245. Copyright (C) 2017 American Megatrends, Inc. BIOS Date: 07/06/2017 01:44:00 Ver: 0ACBZ028 ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 180 TLV Name Code Len Value -------------------- ---- --- ----- Product Name 0x21 9 S4148T-ON Part Number 0x22 6 0JD8R7 Serial Number 0x23 20 CN0JD8R7282987CN0053 Base MAC Address 0x24 6 E4:F0:04:80:EA:CC Manufacture Date 0x25 19 12/23/2017 16:32:17 Device Version 0x26 1 1 Label Revision 0x27 3 A01 MAC Addresses 0x2A 2 256 Manufacturer 0x2B 5 28298 Country Code 0x2C 2 CN Vendor Name 0x2D 8 Dell EMC Service Tag 0x2F 7 6SCCXC2 Vendor Extension 0xFD 4 0x00 0x00 0x02 0xA2 Platform Name 0x28 30 x86_64-dellemc_s4148t_c2338-r0 ONIE Version 0x29 10 3.33.1.1-6 Diag Version 0x2E 10 3.33.3.1-6 CRC-32 0xFE 4 0xD89AF6DE Checksum is valid. S4148F-ON Version 2.17.1245. Copyright (C) 2017 American Megatrends, Inc. BIOS Date: 12/04/2017 20:42:30 Ver: 0ACBZ028 ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 179 TLV Name Code Len Value -------------------- ---- --- ----- Product Name 0x21 9 S4148F-ON Part Number 0x22 6 0R2RKC Serial Number 0x23 20 TW0R2RKC2829872D0046 Base MAC Address 0x24 6 14:18:77:18:2C:B8 Manufacture Date 0x25 19 02/13/2017 19:32:31 Device Version 0x26 1 1 Label Revision 0x27 3 X01 MAC Addresses 0x2A 2 256 Manufacturer 0x2B 5 28298 Country Code 0x2C 2 TW Vendor Name 0x2D 8 DELL EMC Service Tag 0x2F 7 CM31XC2 Vendor Extension 0xFD 4 0x00 0x00 0x02 0xA2 Platform Name 0x28 29 x86_64-dellemc_s4100_c2338-r0 ONIE Version 0x29 10 3.33.1.1-4 Diag Version 0x2E 10 3.33.3.0-1 CRC-32 0xFE 4 0x42273778 Checksum is valid. AS7712_32X Version 2.16.1242. Copyright (C) 2013 American Megatrends, Inc. BIOS Date: 09/08/2015 11:15:24 Ver: ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 168 TLV Name Code Len Value -------------------- ---- --- ----- Manufacture Date 0x25 19 10/28/2015 20:33:51 Label Revision 0x27 4 R0AB Manufacturer 0x2B 6 Accton Country Code 0x2C 2 TW Base MAC Address 0x24 6 CC:37:AB:63:8B:84 Serial Number 0x23 14 771232X1541003 Part Number 0x22 13 FP3ZZ7632014A Product Name 0x21 15 7712-32X-O-AC-F MAC Addresses 0x2A 2 131 Vendor Name 0x2D 8 Edgecore Diag Version 0x2E 7 0.0.5.4 Platform Name 0x28 27 x86_64-accton_as7712_32x-r0 ONIE Version 0x29 13 2018.11.00.02 CRC-32 0xFE 4 0x9208666A Checksum is valid. Z9100-ON Version 2.17.1245. Copyright (C) 2017 American Megatrends, Inc. BIOS Date: 02/22/2017 21:20:05 Ver: 0ACBZ028 ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 168 TLV Name Code Len Value -------------------- ---- --- ----- Product Name 0x21 8 Z9100-ON Part Number 0x22 6 04HW8N Serial Number 0x23 20 CN04HW8N7793163I0010 Base MAC Address 0x24 6 4C:76:25:E8:D7:C0 Manufacture Date 0x25 19 03/19/2016 12:39:24 Device Version 0x26 1 1 Label Revision 0x27 3 A00 Platform Name 0x28 26 x86_64-dell_z9100_c2538-r0 ONIE Version 0x29 8 3.23.1.3 MAC Addresses 0x2A 2 384 Manufacturer 0x2B 5 77931 Country Code 0x2C 2 CN Vendor Name 0x2D 4 DELL Diag Version 0x2E 6 01_010 Service Tag 0x2F 7 2QWRG02 Vendor Extension 0xFD 7 0x00 0x00 0x02 0xA2 0x2D 0x46 0x46 CRC-32 0xFE 4 0x3B190E49 Checksum is valid. AS7816_64X Version 2.19.1269. Copyright (C) 2018 American Megatrends, Inc. BIOS Date: 10/05/2018 08:57:44 Ver: AS7816-64X V36 20181004 ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 171 TLV Name Code Len Value -------------------- ---- --- ----- Manufacture Date 0x25 19 11/02/2018 16:32:21 Label Revision 0x27 4 R01A Platform Name 0x28 27 x86_64-accton_as7816_64x-r0 Manufacturer 0x2B 6 Accton Country Code 0x2C 2 TW Vendor Name 0x2D 8 Edgecore Product Name 0x21 17 7816-64X-O-AC-F-R Part Number 0x22 13 FP3ZZ7664020A Serial Number 0x23 14 781664X1843004 Base MAC Address 0x24 6 B8:6A:97:73:6A:3E MAC Addresses 0x2A 2 300 ONIE Version 0x29 13 2018.11.00.02 Diag Version 0x2E 8 0.1.0.17 CRC-32 0xFE 4 0x84DD5474 Checksum is valid. Z9264F-ON Version 2.19.1266. Copyright (C) 2018 American Megatrends, Inc. BIOS Date: 09/17/2018 21:25:57 Ver: 0ACHI032 ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 181 TLV Name Code Len Value -------------------- ---- --- ----- Product Name 0x21 9 Z9264F-ON Part Number 0x22 6 0RWYT4 Serial Number 0x23 20 CN0RWYT4DND008660010 Base MAC Address 0x24 6 20:04:0F:05:D4:97 Manufacture Date 0x25 19 06/06/2018 03:00:21 Device Version 0x26 1 1 Label Revision 0x27 3 A00 Platform Name 0x28 30 x86_64-dellemc_z9264f_c3538-r0 ONIE Version 0x29 10 3.42.1.9-3 MAC Addresses 0x2A 2 640 Manufacturer 0x2B 5 DND00 Country Code 0x2C 2 CN Vendor Name 0x2D 8 Dell EMC Diag Version 0x2E 11 3.00.3.41-1 Service Tag 0x2F 7 20GKXC2 Vendor Extension 0xFD 4 0x00 0x00 0x02 0xA2 CRC-32 0xFE 4 0xD8EFCB81 Checksum is valid. ONIE:/ # AS5812_54T Version 2.16.1242. Copyright (C) 2013 American Megatrends, Inc. BIOS Date: 08/20/2015 10:55:33 Ver: A02 0820 ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 168 TLV Name Code Len Value -------------------- ---- --- ----- Manufacture Date 0x25 19 08/11/2016 16:36:46 Diag Version 0x2E 7 1.0.0.5 Label Revision 0x27 4 R01A Platform Name 0x28 27 x86_64-accton_as5812_54t-r0 ONIE Version 0x29 13 2015.11.00.01 Manufacturer 0x2B 6 Accton Country Code 0x2C 2 TW Base MAC Address 0x24 6 C4:39:3A:FB:BF:6C Serial Number 0x23 14 581254T1631023 Part Number 0x22 13 FP1ZZ5654031A Product Name 0x21 15 5812-54T-O-AC-F MAC Addresses 0x2A 2 74 Vendor Name 0x2D 8 Edgecore CRC-32 0xFE 4 0xCBA5E40E Checksum is valid. HPE AL 6921-54X Version 2.16.1242. Copyright (C) 2013 American Megatrends, Inc. BIOS Date: 08/20/2015 10:55:33 Ver: ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 231 TLV Name Code Len Value -------------------- ---- --- ----- Manufacture Date 0x25 19 05/24/2016 14:49:30 Diag Version 0x2E 7 1.0.0.3 Label Revision 0x27 4 R01A Platform Name 0x28 27 x86_64-accton_as5812_54x-r0 ONIE Version 0x29 13 2015.11.00.01 Manufacturer 0x2B 6 Accton Country Code 0x2C 2 TW Base MAC Address 0x24 6 E0:07:1B:CB:20:50 Serial Number 0x23 10 TW65JQH009 Part Number 0x22 13 F0P8J5654000A Product Name 0x21 64 HPE Altoline 6921 48SFP+ 6QSFP+ x86 ONIE AC Front-to-Back Switch MAC Addresses 0x2A 2 74 Vendor Name 0x2D 26 Hewlett Packard Enterprise CRC-32 0xFE 4 0xADE27C84 Checksum is valid. AS5712_54X Version 2.16.1242. Copyright (C) 2013 American Megatrends, Inc. BIOS Date: 11/20/2014 10:55:31 Ver: ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 167 TLV Name Code Len Value -------------------- ---- --- ----- Manufacture Date 0x25 19 12/18/2014 11:22:02 Diag Version 0x2E 7 2.0.0.7 MAC Addresses 0x2A 2 74 Manufacturer 0x2B 6 Accton Country Code 0x2C 2 TW Vendor Name 0x2D 8 Edgecore Base MAC Address 0x24 6 70:72:CF:B7:65:44 Part Number 0x22 13 FP1ZZ5654001A Serial Number 0x23 14 571254X1419017 Label Revision 0x27 3 R0A Product Name 0x21 15 5712-54X-O-AC-F Platform Name 0x28 27 x86_64-accton_as5712_54x-r0 ONIE Version 0x29 13 2015.11.00.05 CRC-32 0xFE 4 0x37B6E65B Checksum is valid. N3248PXE-ON Version 2.19.1266. Copyright (C) 2019 American Megatrends, Inc. BIOS Date: 06/18/2019 23:21:39 Ver: 0ACHI040 ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 186 TLV Name Code Len Value -------------------- ---- --- ----- Product Name 0x21 11 N3248PXE-ON Part Number 0x22 6 0WYGRV Serial Number 0x23 20 TW0WYGRVDNT0097I0012 Base MAC Address 0x24 6 50:9A:4C:E6:7B:70 Manufacture Date 0x25 19 07/18/2019 17:41:23 Device Version 0x26 1 1 Label Revision 0x27 4 X01A Platform Name 0x28 32 x86_64-dellemc_n3248pxe_c3338-r0 ONIE Version 0x29 10 3.45.1.9-4 MAC Addresses 0x2A 2 128 Manufacturer 0x2B 5 DNT00 Country Code 0x2C 2 TW Vendor Name 0x2D 8 Dell EMC Diag Version 0x2E 11 3.00.3.41-2 Service Tag 0x2F 7 37QFXC2 2. Installing PicOS® on Bare Metal Switches 2.1 Introduction When using ONIE installer to install PicOS®, the installer reinstalls the software, rebuilds all the PicOS® file system. This can erase the configuration files and system logs from the previous installation. After a successful ONIE installation of PicOS® 4.x, the system generates multiple system partitions including PicOS® (partition size: 2G), PicOS®2 (partition size: 2G) and User-Data partitions. Among them, PicOS® and PicOS®2 are two independent system boot partitions. One of them is the active partition on which the running system resides, and the other is the inactive partition. The two-system-boot-partition feature allows the system to revert to a previous version of the installed software package when the it fails to upgrade PicOS® by using upgrade2 command. The ONIE installer removes all partitions to rebuild a brand new OS only when there is no User-Data partition. However, if there exists a User-Data partition (for example, install a new version 4.0.1 from the old one 4.0.0), the ONIE installer only rewrites the "PicOS®" partition, installs the new installation package to this partition and sets the system on "PicOS®" partition as the default and sole boot system. User-Data partition is a reserved partition which is not affected by ONIE installer and upgrade unless user manually removes it. User-Data partition uses all the available space left on the disk. Users can use this partition to store files and data. This document describes how to install PicOS® 4.x software using ONIE installer. 2.2 Installation Notes and Tools The installation methods used to install a new PicOS® are traditional installation and nos-boot-mode installation. You can choose a suitable installation method that is convenient and appropriate for your installation environment. If you want to install PicOS® through a console port, refer to sections "Traditional Installation" or "Nos-boot-mode Installation". If you want to install the PicOS® through a non-console port (through the management port), refer to section "Nos-boot-mode Installation". You need to log in through the console port of the switch and perform the ONIE installation. Other NOSes including user data will be removed when install PicOS® under ONIE environment. When the ONIE installer is used to downgrade the PicOS® version from version 4.x to PicOS® 3.x or lower versions, we first need to use ONIE to uninstall the higher version PICOS before proceeding with installing PicOS® 3.x or a lower version. On the ARM platform, execute the onie_uninstaller command at the ONIE prompt to uninstall the current version PicOS®. On the x86 platform, select the "ONIE: Uninstall OS" option in the GRUB menu to uninstall the current version PicOS®. If you enter GRUB rescue mode and the switch has GPT format partition, you can use the following commands to reset the GRUB boot variable to enter ONIE GRUB and then install PicOS®. grub rescue> set prefix=(hd0,gpt2)/grub grub rescue> set root=(hd0,gpt2) grub rescue> insmod normal grub rescue> normal Do not plug in the USB disk during onie-nos-installer process until ONIE starts up. If you have plugged in the USB disk before the installation operation, ONIE will find the installer on the USB disk when beginning the installation. On AS4610 series switches, when installation is complete, the installer will display: Please take out the usb disc, then remove the USB disk within 10 seconds after installation successful, and before machine restarts. All X86 platforms share one installation and upgrade package with the name fixed as: onie-installer-picos-VERSION-x86.bin, where VERSION is the release version. X86 platform are listed below: FS N9550-32D FS N8550-64C FS N5850-48S6Q FS N8550-48B8C FS N8550-32C FS N8560-64C FS N8550-24CD8D FS N9600-64OD Edgecore AS4630-54PE Edgecore AS5712-54X Edgecore AS5812-54T Edgecore AS5812-54X Edgecore AS7312-54X Edgecore AS7326-56X Edgecore AS7712-32X Edgecore AS7726-32X Edgecore AS7816-64X Edgecore AS5835-54X Edgecore AS9716-32D DELL N3248P-ON DELL N3248PXE-ON DELL N3224PX-ON DELL N3248X-ON DELL S4048-ON DELL S4148F-ON DELL S4148T-ON DELL S4128F-ON DELL S5224F-ON DELL S5296F-ON DELL S5212F-ON DELL S5248F-ON DELL Z9100-ON DELL Z9264F-ON DELL N3224T-ON DELL S4128T-ON 2.2.1 What is ONIE ONIE (Open Network Install Environment) is an open source project of OCP (Open Compute Project). ONIE provides the environment to install any network operating system on a bare metal network switch. ONIE liberates users from captive pre-installed network operating systems, like the Cisco IOS, and provides them with a choice. ONIE is a small Linux operating system that comes pre-installed as firmware on bare metal network switches. ONIE acts as an enhanced boot loader, extending the features provided by U-Boot. ONIE is used to install Pica8 PicOS® on compatible switches. The bare metal switches listed in the PICOS Hardware Compatibility List must be pre-loaded with ONIE prior to installing PicOS®. 2.3 Traditional Installation NOTE: You need to log in through the console port of the switch and perform the ONIE installation described in this section. The installation method described in this section only applies to platforms that have pre-installed ONIE. 2.3.1 Mind Map of Installation Process Figure1 shows the mind map of PicOS® installation process. Figure1 Mind Map of PicOS® installation process image.png 2.3.2 Manual Installation Process The following example describes the installation of PicOS® via manual installation method. Step1 Make sure that the installation package of .bin file has been load to the server (server could be HTTP, TFTP, or an FTP server or the switch local directory depending on the actual installation environment). Step2 Enter ONIE installation environment. The process is different on the following two types of platforms: ARM Platforms (AS4610 Series Switches) x86 Platform ARM Platforms (AS4610 Series Switches) a) Verify that the switch is pre-loaded with ONIE, which will be used to load PicOS® on the switch. Power on the switch and interrupt the boot sequence by pressing any key when the following line is shown: Hit any key to stop autoboot: b) User will then reach the U-Boot command prompt indicated by ->. Run the printenv command at the U-Boot prompt. If the information displayed contains keywords like onie_initargs and onie_machine, the switch is pre-loaded with ONIE. LOADER->printenv active=image1 autoload=no baudrate=115200 bootcmd=run check_boot_reason;run PicOS_bootcmd;run onie_bootcmd bootdelay=10 check_boot_reason=if test -n $onie_boot_reason; then setenv onie_bootargs boot_reason=$onie_boot_reason; run onie_bootcmd; fi; consoledev=ttyS0 dhcp_user-class=arm-accton_as4610_54-r0_uboot dhcp_vendor-class-identifier=arm-accton_as4610_54-r0 ethact=eth-0 ethaddr=00:18:23:30:E7:8F fdtaddr=0xc00000 fpboot=setenv bootargs console=${consoledev},${baudrate} maxcpus=2 mem=1024M root=/dev/ram ${mtdparts} ubi.mtd=4 ethaddr=$ethaddr quiet gatewayip=192.168.0.1 initrd_high=0x80000000 ipaddr=192.168.0.1 loadaddr=0x70000000 loads_echo=1 mfg=mfg mfgdiags=run fpboot ; nand read ${loadaddr} diags ; bootm ${loadaddr} mfgdiags_recovery=nand read ${loadaddr} diags2 ; nand erase.part diags ; nand write ${loadaddr} diags mtdids=nand0=nand_iproc.0 mtdparts=mtdparts=nand_iproc.0:1m(uboot),2m(shmoo),1m(nenv),12m(onie),3992m(open),12m(onie2),2m(vpd),6m(sys_eeprom),16m(diags),16m(diags2),32m(diags_fs) netmask=255.255.255.0 nos_bootcmd=true onie_args=run onie_initargs onie_platformargs onie_bootcmd=echo Loading Open Network Install Environment ...; echo Platform: $onie_platform ; echo Version : $onie_version ; nand read $loadaddr $onie_start 0x00c00000 && run onie_args && bootm ${loadaddr} onie_dropbear_dss_host_key=begin-base64@600@d#AAAAB3NzaC1kc3MAAACBAIN7HOS7UGtQ+RS9R5Rdim9s4iadCBQ9SEFnHJZ2#ulK15hN2p1BOJ1Mf4qb/oHFGIt8hvopq157ejsJcSPuR9scXE2aYQO7r1+Ie#1MKoR3HyEFKgPhNUr0qYNiIaWGw2UUXivLUlhjmaPhjItsttb6AezNB6N1ap#TmIeEUse0NQBAAAAFQDndwbRrSsw6G/W4wd0LJVAjuyq2QAAAIAe/zGPyPNn#UwwV+i+j3l1W9IFhjA/ovXfX7PQtjHB7OJcInSpOA2gXLXHU2kYDkn+ymJQI#8Tn558nLHq64n9hIJzwaQH4ajMipBNwqR0WtpPXEaow9InDzjs+qFY0HAcTv#7DMEY9BGiJAUUSSCSFZ9dEYHIWUdk6WIpDUMX4b2ewAAAIB6bC+fHzr+Qaet#GjzynI0tApbzyydXKuIiIH6EDh2QEaP0E+TSxJ+C4xfyBAp1j0kvj0IYWR2P#H9ur0RaxDaCmKwIQs1gTJh/137Yd+OsqEV3JnrZxlEKk2DmI5c2wrGtl4oUp#XJfc+viahpFeCsGzsqGHHADWNsjlpKt457QCuQAAABUAk5406cTH4nZO0qlj#6irYf4WA65E=#====# onie_dropbear_rsa_host_key=begin-base64@600@r#AAAAB3NzaC1yc2EAAAADAQABAAAAgQCMTqwNhnJpuSLYAdRA/jjm1lyBaJF1#ovs3Hp0G7XkYnY4+JNPTCYgnmfMQnM83PQncuy89AqehJ2V22LGjpRiqT56K#MRr+hQoSWEbAObRd1azZF45pbxiQaQiQxNzIKbHDDWlGlycXfv8w9ZCElbxj#Ja7bkwmwg9EsBlW0d5u0BQAAAIAFr0FOyfn0OR1FiatvF624Aorcbl9oV/pc#JRghGfl8SxPihizz4bC7xAPCUkwd9ZHi+M2E6AjhIV69xjFKS0vYuQplvl8G#9R8YsnmP5B45TyLE3dW5V2/g+LQERQdFpRaSsPqEPHSlXPq4XHLGLRFItEBt#ohp41Qm+eA6efsAMIQAAAEEA4Y90xi8N1SuwjRk53fqpP8dC+FPnU850XtC1#cKG0rBt6v9qD+BTxxfE6GEpYM+N0fLyECbgBjA2LQF6CG3G15QAAAEEAnz3v#3POrcsMK2LkSNjWzAhzUqOWyOaNlhcvgh+2Xfj2tHyOTpZ09gCm483v1rui9#63uYu4QQurpATrHMcLIjoQ==#====# onie_initargs=setenv bootargs quiet console=$consoledev,$baudrate onie_machine=accton_as4610_54 onie_machine_rev=0 onie_platform=arm-accton_as4610_54-r0 onie_platformargs=setenv bootargs $bootargs serial_num=${serial#} ${platformargs} eth_addr=$ethaddr $onie_bootargs $onie_debugargs onie_recovery=nand read ${loadaddr} onie2 ; nand erase.part onie ; nand write ${loadaddr} onie onie_rescue=setenv onie_boot_reason rescue && boot onie_start=onie onie_sz.b=0x00c00000 onie_uninstall=setenv onie_boot_reason uninstall && boot onie_update=setenv onie_boot_reason update && boot onie_vendor_id=27658 onie_version=master-201603091701-dirty PicOS_bootcmd=usb start;run platformargs;setenv bootargs root=/dev/sda1 rw noinitrd console=$consoledev,$baudrate rootdelay=10 $mtdparts;ext2load usb 0:1 $loadaddr boot/uImage;bootm $loadaddr platform=accton_as4610_54 platformargs=mtdparts=nand_iproc.0:1m(uboot),2m(shmoo),1m(nenv),12m(onie),3992m(open),12m(onie2),2m(vpd),6m(sys_eeprom),16m(diags),16m(diags2),32m(diags_fs) maxcpus=2 mem=1024M ramdiskaddr=0x3000000 serial#=A626P1DL174300014 serverip=192.168.0.10 stderr=serial stdin=serial stdout=serial ubifscfg=ubi part nand0,4 0x0; ubifsmount fs ver=U-Boot 2012.10-gcbef171 (Mar 09 2016 - 17:01:14) - ONIE master-201603091701-dirty Environment size: 3992/65532 bytes c) From U-Boot prompt, boot ONIE in rescue mode. LOADER-> run onie_rescue x86 Platform On x86 platform, it uses GRUB menu to install OS via ONIE. a) Reboot the system, and enter ONIE installation environment from the GRUB menu: +----------------------------------------------------------------------------+ | PicOS | |*ONIE | | | | | | | | | | | | | | | | | | | | | +----------------------------------------------------------------------------+ Use the ^ and v keys to select which entry is highlighted. Press enter to boot the selected OS, `e' to edit the commands before booting or `c' for a command-line. b) From GRUB prompt, choose ONIE: Rescue to Install OS, boot ONIE in rescue mode. GNU GRUB version 2.02~beta2+e4a1fe391 +----------------------------------------------------------------------------+ |*ONIE: Install OS | | ONIE: Rescue | | ONIE: Uninstall OS | | ONIE: Update ONIE | | ONIE: Embed ONIE | | DIAG: Accton Diagnostic | | | | | | | | | | | | | +----------------------------------------------------------------------------+ Step3 Run onie-nos-install command as follows to manually install PicOS®. Install via TFTP ONIE# onie-nos-install tftp:///PICOS.bin Install via FTP When installing via FTP, you need to type username and password of the FTP server on which the image file is loaded. ONIE# onie-nos-install ftp://username:password@/PICOS.bin Install via HTTP ONIE# onie-nos-install http:///PICOS.bin Install from Local Directory a) In ONIE rescue mode, copy the image file to the current directory. ONIE# scp username@/PICOS.bin . b) Run onie-nos-install command to start installation. ONIE# onie-nos-install PICOS.bin For example, ONIE:/ # onie-nos-install onie-installer-picos-4.0.0-8b1219e112-x86.bin discover: Rescue mode detected. No discover stopped. ONIE: Executing installer: onie-installer-picos-4.0.0-8b1219e112-x86.bin Verifying image checksum ... OK. Preparing image archive ... OK. [1] PICOS L2/L3 (default) [2] PICOS Open vSwitch/OpenFlow Enter your choice (1,2):1 PICOS L2/L3 is selected. ONIE installation will overwrite the configuration file of existing system. It is recommended to follow the upgrade procedure to upgrade the system. Press any key to stop the installation... 10 9 8 7 6 5 4 3 2 1 ... The installer runs automatically, before start installation, it will prompt to choose the option to make PicOS® to boot into L2/L3 or OVS mode. If not selected, then PicOS® boots into L2/L3. After finishing installation, the device reboots automatically, the system then comes up running the new network operating system. NOTE: After the system restarts, you need to enter the username and password, the initial login username is admin and password is pica8. After the username and password are entered, user will be asked to choose a new password for admin. This is the only post installation step after which the PicOS® operating system can be used. 2.3.3 Automatic Installation Process The automatic installation process uses the DHCP message exchange process to download and install software packages. Step1 Make sure the switch is connected to DHCP and HTTP servers and the PicOS® installation software package is downloaded to the HTTP server. a) DHCP server configuration: define the path of the installation package and then start DHCP server service: host pica8-3922 { hardware ethernet 70:72:cf:12:34:56; fixed-address 192.168.2.50; option default-url = "http://192.168.2.42/onie-installer-picos-4.0.0-8b1219e112-x86.bin"; } b) Check if the .bin installation file is loaded onto the HTTP server: root@dev:/var/www# ls index.html onie-installer-powerpc.bin Step2 Install PicOS® via ONIE. The process is different on the following two types of platforms: ARM Platforms (AS4610 Series Switches) x86 Platform ARM Platforms (AS4610 Series Switches) a) Verify that the switch is pre-loaded with ONIE, which will be used to load PicOS® on the switch. Power on the switch and interrupt the boot sequence by pressing any key when the following line is shown: Hit any key to stop autoboot: b) User will then reach the U-Boot command prompt indicated by ->. Run the printenv command at the U-Boot prompt. If the information displayed contains keywords like onie_initargs and onie_machine, the switch is pre-loaded with ONIE. LOADER-> printenv active=image1 autoload=no baudrate=115200 bootcmd=run check_boot_reason;run PicOS_bootcmd;run onie_bootcmd bootdelay=10 check_boot_reason=if test -n $onie_boot_reason; then setenv onie_bootargs boot_reason=$onie_boot_reason; run onie_bootcmd; fi; consoledev=ttyS0 dhcp_user-class=arm-accton_as4610_54-r0_uboot dhcp_vendor-class-identifier=arm-accton_as4610_54-r0 ethact=eth-0 ethaddr=00:18:23:30:E7:8F fdtaddr=0xc00000 fpboot=setenv bootargs console=${consoledev},${baudrate} maxcpus=2 mem=1024M root=/dev/ram ${mtdparts} ubi.mtd=4 ethaddr=$ethaddr quiet gatewayip=192.168.0.1 initrd_high=0x80000000 ipaddr=192.168.0.1 loadaddr=0x70000000 loads_echo=1 mfg=mfg mfgdiags=run fpboot ; nand read ${loadaddr} diags ; bootm ${loadaddr} mfgdiags_recovery=nand read ${loadaddr} diags2 ; nand erase.part diags ; nand write ${loadaddr} diags mtdids=nand0=nand_iproc.0 mtdparts=mtdparts=nand_iproc.0:1m(uboot),2m(shmoo),1m(nenv),12m(onie),3992m(open),12m(onie2),2m(vpd),6m(sys_eeprom),16m(diags),16m(diags2),32m(diags_fs) netmask=255.255.255.0 nos_bootcmd=true onie_args=run onie_initargs onie_platformargs onie_bootcmd=echo Loading Open Network Install Environment ...; echo Platform: $onie_platform ; echo Version : $onie_version ; nand read $loadaddr $onie_start 0x00c00000 && run onie_args && bootm ${loadaddr} onie_dropbear_dss_host_key=begin-base64@600@d#AAAAB3NzaC1kc3MAAACBAIN7HOS7UGtQ+RS9R5Rdim9s4iadCBQ9SEFnHJZ2#ulK15hN2p1BOJ1Mf4qb/oHFGIt8hvopq157ejsJcSPuR9scXE2aYQO7r1+Ie#1MKoR3HyEFKgPhNUr0qYNiIaWGw2UUXivLUlhjmaPhjItsttb6AezNB6N1ap#TmIeEUse0NQBAAAAFQDndwbRrSsw6G/W4wd0LJVAjuyq2QAAAIAe/zGPyPNn#UwwV+i+j3l1W9IFhjA/ovXfX7PQtjHB7OJcInSpOA2gXLXHU2kYDkn+ymJQI#8Tn558nLHq64n9hIJzwaQH4ajMipBNwqR0WtpPXEaow9InDzjs+qFY0HAcTv#7DMEY9BGiJAUUSSCSFZ9dEYHIWUdk6WIpDUMX4b2ewAAAIB6bC+fHzr+Qaet#GjzynI0tApbzyydXKuIiIH6EDh2QEaP0E+TSxJ+C4xfyBAp1j0kvj0IYWR2P#H9ur0RaxDaCmKwIQs1gTJh/137Yd+OsqEV3JnrZxlEKk2DmI5c2wrGtl4oUp#XJfc+viahpFeCsGzsqGHHADWNsjlpKt457QCuQAAABUAk5406cTH4nZO0qlj#6irYf4WA65E=#====# onie_dropbear_rsa_host_key=begin-base64@600@r#AAAAB3NzaC1yc2EAAAADAQABAAAAgQCMTqwNhnJpuSLYAdRA/jjm1lyBaJF1#ovs3Hp0G7XkYnY4+JNPTCYgnmfMQnM83PQncuy89AqehJ2V22LGjpRiqT56K#MRr+hQoSWEbAObRd1azZF45pbxiQaQiQxNzIKbHDDWlGlycXfv8w9ZCElbxj#Ja7bkwmwg9EsBlW0d5u0BQAAAIAFr0FOyfn0OR1FiatvF624Aorcbl9oV/pc#JRghGfl8SxPihizz4bC7xAPCUkwd9ZHi+M2E6AjhIV69xjFKS0vYuQplvl8G#9R8YsnmP5B45TyLE3dW5V2/g+LQERQdFpRaSsPqEPHSlXPq4XHLGLRFItEBt#ohp41Qm+eA6efsAMIQAAAEEA4Y90xi8N1SuwjRk53fqpP8dC+FPnU850XtC1#cKG0rBt6v9qD+BTxxfE6GEpYM+N0fLyECbgBjA2LQF6CG3G15QAAAEEAnz3v#3POrcsMK2LkSNjWzAhzUqOWyOaNlhcvgh+2Xfj2tHyOTpZ09gCm483v1rui9#63uYu4QQurpATrHMcLIjoQ==#====# onie_initargs=setenv bootargs quiet console=$consoledev,$baudrate onie_machine=accton_as4610_54 onie_machine_rev=0 onie_platform=arm-accton_as4610_54-r0 onie_platformargs=setenv bootargs $bootargs serial_num=${serial#} ${platformargs} eth_addr=$ethaddr $onie_bootargs $onie_debugargs onie_recovery=nand read ${loadaddr} onie2 ; nand erase.part onie ; nand write ${loadaddr} onie onie_rescue=setenv onie_boot_reason rescue && boot onie_start=onie onie_sz.b=0x00c00000 onie_uninstall=setenv onie_boot_reason uninstall && boot onie_update=setenv onie_boot_reason update && boot onie_vendor_id=27658 onie_version=master-201603091701-dirty PicOS_bootcmd=usb start;run platformargs;setenv bootargs root=/dev/sda1 rw noinitrd console=$consoledev,$baudrate rootdelay=10 $mtdparts;ext2load usb 0:1 $loadaddr boot/uImage;bootm $loadaddr platform=accton_as4610_54 platformargs=mtdparts=nand_iproc.0:1m(uboot),2m(shmoo),1m(nenv),12m(onie),3992m(open),12m(onie2),2m(vpd),6m(sys_eeprom),16m(diags),16m(diags2),32m(diags_fs) maxcpus=2 mem=1024M ramdiskaddr=0x3000000 serial#=A626P1DL174300014 serverip=192.168.0.10 stderr=serial stdin=serial stdout=serial ubifscfg=ubi part nand0,4 0x0; ubifsmount fs ver=U-Boot 2012.10-gcbef171 (Mar 09 2016 - 17:01:14) - ONIE master-201603091701-dirty Environment size: 3992/65532 bytes c) Input command run onie_bootcmd, which will automatically install PicOS® on the switch. LOADER -> run onie_bootcmd Loading Open Network Install Environment ... Platform: arm-accton_as4610_54-r0 Version : 2021.09.00.03 WARNING: adjusting available memory to 30000000 ## Booting kernel from Legacy Image at 02000000 ... Image Name: as4610_54x.1.6.1.3 Image Type: ARM Linux Multi-File Image (gzip compressed) Data Size: 3514311 Bytes = 3.4 MiB Load Address: 00000000 Entry Point: 00000000 Contents: Image 0: 2762367 Bytes = 2.6 MiB Image 1: 733576 Bytes = 716.4 KiB Image 2: 18351 Bytes = 17.9 KiB Verifying Checksum ... OK ## Loading init Ramdisk from multi component Legacy Image at 02000000 ... ## Flattened Device Tree from multi component Image at 02000000 Booting using the fdt at 0x02355858 Uncompressing Multi-File Image ... OK Loading Ramdisk to 2ff4c000, end 2ffff188 ... OK Loading Device Tree to 03ff8000, end 03fff7ae ... OK Cannot reserve gpages without hugetlb enabled setup_arch: bootmem as4610_54x_setup_arch() arch: exit pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01) sd 0:0:0:0: [sda] No Caching mode page present sd 0:0:0:0: [sda] Assuming drive cache: write through sd 0:0:0:0: [sda] No Caching mode page present sd 0:0:0:0: [sda] Assuming drive cache: write through sd 0:0:0:0: [sda] No Caching mode page present sd 0:0:0:0: [sda] Assuming drive cache: write through ONIE: Using DHCPv4 addr: eth0: 192.168.2.77 / 255.255.255.0 discover: installer mode detected. Running installer. Please press Enter to activate this console. ONIE: Using DHCPv4 addr: eth0: 192.168.2.77 / 255.255.255.0 ONIE: Starting ONIE Service Discovery ONIE: Executing installer: http://192.168.2.42/onie-installer-picos-4.0.0-8b1219e112-x86.bin Verifying image checksum ... OK. Preparing image archive ... OK. PicOS installation .............................................. ./var/local/ ./var/run Setup PicOS environment ... .............................................. XorPlus login: admin Password: You are required to change your password immediately (root enforced) Changing password for admin. (current) UNIX password: Enter new UNIX password: Retype new UNIX password: admin@XorPlus$ x86 Platform On x86 platform, it uses GRUB menu to choose install OS via ONIE. a) Reboot the system, and enter ONIE installation environment from the GRUB menu: +----------------------------------------------------------------------------+ | PicOS | |*ONIE | | | | | | | | | | | | | | | | | | | | | +----------------------------------------------------------------------------+ Use the ^ and v keys to select which entry is highlighted. Press enter to boot the selected OS, `e' to edit the commands before booting or `c' for a command-line. b) From GRUB prompt, choose ONIE: Rescue to Install OS, boot ONIE in rescue mode. GNU GRUB version 2.02~beta2+e4a1fe391 +----------------------------------------------------------------------------+ |*ONIE: Install OS | | ONIE: Rescue | | ONIE: Uninstall OS | | ONIE: Update ONIE | | ONIE: Embed ONIE | | DIAG: Accton Diagnostic | | | | | | | | | | | | | +----------------------------------------------------------------------------+ The installer runs and will reboot the system after installation is complete. NOTE: After the system restarts, you need to enter the username and password, the initial login username is admin and password is pica8. After the username and password are entered, user will be asked to choose a new password for admin. This is the only post installation step after which the PicOS® operating system can be used. 2.4 Nos-boot-mode Installation NOTE: The installation method described in this section applies to installation through both the console port and the management port. The installation method described in this section only applies to platforms that have pre-installed ONIE. The installation methods described in "Traditional Installation" must be performed through the console port. If you want to install the system through a non-console port, you can use the nos-boot-mode command to perform the installation which is described in this section. Usage of nos-boot-mode command: admin@Xorplus$sudo nos-boot-mode USAGE install or uninstall NOS(es) SYNOPSIS nos-boot-mode [install|uninstall] DESCRIPTION install- Install NOS uninstall- Remove all NOS(es) including PicOS® When nos-boot-mode install command is executed, PicOS® will switch to ONIE install mode, and the user should go on to complete the subsequent installation. The steps for the manual installation process and the automatic installation process using the nos-boot-mode install command are described below. When nos-boot-mode unsinstall command is executed, the system will remove all NOS(es) including PicOS® from the device. Therefore, it is suggested to use the nos-boot-mode unsinstall command with caution. 2.4.1 Manual Installation Process Step1 Make sure that the installation package of .bin file has been loaded to the server (server could be HTTP, TFTP, or an FTP server or the switch local directory depending on the actual installation environment). Step2 Execute the nos-boot-mode install command to enter ONIE installation environment. admin@Xorplus:~$ sudo nos-boot-mode install Step3 Type “yes” when the below prompt is shown, which will take the system will to ONIE install mode. Type 'yes' to install NOS! Type 'no' to exit [no]/yes: Step4 Run onie-nos-install command as follows to manually install PicOS®. Install via TFTP ONIE# onie-nos-install tftp:///PICOS.bin Install via FTP When installing via FTP, you need to type in the username and password for the FTP server on which the image file is loaded. ONIE# onie-nos-install ftp://username:password@/PICOS.bin Install via HTTP ONIE# onie-nos-install http:///PICOS.bin Install from Local Directory a) In ONIE rescue mode, copy the image file to the current directory. ONIE# scp username@/PICOS.bin . b) Run onie-nos-install command to start installation. ONIE# onie-nos-install PICOS.bin For example, ONIE:/ # onie-nos-install onie-installer-picos-4.0.0-8b1219e112-x86.bin discover: Rescue mode detected. No discover stopped. ONIE: Executing installer: onie-installer-picos-4.0.0-8b1219e112-x86.bin Verifying image checksum ... OK. Preparing image archive ... OK. [1] PICOS L2/L3 (default) [2] PICOS Open vSwitch/OpenFlow Enter your choice (1,2):1 PICOS L2/L3 is selected. ONIE installation will overwrite the configuration file of existing system. It is recommended to follow the upgrade procedure to upgrade the system. Press any key to stop the installation... 10 9 8 7 6 5 4 3 2 1 ... The installer runs automatically, before start installation, it will prompt to choose the option to make PicOS® to boot into L2/L3 or OVS mode. If not selected, then PicOS® boots into L2/L3. After finishing installation, the device reboots automatically, the system then comes up running the new network operating system. NOTE: After the system restarts, you need to enter the username and password, the initial login username is admin and password is pica8. After the username and password are entered, user will be asked to choose a new password for admin. This is the only post installation step after which the PicOS® operating system can be used. 2.4.2 Automated Installation Process The automatic installation process uses the DHCP message exchange process to download and install software packages. Step1 Make sure the switch is connected to DHCP and HTTP servers, and the PicOS® installation software package is downloaded to the HTTP server. a) DHCP server configuration: define the path of the installation package and then start DHCP server service: host pica8-3922 { hardware ethernet 70:72:cf:12:34:56; fixed-address 192.168.2.50; option default-url = "http://192.168.2.42/onie-installer-picos-4.0.0-8b1219e112-x86.bin"; } b) Check if the .bin installation file is loaded onto the HTTP server: root@dev:/var/www# ls index.html onie-installer-powerpc.bin Step2 Execute the nos-boot-mode install command to enter ONIE installation environment. admin@Xorplus$ sudo nos-boot-mode install Step3 Type “yes” when the below prompt is shown, and the system will automatically complete the installation. Type 'yes' to install NOS! Type 'no' to exit [no]/yes: The installer runs automatically and will reboot the system after installation is completed. NOTE: After the system restarts, you need to enter the username and password, the initial login username is admin and password is pica8. After the username and password are entered, user will be asked to choose a new password for admin. This is the only post installation step after which the PicOS® operating system can be used. 2.5 Verifying Version after Installation After system reboots automatically, the system will come up running the new network operating system. admin@Xorplus> show version Copyright (C) 2009-2022 Pica8, Inc. =================================== Hardware Model : as7312_54x Linux System Version/Revision : 4.0.0/8b1219e112 Linux System Released Date : 5/18/2021 L2/L3 Version/Revision : 4.0.0/8b1219e112 L2/L3 Released Date : 5/18/2021 OVS/OF Version/Revision : 4.0.0/8b1219e112 OVS/OF Released Date : 5/18/2021 2.6 Appendix: Troubleshooting Installation/Upgrade Failure on AS7326-56X Installation or upgrade failure (for example, the switches cannot boot up after install) may occur on the old AS7326-56X hardware models (revision is R01F and before). When booting PicOS® on AS7326-56X and detect hardware rev R01F, the system will log a warning message to prompt the hardware revision R01F is a pre-production hardware reversion: "This hardware revision R01F is a pre-production hardware rev, PicOS® has applied a work around to work with PicOS®. Support will be provided on a best effort basis". To work around the issue, first we need to check the “Label Revision”. If it is an old hardware model (revision is R01F or before), then, we can perform the following provided solution after installation/upgrade to solve the problem. 2.6.1 Check Label Revision Under ONIE prompt, run “onie_syseeprom” to get the “Label Revision”. ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 166 TLV Name Code Len Value -------------------- ---- --- ----- Manufacture Date 0x25 19 04/27/2019 02:10:06 Label Revision 0x27 4 R01B Platform Name 0x28 27 x86_64-accton_as7326_56x-r0 ONIE Version 0x29 13 2018.05.00.05 Manufacturer 0x2B 6 Accton Diag Version 0x2E 7 0.0.1.0 Base MAC Address 0x24 6 80:A2:35:81:D5:F0 Serial Number 0x23 14 732656X1916012 Country Code 0x2C 2 TW Part Number 0x22 13 FP4ZZ7656005A Product Name 0x21 15 7326-56X-O-AC-F MAC Addresses 0x2A 2 256 Vendor Name 0x2D 6 Accton CRC-32 0xFE 4 0xC3D3F2DE Checksum is valid. ONIE:/ # 2.6.2 Solution You can follow the steps below after installation/upgrade, to fix the problem of installation and upgrade failure on the old AS7326-56X hardware model (revision R01F or before). Step1 Power cycle the switch. Step2 From the GRUB menu, choose “ONIE” to enter ONIE GRUB menu: +----------------------------------------------------------------------------+ | PicOS | |*ONIE | | | | | | | | | | | | | | | | | | | | | +----------------------------------------------------------------------------+ Use the ^ and v keys to select which entry is highlighted. Press enter to boot the selected OS, `e' to edit the commands before booting or `c' for a command-line. Step3 From ONIE GRUB menu, choose “ONIE: Rescue” to launch ONIE in Rescue mode. GNU GRUB version 2.02~beta2+e4a1fe391 +----------------------------------------------------------------------------+ | ONIE: Install OS | |*ONIE: Rescue | | ONIE: Uninstall OS | | ONIE: Update ONIE | | ONIE: Embed ONIE | | DIAG: Accton Diagnostic | | | | | | | | | | | | | +----------------------------------------------------------------------------+ Step4 Press Enter to display the ONIE prompt. Step5 Mount PicOS® partition with label is “PicOS®”. ONIE:/ # blkid /dev/sda7: LABEL="User-Data" UUID="be63cef8-4560-4c48-ab5a-8f7ced5a950b" /dev/sda6: LABEL="PicOS2" UUID="f589e53f-4cd1-44ba-8384-f339f4e2b2ac" /dev/sda5: LABEL="PicOS" UUID="8ca5f7ed-5a15-4a2a-944c-4d8872647bf5" /dev/sda4: LABEL="PICOS-GRUB" UUID="782a1372-4b66-4783-b920-dab1df8ec6e4" /dev/sda3: LABEL="ACCTON-DIAG" UUID="3e4117d0-1926-472a-9d9e-08883df83d40" /dev/sda2: LABEL="ONIE-BOOT" UUID="1a90abd8-f065-4f7a-90a0-af122b8805fa" ONIE:/ # ONIE:/ # mount /dev/sda5 /mnt Step6 Execute the following command to modify the I2C access address. ONIE:/ # sed -I "s/0x57/0x56/" /mnt/etc/rc_hw.sh ONIE:/ # sync Step7 Unmount the PicOS® partition. ONIE:/ # unmount /dev/sda5 Step8 Reboot the switch. ONIE:/ # reboot 3. Upgrading PicOS® from Version 4.0.0 or Later Using Upgrade Command NOTE: This document ONLY applies to upgrade from version 4.0.0 or the later version using the upgrade command. If you want to upgrade PicOS® from the version before 4.0.0, use ONIE installation process described in "Installing PicOS® on Bare Metal Switches". This upgrading guide is not available for FS S5810 Series and S5860 Series switches. N8560-32C uses the ONIE method for upgrade described in this guide, while the installation uses Rboot method, please refer to "Installing PicOS® for FS S5810/S5860 Series and N8560-32C Switches" for details on the installation process. The installation package name for N8560-32C includes the suffix '-rboot', for example, N8560_picos-4.4.5-9bca0916a3-rboot.bin. The upgrade package, on the other hand, includes the suffix '-x86', such as picos-4.4.5-9bca0916a3-x86.bin. 3.1 Partitioning PicOS® 4.0.0 have multiple system partitions including PicOS® (partition size: 2G), PicOS®2 (partition size: 2G) and User-Data partitions. Among them, PicOS® and PicOS®2 are two independent system boot partitions. One of them is the active partition on which the running system resides, and the other is the inactive partition. The two-system-boot-partition feature allows the system to be reverted to a previous version of the installed software package when it fails to upgrade PicOS®. User-Data partition is a reserved partition which is not affected by ONIE installer and upgrade unless user manually removes it. User-Data partition uses all the available space left on the disk after installation. Users can use this partition to store files and data. 3.2 Supported Platforms PicOS® 4.x software requires to run on a high performance device, only the platforms listed in Switch Machine Outline and System Characteristics are supported upgrading to PicOS® 4.x. 3.3 Preparation before Upgrading NOTE If routed interface is configured, before upgrade, make sure that routed interface name and sub-interface name in the configuration file start with the string "rif-". Otherwise, upgrade will fail due to configuration error. Table 1 Checklist before Upgrading No. Checking Items Checking Standard Results 1 Checking the Running PicOS® Version The currently running system software version is lower than the software version to be installed 2 Checking License Validation Run the license -s command to verify that the license expiration date extends beyond the planned upgrade date. If the license is close to expiration, consider renewing it to avoid interruptions. 3 Building Upgrade Environment Build a different upgrade environment according to the need 4 Getting the Required Upgrade Software Obtain the required supported upgrade software 5 Backing up Important Data in Flash All the important data in Flash is backed up 6 Checking Available Flash Space Flash space is enough to save upgrading package and other files 3.3.1 Checking the Running PicOS® Version Use the version command to check the version of the running system software. admin@PICOS:~$ version Copyright (C) 2009-2024 Pica8, Inc. =================================== Base ethernet MAC Address : d4:31:27:c9:e4:51 Hardware Model : S5860-48MG-U Linux System Version/Revision : 9.8.7-main/fd87d25a10 Linux System Released Date : 10/12/2024 L2/L3 Version/Revision : 9.8.7-main/fd87d25a10 L2/L3 Released Date : 10/12/2024 OVS/OF Version/Revision : 9.8.7-main/fd87d25a10 OVS/OF Released Date : 10/12/2024 3.3.2 Checking License Validation Before performing an upgrade, users can run the license -s command to check if the current license has expired, ensuring it is valid and preventing upgrade failure due to license expiration. admin@PICOS:~$ license -s { "Type":"1GE", "Feature":["Base Product", "Layer3", "OpenFlow"], "Support End Date":"2025-10-28", "Hardware ID":"ACD2-F77A-BBA3-2849", "Site Name":"PICA8" } 3.3.3 Building Upgrade Environment Please make sure that you have set up an HTTP, TFTP or FTP protocol upgrading environment, the basic requirements are as follows: PC can log in to the device through serial or SSH. The communication between the server and the device works well. The upgrading file used by the device has already been stored on the server. 3.3.4 Getting the Required Upgrade Software Please contact Pica8 technical support engineers at the following website for the latest version of upgrade software. https://www.pica8.com/support/ 3.3.5 Backing up Important Data in Flash Before upgrading, save the important data in Flash to the local PC through FTP or TFTP, and then upload it to the switch after the upgrade is completed. 3.3.6 Checking the Available Flash Space Use the df -h command to check the available flash space for saving the upgrade package. admin@PICOS:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 989M 0 989M 0% /dev overlay 706M 57M 650M 8% / tmpfs 1009M 0 1009M 0% /dev/shm tmpfs 404M 5.9M 398M 2% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 50M 192K 50M 1% /tmp /dev/ubi1_0 863M 376M 483M 44% /mnt/open 3.4 Upgrading Notes Downgrade from PicOS® version 4.x to 3.x or to a lower version is NOT supported by using upgrade command. You can use ONIE installation when you want to downgrade. For details about ONIE installation, please refer to "Installing PicOS® on Bare Metal Switches". License check is performed for upgrade: If PicOS® has a license installed before the upgrade, the license will be copied and activated after the upgrade. Please check this section for the PICOS Licenses. If there is no license installed prior to upgrade, upgrade2 process can proceed but only the first four ports and the first two uplink ports (if exist) on the newly upgraded system can be used. If the service has expired, it is not allowed to upgrade a major release (e.g. 4.1 to 4.2). However, it will not affect upgrading to a minor release (e.g. 4.1.1 to 4.1.2). You can log in to the switch through its console port or using SSH. After successful login, you can run commands on the command line interface (CLI) to upgrade the device. When using FTP/TFTP to download the image, user should verify that the "binary" mode is being used. If the "binary" transfer mode is not being used, the image might be modified during download, and the upgrade will fail during the MD5 check. When upgrading, the installer checks whether there is a user-data partition. If there exists a User-Data partition, the installer only rewrites the running system boot partition (PicOS®/ PicOS®2) and installs the new installation package to this partition. However, if there is no User-Data partition, the installer removes all the partitions to rebuild a brand new NOS. All X86 platforms share one installation and upgrade package with the name fixed as: onie-installer-picos-VERSION-x86.bin, where VERSION is the release version. X86 platforms are listed below: FS N9550-32D FS N8550-64C FS N5850-48S6Q FS N8550-48B8C FS N8550-32C Edgecore AS4630-54PE Edgecore AS5712-54X Edgecore AS5812-54T Edgecore AS5812-54X Edgecore AS7312-54X Edgecore AS7326-56X Edgecore AS7712-32X Edgecore AS7726-32X Edgecore AS7816-64X Edgecore AS5835-54X Edgecore AS9716-32D DELL N3248P-ON DELL N3248PXE-ON DELL N3224PX-ON DELL N3248X-ON DELL S4048-ON DELL S4148F-ON DELL S4148T-ON DELL S4128F-ON DELL S5224F-ON DELL S5296F-ON DELL S5212F-ON DELL S5248F-ON DELL Z9100-ON DELL Z9264F-ON DELL N3224T-ON DELL S4128T-ON During the upgrade process, please ensure that the power supply is functioning normally; otherwise, power interruption during the upgrade process could cause unpredictable problems. In previous 4.x.x versions, PicOS® allows the configuration of route leaking by importing BGP IPv4 routes from one user-defined VRF into another user-defined VRF, for example: set protocols bgp vrf vrf1 local-as 1 set protocols bgp vrf vrf1 ipv4-unicast import vrf vrf2 set protocols bgp vrf vrf2 local-as 2 That will cause configuration from PicOS® CLI is not consistent with FRR configuration. Specifically, FRR will add "set protocols bgp local-as 1" (local as number is same as the value in vrf1) to its configuration automatically, which is not in PicOS® CLI. From version 4.4.0, if "set protocols bgp local-as 1" is not configured, the above configurations are not allowed. Based on the above reasons, users are required to manually add the command "set protocols bgp local-as 1" (local as number is same as the value in vrf1) before the upgrade, if there's above configuration exists in the pre-upgrade version, thus to ensure that the configuration can be loaded successfully after the upgrade. 3.5 Usage of Upgrade Command admin@PicOS®:~$ sudo upgrade USAGE Upgrade system with local new image SYNOPSIS upgrade [image_name] [factory-default] DESCRIPTION image_name - Image with bin format file(*.bin) factory-default - Recovery configuration to factory default PicOS® upgrade is done via the command "upgrade" in bash (launching a shell script named "upgrade.sh"). This script will upgrade the image and back up configuration files automatically. The format of the upgrade package is *.bin. The option no-md5-check is removed from PicOS® 3.7.0 and later versions. If there is an MD5 file in the /cftmp directory, the upgrade script will check package integrity with MD5. Else if there is no MD5 file in the /cftmp directory, then skip the MD5 check step. The option factory-default is used to reset the configuration to factory default when performing upgrade. This option retains the license files from the previous version. If you want to backup a file during upgrade, use option backup-file=(*.lst) to define your own backup file list. The usage of option backup-file=(*.lst) is described in the below section. 3.5.1 Usage of Backup-file=(*.lst) Option During the upgrade process, the switch can automatically back up the following files in the following directories from the previous PicOS® system: /etc/passwd /etc/shadow /etc/group /etc/gshadow /etc/resolv.conf ./etc/network/interfaces /etc/picos/picos_start.conf /etc/picos/switch-public.key /etc/picos/pica.lic /pica/config/pica_startup.boot /pica/config/pica.conf.01 /pica/config/pica.conf.02 /pica/config/pica.conf.03 /pica/config/pica.conf.04 /pica/config/pica.conf.05 /ovs/ovs-vswitchd.conf.db /ovs/function.conf.db /ovs/config/meters /ovs/config/groups /ovs/config/flows /ovs/var/lib/openvswitch/pki/ /var/log/report_diag.log /var/log/report_diag.log.1 /var/log/report_diag.log.2 /var/log/report_diag.log.3 /var/log/report_diag.log.4 /var/log/report_diag.log.5 /cftmp/upgrade.log /cftmp/upgrade2.log /cftmp/auto/ If you want to save user files that are not in the above default backup file list, you need to first create or specify a .lst file and then add all those files that need to be backed up to this .lst file. You can use the backup-file=(*.lst) option to achieve this, where (*.lst) is the user created file with .lst format or specify the path to this file, for example: admin@PICOS:~$ sudo upgrade backup-file=/admin/back_files.lst onie-installer-picos-4.0.1-x86.bin For example, if you want to backup /home/admin/a.txt file during the process, then add /home/admin/a.txt to back_files.lst.In this example, back_files.lst is a user created file. The user has already added the file to back_files.lst that needs to be saved in the event of power off. admin@PICOS:~$ cat /admin/back_files.lst /home/admin/a.txt The above operations ensure that user can backup their important files with backup-file=(*.lst) option during the upgrade process. 3.6 Upgrading Procedure The upgrading procedure in this document gives an example of upgrading AS7312-54X from PicOS® 4.0.0 to 4.0.1. Step1 Copy the upgrade package (in the form of .bin) and the MD5 file to /cftmp directory by either FTP, TFTP, HTTP or SCP according to the actual upgrade environment. The following example uses the SCP method. admin@PICOS:~$ sudo scp pica8@10.10.50.16:/tftp/build/4.0.1/as7312_54x/onie-installer-picos-4.0.1-x86.bin /cftmp admin@PICOS:~$ sudo scp pica8@10.10.50.16:/tftp/build/4.0.1/as7312_54x/onie-installer-picos-4.0.1-x86.bin.md5 /cftmp NOTE: If management VRF is enabled, and the FTP/TFTP/HTTP/SCP server is connected via the Eth0/1 port, you need to add the string sudo ip vrf exec mgmt-vrf before the SCP command when executing the scp operation. The format is as follows: admin@Xorplus:~$ sudo ip vrf exec mgmt-vrf scp pica8@10.10.50.22:/tftp/build/4.0.1/as7312_54x/onie-installer-picos-as7312_54x-4.0.1-91bb175.bin /cftmp admin@Xorplus:~$sudo ip vrf exec mgmt-vrf scp pica8@10.10.50.22:/tftp/build/4.0.1/as7312_54x/onie-installer-picos-as7312_54x-4.0.1-91bb175.bin.md5 /cftmp If sudo ip vrf exec mgmt-vrf is not added, find the next hop routing information from the default VRF. For the usage of VRF, refer to the VRF Configuration Guide. Step2 Execute the sync operation. admin@PICOS:~$ sync Step3 Change directory to /cftmp. admin@PICOS:~$ cd /cftmp admin@PICOS:/cftmp$ Step4 Run the upgrade command. admin@PICOS:/cftmp$ sudo upgrade onie-installer-picos-4.0.1-x86.bin After finishing upgrade will reboot automatically, the system will come up running the new network operating system. 3.7 Verifying Version after Upgrading admin@PICOS:~$ version Copyright (C) 2009-2024 Pica8, Inc. =================================== Base ethernet MAC Address : d4:31:27:c9:e4:51 Hardware Model : as7312_54x Linux System Version/Revision : 9.8.7-main/fd87d25a10 Linux System Released Date : 10/12/2024 L2/L3 Version/Revision : 9.8.7-main/fd87d25a10 L2/L3 Released Date : 10/12/2024 OVS/OF Version/Revision : 9.8.7-main/fd87d25a10 OVS/OF Released Date : 10/12/2024 3.8 Appendix: Troubleshooting Installation/Upgrade Failure on AS7326-56X Installation or upgrade failure (for example, the switches cannot boot up after install) may occur on the old AS7326-56X hardware models (revision is R01F and before). When booting PicOS® on AS7326-56X and detect hardware rev R01F, the system will log a warning message to prompt the hardware revision R01F is a pre-production hardware reversion: "This hardware revision R01F is a pre-production hardware rev, PicOS® has applied a work around to work with PicOS®. Support will be provided on a best effort basis". To work around the issue, first we need to check the “Label Revision”. If it is an old hardware model (revision is R01F or before), then, we can perform the following provided solution after installation/upgrade to solve the problem. 3.8.1 Check Label Revision Under ONIE prompt, run “onie_syseeprom” to get the “Label Revision”. ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 166 TLV Name Code Len Value -------------------- ---- --- ----- Manufacture Date 0x25 19 04/27/2019 02:10:06 Label Revision 0x27 4 R01B Platform Name 0x28 27 x86_64-accton_as7326_56x-r0 ONIE Version 0x29 13 2018.05.00.05 Manufacturer 0x2B 6 Accton Diag Version 0x2E 7 0.0.1.0 Base MAC Address 0x24 6 80:A2:35:81:D5:F0 Serial Number 0x23 14 732656X1916012 Country Code 0x2C 2 TW Part Number 0x22 13 FP4ZZ7656005A Product Name 0x21 15 7326-56X-O-AC-F MAC Addresses 0x2A 2 256 Vendor Name 0x2D 6 Accton CRC-32 0xFE 4 0xC3D3F2DE Checksum is valid. ONIE:/ # 3.8.2 Solution You can follow the steps below after installation/upgrade, to fix the problem of installation and upgrade failure on the old AS7326-56X hardware model (revision R01F or before). Step1 Power cycle the switch. Step2 From the GRUB menu, choose “ONIE” to enter ONIE GRUB menu: +----------------------------------------------------------------------------+ | PicOS | |*ONIE | | | | | | | | | | | | | | | | | | | | | +----------------------------------------------------------------------------+ Use the ^ and v keys to select which entry is highlighted. Press enter to boot the selected OS, `e' to edit the commands before booting or `c' for a command-line. Step3 From ONIE GRUB menu, choose “ONIE: Rescue” to launch ONIE in Rescue mode. GNU GRUB version 2.02~beta2+e4a1fe391 +----------------------------------------------------------------------------+ | ONIE: Install OS | |*ONIE: Rescue | | ONIE: Uninstall OS | | ONIE: Update ONIE | | ONIE: Embed ONIE | | DIAG: Accton Diagnostic | | | | | | | | | | | | | +----------------------------------------------------------------------------+ Step4 Press Enter to display the ONIE prompt. Step5 Mount PicOS® partition with label is “PicOS®”. ONIE:/ # blkid /dev/sda7: LABEL="User-Data" UUID="be63cef8-4560-4c48-ab5a-8f7ced5a950b" /dev/sda6: LABEL="PicOS2" UUID="f589e53f-4cd1-44ba-8384-f339f4e2b2ac" /dev/sda5: LABEL="PicOS" UUID="8ca5f7ed-5a15-4a2a-944c-4d8872647bf5" /dev/sda4: LABEL="PICOS-GRUB" UUID="782a1372-4b66-4783-b920-dab1df8ec6e4" /dev/sda3: LABEL="ACCTON-DIAG" UUID="3e4117d0-1926-472a-9d9e-08883df83d40" /dev/sda2: LABEL="ONIE-BOOT" UUID="1a90abd8-f065-4f7a-90a0-af122b8805fa" ONIE:/ # ONIE:/ # mount /dev/sda5 /mnt Step6 Execute the following command to modify the I2C access address. ONIE:/ # sed -I "s/0x57/0x56/" /mnt/etc/rc_hw.sh ONIE:/ # sync Step7 Unmount the PicOS® partition. ONIE:/ # unmount /dev/sda5 Step8 Reboot the switch. ONIE:/ # reboot 4. Upgrading PicOS® from Version 3.0 or Later Using Upgrade2 4.1 Introduction NOTE： This document only applies to PicOS® upgrade from version 3.0 or later version using command upgrade2. If you want to upgrade PicOS® from the version before 3.0, use ONIE installation process described in "Installing PicOS® on Bare Metal Switches". You cannot do a standard upgrade from 3.x to 4.x. This is because 3.x config and 4.x config are not compatible, and PicOS® 4.x will not be able to boot with 3.x config after the upgrade. In order to upgrade from 3.x to 4.x, you MUST convert the configuration to 4.x before upgrade, see section "Converting Configuration to 4.x before Upgrade (when Upgrade from Version 3.x to 4.x)" in this guide for details. This upgrading guide is not available for FS S5810 Series and S5860 Series switches. N8560-32C uses the ONIE method for upgrade described in this guide, while the installation uses Rboot method, please refer to "Installing PicOS® for FS S5810/S5860 Series and N8560-32C Switches" for details on the installation process. The installation package name for N8560-32C includes the suffix '-rboot', for example, N8560_picos-4.4.5-9bca0916a3-rboot.bin. The upgrade package, on the other hand, includes the suffix '-x86', such as picos-4.4.5-9bca0916a3-x86.bin. PicOS® 4.0.0 and later versions have multiple system partitions including PicOS® (partition size: 2G), PicOS®2(partition size: 2G) and User-Data partitions. Among them, PicOS® and PicOS®2 are two independent system boot partitions. One of them is the active partition on which the running system resides, and the other is the inactive partition. The two-system-boot-partition feature allows the system to revert to a previous version of the installed software package when it fails to upgrade PicOS® by using upgrade2 command. User-Data partition is a reserved partition which is not affected by ONIE installer and upgrade unless user manually removes it. User-Data partition uses all the available space left on the disk. Users can use this partition to store files and data. When running upgrade2, the new version PicOS® image will be installed and boot onto the inactive partition automatically. Afterwards, the inactive partition will switch to active partition automatically when the switch boots up normally after the upgrading is finished, while the other partition where the old version resides will become the inactive partition. Upgrade2 method supports system rollback function. The "nos-rollback" command can be used to revert to a previous version of the installed software package. Moreover, if it fails to upgrade, the system can automatically rollback to the old system. This can reduce the network interruption risk due to the failure of system upgrade process and ensure the systems’ continuous availability. You can refer to section "Rollback Procedure" in this page for details. The system also supports the upgrade method for PicOS® version upgrade, you can refer to the document "Upgrading PicOS® from Version 4.0.0 or Later Using Upgrade Command" for details. We recommend using upgrade2 method to upgrade the NOS as it includes system backup and rollback features. 4.2 Preparation before Upgrading Table 1. Checklist before Upgrading No. Checking Items Checking Standard Results 1 Checking the Running PicOS® Version The currently running system software version is lower than the software version to be installed. 2 Checking License Validation Run the license -s command to verify that the license expiration date extends beyond the planned upgrade date. If the license is close to expiration, consider renewing it to avoid interruptions. 3 Building Upgrade Environment Build an upgrade environment according to the need. 4 Getting the Required Upgrade Software Obtain the required supported upgrade software. 5 Backing up Important Data All the important data was backed up. 6 Converting Configuration to 4.x before Upgrade (when Upgrade from Version 3.x to 4.x) 4.x configuration is generated from 3.x configuration file. 4.2.1 Checking the Running PicOS® Version Use the version command to check the version of the running system software. admin@PICOS:~$ version Copyright (C) 2009-2024 Pica8, Inc. =================================== Base ethernet MAC Address : 64:9d:99:d8:fd:07 Hardware Model : S3410-48TS-P PICOS Release/Commit : 4.4.4-s3000/eaf8c6573d PICOS Released Date : 09/14/2024 admin@PICOS:~$ 4.2.2 Checking License Validation Before performing an upgrade, users can run the license -s command to check if the current license has expired, ensuring it is valid and preventing upgrade failure due to license expiration. admin@PICOS:~$ license -s { "Type":"1GE", "Feature":["Base Product", "Layer3", "OpenFlow"], "Support End Date":"2025-10-28", "Hardware ID":"ACD2-F77A-BBA3-2849", "Site Name":"PICA8" } 4.2.3 Building Upgrade Environment Please make sure that you have set up an HTTP, TFTP or FTP protocol upgrading environment, the basic requirements are as follows: PC can log in to the device through serial or SSH. The communication between the server and the device works well. The upgrading file used by the device has already been stored on the server. 4.2.4 Getting the Required Upgrade Software Please contact Pica8 technical support engineers at the following webpage for the latest version of upgrade software. https://www.pica8.com/support/ 4.2.5 Backing up Important Data Before upgrading, save the important data, e.g. the configuration file, to the local PC through FTP or TFTP, and then upload it to the switch after the upgrade is completed if needed. 4.2.6 Converting Configuration to 4.x before Upgrade (when Upgrade from Version 3.x to 4.x) NOTE: When upgrade PicOS® from version 3.x to 4.x: When executing the upgrade2 command, no other option is supported except the option image_name. Backup the configuration file before upgrading. The OVS configuration for crossflow before the upgrade will be saved and restored automatically after the upgrade. You cannot do a standard upgrade from 3.x to 4.x. This is because 3.x configuration and 4.x configuration are not compatible, and PicOS® 4.x will not be able to boot with 3.x configuration after the upgrade. In order to upgrade from 3.x to 4.x, follow the procedure below to prepare the 4.x configuration file before upgrade: Create directory /pica/config-4.x/. Contact Pica8 support to convert the 3.x configuration to 4.x configuration in configuration file pica_startup.boot. Copy the 4.x configuration file (converted from 3.x configuration file in step 2) into the directory /pica/config-4.x just created. After upgrading from 3.x to 4.x and after rebooting, PicOS® 4.x will look for the 4.x configuration in /pica/config-4.x. After completing these steps, the 4.x configuration file is ready and you can continue with the upgrade process. If these steps are not performed before upgrade, the system will load the default configuration file of 4.x and the 3.x configuration will not be loaded after upgrade. However, if this happens unexpectly, you can also remedy by loading the 4. x configuration file after upgrade, follow the steps below: Copy the 4.x configuration file pica_startup.boot (already converted from 3.x configuration file) into the directory /pica/config/. Run the load override command to load 4. x configuration. admin@PICOS# load override /pica/config/pica_startup.boot admin@PICOS# Loading config file... Config file was loaded successfully. Upgrading Notes 4.3 Upgrading Notes This upgrade2 guide only applies to PicOS® upgrade from version 4.0.0 or the later versions. When using upgrade2 to upgrade PicOS®, you should make sure the “PicOS®2” partition exists. When using upgrade2 to upgrade PicOS®, you should make sure the partition type is GPT. When using upgrade2 to upgrade PicOS®, you should make sure that ONIE is pre-loaded. License check is performed for upgrade: If PicOS® has a license installed before the upgrade, the license will be copied and activated after the upgrade. Please check this section for the PICOS Licenses. If there is no license installed prior to upgrade, upgrade2 process can proceed but only the first four ports and the first two uplink ports (if exist) on the newly upgraded system can be used. If the license has expired, it is not allowed to upgrade a major release (e.g. 4.1 to 4.2). However, it will not affect upgrading to a minor release (e.g. 4.1.1 to 4.1.2). You can log in to a device through its console port or using SSH. After successful login, you can run commands on the command line interface (CLI) to upgrade the device. When using FTP/TFTP to download the image, user should verify that the "binary" mode is being used. If the "binary" transfer mode is not being used, the image might be modified during download, and the upgrade will fail during the MD5 check. The image is platform dependent, that is, the image should be consistent with the platform, otherwise the upgrade script will abort. An upgrage2.log file in /cftmp directory will be created which will contain all the logs related to the upgrade2 process. All X86 platforms share one installation and upgrade package with the name fixed as: onie-installer-picos-VERSION-x86.bin, where VERSION is the release version. X86 platform are listed below: FS N9550-32D FS N8550-64C FS N5850-48S6Q FS N8550-48B8C FS N8550-32C Edgecore AS4630-54PE Edgecore AS5712-54X Edgecore AS5812-54T Edgecore AS5812-54X Edgecore AS7312-54X Edgecore AS7326-56X Edgecore AS7712-32X Edgecore AS7726-32X Edgecore AS7816-64X Edgecore AS5835-54X Edgecore AS9716-32D DELL N3248P-ON DELL N3248PXE-ON DELL N3224PX-ON DELL N3248X-ON DELL S4048-ON DELL S4148F-ON DELL S4148T-ON DELL S4128F-ON DELL S5224F-ON DELL S5296F-ON DELL S5212F-ON DELL S5248F-ON DELL Z9100-ON DELL Z9264F-ON DELL N3224T-ON DELL S4128T-ON In previous 4.x.x versions, PICOS allows the configuration of route leaking by importing BGP IPv4 routes from one user-defined VRF into another user-defined VRF, for example: set protocols bgp vrf vrf1 local-as 1 set protocols bgp vrf vrf1 ipv4-unicast import vrf vrf2 set protocols bgp vrf vrf2 local-as 2 That will cause configuration from PicOS® CLI is not consistent with FRR configuration. Specifically, FRR will add "set protocols bgp local-as 1" (local as number is same as the value in vrf1) to its configuration automatically, which is not in PicOS® CLI. From version 4.4.0, if "set protocols bgp local-as 1" is not configured, the above configurations are not allowed. Based on the above reasons, users are required to manually add the command "set protocols bgp local-as 1" (local as number is same as the value in vrf1) before the upgrade, if there's above configuration exists in the pre-upgrade version, thus to ensure that the configuration can be loaded successfully after the upgrade. 4.4 Usage of Upgrade2 Command admin@PICOS:~$ sudo upgrade2 USAGE Upgrade system with local new image SYNOPSIS upgrade2 [image_name] [factory-default] DESCRIPTION image_name - Image with bin format file(*.bin) factory-default - Recovery configuration to factory default admin@PICOS:~$ For PicOS® go2cli version, users can run the upgrade2 command under CLI operational mode or configuration mode: admin@PICOS> upgrade2 image-file xx.bin Possible completions: <[Enter]> Execute this command backup-file Specify a user defined backup list(*.lst) factory-default Recovery configuration to factory default use-prev-config Use previous configuration, and syslog trace admin@PICOS# run upgrade2 image-file xx.bin Possible completions: <[Enter]> Execute this command backup-file Specify a user defined backup list(*.lst) factory-default Recovery configuration to factory default use-prev-config Use previous configuration, and syslog trace PicOS® upgrade is done via the command "upgrade2" in bash (launching a shell script named "upgrade2.sh"). This script will upgrade the image and backup configuration files automatically. Image name is in the form of .bin, which should be copied to the /cftmp directory before running upgrade2 command. The option factory-default is used to reset the configuration to factory default when performing upgrade, but it retains the license files from the previous version. If you want to use the old configuration file in the new version, you can add the use-prev-config option when issuing upgrade2 command. The usage of option use-prev-config is described in the section Usage of Use-prev-config Option. If you want to backup a file during upgrade, use backup-file=(*.lst) option to define your own backup file list. The usage of backup-file=(*.lst) option is described in the section Usage of Backup-file=(*.lst) Option. 4.4.1 Usage of use-prev-config Option The main function of use-prev-config option is to decide whether to load the previous configuration file after a system reboot when performing upgrade2 or rollback to another version. If there is a command line in the old version configuration file that is not supported in the new system, with use-prev-config option that command will be skipped and continue loading the remaining configuration. By default, upgrade2 or rollback is performed without use-prev-config option. The following table describes the usage of use-prev-config option when performing upgrade2 or rollback. upgrade2 (From old version to new version) rollback (From current version to old version) with use-prev-config 1. Load the configuration file of old version after system reboot. 2. If there is a command line in the old version configuration file that is not supported in the new system, skip it and continue loading the remaining configuration. 1. Load the configuration file of current version after system reboot. 2. If there is a command line in the current configuration file that is not supported in the old system, skip it and continue loading the remaining configuration. without use-prev-config 1. Load the configuration file of old version after reboot. 2. If there is a command in the old version configuration file that is not supported in the new system, load default configuration file. Load the old version configuration file after rebooting. 4.4.2 Usage of Backup-file=(*.lst) Option During the upgrade process, the switch can automatically back up the following files in the following directories from the previous PicOS® system: /etc/passwd /etc/shadow /etc/group /etc/gshadow /etc/resolv.conf ./etc/network/interfaces /etc/picos/picos_start.conf /etc/picos/switch-public.key /etc/picos/pica.lic /pica/config/pica_startup.boot /pica/config/pica.conf.01 /pica/config/pica.conf.02 /pica/config/pica.conf.03 /pica/config/pica.conf.04 /pica/config/pica.conf.05 /ovs/ovs-vswitchd.conf.db /ovs/function.conf.db /ovs/config/meters /ovs/config/groups /ovs/config/flows /ovs/var/lib/openvswitch/pki/ /var/log/report_diag.log /var/log/report_diag.log.1 /var/log/report_diag.log.2 /var/log/report_diag.log.3 /var/log/report_diag.log.4 /var/log/report_diag.log.5 /cftmp/upgrade.log /cftmp/upgrade2.log /cftmp/auto/ If you want to save user files that are not in the above default backup file list, you need to first create or specify a .lst file and then add all those files that need to be backed up to this .lst file. You can use the backup-file=(*.lst) option to achieve this, where (*.lst) is the user created file with .lst format or specify the file path to this file, for example: admin@PICOS:~$ sudo upgrade2 backup-file=/admin/back_files.lst onie-installer-picos-as7312_54x-4.0.1-cc8d268.bin For example, if you want to backup /home/admin/a.txt file during the process, then add /home/admin/a.txt to back_files.lst. In this example, back_files.lst is a user created file. The user has already added the file to back_files.lst that needs to be saved in the event of power off. admin@PICOS:~$ sudo upgrade2 backup-file=/admin/back_files.lst onie-installer-picos-as7312_54x-4.0.1-cc8d268.bin The above operations ensure that user can backup their important files with backup-file=(*.lst) option during the upgrade process. 4.5 Upgrade Procedure The upgrading procedure in this document gives an example of upgrading from PicOS® 4.0.0 to 4.0.1 using upgrade2 command on AS7312_54X switch. Step1 Copy the upgrade package (in the form of .bin) and the MD5 file to /cftmp directory by either FTP, TFTP, HTTP or SCP according to the actual upgrade environment. The following example uses the SCP method. admin@PICOS:~$ sudo scp pica8@10.10.50.22:/tftp/build/4.0.1/as7312_54x/onie-installer-picos-as7312_54x-4.0.1-cc8d268.bin /cftmp admin@PICOS:~$ sudo scp pica8@10.10.50.22:/tftp/build/4.0.1/as7312_54x/onie-installer-picos-as7312_54x-4.0.1-cc8d268.bin.md5 /cftmp NOTE: If management VRF is enabled, and the FTP/TFTP/HTTP/SCP server is connected via the Eth0/1 port, you need to add the string sudo ip vrf exec mgmt-vrf before the SCP command when executing the scp operation. The format is as follows: admin@Xorplus:~$ sudo ip vrf exec mgmt-vrf scp pica8@10.10.50.22:/tftp/build/4.0.1/as7312_54x/onie-installer-picos-as7312_54x-4.0.1-cc8d268.bin /cftmp admin@Xorplus:~$sudo ip vrf exec mgmt-vrf scp pica8@10.10.50.22:/tftp/build/4.0.1/as7312_54x/onie-installer-picos-as7312_54x-4.0.1-cc8d268.bin.md5 /cftmp If sudo ip vrf exec mgmt-vrf is not added, find the next hop routing information from the default VRF. For the usage of VRF, refer to the VRF Configuration Guide. For PicOS® go2cli version, users can run the scp command under CLI operational mode or configuration mode: Φ Download a file: file scp get remote-file [local-file local-file-path] ip-address : [vrf ] Φ Upload a file: file scp put local-file [remote-file ] ip-address : [vrf ] Step2 Execute the sync operation. admin@PICOS:~$ sync Step3 Change directory to /cftmp. admin@PICOS:~$ cd /cftmp Step4 Run upgrade2 command to begin upgrading. admin@PICOS:~$ sudo upgrade2 onie-installer-picos-as7312_54x-4.0.1-cc8d268.bin After finishing upgrade, the switch will reboot automatically, the system will come up running the new network operating system. NOTE： For PicOS® go2cli version, users can run the upgrade2 command under CLI operational mode or configuration mode. It will take 20-30 minutes to finish upgrading PicOS®. During the upgrade process, please be patient and do not perform any operation until the upgrade is complete, otherwise, the upgrade may be interrupted. 4.6 Rollback Procedure The upgrade2 method supports system rollback function. The "nos-rollback" command can be used to revert to a previous version of the installed software package. Moreover, if it fails to upgrade, the system can automatically rollback to the old system. NOTE： Usage of nos-rollback command: admin@Xorplus:~$ sudo nos-rollback USAGE Rollback to the previous system after next reboot SYNOPSIS nos-rollback [use-prev-config] DESCRIPTION use-prev-config - Use previous config, and syslog trace For details about the usage of use-prev-config, please refer to Usage of Use-prev-config Option. The rollback procedure is as follows: Step1 Run nos-rollback command for manually rollback. admin@PICOS:~$ sudo nos-rollback USAGE Rollback to the previous system after next reboot SYNOPSIS nos-rollback Checking prerequisites Attribute of current system [OK] Will switch from PICOS-4.4.5GA to the other system! Do you want to continue?[y/N]?y Updating default boot option Modify default boot option [OK] Rollback to the other system successful! Please reboot to enter the other system! admin@PICOS:~$ Step2 Reboot system manually to finish rollback. admin@Xorplus:~$ sudo reboot You need to manually run reboot command to reboot the system after you have issued "nos-rollback" command. After rebooting successfully, the system will come up running the previous version of network operating system. 4.7 Verifying Version after Upgrade admin@PICOS:~$ version Copyright (C) 2009-2024 Pica8, Inc. =================================== Base ethernet MAC Address : d4:31:27:c9:e4:51 Hardware Model : S5860-48MG-U Linux System Version/Revision : 4.4.5GA/7f06432992 Linux System Released Date : 10/14/2024 L2/L3 Version/Revision : 4.4.5GA/7f06432992 L2/L3 Released Date : 10/14/2024 OVS/OF Version/Revision : 4.4.5GA/7f06432992 OVS/OF Released Date : 10/14/2024 4.8 Appendix: Troubleshooting Installation/Upgrade Failure on AS7326-56X Installation or upgrade failure (for example, the switches cannot boot up after install) may occur on the old AS7326-56X hardware models (revision is R01F and before). When booting PicOS® on AS7326-56X and detect hardware rev R01F, the system will log a warning message to prompt the hardware revision R01F is a pre-production hardware reversion: "This hardware revision R01F is a pre-production hardware rev, PicOS® has applied a work around to work with PicOS®. Support will be provided on a best effort basis". To work around the issue, first we need to check the “Label Revision”. If it is an old hardware model (revision is R01F or before), then, we can perform the following provided solution after installation/upgrade to solve the problem. 4.8.1 Check Label Revision Under ONIE prompt, run “onie_syseeprom” to get the “Label Revision”. ONIE:/ # onie-syseeprom TlvInfo Header: Id String: TlvInfo Version: 1 Total Length: 166 TLV Name Code Len Value -------------------- ---- --- ----- Manufacture Date 0x25 19 04/27/2019 02:10:06 Label Revision 0x27 4 R01B Platform Name 0x28 27 x86_64-accton_as7326_56x-r0 ONIE Version 0x29 13 2018.05.00.05 Manufacturer 0x2B 6 Accton Diag Version 0x2E 7 0.0.1.0 Base MAC Address 0x24 6 80:A2:35:81:D5:F0 Serial Number 0x23 14 732656X1916012 Country Code 0x2C 2 TW Part Number 0x22 13 FP4ZZ7656005A Product Name 0x21 15 7326-56X-O-AC-F MAC Addresses 0x2A 2 256 Vendor Name 0x2D 6 Accton CRC-32 0xFE 4 0xC3D3F2DE Checksum is valid. ONIE:/ # 4.8.2 Solution You can follow the steps below after installation/upgrade, to fix the problem of installation and upgrade failure on the old AS7326-56X hardware model (revision R01F or before). Step1 Power cycle the switch. Step2 From the GRUB menu, choose “ONIE” to enter ONIE GRUB menu: +----------------------------------------------------------------------------+ | PicOS | |*ONIE | | | | | | | | | | | | | | | | | | | | | +----------------------------------------------------------------------------+ Use the ^ and v keys to select which entry is highlighted. Press enter to boot the selected OS, `e' to edit the commands before booting or `c' for a command-line. Step3 From ONIE GRUB menu, choose “ONIE: Rescue” to launch ONIE in Rescue mode. GNU GRUB version 2.02~beta2+e4a1fe391 +----------------------------------------------------------------------------+ | ONIE: Install OS | |*ONIE: Rescue | | ONIE: Uninstall OS | | ONIE: Update ONIE | | ONIE: Embed ONIE | | DIAG: Accton Diagnostic | | | | | | | | | | | | | +----------------------------------------------------------------------------+ Step4 Press Enter to display the ONIE prompt. Step5 Mount PicOS® partition with label is “PicOS®”. ONIE:/ # blkid /dev/sda7: LABEL="User-Data" UUID="be63cef8-4560-4c48-ab5a-8f7ced5a950b" /dev/sda6: LABEL="PicOS2" UUID="f589e53f-4cd1-44ba-8384-f339f4e2b2ac" /dev/sda5: LABEL="PicOS" UUID="8ca5f7ed-5a15-4a2a-944c-4d8872647bf5" /dev/sda4: LABEL="PICOS-GRUB" UUID="782a1372-4b66-4783-b920-dab1df8ec6e4" /dev/sda3: LABEL="ACCTON-DIAG" UUID="3e4117d0-1926-472a-9d9e-08883df83d40" /dev/sda2: LABEL="ONIE-BOOT" UUID="1a90abd8-f065-4f7a-90a0-af122b8805fa" ONIE:/ # ONIE:/ # mount /dev/sda5 /mnt Step6 Execute the following command to modify the I2C access address. ONIE:/ # sed -I "s/0x57/0x56/" /mnt/etc/rc_hw.sh ONIE:/ # sync Step7 Unmount the PicOS® partition. ONIE:/ # unmount /dev/sda5 Step8 Reboot the switch. ONIE:/ # reboot 5. Installing Debian Packages on PicOS® PicOS® uses a standard and non-modified Debian Linux distribution. It is very easy to install new packages or software on top of the existing PicOS® packages, using the standard Debian package management system. Here are some installation examples. 5.1 Installing GCC on PicOS® NOTE: If the FTP server is connected via the Eth0/1 port, you need to add the string sudo ip vrf exec mgmt-vrf before the apt-get command when executing the apt-get operation. For example: admin@Xorplus:~$ sudo ip vrf exec mgmt-vrf apt-get update If sudo ip vrf exec mgmt-vrf is not added, find the next hop routing information from the default VRF. For the usage of VRF, refer to the VRF configuration guide. Updating the software list on the source server admin@XorPlus$sudo apt-get update Hit http://ftp.tw.debian.org stable Release.gpg Hit http://ftp.tw.debian.org stable Release Hit http://ftp.tw.debian.org stable/main powerpc Packages Hit http://ftp.tw.debian.org stable/main Translation-en Reading package lists... Done admin@XorPlus$ Installing new software admin@XorPlus$sudo apt-get install make Reading package lists... Done Building dependency tree Reading state information... Done Suggested packages: make-doc The following NEW packages will be installed: make 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 399 kB of archives. After this operation, 1165 kB of additional disk space will be used. WARNING: The following packages cannot be authenticated! make Authentication warning overridden. Get:1 http://ftp.tw.debian.org/debian/ stable/main make powerpc 3.81-8.2 [399 kB] Fetched 399 kB in 6s (64.1 kB/s) Selecting previously unselected package make. (Reading database ... 16155 files and directories currently installed.) Unpacking make (from .../make_3.81-8.2_powerpc.deb) ... Processing triggers for man-db ... fopen: Permission denied Setting up make (3.81-8.2) ... admin@XorPlus$ admin@XorPlus$sudo apt-get install python Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: file libexpat1 libmagic1 mime-support python-minimal python2.7 python2.7-minimal Suggested packages: python-doc python-tk python2.7-doc binutils binfmt-support The following NEW packages will be installed: file libexpat1 libmagic1 mime-support python python-minimal python2.7 python2.7-minimal 0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded. Need to get 5045 kB of archives. After this operation, 18.3 MB of additional disk space will be used. Do you want to continue [Y/n]? Y WARNING: The following packages cannot be authenticated! libmagic1 libexpat1 file mime-support python2.7-minimal python2.7 python-minimal python Authentication warning overridden. Get:1 http://ftp.tw.debian.org/debian/ stable/main libmagic1 powerpc 5.11-2 [201 kB] Get:2 http://ftp.tw.debian.org/debian/ stable/main libexpat1 powerpc 2.1.0-1 [142 kB] Get:3 http://ftp.tw.debian.org/debian/ stable/main file powerpc 5.11-2 [51.7 kB] Get:4 http://ftp.tw.debian.org/debian/ stable/main mime-support all 3.52-1 [35.5 kB] Get:5 http://ftp.tw.debian.org/debian/ stable/main python2.7-minimal powerpc 2.7.3-6 [1753 kB] Get:6 http://ftp.tw.debian.org/debian/ stable/main python2.7 powerpc 2.7.3-6 [2639 kB] Get:7 http://ftp.tw.debian.org/debian/ stable/main python-minimal all 2.7.3-4 [42.6 kB] Get:8 http://ftp.tw.debian.org/debian/ stable/main python all 2.7.3-4 [180 kB] Fetched 5045 kB in 18s (267 kB/s) Selecting previously unselected package libmagic1:powerpc. (Reading database ... 16189 files and directories currently installed.) Unpacking libmagic1:powerpc (from .../libmagic1_5.11-2_powerpc.deb) ... Selecting previously unselected package libexpat1:powerpc. Unpacking libexpat1:powerpc (from .../libexpat1_2.1.0-1_powerpc.deb) ... Selecting previously unselected package file. Unpacking file (from .../file_5.11-2_powerpc.deb) ... Selecting previously unselected package mime-support. Unpacking mime-support (from .../mime-support_3.52-1_all.deb) ... Selecting previously unselected package python2.7-minimal. Unpacking python2.7-minimal (from .../python2.7-minimal_2.7.3-6_powerpc.deb) ... Selecting previously unselected package python2.7. Unpacking python2.7 (from .../python2.7_2.7.3-6_powerpc.deb) ... Selecting previously unselected package python-minimal. Unpacking python-minimal (from .../python-minimal_2.7.3-4_all.deb) ... Selecting previously unselected package python. Unpacking python (from .../python_2.7.3-4_all.deb) ... Processing triggers for man-db ... fopen: Permission denied Setting up libmagic1:powerpc (5.11-2) ... Setting up libexpat1:powerpc (2.1.0-1) ... Setting up file (5.11-2) ... Setting up mime-support (3.52-1) ... Setting up python2.7-minimal (2.7.3-6) ... Linking and byte-compiling packages for runtime python2.7... Setting up python2.7 (2.7.3-6) ... Setting up python-minimal (2.7.3-4) ... Setting up python (2.7.3-4) ... admin@XorPlus$ admin@XorPlus$sudo apt-get install g++ Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: g+-4.6 libstdc+6-4.6-dev Suggested packages: g+-multilib g-4.6-multilib gcc-4.6-doc libstdc6-4.6-dbg libstdc+6-4.6-doc The following NEW packages will be installed: g++ g+-4.6 libstdc+6-4.6-dev 0 upgraded, 3 newly installed, 0 to remove and 17 not upgraded. Need to get 0 B/8383 kB of archives. After this operation, 24.4 MB of additional disk space will be used. Do you want to continue [Y/n]? Y WARNING: The following packages cannot be authenticated! libstdc+6-4.6-dev g-4.6 g+ Authentication warning overridden. Selecting previously unselected package libstdc++6-4.6-dev. (Reading database ... 19555 files and directories currently installed.) Unpacking libstdc+6-4.6-dev (from .../libstdc+6-4.6-dev_4.6.3-14_powerpc.deb) ... Selecting previously unselected package g++-4.6. Unpacking g+-4.6 (from .../g+-4.6_4.6.3-14_powerpc.deb) ... Selecting previously unselected package g++. Unpacking g++ (from .../g++_4%3a4.6.3-8_powerpc.deb) ... Processing triggers for man-db ... Setting up libstdc++6-4.6-dev (4.6.3-14) ... Setting up g++-4.6 (4.6.3-14) ... Setting up g++ (4:4.6.3-8) ... update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode admin@XorPlus$ 5.2 Installing Puppet on PicOS® NOTE: You can see an example of Puppet module to manipulate PicOS® configuration on our Github repository: https://github.com/Pica8/Configuration-Managers If the FTP server is connected via the Eth0/1 port, you need to add the string sudo ip vrf exec mgmt-vrf before the apt-get command when executing the apt-get operation. For example: admin@Xorplus:~$ sudo ip vrf exec mgmt-vrf apt-get update If sudo ip vrf exec mgmt-vrf is not added, find the next hop routing information from the default VRF. For the usage of VRF, refer to the VRF configuration guide. Step1 Use the correct repository for the specific application and CPU on the switch. Pica8 support can help in the choice of repository. admin@PICOS:~$sudo more /etc/apt/sources.list | grep -v "#" deb http://ftp.debian-ports.org/debian/ unstable main For a typical puppet installation, the latest standard debian repo is advised. Step2 Update the debian packages on PicOS®. admin@PICOS:~$ sudo apt-get update Hit http://ftp.tw.debian.org stable Release.gpg Hit http://ftp.tw.debian.org stable Release Hit http://ftp.tw.debian.org stable/main powerpc Packages Hit http://ftp.tw.debian.org stable/main Translation-en Reading package lists... Done admin@PICOS:~$ Step3 Install puppet client and configure it. admin@PICOS:~$ sudo apt-get install puppet Look at the puppet documentation to understand how to connect the puppet client to a puppet server. A simple installation would require at least minor modification on the puppet.conf file. more /etc/puppet/puppet.conf [agent] server = master.local.pica8.com Step4 Verify Puppet installation. admin@PICOS:~$ sudo puppet agent -t Notice: Using less secure serialization of reports and query parameters for compatibility Notice: with older puppet master. To remove this notice, please upgrade your master(s) Notice: to Puppet 3.3 or newer. Notice: See http://links.puppetlabs.com/deprecate_yaml_on_network for more information. Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/instance_id.rb Info: Caching catalog for Roma Info: Applying configuration version '1405148228' Notice: Finished catalog run in 0.35 seconds 5.3 Installing Salt on PicOS® NOTE: You can see an example of Salt module to manipulate PicOS® configuration on our Github repository: https://github.com/pica8/Configuration-Managers If the FTP server is connected via the Eth0/1 port, you need to add the string sudo ip vrf exec mgmt-vrf before the apt-get command when executing the apt-get operation. For example: admin@Xorplus:~$ sudo ip vrf exec mgmt-vrf apt-get update If sudo ip vrf exec mgmt-vrf is not added, find the next hop routing information from the default VRF. For the usage of VRF, refer to the VRF configuration guide. Step1 Use the correct repository for the specific application and CPU on the switch. Pica8 support can help in the choice of repository. admin@PICOS:~$ sudo more /etc/apt/sources.list | grep -v "#" deb http://ftp.debian-ports.org/debian/ unstable main For a typical salt installation, the latest standard debian repo is advised. Step2 Update the debian packages on PicOS® admin@PICOS:~$ sudo apt-get update Hit http://ftp.tw.debian.org stable Release.gpg Hit http://ftp.tw.debian.org stable Release Hit http://ftp.tw.debian.org stable/main powerpc Packages Hit http://ftp.tw.debian.org stable/main Translation-en Reading package lists... Done admin@PICOS:~$ Step3 Install salt-common and salt-minion and configure it admin@PICOS:~$ sudo apt-get install salt-common admin@PICOS:~$ sudo apt-get install salt-minion Look at the salt documentation to understand how to connect the salt-minion to a salt-master. A simple installation would need at least minor modification on the minion configuration file. more /etc/salt/minion # Set the location of the salt master server, if the master server cannot be # resolved, then the minion will fail to start. master: salt.example.com 6. PicOS® Installation and Upgrade Guide for FS S5810 Series, S3410 Series, S5860 Series and N8560-32C Switches 6.1 Upgrading PicOS® for FS S5810/S5860 Series Switches Using Upgrade Command (Login via Console Port) NOTE: This guide is only available for upgrading PicOS® for FS S5810/S5860 Series switches when login via the console port. 6.1.1 Preparation before Upgrading Table 1 Checklist before Upgrading No. Checking Items Checking Standard Results 1 Checking the Running PicOS® Version The currently running system software version is lower than the software version to be installed 2 Checking License Validation Run the license -s command to verify that the license expiration date extends beyond the planned upgrade date. If the license is close to expiration, consider renewing it to avoid interruptions. 3 Building Upgrade Environment Build a different upgrade environment to get the upgrade software according to the need 4 Getting the Required Upgrade Software Obtain the required supporting upgrade software 5 Backing up Important Data in Flash All the important data in Flash is backed up 6 Checking Available Flash Space Flash space is enough to save upgrading package and other files Checking the Running PicOS® Version Use the version command to check the version of the running system software. admin@PICOS:~$ version Copyright (C) 2009-2023 Pica8, Inc. =================================== Base ethernet MAC Address : 64:9d:99:d7:7d:23 Hardware Model : S5810-48TS-P Linux System Version/Revision : 4.3.3.1/4f6f523 Linux System Released Date : 12/10/2023 L2/L3 Version/Revision : 4.3.3.1/4f6f523 L2/L3 Released Date : 12/10/2023 OVS/OF Version/Revision : 4.3.3.1/4f6f523 OVS/OF Released Date : 12/10/2023 Checking License Validation Before performing an upgrade, users can run the license -s command to check if the current license has expired, ensuring it is valid and preventing upgrade failure due to license expiration. admin@PICOS:~$ license -s { "Type":"1GE", "Feature":["Base Product", "Layer3", "OpenFlow"], "Support End Date":"2020-10-28", "Hardware ID":"ACD2-F77A-BBA3-2849", "Site Name":"PICA8" } Building Upgrade Environment Please make sure that you have set up an HTTP, TFTP or FTP protocol upgrading environment to get the upgrade software, the basic requirements are as follows: PC can log in to the device through serial. The communication between the server and the device works well. The upgrading file used by the device has already been stored on the server. Getting the Required Upgrade Software Please contact Pica8 technical support engineers at the following website for the latest version of upgrade software. http://www.pica8.com/support/customer Backing up Important Data in Flash Before upgrading, save the important data (such as the configuration file) in Flash to the local PC through FTP or TFTP, and then upload it to the switch after the upgrade is completed. Where is the PicOS® configuration? OVSDB file L2/L3 Configuration Files Checking Available Flash Space Use the df command to check the available flash space. admin@PICOS:~$ df Filesystem 1K-blocks Used Available Use% Mounted on udev 493028 0 493028 0% /dev overlay 358904 57528 301376 17% / tmpfs 512720 0 512720 0% /dev/shm tmpfs 205088 3256 201832 2% /run tmpfs 5120 0 5120 0% /run/lock tmpfs 51200 292 50908 1% /tmp /dev/ubi1_0 402660 208320 189500 53% /mnt/open 6.1.2 Upgrading Notes The device is not supported to upgrade to a previous version. When using FTP/TFTP to download the image, user should verify that the "binary" mode is being used. If the "binary" transfer mode is not being used, the image can be modified during download, and the upgrade will fail during the md5 check. The image is platform dependent, that is, the image_name should be consistent with the platform, otherwise the upgrade script will abort. The image file is a .bin file, for example S5810-48TS-P-picos-e-4.4.3.2-2f6f578-fs.bin. When upgrading, the installer checks whether there is a user-data partition. If there exists a User-Data partition, the installer only rewrites the running system boot partition (PicOS®/ PicOS®2) and installs the new installation package to this partition. However, if there is no User-Data partition, the installer removes all the partitions to rebuild a brand new NOS. Upgrade operation via upgrade commands is not allowed on non-default system, you can upgrade PicOS® only on default system. When there are more than one PicOS®, the default system is the one automatically booted into after system reboot. During the upgrade process, no power interruption is allowed. 6.1.3 Upgrading Procedure NOTE： Usage of upgrade command: admin@PicOS®:~$ sudo upgrade USAGE Upgrade system with local new image SYNOPSIS upgrade [image_name] [factory-default] DESCRIPTION image_name - Image with bin format file(*.bin) factory-default - Recovery configuration to factory default PicOS® upgrade is done via the command "upgrade" in bash (launching a shell script named "upgrade.sh"). This script will upgrade the image automatically. The file format of the upgrade package is *.bin. If there is an MD5 file in the /cftmp directory, the upgrade script will check package integrity with MD5. Else if there is no MD5 file in the /cftmp directory, then skip the MD5 check step. The option factory-default is used to reset the configuration to factory default when performing upgrade. This option retains the license files from the previous version. The upgrading procedure in this document gives an example of upgrading on S5810-48TS-P from PicOS® 4.4.3.1 to 4.4.3.2. Step1 Stop PicOS® service before upgrade. On FS S5810 Series and S5860 Series switches, use the following command: admin@PICOS:~$ sudo systemctl stop picos Step2 Copy the upgrade package (in the form of .bin) and the MD5 file to /cftmp directory by either FTP, TFTP, HTTP or SCP according to the actual upgrade environment. The following example uses the SCP method. admin@PICOS:~$ sudo scp pica8@10.10.50.22:/tftp/build/daily/s5810/S5810-48TS-P-picos-e-4.4.3.2-2f6f578-fs.bin /cftmp admin@PICOS:~$ sudo scp pica8@10.10.50.22:/tftp/build/daily/s5810/S5810-48TS-P-picos-e-4.4.3.2-2f6f578-fs.bin.md5 /cftmp NOTE: If management VRF is enabled, and the FTP/TFTP/HTTP/SCP server is connected via the Eth0/1 port, you need to add the string sudo ip vrf exec mgmt-vrf before the SCP command when executing the scp operation. The format is as follows: admin@PicOS®:~$ sudo ip vrf exec mgmt-vrf scp pica8@10.10.50.22:/tftp/build/s5810/S5810-48TS-P-picos-e-4.4.3.2-2f6f578-fs.bin /cftmp admin@PicOS®:~$ sudo ip vrf exec mgmt-vrf scp pica8@10.10.50.22:/tftp/build/s5810/S5810-48TS-P-picos-e-4.4.3.2-2f6f578-fs.bin.md5 /cftmp If sudo ip vrf exec mgmt-vrf is not added, find the next hop routing information from the default VRF. For the usage of VRF, refer to the VRF Configuration Guide. Step3 Execute the sync operation. admin@PICOS:~$ sync Step4 Change directory to /cftmp. admin@PICOS:~$ cd /cftmp Step5 Run the upgrade command. admin@PICOS:~$ sudo upgrade S5810-48TS-P-picos-e-4.4.3.2-2f6f578-fs.bin After the upgrade is complete, the system will automatically reboot and run the new network operating system. 6.1.4 Verifying Version after Upgrading admin@PICOS:~$ version Copyright (C) 2009-2023 Pica8, Inc. =================================== Base ethernet MAC Address : 64:9d:99:d7:7d:23 Hardware Model : S5810-48TS-P Linux System Version/Revision : 4.4.3.2/2f6f578 Linux System Released Date : 12/15/2023 L2/L3 Version/Revision : 4.4.3.2/2f6f578 L2/L3 Released Date : 12/15/2023 OVS/OF Version/Revision : 4.4.3.2/2f6f578 OVS/OF Released Date : 12/15/2023 6.2 Upgrading PicOS® by Using Upgrade2 for S5860 Series and S3410 Series Switches (Login via Console Port) NOTE: This guide is only available for upgrading PicOS® for FS S5860 Series and S3410 Series switches when login via the console port. The S3410 Series switches only support upgrade via the upgrade2 method. S5810 Series switches only support the upgrade method and do not support the upgrade2 method. 6.2.1 Introduction PicOS® supports upgrade2 method for system upgrade. There will be two separate systems on the device after the upgrade2 operation: PicOS® and PicOS®2. One of them will be the running system and the other will stay inactive. PicOS® and PicOS®2 system files and their respective configuration files are located in /mnt/open/picos/ of the flash. A list and brief description of these files is as follows. * uImage1.itb * picos1.sqsh * config1/backup_files //User-defined backup files list * config1/backup.tar.gz //Backup of latest.tar.gz * config1/latest.tar.gz //The newest configuration files * uImage2.itb * picos2.sqsh * config2/backup_files * config2/backup.tar.gz * config2/latest.tar.gz The upgrade2 installer installs the new system into the inactive system’s file. The inactive system will be overwritten. After this operation, the new system is the inactive system and then the installer modifies the boot menu to make the newly installed system to be the default boot system. Finally, the system will come up running the new network operating system when boots up normally after the upgrading is finished,. Upgrade2 method supports system rollback function. The "nos-rollback" command can be used to revert to a previous version of the installed software package. Moreover, if it fails to upgrade, the system can automatically rollback to the old system. This can reduce the network interruption risk due to the failure of system upgrade process and ensure the systems’ continuous availability. You can refer to section "Rollback Procedure" for details. We recommend using upgrade2 method to upgrade the NOS as there are functions of system backup and rollback. 6.2.2 Preparation before Upgrading Table 1. Checklist before Upgrading No. Checking Items Checking Standard Results 1 Checking the Running PicOS® Version The currently running system software version is lower than the software version to be installed. 2 Checking License Validation Run the license -s command to verify that the license expiration date extends beyond the planned upgrade date. If the license is close to expiration, consider renewing it to avoid interruptions. 3 Building Upgrade Environment Build an upgrade environment to get the upgrade software according to the need. 4 Getting the Required Upgrade Software Obtain the required supporting upgrade software. 5 Backing up Important Data in Flash All the important data in Flash is backed up. Checking the Running PicOS® Version Use the version command to check the version of the running system software. admin@PICOS# run show version Copyright (C) 2009-2023 Pica8, Inc. =================================== Base ethernet MAC Address : 64:9d:99:d2:04:53 Hardware Model : S5860-24XB-U Linux System Version/Revision : 4.3.3.2/4b5f523 Linux System Released Date : 12/14/2023 L2/L3 Version/Revision : 4.3.3.2/4b5f523 L2/L3 Released Date : 12/14/2023 OVS/OF Version/Revision : 4.3.3.2/4b5f523 OVS/OF Released Date : 12/14/2023 Checking License Validation Before performing an upgrade, users can run the license -s command to check if the current license has expired, ensuring it is valid and preventing upgrade failure due to license expiration. admin@PICOS:~$ license -s { "Type":"1GE", "Feature":["Base Product", "Layer3", "OpenFlow"], "Support End Date":"2020-10-28", "Hardware ID":"ACD2-F77A-BBA3-2849", "Site Name":"PICA8" } Building Upgrade Environment Please make sure that you have set up an HTTP, TFTP or FTP protocol upgrading environment to get the upgrade software, the basic requirements are as follows: PC can log in to the device through serial. The communication between the server and the device works well. The upgrading file used by the device has already been stored on the server. Getting the Required Upgrade Software Please contact Pica8 technical support engineers at the following website for the latest version of upgrade software. http://www.pica8.com/support/customer Backing up Important Data in Flash Before upgrading, save the important data in Flash to the local PC through FTP or TFTP, and then upload it to the switch after the upgrade is completed. 6.2.3 Upgrading Notes Downgrade to an earlier version is NOT supported by using upgrade2 command. When using FTP/TFTP to download the image, user should verify that the "binary" mode is being used. If the "binary" transfer mode is not being used, the image can be modified during download, and the upgrade will fail during the md5 check. The image file is a .bin file, for example S5860-24XB-U-picos-e-4.4.3.2-2f6f578-fs.bin. The image is platform dependent, that is, the image_name should be consistent with the platform, otherwise the upgrade script will abort. 6.2.4 Upgrading Procedure The upgrading procedure in this document gives an example of upgrading on S5860-24XB-U from PicOS® 3.1.0 to 3.1.1 using upgrade2 command. NOTE： Usage of upgrade2 command: USAGE Upgrade system with local new image SYNOPSIS upgrade2 [image_name] [factory-default] DESCRIPTION image_name - Image with bin format file(*.bin) factory-default - Recovery configuration to factory default PICOS upgrade is done via the command "upgrade2" in bash (launching a shell script named "upgrade2.sh"). This script will upgrade the image and back up configuration files automatically. Image name is in the form of .bin from version 3.1.0, which should be copied to the /cftmp directory before running upgrade2 command. The no-md5-check option is removed from PicOS® 3.1.0. If there is an MD5 file in the /cftmp directory, the upgrade script will check package integrity with MD5. Else if there is no MD5 file in the /cftmp directory, then skip the MD5 check step. The option factory-default is used to reset the configuration to factory default when performing upgrade, but it retains the license files from the previous version. Upgrade2 Procedure Step1 Stop PicOS® service before upgrade. On FS S5810 Series and S5860 Series switches, use the following command: admin@PICOS:~$ sudo systemctl stop picos On S3410 Series switches, use the following command: admin@PICOS:~$ sudo /etc/init.d/picos stop Step2 Copy the upgrade package (in the form of .bin) and the MD5 file to /cftmp directory by either FTP, TFTP, HTTP or SCP according to the actual upgrade environment. The following example uses the SCP method. admin@PICOS:~$ sudo scp pica8@10.10.50.22:/tftp/build/daily/S5860-24XB-U/S5860-24XB-U-picos-e-4.4.3.2-2f6f578-fs.bin /cftmp admin@PICOS:~$ sudo scp pica8@10.10.50.22:/tftp/build/daily/S5860-24XB-U/S5860-24XB-U-picos-e-4.4.3.2-2f6f578-fs.bin.md5 /cftmp NOTE: If management VRF is enabled, and the FTP/TFTP/HTTP/SCP server is connected via the Eth0/1 port, you need to add the string sudo ip vrf exec mgmt-vrf before the SCP command when executing the scp operation. The format is as follows: admin@PicOS®:~$ sudo ip vrf exec mgmt-vrf scp pica8@10.10.50.22:/tftp/build/4.4.3.2/S5860-24XB-U/S5860-24XB-U-picos-e-4.4.3.2-2f6f578-fs.bin /cftmp admin@PicOS®:~$ sudo ip vrf exec mgmt-vrf scp pica8@10.10.50.22:/tftp/build/4.4.3.2/S5860-24XB-U/S5860-24XB-U-picos-e-4.4.3.2-2f6f578-fs.bin.md5 /cftmp If sudo ip vrf exec mgmt-vrf is not added, find the next hop routing information from the default VRF. For the usage of VRF, refer to the VRF Configuration Guide. Step3 Execute the sync operation. admin@PICOS:~$ sync Step4 Change directory to /cftmp. admin@PICOS:~$ cd /cftmp Step5 Run upgrade2 command to begin upgrading. admin@PICOS:~$ sudo upgrade2 S5860-24XB-U-picos-e-4.4.3.2-2f6f578-fs.bin After finishing upgrade, the switch will reboot automatically, the system will come up running the new network operating system. NOTE： It will take 20-30 minutes to finish upgrading PicOS®. During the upgrade process, please be patient and do not perform any operation until the upgrade is complete, otherwise, the upgrade may be interrupted. 6.2.5 Rollback Procedure The upgrade2 method supports system rollback function. The "nos-rollback" command can be used to revert to a previous version of the installed software package. Moreover, if it fails to upgrade, the system can automatically rollback to the old system. NOTE： Usage of nos-rollback command: admin@PicOS®:~$ sudo nos-rollback USAGE Rollback to the previous system after next reboot SYNOPSIS nos-rollback The rollback procedure is as follows: Step1 Run nos-rollback command for manually rollback. admin@PICOS:~$ sudo nos-rollback Step2 Reboot system manually to finish rollback. admin@PICOS:~$ sudo reboot You need to manually reboot the system after issued "nos-rollback" command and the system switching takes effect. After rebooting successfully, the system will come up running the previous version of network operating system. 6.2.6 Verifying Version after Upgrading admin@PICOS# run show version Copyright (C) 2009-2023 Pica8, Inc. =================================== Base ethernet MAC Address : 64:9d:99:d2:04:53 Hardware Model : S5860-24XB-U Linux System Version/Revision : 4.3.3.3/4c5a643 Linux System Released Date : 12/26/2023 L2/L3 Version/Revision : 4.3.3.3/4c5a643 L2/L3 Released Date : 12/26/2023 OVS/OF Version/Revision : 4.3.3.3/4c5a643 OVS/OF Released Date : 12/26/2023 6.3 Installing PicOS® for FS S5810/S5860 Series and N8560-32C Switches NOTE: N8560-32C uses the Rboot method for installation described in this guide, while upgrade still uses the ONIE method, please refer to "Upgrading PicOS® from Version 3.0 or Later Using Upgrade2" and "Upgrading PicOS® from Version 4.0.0 or Later Using Upgrade Command" for details on the upgrade process. The installation package name for N8560-32C includes the suffix '-rboot', for example, N8560_picos-4.4.5-9bca0916a3-rboot.bin. The upgrade package, on the other hand, includes the suffix '-x86', such as picos-4.4.5-9bca0916a3-x86.bin. Caution: When an incorrect installation file is detected, the system will display the error message “Ignore ERRORS? [YES/NO]:”. Users need to enter “no” to prevent the system from using the incorrect installation file for system installation. Do NOT enter “yes”, or the system will proceed with the incorrect installation file, which may cause a system crash. image.png PicOS® system can be installed under the Rboot menu through TFTP protocol for FS S5810 and S5860 Series switches. The following steps describe the installation procedure. Step1 Power off and on to force restarting the switch, then press Ctrl+C to enter Rboot menu. Step2 (Optional) If the TFTP server and the switch are in the different network segment, configure the gateway address first (if they are in the same network segment, no need to do this step). a) Enter 4 in the Rboot menu to access the Scattered utilities menu. image.png b) Enter 7 to set the gateway IP and IP netmask. c) Then, press Ctrl+Z to go back to the Rboot menu. Step3 In Rboot menu, enter 0 to access Tftp utilities menu, and then enter 2 to perform TFTP upgrade. image.png Step4 Use TFTP protocol to download the installation files, and then install PicOS®. a) Configure the TFTP parameters. Local IP is the management interface IP of the switch, Remote IP is the IP of the TFTP server, Filename is the installation image directory and name located on the TFTP server. b) After downloaded the installation image successfully, you need to input Y manually, then the system will automatically start system installation process. image.png Wait a few minutes before the installation process is completed. When Success is displayed, it indicates that the installation process is successfully completed. Step5 Press Ctrl+Z to go back to the Rboot menu, then type 2 to reboot the switch. image.png Then the device reboots and comes up running the new network operating system. Users need to enter the username and password after the system restarts, the initial login username is admin and password is pica8. Then users will be asked to set a new password for admin. This is the only post installation step after which the PicOS® operating system can be used. 6.4 Installing PicOS® for FS S3410 Series Switches NOTE: The installation package name for S3410 includes the suffix '-rboot', for example, S3410-PicOS-4.4.5.14-2963b1e57b-rboot.bin. PicOS® system can be installed under the Uboot menu through TFTP protocol for FS S3410 Series switches. The following steps describe the installation procedure. Step1 Power off and on to force restarting the switch, then press Ctrl+C to enter Uboot menu. Step2 Enter 0 and then 1 in the Uboot menu to perform the installation process. image.png Step3 Use TFTP protocol to download the installation files, and then install PicOS®. a) Configure the TFTP parameters. Local IP is the inband management interface IP of the switch, Remote IP is the IP of the TFTP server, Filename is the installation image directory and name located on the TFTP server. image.png b) After downloaded the installation image successfully, you need to input Y manually, then the system will automatically start system installation process. c) Wait a few minutes before the installation process is completed. When Success is displayed, it indicates that the installation process is successfully completed. Step4 Press Ctrl+Z to go back to the Uboot menu, then type 2 to reboot the switch. image.png Then the device reboots and comes up running the new network operating system. Users need to enter the username and password after the system restarts, the initial login username is admin and password is pica8. Then users will be asked to set a new password for admin. This is the only post installation step after which the PicOS® operating system can be used. 6.5 Upgrading PicOS® for FS S5810/S5860 Series Switches Using Upgrade Command (Login via Eth0 or Inband Management Interface) NOTE: This guide is only available for upgrading PicOS® for FS S5810/S5860 Series switches when login via the eth0 or inband management interface, and the supported version should be 4.4.5.7 or later versions before upgrade. During the upgrade process, the user connections and services will be interrupted. This means that users may lose access to the system, and any processes or transactions being handled by the services will be paused until the upgrade is complete. After the upgrade, users will need to reconnect to resume normal operations, and services will be restored. 6.5.1 Preparation before Upgrading Table 1 Checklist before Upgrading No. Checking Items Checking Standard Results 1 Checking the Running PicOS® Version The currently running system software version is lower than the software version to be installed 2 Checking License Validation Run the license -s command to verify that the license expiration date extends beyond the planned upgrade date. If the license is close to expiration, consider renewing it to avoid interruptions. 3 Building Upgrade Environment Build a different upgrade environment to get the upgrade software according to the need 4 Getting the Required Upgrade Software Obtain the required supporting upgrade software 5 Backing up Important Data in Flash All the important data in Flash is backed up 6 Checking Available Flash Space Flash space is enough to save upgrading package and other files Checking the Running PicOS® Version Use the version command to check the version of the running system software. admin@PICOS:~$ version Copyright (C) 2009-2024 Pica8, Inc. =================================== Base ethernet MAC Address : 64:9d:99:d7:7d:23 Hardware Model : S5860-48MG-U Linux System Version/Revision : 4.4.5.7/4f6f523 Linux System Released Date : 10/10/2024 L2/L3 Version/Revision : 4.4.5.7/4f6f523 L2/L3 Released Date : 10/10/2024 OVS/OF Version/Revision : 4.4.5.7/4f6f523 OVS/OF Released Date : 10/10/2024 Checking License Validation Before performing an upgrade, users can run the license -s command to check if the current license has expired, ensuring it is valid and preventing upgrade failure due to license expiration. admin@PICOS:~$ license -s { "Type":"1GE", "Feature":["Base Product", "Layer3", "OpenFlow"], "Support End Date":"2020-10-28", "Hardware ID":"ACD2-F77A-BBA3-2849", "Site Name":"PICA8" } Building Upgrade Environment Please make sure that you have set up an HTTP, TFTP or FTP protocol upgrading environment to get the upgrade software, the basic requirements are as follows: PC can log in to the device through eth0 or inband management interface. The communication between the server and the device works well. The upgrading file used by the device has already been stored on the server. Getting the Required Upgrade Software Please contact Pica8 technical support engineers at the following website for the latest version of upgrade software. http://www.pica8.com/support/customer Backing up Important Data in Flash Before upgrading, save the important data (such as the configuration file) in Flash to the local PC through FTP or TFTP, and then upload it to the switch after the upgrade is completed. Where is the PicOS® configuration? OVSDB file L2/L3 Configuration Files Checking Available Flash Space Use the df command to check the available flash space. admin@PICOS:~$ df Filesystem 1K-blocks Used Available Use% Mounted on udev 493028 0 493028 0% /dev overlay 358904 57528 301376 17% / tmpfs 512720 0 512720 0% /dev/shm tmpfs 205088 3256 201832 2% /run tmpfs 5120 0 5120 0% /run/lock tmpfs 51200 292 50908 1% /tmp /dev/ubi1_0 402660 208320 189500 53% /mnt/open 6.5.2 Upgrading Notes The device is not supported to upgrade to a previous version. When using FTP/TFTP to download the image, user should verify that the "binary" mode is being used. If the "binary" transfer mode is not being used, the image can be modified during download, and the upgrade will fail during the md5 check. The image is platform dependent, that is, the image_name should be consistent with the platform, otherwise the upgrade script will abort. The image file is a .bin file, for example S5810-48TS-P-picos-e-4.4.5.7-2f6f578-fs.bin. Please find the log file related to PicOS® upgrade process at /mnt/open/picos/config2/upgrade.log and /mnt/open/picos/config1/upgrade.log. This log file contains detailed information about the steps performed during the upgrade, including any errors or warnings that occurred. It can be used to troubleshoot issues or verify that the upgrade was completed successfully. When upgrading, the installer checks whether there is a user-data partition. If there exists a User-Data partition, the installer only rewrites the running system boot partition (PicOS®/ PicOS®2) and installs the new installation package to this partition. However, if there is no User-Data partition, the installer removes all the partitions to rebuild a brand new NOS. Upgrade operation via upgrade commands is not allowed on non-default system, you can upgrade PicOS® only on default system. When there are more than one PicOS®, the default system is the one automatically booted into after system reboot. During the upgrade process, power interruption is not allowed. 6.5.3 Upgrading Procedure NOTE： Usage of upgrade command: admin@PicOS®:~$ sudo upgrade USAGE Upgrade system with local new image SYNOPSIS upgrade [image_name] [factory-default] DESCRIPTION image_name - Image with bin format file(*.bin) factory-default - Recovery configuration to factory default PicOS® upgrade is done via the command "upgrade" in bash (launching a shell script named "upgrade.sh"). This script will upgrade the image automatically. The file format of the upgrade package is *.bin. If there is an MD5 file in the /cftmp directory, the upgrade script will check package integrity with MD5. Else if there is no MD5 file in the /cftmp directory, then skip the MD5 check step. The option factory-default is used to reset the configuration to factory default when performing upgrade. This option retains the license files from the previous version. The upgrading procedure in this document gives an example of upgrading on S5860-48MG-U from PicOS® 4.4.5.7 to 4.4.5.8. Step1 Copy the upgrade package (in the form of .bin) and the MD5 file to /cftmp directory by either FTP, TFTP, HTTP or SCP according to the actual upgrade environment. The following example uses the SCP method. admin@PICOS:~$ sudo scp pica8@10.10.50.22:/tftp/build/daily/s5860/S5860_picos-4.4.5GA-7f06432992.bin /cftmp admin@PICOS:~$ sudo scp pica8@10.10.50.22:/tftp/build/daily/s5860/S5860_picos-4.4.5GA-7f06432992.bin.md5 /cftmp NOTE: If management VRF is enabled, and the FTP/TFTP/HTTP/SCP server is connected via the Eth0/1 port, you need to add the string sudo ip vrf exec mgmt-vrf before the SCP command when executing the scp operation. The format is as follows: admin@PicOS®:~$ sudo ip vrf exec mgmt-vrf scp pica8@10.10.50.22:/tftp/build/s5860/S5860_picos-4.4.5GA-7f06432992.bin /cftmp admin@PicOS®:~$ sudo ip vrf exec mgmt-vrf scp pica8@10.10.50.22:/tftp/build/s5860/S5860_picos-4.4.5GA-7f06432992.bin.md5 /cftmp If sudo ip vrf exec mgmt-vrf is not added, find the next-hop routing information from the default VRF. For the usage of VRF, refer to the VRF Configuration Guide. Step2 Execute the sync operation. admin@PICOS:~$ sync Step3 Change directory to /cftmp. admin@PICOS:~$ cd /cftmp Step4 Run the upgrade command. admin@PICOS:/cftmp$ sudo upgrade S5860_picos-4.4.5GA-7f06432992.bin Upgrading system... The connection may be interrupted. Please wait a moment to complete the upgrading procedure. admin@PICOS:/cftmp$ NOTE： It will take about 5 minutes to finish upgrading PicOS®. During the upgrade process, please be patient and do not perform any operation until the upgrade is complete, otherwise, the upgrade may be interrupted. During the upgrade process, the user connections and services will be interrupted. This means that users may lose access to the system, and any processes or transactions being handled by the services will be paused until the upgrade is complete. Step5 After the upgrade, users will need to reconnect to resume normal operations, and services will be restored. admin@PICOS:~$ ssh admin@10.10.51.54 6.5.4 Verifying Version after Upgrading admin@PICOS:~$ version Copyright (C) 2009-2024 Pica8, Inc. =================================== Base ethernet MAC Address : d4:31:27:c9:e4:51 Hardware Model : S5860-48MG-U Linux System Version/Revision : 4.4.5.8/7f06432992 Linux System Released Date : 10/14/2024 L2/L3 Version/Revision : 4.4.5.8/7f06432992 L2/L3 Released Date : 10/14/2024 OVS/OF Version/Revision : 4.4.5.8/7f06432992 OVS/OF Released Date : 10/14/2024 6.6 Upgrading PicOS® by Using Upgrade2 for S5860 Series and S3410 Series Switches (Login via Eth0 or Inband Management Interface) NOTE: This guide is only available for upgrading PicOS® for FS S5860 Series and S3410 Series switches when login via the eth0 or inband management interface, and the supported version should be 4.4.5.7 or later versions before upgrade. During the upgrade process, the user connections and services will be interrupted. This means that users may lose access to the system, and any processes or transactions being handled by the services will be paused until the upgrade is complete. After the upgrade, users will need to reconnect to resume normal operations, and services will be restored. The S3410 Series switches only support upgrade via the upgrade2 method. S5810 Series switches only support the upgrade method and do not support the upgrade2 method. 6.6.1 Introduction PicOS® supports upgrade2 method for system upgrade. There will be two separate systems on the device after the upgrade2 operation: PicOS® and PicOS®2. One of them will be the running system and the other will stay inactive. PicOS® and PicOS®2 system files and their respective configuration files are located in /mnt/open/picos/ of the flash. A list and brief description of these files is as follows. * uImage1.itb * picos1.sqsh * config1/backup_files //User-defined backup files list * config1/backup.tar.gz //Backup of latest.tar.gz * config1/latest.tar.gz //The newest configuration files * uImage2.itb * picos2.sqsh * config2/backup_files * config2/backup.tar.gz * config2/latest.tar.gz The upgrade2 installer installs the new system into the inactive system’s file. The inactive system will be overwritten. After this operation, the new system is the inactive system and then the installer modifies the boot menu to make the newly installed system to be the default boot system. Finally, the system will come up running the new network operating system when boots up normally after the upgrading is finished,. Upgrade2 method supports system rollback function. The "nos-rollback" command can be used to revert to a previous version of the installed software package. Moreover, if it fails to upgrade, the system can automatically rollback to the old system. This can reduce the network interruption risk due to the failure of system upgrade process and ensure the systems’ continuous availability. You can refer to section "Rollback Procedure" for details. We recommend using upgrade2 method to upgrade the NOS as there are functions of system backup and rollback. 6.6.2 Preparation before Upgrading Table 1. Checklist before Upgrading No. Checking Items Checking Standard Results 1 Checking the Running PicOS® Version The currently running system software version is lower than the software version to be installed. 2 Checking License Validation Run the license -s command to verify that the license expiration date extends beyond the planned upgrade date. If the license is close to expiration, consider renewing it to avoid interruptions. 3 Building Upgrade Environment Build an upgrade environment to get the upgrade software according to the need. 4 Getting the Required Upgrade Software Obtain the required supporting upgrade software. 5 Backing up Important Data in Flash All the important data in Flash is backed up. Checking the Running PicOS® Version Use the version command to check the version of the running system software. admin@PICOS:~$ version Copyright (C) 2009-2024 Pica8, Inc. =================================== Base ethernet MAC Address : 64:9d:99:d7:7d:23 Hardware Model : S5860-48MG-U Linux System Version/Revision : 4.4.5.7/4f6f523 Linux System Released Date : 10/10/2024 L2/L3 Version/Revision : 4.4.5.7/4f6f523 L2/L3 Released Date : 10/10/2024 OVS/OF Version/Revision : 4.4.5.7/4f6f523 OVS/OF Released Date : 10/10/2024 Checking License Validation Before performing an upgrade, users can run the license -s command to check if the current license has expired, ensuring it is valid and preventing upgrade failure due to license expiration. admin@PICOS:~$ license -s { "Type":"1GE", "Feature":["Base Product", "Layer3", "OpenFlow"], "Support End Date":"2020-10-28", "Hardware ID":"ACD2-F77A-BBA3-2849", "Site Name":"PICA8" } Building Upgrade Environment Please make sure that you have set up an HTTP, TFTP or FTP protocol upgrading environment to get the upgrade software, the basic requirements are as follows: PC can log in to the device through SSH. The communication between the server and the device works well. The upgrading file used by the device has already been stored on the server. Getting the Required Upgrade Software Please contact Pica8 technical support engineers at the following website for the latest version of upgrade software. http://www.pica8.com/support/customer Backing up Important Data in Flash Before upgrading, save the important data in Flash to the local PC through FTP or TFTP, and then upload it to the switch after the upgrade is completed. 6.6.3 Upgrading Notes Downgrade to an earlier version is NOT supported by using upgrade2 command. When using FTP/TFTP to download the image, user should verify that the "binary" mode is being used. If the "binary" transfer mode is not being used, the image can be modified during download, and the upgrade will fail during the md5 check. The image file is a .bin file, for example S5860-24XB-U-picos-e-4.4.5.7-2f6f578-fs.bin. Please find the log file related to PicOS® upgrade process at /mnt/open/picos/config2/upgrade2.log and /mnt/open/picos/config1/upgrade2.log. This log file contains detailed information about the steps performed during the upgrade, including any errors or warnings that occurred. It can be used to troubleshoot issues or verify that the upgrade was completed successfully. The image is platform dependent, that is, the image_name should be consistent with the platform, otherwise the upgrade script will abort. During the upgrade process, power interruption is not allowed. 6.6.4 Upgrading Procedure The upgrading procedure in this document gives an example of upgrading on S5860-48MG-U from 4.4.5.7 to 4.4.5.8 using upgrade2 command. NOTE： Usage of upgrade2 command: USAGE Upgrade system with local new image SYNOPSIS upgrade2 [image_name] [factory-default] DESCRIPTION image_name - Image with bin format file(*.bin) factory-default - Recovery configuration to factory default PicOS® upgrade is done via the command "upgrade2" in bash (launching a shell script named "upgrade2.sh"). This script will upgrade the image and back up configuration files automatically. Image name is in the form of .bin from version 3.1.0, which should be copied to the /cftmp directory before running upgrade2 command. The no-md5-check option is removed from PicOS® 3.1.0. If there is an MD5 file in the /cftmp directory, the upgrade script will check package integrity with MD5. Else if there is no MD5 file in the /cftmp directory, then skip the MD5 check step. The option factory-default is used to reset the configuration to factory default when performing upgrade, but it retains the license files from the previous version. Upgrade2 Procedure Step1 Copy the upgrade package (in the form of .bin) and the MD5 file to /cftmp directory by either FTP, TFTP, HTTP or SCP according to the actual upgrade environment. The following example uses the SCP method. admin@PICOS:~$ sudo scp pica8@10.10.50.22:/tftp/build/daily/s5860/S5860_picos-4.4.5GA-7f06432992.bin /cftmp admin@PICOS:~$ sudo scp pica8@10.10.50.22:/tftp/build/daily/s5860/S5860_picos-4.4.5GA-7f06432992.bin.md5 /cftmp NOTE: If management VRF is enabled, and the FTP/TFTP/HTTP/SCP server is connected via the Eth0/1 port, you need to add the string sudo ip vrf exec mgmt-vrf before the SCP command when executing the scp operation. The format is as follows: admin@PicOS®:~$ sudo ip vrf exec mgmt-vrf scp pica8@10.10.50.22:/tftp/build/s5860/S5860_picos-4.4.5GA-7f06432992.bin /cftmp admin@PicOS®:~$ sudo ip vrf exec mgmt-vrf scp pica8@10.10.50.22:/tftp/build/s5860/S5860_picos-4.4.5GA-7f06432992.bin.md5 /cftmp If sudo ip vrf exec mgmt-vrf is not added, find the next hop routing information from the default VRF. For the usage of VRF, refer to the VRF Configuration Guide. Step2 Execute the sync operation. admin@PICOS:~$ sync Step3 Change directory to /cftmp. admin@PICOS:~$ cd /cftmp Step4 Run upgrade2 command to begin upgrading. admin@PICOS:/cftmp$ sudo upgrade2 S5860_picos-4.4.5GA-7f06432992.bin Upgrading system... The connection may be interrupted. Please wait a moment to complete the upgrading procedure. admin@PICOS:/cftmp$ NOTE： It will take about 5 minutes to finish upgrading PicOS®. During the upgrade process, please be patient and do not perform any operation until the upgrade is complete, otherwise, the upgrade may be interrupted. During the upgrade process, the user connections and services will be interrupted. This means that users may lose access to the system, and any processes or transactions being handled by the services will be paused until the upgrade is complete. Step5 After the upgrade, users will need to reconnect to resume normal operations, and services will be restored. admin@PICOS:~$ ssh admin@10.10.51.54 6.6.5 Rollback Procedure The upgrade2 method supports system rollback function. The "nos-rollback" command can be used to revert to a previous version of the installed software package. Moreover, if it fails to upgrade, the system can automatically rollback to the old system. NOTE： Usage of nos-rollback command: admin@PicOS®:~$ sudo nos-rollback USAGE Rollback to the previous system after next reboot SYNOPSIS nos-rollback The rollback procedure is as follows: Step1 Run nos-rollback command for manually rollback. admin@PICOS:~$ sudo nos-rollback Step2 Reboot system manually to finish rollback. admin@PICOS:~$ sudo reboot You need to manually reboot the system after issued "nos-rollback" command and the system switching takes effect. After rebooting successfully, the system will come up running the previous version of network operating system. 6.6.6 Verifying Version after Upgrading admin@PICOS:~$ version Copyright (C) 2009-2024 Pica8, Inc. =================================== Base ethernet MAC Address : d4:31:27:c9:e4:51 Hardware Model : S5860-48MG-U Linux System Version/Revision : 4.4.5.8/7f06432992 Linux System Released Date : 10/14/2024 L2/L3 Version/Revision : 4.4.5.8/7f06432992 L2/L3 Released Date : 10/14/2024 OVS/OF Version/Revision : 4.4.5.8/7f06432992 OVS/OF Released Date : 10/14/2024 7. PicOS® Debian Package Upgrade User Guide 7.1 Overview PicOS® provides five Debian packages from release 3.2 to let users upgrade some of the available components manually, or reinstall PicOS® components in case some of them were broken. Available PicOS® Debian packages and the dependencies between them are described below: picos-linux PicOS® Linux Kernel, drivers and switching ASIC kernel modules picos-vasic PicOS® VASIC and line card management libraries and utilities Depends on picos-linux picos-xorplus PicOS® Layer 2 and Layer 3 software package Depends on picos-vasic, picos-utils picos-ovs PicOS® OVS package “picos-ovs” will have its own lib to access peripherals (such as FAN and PSU and LED) via sysfs Depends on picos-vasic, picos-utils picos-utils PicOS® common utilities and configuration files System config files, systemd units Common utility such as ZTP/diag In this way, we do not need to upgrade the entire PicOS® system if version changes only appear in one or several of the components. This provides an efficient and effective method of upgrading PicOS® system. NOTE： Some PicOS® component packages would depend on other parts, so the dependent ones should be installed first if they do not exist on the system. 7.2 How to use When new releases of PicOS® components have been made available to fix urgent issues, users can get the Debian packages from PicOS® support team. For example, the package users get might be "picos-xorplus-s4100-3.2.3-9dc8d94.deb" saved in the working directory. To install the package, the following command is OK: admin@Xorplus:~$ sudo dpkg -i picos-xorplus-s4100-3.2.3-9dc8d94.deb After finishing upgrade, the switch will reboot automatically, the system will come up running the PicOS® operating system with the new PicOS® component. NOTE： If certain PicOS® components have been removed from the running Linux system, this operation would be an installation instead of upgrade. In this case, users need to confirm the model compatibility manually by inputting Yes or Y at the prompt `Are you sure the model is MODEL (yes/no)?` 7.3 Verifying after Upgrade We can use the following command to check the status of PicOS® Debian packages after upgrade. admin@Xorplus:~$ dpkg -l | grep picos- ii picos-linux ii picos-ovs ii picos-utils ii picos-vasic ii picos-xorplus Here two “i” represent normal, the first one indicates that the package has been installed successfully. The second “i” indicates the installation dependencies between the components and configuration operations are successfully completed met. 7.4 Appendix PicOS® component package uninstall operation is provided as follows. NOTE： Uninstall the PicOS® component packages may cause severe system errors, we strongly recommend not to uninstall any of the PicOS® component package. The uninstall operation will uninstall all packages that depend on this package, either directly or indirectly. admin@Xorplus:~$ sudo apt remove picos-utils Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: picos-utils 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded. After this operation, 0 B of additional disk space will be used. Do you want to continue? [Y/n]

400G AI PicOS® Switches Competitive Comparison

11-09-2025 - 400G AI PicOS® Switches Competitive Comparison Overview This document provides a side-by-side comparison of FS 400G AI PicOS® Switches, focusing on hardware specification and software features. Product Comparison Models N9550-32D N8610-32D N8610-64D N9550-64D Product Introduction Models Product Overview Application Scenarios N9550-32D The N9550-32D Switch supports very large, dense, and fast data center fabrics that are based on proven Internet-scale technology. It offers flexible 100GbE, 200GbE, and 400GbE interfaces for server and intra-fabric connectivity. The N9550-32D is an optimal choice for spine-and-leaf deployments in enterprise, service provider, and cloud provider data centers, as well as routing use cases in metro environments. AI Data Center Leaf/Spine Data Center Fabric Spine N8610-32D N8610-32D switch is a high density and power efficient fixed configuration data center switch, providing up to 32 ports of 400/100G in a single rack unit enabling efficient high radix clusters that maximize 12.8 Tbps bandwidth performance of compute and storage by eliminating bottlenecks between leaf and spine tiers. N8610-32D switch delivers wirespeed layer 2 and layer 3 with flexible resources, robust traffic management and load balancing features, optimized for AI generated content lossless network. AI Data Center Leaf/Spine Data Center Fabric Spine/Super Spine IP Storage Networking N8610-64D N8610-64D switch is a high-radix class switch, dedicated for high-bandwidth network switching devices supporting up to 64× 400/100GbE ports in 2U form factor. This makes it optimal for AI data center deployments and spine and super-spine roles within IP fabrics. The additional RoCEv2 capabilities supports IP storage deployments where instead of relying on deep buffer switching, the QoS mechanisms such as priority flow control (PFC) and explicit congestion notification (ECN) deliver high performance for the storage workloads. AI Data Center Leaf/Spine Data Center Fabric Spine/Super Spine IP Storage Networking N9550-64D N9550-64D switch is a high-radix class switch, dedicated for high-bandwidth network switching devices supporting up to 64× 400/100GbE ports in 4U form factor. This makes it optimal for AI data center deployments and spine and super-spine roles within IP fabrics. The additional RoCEv2 capabilities supports IP storage deployments where instead of relying on deep buffer switching, the QoS mechanisms such as priority flow control (PFC) and explicit congestion notification (ECN) deliver high performance for the storage workloads. AI Data Center Leaf/Spine Data Center Fabric Spine/Super Spine IP Storage Networking Product Comparison Hardware Specifications Overview High-performance, high-density 400GbE AI fabric PicOS® switches Models N9550-32D EkjvbX8nAo3gJFxWBa9cUrKinXd.png N8610-32D image.png N8610-64D COHLbCpWGolbOWxCpCgcyfXkn5b.jpg N9550-64D HmxzbJ5FYoY583xzsOWcaVjendf.jpg Switch Chip BCM56980 Tomahawk 3 BCM56993 Tomahawk 4 BCM56990 Tomahawk 4 BCM56990 Tomahawk 4 Form Factor 1U 1U 2U 4U Performance Switching Capacity 12.8 Tbps 12.8 Tbps 25.6 Tbps 25.6 Tbps Forwarding Rate 5,210 Mpps 5,079 Mpps 10,420 Mpps 10,300 Mpps CPU Intel® Xeon® Processor D-1518 4-Core Intel® Xeon® Processor D-1713NTE 4-Core 2.2GHz Intel® Xeon® Processor D-1548 8-Core 2.0GHz Intel® Xeon® Processor D-1627 4-Core 2.9GHz DDR4 8GBx 2 SO-DIMM 32GB DDR4 SO-DIMM with ECC 16GBx 2 SO-DIMM 16G Packet Buffer 64MB 56.83MB 113.66MB 113.66MB Jumbo Frame 9K 9K 9K 9K MAC Address 6.4K 8K 8K 8K Flash Memory m.2 64GB MLC 128GB m.2 SATA SSD with PLP support 128GBx 1 240GB Latency 0.94μs (64B) TBD TBD TBD Ports Connector 32x 400GbE QSFP-DD 32x 400GbE QSFP-DD 64x 400GbE QSFP-DD 64x 400GbE QSFP-DD Max. Ports with Breakout 64x 200GbE or 128x 100GbE 64x 200GbE or 128x 100GbE 128x 200GbE or 256x 100GbE 128x 200GbE or 256x 100/50GbE 100/1000Base-T Management Port 1x RJ-45 1x RJ-45 1x RJ-45 1x RJ-45 Console Port 1 1x RJ-45 1x Micro USB Console 1 1 USB Port 1 1 1 1 Power and Cooling Hot-swap Power Supplies 2 (1+1 Redundancy), AC 2 (1+1 Redundancy), AC 2 (1+1 Redundancy), AC 4 (3+1 Redundancy), AC Hot-swappable Fans 6 (5+1 Redundancy) 6 (5+1 Redundancy) 4 (3+1 Redundancy) 6 (5+1 Redundancy) Airflow Front-to-Back Front-to-Back Front-to-Back Front-to-Back Max. Power Consumption TBD Max. power draw: 1,370W Max. power draw: 2,100W Max. power draw: 1,852W Power Max. Rating 1,300W 1,500W 2,400W 1,600W Environmental Operating Temperature 0°C to 45°C (32°F to 113°F) 0°C to 45°C (32°F to 113°F) 0ºC to 40ºC (32°F to 104°F) 0ºC to 40ºC (32°F to 104°F) Storage Temperature -40°C to 70°C ( -40°F to 158°F) -40ºC to 70 ºC (-40°F to 158°F) -40ºC to 70 ºC (-40°F to 158°F) -40°C to 70°C (-40°F to 158°F) Operating Humidity 5% to 95% (Non-condensing） 5% to 95% (Non-condensing) 5% to 95% (Non-condensing) 5% to 95% (Non-condensing) Physical Specifications Weight 24.38 lbs (11.09 kg) 26.12 lbs (11.85 kg) 47.4 lbs (21.5 kg) 81.57 lbs (37 kg) Dimensions (Hx Wx D) 1.69"x 17.25"x 21.1" (43x 438x 536mm) 1.71"x 17.26"x 23.23" (43.5x 438.4x 590mm) 3.43"x 17.32"x 25.56" (87x 440x 649.2mm) 6.89"x 17.32"x 29.92" (175x 440x 760mm) Electrical Input Voltage 100-240 VAC 100-127 VAC, 200-240 VAC 100-127 VAC, 200-240 VAC 100-240 VAC Frequency 50-60Hz 50-60Hz 50-60Hz 50-60Hz Current 12A Max. 100-127 VAC, 50-60Hz, 12A Max. 200-240 VAC, 50-60Hz, 8A Max. 100-127 VAC, 50-60Hz, 12A Max. 200-240 VAC, 50-60Hz, 15A Max. 10A Max. Software Features The following table lists the Software Features supported by FS 400GbE AI fabric PicOS® switches. Overview High-performance, high-density 400GbE AI fabric PicOS® switches Models N9550-32D EkjvbX8nAo3gJFxWBa9cUrKinXd.png N8610-32D image.png N8610-64D COHLbCpWGolbOWxCpCgcyfXkn5b.jpg N9550-64D HmxzbJ5FYoY583xzsOWcaVjendf.jpg Operating System PicOS® PicOS® PicOS® PicOS® Management Method SNMPv1/v2c/v3, Telnet, SSH 2.0, SSL, FTP SNMPv1/v2c/v3, Telnet, SSH 2.0, SSL, FTP SNMPv1/v2c/v3, Telnet, SSH 2.0, SSL, FTP SNMPv1/v2c/v3, Telnet, SSH 2.0, SSL, FTP Recommended Software Version PicOS® 4.5.0M2 and Later TBD TBD TBD IPv4 Routing OSPFv2, BGP OSPFv2, BGP OSPFv2, BGP OSPFv2, BGP IPv6 Routing OSPFv3, BGP OSPFv3, BGP OSPFv3, BGP OSPFv3, BGP Multicast Routing PIM SM/SSM PIM SM/SSM PIM SM/SSM PIM SM/SSM RoCEv2 Yes Yes Yes Yes PFC/ECN Yes Yes Yes Yes DLB Yes Yes Yes Yes RoCE EasyDeploy Yes Yes Yes Yes MLAG Yes Yes Yes Yes EVPN-VXLAN No Yes Yes Yes MOD (Mirror On Drop) No Yes Yes Yes PTP* Yes No No No DCBX* Yes No No No AmpCon-DC Management Platform Yes Yes Yes Yes *Expected to be available in Q1 2026 Online Resources Datasheet N9550-32D Switch Datasheet N8610-32D Switch Datasheet N8610-64D Switch Datasheet N9550-64D Switch Datasheet Hardware Guide N9550-32D Switch Hardware Guide N8610-32D Switch Hardware Guide N9550-64D Switch Hardware Guide

N8610-32D Switch Hardware Guide

04-09-2025 - N8610-32D Switch Hardware Guide About This Guide This guide provides step-by-step instructions for installing the hardware and performing the initial software configuration of the N8610-32D switch. After completing the installation and basic configuration procedures covered in this guide, you could refer to the PicOS® documentation for information about further software configuration. 1. N8610-32D Overview 1.1 System Overview N8610-32D switch is a high density and power efficient fixed configuration data center switch, providing up to 32 ports of 400/100G in a single rack unit enabling efficient high radix clusters that maximize 12.8 Tbps bandwidth performance of compute and storage by eliminating bottlenecks between leaf and spine tiers. N8610-32D switch delivers wirespeed layer 2 and layer 3 with flexible resources, robust traffic management and load balancing features, optimized for AI generated content lossless network. 1.1.1 Benefits of the N8610-32D Industry-leading wire speed — N8610-32D switch offers 400-Gbps wire speed. Include advanced PicOS® features — MLAG, RoCEv2, PFC, ECN and DLB High-performance packet processing and local storage — Powered by Broadcom BCM56993 Tomahawk 4 with 128GB SSD. Support for channelization — Using breakout cables, each of the 32 400GbE QSFP-DD ports can be broken into 2x 200GbE or 4x 100GbE ports, increasing the total number of supported 200GbE ports per switch to 64 and 100GbE ports per switch to 128. 1.1.2 System Software and Hardware and Software Features The N8610-32D switch runs the PicOS® operating system and delivers Layer 2 and Layer 3 switching, routing, and security services. Table 1. Hardware and Software Features Supported on N8610-32D Switch Model Switch Model Supported System Hardware Features Aggregate Throughput (Bidirectional) Software Features N8610-32D PicOS® Broadcom BCM56993 Tomahawk 4 Chip Intel® Xeon® Processor D-1713NTE 4-Core CPU 32GB SO-DIMM memory 128-GB SSD storage 25.6 Tbps Feature-rich automation capabilities with support for Python, Ansible, and zero-touch provisioning (ZTP) Advanced PicOS® features such as MLAG, RoCEv2, PFC, ECN and DLB. 1.1.3 Channelization in N8610-32D N8610-32D switch supports channelization. You can channelize the 400GbE quad small form-factor pluggable double density (QSFP-DD) ports into interfaces by connecting breakout cables and by using CLI configuration. Table 2. Channelization in N8610-32D Switch Model Ports Port Speed Supported Channelization N8610-32D 1-32 400GbE 4x 100GbE Interfaces 2x 200GbE Interfaces 1-32 100GbE 4x 25GbE Interfaces 1-32 40GbE 4x 10GbE Interfaces Figure 1 shows the front view of the N8610-32D switch. image.png Figure 1. Figure 2 shows the rear view of the N8610-32D switch. image.png Figure 2. Figure 3 shows the components on the front and rear of a N8610-32D switch. image.png Figure 3. 1.2 Chassis 1.2.1 Chassis Physical Specifications The N8610-32D switch is a rigid sheet-metal structure that houses all components of the switch. Table 3. Physical Specifications of the N8610-32D Switch Model Model Height Width Depth Weight N8610-32D 1.71" (4.35cm) 17.26" (43.84cm) 23.23" (59cm) excluding fan and power supply handles 26.12 lbs (11.85 kg) two power supplies and fans installed 1.2.2 Field-Replaceable Units Field-replaceable units (FRUs) are components that you can replace at your site. The FRUs in N8610-32D switch are hot-removable and hot-insertable: You can remove and replace them without powering off the switch or disrupting switch functions. The N8610-32D switch has the following FRUs: Power supplies Fan modules Transceivers Note: Transceivers are not part of the shipping configuration. If you want to purchase any of these components, you must order them separately. 1.2.3 Chassis Status LEDs The front panel of the N8610-32D switch features five chassis status LEDs labeled Loc, Diag, PS1, PS2 and FAN (see Figure 4). 画板 1(6).jpg Figure 4. 1.2.4 LEDs on the Management Port The N8610-32D switch has a RJ-45 management port on the front panel. The figure below shows the location of the management port on the N8610-32D and the LEDs on the port (see Figure 5). The LED indicates link activity of the port. 画板 1 拷贝(5).jpg Figure 5. 1.2.5 Network Port LEDs Each QSFP-DD port on N8610-32D switch has four LEDs that show the link activity of the port (see Figure 6). img_v3_02nj_4fae3f8e-c368-48f7-a3f0-93a82ef3ea2g.jpg Figure 6. 1.3 Cooling System The cooling system in N8610-32D switch consists of fan modules and built-in fans in the power supplies. The airflow direction depends on the fan modules and power supplies installed in the switch. You can order a N8610-32D switch that supports front-to-back airflow (air enters through the front of the switch). The fan modules are hot-removable and hot-insertable field-replaceable units (FRUs) installed in the rear panel of the switch: You can remove and replace them without powering off the switch or disrupting switch functions. 1.3.1 Fan Modules We ship N8610-32D switch with six fan modules (5+1 redundancy) preinstalled in the rear panel. The fan module is available in one airflow direction: Front-to-back (cold air enters through the front of the switch and hot air exhausts through the back of the switch). 1.3.2 Models and Airflow Direction Table 4. Airflow Direction in N8610-32D Switch Model Fan Modules and Power Supplies Direction of Airflow in the Fan Modules and Power Supplies We ship the switch with six fan modules (with front-to-back airflow) and two AC power supplies (with a red ejector lever). Front-to-back—Cold air intake to cool the chassis is through the vents on the front panel of the chassis, and hot air exhausts through the vents on the rear panel of the chassis. N8610-32D Model with Front-to-Back Airflow In the N8610-32D switch models that have front-to-back airflow, the cold air intake to cool the chassis is through the vents on the front panel of the switch and hot air exhausts through the vents on the rear panel (see Figure 7). image.png Figure 7. 1.3.3 How to Position the Switch In front-to-back airflow, hot air exhausts through the vents on the rear panel of the switch. For front-to-back airflow, install the fan modules with the air intake side (typically the side with the fan blades visible or the grille pattern) facing the cold aisle, and the air exhaust side (typically the side with the power connector or handle) facing the hot aisle (see Figure 8). image.png Figure 8. 1.3.4 N8610-32D Fan Module Status There is a status LED for each fan module on N8610-32D switch, next to the fan module slot on the rear panel of the chassis, indicating the status of the fan module. 1.4 Power System The smart power module for the N8610-32D supports power consumption management and hot swapping. It can obtain the output power, output current, and operating temperature in real time. Cautions： To improve system stability and availability, you are advised to configure 1 + 1 power redundancy. The chassis configured with power redundancy works in current-sharing mode. At least one power module is required. If any slot is unoccupied, install a filler panel to enable proper airflow and to keep dust out of the chassis. Unplug the power cord before installing or removing the power module. 1.4.1 AC Power Supply in N8610-32D Switch You can install up to two power supplies in the power supply slots in the rear panel of the N8610-32D switch chassis. On the N8610-32D switch, the slots are labeled PSU1 and PSU2. Figure 9 shows the AC power supply for a N8610-32D switch. N8610-32D电源指示图1.jpg Figure 9. Table 5. Components of AC Power Supply No. Component Description 1 LED Power status LED 2 Fan Forward fan 3 Handle Handle of the power module 4 Power connector Three-pin connector 5 Latch Latch of the power module 1.4.2 AC Power Supply Specifications N8610-32D switch supports 1500W AC power supplies. We ship the N8610-32D switch model with two AC power supplies preinstalled in the rear panel of the chassis. You can install it without powering off the switch or disrupting the switching function. Table 6. Technical Specifications for AC Power Supplies Item Specification Input voltage 100-127VAC 200-240VAC Input frequency 50-60Hz Input current 12A MAX (100-127VAC) 8A MAX (200-240VAC) Hot swapping Supported Cooling Front-to-rear airflow (air exhaust on power module panel) Overvoltage protection Supported Overcurrent protection Supported Over-temperature protection Supported 1.4.3 Power Cord Specifications A detachable AC power cord is supplied with the AC power supplies. The plug end of the power cord fits into the power source outlet that is standard for your geographical location. Table 7. Specifications of the AC Power Cord Countries Power Cord Standard Male Plug Female Connector Voltage Compatibility Maximum Input Amps United States, Canada, Mexico, Puerto Rico, Guam, Japan, Virgin Islands (U.S.) US NEMA 5-15P IEC60320 C13 100-250VAC 10A United Kingdom, Hong Kong, Singapore, Malaysia, Maldives, Qatar, India UK BS1363 IEC60320 C13 100-250VAC 10A Continental Europe, South Africa, Switzerland, Italy, Indonesia EU CEE 7 IEC60320 C13 100-250VAC 10A China, Australia, New Zealand, Argentina CN GB16A IEC60320 C13 100-250VAC 10A 1.4.4 LEDs on the AC Power Supplies Table 8. LED on the AC Power Supply for N8610-32D LED Color Description Power status LED Green The power module is operating normally. Red The power module is operating abnormally. 2. Site Planning, Preparation, and Specifications 2.1 Site Guidelines and Requirements The equipment must be installed indoors for normal operation and prolonged service life. The following sections provide specific information to help you plan for a proper operating environment. 2.1.1 Floor Loading Ensure that the floor under the rack supporting the chassis is capable of supporting the combined weight of the rack and all the other components. 2.1.2 Airflow To ensure adequate airflow through the chassis, maintain a minimum clearance of 20 cm (7.87 in.) around air vents. Route the cables and power cords through the cable management brackets to avoid blocking air intake vents. Dust the equipment every three months to prevent blocking the ventilation openings on the housing. 2.1.3 Space You are advised to have a pathway of 0.8 meters (2.62 ft.) wide in the equipment room. This space ensures that you can remove the components and perform routing maintenance easily. The front and rear of the chassis must remain unobstructed to ensure adequate airflow and prevent overheating inside the chassis. 2.1.4 Temperature To ensure normal operation and prolonged service life of the equipment, maintain an appropriate temperature in the equipment room. Otherwise, the equipment may be damaged. A high temperature can accelerate the aging process of insulation materials, greatly reducing the availability of the equipment and severely affecting its service life. For the device’s operating temperature requirements, please refer to the product datasheet. Note: The operating temperature is measured at a point that is 1.5 m (4.92 ft.) above the floor and 0.4m (1.31 ft.) before the equipment with no protective plates in front or at the back of the equipment. 2.1.5 Humidity To ensure normal operation and prolonged service life of the equipment, maintain an appropriate humidity in the equipment room. Otherwise, the equipment may be damaged. In an environment with a high relative humidity, the insulating material is prone to poor insulation or even electricity leakage. In an environment with a low relative humidity, the insulating strip may dry and shrink, resulting in screw loosening. Furthermore, internal circuits are prone to static electricity For the device’s operating humidity requirements, please refer to the product datasheet. Note: The operating humidity is measured at the point that is 1.5 m (4.92 ft.) above the floor and 0.4 m (1.31 ft.) before the equipment with no protective plates in front or at the back of the equipment. 2.1.6 Cleanliness The indoor dust takes on a positive or negative static electric charge when falling on the switch, causing poor contact of the metallic joint. Such electrostatic adhesion may occur more easily when the relative humidity is low, not only affecting the service life of the switch, but also causing communication faults. The following table lists the requirements for the dust and particles in the equipment room: Table 9. Dust and Particle Requirement Minimum Dust and Particle Diameter Unit Maximum Quantity 0.5 μm particles/m³ 3.5 × 10⁵ 5 μm particles/m³ 3.0 × 10³ Apart from dust, there are also requirements on the salt, acid, and sulfide in the air of the equipment room. These harmful substances will accelerate metal corrosion and component aging. Therefore, the equipment room should be properly protected against harmful gases, such as sulfur dioxide and hydrogen sulfide. The following table lists limits on harmful gases. Table 10. Gas Requirement Gas Average Maximum (mg/m³) mg/m³ cm³/m³ mg/m³ cm³/m³ Sulfur Dioxide (SO₂) 0.3 0.11 1.0 0.37 Hydrogen Sulfide (H₂S) 0.1 0.071 0.5 0.36 Chlorine (Cl) 0.1 0.034 0.3 0.1 Nitrogen Oxides (NO) 0.5 0.26 1.0 0.52 Note: The average value is measured over one week. The maximum value is the upper limit of the harmful gas measured in one week for up to 30 minutes every day. 2.1.7 System Grounding A reliable grounding system is the basis for stable and reliable operation, which is indispensable for preventing lightning strikes and interference. Carefully check the grounding conditions at the installation site according to the grounding specifications, and complete grounding properly based on the site situation. Safety Grounding Ensure that the rack and power distribution system are securely grounded. Otherwise, electric shocks may occur when the insulation resistance between the power module and the chassis becomes small. Note: The building should provide a protective ground connection to ensure that the equipment is connected to a protective earth. Lightning Grounding The surge protection system is an independent system consisting of a lightning rod, a downlead conductor, and a connector connected to the grounding system. The grounding system is usually used for power reference grounding and safety grounding of the rack. EMC Grounding Grounding for the EMC design includes shielded grounding, filter grounding, noise, interference suppression, and level reference. The grounding resistance should be smaller than 1-ohm. Connect the grounding terminal to the ground before operating the equipment. There is a grounding stud in the lower left corner of the rear panel. They are pasted with a conspicuous label. image.png 2.1.8 Preventing Electromagnetic Interference Electromagnetic interference mainly comes from outside the equipment or application system and affects the equipment through capacitive coupling, inductive coupling, electromagnetic waves, and other conduction modes. Interference prevention measures should be taken for the power supply system. Keep the equipment far away from the grounding facility and surge protector facility of the power device. Keep the equipment far away from high-frequency current devices such as the high-power radio transmitting station and radar launcher. Take electromagnetic shielding measures when necessary. 2.1.9 Surge Protection Although the equipment can guard against lightning strikes, strong lightning strikes may still damage the equipment. Take the following surge protection measures: Ensure that the grounding wire of the rack is in good close contact with the ground. Ensure that the neutral point of the AC power socket is in close contact with the ground. You are advised to install a power arrester in front of the power input end to enhance surge protection for the power supply. 2.2 Management Cable Specifications and Pinouts 2.2.1 Console Port Connector Pinout Information The console port on PicOS® devices is an RS-232 serial interface, using an RJ-45 connector to connect to a console management device. The default baud rate for the console port is 115200 baud. 2.2.2 RJ-45 Management Port Connector Pinout Information The RJ-45 connector on PicOS® network devices provides the following pinout details for the management port. Table 11. Pin Signal Definition Table for 1000BASE-T Pin MDI Mode MDI-X Mode 1 Media Dependent Interface A+ Media Dependent Interface B+ 2 Media Dependent Interface A- Media Dependent Interface B- 3 Media Dependent Interface B+ Media Dependent Interface A+ 4 Media Dependent Interface C+ Media Dependent Interface D+ 5 Media Dependent Interface C- Media Dependent Interface D- 6 Media Dependent Interface B- Media Dependent Interface A- 7 Media Dependent Interface D+ Media Dependent Interface C+ 8 Media Dependent Interface D- Media Dependent Interface C- 3. Initial Installation and Configuration 3.1 Unpack and Mount the N8610-32D Switch Below is an optimized guide for unpacking and preparing the N8610-32D switch for installation, including key precautions and potential risk alerts. 3.1.1 Parts Inventory (Packing List) for a N8610-32D Switch The switch shipment includes a packing list. Check the parts you receive with the switch against the items in the packing list. Table 12. Inventory of Components Provided with a N8610-32D Switch Component Quantity Power Cord 2 Grounding Cable 1 Console Cable 1 Front-Post Bracket 2 Rear-Post Bracket 2 Rear Mounting Ear 2 Bracket-Mounting Screw 20 Ear-Locking Screw 2 3.1.2 Mount the N8610-32D Switch on a Rack Make sure the previously mentioned “2.1 Site Guidelines and Requirements” have been met before you begin the installation. Plan for the installation site, networking mode, power supply, and cabling in advance. Then, wear an ESD wrist strap, place the switch, and mount it onto the rack. 3.1.2.1 Installation Requirements Before you begin the installation, make sure that you have the following: Standard-sized, 19-inch-wide rack with a minimum of 1U height available. Category 5e or higher RJ45 Ethernet cables, and fiber optical cables. Phillips screwdriver and adjustable wrench. ESD bracelet, ESD gloves, or ESD clothing. Cable ties, marker, and utility knife. 3.1.2.2 Installation Guidelines Please verify that the front and rear brackets of the rack are in the right locations before mounting. If the front brackets are too close to the front door, there will not be sufficient clearance between the front panel and the door. As a result, the front door cannot be closed after Ethernet cables and optical fibers are connected to the chassis. Generally, maintain a minimum clearance of 10mm (0.39 in.) between the front panel and the front door. Before installation, verify the following guidelines are met: The rack has been secured. The various components in the rack have been installed. There are no obstacles inside or around the rack when installing the switch. 3.1.2.3 Mount the Brackets Attach the front-post brackets and rear-post brackets to the two sides of the switch using the bracket-mounting screws. image.png 3.1.2.4 Mount the Chassis on the Rack The chassis can be installed on a standard 19-inch EIA rack. Mount the chassis on the rack with its front panel facing forward. You are advised to use a tray or guide rails to assist in installing the chassis on the rack. Secure the switch to the rack by fastening the front and rear mounting ears with self-provided screws and cage nuts. image.png Lock the position of the rear mounting ears using the ear-locking screws. image.png 3.1.3 Mount the Chassis on the Workbench If a standard 19-inch EIA rack is not available, mount the switch on a clean workbench. Lay the chassis flat on the workbench and ensure adequate airflow around the chassis. image.png 3.2 Connect the N8610-32D to Power 3.2.1 Connect the N8610-32D Switch to Earth Ground 3.2.1.1 Installation Guidelines A reliable grounding system is the basis for stable and reliable operation, which is indispensable for preventing lightning strikes and interference. The chassis has two grounding studs on its rear panel. Connect the grounding stud to the grounding terminal of the rack and then connect the grounding terminal to the grounding bar of the equipment room. The cross-sectional area of the grounding wire is determined by the maximum possible current. The grounding wire should be of a good conduction quality. Never use bare wires. The combined grounding should have a grounding resistance of less than 1-ohm. 3.2.1.2 Procedure Connect one end of the grounding cable to a proper earth ground, such as the rack in which the switch is mounted. Securely connect the other end of the grounding cable to the switch back panel using the screw and the washer. image.png Danger Warnings: To ensure personal and equipment safety, it is necessary to ground the switch properly. The resistance between the chassis and the ground must be less than 0.1-ohm. The maintenance personnel should check whether the AC power socket is reliably connected to the building's protective ground. If not, the maintenance personnel should use a protective grounding wire to connect the protective ground terminal of the AC power socket to the building's protective ground. The power cord must be plugged into the power socket connected to the earth's ground. The power socket must be installed near the equipment in an easily accessible location. When installing or replacing the unit, the ground connection must always be made first and disconnected last. 3.2.2 Connect Power to N8610-32D Switch Wear an ESD-preventive wrist strap before proceeding with the following operation. 3.2.2.1 Power Module Installation Remove the power module from its packing materials. Make sure the input indicators meet the requirements. Remove the filler panel from the slot by unscrewing the captive screw. Keep the panel with the nameplate facing upwards. Grasp the handle with one hand and place your other hand underneath the power module to support it. Slide the power module along the guide rails into the slot until the module plugs into the receptacle at the back of the slot. Warning: Slide the power module all the way into the chassis gently. Align the power module in the right orientation to the open power slot. If you are not able to push the power module all the way into the slot, carefully slide the module out of the slot, align the module with guide rails, and reinstall the module. All fan and power modules must have the same airflow direction or else an error can occur. 3.2.3 Post-installation Check Note: Before checking the installation, ensure that all power is turned off and disconnected to prevent personal injury and damage to the switch components. The external power supply matches the power distribution system. The front and rear doors of the rack can close properly after installation has been completed. The rack has been completely fastened, which will not move or tilt. The chassis has been mounted on the rack and all cables have been fastened to the rack. Select the proper fan module and tighten captive screws. Select the proper power module. The power module is completely seated in the slot. At least two personnel are required to power on the chassis. Do not service the chassis before it is powered off. Carefully check your work area for possible hazards, such as ungrounded power extension cables, missing safety grounds, and moist floors. Do not subject the equipment to dampness and avoid liquid inside the equipment. Locate the emergency power-off switch in the room. In the case of an electrical accident, you will be able to quickly turn off the power. Never assume that power is disconnected from a circuit. Instead, always check. The power cord is plugged into the power module and retained there. The power cord is long enough to avoid over-extension. The power socket is connected to the earth ground as required with a rated current of at least 10 A. Each power module receives power from a power socket. If a slot is to remain empty, install a filler panel to allow for adequate airflow and to keep dust out of the chassis. 3.2.4 Connect the Power Cable Install one or two AC PSUs and connect them to an AC power source. image.png Cautions: Make sure the power socket is OFF before connecting the power cord. Use a 3-core power cord, with a minimum cross-sectional area of 1.5mm² or 14 AWG per pin. Use a 10A power cord for the AC power supply. Adopt the proper power socket and make sure that the AC power system in the equipment room is capable enough. 3.3 Connect the N8610-32D to the Network 3.3.1 Install a QSFP-DD Transceiver Before you install a transceiver in a device, ensure that you have taken the necessary precautions for safe handling of lasers. Ensure that you have a rubber safety cap available to cover the transceiver. The transceivers for FS devices are hot-removable and hot-insertable field-replaceable units (FRUs). You can remove and replace the transceivers without powering off the device or disrupting the device functions. Note: After you insert a transceiver or after you change the media-type configuration, wait for 6 seconds for the interface to display operational commands. To install a QSFP-DD transceiver: Wrap and fasten one end of an ESD wrist strap around your bare wrist, and connect the other end of the strap to the ESD point on the switch. Verify that a rubber safety cap covers the QSFP-DD transceiver. Position the transceiver in front of the port on the device so that the QSFP-DD connector faces the port. Slide the transceiver into the port until the locking pins lock in place. If there is resistance, remove the transceiver and flip it so that the connector faces the other direction. Laser Warning: Do not look directly into a fiber-optic transceiver or into the ends of fiber-optic cables. Fiber-optic transceivers and fiber-optic cable connected to a transceiver emits laser light that can damage your eyes. CAUTION: Do not leave a fiber-optic transceiver uncovered except when inserting or removing cable. The safety cap keeps the port clean and protects your eyes from accidental exposure to laser light. 3.3.2 Connect a Fiber-Optic Cable Before you connect a fiber-optic cable to an optical transceiver installed in a device, ensure that you have taken the necessary precautions for safe handling of lasers. To connect a fiber-optic cable to an optical transceiver installed in a device: Laser Warning: Do not look directly into a fiber-optic transceiver or into the ends of fiber-optic cables. Fiber-optic transceivers and fiber-optic cables connected to transceivers emit laser light that can damage your eyes. If the fiber-optic cable connector is covered with a rubber safety cap, remove the cap. Save the cap. Remove the rubber safety cap from the optical transceiver. Save the cap. Insert the cable connector into the optical transceiver. image.png Secure the cables so that they do not support their own weight. Place excess cable out of the way in a neatly coiled loop. Placing fasteners on a loop helps cables maintain their shape. CAUTION: Do not bend fiber-optic cables beyond their minimum bend radius. An arc smaller than a few inches in diameter can damage the cables and cause problems that are difficult to diagnose. Do not let fiber-optic cables hang free from the connector. Do not allow fastened loops of cables to dangle, which stresses the cables at the fastening point. 3.4 Connect the N8610-32D to External Devices 3.4.1 Connect a Device to a Network for Out-of-Band Management Ensure that you have an Ethernet cable that has an RJ-45 connector at either end. You can monitor and manage a network device, such as a router or a switch, by using a dedicated management channel. Each device has a management port to which you can connect an Ethernet cable with an RJ-45 connector. Use the management port to connect the device to the management device. To connect a device to a network for out-of-band management: Connect the MGMT port of the switch to a computer with a standard RJ45 Ethernet cable. image.png 3.4.2 Connect a Device to a Management Console Using an RJ‑45 Connector You can configure and manage your network devices through a dedicated management channel, using the console port available on each device. 3.4.2.1 Console Port Management Connect the PC to the device's Console port using a console cable, as shown in the image below. Connect the RJ45 end of a console cable to the RJ45 console port of the switch. Connect the DB9 end of the cable to the serial port of a computer. image.png 3.5 Configure PicOS® on the N8610-32D 3.5.1 Default Configuration We ship each N8610-32D switch programmed with a factory-default configuration that contains the values set for each configuration parameter. The default configuration file sets values for system parameters such as the system log and file messages. When you commit changes to the configuration, a new configuration file is created that becomes the active configuration. You can always revert to the factory-default configuration. This topic shows the factory-default configuration file of a N8610-32D switch: system { login { announcement: "" user operator { class: "read-only" } } services { ssh { port: 22 root-login: "deny" protocol-version: "v2" connection-limit: 20 rate-limit: 20 disable: false idle-timeout: 0 } } inband { enable: false } syslog { local-file: "ram" } log-level: "warning" log-facility: 0 timezone: "UTC" } 3.5.2 Connect and Configure N8610-32D The initial configuration of the switch requires the user to connect the terminal or computer to the switch’s console port. Once the user accesses the switch and establishes the CLI (Command Line Interface) through a serial console connection, an IP address is assigned to the management port, and an IP route to the gateway is created. Keep in mind the following points: The console port provides local serial access to the switch. The Ethernet management port is used for out-of-band network management tasks. Before using the management port for the first time, you must assign an IP address to the port. 3.5.2.1 Connect Console Port Before configuring the device for the first time, you need to access it via the console port. The console port is located at the front of the switch. You can connect a terminal or a computer to the console port using a serial or RS-232 cable. Port Settings Use the following port settings to connect the terminal or computer to the switch console port: Baud rate: 115200 Data bits: 8 Stop bits: 1 The default width for terminal sessions through the console port is 80 characters. This means that the terminal client’s width should be at least 80 characters for proper use of the console port. Most terminal clients have a default width of 80 characters. 3.5.2.2 Assign an IP Address to the Management Interface Once initial access to the switch is obtained, the user needs to configure the management IP address and default gateway in either L2/L3 mode or OVS mode. This section explains the configuration in L2/L3 mode. The management IP address is used for maintaining and managing the device. You can configure a static IP address for the management interface eth0, or you can dynamically assign the address via DHCP. If a static IP address is not assigned, the system will default to attempting to obtain the management port IP address dynamically from the DHCP server. Note: When switching from OVS mode to L2/L3 mode, the static IP address of the management port configured before will still be used if there is no user configuration for it in the new mode. 3.5.2.2.1 Configure Management Interface Step1 Set static IP addresses for management interface eth0. set system management-ethernet eth0 ip-address {IPv4 | IPv6} NOTE: If the static IP address is not assigned, the system will try to dynamically obtain the management port IP address from the DHCP server which is also the factory setting. Step2 Set the gateway address for management interface eth0. set system management-ethernet eth0 ip-gateway {IPv4 | IPv6} 3.5.2.2.2 Configuration Example Step1 Set static IP addresses for management interface eth0. admin@Xorplus# set system management-ethernet eth0 ip-address IPv4 192.168.10.5/24 Step2 Set the gateway address for management interface eth0. admin@Xorplus# set system management-ethernet eth0 ip-gateway IPv4 192.168.10.1 Step3 Commit the configuration. admin@XorPlus# commit Step4 Verify the configuration. Run run show system management-ethernet command to view the configuration information, status and traffic statistics information of the management interface. admin@XorPlus# run show system management-ethernet eth0 Hwaddr: 00:18:23:30:e5:72 State: UP Gateway : 192.168.10.1 Inet addr: 192.168.10.5/24 Traffic statistics Input Packets......................3620 Input Bytes........................462971 Output Packets.....................597 Output Bytes.......................75459

N8610-32D Switch Quick Start Guide V1.0

03-07-2025 - For details, please click the attachment icon below to view or download for a good reading experience or resources.

Lossless Network for RDMA White Paper

26-06-2025 - Lossless Network for RDMA White Paper 1. RDMA Over Converged Ethernet (RoCE) 1.1 Principles of RDMA In traditional TCP/IP communication, data transmission between two hosts involves the following core steps: On the sender side: User-to-kernel memory copy: Application data is copied from user space to the socket send buffer in kernel space. Protocol stack encapsulation: The kernel protocol stack adds TCP/IP headers layer by layer to form a complete packet. DMA transfer: The data is transferred from the kernel buffer to the NIC queue via mechanisms like zero-copy (e.g., sendfile) or direct NIC DMA. On the receiver side: Hardware interrupt handling: Upon packet arrival, the NIC triggers an interrupt, and the kernel uses the NAPI mechanism to store the data into the DMA ring buffer. Protocol stack processing: The kernel decapsulates the packet headers, performs checksum verification, and handles tasks such as out-of-order packet reassembly. Ready data copy: Complete message data is copied from the kernel socket receive buffer to user space memory. Context switch: The application is awakened from kernel mode to user mode to process the received data. Performance bottlenecks include: Four context switches (two for send, two for receive) At least two full memory copies (user space ⇄ kernel space) Latency introduced by protocol stack processing (e.g., fragmentation/reassembly, ACK handling) Overhead from interrupt handling and software interrupt scheduling This architecture causes significant performance degradation in high-speed networking environments. This is where RDMA (Remote Direct Memory Access)—with kernel bypass and zero-copy mechanisms—and kernel stack optimization techniques like XDP and eBPF bring their value. image.png 1.2 Overview of RoCE (RDMA over Converged Ethernet) RoCE is an Ethernet-based Remote Direct Memory Access (RDMA) protocol that enables applications to directly read and write memory between hosts without operating system intervention. This significantly reduces communication latency and CPU overhead. By offloading data transfer tasks to the network adapter, RoCE delivers higher throughput and lower system resource consumption. The second-generation protocol, RoCEv2, builds upon the original architecture by introducing the UDP/IP protocol stack. This enhancement enables Layer 3 routing capability, making RoCEv2 well-suited for large-scale deployments in modern data centers. Advantages of Broadcom RoCEv2 Solution Broadcom’s Ethernet adapters offer hardware-level support for RoCEv2, enabling high-performance, low-latency network communication across a wide range of scenarios, including AI/ML workloads, distributed storage, and high-performance computing (HPC). Key benefits include: High Throughput: Leverages NIC hardware acceleration with support for multi-queue, high-concurrency data transfers. Low Latency: Bypasses the kernel protocol stack, significantly reducing end-to-end latency for applications. Low CPU Utilization: The communication data path is offloaded from the CPU, freeing up host processing resources. 1.3 Three Key Features of RoCEv2 Ethernet-Based Routed Communication: RoCEv2 utilizes UDP/IP encapsulation, enabling it to be routed across Layer 3 networks. This makes it well-suited for large-scale data center deployments based on Spine-Leaf architectures and cross-switch communication. IP QoS Assurance: RoCEv2 supports DiffServ (DSCP) and VLAN priority (802.1p), allowing network devices to perform priority-based traffic scheduling and control. This ensures stable performance for latency-sensitive and mission-critical workloads. IP-Based Congestion Control: RoCEv2 leverages ECN (Explicit Congestion Notification) to enable sender-side rate adjustment based on real-time network feedback. This mechanism effectively mitigates congestion and improves overall transmission efficiency. To further enhance a RoCEv2 network architecture, DCB (Data Center Bridging) configurations can be integrated to enable coordinated optimization of priority-based flow control (PFC), ECN-based congestion management, and queue scheduling strategies. image.png RoCEv2 introduces a network congestion detection and control mechanism based on Explicit Congestion Notification (ECN) and Congestion Notification Packets (CNP) to dynamically adjust data transmission rates and alleviate congestion along the transport path. Core Mechanism: ECN Marking (Congestion Experienced, CE): When a switch or the receiver-side RNIC detects network congestion, it sets the ECN field in the IP header of a packet to indicate that congestion has occurred. Who marks it? Primarily the switches (or ECN-capable routers) in the network. How is it marked? The ECN field in the IP header is set to 11 (CE) to indicate congestion. When is it marked? When the internal buffer or queue length of the switch exceeds a predefined threshold, signaling congestion on the link. Congestion Notification Packet (CNP): Upon receiving a packet with an ECN mark, the receiver-side RNIC generates a special CNP and sends it back to the sender to request a reduction in transmission rate. Who sends it? The receiver's RNIC, upon detecting ECN-marked packets. Who receives it? The original sender's RNIC. Purpose: To notify the sender that congestion has occurred along the transmission path and that it should reduce its sending rate. image.png The RDMA data flow typically involves the following key steps: Application Memory Registration: The application registers a user-space memory region with the RDMA driver using an RDMA API (e.g., ibv_reg_mr). This grants the RDMA Network Interface Card (RNIC) direct access to the memory via DMA and sets local and remote access control policies. RDMA Connection Establishment: A connection is established using a Queue Pair (QP) and an RDMA transport protocol such as InfiniBand, RoCE, or iWARP. Data Transmission: The sender-side NIC performs a direct memory read (via DMA) from the registered local memory. Data is transferred over the PCIe bus to the local RDMA NIC (RNIC). The NIC transmits the data over the InfiniBand/RoCE/iWARP network directly to the remote RNIC. The remote RNIC writes the data via PCIe into the receiver’s registered memory region. Completion and Notification: Once the transmission is complete, the RDMA device updates the relevant status. The application can immediately access the data in memory without additional copying. image.png 1.4 Recommended RoCE RDMA Configuration Guidelines (Example) Item Recommended Setting RoCE Mode Use RoCEv2 (UDP encapsulation, ECN supported) ECN Enable ECN marking on switches when buffer thresholds are exceeded CNP Response Driver supports it by default; DCQCN needs to be enabled PFC Enable PFC only for RDMA-specific traffic classes (e.g., TC1) Buffer Allocate at least 512KB to 1MB per enabled RDMA traffic class (TC) MTU Recommend enabling Jumbo Frames (e.g., MTU 9000) DSCP / PCP Ensure consistent mapping: DSCP → TC → Priority → PFC 2. RoCE Deployment on FS Switches This solution is designed to support low-latency, lossless Ethernet environments based on RoCEv2, especially for application scenarios requiring high bandwidth and low latency—such as AI training, high-performance computing (HPC), and distributed storage. 2.1 Device Selection Guidelines (Recommended Models) Broadcom-based Switching ASICs Ensure RoCE Feature Support Support for lossless Ethernet technologies, including IEEE 802.1Qbb PFC, ECN, and QoS scheduling Built-in support for ECMP, DCQCN, and other congestion control enhancements at the ASIC level Broad compatibility with mainstream RDMA NIC vendors such as Mellanox, Broadcom NetXtreme-E and Intel PicOS: A Flexible and Stable Network OS Supports hybrid CLI (Cisco-like syntax) and Linux Shell for versatile operations Open network OS with support for Ansible, ZTP, SNMP, and custom scripting Unified configuration of PFC, ECN, and priority queues, ideal for large-scale RDMA traffic environments Model Port Configuration Switching Capacity Feature Support Recommended Use Cases N9600-64OD 64 × 800G OSFP 51.2 Tbps RoCEv2, GLB/DLB Cloud computing, AI/ML clusters N8650-32OD* 32 × 800G OSFP 25.6 Tbps RoCEv2, GLB/DLB Cloud computing, AI/ML clusters N8520-32D* 32 × 400G QSFP-DD 12.8 Tbps RoCEv2, MLAG, EVPN-VXLAN Edge/DC interconnect (DCI) N8610-32D* 32 × 400G QSFP-DD 12.8 Tbps RoCEv2, DLB Deep learning networks N9550-32D 32 × 400G QSFP-DD 12.8 Tbps PFC, ECN, QoS, M-LAG AI clusters, large-scale RoCE deployments N9550-64D 64 × 400G QSFP-DD 25.6 Tbps RoCEv2, DLB Hyperscale data centers, RDMA storage clusters N8610-64D* 64 × 400G QSFP-DD 25.6 Tbps RoCEv2, DLB Hyperscale data centers, RDMA storage clusters N8550-24C8D 24 × 200G QSFP56 + 8 × 400G QSFP-DD 2 Tbps RoCEv2, MLAG, EVPN-VXLAN Edge/DC interconnect (DCI) N8550-48BC 48 × 25G SFP28 + 8 × 100G QSFP28 2 Tbps EVPN-VXLAN, MLAG, Multihoming Medium-scale cluster edge access * Indicates the device is under development Data Rate Model & Features 800G N9600-64OD (2U) 64 x 800G OSFP TH5 BCM78900 image.png N8650-32OD* (1U) 32 x 800G OSFP TH5 BCM78902 image.png 400G N9550-64D (4U) 64 x 400G QSFP-DD TH4 BCM56990 image.png N8610-64D* (2U) 64 x 400G QSFP-DD TH4 BCM56990 image.png N9550-32D (1U) 32 x 400G QSFP-DD TH3 BCM56980 image.png N8610-32D* (1U) 32 x 400G QSFP-DD TH4 BCM56993 image.png N8520-32D* (1U) 32 x 400G QSFP-DD TD4-X11 BCM56880 image.png 100/200G N8550-24CD8D (1U) 24 x 200G QSFP56, 8 x 400G QSFP-DD TD4-X9 BCM56780 image.png N8550-64C (2U) 64 x 100G QSFP28 TH2 BCM56970 image.png N8550-32C (1U) 32 x 100G QSFP28 TD3-X7 BCM56870 image.png N8560-32C (1U) 32 x 100G QSFP28 TD3-X7 BCM56870 image.png 10/25G N8550-48B8C (1U) 48 x 25G SFP28, 8 x 100G QSFP28, 2 x 10G SFP+ TD3-X7 BCM56873 image.png N5570-48S6C (1U) 48 x 10G SFP+, 6 x 100G QSFP28 TD3-X5 BCM56771 image.png N5850-48X6C (1U) 48 x 10G RJ45, 6 x 100G QSFP28 TD3-X5 BCM56771 image.png N5850-48S6Q (1U) 48 x 10G SFP+, 6 x 40G QSFP+ TD2+ BCM56864 image.png 2.2 Switch Configuration Recommendations for RoCE Networks Feature Recommended Setting Description Port MTU 9216 Must be larger than RoCE packet size to prevent fragmentation RoCE Traffic Priority Priority 3 Recommended fixed priority for PFC traffic control Enable PFC priority-flow-control priority 3 enable Enable PFC on the specified priority; ensure consistency with server settings Enable ECN ecn enable Enable ECN marking for congestion feedback DSCP/COS Mapping DSCP 4791 → CoS 3 → TC 3 Ensure RoCEv2 traffic matches the configured lossless queue QoS Queue Binding Bind each priority to a dedicated queue Guarantee isolation and determinism for RoCE traffic Lossless Queue Routing Based on UDP port 4791 or VLAN/QoS tag Ensure RoCE traffic enters the lossless priority queue Broadcast/Unknown Multicast Disable if unnecessary Reduce interference and preserve RDMA path stability 2.3 RoCE Configuration Steps on FS Switches (Example: N8560/N9550 Series) Step 1: Enable PFC (Priority Flow Control) Configure PFC on all relevant ports of the switch to prevent packet loss by pausing traffic during periods of congestion. The following steps complete the configuration: Create a PFC configuration profile named pfc1. Apply the PFC profile to port te-1/1/1. admin@PICOS# set class-of-service pfc-profile pfc1 admin@PICOS# set class-of-service interface te-1/1/1 pfc-profile pfc1 admin@PICOS# commit Display service statistics for the specified interface. In the PFC framework, classes 0 through 7 correspond to the 802.1p priority values. If port te-1/1/1 receives a PFC frame, the RxPFC counter will increase by 1. If port te-1/1/1 sends a PFC frame, the TxPFC counter will increase by 1. admin@PICOS# run show class-of-service interface te-1/1/1 Interface : te-1/1/1 802.1P Priority Flow Control RxPFC TxPFC 0 true 0 500 1 true 0 0 2 true 0 71 3 true 0 0 4 true 0 0 5 true 0 0 6 true 0 102 7 true 0 0 trust mode : ieee-802.1 Default ieee-802.1 : 0 Default dscp : 0 Default inet-precedence : 0 Local-priority Queue-Schedule Code-points 0 SP,0kbps 1 SP,0kbps 2 SP,0kbps 3 SP,0kbps 4 SP,0kbps 5 SP,0kbps 6 SP,0kbps 7 SP,0kbps Step 2: Configure PFC Buffers Once PFC is enabled, default buffer space is allocated for priority queues. To make optimal use of available buffer resources, you can fine-tune buffer thresholds for specific queues as needed. The following commands complete the configuration: Set the MTU of interface te-1/1/1 to 9000. Set the guaranteed buffer limit for PFC queue 3 on interface te-1/1/1 to 24,000 cells. Set the static threshold for the shared buffer of PFC queue 1 on interface te-1/1/1 to 10,000 cells. Set the offset for the shared buffer of PFC queue 1 on interface te-1/1/1 to 3,000 cells. admin@PICOS# set interface gigabit-ethernet xe-1/1/1 mtu 9000 admin@PICOS# set interface gigabit-ethernet te-1/1/1 ethernet-switching-options buffer ingress-queue 3 guaranteed 24000 admin@PICOS# set interface gigabit-ethernet te-1/1/1 ethernet-switching-options buffer ingress-queue 1 threshold 10000 admin@PICOS# set interface gigabit-ethernet te-1/1/1 ethernet-switching-options buffer ingress-queue 1 reset-offset 3000 admin@PICOS# commit Step 3: Configure PFC Watchdog The PFC Watchdog helps detect and recover from PFC deadlock conditions. Use the following commands to complete the configuration: Enable PFC Watchdog on queue 5 of interface te-1/1/1. Set the deadlock detection interval to 10 × 100 ms, where 100 ms is the default granularity of the detection timer. Set the recovery timeout to 10 × 100 ms, where 100 ms is the default granularity of the recovery timer. admin@PICOS# set class-of-service interface te-1/1/1 pfc-watchdog code-point 5 enable true admin@PICOS# set class-of-service pfc-watchdog code-point 5 detect-interval 10 admin@PICOS# set class-of-service pfc-watchdog code-point 5 restore-interval 10 admin@PICOS# commit After configuration: Use the command show pfc-watchdog config to view the PFC Watchdog configuration. admin@PICOS# run show pfc-watchdog config PORT ACTION QUEUE DETECTION TIME RESTORATION TIME ---------- ----------- ------------ ---------------- ------------------ te-1/1/1 forward 5 1000 1000 Use the command show pfc-watchdog statistics to view statistics, including: The number of PFC pause storms detected and recovered on each queue of the interface. The number of packets dropped due to PFC deadlock. admin@PICOS# run show pfc-watchdog stats QUEUE STATUS STORM DETECTED/RESTORED TX OK/DROP TX LAST OK/DROP ------------ ----------- ------------------------- ---------------- ----------------- te-1/1/1:5 stormed 9/8 82072626556/0 32053822365/0 Step 4: Configure ECN (Explicit Congestion Notification) Users should continuously monitor network performance and ECN marking rates. ECN threshold values should be adjusted dynamically as needed to adapt to changing network conditions. Use the following commands to complete the configuration: Enable WRED on queue 0 of interface te-1/1/1. Set the maximum threshold to 400 and the minimum threshold to 200 on queue 0 of interface te-1/1/1. Set the packet drop probability to 50% on queue 0 of interface te-1/1/1. Enable ECN on queue 0 of interface te-1/1/1. admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 enable true admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 max_thresh 400 admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 min_thresh 200 admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 drop_probability 50 admin@PICOS# set interface gigabit-ethernet te-1/1/3 wred queue 0 ecn_thresh 1 admin@PICOS# commit After configuration: Use the command to display WRED configuration for the specified interface. admin@PICOS# run show interface gigabit-ethernet te-1/1/1 wred Queue Num Min Thresh Max Thresh Drop Probability ECN Thresh Status ---------- ---------- ---------- ---------------- ----------- ---------- 0 200 400 50% Enabled Enabled 1 0 0 0% Disabled Disabled 2 0 0 0% Disabled Disabled 3 0 0 0% Disabled Disabled 4 0 0 0% Disabled Disabled 5 0 0 0% Disabled Disabled 6 0 0 0% Disabled Disabled 7 0 0 0% Disabled Disabled Step 5: Enable Dynamic Load Balancing for ECMP By enabling dynamic load balancing, Equal-Cost Multi-Path (ECMP) routing can distribute traffic more evenly across multiple member links. This maximizes load balancing efficiency among ECMP paths. admin@PICOS# set interface ecmp hash-mapping dlb-normal admin@PICOS# commit 2.4 RoCE EasyDeploy Initialization RoCE EasyDeploy is a feature designed to simplify the deployment and configuration of RoCE (RDMA over Converged Ethernet) on switches. It enables seamless integration with servers to optimize network performance. This feature allows users to easily switch between lossless and lossy modes, ensuring optimal performance in diverse network environments. Features & Benefits Simplifies RoCE deployment and configuration on switches Enables fast switching between lossless and lossy transmission modes Fewer configuration steps with optimized default parameters Flexible interface-level control Enhances network stability and ease of maintenance Limitations & Guidelines Supported only on Tomahawk2, Trident3-X7, and Tomahawk3 platforms The number of ECN and PFC queues on the switch must align with server-side configuration Post-deployment fine-tuning is supported and recommended based on network conditions Continuous monitoring of RoCE statistics is required for performance assurance Configuration Example Set RoCE mode to lossless Apply RoCE mode to all interfaces admin@PICOS# set class-of-service roce mode lossless admin@PICOS# set class-of-service roce apply all admin@PICOS# commit Verify RoCE configuration after deployment admin@PICOS# run show class-of-service roce status applied mode lossless congestion-control congestion-mode ECN enabled-queue 3 max-threshold 1500000 bytes min-threshold 150000 bytes probability 100 pfc pfc-priority 3 rx-enabled enabled tx-enabled enabled trust trust-mode dscp RoCE PCP/DSCP->LP mapping configurations =========================================== local-priority dscp ------------ ------------------- 0 0,1,2,3,4,5,6,7 1 8,9,10,11,12,13,14,15 2 16,17,18,19,20,21,22,23 3 24,25,26,27,28,29,30,31 4 32,33,34,35,36,37,38,39 5 40,41,42,43,44,45,46,47 6 48,49,50,51,52,53,54,55 7 56,57,58,59,60,61,62,63 RoCE LP->FC mapping and ETS configurations ============================================= local-priority forwarding-class scheduler-weight -------------- ---------------- ---------------- 0 default WRR-8 1 default WRR-8 2 default WRR-8 3 roce WRR-8 4 default WRR-8 5 default WRR-8 6 cnp SP 7 default WRR-8 3 RoCE Configuration on Ethernet NICs 3.1 Hardware Requirements A low-frequency CPU can increase RDMA polling latency. Since RDMA relies on Direct Memory Access (DMA) for high-speed data transfers, proper hardware provisioning is essential. CPU Configuration PCIe 5.0 Support: The CPU must support a PCIe 5.0 controller with at least 40 PCIe 5.0 lanes to ensure sufficient bandwidth. High-Frequency, Multi-Core: At least 16 cores / 32 threads are required to handle high RDMA concurrency. NUMA Optimization: Use NUMA binding to ensure RDMA test processes and NICs are located on the same NUMA node. Use tools like numactl --cpunodebind, numactl --membind, or taskset to reduce latency caused by cross-node access. Category Low-Latency RDMA(HPC, Small Packet) High-Throughput RDMA(AI/Storage, Large Packet) CPU Frequency ≥ 3.5 GHz (Ideal: 4.0 GHz+) ≥ 2.5 GHz (Recommended: 3.0 GHz+) Core Count 16–32 cores (Single-core performance priority) 32–128 cores (Multi-core scaling priority) PCIe Version PCIe 5.0 x16 (Required) PCIe 5.0 x16 (Required) Arch Optimization Disable C-States, pin to high-performance cores NUMA binding, optimized for parallelism Recommended CPUs Intel Xeon 8468H (3.8GHz, 48C)AMD EPYC 9554 (3.35GHz, 64C) AMD EPYC 9754 (3.1GHz, 128C)Intel Xeon 8490H (3.5GHz, 60C) Memory Configuration High Bandwidth: Use DDR5 4800 MT/s or higher. For example, DDR5 at 4400 MT/s (2DPC) delivers ~35.2 GB/s per channel (≈281.6 Gbit/s). Note: Intel 4th Gen Silver CPUs support up to 4000 MT/s. Sufficient Capacity: RDMA requires a large amount of pre-pinned memory. 128 GB is the minimum recommended. 1 TB or more may be needed for large-scale tests. Ensure ulimit -l unlimited is set to enable large locked memory via mlock(), avoiding swap-related performance degradation. NUMA Affinity: Bind memory to the same NUMA node as the RDMA process. Use numactl -N or taskset -c Memory Category Low-Latency RDMA (HPC, Small Packet) High-Throughput RDMA (AI/Storage, Large Packet) Memory Capacity ≥ 64 GB (128 GB+ recommended) ≥ 128 GB (512 GB+ recommended) Memory Channels ≥ 4 channels (8+ recommended) ≥ 8 channels Memory Speed DDR5 4800 MT/s or higher DDR5 4800 MT/s or higher NUMA Binding Required (bind to same NUMA node) Required (bind to same NUMA node) HugePages Enable 2MB/1GB HugePages Enable 2MB/1GB HugePages BIOS Configuration (Refer to vendor documentation for exact settings) Disable virtualization Set CPU mode to performance Disable C-States and P-States to reduce frequency fluctuations and latency Disable IOMMU, or set it to pass-through (PT) mode UEFI/BIOS Area Value BIOS -> Processor Settings Logical Processor = Disable Virtualization Technology = Disable SubNumaCluster = Disable MADt Core cluster = Linear BIOS -> Integrated Devices Global SRIOV = Disable BIOS -> System Profile Setting Server System Profile = Performance Workload = Not Configured BIOS -> System Security AC Recovery Delay = Random (highly recommended) 3.2 NIC Selection 3.2.1 Broadcom Part Number ASIC Chip Ports Connector BCM957504-P425G BCM57504 4x 25G SFP28 BCM957508-P2100G BCM57508 (Thor1) 2x 100G 1x 200G QSFP56 BCM957508-P1200G BCM57508 (Thor1) 1x 200G QSFP56 BCM957608-P2200G BCM57608 (Thor2) 2x 200G 1x 400G QSFP-DD BCM957608-P1400G BCM57608 (Thor2) 1x 400G QSFP112-DD Step 1: Install IP and RoCE Drivers All validation and testing are based on Ubuntu 22.04 LTS, using the Broadcom BCM957608-P1400G network adapter. The driver package version used is 232.1.132.8. Download the appropriate driver installation package from the Broadcom Ethernet network adapter Driver download page. The package includes an automated tool for driver installation and system configuration. After extracting the ZIP file, the contents are placed in a directory, referred to below as Ubuntu_DIR. The installer provides the following: Installation of included drivers, firmware, RDMA libraries, and utilities (niccli) Configuration of interface priorities for both IP and RoCE Sample switch configuration templates Installation of additional system packages: ansible, libibverbs-utils, rdmacm-utils, perftest, gcc, make, rpmbuild, kernel headers, etc. 代码块 root@FS1:~# sudo apt install -y automake autoconf libtool libibverbs-dev ibverbs-utils infiniband-diags perftest ethtool root@FS1:~# sudo tar -zxvf bcm_232.1.132.8c.tar.gz -C Ubuntu_DIR root@FS1:~# sudo cd Ubuntu_DIR/bcm_232.1.132.8c/utils/linux_installer root@FS1:~# sudo bash install.sh -i -i -f -g # =bus id Verify if the specified version is installed 代码块 root@FS1:~# sudo dmesg | grep -i bnx | grep -i ver [ 4.285625] bnxt_en: module verification failed: signature and/or required key missing - tainting kernel [ 4.327736] Broadcom NetXtreme-C/E/S driver bnxt_en v1.10.3-232.0.155.5 [ 45.158387] bnxt_re: Broadcom NetXtreme-C/E RoCE Driver bnxt_re v232.0.155.5 (December 10, 2024) [ 7655.824701] Broadcom NetXtreme-C/E/S driver bnxt_en v1.10.3-232.0.155.5 [ 7664.413538] bnxt_re: Broadcom NetXtreme-C/E RoCE Driver bnxt_re v232.0.155.5 (December 10, 2024) [ 7676.806627] Broadcom NetXtreme-C/E/S driver bnxt_en v1.10.3-232.0.155.5 [ 7685.553099] bnxt_re: Broadcom NetXtreme-C/E RoCE Driver bnxt_re v232.0.155.5 (December 10, 2024) root@FS1:~# modinfo bnxt_en | grep -i ver version: 1.10.3-232.0.155.5 description: Broadcom NetXtreme-C/E/S network driver srcversion: A88BAEC6211445D95464C78 vermagic: 6.8.0-57-generic SMP preempt mod_unload modversions root@FS1:~# modinfo bnxt_re | grep -i ver version: 232.0.155.5 description: Broadcom NetXtreme RoCE Driver srcversion: B738B8F51148E0DA45D7322 depends: ib_uverbs,ib_core,bnxt_en vermagic: 6.8.0-57-generic SMP preempt mod_unload modversions root@FS1:~# dkms status|grep -i bnxt bnxt_en/1.10.3.232.0.155.5, 6.8.0-57-generic, x86_64: installed bnxt_re/232.0.155.5, 6.8.0-57-generic, x86_64: installed Step 2: Manually Configure RoCE Settings The default RoCE configuration is as follows: RoCEv2 (RDMA over IPv4) is enabled Congestion Control and PFC are enabled RoCE traffic is marked with DSCP 26 on priority 3 RoCE CNP traffic is marked with DSCP 48 on priority 7 MTU is set to 1500 bytes root@FS1:~# cat /etc/bnxt_re/bnxt_re.conf ENABLE_FC=1 FC_MODE=3 ROCE_PRI=3 ROCE_DSCP=26 CNP_PRI=7 CNP_DSCP=48 ROCE_BW=50 UTILITY=3 Switch ECN + PFC Synchronization Example qos map dscp 26 to traffic-class 3 qos map dscp 48 to traffic-class 7 interface Ethernet1/1 mtu 1500 tx-queue 3 random-detect ecn minimum-threshold 500 kbytes maximum-threshold 1500 kbytes max-mark-probability 20 priority-flow-control mode on priority-flow-control priority 3 no-drop Manually Modify NIC RoCE Configuration The NIC has RDMA enabled by default. You can verify and configure it using the NICCLI tool. Enable the RDMA option 代码块 To check the option value: sudo niccli -i nvm -getoption support_rdma -scope # Example: root@FS1:~# sudo niccli -i 1 nvm -getoption support_rdma -scope 0 ------------------------------------------------------------------------------- NIC CLI v232.0.153.0 - Broadcom Inc. (c) 2024 (Bld-106.52.39.138.16.0) ------------------------------------------------------------------------------- support_rdma = True To enable the option: sudo niccli -i nvm -setoption support_rdma -scope -value 1 # value 0: Default # value 1: RoCE # Example: root@FS1:~# sudo niccli -i 1 nvm -setoption support_rdma -scope 0 -value 1 -------------------------------------------------------------------------------- NIC CLI v232.0.153.0 - Broadcom Inc. (c) 2024 (Bld-106.52.39.138.16.0) ------------------------------------------------------------------------------- support_rdma is set successfully Enable the RoCE performance profile on the NIC To check the option value: sudo niccli -i nvm -getoption performance_profile # Example: root@FS1:~# sudo niccli -i 1 nvm -getoption performance_profile ------------------------------------------------------------------------------- NIC CLI v232.0.153.0 - Broadcom Inc. (c) 2024 (Bld-106.52.39.138.16.0) ------------------------------------------------------------------------------- performance_profile = default To enable the option: sudo niccli -i nvm -getoption performance_profile # value 0: Default # value 1: RoCE # Example: root@FS1:~# sudo niccli -i 1 nvm -setoption performance_profile -value 1 ------------------------------------------------------------------------------- NIC CLI v232.0.153.0 - Broadcom Inc. (c) 2024 (Bld-106.52.39.138.16.0) ------------------------------------------------------------------------------- performance_profile is set successfully Please do the device reset to apply the configuration Enable PCIe relaxed ordering on the NIC To check the option value sudo niccli -i nvm -getoption pcie_relaxed_ordering # Example: root@FS1:~# sudo niccli -i 1 nvm -getoption pcie_relaxed_ordering ------------------------------------------------------------------------------- NIC CLI v232.0.153.0 - Broadcom Inc. (c) 2024 (Bld-106.52.39.138.16.0) ------------------------------------------------------------------------------- pcie_relaxed_ordering = Disabled To enable the option: sudo niccli -i nvm -setoption pcie_relaxed_ordering -value 1 # Example: root@FS1:~# sudo niccli -i 1 nvm -setoption pcie_relaxed_ordering -value 1 ------------------------------------------------------------------------------- NIC CLI v232.0.153.0 - Broadcom Inc. (c) 2024 (Bld-106.52.39.138.16.0) ------------------------------------------------------------------------------- pcie_relaxed_ordering is set successfully Please reboot the system to apply the configuration Modify congestion control (cc) parameters Option Description cc_mode 0 is used for DCQCN-D (deterministic marking); 1 is for interoperability with DCQCN-P (probabilistic marking). Default is 1. ecn_enable Enable ECN. g Weight for running average in rate calculation: weight is 2^g, where g is between 1 and 9. Default is 8. init_cr Initial rate when QP is created or before inactivity_th expires. Value between 0–1023, representing a percentage of link BW. Required when PF is enabled. init_tr Target rate after inactivity_th expires and QP is created. Value between 0–1023, representing a percentage of link BW. rtt Round-trip time (RTT) in nanoseconds, used to calculate the interval between CNP and actual packet processing. Default is 100000 ns. inact_cp If QP is inactive, the counter used for suppression logic will decay after the threshold expires. Default is 100000 microseconds. cnp_dscp DSCP value for CNP packets. roce_dscp DSCP value for RoCE traffic. cnp_prio Priority for CNP packets. roce_prio Priority for RoCE traffic. apply Apply the settings. mkdir -p /sys/kernel/config/rdma_cm/bnxt_re0 echo RoCE v2 > /sys/kernel/config/rdma_cm/bnxt_re0/ports/1/default_roce_mode mkdir -p /sys/kernel/config/bnxt_re/bnxt_re0 cd /sys/kernel/config/bnxt_re/bnxt_re0/ports/1/cc/ echo -n 0x1a > roce_dscp echo -n 0x3 > roce_prio echo -n 0x1 > disable_prio_vlan_tx echo -n 0x1 > ecn_enable echo -n 0x30 > cnp_dscp echo -n 0x7 > cnp_prio #After setting all the above parameters, apply the values to HW echo -n 0x1 > apply Configure Congestion Control Using bnxt_setupcc.sh The bnxt_setupcc.sh tool uses niccli or lldptool to configure network parameters with a single command. An example is shown below: 代码块 root@FS1:~# sudo bnxt_setupcc.sh -d bnxt_re0 -i ens4f0np0 -m 3 -s 26 -p 48 -r 3 -c 7 -u 3 ENABLE_PFC = 1 ENABLE_CC = 1 ENABLE_DSCP = 1 ENABLE_DSCP_BASED_PFC = 1 L2 50 RoCE 50 Using Ethernet interface ens2np0 and RoCE interface bnxt_re0 Setting pfc/ets 0000:2a:00.0 ------------------------------------------------------------------------------- NIC CLI v232.0.153.0 - Broadcom Inc. (c) 2024 (Bld-106.52.39.138.16.0) ------------------------------------------------------------------------------- BCM57608> AppTLV deleted successfully. BCM57608> AppTLV deleted successfully. BCM57608> AppTLV deleted successfully. BCM57608> Enhanced transmission selection (ets) configured successfully. BCM57608> pfc configured successfully. BCM57608> AppTLV configured successfully. BCM57608> AppTLV configured successfully. BCM57608> AppTLV configured successfully. BCM57608> Settings Default to use RoCE-v2 root@FS1:~# sudo niccli -i 1 getqos /opt/niccli/niccli.x86_64: /lib/x86_64-linux-gnu/libnl-3.so.200: no version information available (required by /opt/niccli/niccli.x86_64) ------------------------------------------------------------------------------- NIC CLI v232.0.153.0 - Broadcom Inc. (c) 2024 (Bld-106.52.39.138.16.0) ------------------------------------------------------------------------------- IEEE 8021QAZ ETS Configuration TLV: PRIO_MAP: 0:0 1:0 2:0 3:1 4:0 5:0 6:0 7:2 TC Bandwidth: 50% 50% 0% TSA_MAP: 0:ets 1:ets 2:strict IEEE 8021QAZ PFC TLV: PFC enabled: 3 IEEE 8021QAZ APP TLV: APP#0: Priority: 7 Sel: 5 DSCP: 48 APP#1: Priority: 3 Sel: 5 DSCP: 26 APP#2: Priority: 3 Sel: 3 UDP or DCCP: 4791 TC Rate Limit: 100% 100% 100% 0% 0% 0% 0% 0% Parameter Description： Option Description -d RoCE Device Name (for example, bnxt_re0, bnxt_re_bond). -i Ethernet Interface Name (for example, p1p1 or for bond, specify secondary interfaces like -i p6p1 -i p6p2). -r [0-7] RoCE packet priority. -s VALUE RoCE packet DSCP value. -c [0-7] RoCE CNP packet priority. -p VALUE RoCE CNP packet DSCP value. -v VALUE Use specified VLAN ID instead of DSCP. -b VALUE RoCE Bandwidth percentage for ETS configuration. Default is 80%. -h Display help. -m [1-3] 1 – PFC only.2 – CC only.3 – PFC + CC mode. -u [2-3] Utility to configure QoS settings.2 – Use lldptool3 – Use niccli utility 3.2.2 NVIDIA Part Number ASIC Chip Ports Connector MCX512A-ACAT ConnectX-5 2x 25G SFP28 MCX623106AN-VDAT ConnectX-6 2x 100G QSFP56 MCX653106A-HDAT ConnectX-6 2x 200G QSFP56 MCX755106AS-HEAT ConnectX-7 2x 200G QSFP112 MCX75310AAS-HEAT ConnectX-7 1x 400G OSFP Step 1: Install IP and RoCE Drivers All testing is conducted on Ubuntu 22.04 LTS, using the BCM957608-P1400G NIC. The installation package version used is 232.1.132.8. For NVIDIA NICs, both RoCE and IP drivers are integrated into the MLNX_OFED package. Multiple installation methods are available. You can download the MLNX_OFED version that matches your system from the NVIDIA official driver page. The MLNX_OFED package includes all necessary components for NVIDIA NICs (e.g., ConnectX-5/6/7), including: Kernel modules and drivers: mlx5_core, ib_uverbs, ib_core User-space protocol libraries and development APIs: for RDMA and RoCE application development Device management and debugging tools Performance testing and benchmarking tools wget https://content.mellanox.com/ofed/MLNX_OFED-/MLNX_OFED_LINUX--ubuntu22.04-x86_64.tgz tar -xvzf MLNX_OFED_LINUX-*.tgz cd MLNX_OFED_LINUX-* sudo ./mlnxofedinstall --add-kernel-support Verify if the specified version is installed 代码块 root@FS1:~# sudo dmesg | grep -i mlx5 | grep -i ver [115345.338018] mlx5_core 0000:45:00.0: mlx5_pcie_event:304:(pid 3648990): PCIe slot advertised sufficient power (75W). [116552.824757] mlx5_core 0000:45:00.0: firmware version: 28.43.1014 [116553.127338] mlx5_core 0000:45:00.0: mlx5_pcie_event:304:(pid 3732904): PCIe slot advertised sufficient power (75W). [116578.352357] mlx5_core 0000:1f:00.0: firmware version: 28.43.1014 [116578.657001] mlx5_core 0000:1f:00.0: mlx5_pcie_event:304:(pid 3732904): PCIe slot advertised sufficient power (75W). root@FS1:~# sudo modinfo mlx5_core | grep -i ver version: 24.07-0.6.1 description: Mellanox 5th generation network adapters (ConnectX series) core driver srcversion: DBC460DDA01C102939C89DC vermagic: 6.8.0-52-generic SMP preempt mod_unload modversions root@FS1:~# sudo modinfo mlx5_ib |grep -i ver description: Mellanox 5th generation network adapters (ConnectX series) IB driver srcversion: 84F601BAD872C13B202813E depends: mlx5_core,ib_uverbs,ib_core,macsec,mlx_compat vermagic: 6.8.0-52-generic SMP preempt mod_unload modversions root@FS1:~# sudo dkms status |grep -i mlnx-ofed mlnx-ofed-kernel/24.07.OFED.24.07.0.6.1.1, 6.8.0-52-generic, x86_64: installed mlnx-ofed-kernel/24.07.OFED.24.07.0.6.1.1, 6.8.0-57-generic, x86_64: installed Step 2: Manually Configure RoCE Settings Zero Touch RoCE (ZTR) is a plug-and-play RDMA auto-optimization solution that automatically configures key parameters such as PFC, ECN, and DSCP without manual intervention. This ensures lossless RoCEv2 traffic over Ethernet. The default RoCE configuration includes: RoCEv2 (RDMA over IPv4) is enabled PFC is enabled ECN is enabled on all priority queues DCQCN is configured based on ECN RoCEv2 CNP packets are marked with DSCP 46 MTU is set to 1500 bytes 代码块 root@FS1:~# sudo mlnx_qos -i ens2np0 -a DCBX mode: OS controlled84 Priority trust state: dscp dscp2prio mapping: prio:0 dscp:07,06,05,04,03,02,01,00, prio:1 dscp:15,14,13,12,11,10,09,08, prio:2 dscp:23,22,21,20,19,18,17,16, prio:3 dscp:31,30,29,28,27,26,25,24, prio:4 dscp:39,38,37,36,35,34,33,32, prio:5 dscp:47,46,45,44,43,42,41,40, prio:6 dscp:55,54,53,52,51,50,49,48, prio:7 dscp:63,62,61,60,59,58,57,56, default priority: Receive buffer size (bytes): 19872,243072,0,0,0,0,0,0,max_buffer_size=2069280 Cable len: 7 PFC configuration: priority 0 1 2 3 4 5 6 7 enabled 0 0 0 0 0 0 0 0 buffer 1 1 1 1 1 1 1 1 tc: 0 ratelimit: unlimited, tsa: vendor priority: 0 tc: 1 ratelimit: unlimited, tsa: vendor priority: 1 tc: 2 ratelimit: unlimited, tsa: vendor priority: 2 tc: 3 ratelimit: unlimited, tsa: vendor priority: 3 tc: 4 ratelimit: unlimited, tsa: vendor priority: 4 tc: 5 ratelimit: unlimited, tsa: vendor priority: 5 tc: 6 ratelimit: unlimited, tsa: vendor priority: 6 tc: 7 ratelimit: unlimited, tsa: vendor priority: 7 Manually Modify NIC RoCE Configuration RDMA is enabled by default on the NIC. You can use the mlnx_qos tool to verify and configure RoCE-related settings. DCQCN-ECN Configuration 代码块 root@FS1:~# sudo mlxconfig -d mlx5_0 query|grep -i ROCE_CC_PRIO_MASK ROCE_CC_PRIO_MASK_P1 255 root@FS1:~# mlxconfig -d mlx5_0 set ROCE_CC_PRIO_MASK_P1 =0x08 Device #1: ---------- Device type: ConnectX7 Name: MCX75310AAS-NEA_Ax Description: NVIDIA ConnectX-7 HHHL Adapter card; 400GbE / NDR IB (default mode); Single-port OSFP; PCIe 5.0 x16; Crypto Disabled; Secure Boot Enabled; Device: mlx5_2 Configurations: Next Boot New ROCE_CC_PRIO_MASK_P1 255 0x10(16) Apply new Configuration? (y/n) [n] : Note: 255 in binary is 11111111, which corresponds to 8 priority queues. This means DCQCN is enabled for all priorities (Priority 0–7) — RoCE congestion control will apply to all traffic. To enable DCQCN only on Priority 3, convert the binary value 00001000 to hexadecimal, which equals 0x10 RoCE_np Settings 代码块 To check cnp_dscp options root@FS1:~# cat /sys/class/net/ens2np0/ecn/roce_np/cnp_802p_prio 6 root@FS1:~# cat /sys/class/net/ens2np0/ecn/roce_np/cnp_dscp 48 root@FS1:~# cat /sys/class/net/ens2np0/ecn/roce_np/enable/* 1 1 1 1 1 1 1 1 # Indicates that all queues have RoCE_np enabled by default. No modification is needed unless customization is required. To temporary configuration root@FS1:~# echo 5 >/sys/class/net/ens2np0/ecn/roce_np/cnp_802p_prio 5 root@FS1:~# echo 36 >/sys/class/net/ens2np0/ecn/roce_np/cnp_802p_prio 36 To persistent configuration mlxconfig -d ibdev_name or /dev/mst/ -y s CNP_DSCP_P1= CNP_802P_PRIO_P1= #example root@FS1:~# sudo mlxconfig -d mlx5_0 -y set CNP_DSCP_P1=36 CNP_802P_PRIO_P1=5 Device #1: ---------- Device type: ConnectX7 Name: MCX75310AAS-NEA_Ax Description: NVIDIA ConnectX-7 HHHL Adapter card; 400GbE / NDR IB (default mode); Single-port OSFP; PCIe 5.0 x16; Crypto Disabled; Secure Boot Enabled; Device: mlx5_0 Configurations: Next Boot New CNP_DSCP_P1 40 36 CNP_802P_PRIO_P1 7 5 Apply new Configuration? (y/n) [n] : y Applying... Done! -I- Please reboot machine to load new configurations. Reboot machine or reset fw root@FS1:~# mlxfwreset -d mlx5_0 -l 3 -y r Requested reset level for device, /dev/mst/mt4129_pciconf2: 3: Driver restart and PCI reset Continue with reset?[y/N] y -I- Sending Reset Command To Fw -Done -I- Stopping Driver -Done -I- Resetting PCI -Done -I- Starting Driver -Done -I- Restarting MST -Done -I- FW was loaded successfully. PFC Configuration To check pfc configrution root@FS1:~# sudo mlnx_qos -i ens2np0 DCBX mode: OS controlled Priority trust state: dscp dscp2prio mapping: prio:0 dscp:07,06,05,04,03,02,01,00, prio:1 dscp:15,14,13,12,11,10,09,08, prio:2 dscp:23,22,21,20,19,18,17,16, prio:3 dscp:31,30,29,28,27,26,25,24, prio:4 dscp:39,38,37,36,35,34,33,32, prio:5 dscp:47,46,45,44,43,42,41,40, prio:6 dscp:55,54,53,52,51,50,49,48, prio:7 dscp:63,62,61,60,59,58,57,56, default priority: Receive buffer size (bytes): 0,156096,0,0,0,0,0,0,max_buffer_size=4151520 Cable len: 7 PFC configuration: priority 0 1 2 3 4 5 6 7 enabled 0 0 0 1 0 0 0 0 buffer 0 0 0 1 0 0 0 0 tc: 0 ratelimit: unlimited, tsa: vendor priority: 1 tc: 1 ratelimit: unlimited, tsa: vendor priority: 0 tc: 2 ratelimit: unlimited, tsa: vendor priority: 2 tc: 3 ratelimit: unlimited, tsa: vendor priority: 3 tc: 4 ratelimit: unlimited, tsa: vendor priority: 4 tc: 5 ratelimit: unlimited, tsa: vendor priority: 5 tc: 6 ratelimit: unlimited, tsa: vendor priority: 6 tc: 7 ratelimit: unlimite To enable PFC for priority root@FS1:~# mlnx_qos -i ens2np0 --pfc 0,0,1,0,0,0,0,0 DCBX mode: OS controlled Priority trust state: dscp dscp2prio mapping: prio:0 dscp:07,06,05,04,03,02,01,00, prio:1 dscp:15,14,13,12,11,10,09,08, prio:2 dscp:23,22,21,20,19,18,17,16, prio:3 dscp:31,30,29,28,27,26,25,24, prio:4 dscp:39,38,37,36,35,34,33,32, prio:5 dscp:47,46,45,44,43,42,41,40, prio:6 dscp:55,54,53,52,51,50,49,48, prio:7 dscp:63,62,61,60,59,58,57,56, default priority: Receive buffer size (bytes): 19872,156096,0,0,0,0,0,0,max_buffer_size=4151520 Cable len: 7 PFC configuration: priority 0 1 2 3 4 5 6 7 enabled 0 0 1 0 0 0 0 0 buffer 0 0 1 0 0 0 0 0 tc: 0 ratelimit: unlimited, tsa: vendor priority: 1 tc: 1 ratelimit: unlimited, tsa: vendor priority: 0 tc: 2 ratelimit: unlimited, tsa: vendor priority: 2 tc: 3 ratelimit: unlimited, tsa: vendor priority: 3 tc: 4 ratelimit: unlimited, tsa: vendor priority: 4 tc: 5 ratelimit: unlimited, tsa: vendor priority: 5 tc: 6 ratelimit: unlimited, tsa: vendor priority: 6 tc: 7 ratelimit: unlimited, tsa: vendor priority: 7 root@FS1:~# mlnx_qos -i ens2np0 --pfc 0,0,1,0,0,0,0,0 DCBX mode: OS controlled Priority trust state: dscp dscp2prio mapping: prio:0 dscp:07,06,05,04,03,02,01,00, prio:1 dscp:15,14,13,12,11,10,09,08, prio:2 dscp:23,22,21,20,19,18,17,16, prio:3 dscp:31,30,29,28,27,26,25,24, prio:4 dscp:39,38,37,36,35,34,33,32, prio:5 dscp:47,46,45,44,43,42,41,40, prio:6 dscp:55,54,53,52,51,50,49,48, prio:7 dscp:63,62,61,60,59,58,57,56, default priority: Receive buffer size (bytes): 19872,156096,0,0,0,0,0,0,max_buffer_size=4151520 Cable len: 7 PFC configuration: priority 0 1 2 3 4 5 6 7 enabled 0 0 1 0 0 0 0 0 buffer 0 0 1 0 0 0 0 0 tc: 0 ratelimit: unlimited, tsa: vendor priority: 1 tc: 1 ratelimit: unlimited, tsa: vendor priority: 0 tc: 2 ratelimit: unlimited, tsa: vendor priority: 2 tc: 3 ratelimit: unlimited, tsa: vendor priority: 3 tc: 4 ratelimit: unlimited, tsa: vendor priority: 4 tc: 5 ratelimit: unlimited, tsa: vendor priority: 5 tc: 6 ratelimit: unlimited, tsa: vendor priority: 6 tc: 7 ratelimit: unlimited, tsa: vendor priority: 7 To set dscp for priority root@FS1:~# mlnx_qos -i ens2np0 --dscp2prio=set,30,2 DCBX mode: OS controlled Priority trust state: dscp dscp2prio mapping: prio:0 dscp:07,06,05,04,03,02,01,00, prio:1 dscp:15,14,13,12,11,10,09,08, prio:2 dscp:30,23,22,21,20,19,18,17, prio:2 dscp:16, prio:3 dscp:31,29,28,27,26,25,24, prio:4 dscp:39,38,37,36,35,34,33,32, prio:5 dscp:47,46,45,44,43,42,41,40, prio:6 dscp:55,54,53,52,51,50,49,48, prio:7 dscp:63,62,61,60,59,58,57,56, default priority: Receive buffer size (bytes): 19872,156096,0,0,0,0,0,0,max_buffer_size=4151520 Cable len: 7 PFC configuration: priority 0 1 2 3 4 5 6 7 enabled 0 0 1 0 0 0 0 0 buffer 0 0 1 0 0 0 0 0 tc: 0 ratelimit: unlimited, tsa: vendor priority: 1 tc: 1 ratelimit: unlimited, tsa: vendor priority: 0 tc: 2 ratelimit: unlimited, tsa: vendor priority: 2 tc: 3 ratelimit: unlimited, tsa: vendor priority: 3 tc: 4 ratelimit: unlimited, tsa: vendor priority: 4 tc: 5 ratelimit: unlimited, tsa: vendor priority: 5 tc: 6 ratelimit: unlimited, tsa: vendor priority: 6 tc: 7 ratelimit: unlimited, tsa: vendor priority: 7 root@FS1:~# ethtool -S ens2np0 | grep pause rx_pause_ctrl_phy: 0 tx_pause_ctrl_phy: 0 rx_prio2_pause: 0 rx_prio2_pause_duration: 0 tx_prio2_pause: 0 tx_prio2_pause_duration: 0 rx_prio2_pause_transition: 0 tx_pause_storm_warning_events: 0 tx_pause_storm_error_events: 0 3.3 Verify RoCE Configuration After driver installation and configuration, perform the following steps to verify the RoCE setup: Check the GUID of the RoCE Interface Use ibv_devices to confirm whether a GUID is available Use ibv_devinfo to confirm the presence of a GUID and obtain additional details about the RoCE interface 代码块 root@FS1:~# ibv_devices device node GUID ------ ---------------- mlx5_1 a088c20300d938f8 mlx5_0 a088c20300d92ec0 root@FS1:~# ibv_devices hca_id: mlx5_1 transport: InfiniBand (0) fw_ver: 28.43.2566 node_guid: a088:c203:00d9:38f8 sys_image_guid: a088:c203:00d9:38f8 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 4096 (5) sm_lid: 0 port_lid: 65535 port_lmc: 0x00 link_layer: Ethernet hca_id: mlx5_0 transport: InfiniBand (0) fw_ver: 28.43.2566 node_guid: a088:c203:00d9:2ec0 sys_image_guid: a088:c203:00d9:2ec0 vendor_id: 0x02c9 vendor_part_id: 4129 hw_ver: 0x0 board_id: MT_0000000838 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet Use the perftest package utilities to verify traffic server#ib_write_bw -d bnxt_re0 -F -x 3--report_gbits client#ib_write_bw -d bnxt_re0 -F -x 3--report_gbits 192.168.2.30 --run_i --qp 16 4 RoCE Performance Testing and Results Performance measurements were taken on a cluster using Broadcom/NVIDIA NICs (200GE/400GE) and FS switches (200GE/400GE). Server Switch NIC Benchmarks Model: Dell R860 CPU: 6448H Memory Type: DDR4 -5600 MT/s Memory: 512 GB (128GB/Socket) Kernel: 6.8.0-57-generic (Ubuntu22.04) Model: N9550-32D Hardware Revision: - Software Version: 4.5.0E/3b574830da Model: Broadcom P1400G Driver Version: 1.10.3-232.0.155.5 Firmware Version: 230.2.36.0/pkg 230.2.37.0 Congestion Control OSU: DCQCN-p Perftest: 6.23 Model: MCX75310AAS-NEAT Driver Version: 24.10-2.1.8 Firmware Version: 28.43.2566 Congestion Control OSU: DCQCN-p image.png The performance results from various benchmark tests are summarized below: Category Test Broadcom NVIDIA RDMA Point-to-Point Bandwidth Tool: ib_write_bw, ib_read_bw Unidirectional, Bidirectional Bandwidth 392.25 GB/sec 684 GB/sec 393 GB/sec 695 GB/sec RDMA Point-to-Point Latency Tool: ib_write_lat Small packet RTT latency test: 64B, 256B, 512B 3.58 t_avg [usec] 3.75 t_avg [usec] 3.75 t_avg [usec] 2.19 t_avg [usec] 2.92 t_avg [usec] 2.93 t_avg [usec] RDMA Multi-Node Scalability Tool: ib_write_bw Simulated multi-node RDMA access to a single target node node1: 130 GB/sec node2: 131 GB/sec node3: 129 GB/sec total: 390 GB/sec node1: 131 GB/sec node2: 128 GB/sec node3: 134 GB/sec total: 393 GB/sec Long-Term Stability Test Tool: ib_write_bw 12+ hours of sustained high-pressure traffic; packet loss, performance fluctuation recorded 392 GB/sec 392 GB/sec

N8610-32D Instruction Manual of Safety and Compliance Information

25-06-2025 - For details, please click the attachment icon below to view or download for a good reading experience or resources.