Traditionally perceived as a chip-to-chip, single-host interconnect technology, PCIe (PCI Express) over fiber is making inroads into switch fabrics, challenging and potentially replacing previous interconnect technologies in embedded systems.
In a new white paper, authors from Samtec and Dolphin ICS explain the use of PCIe for backplane expansion using fiber optics, including its applications and limitations for high-performance embedded systems, SoCs, and microcontrollers. The paper focuses on the hardware and architectural options available to embedded system designers when using PCIe as a high-performance connectivity solution for peripheral expansion. It includes examples of flexible configurations, which are particularly important when several host computers are interconnected.
Technical Concerns
Engineers designing embedded systems typically focus on optimizing capital equipment usage while enhancing overall performance. This includes particular attention to reliability and long-lifetimes for these high-performance solutions, particularly if the systems will be deployed in challenging environments where troubleshooting and part replacement can be difficult.
In many applications, the real-time aspect is a central consideration, and the system must be able to process a high volume of data with extremely low latency. Engineers use specialized hardware solutions such as FPGAs, ASICs, GPUs, and advanced storage solutions to meet these demands effectively. However, when building large systems where the interconnection between I/O or processor must be over several meters, ensuring low latency, high bandwidth, real-time aspect, and signal integrity becomes a major concern.
Flexibility and Expansion
PCIe over fiber unlocks new possibilities for remote PCIe expansion, expanding PCIe peripherals to external chassis using optical fiber. For instance, a host or server can be positioned up to 100 meters away from the I/O, while still harnessing the full benefits of low latency and high bandwidth associated with PCIe. And, using the PCIe protocol makes these systems backwards compatible with existing systems.
For example, the figure shows one way to connect 16 peripherals using PCIe over optical fiber with passive backplanes (such as the Dolphin backplane IBP-G4X16-5 shown). In this example, the host is equipped with an MXH945 card (Dolphin ICS) where each of the four lanes of fiber is connected to the target board (MXH948) in the backplane using Samtec FireFly™ Micro Flyover System™ optical cable assemblies (Figure 2) and standard MPO-based optical fiber cables. The 16x PCIe bus originating from the target board is then split into four 4x PCIe Gen4 links. This type of configuration supports any expansion board compliant with PCIe standards.
Applications
PCIe-over-optics interconnects enable high data throughput, coherency, and low latency in data center, edge infrastructure, AI/ML, and embedded system applications. Because memory duplication is not needed, CPUs can borrow resources from other systems, and GPUs or NICs can be landed between CPUs. As a result, running embedded applications over PCIe fabric can provide the capability of a small high-performance computer (HPC) while reducing hardware costs. The potential is promising; the good news is that the building blocks exist today.
More Information:
Why use FlyOver Systems? Samtec White Paper
PCI Express Family of Products
PCIe 6.0: From IP To Interconnect In High-Performance Computing – The Samtec Blog
Samtec and Rohde & Schwarz Discuss Testing PCIe 5.0/6.0 Cable Implementations – The Samtec Blog
Leave a Reply