While we are only a few months away from Intel Xeon Ice Lake processors and the company’s move to provide PCIe Gen4 platforms, PCIe Gen4 is going to be a relatively short-timeframe connectivity option even though AMD, Arm, and IBM POWER9 have supported the standard for some time. The industry is instead looking ahead to PCIe Gen5 which will effectively double speeds but also bring in the CXL era (CXL runs atop PCIe Gen5.) We are now starting to see PCIe Gen5 ecosystem products launch, including the Microchip Switchtec PCIe 5.0 switches.
Microchip Switchtec PCIe 5.0 Switches
The Microchip Switchtec PCIe 5.0 switch series is trying to get out ahead of the PCIe Gen5 roll-out. PCIe switches effectively allow PCIe endpoints to have a communications fabric in some ways similar to the Ethernet switch you may have in your office (or home office given the current environment.)
The reason for moving to PCIe Gen5 is simple, we get twice the speed of PCIe Gen4. On the Intel Xeon platform, for example, we first saw PCIe Gen3 with the Xeon E5-2600 V1 generation in 2012. Nine years later, an eternity in computer architecture, Intel’s generally available Xeons will still support PCIe Gen3. Meanwhile, we have seen the industry transition from 10/40GbE to 25/100/200/400GbE, hard drives to NVMe storage, massive GPUs for compute, and from 8 core CPUs to 64 core and 80 core designs with bigger chips coming. PCIe Gen3 became a major bottleneck and so the industry needs to speed its cadence.
We previously covered the Microchip XpressConnect PCIe Gen5 and CXL 2.0 Retimers launched in 2020. The new Switchtec PCIe Gen5 switches are designed to work with these retimers. One small caveat is that we asked Microchip and these new switches will not support CXL.
The all-Microchip solution has a <10ns latency on its hops which it has been working with Intel to implement lower system latency than the ~60ns the PCIe Gen5 spec states. Intel is working with partners on the PCIe Gen5 ecosystem since it will be very late to the PCIe Gen4 ecosystem. We see Intel investing in Gen4 to achieve feature parity with other vendors who have been providing Gen4 for some time. We see Intel’s investment in Gen5 as a step beyond this. Working with Microchip, the companies can create chains where each step has low latency. What is not shown is the end devices, but the key Microchip shows is creating lower latency chains since each hop can add quite a bit.
As part of this increased industry cadence around PCIe generations, systems designers do not have the luxury of almost a decade with a technology as we saw in PCIe Gen3. Instead, systems will need to be brought online faster and that means storage disk shelves, multi-GPU boxes, and other common PCIe switch topologies will need to go through design and validation cycles much faster. Microchip has ChipLink to help with that process. To an end-user this may not seem exciting, but for those designing boxes with these components, having the ability to get data from the PCIe switch and retimers helps speed validation.
SKUs range from 28 to 100 lane configurations. Other than the smallest 28-lane design, the 36-100 lane designs one may notice are effectively Z * 16 + 4 in size. PCIe x16 lanes are common so the SKUs are being designed around multiples of those, although they can support bifurcation into smaller links. The extra four lanes we are told are still on the same switch crossbar fabric but are added for customers who wanted extra lanes to connect management controllers as an example. One must remember that a PCIe Gen5 x4 lane set is roughly equivalent to the bandwidth from a PCIe Gen3 x16 slot.
In terms of availability, the switch chips are now sampling ahead of PCIe Gen5 roll-outs likely in 2022.
Final Words
It is good to see the industry ramp to PCIe Gen5. For the broader industry, it is important that Microchip can get ahead of the systems design cycles to provide an alternative to the Broadcom PLX switch series. When PLX was purchased, PCIe switch prices rose a substantial amount which hurt NVMe adoption in the early days. Having a strong second source is important in the market.
If reading this you want to dive into why PCIe Gen4 is expected to be quickly displaced see our The 2021 Intel Ice Pickle How 2021 Will be Crunch Time piece.
It’s disappointing to hear that these don’t support CXL. With DDR5 and CXL it really feels like we are on the verge of a step-change increase in server capability. I’m excited about the potential for disaggregation of memory resources, but without CXL, PCIe 5.0 is less to look forward to. I wonder why, given their partnership with Intel, they chose to forgo it?
> “PCIe Gen4 is going to be a relatively short-timeframe connectivity option even though …”.
I think PCIe 5 will be the (recent) short lived generation.
PCIe 6 (Gen6) designware and IP is available. I suspect that consumers will be happy with PCIe 4 and its affordability, while some businesses will adopt PCIe 6 before 2023.
Sources:
https://www.servethehome.com/intel-labs-day-2020-a-photonics-integration-future/
https://blogs.synopsys.com/vip-central/2020/03/16/256gb-s-ready-set-go-pcie-gen6-verification-ip/
https://www.anandtech.com/show/14559/pci-express-bandwidth-to-be-doubled-again-pcie-60-announced-spec-to-land-in-2021