At OCP Summit 2023, Meta and AMD showed off a demo of CXL Memory, or a Type-3 device. We spotted this in a corner of the show floor and figured we would give it some coverage since we previously covered QCT demoing Astera Labs Leo CXL Memory Expansion on Intel Sapphire Rapids at SC22. It is a bit strange, but while AMD EPYC 9004 “Genoa” processors support CXL, most of the Type-3 memory device demos we have seen to date have used the Sapphire Rapids Intel Xeon which does not support CXL Type-3 devices (but they work.) It was great to see AMD with a hyper-scale demo.
Meta AMD CXL 2.0 Memory Expansion Demo at OCP Summit 2023
Here was the demo board which was pretty easy to miss. It has four DIMM slots and then a chip in the middle.
At the far end of the board you can see a PCIe x16 connector.
We asked the folks staffing the demo which CXL chip is being used. Normally we see Astera Labs or Microchip these days. The representatives declined to tell us who makes the chip under the heat sink and fan.
Since this was an AMD demo, we see a dual SP5 socket system with a fun array of fans over the DIMM slots and liquid cooling for the CPUs.
The CXL memory expansion demo board is plugged into a PCIe Gen5 slot in the expansion area. One will also notice that the. DDR5 slots on the motherboard are populated with only two DIMMs to boot the system.
Final Words
CXL memory expansion devices serve two purposes. First, they can add memory capacity and bandwidth to systems, expanding the capabilities of those systems. Second, and important for a company like Meta, CXL Type-3 devices are being looked at as a way to re-use DRAM. If you think about how many DIMMs that hyper-scalers like Meta use, then the opportunity to make memory controllers that can interface between DDR4 or DDR5 today on one end and CXL on the other end offer an enormous opportunity for cost savings. For some perspective, many devices within a system, are still using DDR4 even though modern server CPUs have transitioned to DDR5. Memory is often a top, or near the top, data center hardware expense item, so finding a way to tier memory to slower capacity tiers over CXL has a lot of promise.
Now, what we really wanted to see was a MI300 with CXL memory expansion, but that will have to wait until December it seems.
If it’s not fabric-connected, what’s the point? And if it is, you can build a whole mesh of conventional cpu+dimm systems with a fast, coherent shared-memory fabric. So great, we’ve reinvented the Altix.
An MI300 with CXL memory reminds me of the Radeon Pro SSG (which used a 2TB SSD for expansion memory), but better than the Altrix UV.