At OCP Summit 2024, we saw one more CXL monster, the Lenovo ThinkSystem SR860 V3. This system is listed on Lenovo’s website as having “Up to 16TB of TruDDR5 memory in 64x slots.” At the OCP Summit 2024, we saw this beast and how Lenovo is doing it with the help of CXL and Astera Labs Leo.
Lenovo Has a CXL Memory Monster with 128x 128GB DDR5 DIMMs
The idea of having a four-socket server is not new. A challenge, however, is that with a 8 channel Intel Xeon, you are limited by the number of DDR5 DIMM slots. Each 8-channel Xeon supports 2 DIMMs Per Channel (2DPC) for 16 DIMMs per socket. Four sockets gives us 64 DDR5 SODIMMs. Filling those with 128GB DDR5 RDIMMs only gets one to a paltry 8TB of memory. For some, a large consumer SSD-sized memory capacity is simply not enough.
At OCP Summit 2024, we saw Lenovo’s answer to add another 64 DIMMs to the system.
Using stacked boards with Astera Labs Leo CXL controllers, Lenovo is able to add many memory slots to the system.
Better said, up to 64 DDR5 DIMM slots. Each Astera Labs Leo CXL memory controller can handle up to four DDR5 DIMMs.
With this option installed, the top part of the chassis becomes a giant memory forest. The four CPUs sit on the motherboard at the bottom each with 16 DDR5 DIMM slots. Above that, is this CXL memory forest with another 64 DIMM slots.
The 64 DIMMs are connected directly to the Xeon CPUs, and then 64 additional DIMM slots are connected via CXL memory expansion. That gives us 128 DIMM slots. With 128GB RDIMMs, that is 16TB of memory capacity.
Something neat here is that the configuration with four CPUs and 128 DIMMs can also support four double-width GPUs.
That makes these very big systems.
Final Words
For some scale-up workloads where massive memory capacity is needed, solutions like these using CXL Type-3 memory create topologies that would not be possible otherwise. The recent AI build-out has delayed the CXL deployment, but these solutions are coming. What is more, with CXL 2.0 (or really CXL 3.1), the vision is that in the future, one can hook up multiple memory expansion shelves, like the Inventec 96 DIMM CXL Expansion Box to CXL switches, building massive memory pools and then allocating and sharing them among different systems.
CXL is getting fun again. In Q1 2025 with Granite Rapids-SP we expect to see eight CPU systems like the Inventec with even more memory that will make this Lenovo 4-way system look small in comparison.
