Recently, we had the chance to use an ASUS RS520QA-E13-RS8U system with four Montage CXL memory controllers. While the CXL Type-3 memory expansion devices may act like memory in the system, but it might be useful to see how they operate. We thought it would be worth showing you how to take a quick look at the CXL memory in a system.
Quick Guide to Show CXL Memory in a System
The first hint that you have CXL memory in a system might be the fact that you have a lot of memory in a system. For example, we have a single socket node with 1.2TB of memory using 64GB DIMMs.
$ free -mh
total used free shared buff/cache available
Mem: 1.2Ti 4.0Gi 1.2Ti 85Mi 3.9Gi 1.2Ti
Swap: 2.0Gi 0B 2.0Gi
When we do lscpu, we see something different in the NUMA section. Here you can see we have a single socket server, but with NUMA node(s): 2. You can see that the NUMA node0 CPUs: 0-255 since we have a 128 core AMD EPYC 9755 processor. NUMA node1 does not have CPUs attached.
From here, we have a fairly good idea that we have a system with CXL Type-3 memory. In systems with CXL Type-3 memory you can lspci | grep CXL and find CXL devices easily. You can drill down into each of those devices using their addresses.
If you have numactl package installed, you can see the topology and latencies using numactl -H.
You can see this vistually using lstopo where we can see 768GB installed on the CPU (12x 64GB) and 512GB installed via CXL memory (8x 64GB.)
In Debian/ Ubuntu you can do a quick apt install ndctl and that will install the CXL tool. There we can do a cxl list -Mu (u makes this human readable) to see the CXL devices in a system.
Something that folks may find fun here is that the command shows pmem size as part of the origins of CXL started with Intel and Optane. Still, we might see persistent CXL modules in the mix.
Final Words
This was just a quick guide to show some of the basics. The world of CXL memory expansion is still not quite the plug-and-play experience that we expect in other device classes. Hopefully, over time, things will not be as reliant upon having the right firmware versions in systems, CXL controllers, and matching DIMMs. Since we are seeing more systems with CXL memory expansion devices in this generation, we thought it might be worth showing a few commands that are useful to folks when they see these systems. If you have more, feel free to add them to the comments.
Excited to see where this goes!
“as part of the origins of CXL started with Intel and Optane”
I think Optane would have been a good fit for CXL. Such a pity Intel dragged it behind the barn..