Intel Xeon 6900P Memory Subsystem is FAST
The Intel Xeon 6900P has an absolutely screaming memory subsystem. First off, let us talk about DDR5-6400. This is what a DDR5-6400 64GB RDIMM looks like on the front.
Here is the back side. Both sides will look familiar to folks that have been using DDR5 in servers for a few years now.
If you want to learn more about DDR5, see our Why DDR5 is Absolutely Necessary in Modern Servers and the associated video:
New for this generation is the MRDIMM. Here is what one looks like on the front:
Here is the back. On both sides, you will notice the buffer chips at the bottom that make the MCR DIMM/ MRDIMM work.
To understand what is going on here, the DIMMs are able to read from two ranks at the same time instead of one. As a result, the MRDIMMs are able to achieve 8800MT/s transfer rates up from 6400MT/s.
The result is faster memory speeds running at 8800MT/s.
Of course, we should take a second and just note that there is a big caveat here, and it is somewhat strange. JEDEC will have a MRDIMM spec, and the MRDIMMs for Xeon 6 with P cores is actually not that spec, instead it is the MCR DIMM spec. When we use the MRDIMM with Xeon 6, we really mean MCR DIMM. The concepts are largely similar, but it is unlikely when AMD supports MRDIMMs that you will take a Xeon 6 MRDIMM (really a MCR DIMM) and be able to use it in an AMD system.
We covered this for our Substack subscribers, but the current MRDIMM/ MCR DIMM for Xeon 6 is the same thing. We also noted that during some of the standards meetings, MRDIMM was pronounced “Mr. DIMM.” If you do not believe this, here is Micron’s FAQ for its current generation of MRDIMMs for Intel platforms.
On the one hand, we wish Intel kept MCR DIMM. It is a bit confusing to call the MCR DIMM a MRDIMM, when the MRDIMMs are going to be a different standard in the future. With that said, Intel offers MRDIMM/ MCR DIMM, and that is going to take its parts to a new level. Notable here is that AMD is absent. So Micron’s MRDIMMs, albeit MCR DIMMs, are for Intel platforms, but not AMD. AMD will say this is a one generation iteration of a multiplexed DIMM and that it is holding the ground of only adopting once MRDIMMs are mainstream. Intel will counter with the fact that if you need memory bandwidth, it has a high-speed solution in this generation so you do not have to wait for a future technology and generation of chips.
Just to give you some sense of how massive the platform bandwidth is, here is a STREAM run using DDR5.
That is good, but not necessarily superior. Here is the 128-core version.
Those are some big numbers. Scaling to two sockets was around twice that amount as we would expect.
Just to give you some sense, NVIDIA markets its Grace CPU as having huge memory bandwidth. This is the Grace side of a GH200 system with 480GB of memory:
Doubling that STREAM Triad number would give us around 660GB/s for a full Grace Superchip with 144 cores and 960GB of memory. Intel is now closing in on a 960GB Grace Superchip. The benefit of Intel’s solution is that you can customize your DIMM configuration. There is also room to go up from where we are.
P-core, E-Core, Granite Rapids, Sierra Forest, let us get to that next.
Wow can’t even hide Patrick’s love affair with Intel anymore can we ? Intel has not even properly launched this but yet it’s 128c Intel vs 96c Genoa, but AMD will have same 128c in 2 weeks time……just be honest finally and call it servingintel.com ;-)
Yawn… Still low on PCIe lanes for a server footprint when GPUs and NVME storage is coming fast and furious. Intel needs to be sold so someone can finally innovate.
Whether love or not, the numbers are looking good. For many an important question will be yield rates and pricing.
I wonder why Epyc is missing from the povray speed comparison.
One thing I’d like to see is a 4-core VM running Geekbench 6 while everything else is idle. After that Geekbench for an 8-core VM, 16-core, 32-core and so forth under similar circumstances. This sort of scaling analysis would help determine how well balanced the MCRDIMM memory subsystem is to the high-core-count processors–just the kind of investigative journalism needed right now.
As an asside, I had to work over eight captchas for this post.
The keyword would be availability. I checked just now, and these newer parts don’t have 1k Tray Pricing published yet. So not sure when would they be available. It felt painful to restrict the On-Premise Server procurement specification at 64 cores to get competitive bidding across vendors. Hope for the best.