3rd Generation Xeon Scalable Big UPI Link Update
The UPI change is perhaps the other big feature with this generation. One gets 6x UPI links. These links still operate at 10.4GT/s which makes sense if one thinks of this as a Cascade Lake refinement. With Ice Lake and PCIe Gen4 companies will have to do a lot of signaling work in and out of sockets and enhance motherboard PCB quality so that is a more logical time to move to 11.2GT/s UPI.
To understand why six UPI links are important, we can go back to the original Skylake multi-socket topology diagram from our 2017 Intel Xeon Scalable Processor Family Platform Level Overview.
As you can see, in a 4-socket 3x UPI server, each CPU uses one of its three UPI links to communicate with each of the other three CPUs in a server. Conversely, in a 2-socket configuration, either two or three UPI links are available between sockets. The Cascade Lake “Refresh” SKUs had only two UPI links, as did the original Xeon Gold 5100/ 5200 series for example. With a 6x UPI configuration, one can have three pairs of UPI links meaning each CPU can use two UPI links to communicate with every other CPU. That is the magic of Cooper Lake 4-socket. Whether you are using a Cascade Refresh CPU or a 4-socket Cooper Lake CPU, you get two socket-to-socket UPI links. Of course, one can use non-refresh SKUs in 3x UPI dual-socket configurations and get more bandwidth, but that is more of a niche case these days.
8-socket configurations are available, with Cooper, but many fewer vendors will support those. For example, Lenovo is not announcing an 8-socket platform even though it has an 8-socket Cascade Lake platform. That is partly due to customer refresh cycles.
3rd Generation Xeon Scalable Memory Update
Perhaps the third big prong of the Cooper Lake change is the memory update. There are both speed changes as well as updated DCPMM support. It will also mark the end of an era on the memory side with a bridge to the future.
The first big change on the memory side is that we get some DDR4-3200 support. That DDR4-2933 support is not on all SKUs, just the Platinum parts. A cynic could say that it is only reserved for the high-end SKUs on the highest end parts so that Intel could claim DIMM speed parity with AMD and a slew of Arm vendors. In 2020, the fact is that DDR4-3200 is already a fairly mature technology, but Intel was planning for DDR4-3200 in Whitley which is being released later than anticipated. From what we understand, just getting DDR4-3200 support at all is something that the Intel engineering teams had to push to be able to get out of the current memory controllers. For standard RDIMMs it is 1DPC (DIMM per channel) support but if using LRDIMMs or 3DS memory we expect this to work in 2DPC mode. Remember, Intel Xeon memory controllers have an additional task, they need to support Optane DCPMM as well so Intel engineers have additional complexity.
On the Optane DCPMM support, we get Intel Optane DC Persistent Memory 200 Modules in 128GB, 256GB, and 512GB capacities again. These are now called the “Data Center Intel Optane Persistent Memory 200 series”, “Intel Optane PMem 200 series.” Given that, it seems like the Optane DCPMM first-generation will be the Optane PMem 100 series. The new modules were codenamed “Barlow Pass.”
A few notes. First, Optane PMem uses a new controller. The PMem 100 “Apache Pass” modules utilized a controller codenamed “Elk Valley.” With the new PMem 200 “Barlow Pass” generation we get the “Barlow Valley” controller. The key here is that it enables more bandwidth per channel and better performance at lower power consumption than the first generation part. Intel is saying 38% performance increase per channel and 25% more memory bandwidth in aggregate than previous generations. A big portion of that gain is the move to DDR4-3200 which we will discuss later. The other portion is the faster silicon.
Intel, says Barlow Pass or Optane PMem 200 supports 3200MT/s speeds. Intel also says it supports Memory Mode. Those are both true statements, however, not for Cooper Lake which is being launched alongside the new modules. Breaking these down:
- Optane PMem 200 is limited to 2666MT/s support, as it is in the Cascade Lake Xeon generation (2nd Generation Intel Xeon Scalable.)
- Optane PMem 200 does not support Memory Mode on Cooper Lake, nor obviously, any hybrid/ mixed mode. On the 3rd Generation Intel Xeon Scalable designs, PMem 200 is limited to App Direct mode. One will notice that the benchmarks and case studies Intel offered with PMem 200 are for applications such as SAP HANA. SAP HANA is App Direct only.
Of course, let us be 100% clear here. Memory Mode in Cascade Lake where the DCPMM or now Optane PMem 100 is used in a non-persistent manner to simply extend a server’s memory footprint is very popular. Intel has logic to push cold memory pages to Optane PMem 100 modules while keeping hot pages in DRAM. For many, including some of STH’s future hosting servers (we just got PMem 100 modules the day before this will go live for this purpose) this is a great feature that greatly reduces memory costs. Rumor has it, a large percentage of buyers use Memory Mode. That is not available in Cooper Lake.
Since Memory Mode is listed in PMem 200 specs, we thus know it will make a comeback in the mainstream Whitley/ Ice Lake Xeon generation.
With the same 2666MT/s speed and no Memory Mode, one may logically wonder if first-generation Optane PMem 100 modules can be used in Cooper Lake. Officially, this is not a supported configuration. Interestingly enough, one also cannot utilize PMem 200 in 8-socket configurations either. If you buy a 512GB PMem 200 part with Cooper, it is actually less flexible than a 512GB PMem 100 part with Cascade Lake with respect to Memory Mode.
While Intel is technically correct marketing that the Optane PMem 200 modules are capable of 3200MT/s and Memory Mode, and that they can be used with Cooper Lake, there is a fine line here. All of that cannot happen together. There are likely a huge number of journalists and analysts, even technical ones, who will miss this. We confirmed the above details with Intel prior to the announcement. Intel confirmed all of the above except Ice Lake Xeon supporting Memory Mode, but since Memory Mode is in the product brief, we know when that is coming back since it is the only logical place for that to re-appear.
Next, let us focus on PCIe support.
Why doesn’t anyone else have info on the PMem 200 in memory mode that you’re talking about on page 3. STH is the only place reporting that.
Fred we confirmed this with Intel before the piece went live. To be fair, it was very hard to figure out based on what Intel released in their materials.
This is the most in-depth coverage of Cooper I’ve seen. There’s crazy detail here. excellent work Patrick and STH
Does anyone know what the “MOQ” refers to in the Optane specs, the values (4 and 50) align with the existing Optane DIMMs but could never find what it referred to.
Thanks
“a stopgap step”
From the processor king that has been pumping $Bs Q after Q?
AMD for pumping out products, but Intel for pumping in $$$…
@binkyto… MGTFY.com. OH what help a google could be. MOQ typically refers to Minimum Order Quantity. however…if you had actually googled “Optane” + “MOQ”, without the quotes of course, you’d have come to this link:
https://ark.intel.com/content/www/us/en/ark/products/190348/intel-optane-persistent-memory-128gb-module.html
wherein it says:
Intel® Optane™ Persistent Memory 128GB Module (1.0) 4 Pack
Ordering Code
NMA1XXD128GPSU4
Recommended Customer Price
$1499.00
Intel® Optane™ Persistent Memory 128GB Module (1.0) 50 Pack
Ordering Code
NMA1XXD128GPSUF
Recommended Customer Price
$1499.00
4 pack and 50 packs. they have different sku’s. you’r welcome.
Isn’t MOQ minimum order quantity BinkyTo?
Nice writeup STH.
That Optane PMem 200 I’ve read Anandtech, Toms, and nextplatform and none of them mentioned it in their articles.
Fire.
Keep cutting through the marketing BS at these companies.
Can’t wait to see how it stacks up with the exiting parts and Epyc 7xx2
The use of place names to distinguish these products creates confusion. Reading articles about Intel CPUs requires one to have a decoder ring handy.
Intel should stop being obscurantist.
Is the TDP 15W or 18W? If you go to their site it still mentions 18W.
Also some of their datasheet still mentions the specs for the 100 series. Like here: https://www.tomshardware.com/news/intel-announces-xeon-scalable-cooper-lake-cpus-optane-persistent-memory-200-series
Will the rated 8.1GB/3.15GB R/W bandwidth improve with the Icelake platform? Why does the datasheet say it supports up to 2666MT/s?
Intel got hurt bad by their 10nm process. The delays that caused allowed AMD to leapfrog. The question now is who will be first to market with PCIe Gen 5. That will be an enormous advantage in the server space.
I love competition. I really do.
I’m going to have to dig into this further tomorrow, but what has really caught my eye this week is working Sapphire Rapids chips in Intel labs. Sapphire Rapids is supposed to have both DDR5 and PCIe 5, and they are targeting a 2021 release. The reporting on AMD’s roadmap that I have seen has Genoa and the SP5 socket (with DDR5 and PCIe 5) coming out in 2022. I wonder how many months Intel powered servers will have those features before AMD’s get them? DDR5 looks like a huge upgrade. I thought AMD’s technical dominance in servers would last a couple more years, I might have been wrong.
@Wayne Borean: From a business point of view, Amd has done nothing to really stress Intel as regards profits/revenue over the last 3 years since Epyc has been on the market. Most of the marketshare that Epyc has gained over the last 3 years is miniscule when you look at the revenu/profit that Amd earns from Epyc.
If Amd cannot compete on a volume basis with Intel when Zen 3 come out, Intel will continue to dominate in the only ONE true area that counts for any business: increasing revenue/profits over your competitors
Amd has amazing performance compared to Intel but at what cost??
Once Ice Lake server comes out in volume then and only then will we see a head to head competition and based on the last 3 years when Amd clearly had the performance lead, it does not look too good for Amd in the server space!!!