Cavium ThunderX2 and OCP Platform Details

Yesterday we covered the Qualcomm Centriq 2400 launch and as part of that launch series, we noted that there is a bigger, faster ARM server CPU already available, the Cavium ThunderX2. You can read a bit more about that in our Analyzing Key Qualcomm Centriq 2400 Market Headwinds piece. Quietly, Cavium also released v1.0 specs for an OpenCompute motherboard for dual Cavium ThunderX2 CPUs and it shows off the monster processing power of the ThunderX2 platform. Cavium has been shipping the ThunderX2 for some time from what we understand and we have seen it running. The Cavium ThunderX2 has been very quiet and many of the specs have not been public until now.

The Cavium ThunderX2 OCP Motherboard Details

There is an absolute ton to cover in this motherboard and that tells us a great deal about the ThunderX2 platform. It is probably the largest public disclosure to date on ThunderX2.

You will notice that this differs significantly from what we saw at the OCP summit for Microsoft Project Olympus earlier in 2017.

Cavium ThunderX2 Memory

First off, we see a 24x DIMM slot configuration. This is actually 8x DIMM channels per ThunderX2 chip and then half of those channels have 2DPC operation for 12 DIMMs per CPU. AMD EPYC also has 8x DIMMs per socket. The Intel Xeon Scalable CPUs and Qualcomm Centriq 2400 have 6 channel memory controllers per socket. That means that the Cavium ThunderX2 has 33% more memory channels per socket than Intel and Qualcomm and has the same as the AMD EPYC 7000 series. From the motherboard specs we see support for large DIMMs and 1TB listed. Each Qualcomm Centriq 2400 can only handle 768GB of RAM in comparison and going past 768GB of RAM on the Intel Xeon Scalable family carries a $3000 premium.

Supported DDR4 speeds are DDR4-2400 in 2 DIMM per channel and DDR4-2666 in 1 DIMM per channel mode. That is equivalent to current generation platforms from other vendors.

The Microsoft Project Olympus motherboard (pictured and linked above) has 16 DIMMs per CPU for 32 total. Using 64GB DIMMs that would give a 2TB capacity for ThunderX2 although only 1.5TB on the OCP reference platform released.

Cavium ThunderX2 CPU Configuration

The OCP motherboard specification suggests a figure of up to 195W TDP per CPU. Current, publicly available, generation Intel Xeon Scalable CPUs top out at around 205W and AMD EPYC at 180W. The Centriq 2400 series is a paltry 125W TDP in comparison.

We also find that the ThunderX2 can handle up to two sockets in this motherboard. That is similar to the AMD EPYC and some Intel Xeon Scalable CPUs. Qualcomm Centriq 2400 is limited to a single socket while some Intel Xeon Scalable configurations can scale to 4 socket, 8 socket and beyond.

One other generational change is that the Cavium ThunderX2 is now socketed instead of soldered to a motherboard.

Cavium ThunderX2 PCIe and SATA Configuration

The item that immediately jumped off the page was the PCIe and SATA configuration of the OCP motherboard. There are a total of 108 PCIe 3.0 slots emanating from the dual socket configuration that manifest themselves in device connectors. That is very high for this generation. AMD EPYC has a total of 128 PCIe 3.0 lanes possible in a dual socket configuration, however, SATA and BMC devices will take some of these. The Intel Xeon Scalable platform has up to 96 PCIe 3.0 lanes plus additional I/O in the form of PCH for low-speed networking and SATA and an option for another 32x PCIe lanes for integrated Omni-Path fabric. Qualcomm Centriq 2400 has 32 PCIe lanes and 8x SATA III ports.

When looking at the SATA side, we find the OCP platform has 4x SATA III 6.0Gbps ports two emanating from each CPU and there is an additional x1 PCIe lane for the BMC.

Cavium ThunderX2 is certainly competitive on the PCIe expansion side, which is something that was missing in ThunderX (1).

Final Words

Overall, the Cavium ThunderX2 is extremely competitive in terms of features and it completely dwarfs the Qualcomm Centriq 2400. In fact, from a platform capabilities perspective, we see the ThunderX2 OCP motherboard is much more in line with the dual socket Intel Xeon Scalable and AMD EPYC 7000 series offerings. We will share more on the ThunderX2 architecture as soon as we can. Astute readers will note that this looks a lot less like the original ThunderX2 product with high-speed networking onboard and more like the Broadcom Project Vulcan platform that Broadcom shuttered.