At the very top-end of the server market, x86 loses steam. Here players like the IBM POWER9 are able to offer some really cool technology. The latest generation IBM POWER9 CPUs now have new machines available with PCIe 4.0 and up to 4 sockets per node.
IBM POWER9 E950 and E980 Overview
IBM made a handy table of the two systems it is launching in its scale-up architecture. While IBM offers systems like we saw in US DoE Announces Summit with IBM Power and NVIDIA Tesla, there are customers that want more POWER9 to compute resources in a box over GPUs.
The POWER9 chips in these systems are the 8-way SMT capable variants so a 48 core machine is a 384 thread box. The 192 core E980 then becomes a 1536 thread machine.
We find the core counts really intriguing. If you divide the “# of cores” by the four sockets available in each node, you get 8, 10, 11, 12 core options for each CPU. It has been a long time since we have seen a multi-core CPU that had an odd number of cores on the x86 side. Perhaps back to the AMD tri-core days in 2010-2011.
IBM Power E950
The IBM Power E950 is a 4U, 4-socket server. Aside from being an AIX powerhouse, one of its other claims to fame is expansion. It has 10x PCIe Gen4 slots (two x16 per socket plus two x8’s in the system.) The x86 ecosystem is still on PCIe Gen3 as of this writing. We do not expect x86 PCIe Gen4 until 2019 with AMD EPYC “Rome” and Intel Xeon’s “Cooper Lake” variants.
In terms of memory capacity, we are starting to get 4 socket Intel Xeon Platinum systems in the STH lab. They are all 48 DIMM designs. That means that using “M” series CPUs one can get up to 6TB of RAM in a 4-way Intel Xeon Platinum / Gold system. STH was the first to confirm that the next-generation “Cascade Lake” will support up to 3.84TB of memory per socket, bringing it closer to POWER9. IBM Is offering up to 16TB via 128x 128GB DIMMs. 128GB LRDIMMs to very large customers (think hyperscale) the last we heard cost around $2000 each, and more if you are not building out data centers at scale. Expect to spend at least a quarter of a million per system just on RAM if you want to get 16TB of memory today in one of these systems if you work at a large hyperscaler.
IBM Power E980
The larger of the two systems is the IBM Power E980. This is for when a 4-socket system is simply too small. The IBM Power E980 has up to four 5U quad socket nodes that can work together to provide up to 16 sockets in a single system. That also means up to 64TB of memory.
IBM has a proprietary interconnect that has a much higher link density than Intel Xeon Scalable. We covered this in our Intel Xeon Scalable Processor Family Platform Level Overview. In the x86 world, the higher-end Intel Xeon Platinum and Gold CPUs only have 3 UPI links which make sense for smaller topologies but require a custom interconnect to hit the 16-way configurations.
With the IBM Power E980, there are features such as I/O expansion shelves for PCIe Gen4 devices and all of the other features one would expect from a system of this scale.
Final Words
The IBM Power E950 and Power E980 are the systems that go after parts of the market where IBM excels in. IBM Power simply out-scales Intel Xeon in these areas. There is a trend toward scale out rather than scaling up which benefits x86. At the same time, customers have workloads that run on AIX and Power. There are other customers that simply need more RAM capacity in a box or more high-speed I/O connectivity. For these customers, Intel Xeon still cannot touch. With Cascade Lake later this year and Optane Persistent Memory DIMMs, Intel will be much closer in memory capacity (3.84TB per socket versus 4TB for POWER9), but will be at a significant feature defecit for these scale-up systems.
Some editor work needed: “IBM POEWR9” — simply typo and not so logical “The latest generation IBM POWER9 CPUs now have new 4-socket machines available with PCIe 4.0 and up to 4 sockets per node.” e.g. 4-socket machine and up to 4 sockets per node. Probably some reformulation would solve this.
Otherwise thanks for the article!