Top500 November 2019 Our New Systems Analysis

At SC19 the Top500 November 2019 edition was released. Twice per year, a new Top500.org list comes out essentially showing the best publicly discussed Linpack clusters. We take these lists and focus on a specific segment: the new systems. Our previous edition published around SC18 last year you can revisit at Top500 June 2019 Our New Systems Analysis. In that list, we saw relatively mundane movement. Architectures were relatively similar and Lenovo had 47 of the 94 new systems. With the November update, we have 102 new systems and Lenovo has “only” 34 of them. More vendors have their designs added and those designs are more heterogeneous than we saw on the June 2019 list.

Top500 New System CPU Architecture Trends

In this section, we simply look at CPU architecture trends by looking at what new systems enter the Top500 and the CPUs that they use.

In June 2019, 94 of the 95 new systems were all Intel. In November 2019, AMD claims four new systems, IBM added three, and Fujitsu claims one spot. While still a virtual lock, Intel’s share has decreased from 98.9% to 92.2% over the last six months while still adding the same number of servers. We are going to discuss that in more detail shortly.

This is extremely interesting. Not only do we see the Fujitsu A64fx on the list, we see it takes as many spots as the Intel Xeon D-1571, Intel’s embedded CPU part based on Broadwell-DE. This is essentially what we saw launched in 2015 (see Intel Xeon D Broadwell-DE SoC.) AMD has added two EPYC 7001 “Naples” systems and two AMD EPYC 7H12 “Rome” systems to the list.

Interestingly enough here, if you remove the E5/ E7 V3/ V4 chips that were not launched in the last 3 years, Intel would have claimed only 75 of 83 new systems with a 90.4% share.

CPU Cores Per Socket

Here is an intriguing chart, looking at the new systems and the number of cores they have per socket.

Here we can see that 20 cores is still the most popular option. This chart in the June 2019 list only scaled to up to 28 cores. We now have 7 systems with more than 32 cores on the list. Of those, 4 are AMD EPYC, two are Intel Xeon Platinum 9242 48 core parts, and one is the Fujitsu A64fx. Here is a list of all the CPUs added to the list:

As one can see, the Intel Xeon E5-2673 V4 part is the most popular part. These E5-2673 V4 systems are Lenovo systems are built for a large web company with 10GbE. Also very interesting here is that Intel’s highly touted Intel Xeon Platinum 9282 56-core part did not make the list while AMD’s top bin SKU, the AMD EPYC 7H12 that was launched months later, claimed the same number of spots as the Cascade Lake-AP line.

We are also going to note that despite the fact that the 2nd Gen Intel Xeon Scalable line launched around 7 months prior to this list, the 1st generation takes around 4x the number of spots. This is despite the fact that Intel added a new VNNI feature, security mitigations, and both cores/ clock speed to many popular SKUs at the same price levels.

Accelerators or Just NVIDIA?

Unlike in the June 2019 list, NVIDIA is not the only accelerator vendor for the new systems. Here is a breakdown:

NVIDIA retains 42 of the 44 accelerator systems, but this time it is not a complete shutout of other accelerator technologies.

AMD GPUs along with the PEZY solution both make the list in this cut. Four of the Tesla V100 systems are listed as SXM2.

This is still a mostly NVIDIA game, but with Exascale systems coming soon, and we know about AMD with Frontier and Intel Xe HPC GPUs for that era, we may see a change over the next few lists as high-end systems get more diverse with accelerators.

Fabric and Networking Trends

One may think that custom interconnects, Infiniband, and Omni-path are the top choices on the Top500 list’s new systems. Instead, we see 60 of the 102 new systems or 59%, using Ethernet. This is down from around 75% in June 2019.

It is somewhat surprising here that OPA remains so popular. Many have known Intel’s current interconnect was being sunset since SC18, and the product line officially has no roadmap beyond 100Gbps.

When we look at a breakdown by type and speed, here is what we get:

As one can see, common web-scale interconnects, namely 10GbE and 25GbE are extremely popular. These are lower performance and lower cost interconnects typically used by hyper-scale and similar companies in general-purpose infrastructure. If we drill into which manufacturers are using 10GbE, 25GbE, and 40GbE to be consistent with our June 2019 analysis, here is what we get:

As one can see, Lenovo maintains 10GbE supremacy, although their dedicated HPC clusters like the Harvard Cannon do not use lower-performance Ethernet. Sugon is now the leader in the new Top500 system 25GbE Linpack clusters. This also shows that Chinese vendors are focusing efforts on those technologies which are consistent enough to be a regional variation.

Final Words

Change is good. One thing is certain from SC19, we are going to see the towering leads that Intel and NVIDIA have in the CPU and accelerator spaces come under pressure from new entrants. Another key message is that Intel and NVIDIA’s dominance is actually being driven by products announced in 2017. If we remove x86 CPUs that were released in 2017 and earlier from the list, (Intel Xeon Scalable “Skylake-SP” and AMD EPYC 7001 “Naples”) we get a very different view:

Intel is still dominant with that view but holds only 73% new Top500 system share based on that. One can argue that Power9 was a 2017 “launch” but realistically it hit a similar product cycle window as an x86 “launch” in 2018 so we included those three systems here. NVIDIA for its part has lost a 100% share position but is doing so with a 2017 V100.

As we move to Exascale, we are seeing vendors embrace more diverse technology. It will be interesting to see how this continues to evolve over the next few years as strength built through previous product cycles wanes.