AMD EPYC 7281 Dual Socket Linux Benchmarks and Review

When we published the first benchmarks of dual AMD EPYC 7251 CPUs one of the not-so-subtle undertones was that we had another configuration that we preferred. While the AMD EPYC 7251 is a great chip, AMD’s product strategy has an offering, we believe, is a much better value: dual AMD EPYC 7281 processors. The reason for this is simple, AMD EPYC 7281 CPUs are $650 CPUs with 16 cores, 32 threads, essentially twice that of the AMD EPYC 7251’s for a relatively small price premium.

Key stats for the AMD EPYC 7281: 16 cores / 32 threads, 2.1GHz base and 2.7GHz turbo with 32MB L3 cache. The CPU features a 155/170W TDP. Here is the AMD product page with the feature set. Here is the lscpu output for the processor:

What gets interesting with the dual AMD EPYC 7281 is the fact that there are eight NUMA nodes across the two sockets. AMD EPYC architecture has two DDR4 channels per die and four die per package/ socket. That means that each die will have four of a possible 8 cores enabled to yield 16 cores total. Each die is its own NUMA node. So four NUMA nodes and 16 cores per socket and eight NUMA nodes and 32 cores in a dual socket machine.

If you want to see latency for dual socket EPYC, we did dual AMD EPYC 7601 latency testing a few months ago both using DDR4-2400 and DDR4-2666 memory speeds. Memory speeds directly impact the Infinity Fabric.

One of the biggest questions we get is regarding the performance of dual socket AMD EPYC on lower-end SKUs because of the added Infinity Fabric overhead. The solid performance of dual socket Infinity Fabric is essential for AMD offering a competitive EPYC solution versus Intel.

Test Configuration

For our tests we have been using a Supermicro Ultra platform configured as follows:

• System: Supermicro 2U Ultra EPYC Server (AS-2023US)
• CPUs: 2x AMD EPYC 7281 16-core/ 32-thread CPUs
• RAM: 256GB (16x16GB DDR4-2666)
• OS SSD: Intel DC S3710 400GB
• NIC: Mellanox ConnectX-3 Pro 40GbE

We wanted to highlight one important point of this system. It was the first AMD EPYC system in our labs and it is one that our readers have already started to deploy. We have met (in person) readers who started deploying these servers in August 2017. There is a lot of discussion regarding when one can actually buy EPYC systems. The Supermicro AS-2023US has been on sale to customers for over a month already.

While the system may run with both DDR4-2400 and DDR4-2666 RAM, we stand by our recommendation that anyone purchasing an AMD EPYC 7000 series system, especially a dual socket one, use only DDR4-2666.

This configuration has seen at least eight teardowns to install different CPUs and has proven to be extremely serviceable. The fact that it follows the same form factor and has similar expansion options to the Intel-based versions of the Supermicro 2U Ultra platforms makes it easily interchangeable within a rack. Those who are accustomed to the Intel versions simply need to add a T20 driver for the AMD SP3 socket and they will be right at home working on the AS-2023US.

As of late September 2017, if we are recommending EPYC dual-socket servers, this is what we would pick.

Awkwardly Picking the Comparison Set

Picking a comparison set for the dual AMD EPYC 7281 configuration was an awkward enterprise. After seeing the single AMD EPYC 7351P and dual AMD EPYC 7251 configurations, we know that Intel Xeon Silver is no match for the EPYC. At the same time, these 7281’s are two $650 CPUs. Even with street pricing now around $720, that is still in the middle of the Xeon Silver range, specifically near the Intel Xeon Silver 4114. So for a dual-socket Intel comparable, Xeon Silver is the competition in this price bracket and specifically the dual Xeon Silver 4114 configuration.

The Xeon Gold 5100 series is one that we have heard an astoundingly little interest in at STH and with our DemoEval service. Intel essentially has three performance variants in the Xeon Gold 5100 series that cost about as much as two AMD EPYC 7281 chips, the Xeon Gold 5115, 5118 and 5122. Here is the Intel ARK comparison.

• The Xeon Gold 5122 is a 4 core / 8 thread 3.6-3.7GHz chip priced at around $1200. Even with the higher clock speeds, given what we saw with the Xeon Silver 4112 (also 4 core / 8 thread and DDR4-2400) we did not see a point in that exercise against 32 AMD cores with 64 threads. If you want lower core count Xeons, the Gold 6134 is our preferred option and we have bought some for the STH lab.
• The Xeon Gold 5115 (10 cores/ 20 threads 2.4-3.2GHz) and Xeon Gold 5118 (12 cores/ 24 threads 2.3-3.2GHz) are also in the $1220-$1300 range, or about the same price as two EPYC 7281 chips. Essentially AMD has 32 cores where Intel is offering 10-12. That seemed like too awkward of a comparison.

The takeaway here is that AMD is using pricing competitively to push potential buyers up the stack while Intel is relying upon TCO economics of consolidating more performance on a socket.

As a result, we are going to bring in some of the lower-end Intel Xeon Gold 6100 series parts when they can be competitive and otherwise stick to the Xeon Silver line since those are the most price competitive with the AMD EPYC 7281.

On the AMD EPYC comparison side, things get interesting for a different reason. If one wants more memory capacity, in this price bracket, the decision is either to move down the stack to the 7251 or to move up the stack to the 7301. We have not procured a second EPYC 7301 yet but we do have data comparing the 7251, 7281 and 7301 CPUs in single socket configurations that can be found in our AMD EPYC 7351P review.

AMD has another option here. One can alternatively eschew having twice the memory slots. Here one can use a less expensive motherboard retaining 128 I/O lanes for PCIe in a single socket configuration. To further entice buyers, AMD also has its “P” line of CPUs. There one can move up to 24 cores with an AMD EPYC 7401P for only $1075, less than the price of two EPYC 7281’s. For $2100 one can get 32 cores and 64MB L3 cache, essentially equaling the dual EPYC 7281 specs, using the EPYC 7551P. We are going to bring in the EPYC 7401(P) results in many of our charts to show what that option would look like since that is most competitive from a price perspective.

In the last three months, the STH lab has churned out tens of thousands of runs on about 150 different CPU configurations all under the same data center test conditions. Finding a comparison set is no easy task and one that we give a tremendous amount of thought and effort into what we present.

AMD EPYC 7281 Dual Socket Benchmarks

For this exercise, we are using our legacy Linux-Bench scripts which help us see cross-platform “least common denominator” results we have been using for years as well as several results from our updated Linux-Bench2 scripts. At this point, our benchmarking sessions take days to run and we are generating well over a thousand data points. We are also running workloads for software companies that want to see how their software works on the latest hardware. As a result, this is a small sample of the data we are collecting and can share publicly.

Our position is always that we are happy to provide some free data but we also have services to let companies run their own workloads in our lab, such as with our DemoEval service. What we do provide is an extremely controlled environment where we know every step is exactly the same and each run is done in a real-world data center, not a test bench. If you want to read more about the lengths we go to to ensure real-world environments, see How We Test 2U 4-Node System Power Consumption and the “sandwich” approach we pioneered.

Python Linux 4.4.2 Kernel Compile Benchmark

This is one of the most requested benchmarks for STH over the past few years. The task was simple, we have a standard configuration file, the Linux 4.4.2 kernel from kernel.org, and make the standard auto-generated configuration utilizing every thread in the system. We are expressing results in terms of compiles per hour to make the results easier to read.

Here we see relative parity between the dual AMD EPYC 7281 and Intel Xeon Silver 4114. They compete in a similar price segment, so this is a good result. We also see an example of what we mean by Intel can offer (significantly) higher performance per core for core-based licensing schemes as the dual Intel Xeon Gold 6134 configuration is an expensive and high-clock speed part optimized for performance per core. In the price bracket, the Xeon Silver does not have the same per-core performance. This is also the best Intel Xeon Silver 4114 showing against EPYC.

c-ray 1.1 Performance

We have been using c-ray for our performance testing for years now. It is a ray tracing benchmark that is extremely popular to show differences in processors under multi-threaded workloads.

With c-ray, AMD Zen based architectures always perform well. Here we would use these results mostly to see performance compared to others in the same generation. Of the six different AMD EPYC configurations we are showing here, the EPYC 7281 has the most cores so it predictably takes the top spot.

7-zip Compression Performance

7-zip is a widely used compression/ decompression program that works cross-platform. We started using the program during our early days with Windows testing. It is now part of Linux-Bench.

In terms of compression speed, the dual AMD EPYC 7281 configuration is extremely competitive. The Intel Xeon Silver line does not have the clocks and cores to effectively compete. A more interesting comparison is the EPYC 7251 to dual EPYC 7281 where one can see the marginal cost to get significantly more performance is easily justified based on the performance we are seeing.

On many of our expanded benchmarks, this was a common result comparing the dual EPYC 7251 to the dual EPYC 7281 configuration. That is a primary reason we recommend the EPYC 7281.

NAMD Performance

NAMD is a molecular modeling benchmark developed by the Theoretical and Computational Biophysics Group in the Beckman Institute for Advanced Science and Technology at the University of Illinois at Urbana-Champaign. More information on the benchmark can be found here. We are going to augment this with GROMACS in the next-generation Linux-Bench in the near future. With GROMACS we have been working hard to support Intel’s Skylake AVX-512 and AVX2 supporting AMD Zen architecture. Here are the comparison results for the legacy data set:

GROMACS is a similar test and is frankly more interesting given we take advantage of the newest AVX enhancements. Here we use raw core compute speed and the EPYC 7281 is very fast. On the other hand, this is precisely the type of problem that is an early adopter of AVX2/ AVX-512 so GROMACS is more interesting.

Sysbench CPU test

Sysbench is another one of those widely used Linux benchmarks. We specifically are using the CPU test, not the OLTP test that we use for some storage testing.

These tests tend to favor the AMD Zen architecture, but the Intel Xeon Silver results are informative to compare against the Xeon Gold 6100 series we have in the charts. You can, for example, see a dual Xeon 6134 (8 core each) configuration out-pacing the dual Xeon Silver 4114 (10 core each) by a wide margin.

OpenSSL Performance

OpenSSL is widely used to secure communications between servers. This is an important protocol in many server stacks. We first look at our sign tests:

Here are the verify results:

Moving to higher-core counts is certainly helping these results to a great degree. Due to the high core counts, AMD is extremely competitive with the Intel Xeon E5 V3 and V4 generations. While the Intel Xeon Gold 6100 can get to the level of performance needed to compare to the AMD EPYC, the dual Intel Xeon Silver 4114 is simply not able to compete.

UnixBench Dhrystone 2 and Whetstone Benchmarks

Some of the longest-running tests at STH are the venerable UnixBench 5.1.3 Dhrystone 2 and Whetstone results. They are certainly aging, however, we constantly get requests for them, and many angry notes when we leave them out. UnixBench is widely used so we are including it in this data set. Here are the Dhrystone 2 results:

Here is the whetstone comparison:

One will quickly notice that the Intel Xeon Silver line occupies the lower rungs of these charts. That is part of Intel’s product segmentation strategy and a product of trying to compete on factors other than performance.

GROMACS STH Small AVX2/ AVX-512 Enabled

We have a small GROMACS molecule simulation we previewed in the first AMD EPYC 7601 Linux benchmarks piece. In Linux-Bench2 we are using a “small” test for single and dual socket capable machines. Our medium test is more appropriate for higher-end dual and quad socket machines. Our GROMACS test will use the AVX-512 and AVX2 extensions if available.

One item we brought into this chart was the Intel Xeon Gold 6100 line. The reason for this is simple. We wanted to show how much faster the dual port FMA AVX-512 is here in order to show how Intel’s product segmentation in the sub-$1050 per CPU market is essentially ceding a key performance advantage in the Xeon Silver space. With the higher clock speeds and full AVX-512 implementation, the Xeon Gold 6100 series is an absolute monster.

The Gold 5100, Silver and Bronze series cannot handle dual AMD EPYC 7281 CPUs with their curtailed AVX-512 implementation and lack of clock speeds.

Chess Benchmarking

Chess is an interesting use case since it has almost unlimited complexity. Over the years, we have received a number of requests to bring back chess benchmarking. We have been profiling systems and are ready to start sharing results:

Here we added all but the top two single-socket EPYC options just to show where dual EPYC 7281 falls in the stack. The single and dual Intel Xeon Gold CPU configurations are significantly costlier but we wanted to have some relative sense of where they fall since the Xeon Silver line is not overly competitive.

Market Positioning

As we pointed out in our dual AMD EPYC 7251 review, AMD has a number of competitive vectors. The obvious competition is the Intel Xeon single socket line. As a P series part, AMD is also positioning the CPU against lower-end dual Intel CPU configurations. We also see some competition from the dual AMD EPYC 7251 and the AMD EPYC CPUs around the same price range.

Dual AMD EPYC 7281 v. Dual Intel Xeon Silver

This is an easy one if you are looking for raw performance, just get the AMD EPYC. The dual EPYC 7281 configuration also has more expansion for PCIe, more DDR4 RAM slots and capacity. Here, Intel simply needs to take the Intel Xeon Silver and Xeon Gold 5100 series and spin some new SKUs with competitive compute performance. Otherwise, the sub $1000 CPU market boils down to Intel has low power but not performance oriented parts in the segment.

We are going to discuss this chart more thoroughly in an upcoming piece, but this is the crux of what AMD is doing versus Intel in the sub $1050 CPU market:

While Intel is charging progressively more for the value of consolidating more performance on a single socket, AMD is actually providing a relative discount for moving up the stack from the EPYC 7251.

Dual AMD EPYC 7281 v. AMD EPYC

When comparing the dual AMD EPYC 7281 configuration against other AMD EPYC offerings it is a bit more interesting. The AMD EPYC 7351P is a great single socket CPU, and we will have more numbers on the EPYC 7401(P) parts beyond what has been shown above soon. Still, 32 cores / 64 threads at this price point is an amazing value. Moving up to the EPYC 7301 offers twice as much L3 cache and slightly higher clock speeds which may be worthwhile for some applications. Compared to the dual AMD EPYC 7251, our advice is to just get the dual EPYC 7281 as the cost per core is astoundingly low.

Final Words

While the dual EPYC 7601 configuration is extremely fast, the dual AMD EPYC 7281’s pricing is what makes it so alluring. In the Intel Xeon E5-2600 V4 (Broadwell-EP) or Xeon Gold 6130 generations, 16 cores on a single socket is a $2000+ privilege. The fact that AMD is offering that on a $650 chip is astounding. One can essentially consolidate two Intel Xeon E5-2670 (V1) servers onto a dual AMD EPYC platform which has enormous rack-level cost implications.

On the flip side, the dual AMD EPYC 7281 platform is using significantly more power than dual Xeon 4114, or about 50% more power in our sustained 70% workloads (310W.) With Intel positioning the Xeon Silver line for low-power servers, it has left the entire segment vulnerable for a performance option to emerge. That is what has happened with the AMD EPYC 7281. AMD simply has more cores, more PCIe lanes, more memory channels (running at higher speeds) than Intel has and Xeon Silver is ill-equipped to combat AMD’s competitive entrance in this segment.

If you are looking for an inexpensive virtualization or storage server with 512GB of RAM and 32 cores, the AMD EPYC 7281 is perhaps one of the best values we have seen in this market for some time.

To read more about AMD EPYC architecture we have AMD EPYC 7000 Series Architecture Overview for Non-CE or EE Majors and this handy video explaining AMD versus Intel in this segment: