Gigabyte R270-T61 2U Server Review – 96 core Cavium ThunderX

Upon receiving the Gigabyte R270-T61 you may think it is an ordinary 2U server. That is exactly the point. With 24x 2.5″ hot swap bays adorning the front of the chassis and redundant power supplies in the rear, the server is meant to integrate into existing racks with little fanfare. What powers the server are two 48-core Cavium ThunderX chips. There are no HBAs in the system, no add-in NICs, just a genuine ARMv8 64-bit performance machine. If you are looking for a 64-bit ARM server today, this is the one you want.

Test Configuration

Cavium set us up with a test server that is very solid.

• Server: Gigabyte R270-T61
• CPU: 2x Cavium ThunderX 48 core CPUs onboard (96 cores total)
• RAM: 256GB (8x 32GB) DDR4 RDIMM
• SSD: 2x Samsung PM863 960GB
• OS: Ubuntu 16.04 LTS

We did change the SSD from Cavium’s original configuration to Samsung PM863’s that we use in the lab. Notably, even with 24x 2.5″ bays in the front of the chassis, all available there is no need for add-on HBAs. Likewise, while there is a ton of network connectivity, there are no add-on network controllers. This leaves the PCIe slots free for additional expansion.

Gigabyte R270-T61 Overview

The front of the Gigabyte R270-T61 is a standard 24-bay in a 19″ 2U chassis. These form factors have become extremely popular since 2.5″ SSDs are easy to populate. Just by looking at the server, there is very little indication that anything is different than an industry standard server.

Inside the server, one can see Gigabyte’s motherboard design experience. Even the color scheme is what we would expect from Gigabyte branded motherboards.

There are 8 DDR4 DIMM slots per CPU for 16 DDR4 DIMM slots total. This allows for up to 2TB of RAM to be added to the system. Moving from single DIMM per channel (four per CPU) to two DIMMs per channel (eight per CPU) will drop the DDR4 RAM speeds from 2133MHz to 1866MHz.

Cooling the full 48-core Cavium ThunderX is a bit challenging in a 1U form factor (as with any similar TDP chip) but with the 2U form factor, cooling is much easier. The fan array is relatively simple with four 80mm fans that allow cabling to occur on either side of the fan partition. These fans have ducts that channel air over the 2U passive heatpipe coolers.

With dual Cavium ThunderX 48-core ARM SoCs one sees the clear platform advantage: integration. While this server has a combined 160Gbps of network ports, there are no PCIe add-in controllers found on the system. Likewise, controlling 24x SATA SSDs would normally require the addition of an add-on HBA such as the LSI SAS 9305-16i in addition to onboard motherboard ports. Instead we see six of the eight SFF-8087 ports being used, each cabling four drive bays. Between the networking and storage, these are the key value points of the Gigabyte-Cavium platform.

The PCIe 3.0 slots are actually PCIe 3.0 x8 in a PCIe x16 physical slot, two PCIe 3.0 x8 slots and one non-functional PCIe 3.0 x8 slot. We did get an e-mail from a reader who purchased one of these machines after our Cavium ThunderX Benchmarks Part II: Why enterprise ARM developers need these machines piece. The user was trying to figure out why four PCIe cards would not work in the system, however this is clearly listed on the Gigabyte spec page.

Moving to the rear of the server, one can see significant networking capabilities. The throughput of the machine can be absolutely awesome using the integrated networking. There are four SFP+ 10GbE ports along with three QSFP+ 40GbE ports. That is a total of 160Gbps in aggregate. The sole 1GbE port is the BMC management port.

The system has redundant 750w power supplies. This is a major upgrade to our 1U test system which featured a single PSU design.

Gigabyte R270-T61 Remote Management

A great feature with the Gigabyte R270-T61 remote management is the remote management interface. We did notice a maturation in the remote management interface from the single processor system we first tested months ago to the updated dual processor system. It is nice to see that Gigabyte is adding additional functionality with newer BMC firmware releases. Here is a quick view of the R270-T61’s IPMI management interface. We added the menu bar from the original Gigabyte R120-T30 1U server‘s interface as an easy visual indicator of areas added:

These features extend to standard functionality such as providing iKVM access with remote media mounting. During development cycles, having the ability to boot to an ISO image is handy.

For those looking at these systems for network appliances, the out of band management ports can also be used for Serial-over-LAN functionality:

Overall, these systems are much more advanced than the average ARM development motherboard. One can easily deploy them to a data center and they will function similarly to mainstream enterprise servers.

Gigabyte R270-T61 Networking Performance

With 160Gbps of networking promised on the system, we did want to update test figures from our Cavium ThunderX Benchmarks Part II piece with the dual processor model.

As one can see, there is a lot of potential networking bandwidth in this platform. We have recommended several companies that build networking appliances to get these servers for development as the potential bandwidth is excellent.

Power Consumption

We did take an opportunity to measure power consumption of the system in our lab using our calibrated APC metered-by-outlet PDUs. We test data center power consumption on our 208V 30A power circuits.

• BMC Only: 6.1
• Ubuntu 16.04 Idle: 319W
• Maximum Recorded: 374W

Cavium ThunderX power consumption at OS idle is higher than one may expect from an Intel system. Cavium is utilizing OS power management while Intel relies heavily on hardware power management. Our sense is that with future software updates, we may see the idle power consumption decline on the ThunderX platform.

Final Words

One can certainly see the Cavium ThunderX ecosystem maturing. The dual processor server had a number of updates to the BMC firmware that bring additional features to the platform. You can see our 96 core micro-benchmarks for some perspective on compute performance. Overall, we believe the Gigabyte R270-T61 with dual Cavium ThunderX 48 core processors (96 cores total) to be the best arm64 development platform available to date for anyone looking to deploy into data centers. There are still a few areas we would like to see improved. One area is the rails. Most servers in this price range have tool-less rail systems. While they do add cost to the overall BOM, they greatly reduce installation time. This is doubly so if you are installing these machines into a mid-aisle rack. We also would like to see a redundant mid-plane fan array added. Fans are generally reliable but we prefer higher levels of redundancy with fans either through ducting or through in-line redundant fans. Finally, the pricing and availability of these servers has improved to the point that they can be ordered fairly easily. We have received feedback that they were hard to find a few months ago. If you are having issues finding them, feel free to send me a note at patrick at this domain and I will be happy to send you to someone who can help.