Supermicro X9DRH-IF-NV Review – NVRAM Capable Motherboard

The Supermicro X9DRH-IF-NV is a solution to fit two Intel Xeon E5-2600 (V1 or V2) series processors into a standard EATX form factor. The key differentiation point with this motherboard is the ability to support NVRAM. Supermicro specifically sells the X9DRH-IF-NV with Agigatech NVDIMMs. The interesting point here is that the platform  is largely shared with the Supermicro X9DRH-7TF we reviewed last year, subtracting the 10GbE and LSI controllers and adding NVDIMM support. The Supermicro X9DRH platform is extremely flexible supporting dual Intel Xeon E5 V1 or V2 processors and up to 1TB of RAM alongside a plethora of PCIe slots.

Test Configuration

For this test we were limited to 130w processors as the motherboard supports up to 130w TDP Intel Xeon E5 processors. We utilized the fastest 32nm LGA2011 “server” processors available right now just to generate a bit more heat. We loaded up a healthy dose of RAM and more SSDs to the mix. We also tested both a standard configuration and a configuration that uses the 4x NVDIMM modules we had available for testing.

1. CPUs: 2x Intel Xeon E5-2690 CPUs or 1x Intel Xeon E5-2670
2. Motherboard: Supermicro X9DRH-iF-NV
3. Memory: 8x Samsung 16GB 1600MHz DDR3 ECC RDIMMs or 4x Agigatech AGIGARAM 4GB NVDIMMs and 4x PowerGEM ultracapacitors
   
4. SSDs: 2x Samsung 840 Pro 256GB 4x Seagate 600 Pro 240GB 
5. Operating System: Ubuntu 14.04 LTS and Windows Server 2012 R2, VMware ESXi 5.5, Hyper-V Server 2012 R2, CentOS 6.5

The major caveat here is that one does need to take a special look at cooling with this type of configuration as there are many parts to generate heat.

Supermicro X9DRH-IF-NV Review

The Supermicro X9DRH-IF-NV is a larger EATX motherboard with 13″ x 12.1″ dimensions. It is a large motherboard, but should fit in the majority of full depth chassis. One benefit is that in general, all connectors are easy to reach even on the edges of the motherboard so one can do so even in tighter enclosures.

One important note is that the motherboard utilizes the narrow ILM for LGA 2011. We did a piece on square versus narrow ILM which is certainly an important read for those not familiar. This is one of the two main cooling configurations for the LGA 2011 platform, but one does need to focus on getting the right solution.

The two CPU sockets are slightly offset. As a result, cooling is relatively good. Generally setups where the two processor heatsinks are directly-in line have higher second CPU temps.

The headline feature of this motherboard is support for NVDIMMs. For those wondering, NVDIMMs essentially allow a server to withstand power failure and not lose data stored in RAM. The NVDIMMs sense power loss and then utilize supercapacitors to power the DIMMs dumping RAM to non-volatile flash storage. The entire process of dumping to RAM, restarting then putting data back in RAM from NAND can take under 2 minutes as can be seen in this video:

This is very advanced functionality despite how simple that 2 minute process looks. Generally it requires: NVDIMM hardware, BIOS, motherboard, OS and application support.  The X9DRH-iF-NV supports the ADR trigger method. The best explanation we found was in the Supermicro manual:

The Agigatech AGIGARAM modules fit nicely on the motherboard, especially when the PowerGEM supercapacitors are connected via cables.

In terms of expansion slots there is one PCIe 3.0 x16 slot and six PCIe 3.0 x8 slots. That means one could fit, for example, up to 7 SAS controllers to pair with the NVDIMMs of 7x PCIe SSDs. There are a total of 64 PCIe 3.0 lanes available for add-in cards of the 80 total available with two CPUs.

One great feature is that Supermicro left the ends of the PCIe x8 slots open so if one did want to install larger cards, they likely can.

In terms of internal storage ports the X9DRH-iF-NV has several. There are two SATA III 6.0gbps ports and a total of six SATA II 3.0gbps ports onboard. This is enough for many storage requirements. Further, there is an internal USB Type-A header that can be used as a boot device or as a key. A very common use case is booting a hypervisor or OS off of the USB port in order to free an additional onboard SATA port.

On the rear I/O panel we can see a standard Supermicro design. There are legacy serial port and VGA port. Four USB 2.0 ports can be used for adding external KVM carts, DVD ROMs or other applications. There is a dedicated IPMI port atop two USB 2.0 ports. Finally, two Intel i350-t2 ports provide network connectivity.

In terms of network controllers, one can clearly see the large Intel i350-T2 controller next to a very small Realtek network controller (black package to the left of the Intel).

Overall, the platform has all of the features one would want if they were looking for a NVDIMM solution without other features such as 10GbE and LSI SAS RAID.

FLIR Thermal Imaging

We used our new FLIR thermal imager to take a look at the motherboard in our standard thermal test bed. Looking at the thermal images, one can see the heat is predictably behind the CPU socket. The temperature scale is our normal 22C to 60C range and one can see that the hottest regions are around 45C even with all network ports running and the CPU heat soaked for several hourse. 45C is well within tolerances for components.

For this test we did something slightly different. We utilized the NVDIMMs alongside a single Intel Xeon E5-2670 processor. This was to give a sense of what kind of heat a socket would put out. As one can see, due to the 4U active cooler the processor stays cool and the NVDIMMs stay well within operational tolerances as well. All components are adequately cooled even in the test bed that has less than extreme airflow.

Supermicro Remote Monitoring and Management

Supermicro’s IPMI and KVM-over-IP as described a few times on this site, allows for a lot of deployment flexibility. Things such as fan speeds, chassis intrusion sensors, thermal sensors, and etc. can be monitored remotely. Alerts can be setup to notify the admins of issues. Beyond this, the Nuvoton BMC chip also allows for remote power control. You can do remote power up, power down, and reset of the server in the event that it becomes unresponsive. In fact, the test system has never had a keyboard, mouse, CD/DVD ROM, or monitor hooked up to it, even after multiple BIOS tweaks and operating system/ hypervisor installations.

Another important feature is the ability to remotely mount CD images and floppy images to the machine over the dedicated management Ethernet controller. This keeps maintenance traffic off of the primary Intel NICs. At the same time it removes the need for an optical disk to be connected to the Supermicro motherboard. Combined with the onboard USB 2.0 header, mounting OS installation or recovery images remotely is a very simple affair.

In combination with the baseboard level management features just described, KVM-over-IP features of the Supermicro board are not to be underestimated. One can log on to the server using a web browser with Java platform support to administer the server. Or one can go through Supermicro’s IPMIview software and have remote console capabilities, with mouse support included. Oftentimes, users opine that shell access is enough to troubleshoot. However this BMC level access to the remote system allows one to immediately see things such as a hung and incomplete boot process, frozen servers, as well as manage motherboard and add-in card BIOS remotely. Supermicro provides a well featured WebGUI to its IPMI management system for each server.

IPMI 2.0 features, including the KVM-over-IP features are essential for a server-class motherboard unless one has an external KVM-over-IP unit and a network addressable power source. For users with redundant PSUs, as will be the most common use case with this motherboard, remotely toggling on/ off multiple power supply ports across multiple PDUs can be difficult making the motherboard option preferred.

Conclusion

This is certainly a motherboard that is tailored for a specific purpose. If you do want the ADR trigger for NVDIMMs the motherboard is a key component. The overall layout is very similar to the Supermicro X9DRH-7TF we reviewed and that motherboard is one of our highest rated ever. The flexibility of the X9DRH base is very strong. Of course, there is a price premium for adding Agigatech AGIGARAM and having the supporting motherboard in place. That cost will be easily offset in applications such as in memory databases is an example where one can get significantly more performance with the solution.