Dual AMD Socket G34 motherboards offer a lot of expandability and high core counts at competitive prices and the Supermicro H8DG6-F is no exception. With AMD’s current Opteron 6100 series Magny-cours socket G34 chips one can pack from eight to twenty-four cores on a dual socket motherboard at a reasonable price. With AMD’s upcoming Interlagos series of Bulldozer based Opterons the core count per socket is expected to rise to sixteen meaning that a single dual socket board like the Supermicro H8DG6-F will be able to accept up to thirty-two cores. Furthermore, the G34 platform scales to four sockets and has an upgrade path making it attractive for many businesses looking at future upgrade paths. As for the H8DG6-F, it is perhaps the most feature-packed dual socket G34 board reviewed to date.
Test Configuration
To look at a typical use case I decided to use dual Opteron 6166 HE chips in this review. Eight DIMMs were also used which is very important with AMD socket G34 chips because using less than four memory channels per chip leads to severe memory bandwidth constraints.
- CPUs: 2x AMD Opteron 6166 HE’s
- Motherboard: Supermicro H8DG6-F
- Memory 8x 2GB ECC DDR3 UDIMMs
- OS Drive: OCZ Agility 2 120GB
- Enclosure: Norco RPC-4224
- Power Supply: Seasonic X650 (650w Gold level power supply)
One of the most interesting notes is that the 650w power supply seemed to be over-matched for this configuration. Under 100% load, the dual Opteron 6166 HE’s and the Supermicro H8DG6-F platform managed to pull under 220w at the power socket. This is a very important figure because even at load, the entire twenty-four core platform is pulling around 9w per core!
Board Layout
The Supermicro H8DG6-F is an EATX motherboard design which is very common among dual processor (DP) servers. Smaller ATX form factors are not suitable for the amount of PCB the CPUs and memory slots occupy.
Some features immediately stand out with the Supermicro H8DG6-F. First, the dual AMD G34 sockets are flanked by eight DIMM slots each for a total of sixteen DDR3 DIMM slots. One can use up to 4GB ECC DDR3 UDIMMs in each slot for a total of 64GB or 16GB registered ECC DIMMs for 256GB maximum memory capacity on the board. Next, one can see the onboard dual AMD SR5690 northbridge and SP5100 southbridge chipset combination. With the dual northbridge chips, the LSI SAS2008 onboard and dual CPU platform, the H8DG6-F is similar to the Intel-based Supermicro X8DTH-6F. The similarity continues with the multitude of PCIe expansion slots.
The Supermicro H8DG6-F includes six PCIe slots, three PCIe x16 slots, and three PCIe x8 slots (one x8 and two x4 electrical). This is a lot of potential PCIe bandwidth for things like 10GbE adapters, Infiniband adapters, storage controllers, and PCIe flash drives. Compared to both the Tyan S8236WGM3NR and S8230GM4NR boards previously reviewed, the Supermicro H8DG6-F has more usable PCIe lanes and physical slots making it quite flexible.
Augmenting the PCIe expansion capabilities, the Supermicro H8DG6-F has a higher-end feature set when it comes to SAS and SATA connections. There are two SFF-8087 connectors for the onboard LSI SAS2008 controller in the lower rear corner of the board which is fairly standard since in a storage chassis, the SFF-8087 ports are close to the drive back planes. The LSI SAS2008 part is very well supported and has drivers for virtually every OS and also works well with VMware ESXi out of the box. Along with the LSI SAS2008 controller, the H8DG6-F has SATA II 3.0gbps ports all connected through the AMD SP5100 southbridge. Between the 7-pin SATA connectors and the SFF-8087 SAS connectors the Supermicro H8DG6-F also includes two internal USB headers where one can install internal USB flash drives.
On the Supermicro H8DG6-F the rear I/O panel connectors are fairly standard fare for server motherboards. The rear I/O panel includes two USB ports, a VGA port, serial port and three RJ45 connectors. The RJ45 connector closest to the PS/2 ports is used for IPMI 2.0 administration and monitoring and the two right most connectors are linked to an Intel 82576 NIC. As has become my standard practice, I did not connect a keyboard, mouse, or monitor to the H8DG6-F so after quickly connecting three RJ45 connectors I was able to setup BIOS, install an operating system, and have a fully operational system.
Like its Intel-based brethren, the Supermicro X8DTH-6F, the Supermicro H8DG6-F is a very solid motherboard with a standard layout and features that are desirable in higher-end platforms.
Features
Supermicro’s IPMI and KVM-over-IP as described a few times on this site, allows for a lot of deployment flexibility. Things such as fan speeds, chassis intrusion sensors, thermal sensors, and etc. can be monitored remotely and alerts setup to notify the administrator of issues. Beyond this, the Winbond WPCM450 BMC chip also allows for the remote power up, power down, and reset of the server in the event that it becomes unresponsive. In fact, the test system has never had a keyboard, mouse, CD/DVD ROM, or monitor hooked up to it, even after multiple BIOS tweaks and three operating system installations. One such tweak was when I accidentally flipped the switch on node interleaving for a Linux installation:
The wrong node interleaving setting made a Ubuntu 11.04 installation impossible. Without the KVM-over-IP functionality, I would not have been able to remotely recover the system as it was stuck in a loop whereby the OS never fully loaded. Although one can administer many systems through SSH, VNC, X11, and RDC these days, KVM-over-IP is the only way one can make BIOS level tweaks.
Another important feature is the ability to remotely mount CD images and floppy images to the machine over the dedicated management Ethernet controller. This keeps maintenance traffic off of the primary Intel NICs and at the same time removes the need for an optical disk to be connected to the Supermicro H8DG6-F. Combined with the dual onboard USB 2.0 headers, mounting OS installation or recovery images remotely is a very simple affair.
In combination with the baseboard level management features just described, KVM-over-IP features of the Supermicro H8DG6-F are not to be underestimated. One can log on to the server using either a web browser with Java platform support, or through Supermicro’s IPMIview software and have remote console capabilities, with mouse support included. Oftentimes, users opine that shell access is enough to troubleshoot, but this BMC level access to the remote system allows one to immediately see things such as a hung and incomplete boot process, frozen servers, as well as manage motherboard and add-in card BIOS remotely. I believe that the IPMI 2.0 features, including the KVM-over-IP features are essential for a server-class motherboard unless one has an external KVM-over-IP unit and a network addressable power source. For users with redundant PSUs, as will be the most common use case with this motherboard, remotely toggling on/ off multiple power supply ports across multiple PDUs can be difficult making the motherboard option preferred.
Conclusion
With the dual AMD SR5690, dual socket G34 and accompanying sixteen DIMM slots for up to 256GB of RAM, and multitude of PCIe slots the board has plenty of room for expandability. Since the Supermicro H8DG6-F is a HyperTransport 3.0 motherboard, one can expect an upgrade path for Interlagos using a simple BIOS update which means that unlike the X8DTH-6F, an aforementioned similar Intel platform, the H8DG6-F has future upgrade possibilities to accompany its expandability. The inclusion of features such as the LSI SAS2008 controller that provides both 6.0gbps SAS 2 and SATA III interfaces for up to eight drives, the H8DG6-F is well suited to be a storage server platform. Between the expansion slot count, built-in LSI SAS2008 controller, and remote administration and monitoring features, the Supermicro H8DG6-F has a feature set well suited to both powerful but power efficient storage servers and virtualization servers where physical core count is important.
I own one of these boards and I absolutely love it. I wish you’d completed this review before I bought mine, because they’re so hard to find, and so expensive, I wasn’t sure it would be worth the added time and money over a $350 ASUS board. I think I made the right choice :)
Hello Stephen
how much did you buy your H8DG6-F mobo ?
did you bought this for a vmware box ?
Herve
Mine arrived a week ago. I was planning on running ESXi with passthrough for the SAS controller to an openindiana guest. But iommu/passthrough is not working, ESXi says it’s not supported (ACPI IVRS parsing failed). I have fiddled with every BIOS option imaginable to no avail.
Supermicro hasn’t responded yet, so I’m very disappointed.
Hey DD
I am considering the exact same setup you’re working on. Any luck passing it though? Can you pass anything else besides the sas controller?
No, unfortunately not. Passthrough is not working. I tried also some Linux distributions (Debian, Fedora) and Xen/KVM. They won’t even boot without completely disabling acpi… I think the BIOS is severely flawed.
I assume you upgraded the bios, right?
Did you try calling support? What about an RMA? The board might be defective.
Yes, the BIOS is the latest I found (1.0b). I don’t think the board has a (hardware) defect, it’s a buggy BIOS and IPMI tools that are driving me nuts :(
If I disable acpi and don’t try to enable passthrough it’s working solidly.
It may be the special mainboard config with two SR5690 or my setup with only one CPU (with the Bulldozer Opterons approaching I figured I will wait instead of spending money on a second CPU I’m going to replace soon). The two SR5690 are really nice since they provide more PCIe lanes, other dual mobos only have a single one or a SR5670.
Well thanks to your warning I think I might go with a Tyan board with socket C32, using the Opteron 4000 line of chips.
It’s very similar to the h8dg6, it also has duel SR5690, giving a ton of pcie lanes, but the latest bios release notes mentions fixes for iommu. So at least they haven’t forgotten about the function. And to release fixes for it, someone along the way must’ve tested it out, right? :-(
http://tyan.com/product_SKU_spec.aspx?ProductType=MB&pid=687&SKU=600000213
Good news! I contacted Supermicro again, this time through their web form and got a quick reply and a new BIOS (1.0c) and IPMI firmware (maybe my first mail got lost somehow). So far ESXi detected passthrough and I’m currently passing my additional LSI to the OI VM. Booted with the OI live cd and OI detects the LSI. Debian 6.0 live cd also boots now with acpi enabled. Getting late now, continuing testing tomorrow.
Great to hear DD! Did you have a BIOS date for 1.0c?
BIOS date is 06/23/11.
Interesting. DD, how did the testing turn out? I’m planning to buy this board and experiment with vga passthrough, although I will choose another brand if there is doubt about the functioning of the IOMMU on this one.
Passthrough with the LSI controllers ist working without a problem (both the onboard and an additional one). Didn’t try anything else since I read that ESXi has a limit of 2 passthrough devices per VM. VGA passthrough seems to be either very experimental or not supported at all by the different hypervisors if I remember things right.
Man, I’ve been having nothing but problems with my H8DG6-F! I’ve yet to actually get it to turn on.
I’ve dual 6128 8-core Opterons (I’m on my second pair, AMD sent me a second set since Supermicro support recommended that I RMA the original ones that weren’t posting in the board) and a whole bunch of various 4GB (unbuffered, I know I know…it’s what I’ve got though!) DDR3 dimms. I’ve tried bout a 750w Corsair Gold AX750 and a 750w Seasonic Gold X750 and neither manage to get the CPU fan to actually spin.
I do see a flashing green light on the board (DP1 I believe) that shows that there’s at least -some- form of power to the board, but that’s always as far as it gets.
Everything to me points toward faulty PSUs, but they have no problem posting any of my consumer-level systems. Supermicro has said that they don’t have a proprietary pinout for their PSU, but then turns around and say they only have compatibility with their own…
Does anyone have any suggestions? Is there something that I’m not getting here? I’ve built hundreds (when I worked in a computer store 5 years ago) of consumer-level systems, but this is my first modern server-class build.
LOL, nevermind. Apparently between one of my cables being bad and such, I ended up using the EPS12V cable from the Seasonic PSU on my Corsair AX750 (and yes, I know they’re the same) with a splitter, and it posted. Happy Day! :)
Ha! Good to here Kyle!
I tend to use the X-Series PSUs for that reason. Good to hear you are making progress.
@ac: Same problem with the Tyan S8225. Today I tested VMware vSpehere ESXi 5.0 with this board (1x Opteron 4180 and 32GB RAM).
2011-09-20T09:39:45.796Z cpu0:2618)Loading module AMDIommu …
2011-09-20T09:39:45.796Z cpu0:2618)Elf: 1862: module AMDIommu has license VMware
2011-09-20T09:39:45.797Z cpu0:2618)AMDIOMMU: 510: Loading AMD IOMMU driver…
2011-09-20T09:39:45.797Z cpu0:2618)AMDIOMMU: 520: AMD IOMMU driver version 1.26, built on: Aug 18 2011
2011-09-20T09:39:45.805Z cpu0:2618)WARNING: AMDIOMMU: 1312: ACPI IVRS parsing failed!
2011-09-20T09:39:45.805Z cpu0:2618)WARNING: AMDIOMMU: 580: failed to detect any AMD IOMMU device!
2011-09-20T09:39:45.805Z cpu0:2618)AMDIommu failed to load.
2011-09-20T09:39:45.805Z cpu0:2618)WARNING: Elf: 2754: Kernel based module load of AMDIommu failed: Failure
I’ll try to contact the support and see if they will fix the problem.
So long
@ac Today I trioed to use IOMMU on my new Tyan S8225 (with 1x Opteron 4180 and 32GB RAM) and found out the same problem. VMware vSphere ESXi 5.0 shows the same error in the kernel log:
2011-09-20T09:39:45.796Z cpu0:2618)Loading module AMDIommu …
2011-09-20T09:39:45.796Z cpu0:2618)Elf: 1862: module AMDIommu has license VMware
2011-09-20T09:39:45.797Z cpu0:2618)AMDIOMMU: 510: Loading AMD IOMMU driver…
2011-09-20T09:39:45.797Z cpu0:2618)AMDIOMMU: 520: AMD IOMMU driver version 1.26, built on: Aug 18 2011
2011-09-20T09:39:45.805Z cpu0:2618)WARNING: AMDIOMMU: 1312: ACPI IVRS parsing failed!
2011-09-20T09:39:45.805Z cpu0:2618)WARNING: AMDIOMMU: 580: failed to detect any AMD IOMMU device!
2011-09-20T09:39:45.805Z cpu0:2618)AMDIommu failed to load.
2011-09-20T09:39:45.805Z cpu0:2618)WARNING: Elf: 2754: Kernel based module load of AMDIommu failed: Failure
I’ll try to contact the Tyan support for a new BIOS release.
What’s the best source to purchase one of these m/b’s?
Pricey but I like it
Is this still the best bang for your buck esxi setup?
I have to warn everebody, this MB has an unknown issue, afetr 6-7 years of using them in last 6 months two servers on this MB stopped booting, you can turn on server, all lights flash OK but server cannot start POST, all firmware have the last version, all components (RAM, CPU, …), except MB, are OK. Our NTT support suspect that is because of BIOS overheating problem but there is no 100% certainty and supermicro cannot solve the problem. This issue reveals itself after starting the server after a power failure. Summary we have abut 10 servers based on this MB so for know 20% of them are useless.
I have just purchased a Supermicro box including this Mother Board, and 32G of RAM. It booted right up and runs, however when using VMware 15 I get what looks like IRQ comflicts… but during boot of Mint 19 it reports bios flaws…
Trying to get suitable software to flash the BIOS with 318. I’ve got 3.5 now.
Anyone help?
does the system actually work with the OCZ Agility 2 120GB ? I can’t get mine to acknowledge SSD drives
BIOS issue