My main server, the Big WHS now houses over 60TB of storage, runs multiple VM’s, and has over 10 Gigabit NICs. At the heart of this server, is a Supermicro X8ST3-F. It was not the first motherboard I tried in the server, as I originally tried using an ASUS P6T7 WS Supercomputer in the Big WHS, but it has been running solidly since its first installation. Aside from its stability, it also comes with many PCIe slots, an onboard LSI 1068e based 8 port SATA/ SAS controller, dual Intel Gigabit NICs, onboard video, and IPMI 2.0 with KVM over IP.
Test Configuration
For this review, I am using the Big WHS. This is not my standard test configuration, however if one is building a server off of the X8ST3-F instead of a lower-end motherboard, the server is most likely going to have many disks, lots of RAM, and many ethernet ports.
- CPU: Intel Core i7 920
- Motherboard:Â Â Supermicro X8ST3-F
- Memory: Patriot Viper 12GB DDR3 1600
- Case (1): Norco RPC-4020
- Case (2): Norco RPC-4220
- Drives: 12x Seagate 7200rpm 1.5TB, 2x 7200.11 1TB, 12x Hitachi 7200rpm 2TB and 2x 1TB, 8x Western Digital Green 1.5TB EADS, 2x Western Digital Green 2TB EARS.
- SSD: 2x Intel X25-V 40GB
- Controller: Areca ARC-1680LP
- SAS Expanders: 2x HP SAS Expander (one in each enclosure)
- NIC (additional): 2x Intel Pro/1000 PT Quad , Intel Pro/1000 GT (PCI)
- Host OS: Windows Server 2008 R2 with Hyper-V installed
- Fan Controllers: Various
- PCMIG board to power the HP SAS Expander in the Norco RPC-4220
- Main switch – Dell PowerConnect 2724
Expansion slots
One of the key differentiators for the X8ST3-F is the number of expansion slots available. The X58 based X8ST3-F has three PCIe x8 electrical slots and one PCIe x4 electrical slot housed in one x16 physical slot and three x8 physical slots. Beyond this, there are two legacy PCI slots. The four PCIe slots can easily handle quad port NICs, raid cards, HP SAS Expanders, and etc. Beyond these the two PCI slots allow for additional Intel Gigabit NICs. While this may seem like a fairly expandable system, one needs to remember that there are two onboard Intel NICs, a management Realtek 10/100 NIC, and an onboard LSI 1068e controller. Practically speaking the X8ST3-F gives the user the option of configuring plenty of network bandwidth alongside enough hard drive expansion options to handle any enclosure. Case and point, the Big WHS usually has 12 Intel Gigabit NICs (two onboard, eight from Quad port cards, and another two in PCI slots) and port capacity for 46 internal drives.
Another strong possibility for the X8ST3-F is as a NVIDIA CUDA machine using either single or dual slot cards. While consumer boards can handle more GPUs using switched PCIe slots, the X8ST3-F has many onboard features (LSI RAID controller, Intel NICs, IPMI 2.0, and etc) that the consumer boards lack.
Aside from the expansion slots, it is worth mentioning that the Intel LGA1366 platform also offers six DDR3 slots. This is an important difference between the Intel LGA1156 platform and AMD AM2+/ AM3 platforms because one can either utilize more RAM or utilize lower density DDR3 modules.
Integrated LSI SAS1068E RAID Controller
The Supermicro X8ST3-F differentiates itself among many of Supermicro’s other offerings by providing an integrated 8-port LSI SAS1068E controller. The only major downside to this controller is the fact that it only supports raid 0, 1, and 10 in stock form. For a lot of users, this is enough as raid 1 provides redundancy and raid 10 provides redundancy and speed without the overhead of performing parity calculations. One of the cool features is that using a reverse Mini-SAS to SFF-8087 cable (I used a 3ware/ AMCC CBL-SFF8087OCR-06M) one can connect a HP SAS Expander and use the onboard controller for 36+ disks using one expander (a second can be added easily). This is a solid solution for those using OpenSolaris, FreeBSD, or Linux software raid based solutions. The LSI SAS1068E is not the fastest controller by any means, but it is supported by Microsoft Windows Server products as well as open source operating systems alike.
It should be mentioned that one can purchase and use a AOC-IButton68 to enable RAID 5 support on the X8ST3-F’s onboardLSI SAS1068E controller. This iButton based device has a street price of about $75. Frankly, I would advise against this unless there is a need to run RAID 5 on this controller. Dedicated controllers, while costing considerably more, do have faster chips for parity calculations, can run RAID 6, have dedicated cache, and can use battery back up units. Then again, if one really wants basic RAID 5 this is not a bad option.
Dual Integrated Intel Gigabit LAN
Like the Supermicro X8SIL-F, the Supermicro’s X8ST3-F has two Intel 82574L based gigabit NICs onboard. For servers, Intel Gigabit NICs are superior to other offerings. This is for two reasons. First, operating system and Hypervisor (VMWare ESX / ESXi, Microsoft Hyper-V, and etc) support is basically universal for Intel NICs. Second, I have seen much better throughput on Intel NIC to Intel NIC gigabit networks versus networks with Realtek and Marvell network controllers.
Each Intel 82574L supports features such as like TCP/UDP CPU offloading and TSO for lower CPU utilization, PXE/ iSCSI boot, Jumbo frames, two transmit and receive queues per port, a 40KB packet buffer, and etc. The Intel 82574L is the same controller used in Intel’s Gigabit CT (EXPI9301CTBLK) PCIe x1 network adapter. In essence, the Supermicro X8ST3-F saves one $60-70 and a PCIe slot (if one were to get a dual port Intel NIC) or two PCIe slots versus a lower-cost board where Intel NICs would be required separately.
Also of note again, with the X8ST3-F, the Supermicro IMPI 2.0 feature includes the ability to mount CD’s and DVD’s over the onboard 10/100 Realtek controller. Furthermore, the keyboard and video inputs and outputs can also go over the IPMI 2.0/ KVM over IP interface offloading this data from the dual Intel gigabit NICs. The net result is that the Intel NICs will have more bandwidth available for sending and receiving content on the X8SIL-F because it has the third NIC (Realtek) for management functions.
IPMI 2.0 and KVM over IP
One of the best features of Supermicro boards is the IPMI 2.0 and KVM over IP built in to many of the boards. Aside from remote system monitoring of things such as CPU voltages, temperatures, fan speends, and other parameters, Supermicro’s implementation has great remote control features through the web interface. My three favorite IPMI 2.0 and KVM over IP features are depicted below.
One will notice a few things in the above screenshot. First, this interface is slightly different than the X8SIL-F, yet seems to provide the same functionality. The browser window depicted on the left shows remote power control of the server that allows one to remotely reboot and power up or down the server among other things. On the right side of the above screenshot (“Redirection Viewer” window), one can see the Windows Server 2008 R2 login screen as seen through the Java based KVM over IP application. If one were to plug in a monitor to the X8ST3-F’s VGA output, this is the same output they would see. Unlike RDC or VNC which can be used to have remote monitor output in an operating system, Supermicro’s KVM over IP works at the system board level so one can use it for virtually any system maintenence task, including BIOS configuration.
The keyboard and mouse functionalities compliment the VGA over IP so remote administration is greatly added. A final feature of the X8ST3-F’s IPMI 2.0 is the ability to mount CD or DVD ISO files as well as floppy images over the network. To the X8ST3-F’s operating system (and to some extent the BIOS) these network mounted images look like they are local drives meaning that one can easily use this interface to boot and install operating systems or other software even without physical CD/ DVD drives attached. In fact, one could use this interface in a diskless system to run network stored LiveCD operating systems on the server. Overall, Supermicro’s IPMI 2.0 and KVM over IP are great features that are very hard to do without once one becomes accustomed to the convenience. It works well enough that over the past few months I have yet to connect a CD or DVD ROM drive, monitor, keyboard, or mouse to the system.
Dual Internal USB headers
The X8ST3-F has dual internal USB headers. This allows one to add a USB thumb drives to the interior of the enclosure housing the Supermicro X8SIL-F. Small business storage servers, this is a great feature since one can have two USB drives either for dual OSes or one OS USB drive and a secondary firmware update USB drive. FreeNAS, Openfiler, unRaid, untangle and many other operating systems can install directly to a relatively low-cost flash drive saving onboard SAS/SATA port (the X8ST3-F has 8x SAS/SATA ports on the onboard LSI 1068e and six onboard SATA II ports form the ICH10R). Since the USB sticks can be installed inside the enclosure, it offers more protection from becoming wrongfully uninstalled than an externally installed drive. Finally, one can mirror the USB thumb drives in the event that one drive has flash wear out from usage.
Onboard VGA
For video, the Supermicro X8ST3-F provides a Matrox G200eW based GPU that outputs only to analog VGA on the backplane. The saving grace of this solution is the integration with the Winbond WPCM450 BMC which provides IPMI 2.0 features and KVM over IP. As a side note, if you are running a quad core, LGA1366 based Intel CPU, you have the same 8MB of cache as the Matrox G200eW has in DDR2 memory. As a comparison, high-end dedicated GPUs currently have 2GB or more of onboard GDDR5 memory. Of course, the saving grace is that with KVM over IP, most users will not need more than what this Matrox G200eW has to offer.
An important note here is that the onboard GPU does save the need to use a PCIe slot for graphics while at the same time not providing the airflow restrictions of an add-in GPU.
Conclusion
The Supermicro X8ST3-F differentiates itself with its onboard components. First, IPMI 2.0 along with the Matrox G200eW allow for remote server administration. Supermicro’s IPMI 2.0 implementation means that a user does not need to install a cheap video card nor a CD/DVD ROM in the server. Second, the LSI SAS1068E is a solid SAS controller that is well supported in various operating systems and does not require an expansion slot. Furthermore, the LSI controller supports HP SAS Expanders making the X8ST3-F a good choice for storage servers based upon software raid such as ZFS/ Raid-Z or Raid-Z2. Third, the dual onboard Intel 82574L gigabit NICs save a user from having to purchase a dual port Intel Pro/1000 PCIe card and again save an expansion slot.
It is hard to think of a $340 motherboard as a value board, yet the X8ST3-F is if one uses the onboard components. All tallied, $20 for a generic DVD ROM drive and $30 for a cheap VGA card (neither of which will work as well for network administration as Supermicro’s IPMI 2.0 implementation) save a user $50. The LSI SAS1068E is the same ASIC used in the Intel SASUC8I RAID Controller which is a $160 8-port controller. Finally, the two onboard Intel Gigabit NICs are roughly equivalent to an Intel Pro/1000 PT Dual port card which runs approximately $160 also. In essence, the Supermicro X8ST3-F provides lots of expandability plus about $370 worth of onboard hardware for only $340. That is a fairly strong value proposition.
I just finished building my server with this board and currently waiting for a Dell PERC 6/i to finish the install. It’s a great featured filled board as mentioned but seems like supplies are extremely low around the internet as I’m sure this board is soon going to be EOL.
My setup:
i7 920 2.67GHz, 12GB of RAM
VMWare ESXi 4
500GB x3 RAID5 for VMs
1.5TB x4 RAID5 for Data/Backups
VM1: Windows Server 2008 file serving and DFS-r with remote office
VM2: **testing** Windows 7 Ult + vt-d passover of Ceton InfiniTV 4 to be a dedicated Media Center PC
VM2 is going to be a test, in theory i believe it should work out but we will have to wait and see.
How picky is the X8ST3-F about memory? Experience any issues like with the X8SIL-F review?
I originally installed 6GBs of Kingston KVR1066D3N7K3. Then decided I wanted to bump up so I pulled some OCZ OCZ3X1600LV6GK from another workstation while I am waiting on my shipment of another set of Kingston KVR1066D3N7K3.
The Kingston and OCZ seem to work fine together without any problems… The OCZ is a 1.65v while the Kingston is 1.5v
If you are looking into buying the board, I would just go with the Kingston since its an approved set according to Kingston’s website.
Interesting looking board, do you have any idea of the power consumption? In conjunction with a 32nm 1366 quad, this could make the pefect 24/7 vmware lab server.
Maybe Patrick can share his numbers too but mine is clocking in at ~220 watts with few harddrives. Haven’t done any comparisons of heavy load vs idle.
I have had geat luck memory capability wise, admittedly I have been using only non-ECC modules. It seems like the memory compatability is basically the same as the i7-920 (so you need sub 1.65v DDR3 DIMMs) with an X58 chipset.
On the power consumption, I will do this at some point in the near future. I haven’t tested the system as a whole in quite awhile but I would venture to guess it is currently pulling 350-450w at idle. Let’s see how that guess holds up! The big factor in not doing the power consumption figures is the fact that the box runs 24×7 and only comes down very rarely for Windows Server 2008 R2 updates/ patches. I have been very satisfied with the reliability.
this is a great board. I bought it after reading your original review. I’m using the onboard LSI, with reverse breakouts into a HP expander, in a norco case. Works great for my WHS in JBOD. the LSI needs a firmware update to run JBOD, but the SM tech support essentially walked me through it.
Hey Patrick,
Cybertron from Anandtech here. I’m really diggin’ this mobo, but I’m torn about the raid config. I’m looking at spending a few extra bucks and getting the Supermico Mobile Rack with a “Smart chip” inside that will tell me when the drives fail, and I’m thinking I should go with a proper Raid controller.
My options seem to be:
W/ Add-on Raid card: plug 5x 2TB in to RAID5 array, and 2x RAID0 (sas) array.
W/O Add-on Raid: Plug 5x 2TB into the Intel ICH10 controller and do ZFS (like mentioned in the thread) and then 2x RAID0 (sas) array on the LSI onboard.
My problem is, this motherboard is so expensive, I really dont want to buy a $500+ raid card on top of it.
I think that is right, unless you decide to do raid 6 (I am primarily off of raid 5 right now). Also at 10+ drives, you want to think about having a hot spare. I am not a huge fan of raid 0 because downtime due to disk failure almost always causes some level of pain in restoring.
This can be accomplished either with ZFS (Raid DP) or with a good controller.
I do not have a Supermicro Mobile Rack yet, but I may try reviewing more of their products in the future.
The main value of the X8ST3-F is the onboard components. The dual Intel NICs save not only expansion slots, but they cost quite a bit if you try adding them to a native Realtek NIC board. Likewise with the LSI controller.
Another option, of course, is to get a lower-cost Supermicro motherboard without the onboard controller if you are going the dedicated raid card route. Supermicro makes some solid LGA1156 boards if you are OK moving to that type of platform. I actually reviewed the X8SIL-F (LGA1156 and no LSI controller) that sits in the extra storage enclosure for my main server based upon this board.
Really, I think the starting point for you needs to be:
1. What OS(es) do I want to use. (If you don’t want OpenSolaris or FreeBSD, then going dedicated raid is an obvious choice).
2. How much do I want to expand?
I will say that it pays to be thorough in planning. I originally bought an Asus P6T7 WS Supercomputer motherboard that this X8ST3-F replaced. If I had bought the right board in the first place, I would have saved some money.
Patrick,
I’m going to have to play with ZFS since my last implementation did not go so well. I’m very curious to see your suggested EON in action or Nexenta. I guess I’m still torn between a single and a dual machine setup, but I’m pretty sure I’ll settle on a single. I too have made the same mistake by going with a less-than-server grade hardware. I wont go with anything other than supermicro, tyan, or the like.
I see there are a couple boars like the X8ST3-F that dont have the LSI controller that can be had for a bit less. And my reason for RAID0 is because all these VM’s are in a test bed and I dont care if they die. I’ll make backups of them after I get them configured how I want. I just want the speed of SAS so I can have 5 or so machines running at the same time.
As for expansion… I’d like to *think* that I may need more room in the future but the fact is, I’ve not used but 1TB of my 4TB NAS. I do plan to start more DVD ripping, etc, but I dont see exceeding the planned 6-9TB config. I’m thinking either 5x 2TB in RAID5 or 8x 2TB in raid 6 with a hot spare. That should take care of my needs for a long time to come.
So the last question is i7 vs. x3440(etc) for the processing of VM’s. Are you happier with one than the other?
One thing to consider is that we are going to see 3-4TB drives in the next 7 months. So storage is going to keep getting cheaper, and the number of drives required for a given capacity is going to keep decreasing.
The i7 v. X3440 and X3460 (I haven’t posted the review of that yet) is basically about memory and PCIe lanes v. ECC. With the i7 you can easily add 12GB of ram for about $300 using 2GB sticks. With the X34xx series CPUs you are going to want to use 4GB sticks. Also, the i7 and X8ST3-F are much less picky about RAM. The i7 has more memory bandwidth which is also important for running multiple VMs. CPU wise, both are 4 core with hyperthreading and realistically, are pretty close in speed since they are basically i7 8xx series CPU’s with ECC support. The X3440 and X3460 both have lower power consumption than the i7 920. Realistically, the i7 LGA1366 series is faster but for a lot of applications, there isn’t a huge difference.
Patrick,
Have you successfully update the BIOS via IMPI utility?
I have not, primarily because that means taking the box down. In the last 60 days it has been rebooted one time for Windows 2008 Server R2 patches. With that kind of stability (frankly overkill for my application), I have had little reason to change the BIOS. I think you can do it though.
does the on-board RAID controller support JBOD?
thinking for use as a WHS mobo.
Yes, with one major caveat. I think you will need to flash the controller (in DOS which is easy with IPMI 2.0/ KVM-over-IP) using the LSI IT firmware to enable JBOD mode.
I’m interested in implementing the same idea as Michael M in the first comment… using vt-d to pass through a tuner to a Windows 7 vm. Michael if you still check this post, did that work with this board? What hypervisor did you use? Does anyone have any experience with vt-d on this board in general?
Thanks!
Lucky for you Luke that I randomly checked back this post :P
I have not been able to test out an InfiniTV 4. However, I have tested this out with 2 analog tunners + 2 HDHomeRun HD tunners and everything was working pretty smoothly.
My testing environment was a Windows 7 VM passing over the two analog cards and streaming Live TV to a Xbox 360 using WMC extender.
Hi,
thanks for the review of the board.
Do you have any numbers on idle power draw?
The numbers Michael and Patrick published seem way too high.
Could you just measure the board with 3 DIMMs please?
thanks,
Olli
I’m actually away from it now and won’t be able to take another look at it for a while but I believe mine was idling at around 155 watts with 8 drives and i think 7 fans. Its in a rackmount Supermico case… I noticed in the BIOS if I put my fans on full performance I can idle at around 245 watts (100 watt difference just in fan power)… currently they are on power save mode running at low speed except for one that is blowing passive across the RAID controller.
Your review was one of the driving factors for me to purchase a X8ST3-F board to go with an Intel E5640 Xeon I got at a great price. Unfortunately, I also purchased a 3×4 kit of Kingston KVR1333D3E9SK3/12G to go with it. For the life of me, I can’t get the board to recognize all 3 sticks. Is there a trick to what slots I need to put the memory in, or specific BIOS settings I need to enable/disable to access all 3 sticks?
Like all X58 based systems, there are specific slots you should try. If you look at the X8ST3-F manual starting on page 2-8 shows you everything you need for memory installation. Also, I checked the Kingston HCL and it shows that that memory works with the Supermicro X8ST3-F so you should be OK from a compatibility perspective. If it still fails, you may have a bad memory slot (it happens). I would at that point e-mail or call up tech support as they tend to be pretty good.
Great review and nice HQ pictures!
@David: Thought I’d point out that Patrick is wrong about flashing the LSI for JBOD, there’s a convenient jumper (JPS2) located between the left 4 and right 4 SAS headers just below the screw hole that allows you to select the desired MPTSAS BIOS: Close it (default) and the LSI is in Software RAID mode, open it and you’ve got IT mode (JBOD).
Good point Dave, I did this just a few days ago on one of my X8ST3-F’s. Thanks for pointing this out.
I love server boards like these, although this one is a bit too much for my future home NAS. I’m looking at the Supermicro X8SIL-F, but that one only has 6xSATA. The Supermicro X8SI6-F has more hdd capability but 14xSATA is way too much. Is there a 1156 socket Supermicro server board capable of handling 8xSATA? Any chance the next linup for socket 1155 will bring a 8xSATA server board around?
There are non-server boards that incorporate secondary SATA controllers like the JMF363 chip which adds two SATA ports. The big issue here is that support for extra controllers tends to be at best poor.
What you could do is get a 4-port add-in card as SAS cards tend to come in multiples of four ports (4, 8, 12, 16, 24 and etc.)
Patrick or Michael,
It’s been a while since you reviewed this board, and I’m wondering if you guys have any power consumption numbers. I’m interested in this platform, and trying to find out what I will be looking at for a 24/7 system.
Thank you,
Chris
I ordered this board based on this review and couple opinions over Internet.
I was wondering can it handle my non-ECC DDR3 memory and standard disks.
The result is so satisfactory that I would like to share.
I ordered Xeon X5650 and this board from separate suppliers. Moving from AMD FX-6100 and ASUS M5A97.
I put 6 x 4GB Kingston Hyperx 1600MHz (all available 24GB), 3 disks (2 x WD RED 2TB in software mdadm RAID1, 1 x Intel 520 SSD) and 2 x Infiniband 10Gbit cards. PSU is high quality standard 500W ATX. Memory runs at 1333MHz – max available.
First powerup was incredible. FX-6100 is 3-core (+ something like HT, so 6 threads in total) CPU, when X5650 is 6-core (+HT, so 12 threads). All VMs booted instatly! This is why I like Linux – moving from one platform (AMD) to the tother (Intel) works like a charm!
The board is running 3Gbps SATA however there is no noticeable slowdown on SSD disk.
After all was working fine added second disk to RAID1 and resynced (was used as backup).
Now I’m going to write Zabbix template to add monitoring, since so far I didn’t found if someone did it already..
Took only few hours to completly move to different platform.
Has anyone installed Windows Server 2012R2 or 2016 on this board? SuperMicro only lists 2008 as compatible. I have the chance to get a great deal on a system with this board in it, but want to be able to run Server 2016. Anyone know if it will work? Thanks!