It has been a long time since I have posted about The Big WHS. At the last update in May 2010, the machine occupied 8U using two 4U enclosures, and was topping 60TB of raw storage capacity. Since then there have been quite a few developments that I thought I would write about.
The current hardware list is as follows:
- CPU: Intel Xeon W3550
- Motherboard: Supermicro X8ST3-F
- Memory: 24GB ECC unbuffered DDR3 DIMMs (6x 4GB)
- Case (1): Norco RPC-4020
- Case (2): Norco RPC-4220
- Drives: 8x Seagate 7200rpm 1.5TB, 4x Hitachi 5K3000 2TB, 20x Hitachi 7200rpm 2TB, 8x Western Digital Green 1.5TB EADS, 2x Western Digital Green 2TB EARS.
- SSD: 2x Intel X25-V 40GB
- Controller: Areca ARC-1680LP (second one)
- SAS Expanders: 2x HP SAS Expander (one in each enclosure)
- NIC (additional): 1x Intel Pro/1000 PT Quad , 1x Intel Intel Pro/1000 GT (PCI), Intel EXPX9501AT 10GigE
- Host OS: Windows Server 2008 R2 with Hyper-V installed
- Fan Partitions updated with 120mm fans
- PCMIG board to power the HP SAS Expander in the Norco RPC-4220
- Main switch – Dell PowerConnect 2724
The Big WHS Changes
First off, the CPU was changed to a proper Xeon W3550 CPU adding both ECC support as well as slightly higher clock speeds. It was a simple remove and replace for the old Core i7-920 so this was an easy update. Frankly, in any system this large, not using ECC memory is a sub-optimal idea and this helped a lot towards that end. I could see a day though where a DP motherboard is added into the equation which would require another component swap. Memory has been changed to 24GB of ECC DRAM accordingly. I did find through trial and error that not all memory suitable for Intel DP motherboards are suitable for the X58 system. Below see a picture of what 24GB (and hundreds of dollars) worth of incompatible RAM looks like. Bottom line, always check the hardware compatibility lists.
The Areca 1680LP started acting up which caused me to re-evaluate the SAS controller situation. Next steps are moving to a simple HBA architecture. WHS is unable to use all drives, and due to new project load, my storage consumption has slowed significantly. I will likely keep the Hyper-V VM using Drive Extender Storage and getting more elegant with the Windows Home Server 2011 environment in the future. One important note for HP SAS expander owners is that I did see PCB warp due to all of the SFF-8087 connections in the card. The new curvature of the HP SAS Expander can be seen in the below photograph.
I started to use Mellanox Infiniband dual-port connections between the Big WHS and other machines (as can be seen in the photo above. The short story is, on Windows the things work great, on other operating systems they are not as easy to use. I have started testing Intel EXPX9501AT 10GigE adapters and I do like the architecture except that 10GigE switches are too expensive at the moment. For those that read the forums, there is a 10GigE adapter thread with some additional information.
Additional hard drives have been added with some of the older Seagate 7200.11 1.5TB drives removed and the Hitachi 7k2000 1TB drives and Seagate 7200.11 1TB drives were completely removed leaving the minimum capacity of 1.5TB for any installed drives. Total raw capacity is 76TB of rotating disks at the moment.
On the cooling front, I had a fan controller become non-functional so I did swap out the fan partitions for Norco’s 120mm fan partitions (see the test fittings above). As a positive note, this leaves one with enough fan headers on the Supermicro X8ST3-F to power all fans in the primary case off of the motherboard and therefore get remote IPMI 2.0 fan readings. One can see more experiences in the Norco RPC-4020 and RPC-4220 120mm fan partition guide.
From a software side, the big additions have been Microsoft Windows Server 2008 R2 SP1 and now new Windows Home Server 2011 and Windows Small Business Server 2011 Essentials virtual machines.
Conclusion
The maximum capacity I will see with The Big WHS is 80TB, only about 4TB greater than today. Beyond that point, I will be getting much more elegant with the solutions, incorporating tiered storage and a host of other enhancements. 2011 is shaping up to be a very exciting year indeed. One thing that I am glad about is that the Big WHS is no longer a common testbed for this site’s reviews. Multiple test systems have allowed the Big WHS to deliver two consecutive 90 day up time periods with the shutdowns being for a dual hardware installation and Windows Update related installations. Frankly, the stability has been awesome.
Did the warping on the expander cause any issues or is it still in the realm of cosmetic at this point?
Still only cosmetic. That is one of the more noticeable angles.
Only having a fifth of your storage on my WHS, I have come across an issue when trying to use the automted WHS backup functionality. It won’t let me include my large 4TB drive (RAID array) in the backup because it is greater than 2TB. How have you managed to get round this backup limitation?
Since you use so many SFF-8087 connections, Where do you buy your cables? I need to buy 4x 1 meter cables? Got a gooe source?
“Total raw capacity is 76GB of rotating disks at the moment.”
I think you meant 76TB
Thanks Paul. This has been fixed.
I apologize for my newbie question but I am really curious: How exactly the Infiniband controller connecting the Big WHS and other machines?
Thank you!
So I am simply using Infiniband controllers on the connected boxes. Dual port controllers are fairly cheap making this setup inexpensive. That being said, it is not a simple setup at all.
Hey Patrick. I have just read through all your blogs concerning the Big WHS. Whilst my plans aren’t anywhere near as ambitious and my hardware pales into insignificance against that shown, I am in the process of at least adding more drives and RAID options.
At the moment I have a simple Adaptec 5405 with 4 x 1.5 TB Samsung drives as a RAID 5 array, with a 2 WD drive as RAID 1 for the OS on the MB controller and a third as a spare. I run Win 2003 SBS rather than WHS, but this is something i am looking to change. Also, I have been toying with the idea of add multiple VM using ESXi, such as MythBuntu and an Asterix machine.
I’m not as informed as you with regard to all of this and I noted that you have used VM for various software RAID and NAS machines. What would the purpose of using a software based RAID device as well as hardware RAID? How have you got your drive actually configured in arrays, and what RAID version do you find most acceptable for which purpose?
One other point. I have read (no idea where now), that actually running RAID/NAS OS systems as VM is not that brilliant and that they do not behave always as expected. How have you found this in your setup.
Chris, lots of good questions. May I suggest you ask these in the forums. It would let more people weigh in. Vt-d is basically a requirement in NAS OS in VM at this point.
Thanks Patrick – I did just that. Was sure where to post so it went into the DIY builds. Please post here to answer these questions – http://forums.servethehome.com/showthread.php?378-Questions-arising-from-the-Big-WHS-blog-RAID-types-and