STH Colocation – May 2014 Update

After a quick hop out to Las Vegas, Nevada the STH colocation architecture has received a significant upgrade in May 2014. We originally posted our plans almost two months ago. A few iterations later and everything is now up and running. In total the node count has gone from 6 to 9 while only adding 1 amp to the setup (running right now at 6amps with most of the architecture idle.) This is a very impressive result as there have been major upgrades in terms of potential CPU power, drive speed and capacity, memory capacity and even interconnect capabilities. In keeping with tradition we take a look at some of the hardware that went into the colocation facility.

For those that did not read the forum thread, we got a great deal on a Supermicro Twin SYS-6027TR-D71FRF which is a monstrous system. It has 2x Intel Xeon E5 dual socket systems in a single 2U enclosure. Both systems are fed by a pair of redundant power supplies. Each node comes standard with a LSI SAS 2108 RAID controller and Mellanox ConnectX-3 56gbps FDR Inifiniband/ 40GbE onboard. Compared to the previous Dell PowerEdge C6100′s these are significantly more powerful. Here is how the internals are configured. 

We did an initial test of the storage and networking subsystems with only one processor. The next step was adding RAM. This was somewhat tricky as there are only 4 DIMM slots per CPU for a total of 8 slots per node and 16 per chassis. Using dual Intel Xeon E5-2665′s in each node and filling each DIMM slot with 16GB Samsung ECC RDIMMs yielded 16 cores, 32 threads and 128GB of RAM per node.

One other excellent feature of the Supermciro SYS-6027TR-D71FRF is that it has three PCIe expansion slots to go along with its onboard Infiniband and SAS RAID controllers. Two are full height and one is half height but there is enough room for many types of cards.

In terms of drives we ended up using a stack of Crucial M500′s. With lots of RAM in the colocation cabinet there is simply not much disk writing going on for a website like STH. Each of these Supermicro nodes got 2x Seagate 600 Pro 240GB “OS” drives in RAID 1, 2x Crucial M500 480GB “storage” drives in RAID 1 and 2x 4TB Hitachi CoolSpin 4TB drives in RAID 1 simply to act as backup targets. Crucial M500 240GB drives also found their way into the pfsense nodes.

Colocation cabinet redo

The impacts were that we increased density significantly. We replaced 24 Nahelem EP cores (L5520) with 24 Westmere-EP cores (X5650 and L5639) while keeping the same 216GB of RAM total we had in the two Dell C6100 chassis earlier. We added 32x Xeon E5-2665 cores and 256GB of RAM, over 4TB of SSD and 8TB of disk capacity and got a fast Mellanox FDR Hyper-V replication link between the two main Hyper-V nodes: