Dirt Cheap Data Warehouse – Low Cost Database and Storage Servers

The Dirt Cheap Data Warehouse is a project to build low cost database and storage servers for development. Jeff Shukis both author on STH and active forum moderator recently detailed his very inexpensive data warehouse project. As a Silicon Valley start-up costs must remain in check while at the same time providing a lot of performance. The Dirt Cheap Data Warehouse is built using mostly recycled HP and Dell gear, including several 8 CPUs per 2U Dell C6100 XS23-TY3 (Intel) and the Dell C6145 (AMD) cloud servers. In the forum post, Jeff highlights the design philosophy of the Dirt Cheap Data Warehouse:

The secret to the Dirt Cheap Data Warehouse is this: Used generation-old server technology from eBay, plus a large number of consumer-grade SSD drives, carefully selected, configured, and deployed to avoid resource-wasting bottlenecks and to maximize query throughput and query throughput per dollar.

From the pictures, primary storage is 2.5″ consumer grade SSDs. The main storage uses a HP DL585 G7 quad Opteron 6000 series platform and 10 disk controllers (over 100 SAS channels) and handles over 16GB/s in Oracle database query throughput.

Further, both gigabit networking plus a 40gbps QDR Infiniband is in play. There are a total of 48 gigabit ports using a Dell switch. Further, a 40gbps QDR Infiniband network is handled by its own switch.

This storage and networking backend feeds a server farm. One can see that there are at least 16 compute nodes in the below picture, 12 dual CPU and 4 quad CPU.

One key to setting this all up on a budget is carefully selecting previously used parts. The STH Forums includes a Great Deals forum where users trade finds on ebay and elsewhere on great server hardware. Many of these components were purchased at a small fraction of the cost new due in large part to that forum.

Bottom line, this is an awesome project that one needs to check out. Feel free to submit your own home labs in the forum and they may get featured on the main site like this and the 50TB VMware lab.