Today we have an article that stems from a personal project. Ever since I first saw the CS381, even before our SilverStone CS381 review, I wanted to turn one into a ZFS storage server. Usually, these types of builds are extremely easy. This ended up being harder precisely because of the mATX form factor. In this article, we are going to talk about the thought process behind the build, and what you can take away from it. The particular system was planned as a FreeNAS build, but we are going to let this system run TrueNAS Core starting with nightly builds. You can apply everything here to FreeNAS today, or we expect TrueNAS Core in a few months as that is launched.
Building a FreeNAS TrueNAS Core ZFS mATX Appliance Video
Since this is one where you may want more angles, we have an accompanying video to this article.
For more information, including part numbers, we have the rest of this article.
Getting Started: Choosing the Form Factor
The SilverStone CS381 is a chassis we first saw from SilverStone at Computex 2018. It promises an 8-bay storage platform for mATX servers with large (quiet) fans and room for features such as GPUs.
During our review, we found that there were a few updates, but that system still works exceptionally well.
Aside from the internal 3.5″ drive bays, the platform has access to 2-4 additional 2.5″ mounting points for SSDs, making it a great hybrid platform.
As we saw in our FreeNAS Mini XL Plus Review 8-bays and 10GbE, the 8-bay FreeNAS platform works very well, so we wanted to replicate in a system with larger fans. One major note here is that the CS381 is a 28L design measuring 400mm (W) x 225mm (H) x 316mm (D) or 15.75″ (W) x 8.86″ (H) x 12.44″ (D). It feels like a large system even though it is based around a mATX size motherboard.
Motherboard and CPU Selection
This was a surprisingly tricky system. On the consumer side, there are plenty of mATX options out there. On the server-side, mATX, once a popular size, has become significantly less so. Since this is designed to be a server, we wanted something with remote management capabilities as well.
Intel Atom C3000 Series Option
The original thought was to go with an Intel Atom C3000 series solution. We could get 16 cores, 10GbE, and have a very low power server.
One of the biggest challenges here is that the Atom C3000 platforms are generally mITX or FlexATX size. As a result, they can be handled in smaller form factor chassis, and one does not need the larger CS381 size chassis. For some context, in the 8-bay ZFS server world, the Atom C3000 series is probably the best-fit solution from a SoC perspective, but we needed the right platform. We could, of course, use a smaller platform in the CS381, but we wanted to use mATX.
Perhaps Bigger: Intel Xeon Scalable Solution
If we could not get an mATX Atom C3000 solution, the next thought was going big. Specifically, getting an Intel Xeon Scalable solution. We have dozens of Xeon Scalable CPUs to use, so this seemed like an easy option.
We reviewed the Supermicro X11SPM-TPF mATX Intel Xeon Scalable Motherboard, and it seemed like a perfect fit. It had SFP+ 10GbE, multiple expansion slots, enough onboard SATA to power all of the drives, 6x DIMM slots, and plenty of PCIe all perfect to go bigger and do a converged storage and virtualization platform.
Then we found a snag. Due to the positioning of the CPU socket and memory, we could not get an easy and quiet cooling solution for the Xeon Scalable CPU. We would effectively need a liquid cooling solution. The closed-loop solution on the market for the LGA3647 120mm fan applications is the Dynatron L13. We tried for two months and were unable to buy one. That would have worked and made for a unique solution; alas, it was not available. With LGA3647, one affixes the CPU to the heatsink then installs the assembly into the socket. As a result, other liquid coolers that would fit the chassis do not work on LGA3647. We could have used a lower profile solution, but I did not want a loud NAS.
Intel Xeon E-2200 Series
This was perhaps the most logical option. There are a lot of Xeon E-2100/ Xeon E-2200 series platforms on the market, especially in mATX. Cooling is not an issue here since there are many low power options.
The challenge here is that these chips felt a bit like no-mans land. We know that the eight high-speed cores offer excellent performance, but it is not exactly inexpensive, it is not exactly low power. We do not get RDIMM support for higher capacities for a server that could use that much power with enough memory. Maybe it was the fact that we had just done over two dozen pieces on the series, but I just did not feel this was the right fit for me especially given the lower-cost pre-built solutions out there based on Xeon E-2200 series.
Still, this is one that we think you should look at. In the space, mATX options were more popular until 2015 or so when Intel started introducing single-chip embedded solutions (Xeon D-1500, D-2100, and Atom C3000.) Removing the PCH from the motherboard greatly reduced motherboard footprints. It also dropped volumes for the server entry Xeon series. As a result, we saw the number of mATX platforms here shrink. Still, this is probably one of the most vibrant mATX ecosystems. The Supermicro X9SCM-F was an absolutely killer platform I look back on fondly.
Let us continue by looking at some other options.
They should sell a FreeNAS MiniXXL with this setup for those that need more than Atom but still want a smaller FF.
Stop trying to make me subscribe to watch your content is stupid and doesn’t work on mobile
We do not have subscribe to watch filters. Everything is on YouTube or the STH main site without a registration wall.
Any performance issues with semi-disregarding the generally accepted wisdom/paradigms of the TrueNAS/FreeNAS gurus of recommending 1 GB of RAM per TB of storage? (Any benefits at all of running 128 GB over 64 GB?)
@Mark Dotson,
that 1GB RAM per 1TB storage is only partially true. Look at the offical hardware requirements/recommendations for FreeNAS: https://www.freenas.org/hardware-requirements/
In short (and not considering any RAM requirments of any other services running on your FreeNAS box), 8GB RAM is minimum and sufficient for 8 drives. Then, for any additional drive, an additional minimum of 1GB per drive is recommended. Note that those requirements/recommendations are not based on a RAM-per-storage-capacity metric, but rather based on RAM-per-drive.
Note that additional RAM can and will be utilized by the ZFS ARC for increased performance. Like with any other cache, how much of a performance gain a particular cache size will bring depends significantly on the usage scenario(s) that define the cache access patterns and the tiered cache hierarchy configuration (ARC and L2ARC).
What a coincidence!
I’m just waiting for delivery of the last four disks. Should arrive in the next hour or two.
My build:
CS381
450W Seasonic FOCUS SGX, Full Modular, 80PLUS Gold
X11SSH-CTF
E3-1260L V5 (also 45W, less cores than the D-1451, but higher per core clock)
64GB ECC
8 * DC520 (mix of SE/ISE,SAS/SATA so they’re from different mfg batches)
PMC NV1604 Flashtec NVRAM (if I can ever get it to work – it’s been EOLed so no control s/w) :(
Another option for the motherboard would be the Intel D-21xx based boards. Higher TDP but it has better performance per core. You make up for that with future expansion options or more flexibility now however when considering the boards it’s used on like X11SDV-8C-TP8F from SuperMicro. Lower price and you get 2xSFP+, 2x10Gbase-T, 4x1Gbase-T, Satadom, M2-nvme, pcie-x8 and x16 slots, and its VMWare certified. Perfect for a freenas, soho/lab fw/router, and room for a home automation guest or other light weight options.
Im running the X11SDV-8C-TP8F in the CS81 chassis, adapting a CU cooler to fit was the most difficult part of the build as the CPU sit partly under the drive trays.
How do you like it so far? I just finished building mine. Using a Plink web2208 rackmount case. Still waiting on new drives but I threw my old ones in it and booted it about 30 minutes ago. Havent had time to get esxi on it yet. Project for tomorrow.
Dr. Madness said: “8GB RAM is minimum and sufficient for 8 drives.”
What is your source for this info regarding very RAM amounts?
Perhaps someone should advise should advise their TrueNAS/FreeNAS/ZFS experts they’ve had it wrong for years…
They (the TrueNAS folks) seemed pretty clear in their recommendations for RAM amounts being related to 8 GB min, then add 1 GB of RAM per TB of storage capacity in Terrabytes, and not simply the ‘number of drives’, or, ‘add one GB for each drive above 8’ (which no one has EVER uttered/implied in the TrueNAS forums in the 4 years I’ve been reading them)…; and I’m reasonably certain there would logically be more RAM overhead involved in managing /correcting issues with 8 each 16 TB drives vs. 8 x 80-100 GB drives from 12 years ago, for instance…
EDIT: I do see Dr. Madness’ recommendations now listed officially as the minimal RAM recommendations on the TrueNAS page….; seeking clarification as to why /when the recommended was so drastically lowered… (Certainly makes TrueNAS a more affordable solution, hardware wise, if only 8 GB or 16 GB is sufficient now instead of 128 GB)
Do you need a storage controller that support JBOD for ZFS? Or create 8 RAID-1 arrays on the LSI controller?
1GB of ram / TB is only if you plan on using dedupe. I’ve got 128GB of ram in a box with 96TB Usable (144TB RAW), however that is overkill (I got a GREAT deal on the RAM). When I was running 64GB it was more than enough as well, my machine never struggles. IF you run DEDUPE, then yes it’s recommended, otherwise 32-96GB (depending on what you want to do with the box)
It’s more cost-effective to buy a complete server these days. I had actually considered building my own but ended up buying an HPE server and building around it.
An HPE ML110 Gen 10 with an 8 core Xeon Silver 4208 and 16gb of RAM is only $850:
https://www.provantage.com/hpe-p10812-001~7HEWY349.htm
STH review: https://www.servethehome.com/hpe-proliant-ml110-gen10-review/
Another 4 bay LFF drive cage is $105 (comes with cabling):
https://www.provantage.com/hpe-869491-b21~7HPE931F.htm
Another 16GB RDIMM is $175 (I know you can get memory much cheaper, but for this exercise I’m using real deal HPE labeled memory):
https://www.provantage.com/hpe-815098-b21~7HPE92F8.htm
You can get 8 real HPE drive sleds on eBay for around $60.
https://www.ebay.com/itm/LOT-OF-4-651314-001-HPE-TRAY-FOR-3-5-SAS-SATA-DRIVE-TRAY-DL160-DL380p-G8-G9/133331123957 (I have actually bought from this vendor before)
Likewise, you can get iLO Advanced on eBay for $30:
https://www.ebay.com/itm/HPE-iLO-Advanced-License-LIFETIME-GUIDE-2-3-4-5-ALL-Servers-FAST/253721355479
If you don’t want to use the built-in SATA ports, I bought a Broadcom/LSI 9400-16i HBA for $275 on eBay, though of course older cards are even cheaper.
Apples to apples, that’s $1,220, or just a tad under $1,500 with an HBA, for a much better system with a warranty and support. Even if you paid full retail for iLO Advanced instead of getting it on eBay you still come out way ahead.
I also just built a FreeNAS box with a D1541D4U-2O8R. For me it was the excellent price and feature set of that board that was a no brainier.
I originally planned on going with a C3758 based board, but there was none with SFP+ ports other than the A2SDI-H-TP4F for more than 2x the price.
I ended up getting a tiny 128 GB m.2 2230 for $30 as a boot drive.
I went with 4x16GB DDR4-2400 RDIMMs. It seems to be true on this board that if you only populate 2 RDIMMs the ram only comes up as DDR4 2133. (Still waiting on the extra 2 sticks to verify)
The only thing that i haven’t decided is if I need a L2ARC, and do i go another m.2 nvme (only PCIe2 1x) or SATA SSD?
I built this same setup for my first NAS. I purchased the MB from Newegg for 299.00 for a steal with only 32gb of ram. It runs my plex and unifi software with ease. Next step is more ram and surveillance software. Great article and now I can show some friends. They said it was overkill, but that’s what we do right?
I’m thinking of doing a build in this case myself, but I’m concerned about the HDD temps. Does anyone know how high they are during normal usage?
The Silverstone CS381 seems like a decent case. Better airflow than most, especially if you need the hot swap drives.
I prefer the Fractal Design Node 804. More and larger fans. It can still take full size PCI cards and full size ATX power supplies. It puts the board in a separate airflow pattern than the power supply and drives. A bit unique on form factor since it’s a big (almost) cube.
https://www.fractal-design.com/products/cases/node/node-804/
What a joke, NAS with $1300 motherboard but the HDD shucks…
Did you flash the LSI 3008 into IT mode for this setup?
Hello I’m planing my first FreeNAS build. I’m colorist and need a shared storage solution to work direclty out of.
I’ve been planing a build around the AsRock X570D4U-2L2T, however after reading this article I found a really cheap D1541D4U-2T8R on ebay and was wondering if it would be a better option to get that instead, however it’s a really old design and I don’t know if it makes sense to buy that in 2021…
What would you recommend?
Did you ever measure the fan noise? How did that turn out?