About a month ago, we published a short preview piece on the ASRock Rack D1540D4U-2T8R motherboard and today we have the full review. We spent significant time loading the motherboard with different configurations since it has so many onboard features. The result, an excellent Xeon D platform that is envisioned differently than many of the platforms currently on the market. The ASRock Rack D1540D4U-2T8R, and we would imagine stable mate D1540D4U-2T2O8R, are not meant to be as compact as possible. Instead they trade an extremely compact PCB for a mITX size. With that size, the boards are able to provide more features than smaller counterparts. Whereas many Xeon D motherboards on the market today are aimed at compact edge devices, the ASRock Rack D1540D4U-2T8R is geared more towards a traditional storage server market. That differentiation makes the platform fascinating.
Test Configuration
We used a similar configuration as to the one we used on other Xeon D platforms we have used. Due to the expanded capabilities of the ASRock Rack system, we added a few unique pieces. We also varied our configuration to try out a few potential build options.
- CPU: Intel Xeon D-1540 (embedded)
- Motherboard: ASRock Rack D1540D4U-2T8R
- Memory: 128GB – 4x Samsung 32GB DDR4 2133MHz ECC RDIMMs
, 64GB – 4x 16GB Micron DDR4 2133MHz ECC RDIMMs
- SSD: 6x Intel DC S3700 400GB, 2x Toshiba PX03 1.6TB SAS3 SSDs, 2x SanDisk X400 512GB m.2 SSDs
, 4x 4TB WD RE Nearline SAS hard drives
- Operating Systems: Ubuntu 14.04 LTS, Windows Server 2012 R2
We used a basic configuration for this review however we are working on a much more substantial build. It occurred to us that on our 10Gbase-T networks this system could be fitted with two additional HBAs and turned into a great 4U 24-bay storage machine. We are decommissioning one of our Intel Xeon E5-2670 V2 servers with add-on NICs and replacing it with this motherboard. Our expectation is that we will see substantial power efficiency gains. Most of our systems are now running SFP+ or QSFP+ networking. ASRock Rack does make variants of this board with dual SFP+ 10Gb Ethernet ports as well (which we hope to test soon.)
ASRock Rack D1540D4U-2T8R Overview
The ASRock Rack D1540D4U-2T8R is based on the ASRock Rack D1540D4U platform which is unique platform insofar as it is the only mATX Xeon D platform we have seen thus far. ASRock has motherboard options available with 10Gbase-T networking, SFP+ networking, LSI SAS3008 controllers, and without any of those advanced features. This larger form factor means that ASRock Rack has numerous variants to hit different price points and feature requirements. The ASRock Rack D1540D4U-2T8R we are testing is a mATX form factor board has dual 10Gbase-T ports and a LSI / Avago / Broadcom SAS3008 controller onboard.
The onboard Intel Xeon D-1540 is an 8 core/ 16 thread SoC that utilizes the same Broadwell instruction set as the newly launched Intel Xeon E5-2600 V4 processors. Here is an example of the performance of this SoC on the ASRock Rack platform in Linux-Bench. The chip has as 2.0GHz base and 2.6GHz turbo clocks. It also has a 45w TDP. Intel has introduced a slight update with the Intel Xeon D-1541 which bumps those figures by 100MHz. There is also 12MB L3 cache. Our sense is that these motherboards will be updated with the Xeon D-1541 part in the near future. With that said, STH has a cluster of Xeon D-1540 machines running Ceph and KVM and there is no reason to upgrade them to D-1541 at this point.
In terms of airflow, an important aspect of these Xeon D platforms, the CPU heatsink is an active model. It does look similar to the unit we saw in our Taipei visit to ASRock Rack in October 2015. The main difference seems to be that this is a 3-pin fan whereas the previous version we saw had a 4-pin PWM fan. We would like to see ASRock Rack bring the 4-pin solution back as small fans spinning quickly do generate significant noise.
One can also see that all heatsinks, the two power connectors, 4x DDR4 DIMM slots and other components are laid out for traditional front to back server airflow. The four DDR4 DIMM slots allow for a maximum of 128GB DDR4 total using 32GB RDIMMs.
Using the mATX form factor one gets both a PCIe x16 and x8 slot (both PCIe 3.0 x8 electrical). Our next build with this board will use these slots with HBAs for bulk Ceph and ZFS storage disks which we are very excited about. This is truely the power of the Xeon D platform as having this level of onboard components as well as two PCIe 3.0 x8 electrical slots onboard a Xeon E3 platform is not possible.
Near the PCIe slots, the platform also has provisions to add storage via two m.2 ports for SATA storage using a Marvell 9172 controller. This is a great way to add boot SSDs without utilizing the onboard SATA or SAS3 ports. You can see the small Marvell controller near the bottom m.2 connector.
Beyond the two Marvell based SATA ports we can see six SATA III ports based on the Xeon D SoC. The Xeon D has an onboard PCH that supports six 6.0gbps SATA ports.
Next to these ports we can also see a USB 3.0 Type-A header for an internal USB key. These are popular for ESXi and Nutanix installations as an example.
The ASRock Rack D1540D4U-2T8R additionally has a LSI/ Avago/ Broadcom SAS 3008 controller onboard. This provides 8x SAS3/ SATA III ports via the HBA. The controller is an extremely well supported HBA and one that we recommend at STH. There are less expensive alternatives out there, but the broad software support for the SAS 3008 makes it an excellent choice.
Taking stock of storage, even without PCIe add-in cards or SAS expanders, one can have a system with: 8x SATA III/ SAS3 drives (SAS 3008), 6x SATA III drives (Xeon D SoC) and 2x m.2 SATA drives (Marvell 9172) for up to 16 drives total. That is great on a mATX platform. Furthermore, one can expand the platform with two additional PCIe cards. In comparison, the Supermicro X10SDV-7TP8F we reviewed has more storage ports onboard, but does not have any high-speed SAS3 ports that you would want if you were using SAS3 SSDs or SAS expanders.
The rear I/O will vary based on the model. One can see that there is a dedicated RJ-45 Ethernet port above the two USB 3.0 ports providing out of band management access. The two legacy headers are a DB9 serial header as well as a VGA header. Finally there are two 10Gbase-T network ports providing networking for the motherboard.
Regarding the 10Gbase-T ports, we did notice that they are using the Intel X540 dual port 10Gbase-T controller, not the onboard Xeon D MAC. The X540 is ubiquitous and still has better support than the Xeon D’s X557 so this is a welcome addition.
The PCB is meant to be configured for different models so we can see why this decision was made. On the PCB one can clearly see the portion of the PCB that would otherwise house the SFP+ connectors and PHY.
It appears as though ASRock Rack made the decision that the SFP+ ports would utilize the Xeon D’s onboard MAC while the 10Gbase-T ports would utilize an add-on Intel network controller.
ASRock Rack D1540D4U-2T8R Thermal Imaging and Power Consumption
To capture these images we utilize our FLIR Ex series professional thermal imaging camera and turned off FLIR’s MSX enhancements so we can see components outlined clearly. We put the system under 100% CPU load for a period of 24 hours to let “heat soak” set in prior to taking the images.
We can see the key hot spots being the AST2400 controller, the X540 for the 10Gbase-T and the SAS 3008 controller. The CPU fan kept the Xeon D SoC extremely well cooled.
We used our Extech TrueRMS Power Analyzer 380803 to take measurements at different points of the board’s use.
- BMC only power off – 4.8w
- Ubuntu 14.04 LTS installation screen – 42.6w
- UnixBench 5.1.3 single thread max – 57.5w
- Multi threaded max observed – 99.1w
These results are excellent with the new 45w TDP part showing great performance. If one is careful to select the right power and cooling, these platforms are just about able to fit two systems in a 1A @ 120V lower power density rack with two SSDs.
ASRock Rack Management
ASRock Rack has an industry standard web management interface based on the American Megatrends MegaRAC solution. The ASRock Rack IPMI 2.0 implementation has an array of configuration options and sensor information. One of the more interesting features is the ability to graph sensor information over time. This is the first implementation we have used the live widget with. You can get a live graph of items like CPU temperature, fan speeds, voltages, and PSU health.
ASRock Rack includes remote KVM-over-IP functionality. From what we understand, 80% or more of servers are now supporting IPMI 2.0 and some sort of KVM-over-IP solution. The reason is simple. In a data center, KVM-over-IP saves companies a lot of time by allowing remote administration instead of having to physically access the server. Companies like HP, Dell, Lenovo and Intel are charging more for full KVM-over-IP functionality, but ASRock includes it with the motherboard. That is by far the preferred way to bundle the functionality.
The ASRock Rack Remote management solution provides the remote administrator with the information they need to manage servers far away from the home office.
Conclusion
The ASRock Rack D1540D4U-2T8R is an awesome platform because it can provide high speed networking and excellent storage and expansion capabilities that other smaller offerings cannot match. We are excited for this evolution of the Xeon D platform as it is truly a competitor to the larger Xeon E5 systems. The 16 onboard storage ports, two PCIe 3.0 x8 electrical slots and onboard dual 10Gbase-T networking provide a solid foundation for a storage server with room to expand. We do hope to test the SFP+ equipped version of this board as it has been a very highly requested review at STH.
Given that the SoC comes with 32 PCIe lanes total there must be a trade-off somewhere, right?
Would the PCIE slot support a graphics card, all other things being equal?
The PCIe 3.0 x16 slot is only PCIe 3.0 x8 electrical
We did not test this however I would suggest that there are better options for gaming or machine learning if you were looking at either of those two GPU cases.
My interest in adding a graphics card is for using Remote FX on Hyper-V. This requires a GPU with at least two g ram. This will enable remote workstation use of virtualize applications.
Great review on a very competitive product! This thing will do as well as much more costly servers for a fraction of the cost and TDP from what I have read here. An SoC/embedded Xeon with 16 threads and all the features of a decently specced server board for under $2000…. I can see these doing very well in mass deployment environments as well as higher end home server scenarios. Great product ASRock!