Build #2: DIY Mega Workstation
Taking the above build, we added the Intel Xeon w9-3495X 56-core processor, the current top-end.
Here is a fairly clean setup, albeit we added WiFi to make it easier to run on set.
With that processor, most will likely use a 256GB or 512GB configuration and we used 64GB ECC RDIMMs that we had sitting around the lab.
We changed the cooler to a 120mm Noctua.
This changed the airflow direction. Still, this was plenty to handle the 56-core Xeon.
On the GPU side, we had the high-end NVIDIA RTX 6000 Ada.
Even with all of this, there are a ton of slots available.
Performance-wise, out of the box one can get 70,000 on Cinebench R23, but that is more of a baseline. Tweaking the power level, adding mild overclocks, and so forth can increase these figures by a fairly large amount.
Running through the Linux-based suite we see fairly large gains. At bone stock, it is similar to the Supermicro liquid-cooled workstation (also stock settings) in terms of performance, but well above the Lenovo ThinkStation P7. The Lenovo P7’s air cooling is not as good as Noctua’s and so Lenovo is running its chips at the lower-end of Intel’s power spec range.
We ended up using a Kioxia CM7 for a PCIe Gen5 NVMe SSD in this as a data SSD, but it missed our cut-off for being included in the video. Of course, there is a ton of connectivity to go faster in this build.
DIY is great, but what about if one wants to go faster? That is the third version.
Can you undervolt these cpus? The power consumption is limiting them too much.
@Makis
There are options to apply negative voltage offsets in the BIOS of this board but I suspect that these settings are being ignored in actual operation because power consumption still remains relatively high after applying such offsets.
Despite the high power consumption, these CPUs aren’t typically being thermally constrained, their performance will not benefit from undervolting.
@Patrick
Did you have trouble cooling the memory during more stressful workloads? I’ve got the same platform and ended up having to get water blocks for all the DIMMs to keep them from overheating.
Also I’ve noticed a huge discrepancy (~30%) in performance between running the same FEA workload on Windows vs Linux; I’m thinking its scheduler related.
Was there a reason for having the Noctua cooler blowing down onto the GPU instead of up to the top vents? Curious if having it the other way round would’ve been even better.
The QAT situation on these is … there just isn’t one? I am having trouble interpreting the extremely large Xeon feature matrix for this generation.
Impressive choice! Building 3x Intel Xeon W-3400 workstations and servers with the ASUS Pro WS W790E SAGE SE is a fantastic way to achieve top-notch performance and reliability. The ASUS Pro WS motherboard’s robust features and support for multiple Xeon processors make it an ideal foundation for high-end computing solutions. Well done on your hardware selection!
@Patrick
Any idea if there would be boards from Supermicro, ASRock Rack, etc. for these CPU’s?
Would be interesting to see what a slightly more server-like board would be for these.
@wes What is wrong with Supermicro X13SWA/X13SRA ?
You write, “We will quickly note that we had an EPYC Genoa workstation build that our team thought was too cumbersome to really recommend as it was a bear to get working.” Even if not recommended, it would be interesting to hear about the troubles of the parallel Genoa project.
It seems a shame that it has seven PCIe slots but most of them are impossible to use once you put a GPU or three in. So then you use riser cables but GPUs generally fail to fully utilise a 16x slot. Actually maxing this system out for use with HuggingFace AI models would be intriguing but perhaps the hardware to do so doesn’t really exist…?
I am looking at the CPU-Z specs presented in the video at time 00:49, and am trying to figure out why the Intel Xeon w5-3425 (left), the alpha numeric portion not include in the “Name” field near the top of the “Processor” portion of the frame, like it is for the Intel Xeon w9-3495X (right); does not seem to share logic continuity in the “Clocks (Core #0)” portion of the frame, compared to the w9-3495X. The w5-3425, and rated (advertised) at 3.2GHz with a turbo speed of 4.6GHz, being shown to have a “Core Speed” of just 796.00MHz, the “Multiplier” set to only x8.0, given a “Bus Speed” of 99.75MHz, were as the w9-3495X, and rated at 1.9GHz with a turbo speed of 4.8GHz, displays a more understandable “Core Speed” value of 4800.00MHz, the “Multiplier” set to x48, given a Bus Speed of 100.00MHz. Shouldn’t, presuming all settings are basically identical in the BIOS configuration, the CPU-Z report for the w5-3425, with regards to the “Core Speed” read 4600.00MHz, thus consistent with the advertised turbo speed value, the “Multiplier” value set to x46? Please explain what the values in parenthesis mean, the w5-3425 range being from (5.0-32.0), the w9-3495X range being from (5.0-7.0). I would like to see the industry address the legacy (mid 1990’s) “Bus Speed” of ~100MHz-133MHz.
Do these servers support CXL?
but why was the supermico platform faster. you nonchalantly dropped in that comment as if it was no big deal. well it’s a big deal! lol. is there a simple explanation?
Hi everyone,
great that I found this article. However, I faced memory issues during a similar build but with 2 4090 GPUs. I used RDIMM ECC memory (64GB DDR5 4800MT/s ECC Reg 2Rx4 Module KTL-TS548D4-64G Server Memory) and was surprised with d2,b7, and b9 codes, all memory-related errors.
Can you give any hints and what memory was used in your setup?