Selecting Ultra EPYC Components
As much as we love Sun, 4GB of RAM is not going to cut it. The 530W PSU is not going to help when we have 200W+ CPUs, 250W+ GPUs, and many other components. It was time to pick the components to make something truly, Ultra… EPYC.
Ultra EPYC CPU: AMD EPYC 7371
Current generation AMD EPYC 7000 “Naples” generation CPUs top out at 32 cores. While we have plenty of AMD EPYC 7551P and EPYC 7601 chips in the lab, they suffer from lower clock speeds. As a workstation, one really wants higher clock speeds if possible. We could have gone with an AMD Ryzen Threadripper 2990WX 32-core part, but that would have limited our memory capacity. Plus, this is a Sun Ultra that we were turning into an Opteron-worthy successor.
Enter the AMD EPYC 7371. You can read all about the chip in our AMD EPYC 7371 Review. Price wise, the AMD EPYC 7371 is fairly reasonable at $1550. It is capable of dual-socket operation, however, we are constrained with an ATX case. ATX and even EATX struggle to fit two modern server CPU sockets.
With that said, we are using a 205W TDP CPU here. The Intel Core 2 Duo we are replacing is a 65W part.
Ultra EPYC Motherboard: Gigabyte MZ01-CE0
We are going to have our Gigabyte MZ01-CE0 review coming soon. For now, let us just say it is an AMD EPYC platform that is designed to work well even in an ATX form factor.
The MZ01-CE0 has spaces for four PCIe x16 cards, even with double-width coolers. The solution even has 10GbE built-in which is exciting for a workstation. To fit the ATX form factor, the Gigabyte MZ01-CE0 cannot fit a full 16x DIMM slots so one gets 8x DIMM slots, one to fill each of AMD EPYC’s memory channels. The Sun Ultra 24 had gigabit Ethernet which was a leading feature in 2007.
Ultra EPYC Power Supply: EVGA SuperNOVA 1000W G3
When EVGA got wind of this project, they offered to send an EVGA 1000W G3 power supply. The modular cabling will help since cable management in 2007 was not as robust as today.
I also use an EVGA SuperNOVA 1000W G3 model in my personal Threadripper workstation, so I was happy to use this one. One of the nice things that these PSUs have is the ability to shut off the fan if the load is too low. Being able to have a cache of the same types of cables is also nice.
Ultra EPYC GPU: NVIDIA GeForce RTX 2080 Ti
This one caused a bit of a conundrum. What GPU to use. The NVIDIA Quadro series has gone up in price quite a bit since 2007, so a Quadro RTX was out of the budget. Since we started this build before the new Titan RTX was launched, we are using the NVIDIA GeForce RTX 2080 Ti.
We came close to using the AMD Radeon Vega Frontier Edition in this build, but since we are replacing NVIDIA’s first generation CUDA card, it seemed reasonable to stay NVIDIA.
As a quick note, the outgoing NVIDIA Quadro FX 1700 was a 42W TDP part, we are replacing that with a 250W RTX 2080 Ti. That is an enormous leap.
Ultra EPYC CPU Cooling: Noctua NH-U9 TR4-SP3
Since the Sun Ultra 24 does not have radiator mounts, we needed an air cooler. Further, this is not the largest case, so we needed to find a 4U air cooler for a 205W TDP AMD EPYC.
I re-read William’s Noctua NH-U9 TR4-SP3 AMD EPYC 4U Cooler Review and immediately ordered one from Amazon.
This was probably the biggest configuration miss that we had, but frankly, there are not a lot of great options to remedy in this case. We are going to show why during the build.
Ultra EPYC NVMe Storage: Intel Optane 905P 380GB M.2 SSD
The system has four hot-swap bays. We even ordered an extra four 3.5″ trays just to match. Still, for basic storage, we wanted the best. You can get fast and big storage, but Intel Optane 905P 380GB M.2 NVMe SSD review shows why it is an amazing part.
The M.2 SSD fits directly onto the motherboard port. We also are using an EK cooler that came bundled with the drive. In a server, there is enough airflow. In this workstation, we were worried.
Ultra EPYC SAS3 Storage: Broadcom HBA and Micron 3.2TB SAS3 SSDs
The donor Ultra 24 had a SAS/ SATA backplane. We decided to use it for this build, even though we would recommend against it. In fact, this is the part of the build that we would strongly urge our readers to not follow. We used a Broadcom SAS 9340-8i HBA which is our standard IT/IR-mode HBA. We are using the Micron S630DC 3.2TB SAS3 SSDs for 12.8TB of SAS storage.
This is a part of the build that can be left out for the better. The Micron S630DC 3.2TB drives run too hot. One can use WD coolers to attach them but we had issues there as well. The older SAS/ SATA backplane negotiates at 6gbps SAS2 speeds, but when we add a cooler to keep the SSDs cool while in operation, the links negotiate down to SAS1 3.0gbps speeds. That is absolutely silly.
We could have run large hard drives, but this is 2019, hard drives belong in network storage. They are loud and too slow to sit next to you in a workstation.
Ultra EPYC RAM: 256GB in 32GBx8
The RAM side was easy. 64GB DIMMs are out of our budget. We have 8x DIMM slots. That means 8x 32GB DDR4-2666 RDIMMs.
With AMD Threadripper, 128GB is more of a realistic number using 8x 16GB unbuffered DIMMs. Since this is a high-end workstation, we are using 32GB RDIMMs. That is a great upgrade over the 4GB circa 2007. We also had an 8x 16GB 128GB ECC RDIMM configuration we tried which works well in cases, for example, when you need to put together 48x 32GB RDIMMs for a single node in the lab (review on that one coming soon) and that is all you can spare.
Ultra EPYC Networking: Mellanox 100GbE and EDR
We tend to use a lot of Mellanox ConnectX-3 Pro EN 40GbE cards. They more or less have invaded our labs with only a few Intel and Chelsio 40GbE adapters in the mix. Since this is the Ultra EPYC, why not utilize a network standard that is 100x faster than the 1GbE the system came standard with 12 years ago.
We are using Mellanox network cards almost everywhere these days for our 100GbE networking. The Mellanox VPI cards can handle either 100GbE or 100Gbps EDR Infiniband duties, making this more flexible than one may have assumed. This is a challenge to cool. Using a 40GbE NIC would be much easier to keep cool in this type of case. We wanted to use the extra PCIe x16 lanes that the motherboard has, so that made a lot of sense.
Ultra EPYC: Beauty Mark Update
We could have left the “beauty mark” of the CD/ DVD drive alone and hooked it up via SATA. Instead, we decided to update this to a more modern USB 3.0 hub.
Of course, we found a black faceplate to keep the aesthetically displeasing and defining feature of the Sun Ultra 24 intact. This Kingwin KW525-7U3C looked like it was a worthy upgrade to the optical drive since it allows for an array of USB 3.0 thumb drives. Also, the Gigabyte MZ01-CE0 does not have audio, USB 2.0 and firewire headers galore so one has to make due with a single USB 3.0 front panel header.
Now, let us take a look at the final Ultra EPYC.
This article was so well done. It was cool to get a history lesson as well.
Hey you missed that the case has thumb screws 10 years ago.
We ran datacenters full of Sun gear. They were arrogant but they also made so much technology we still use today.
Thanks for the article. I enjoy these restomod builds. I personally fit modern running gear into old SGI chassis. I currently run a Threadripper in an SGI Fuel chassis and have an Octane ready to go an ITX build with Ryzen 3000 when it is released.
I’m impressed with how accessibel this is. Bend one part. break something else and you’re using EPYC in this old case. It looks like your power button isn’t working either but you’ve got IPMI so you can reboot from your phone if needed.
A $1550 CPU with 8ch ECC RAM?
100Gb, 10Gb, 1Gb NICs?
12.8TB SAS array and Optane?
This is a Sun server not a workstation.
Great PSU tips on p5
It’s not a server, because it does not have dual-psu and rack-case :)
please do a build video if not already done :)
awesome work.. btw :D
The beauty mark comment was spot on. You’re showing why Sun was in decline and about to be sold to Oracle.
Ultra 24? You should have used epyc 7451 to match the number 24 with cores ;)
What’s next, Patrick, a red sports car?
I’m not gonna lie. STH is usually like the authoritative encyclopedia of content and this is totally not.
But you’ve 100% nailed your demographic with this build. No RGB lights and a Sun logo. I can’t remember the last time I’ve read more than 2 or 3 pages on an article STH or other but I couldn’t wait to see more word for word.
I just did a search for other Ultra 24 builds and there aren’t many. Maybe you’ve started something? I’m not going to cut sheet metal for a build, but bending a few bits I’ll do. I can’t wait until my interns this summer see a Sun box under my desk at work.
That AMD easter egg was a 1 in a million find too as others pointed out.
“Sun Ultra workstation families that are powered by the latest multi-core AMD… processors, and run virtually any OS”
Fixed with EPYC by STH Mr. Fowler. #IRememberSun
I love the thought process. You’re using IT SAS like SUN would with ZFS – no SAS RAID for that.
Most people that do these are either gaming people or gaming people doing servers. It’s fun to see something that’s thoroughly modern with a nod to the past.
That’s a mean pedestal workgroup file, VM and deep learning server too.
Really nice, I worked on Sun workstations for many years and have found memories of Sun and Solaris. Solaris was unbreakable Unix, too bad Oracle abandoned it.
Changing gears, man, if you really NEED such hardware in your workstation you must be doing some awesome stuff – and doing it all at the same time.
Nice – I’d pop the black faceplate off and get rid of the white writing. Every port is USB3.0 so why label it? The white writing is a bit of a distraction, but then again I like discreet things.
Having sentimental memories of a 1990 Sun Sparc Workstation that we used to get orbital parameters, if I recall correctly Byte Magazine at the time reviewing the 20MHz RISC based CPU making note of how a spark is the last thing a user would want associated with their computer, thus odd naming selection, am happy if not excited to see this effort to revive a mint condition, classic upright Sun workstation case with new innards.
Actually, Oracle still supports Solaris and releases new versions every now and then even though there are not as many engineers working on it anymore.
Vlad, Solaris has been abandoned. No roadmap, just minor fixes.
Well done! Would love to see a rebuild of a Sun Ultra 40 (M2)! The old AMD version before the Intel switch and the “beauty mark” but very different hardware layout (custom psu).
I love the Sun workstation enclosures. If only someone would make generic enclosures that looked like that.
Does anyone know if an EATX board would fit?
I’ve been lurking at this site for a while now, and I really do appreciate the writing. There is one point of criticism though that if you would correct would be appreciated especially when one is doing a build. That item would be the cost breakdown of the build.
did you put solaris on it?
where are you finding these machines for 100? 395 is the cheapest on eBay, please let me know where i can pick a few up for a cool Benjamin!!!!!!
Hello:
Great job! Very neat.
I also have a U24, got it three years ago but for ~$300.
It’s really great hardware, even by 2019 standards.
I was wondering: could you tell me what BIOS version your “barn find’s” motherboard came with? =-)
Thanks in advance,
JHM
I just finished the same swap on my Sun Ultra 20. I had to slightly file the holes for the PSU because of a slight misalignment, about 1/16″. Otherwise, it just dropped in. I’ll be replacing the IDE DVD Reader with a SATA DVD RW
The Sun Ultra needs an upgrade to 2 x AMD EPYC 7742 :D
1) Can you put a fan BEHIND the drive cage or is there metal that comes down which would render that idea useless?
2) I’m surprised that your 100 Gbps Mellanox cards are only single port and not the dual port variant, especially given the fact that they’re PCIe 3.0 x16/x16 cards being inserted into a PCIe 4.0 x16/x16 slot.
Either way, pretty neat though. It’s a nice looking box. Oracle should totally bring these back.
Got my hands on a 24 Ultra through ewaste today. While the reason for why somebody would dispose of such a beautiful piece of work is beyond me, especially with ebay listings for it going wild, I really am not complaining.
There are some problems ahead, but I am absolutely stoked to bring life back into this machine. Today was an awesome day lol.