At OCP Summit 2024, we saw a system that we have seen a few times but have never covered. The HPE Cray XD670 is a NVIDIA HGX H100 or H200 8 GPU system. The version on the show floor uses CoolIT liquid-cooling as well. Still, if it looks familiar, we might know why.
HPE Cray XD670 NVIDIA HGX with CoolIT Liquid Cooling Shown
The unit that HPE brought was the liquid-cooled version of the XD670, but there is an air-cooled version as well. The XD670 has the GPU tray on the bottom and the CPU on the top along with the storage and many of the expansion slots.
This is a better design than the Dell PowerEdge XE9680 since it has a directly serviceable NVIDIA HGX H100/ H200 tray. Unlike the Dell it requires one to pull the entire chassis out of the rack to replace GPUs, the HPE has a tray with liquid cooling blocks on each GPU.
The design with the rails and the tray is also similar to the ASRock Rack HGX H200 platform, albeit this is a shorter 5U platform. We will show off the ASRock Rack version soon. Still, the three systems share this bottom HGX tray and latch design. In this HPE server, the liquid tubes come out of the front of the chassis.
Along with the GPUs, the Intel Xeon CPUs are also liquid-cooled using CoolIT liquid-cooling blocks. The blue motherboard is a giveaway that this is a Gigabyte system. Most HPE ProLiant servers and many Dell PowerEdge servers use Inventec motherboards, which is a bit different. Many Cray designs have used Gigabyte and Supermicro over the years so seeing Gigabyte here is not unusual.
Looking at the back of the system, we see six power supplies, multiple low-profile expansion slots that are CPU-connected on top, six fans are in the center. We then have eight low-profile slots on the bottom in two removable sleds.
This is still nowhere near as dense, sophisticated, or serviceable as the Supermicro 4U Universal GPU System that we showed Inside the 100K GPU xAI Colossus Cluster. Still, it is a functional system that is a step above Dell.
Final Words
At this point, some may wonder why we say this looks a lot like the Gigabyte G593-SD0. Here is a look at that system. About a year ago, HPE was the third largest customer for Gigabyte for this system.
Overall, it was cool to see the HPE Cray XD670 liquid-cooled server at OCP Summit 2024. With SC24 a few weeks away, we hope HPE also has some new hardware it is readying for the show.
Slightly awkward sentence structure here:
“Unlike the Dell it requires one to pull the entire chassis out of the rack to replace GPUs, the HPE has a tray with liquid cooling blocks on each GPU.”
So is the Dell better, or the HPE better? Because it sounds like the HPE requires you to pull the entire chassis out, but then you go on to explain that the HPE has a tray for the GPUs.
Speaking of ODMs and subcontracting; I was a little surprised by how prominently “CoolIT” was presented in the design. Do they have sufficient presence/reputation for not leaking/a de-facto standard for fittings at the rack level that customers specifically want them rather than whatever ODM HP feels like farming out to?
It’s obviously not uncommon for various supplier brands to not be entirely scrubbed(fan labels, PSUs, sometimes some embossing on a CPU socket or heatsink); but they tend not to feature quite as prominently as the CoolIT ones do here.
James – HPE’s HGX tray slides out to service on rails. Dell’s is perhaps the worst in the industry since it requires pulling the chassis out of the rack and disassembly to get to the HGX tray.