Dell PowerEdge R760 Internal Hardware Overview
We are going to work from front to back inside this server.
First, we are going to start with the fans. There are six hot-swappable fans. Dell has one of the smoothest hot-swap fan designs in the industry.
What is more, the entire fan partition can be removed from the server via two latches. This is very useful.
Dell has a lot of customized flat cabling and such in the server to route cables through the chassis. Still, it also has connectors on the front portion of the motherboard. Removing the fan partition quickly means that one has access to this as well as servicing front-mounted PERC cards for example.
Here is just another look at how much cabling comes off of the front of the motherboard.
The fans push air through a very nice airflow guide. Dell’s airflow guides are some of the best in the industry.
Those airflow guides push air through the CPU heatsinks and the memory. We showed a size comparison in the video, but it is amazing to see how small of heatsinks Dell is using here compared to what Microsoft uses in its Azure servers of this generation.
In terms of processors, we get dual 4th Gen Intel Xeon Scalable “Sapphire Rapids” processors. That means up to 56/60 cores per socket and two sockets. The linked article above has all of the new features of the new CPUs such as AMX AI acceleration, built-in QuickAssist crypto/ compression acceleration (on some SKUs), and more.
With the new CPUs, we get 8 channels of DDR5, up to DDR5-4800 but that is SKU and population dependent. This server supports 2DPC operation for 16 DIMMs per CPU and up to 32 total. If you want to learn more about 1DPC v. 2DPC, check out Why 2 DIMMs Per Channel Will Matter Less in Servers.
You can also learn more about DDR5 RIMMs and why they are so different in our Why DDR5 is Absolutely Necessary in Modern Servers piece. This is a great example of the DDR5 design trend.
Another big feature of the new platform is PCIe Gen5. PCIe Gen5 for things like storage depends on options, but the new generation allows for up to 400GbE NICs. We will have a 400GbE networking review including PCIe Gen5 cards and a 64x 400GbE switch about two weeks after this review goes live. That is only possible because of the new PCIe Gen5 generation.
With the PCIe Gen5 transition, long motherboard PCIe traces are out. Cables must be used to span long distances. As such, we are seeing server motherboards shrink in depth. That is why this motherboard stops before the rear I/O area. This is going to be more common as the OCP DC-MHS platform takes off across the industry.
Behind the CPU we can see the impacts of this. The main PCIe Gen5 riser slots and riser high-density connectors do not go past the Intel Emmitsburg PCH. Instead, these connectors are closer to CPUs to shorten PCB trace runs.
Here is another view of the OCP NIC 3.0 internal lock slot which sits at the end of the motherboard.
Next, let us get to management and performance.
On BOSS Cards: Previous generations required a full power down and removing the top lid to access the M.2 card to change BOSS sticks. This new design allows hotswapping BOSS modules without having to remove the device from the rack. Very handy for Server Admins.
I’m loving the server. We have hundreds of r750’s and the r740xa’s you reviewed so we’ll be upgrading late this year or next thx to the recession.
Pricing comments standout here.
I’m in disbelief that Dell or HPE doesn’t just buy STH to have @Patrick be the “Dude you’re gettin’ a PowerEdge” guy. They could pay him seven figures a year and it’d increase revenue by nine to whoever does it. I’d be sad to lose STH from that tho
Shhhh….that worked with Anand, Kyle, Ryan, Cyril, Geoff, Scott….
I had a look at these server and preference the lenovo above them, I got a quote for a bunch of NvME drives and they wanted to put 2 Percs in to cover the connectivity their solution was expensive in inflexible especially if you put the NvME on the raid controllers I think it only does Gen3 and only 2 lanes per dive or something like that
the solution I got from lenovo with the new genoa chip was cheaper , faster and better because of their anybays and the pricing on the U3 drives was pretty good as well especially b/c I only wanted 16 cores and they have a single CPU version
you need to take the time to real the full technical documentation for this server withe the options and how stuff connects.
Agreed that the pricing is bonkers. I built a R7615 on Dell public, and it came out to $110k, while on Dell Premier it was $31k. A Dell Partner had built that out and tried to say $60k was a reasonable price, based on the lack of discounts Dell gave them. Mind you a comparable Lenovo or Thinkmate was $5-7k less than the $31k.
Second is their insane pricing on NVMe drives in particular. In a world where you can get Micron 9400 Pro 32TB’s for ~$4,200 retail (from CDW no less), Dell asking $6,600 for a 15.36TB Kioxia CD6-R (which is their “reasonable” Premier price) is shameful! I think they are actively trying to drive people away from NVMe, despite the PERC12 being a decent NVMe RAID controller for most use cases. Whenever I talk to a Dell rep / partner, they try to push me to SAS SSDs since NVMe “are so expensive”. You’re the ones making it so damn expensive!
Dell really should get more criticism / kick back for these tactics.
Tam, FYI the PERC12 is based on the Broadcom SAS4116W chipset and has a 16x PCIe Gen 4 interface and I believe is 2x Gen 4 lanes per SSD. Still a bottleneck if you are going for density of drives, but fairly high performing if you can split it out to a few more half capacity drives. E.g. four 15TB drives instead of two 32TB drives.
There are a couple of published benchmarks showing the difference, for example:
https://infohub.delltechnologies.com/p/dell-poweredge-raid-controller-12/
So, if you do need hardware NVMe RAID, such as for VMWare with local storage, the PERC12 looks like a solid solution. But they do have direct connect options to bypass using a PERC if you prefer that.
Thanks for that link Adam. The Broadcom docs for this gen chip, with 2, 8x connectors (8 PCIe lanes per connector), show that # of lanes assigned can vary by connector, and can be 1, 2 or 4 per drive, but must be the same for every drive on a specific connector. See page 12 of
https://docs.broadcom.com/doc/96xx-MR-eHBA-Tri-Mode-UG
Keep in mind though that the backplane needs to support matching arrangements. Not sure what Dell is doing there. It’s an issue on at least some Supermicro AFAIK.
So glad to see the latency came down to NVMe levels in that PERC12 testing and write IOPS improved so much. As a Windows shop, hardware RAID is still our standard.
Agreed on the silliness of the Dell drive pricing though – had to go Supermicro due to it on some storage servers, though I much prefer Dell fit/finish, driver/firmware and iDRAC for our small shop. I was able to get around it by finding a Dell outlet 740xd that met our needs one time as well (and was packed with drives so risk of needing to upgrade was very low).
Whereas I can buy a retail drive (of approved/qualified P/N) and put it in my Supermicro servers.
SIX HUNDRED WATTS IDLE AND 2000 DOLLAR MEMORY STICKS? What is this, man.
Do you know where the power is going for a 600w idle figure? You mentioned a specific high performance state config; is that some sort of ‘ASPM is a lie, nothing gets to sleep ever, remain in your most available C-states, etc.’ configuration(and, if so, are there others?); or is there just a lot going on between the lowest power those two Xeons can draw without going to sleep and the RAM, PERC, etc?
I wouldn’t be at all surprised to see 2x 300w nominal Xeons demand basically all the power you can supply when bursting under load, that’s normal enough; I’m just shocked to see that idle figure; especially when older Dells in the same basic vein, except with older CPUs that have less refined power management, less refined/larger process silicon throughout the system generally, plus 12x 3.5in disks put in markedly lower figures.
I think the R710 I was futzing with the other day was something like 100-150w; and that was with a full set of mechanical drives, full RAM load, much, much, less efficient CPUs, etc. Lower peak load, obviously, since the CPUs just can’t draw anything like the same amount the newer ones can; but at idle I would have expected it to be worse across the board.
thanks for the reply Adam, I did realize that after I posted it but you cant edit I dont think , the other thing was the requirement I was quoted to have 2 percs in stalled. If I was building some think “special” especially with NvMEs I would go the intel server you can really muck around with them , support is a bit of a pain unless you a big guy, but for what you save you can afford to actually buy a couple of spares, like a mainboard ect, they have connectivity on their boards directly behind the drive bays with MCIO connectors all run flat out if you needed it. I have a unit here with VROC I was testing but got some funny results and dont have the coin to buy really good drives to see if issue still arises.
I would love STH to actually build a VROC server and do some numbers with some quality components since you probably have most of it hanging around
I completely agree about VROC, I requested that in a comment on another STH article. Since VROC is supported by VMWare, I’m really curious what performance looks like versus software raid, ZFS & a current gen hardware RAID controller.
Also as a general question for all readers- does anyone know in practice how this generation of Dell servers handles non-Dell NVMe drives? What little Dell publishes just states there are certain NVMe features that are required.
Maybe STH could test tossing in a couple of Micro 9400 drives into the R760?
I created a thread about VROC on the forums and shared my very limited experience. Figured we do not want to pollute these comments further. I agree there is a dearth of information available on it which is a problem as there is a need for such solutions.
https://forums.servethehome.com/index.php?threads/intel-vroc-2023.39815/
Super thread on VROC.
Yuno put it about as well as anyone, Dell has lost their damn minds and their entire company must be high to think this crap is okay. I just saw today they’re still selling a desktop with an nvidia 1650 in it for almost $1000 ( https://www.dell.com/en-us/shop/desktop-computers/vostro-tower/spd/vostro-3910-desktop/smv3910w11ps75397 ), what kind of bizarro world are we living in?
What kind of performance are you seeing with the BOSS setup? I just received a PowerEdge R360 with a BOSS-N1 configured for RAID1 and I’m getting 1400MB/s READ and 222MB/s WRITE. I can’t find any reviews stating the read/write performance.