Dell PowerEdge R760 iDRAC 9 Management
This generation still uses Dell’s iDRAC 9 management. To be clear, iDRAC 9 is probably the biggest differentiator in the PowerEdge R760. With iDRAC 9, data can be collected on many components and configurations set. These can then feed into Dell’s OpenManage solution so organizations can monitor and manage fleets of servers easily. This is not all free functionality, but for organizations that want an easier solution than cobbling together open-source or 3rd party tools, this is very useful.
Many are familiar with iDRAC 9 because it has been around for some time. It has features to manage storage without having to go into OS or UEFI tools.
One great feature is that one can set BIOS settings directly from the web interface. Not all vendors have this feature. We showed How To Change Core Count and TDP of Intel Xeon Y CPUs on Dell PowerEdge using this feature.
Along with the iDRAC BMC, Dell also has optional software agents like the iDRAC Service Module (see How to Install the Dell iDRAC Service Module) that can be used to gather information via the OS agent and pass it through the management stack.
This data is then brought into iDRAC and then OpenManage.
Dell also has features like iKVM support. We have iDRAC 9 enterprise on this system, so we have the full HTML5 iKVM functionality.
We also have virtual media support via the GUI.
Overall, Dell iDRAC 9 is a solution that is very good and that makes PowerEdge customers stay with PowerEdge.
Next, let us get to the performance.
Dell PowerEdge R760 Performance
The CPUs installed in this unit are Intel Xeon Platinum 8452Y units. We configured them to P1 36 cores, 300W TDP, and a 2.0GHz base clock which is the maximum core count for these Intel CPUs.
We had the opportunity to test a number of servers with the same SKUs recently and so we had some decent generational comparison points.
If you saw our recent Putting the Bare Metal Server in the PhoenixNAP Bare Metal Cloud article or video, then you saw some of this testing already.
We used an on-prem Supermicro server, a cloud bare metal server at PhoenixNAP, and then the PowerEdge R760 and ran the same workloads ranging from virtualization to some microbenchmarks, to AVX-512 HPC workloads, database workloads, compile workloads, an even utilized the new Intel AMX AI extensions for inference.
As one can see, we ended up fairly close between the three servers with the same CPU SKUs. There is always some test variation, and we could not control the cloud data center, but the key finding is that the Dell PowerEdge R760 with these 300W CPUs was in-line with expectations.
Next, let us get to power consumption.
On BOSS Cards: Previous generations required a full power down and removing the top lid to access the M.2 card to change BOSS sticks. This new design allows hotswapping BOSS modules without having to remove the device from the rack. Very handy for Server Admins.
I’m loving the server. We have hundreds of r750’s and the r740xa’s you reviewed so we’ll be upgrading late this year or next thx to the recession.
Pricing comments standout here.
I’m in disbelief that Dell or HPE doesn’t just buy STH to have @Patrick be the “Dude you’re gettin’ a PowerEdge” guy. They could pay him seven figures a year and it’d increase revenue by nine to whoever does it. I’d be sad to lose STH from that tho
Shhhh….that worked with Anand, Kyle, Ryan, Cyril, Geoff, Scott….
I had a look at these server and preference the lenovo above them, I got a quote for a bunch of NvME drives and they wanted to put 2 Percs in to cover the connectivity their solution was expensive in inflexible especially if you put the NvME on the raid controllers I think it only does Gen3 and only 2 lanes per dive or something like that
the solution I got from lenovo with the new genoa chip was cheaper , faster and better because of their anybays and the pricing on the U3 drives was pretty good as well especially b/c I only wanted 16 cores and they have a single CPU version
you need to take the time to real the full technical documentation for this server withe the options and how stuff connects.
Agreed that the pricing is bonkers. I built a R7615 on Dell public, and it came out to $110k, while on Dell Premier it was $31k. A Dell Partner had built that out and tried to say $60k was a reasonable price, based on the lack of discounts Dell gave them. Mind you a comparable Lenovo or Thinkmate was $5-7k less than the $31k.
Second is their insane pricing on NVMe drives in particular. In a world where you can get Micron 9400 Pro 32TB’s for ~$4,200 retail (from CDW no less), Dell asking $6,600 for a 15.36TB Kioxia CD6-R (which is their “reasonable” Premier price) is shameful! I think they are actively trying to drive people away from NVMe, despite the PERC12 being a decent NVMe RAID controller for most use cases. Whenever I talk to a Dell rep / partner, they try to push me to SAS SSDs since NVMe “are so expensive”. You’re the ones making it so damn expensive!
Dell really should get more criticism / kick back for these tactics.
Tam, FYI the PERC12 is based on the Broadcom SAS4116W chipset and has a 16x PCIe Gen 4 interface and I believe is 2x Gen 4 lanes per SSD. Still a bottleneck if you are going for density of drives, but fairly high performing if you can split it out to a few more half capacity drives. E.g. four 15TB drives instead of two 32TB drives.
There are a couple of published benchmarks showing the difference, for example:
https://infohub.delltechnologies.com/p/dell-poweredge-raid-controller-12/
So, if you do need hardware NVMe RAID, such as for VMWare with local storage, the PERC12 looks like a solid solution. But they do have direct connect options to bypass using a PERC if you prefer that.
Thanks for that link Adam. The Broadcom docs for this gen chip, with 2, 8x connectors (8 PCIe lanes per connector), show that # of lanes assigned can vary by connector, and can be 1, 2 or 4 per drive, but must be the same for every drive on a specific connector. See page 12 of
https://docs.broadcom.com/doc/96xx-MR-eHBA-Tri-Mode-UG
Keep in mind though that the backplane needs to support matching arrangements. Not sure what Dell is doing there. It’s an issue on at least some Supermicro AFAIK.
So glad to see the latency came down to NVMe levels in that PERC12 testing and write IOPS improved so much. As a Windows shop, hardware RAID is still our standard.
Agreed on the silliness of the Dell drive pricing though – had to go Supermicro due to it on some storage servers, though I much prefer Dell fit/finish, driver/firmware and iDRAC for our small shop. I was able to get around it by finding a Dell outlet 740xd that met our needs one time as well (and was packed with drives so risk of needing to upgrade was very low).
Whereas I can buy a retail drive (of approved/qualified P/N) and put it in my Supermicro servers.
SIX HUNDRED WATTS IDLE AND 2000 DOLLAR MEMORY STICKS? What is this, man.
Do you know where the power is going for a 600w idle figure? You mentioned a specific high performance state config; is that some sort of ‘ASPM is a lie, nothing gets to sleep ever, remain in your most available C-states, etc.’ configuration(and, if so, are there others?); or is there just a lot going on between the lowest power those two Xeons can draw without going to sleep and the RAM, PERC, etc?
I wouldn’t be at all surprised to see 2x 300w nominal Xeons demand basically all the power you can supply when bursting under load, that’s normal enough; I’m just shocked to see that idle figure; especially when older Dells in the same basic vein, except with older CPUs that have less refined power management, less refined/larger process silicon throughout the system generally, plus 12x 3.5in disks put in markedly lower figures.
I think the R710 I was futzing with the other day was something like 100-150w; and that was with a full set of mechanical drives, full RAM load, much, much, less efficient CPUs, etc. Lower peak load, obviously, since the CPUs just can’t draw anything like the same amount the newer ones can; but at idle I would have expected it to be worse across the board.
thanks for the reply Adam, I did realize that after I posted it but you cant edit I dont think , the other thing was the requirement I was quoted to have 2 percs in stalled. If I was building some think “special” especially with NvMEs I would go the intel server you can really muck around with them , support is a bit of a pain unless you a big guy, but for what you save you can afford to actually buy a couple of spares, like a mainboard ect, they have connectivity on their boards directly behind the drive bays with MCIO connectors all run flat out if you needed it. I have a unit here with VROC I was testing but got some funny results and dont have the coin to buy really good drives to see if issue still arises.
I would love STH to actually build a VROC server and do some numbers with some quality components since you probably have most of it hanging around
I completely agree about VROC, I requested that in a comment on another STH article. Since VROC is supported by VMWare, I’m really curious what performance looks like versus software raid, ZFS & a current gen hardware RAID controller.
Also as a general question for all readers- does anyone know in practice how this generation of Dell servers handles non-Dell NVMe drives? What little Dell publishes just states there are certain NVMe features that are required.
Maybe STH could test tossing in a couple of Micro 9400 drives into the R760?
I created a thread about VROC on the forums and shared my very limited experience. Figured we do not want to pollute these comments further. I agree there is a dearth of information available on it which is a problem as there is a need for such solutions.
https://forums.servethehome.com/index.php?threads/intel-vroc-2023.39815/
Super thread on VROC.
Yuno put it about as well as anyone, Dell has lost their damn minds and their entire company must be high to think this crap is okay. I just saw today they’re still selling a desktop with an nvidia 1650 in it for almost $1000 ( https://www.dell.com/en-us/shop/desktop-computers/vostro-tower/spd/vostro-3910-desktop/smv3910w11ps75397 ), what kind of bizarro world are we living in?
What kind of performance are you seeing with the BOSS setup? I just received a PowerEdge R360 with a BOSS-N1 configured for RAID1 and I’m getting 1400MB/s READ and 222MB/s WRITE. I can’t find any reviews stating the read/write performance.