A Broader Industry Discussion and Concern
Let us be clear, while iDRACula focuses on Dell EMC iDRAC, there is a broader industry concern here. The company had resources mobilized to understand and reproduce what happened on the STH forums within hours. Other vendors do not have this response time.
In order to remotely execute an iDRACula firmware swap, you need to have remote access to the BMC and an existing login. There are still a startling number of IPMI management interfaces directly addressable via a public IP. Further, the default logins are often left in place. Here is a list of the default logins I wrote down in 90 seconds giving myself the task:
- Dell EMC iDRAC: root/ calvin
- Supermicro: ADMIN/ ADMIN
- Cisco UCS: admin / no password
- Other Cisco: admin/ admin or cisco/ cisco
- Intel Remote Management Module: admin/ password
- Lenovo IMM:Â USERID/ PASSW0RD
- Gigabyte Server: admin/ password
- QCT Networking: admin/ no password
- ASUS ASBM: admin/ admin
- APC: apc/ apc
- ASRock Rack: admin/ admin
These were a few top of mind default logins. Exposing interfaces with default logins to the Internet is not a leading security practice, yet it is something done on a regular basis.
Newer Dell iDRAC implementations, along with those like Cisco UCS, prompt users to change default credentials immediately. Those in the industry can probably attest that there are a lot of “root/ calvin” Dell EMC servers with that login active. Likewise, ADMIN/ ADMIN on Supermicro servers and some appliances that use Supermicro as a building block is common and often exposed to the internet.
While many will dismiss the physical access variant, the nature of the vulnerability makes it quite the intriguing vector. We often think about servers operating in a data center. There is a step just before that. A server is rarely fully assembled on-site. Even if it is, motherboards with the BMCs are created elsewhere. That shipping process exposes servers, prior to installation in a data center to various entities in a supply chain. Larger enterprises often order servers not in a box with a tamper-proof seal, but instead as pre-integrated racks ready to wheel into the data center and install. That gives ample touch points where a malicious actor could use this vulnerability to compromise a BMC. Once compromised, a user would not be able to see that this has taken place.
Imagine if a financial institution could remotely program all 1000 nodes of an exchange or competitive firms servers to reboot at exact times by loading a script to do that before the servers were installed. Also, state actors who can inspect shipments of servers during standard customs screenings would have access to utilize this vulnerability with a reason to delay servers and break security tamper seals.
You may assume that just because this impacts the previous generation servers it is not an issue since the new 14th generation PowerEdge servers are now shipping. Remember, Intel significantly raised prices on its chips with this Xeon Scalable generation, and the previous generation Intel Xeon E5-2600 V4 servers that are susceptible to iDRACula are still shipping in quantity in 2018.
Infrastructure’s largest consumers such as Microsoft and Google are taking management firmware very seriously. They are creating a hardware root of trust chain to validate that the correct software and hardware is in a server. Dell EMC and HPE have been pushing similar solutions. Facebook has been active with the OpenBMC project to push auditable firmware on to BMC devices.
From an industry perspective, securing BMCs needs to be a priority. They are used both by single server shops as well as the largest infrastructure providers. BMCs make servers manageable without full-time data center staff or expensive remote hands. They are what make the digital infrastructure run and provide management for the machines that trillions of dollars worth of global economic activity rely upon.
Final Words
After hearing about this vulnerability, I wanted to give it a memorable name, hence calling it iDRACula. The importance of BMCs, the prevalence of the technology in Dell EMC servers along with other brands, and the potential for vulnerabilities like iDRACula mean that this is is an area of data center technology that needs your attention. Although iDRACula focused on Dell EMC servers, we know of other management solutions that are more vulnerable. Dell EMC has been shipping a new generation of servers with updated hardware to address this type of vulnerability for over a year now. On the other hand, there are large numbers of legacy PowerEdge products still in the field.
We hope that reading this article, STH’s IT professional readers will take a moment to ensure that, at a minimum, their iDRAC (and other IPMI management) firmware is updated, and that iDRAC interfaces are not on public networks and do not have default logins active. Take this article and have a discussion with your teams about iDRACula and how you may have exposed BMCs. The prospect of a malicious actor utilizing these vulnerabilities with great effect is too great to not spend a few minutes reviewing this.
The next step questions for your team today would be:
- If this can be done on a Dell EMC PowerEdge, generally considered one of if not the best management solutions out there, what about our other servers?
- Could someone have intercepted the servers and applied a corrupted firmware image in transit. These are exactly the questions that the hyper-scale community like Google and Microsoft see as a big enough threat that they are introducing their own hardware solutions to give their customers peace of mind.
Even casual industry observers recognize the power of remote management. That power makes scale-out automation of servers possible as well as small deployments for startup organizations. The other side of this power equation is that it is a great power that needs to be guarded. We urge our users to take this opportunity and evaluate their risk profile for an iDRACula like vulnerability.
Holy shit. Good job presenting. I’ve sent this to our team as a reminder we need to take this kind of security seriously and not go lax over time. I saw the post in the forums before it was taken down and I’d not put 2 and 2 together.
Dell is supposed to be the best at this………
I’ve know you could do this for awhile on regular ipmi servers.
Convenience comes with a price.
We were about to buy R730 but now we’re going to look at the R740 or R740xd.
So if I buy a new PowerEdge R630 today, ship it to our South American hub the person installing it tomorrow could do this to the server before I even see it is online? iDRAC is pre-boot so I’d have no way of knowing.
Is this an end of month Dell 14th gen sales promotion?
Was there a CVE submitted for this? Or does it not need one as an old firmware with a known vulnerability was used?
Aaron: a CVE is in the process of being submitted by dell, yes. While the remote process does involve rolling back to an older firmware, the issue/actual exploit lies in the fact the latest firmware has some holes that when used, allow you to then re-flash to the latest firmware and keep exploits in place undetected
Dell is still selling idrac 8 in new servers that can be exploited like this
Ok, so by handing a thief the key to your car and/or the key to your garage (because you chose to trust them)…. They can now take the motor out and modify it and put it back (hey that sounds fun)…. Helllllo you gave them the keys!!! They can turn it on or off anytime and have their way with it at will just by utilizing the key!! Or better yet, if I’m the owner and I hold the key I can have MY way with MY server, my choice of handing out the key or leaving the door open is the obvious security vulnerability at large. While a valid security flaw this is just a bit over fluffed. People need to learn more about enterprise best practices to prevent this, such as management vlans blocked from the internet, video surveillance and physical access control… rather than only buy 14th gen servers or learn the hard way.
John, you’re missing a huge part of the server deployment process, the supply chain. You can guard against physical access all you want once you have the equipment in-hand, but it doesn’t matter if anyone who had it previously had bad intentions, especially given as-of-now anything malicious they could potentially load will survive standard firmware re-flashes. Using your analogy, it’s more akin to buying a car that runs unsigned code on the engine control unit, and a pissed off employee at the dealer/tow company/etc loaded in their own modified firmware. The car seems perfectly fine, you keep it secure in the garage and nobody else has the keys, but one day that modified code is going to slam you into a wall.
Is this likely to happen to rinky-dink IT companies with not much to steal from? Doubtful, the chances are very low. is this likely to happen to large providers, financial institutions, etc? Yes, it already has happened. There is a reason Facebook, Microsoft, OpenBMC etc have spent millions solving the unsigned firmware issue, they have seen the damage it can do. 5 minutes of physical access to hardware to load something that will never be cleaned away is much easier for an entity to achieve in supply chains then you think, just ask any of the victims of previous exploits
This is why you never ignore defense in depth. Physically lock and secure your datacenters. IPMI traffic should be in an isolated vlan with no direct internet or end user access.
I was so excited to read about a new BMC vulnerability but this doesn’t apear to be anything new at all. Speaking strictly about the article, it seems to be a sensationalized piece about “scary BMCs oooh”. Nothing the security world doesn’t know about already. And please for the love of… stop naming these things. “integrated Dell Remote Access Controller unauthorized load access”… really??? You’re kinda grasping with that one.
If you’ve got physical access to a server you’ve already won. Not even talking about the iDRAC there are other interesting attack surfaces available as well (Internal SD card? Internal USB port? etc) but yeah the completely hidden and persistent ones are the best.
Speaking as someone who has actually discovered such a vulnerability in 2016 (CVE-2016-5685, local iDRAC8 root shell: https://www.cvedetails.com/cve/CVE-2016-5685/) I’m not sure how downgrading to an old, known to be vulnerable iDRAC release and exploting it is news. Did I miss something or is the -f[orce] flag that can be used to downgradeFW not a thing with the Dell Update Package anymore? I’m asking honestly. I haven’t checked. But that’d be an easy way to downgrade all the things by booting via Live USB.
I mean, if you managed to get root on the iDRAC on the old pre-patched version all by yourself (without hints) then in that case mad props to you for figuring it out. The fact that you’re now attacking a persistent file system and that this can’t be quickly fixed by reflashing to the same or newer version of the Firmware is known to Dell (and should be pretty obvious to anyone really) but clearly it wasn’t meant to be broadcast to the world…
Finally, the 14th Gen servers can and will be exploited… It will be harder to make these changes persistent but nothing says “challenge accepted” like a promise of complete protection from security bugs. If you have an iDRAC9 with a very early initial release of the FW you can drop to a root shell pretty easily too if you “invoke” some “debug” stuff: https://pbs.twimg.com/media/DIm-fhAUQAE8VaC.jpg
Luckily that was a very early FW release and was quickly patched…
Anyway, good on you for keeping BMCs on peoples radars as attack surfaces.
Is there an official dell support page for this? What is the CVE number?
THanks
“Please keep in mind you need either physical access to the PowerEdge server. To do this remotely, you must have existing iDRAC credentials.”
If someone has physical access to your servers you have bigger problems
Ditto iDRAC credentials..
@SA, apologies, just now seeing this. downgrading to a known exploitable firmware version is not the news, that’s however all STH was comfortable mentioning about the process and I don’t blame them. Now that a CVE has been assigned internally and a patch is coming out in a couple weeks I can probably say more: the larger exploit here is not being able to downgrade, but the process of downgrading, utilizing something that allows you to make arbitrary changes to the filesystem, and then upgrading back to the very latest “secure” firmware but keeping your changes in place undetected, persisting privilege elevation exploits regardless of how new firmware you flash to. According to dell any persistent changes like this should not have been possible, but clearly they are. There’s a couple other fun things too that wouldn’t have been good to publish months before a patch. I will probably publish more details on the github after the patch is released, but as you’ve guessed, since they don’t have hardware root of trust, one of these methods is unpatchable, and will always be possible.
And yes, very early versions of IDRAC9/14th gen firmware allowed you root access. It’s not of much use though, as you cannot load and persist anything. You can try, but on next power-on the entire boot chain from bootloader to entire EMMC flash is key checked against burned in keys in the proc and it will refuse to boot, as I’m sure you’re aware. That’s not to say there’s not some extremely niche bug in the signature checking chain somewhere, but it would certainly be far from arbitrary to exploit