While we have been focusing a lot on NAS crypto locking with DeadBolt and a recent wave of attacks, there was another one that came out at the end of 2021 targeting older versions of HP/HPE iLO. Specifically, a research team found a rootkit that can impact older generations of hardware using iLO 4 and earlier, and another researcher found over twenty thousand of these iLO 4 controllers connected directly to the Internet.
Outdated HPE iLO Interfaces Exposed to the Internet and Rootkits
Let us get into the two parts of this. First, let us discuss the rootkit briefly. Then, let us discuss the public scope of exposure on the Internet.
The iLO rootkit is called Implant.ARM.iLOBleed.a that was disclosed at the very end of 2021 (see here for reference.) We missed picking this one up since it was on December 28. The basic idea is that HP/ HPE iLO firmware can be altered and then the rootkit installed, leading to a persistent threat and attackers able to erase disks and more. This is specifically targeted at iLO 4 and below servers even the HPE Moonshot iLO like we are showing above. If you are on ProLiant Gen10 servers, then you likely have iLO 5 but G9 and older servers can be vulnerable. In newer servers, there is a much stronger concept of firmware security and secure boot that would likely prevent this type of attack.
As a quick aside, if you saw our AMD PSB Vendor Locks EPYC CPUs for Enhanced Security at a Cost and Lenovo Vendor Locking Ryzen-based Systems with AMD PSB pieces, this is one of the types of firmware attacks that that feature is designed to mitigate. Effectively, an attacker compromising firmware can get low-level access to the system.
Of course, the prerequisite to loading the firmware is getting access to the device. In our Basic BMC and IPMI Management Security Practices, we note a poor way to set up a server management BMC interface, on any server (not just HPE) is to have it directly accessible from the Internet.
A researcher at sans.edu, Jan Kopriva, used Shodan plus some clever favicon hashing to determine the number of iLO systems that follow the worst practice diagram above. Check out the full analysis here. The net result is that over 20,000 iLO BMCs are directly accessible via the Internet. Many of these, are also running out-of-date firmware. This is important because pre-iLO 2.5.3 firmware has an authentication bypass CVE scoring just shy of 10. There are also quite a few other vulnerabilities in iLO that, as one would expect, get patched regularly. That is a lot of potentially vulnerable systems.
Direct access to the public Internet, a well-known authentication bypass, and a persistent firmware rootkit on systems without the same level of platform security is not a good recipe for those operating servers.
Final Words
As with our recent QNAP pieces, this is one of those areas where we are urging our readers to update iLO firmware. For many legacy systems, they may not have been updated to the latest firmware before support contracts ran out, so that can be challenging given HPE’s paywalled support model. We are also suggesting that our readers double-check that their iLO interfaces and BMC interfaces, in general, are not exposed to the public Internet. Attacks like these are one of the reasons platform firmware security has become such a hot topic. Even if you are not impacted by these, it is always good to set a reminder to keep firmware updated because we know that not all vulnerabilities are disclosed.
Seriously, people that expose iLOs, iDRAC or any kind of management-interface, to the internet should be shot. Survivors should be shot again (just kidding). There is no excuse for this.
As ridiculous as it sounds, having an ILO or IDRAC exposed to the internet is how some cloud providers give you access to physical servers you pay for.
I was only given a read only account that could access the console. That was more than enough for me to be able to do an account reset and gain full access to the entire system (It was the only way I could reboot the iDRAC to release a Virtual DVD that was stuck).
Needless to say, I no longer use that cloud provider.
@John:
Like an old MASH (TV series) goes, they “should get life in front of a firing squad”….
But you are completely correct, as is the article’s suggestion in it’s final paragraph.
What are people thinking that connect stuff like this and those NAS boxes and such to the Internet…even if they take proper steps to secure it.
Anything attached to the Internet will eventually be hacked. It’s not not a matter of IF but of WHEN.
Sadly some vendors require a paid service plan subscription for admins to be able to download firmware etc that keeps small business or homelab users from updating their systems. IMHO those should be available freely since the product was somehow ‘defective’ or ‘unsafe’ to begin with!
Ugh. I’ve had to clean up management interfaces for clients on premise and at the datacenter more times than I can count. Many act as if I’m asking for firstborn just to get it moved to a seperate network, let alone employing other security methods. This issue sadly is more widespread than people think in LARGE, enterprise sized companies.
@John
There are some small corporations that need these for specific issues and just hook them up. They don’t have the IT staff with the knowledge to do this correctly. As bad as having your iLO, iDRAC, IPMI, etc… exposed to the internet I have seen worse as a consultant. I’ve seen an entire ESXi host directly attached to the internet. It wasn’t even behind a firewall. Again small companies not having the correct support staff. If a large corporation does this there is no excuse since they have the money for the correct staffing.
This problem was found and corrected by HPE in 2018. Old news that was resolved. Customer bulletins were sent out to have customers update iLO.
IKnowProLiant if you read the sans.edu article, they’re looking for and found iLOs that had not been updated and the rootkit is new. The paywall firmware means not everyone’s gonna be updating. I think that’s the point they’re making that there are still vulnerable devices out and on the Internet
@Sacrilego – did they at least limit inbound packets to your IP or gateway IP?
@emerth – I wish, it was open for anyone to access. The only protection was a randomly generated username and password. I brought it to their attention but was told that it was secure enough. I canceled within the month as I wasn’t satisfied with their response and the performance.
Most corporations don’t connect their iLo’s, DRAC’s and management interfaces to a network rail that has an internet gateway. Cloud providers usually put their customer management rails behind a firewall with a secure ipKVM portal in place.
I just checked, iLo 3/4/5 updates are available without login.
Because there is a barrier in the firmware, not everyone is going to update.