Home Lab and Secondary Server Market Killer
Thinking about the lifecycle of a server for a moment, when it is first sold with an AMD EPYC processor, the fuses are blown, and the server OEM effectively brands the AMD EPYC CPU in its ecosystem. Beyond that chain, we now have challenges. These are where the new security paradigm will introduce new hurdles.
Today, there is a healthy grey market for CPUs. Many of the Intel Xeon Bronze and low-end EPYC 7232P / 7252 system sales are actually because those CPUs can often be removed and a higher-end CPU replaced in the socket at a lower cost than it is offered by OEMs. The reverse is also true that a discount on an entire server may make it viable to order a higher-end SKU and then downgrade the processor. Extra CPUs can be purchased for projects, testing, or advanced stock. Perhaps a partner needs to buy 20 more CPUs to hit a program tier limit, so a partner simply just purchase extras to resell so they can move up a tier. These are some of the ways that even “new” CPUs can hit the secondary market. If any of these CPUs are from say a Dell server that has turned on and one-time-programmable fuse blown, then when that CPU is resold on the grey market, it may not work in systems by other vendors. For a grey market reseller, this is an enormous headache, especially if they do not know about the feature. If you do have a reseller, send them this article and ask them to detail what they are doing about their supply chain.
Beyond the early grey market, decommissioned servers are often repurposed as used systems. These can be for home labs, or simply companies that want to use the lower-cost used gear. Something about that market that is fascinating is that often these systems are not repurposed as a whole. STH’s first colocation web hosting servers were Dell C6100 systems that were repurposed Twitter and Facebook systems. Today’s hyper-scale servers likely would not work in many environments so recycling companies pull CPUs, RAM, and potentially other components that can be used to upgrade or augment other existing systems. That can also happen in enterprise refreshes where recyclers break down enterprise servers to make certain valuable components easier to resell. For security “dumb” processors of yesteryear, this is a normal model. Once these field programmable fuses are blown it gets more complex. Moving a processor to a new system can have a high likelihood of the processor not working.
So I asked Dell EMC about this. Not all vendors are currently using the PSB feature, so by enabling it, it effectively makes a Dell EMC system or systems from other vendors who enable this feature less able to be re-used. That limit of reusability severely curtails useful lifespans of systems, and is, therefore, less green. Here is Dell EMC’s statement to STH:
As you rightly point out, the AMD Platform Secure Boot Feature (PSB) is a mitigation for firmware Advanced Persistent Threats. This allows us to establish an unbroken chain of trust from AMD’s silicon root of trust to our BIOS and then from the BIOS to the OS Bootloader using UEFI secure boot. This provides a very powerful defense against remote and local attackers seeking to embed malware into a platform’s firmware.
We design PowerEdge servers with security built-in as the security of our products is critical to helping ensure our customers’ data and systems are protected. Given the pervasiveness and increasing sophistication of these ongoing persistent threats, we decided to enable the PSB function that AMD makes available. The tradeoff, as you pointed out, is that the CPU would only be able to operate in another Dell EMC PowerEdge server. However, we feel that’s a rather limited use case for how customers look to decommission old equipment and wanted to err on the side of security.
We have a decades-long focus on sustainability, and are fully committed to accelerating the circular economy and diverting all e-waste from landfills (more information here about our efforts and our sustainability goals). We also provide recycling solutions that protect our customers’ data, safeguard their brand reputation, reuse precious materials and responsibly recycle in compliance with local regulatory guidelines, such as the EPA and WEEE legislation and waste regulations. (Source: Dell EMC statement to STH.)
My position is that this feature will lead to more e-waste, but Dell points out it is pro-environment and recycling. We will let our readers think about the implications having heard both sides and draw their own conclusions.
At the end of the day, vendors and their customers are happy to have better security so these are touted features. It is also fairly easy to see how they will impact the secondary markets for used servers, CPUs, either in a professional or even a “home lab” scenario. Imagine the confusion it will cause if the source of CPUs is not tracked down to the level of which firmware is being used. In theory, a vendor servicing multiple hyper-scalers could use the same motherboard, but load different cryptographic signatures for the firmware used by each end customer. Once a CPU is removed from a system after fuses are blown, there is no easy way to visually tell what has previously transpired. That record needs to travel with CPUs which is easy enough, except today’s ecosystem does not track this information.
We have been focusing on AMD EPYC CPUs, but it goes beyond AMD.
Stepping Back: A Look at the Future
It is important to remember that these security features are being implemented by AMD ahead of Intel, but they are being demanded by customers. At some point, Intel will have to match this feature set. Intel’s Management Engine found in its Lewisburg PCH is not as feature-rich, and so Intel will be forced to change. The LBG-R refresh part is already making major moves, but at some point, other modern architectures do not have PCHs so Intel is likely to follow suit here.
HPE told us, for example, they do work with the LBG ME, but the functionality it is using to secure Xeon platforms is not as robust as what AMD is offering. The bottom line is that Intel will need to meet customer demands in this space especially given AMD’s lead. Ice Lake Xeons will see a move in this direction but there is a security roadmap Intel is pursuing for the Xeon line. As a fun piece of trivia knowledge, this means that every Intel Xeon Scalable server has multiple Intel Quark cores inside. If you thought Quark is a dead architecture, it is alive in mainstream server platforms today. Indeed, one could even say that Facebook is deploying Quark servers even in its new Cooper Lake generation.
Outside of x86, IBM POWER10 is making a push for enhanced security, so the will need to have a silicon root of trust to enable their security feature set.
Even Arm offerings are going to need to add these types of security processors. We originally wrote this article with Marvell ThunderX3 before it was canceled (although Marvell OCTEON is still moving in this direction) but we can see Ampere receiving the same security mandates from customers and implementing them in its designs.
Ampere Altra has control processors that handle many of its security features. Assuming vendors want CPUs to validate against signed platform firmware, Ampere will have to do something similar.
Neither the IBM Power10 or Ampere Altra are commercially available to purchase for a site like STH today given where we are in their lifecycles, so we are not going to comment on features of these platforms.
The key takeaway from this section is that we have focused on AMD here because they are a few years ahead of Intel on addressing this requirement. Other vendors to meet hyper-scale, government, and some enterprise requirements will need similar functionality. This is not AMD trying to harm the used market, it is really that the initial customers buying these parts want that functionality.
Final Words
This started out as being a 500-word post clarifying some comments in the Dell EMC PowerEdge C6525 Review and video. It turned into multiple exchanges with vendors and a few large customers to understand what is going on. Hardware security is a big deal especially given potential state-level threats so those that are driving the server industry are making it a requirement that what they buy has a level of security capability they are comfortable with. Vendors are tailoring their products to those requirements, as they should.
For those who buy or sell either used IT equipment or grey-market equipment, this is a new functionality that needs to be on your mind today with AMD EPYC, but then with other platforms in the future. Intel, for its part, benefits in the secondary market for not having the same security features that make AMD EPYC processors attractive in the primary market. Still, it should be practice to document where any AMD EPYC processors are pulled from so that evidence can be provided to potential buyers since these are invisible changes.
One mitigation would be to add an “un-securable” feature to these CPUs. Once this feature is set, they cannot be used with secure firmware/ platforms. That would allow the secondary market to use these chips knowing they are not secure. A consequence here is that doing so would not help the case where a grey market CPU gets fused un-securable and is sold and used in a secure platform. Perhaps if this was some kind of physical external and visible PCB based fuse it would make it clear to buyers that this was irreversibly set. There are no great options, but the industry will have to explore possibilities in the future.
For STH readers, we want you to be aware of the implications of increased platform security so you can inform your colleagues, suppliers, and customers of the changes. If we learned nothing from the PowerEdge C6525 comments on this subject, it is that the market does not have a great grasp of the features and perhaps as importantly, the implications of increased security.
Do you know whether the flash firmware on the motherboard is protected using AMD secure memory encryption? That would protect against the kind of circumvention measures seen in the fake Cisco routers reported on earlier.
In my opinion, this is a very complicated and wasteful way to solve the problem that the flash memory on the motherboard doesn’t have a write protect jumper. I know that jumpers don’t scale in a data center, but there could still be a way to physically write protect the motherboard flash while the server is in operation.
We’re actually looking at EPYC and HPE because of the security security features they have like this that you mention.
I get why this is something vendors and customers would want. However, I feel that this feature is something that really should be toggled when ordering said system ahead of time. Let the end users that want PSB enabled, order their systems from the factory as such. Otherwise it is left off for the rest of the ecosytsem.
This looks more like giving control of CPU to system vendors.
And it seems like any key leak from the vendor means doom for the “security” of the entire CPUs locked to that vendor? Is there any key revocation process in place?
There needs to be a revocation process! What happens when the big cloud providers are done with the hardware and wants to sell it off to get some return for it? It’s a huge waste and adds to an e-waste problem. Part reuse needs to be an option!
I think I’d rather purchase a TPM than have my CPU locked – and what about BIOS upgrades, if there’s any means to do that then the fuse is security by obscurity (or difficulty).
With a TPM it won’t boot without a password, but worse come to worse you still have all the pieces (and can start over). With a blown fuse there’s a tiny chance of losing an expensive CPU (and no mention of support / insurance in the event of damage).
Do they discount their CPUs because they know you’ll only be buying their servers. Would anyone buy something (home / car / stereo / cellphone) knowing that resale value or transferring ownership had severe limitations.
Why it couldn’t be an EEPROM that required keyboard (maybe even IPMI keyboard) entry of a password to upgrade the BIOS isn’t explained.
Not a fan of this ‘helpful’ feature.
Do you have a second source that AMD is enabling this feature you are calling “Platform Secure Boot”?
Guys the purpose of this was to get the discussion started. All great points on the implications.
do-better-journalism – we have statements about a publicly documented feature from AMD and its partners, and it is a documented feature. I am not sure what you mean here.
Likely lifecycle of these systems are as delivered from the factory – that CPU will live inside that Server for the entirety of it’s life. Someone who purchases the system after 3-4-5 years are paying a small fraction of the original list price – and will get what they pay for.
None of these systems will be upgraded at all during the period they are being used by their original purchaser. None. Buy them completely configured, then run them into the ground over that 3-4-5 year period – sell to some entity that sells used hardware – and forklift in new systems.
Only affects a tiny – 1% of the server market at most – as very few Epycs are being used in the first place, and a fraction of those are Dell.
It does suck for small operations like this who like to test enterprise gear – even limited use due to major architectural flaws like Epyc.
When it is not about money, it is about money.
Having the “fuse” be reversible/replaced with a hardware switch is weaker from a security standpoint: you are no longer protected against an insider threat physically undoing your protection.
I think a lot of people commenting are themselves in the secondary market and bemoaning that it’s made your life harder, but to be quite blunt, you are not the customer. At best you are an afterthought, a way to recoup some value from decommissioning old gear. For the large buyers, the security argument is going to be much stronger than making life easier for recyclers.
From an electrical engineering standpoint, the blowable fuse/circuit can be designed in such a way that various security features that depend on root of trust will not operate, effectively rendering inside attacks moot. Ex – the fuse can completely disable registers needed to read in a vendor security key required for the system to cryptographically sign that it is indeed the system it is saying it is with its hardware root of trust intact (ie, the registers will always report the key is 0 in all bits). The system will still boot (if other security features are disabled, lets assume the hypothetical attacker did so), but it will (Rightfully so) immediately give itself away on the datacenter network as having had its security compromised because it will be incapable of reporting itself with the correct cryptographic key since it is incapable of accessing it, while also being incapable of decrypting any locally stored data (ex in persistent dimms or on persistent storage such as SSDs) which was encrypted using the hardware root of trust key that it can nolonger read. The datacenter and the data on the system is thus secured from the attacker.
@Bob Dobbs
Locking the CPUs to the motherboard destroys much of the recycle value of old servers for parts. This affects any customer who was planning to eventually sell the servers as used surplus when they are no longer needed. Not only do people who buy them after 3 to 5 years pay less because it’s less useful, but the resulting waste means companies selling those machines make less from the surplus auction, which then needs to be taken into account in the original depreciation schedule.
Intel does have a comparable feature, its called Boot Guard, and it was rolled out in 2013 starting with Skylake. The firmware is signed by an OEM key. AMD burns the hash of the certificate corresponding to this key into the CPU, while Intel burns it into the Platform Controller Hub (PCH), which is a separate chip soldered into the motherboard. That’s why you’re able to swap CPUs with no issues for Intel systems.
Logically, the firmware belongs to the motherboard, so its root of trust should be tied to a fixed component on the motherboard, not some interchangeable component like the CPU. Tying it to the CPU results in the problems you encountered and there doesn’t seem to be any advantages in doing so. I’ve yet to figure out why AMD adopted this design.
As a consumer just hearing about this *processor* DRM damages the AMD brand in my eyes. The concept of having one-time-programmable ROM on the CPU that can be written to during normal use is ridiculous. This is a PC, not a game console. And it opens the system up to hackers where they will be able to permanently brick CPUs, with no possibility of recovery.
Until the AMD PSP has been cracked I will be seriously considering Intel for my next PC.
We buy Dell servers new and this along with the memory encryption are big reasons we are shifting some spend to AMD over Intel. I don’t care what happens to chips after we’re done with them.
If this can be done on an EPYC is can be likely done on a Ryzen CPU because they are all based on the same hardware IP (intellectual property) blocks. So once a hacker obtains access to the PSP on a Ryzen, assuming they know how to program the eFuse, they can brick the chip permanently. The only fix would be hardware replacement. Thanks AMD. Not only that, the PSP can be used a way of hiding malware on the system. So is this all really an improvement in system security, or is it an improvement in vendor control? And increased vendor control means higher profits for the vendor.
I don’t see what this provides that can not be achieved with existing technologies. Intel has had Trusted Execution Technology (TXT) for a decade or so (vulnerabilities in the implementation notwithstanding; I would not trust AMD to be immune against such anyway). TXT provides a way for a system to prove to a remote system that it runs the correct software across all levels. Also, with a TPM and measured boot + OS partition encrypted with the hash of the measured boot you can make sure that the system doesn’t boot if any layer (firmware, bootloader, etc.) is tampered with.
I don’t believe for a second that the purpose of this is not anti consumer.
If processor can be swapped to another motherboard of the same vendor then this is not a security feature but rather vendor lock-in.
So, another way to force companies to pay for ransomware makers? Now they could simply blow additional fuse essentially killing the machine.
Both PSP and IME had large security holes in the past, it would be safe to assume that they still have some.
AMD should ensure that power for programming the eFuse can be physically disconnected (via a jumper). No software running on the processor should be able to destroy it, ever. On the Xbox 360 it was possible to remove a resistor to prevent eFuse writes, if Microsoft could do that back in 2006 then AMD can certainly do so now.
This does not seem like a security feature, more like a risk. All of this stuff is closed source and security by obscurity, with the added benefit of destroying the resale value of your CPU. I don’t think customers asked for this. What we really need is to have access to all firmware source code and a way to verify INTERNALLY that the correct firmware is being run (or just reflash from a known good copy at every boot). Relying on a vendor for this seems very risky.
I also don’t see a security benefit in tying the CPU to a single mainboard vendor. It seems to me this feature was mostly demanded by Hardware vendors to the detriment of their customers. It’s also very generous of them to offer recycling (likely even for free, yay), I’m sure they do all this out of the goodness of their heart and concern for your security and the environment, not to take perfectly fine used hardware off the market to be able to sell more new hardware.
If you’re designing this, 15 minutes of thought lets you make the processor reusable.
One example approach: The whole point of a “hardware root of trust” is that it knows some crypto keys. Find yourself in a new system? Erase and regenerate those keys. Which is something you should be able to do anyway. Bingo. You’re reusable.
… and don’t BS me with “simple hardware security” pseudo-arguments, either. Blowable fuses don’t add security when all they do is trigger a massive software/microcode apparatus. A fuse can’t recognize a BIOS signing key; all it can do is inform some code that it needs to do so.
The people who design these things are ALWAYS driven by revenue first. Any actual security they provide is simply a convenient excuse that they can use to convince themselves and/or others that they’re helping.
I don’t know about Dell since we’re a HPE shop but I don’t think HPE’s firmware uses Intel Boot Guard but they use AMD PSB.
Someone at work asked if the Xeon servers use Boot Guard and it was awkward when the HPE guy says nope because it isn’t as secure.
We buy millions of dollars of servers each year and this is a feature we want.
Imo, I can understand the vendor point of view of niche use cases. For vendors, what about having a BIOS feature to disable PSB enablement? For most regular customers just toss a boot warning message about it being disabled, similar to how some BIOS configurations warn about TPM being disabled on boot.
In my opinion this is mostly a vendor lock-in than a security feature, and it makes money for AMD and DELL that will sell more new systems because the seconday market will shrink. At the same time we will see more complete system on the secondary market where you could upgrade your cpu only buying a new one, if it still exists, and the cpu without fuses blown will became precious. Think if you extend this approach to SSD disks, memory and devices in general.
For sure this approach is not eco-friendly and it will not help circular economy. About securiy, we will see: often when closed approach are used and the value of products is relatively high a solution to overcame this limitations is found. We will see.
Now, as a “primary” customer, we should check if this security model will impact in our companies (in some small companies and research center, it will impact whenever you needs to move components for whatever needs). If this is a concern simply don’t buy from vendors who are using lock-in mechanism. It’s something that is already happened in the smartphone market (where some vendors lock-in the bootloader).
As a “secondary” customer… customer of secondary market… potentially this could be a big problem leading to headcaches.
And just like with the printer cartridges, people who manage to bypass this ‘protection’ risk a FELONY lawsuit from AMD just as with Lexmark, due to the DMCA anti-circumvention provision. Yes, a FELONY for wanting to use a discarded CPU in a new motherboard.
The appropriate place for the circuitry is in a separate chip. Per se the CPU is not more or less secure if the board firmware has been hacked.
It is difficult to see thus as other than a means to limit aftermarket sale of Epycs while calling it a security feature. AMD is shooting itself in the foot here: every fool knows the aftermarket is a powerful way to bring in new customers in a market with relatively high cost of entry.
This has been a thing with Dell laptops for quite a while now. The article makes very good points about the second-hand market. I’d like to add to it with some experiences of my own that demonstrate how boneheaded this actually is.
As you’re aware, this type of locking is to support signed firmware. The issue for me is that Dell firmware has intentional backdoors. They have included a method for customers to remove UEFI setup/admin passwords by calling Dell Support, proving ownership, and obtaining a one-time code. While I understand they would rather not replace mainboards because a customer forgot their password, this system introduces multiple serious vulnerabilities.
In the firmware I have examined the algorithm used to generate this code is easily extracted from the system firmware itself. Writing a keygen/code generator is trivial. Additionally I do not believe it would be difficult to talk a Dell support rep into giving a code as needed. This makes UEFI setup passwords useless. Which makes secureboot useless as secureboot keys can be changed from setup(A good thing if you don’t trust the Microsoft key..).. Blows away the whole root of trust on the system.
I’ve contacted Dell on several occasions but they do not think this is a serious issue. On my previous laptops I just removed the UEFI module for this backdoor. On my new ones I’m stuck with this vendor-enforced backdoor.
This goes to show that locking people out of their own hardware is an absolutely terrible idea and can actually reduce security! I wish there was some way to push back against this sort of thing.
Interestingly enough, HPE responded today and are now saying they do not use the AMD PSB eFuses after I asked this last week. Added the statement on how HPE is implementing it differently than Dell, on page 1.
Congrats Patrick, you made it to the #1 tech site in Belgium and the Netherlands. :) I’m glad more people find STH and consider it to be the authority on server related tech news.
https://tweakers.net/nieuws/171956/amd-platform-secure-boot-van-epyc-cpus-is-slecht-voor-markt-2e-hands-servers.html
So this is actually a way to make sure you can never ever trust this server. Because the only way to trust what you run is if all the firmware is open. But your epyc is now fused to only run closed source blobs from your vendor?
Yeah, I’m in agreement with the majority sentiment here. This STINKS of vendor lock-in and money grabbing and it’s one that positively will ruin the third-party/fourth-party used parts market if all that has been said pans out true. And, as usual, IT DOESN’T ACTUALLY MAKE ANYTHING MORE SECURE! Just like the DMCA didn’t stop piracy and KYC/AML hasn’t stopped money laundering, this won’t stop the determined uberhacker from working out how this system works and completely circumventing it.
While I agree wholeheartedly on the motive to improve system security and firmware resilience to malicious threats targeted at the motherboard BIOS, using the CPU as the key to this security by means of non-reversible hardware fusing is a BAD IDEA. Surely, this kind of thing can be done WITHOUT locking the processor to a specific board and system brand and model. Seems like the main chipset would be a better place to do this given that the chipset is soldered to the motherboard whereas the CPU is a normally standardized interchangeable part that can go into any system. This is analogous to the previous issue I took regarding the whole concept of brand-specific processor SKUs that I mentioned in the previous post about “SoC containerization”.
AMD’s development teams have delivered some major advances over Intel as of late and it would be to their detriment to try and set this as a precedent. Of course, I’m saying this while knowing that enterprise buyers really couldn’t give two shits about the non-enterprise buyers and what they want and that all of the manufacturers are pursuing this concept in some form and to some extent. I’ll place blame on BOTH the manufacturers AND the enterprise market buyers for this as I feel they are ALL complicit in the whole “money first, environment second, end users’ wishes not at all” marketing model.
Now, you might be reading this and asking “Okay Stephen, what is it that YOU stand to lose in all this? Are YOU a third/fourth-party used parts vendor?”. Well, to be honest, no, I’m not a used parts vendor. However, I AM an amateur chip collector. It has been one of my hobbies for many years, probably since my days in vo-tech school taking classes for my Diploma in computer technology.
AND I RELY COMPLETELY ON THIRD-PARTY AND FOURTH-PARTY SOURCES FOR THE CHIPS AND PROCESSORS I BUY!
I rely on these so-called “grey market” (not a term I prefer to use) vendors for the parts I buy because it is usually the only way I can obtain otherwise hard to get CPUs and other devices. I don’t have lots of money to buy processors brand-new-in-box when they come out. It’s just too expensive. And being unemployed and broke since the end of January 2009 has done me NO FAVORS in dealing with that.
Currently, I buy my stuff from third and fourth parties on Amazon as it’s the only place that I know of that actually sells their gift cards in the local stores where I live. I have yet to see any eBay gift cards so I don’t use that source. I can often find decent chips for anywhere from as low as $3 to up to $20, some of them quite good parts like a 12-core AMD Opteron 6174 or a 6-core Intel Sandy Bridge Xeon E5-2620 in LGA-2011. These were expensive when new and now, after ten-plus years of waiting, I can buy them dirt cheap. That is the beauty of the third-party/fourth-party used parts marketplace. I can buy the processors I’ve always wanted (and there are many, many more I want that I don’t yet have) without having to look for a job that doesn’t exist, work my ass to death and still not have the money to buy them because, for example, Intel wants some $10,000 or so for a Xeon Platinum CPU or AMD wants over $3,000 for one of their Ryzen Threadrippers. Never mind the Cavium/Marvell Thunder X1s and X2s, the Qualcomm Centriqs OR Ampere’s shiny new Altra chips, all of which are currently in the “unobtanium” category right alongside IBM POWER-anythings and z/-anythings, MIPS-anythings, SPARC-anythings or any other mainframe/supercomputer/server processors you can divine from the ether.
This security “feature” of AMD’s will likely make any newer processors more scarce and much harder to buy in the future as more people catch on to this and start sending perfectly usable hardware straight to the e-waste pile. This will hurt me as an amateur CPU collector because this will mean that I may never be able to buy a specific model or type of processor due to them being unavailable. It’s already hard enough to buy used chips as it is just due to the fact that many of the ones I don’t yet have are now old enough to where there’s a really good chance that I might never be able to buy them on account of rarity due to the whole “scrap it for the gold” craze I’ve seen among electronics “collectors” (really just scrappers but who legally purchased, rather than stole, the items they scrapped). Who knows how much of our technological legacy we are losing to this behavior and attitude. Today’s commonplace blah-blah will be tomorrow’s sexy retro-chic (vintage ’80s Commodore 64, anyone?), BUT ONLY IF WE DON’T SCRAP ALL OF IT AND ONLY IF WE STOP ENGAGING IN DESIGN AND SALES PRACTICES THAT MAKE IT OKAY TO WASTE IT IN THIS WAY!! Not to mention the right-to-repair aspect to all of this…
Personally, I think new technology costs far, far too much. I cannot justify upgrading every year or two just because everybody else is doing so. This “security” move seeks to continue that wasteful model of “Trash your old machine! It’s only two years old and still works just fine, but trash it anyway because WE just released this NEW, MORE ‘SECURE’ one for you! You know you want it so just buy it already! Don’t worry, WE’LL TAKE VERY GOOD CARE OF YOUR OLD ONE when you ship it back to us, AT YOUR EXPENSE of course! HEH HEH HEH”. You enterprise buyers can say what you will, but you are a player in this nightmare. Just remember the little CPU collector guy who’s desperately trying to save a small piece of our technological legacy before it all ends up in the scrap recycler’s furnace.
AMD should realize they are tarnishing their own brand reputation with the inclusion of the PSP and the recent CPU lockdown.
Even though it’s a server CPU that’s affected by the lockdown, stories like this are definitely not well received by the enthusiast and gamer communities and draw attention to such anti-features like the PSP. Knowing that there’s a special processor inside the CPU specifically designed to prevent you from unlocking cores, etc. would NOT be good PR for AMD at all. I am using a Ryzen system right now and I regret buying it, I wish I went with Intel instead. At least the management engine has been cracked, unlike AMD’s AFAIK.
It’s about time we looked into a legal response to this behavior, just as with John Deere farm equipment, it will likely not stop unless fines are imposed or some kind of consumer boycott occurs.
Regarding the CPU lockdown, even Intel wouldn’t do such a thing. Surely isn’t it anti-competitive to lock the CPU to a specific system in this way? What would the EU think about this regarding e-waste and recycling? And I believe in Australia the ACCC would crack down very hard on such shenanigans?
These kind of DRM-like ‘security’ features starting being implemented first with phones and consoles and then has spread throughout the entire industry like cancer.
Many of the features of the AMD PSP could be implemented as hardwired logic, no need for a CPU for that. And thus no chance of malware being able to run undiscovered.
It’s like Orwellian doublespeak, in fact the Platform Security Processor might well be making the entire system less secure. Because we cannot inspect the content of the eFuse ROM how do we know if a state level adversary has placed code in there to weaken the system security?
Note: On the nVidia Tegra platform the eFuse ROM can contain executable code to patch the boot-up process, as Nintendo has done with the Switch console. It’s likely that AMD has such similar functionality.
So the PSP could be cracked, and then CPUs can be eFused with malware before shipping the server, and nobody would know that there’s an easily exploitable vulnerability now present.
I guess one of the real purposes of the PSP is to protect AMD’s security and prevent the user from unlocking disabled cores, boosting clock frequencies, retrieving HDCP keys, etc. on both CPUs and GPUs. So it’s partly to prevent the owner from doing what they want with the hardware.
A PSP or Intel ME related hack involving a SCADA system would not be discovered until it’s too late, with potentially extremely severe consequences. AMD is advertising the processor’s PSP as being a security device that is intended to enhance system security. If such a SCADA hack involving the PSP was to result in loss of life for example, what would AMD’s liability be in such circumstances, where the ‘security device’ itself has enabled the system to be hacked in the first place? Taking into account that the ‘security device’ cannot be disabled by the SCADA operator, so they have no choice to use it.
It doesn’t have to be directly controlling the safety critical SCADA equipment, it can be something as indirect as a contractor’s laptop containing Russian / Chinese PSP malware, and this allowing the secure SCADA network to be breached.
The fact is that because of this PSP and eFuse now processors might be considered storage devices, just like hard drives, of which there is a possibility for malware to be present. This is extremely concerning if you are at risk of state-level actors hacking your system (e.g. electrical / telecoms infrastructure operators).
That is why I believe the PSP and ME should be removed completely. Should that not be possible it should be replaced with a processor that is transparent to its internal operation. Once that is easily auditable by the user, and not trying to deliberately hide it’s internal operation from the owner.
Unlocking the EPYC CPU could be done with some kind of JTAG modchip, but this depends on what kind of JTAG security AMD has implemented.
If you want to know where to start, search GitHub for ‘KaveriPI’, if you unpack AMD BIOSDBG.EXE you can find a complete list of processor registers. This is all from 2015 but the PSP is documented in there.
There’s also a Microsoft Access database which has all the JTAG registers, but I don’t have the time to decode the meaning of the fields… It is likely that things have changed (a lot) since then but it still might be enough for a start.
Should the JTAG interface be protected then some kind of fault injection (possibly laser based, unfortunately) might be required to open it up. I guess some of the eFuse bits can be overwritten, maybe there’s a combination which can remove the lock. An innovative recycling company could work on making a jig to automate this somehow…
Some PSP JTAG stuff here (publicly available material from GitHub in 2015, fair use applies):
41469,3529,164000,164999,”SMU_PSP_efuse_ovr_tried”,,1,0,0,0,50,”0000″,0,
41470,3529,164000,164999,”SMU_PSP_FRA_pass_ld_err”,,1,1,1,0,50,”0001″,0,
41471,3529,164000,164999,”SMU_PSP_FRA_pass_ld_cor”,,1,2,2,0,50,”0002″,0,
41472,3529,164000,164999,”SMU_PSP_efuse_pdmb_aes_dis”,,1,3,3,0,50,”0003″,0,
41473,3529,164000,164999,”SMU_PSP_efuse_pcpu_dis”,,1,4,4,0,50,”0004″,0,
41474,3529,164000,164999,”SMU_PSP_efuse_ccp_cyph_dis”,,1,5,5,0,50,”0005″,0,
41475,3529,164000,164999,”SMU_PSP_efuse_FRA_en”,,1,6,6,0,50,”0006″,0,
41477,3529,164000,164999,”SMU_PSP_efuse_proto”,,1,7,7,0,50,”0007″,0,
41478,3529,164000,164999,”SMU_PSP_efuse_secure”,,1,8,8,0,50,”0008″,0,
41552,2352,164000,164999,”SMU_PSP_hard_resetb”,,1,31,31,0,50,”101F”,0,
41553,2352,164000,164999,”SMU_PSP_early_resetb”,,1,30,30,0,50,”101E”,0,
41554,2352,164000,164999,”SMU_PSP_slv_mbus2_reset”,,1,29,29,0,50,”101D”,0,
41555,2352,164000,164999,”PSP_SCAN_MODE_STICKY”,,1,28,28,0,50,”101C”,0,
41568,2352,164000,164999,”PSP_AEB_307_PCPU_RST_DLY_TDR_en_pclk”,,1,15,15,0,50,”100F”,0,
41569,2352,164000,164999,”PSP_AEB_304_PCPU_FORCE_rst_en_pclk”,,1,14,14,0,50,”100E”,0,
41579,2352,164000,164999,”PSP_Resetn”,,1,8,8,0,50,”1008″,0,
43605,555,164000,164999,”PSP_ENABLE_SPARE”,,0,1,1,0,50,”0001″,0,
43642,555,164000,164999,”PSP_SPARE”,,0,7,14,0,50,”0007″,0,
Search Twitter for “EPYC schematics” and find the post by RetiredEngineer to get the JTAG pads.
From what I know already with some of their old chips is that there is a short period of time after deasserting the reset signal where ‘protected’ JTAG registers may be accessed before the SMU (an older on chip CPU for housekeeping/etc) starts to lock things down. So you can disable the SMU before it gets a chance to shut down access. There is a slim chance that that persists on their newer hardware.
Sorry for the spam, just so outraged at CPU DRM and that reusing old CPUs might become a crime under the DMCA (that’s what blew my top).
@ argb
Oh God, I wasn’t even thinking about SCADA systems until you brought it up! Jesus, that’s even WORSE than the damage this could potentially do to the used parts market! And then there’s the gamers and home HPC nerds that this will also hurt. Because you know the tech industry suffers from a very BAD CASE of “monkey see, monkey do” syndrome. Whatever one company rolls out, EVERYBODY IS GONNA WANT SOME OF! Why stop at servers? LET’S PUT THIS “SECURITY” FEATURE IN ALL OF OUR PROCESSORS!!! Kind of puts my Mom’s classic “cat poo” meme right to shame!
Nothing good is going to come out of this and everything you have said is EXACTLY the narrative of the Right to Repair movement. Everyone from Louis Rossmann on down is saying the same things. But alas, we are all at the mercy of our governments. You know governments don’t like to listen to the citizenry. I guess it’s going to take a nation’s spy agency tearing into the national grid of another before our stupid businesses and government “leaders” finally get a clue that this shit can’t be stopped the way they’re doing it. But it sure is effective at taking our freedom to use our own property the way we see fit. Sorry, but I won’t be subjugated so easily and I hold up the damn Pirate Bay as my proof. DMCA and “digital rights management” HAS NOT ONCE stopped them. And I believe the AMD platform security processor thing ain’t gonna stop an uberhacker any more than a speeding bullet to the head would.
No need to apologize for spamming. I wouldn’t call it that anyway. You’re venting and THAT’S A GOOD THING!! It means you’re not holding it in, allowing it to tear up your insides. In fact, I’m kind of enjoying reading your take on this. For what it’s worth, I saved a full copy of your comments, including the JTAG stuff you put up, just in case the “Powers that Be” decide that you’re a “criminal” to be dealt with and your stuff gets deleted.
I’m with you. There NEEDS to be some way for EVERYBODY to push back on this stuff. I’m worried that most of this won’t get to the proper people, if you get my drift.
@Stephen Exactly – it’s taking away users’ ability to do what they want with hardware that they own (with a small chance of being prosecuted criminally – this is utterly shocking).
At least we need the DMCA’s scope limited to cover copy protection *only*, if there are other reasons for cracking the protection (e.g. for curiosity or to unlock cores or boost the clock frequency), than that should be entirely legal. Just like we can take apart our vehicles and there is no penalty for doing that.
I’ve got a lot more than just the JTAG, everything taken from public sources on the Web. On some of their older chips I have gotten code execution on the SAMU (Security Asset Management Unit), which has now been replaced by the PSP. However the SAMU wasn’t involved in the boot process (on PC CPUs and GPUs), it was just an optional DRM processor which you didn’t have to use if you didn’t want to. It’s quite fun to watch this mini-CPU running your own bare metal code BTW…
BTW The SAMU is an AM32 CPU core which is a heavily modified open-source Lattice Mico32 processor. It’s instruction set is completely different from the LM32 but if you know what to search for and which file to dump with a hex editor, you can find a complete description of all the instructions.
https://www.wired.com/2015/04/dmca-ownership-john-deere/
What’s the post code for a locked down epyc? Suspecting I’ve been dealing with this issue.
This is vendor lock-in at its best. Instead of getting used grey-market cpus you’ll have to pay 5x the price and buy them direct from Dell at a premium (and they’ll likely bump up the prices because they’ll know you have no other choice).
IMHO, this should be investigated as antitrust practice.
How long until a government busybody demands backdoor access to this “secure platform”? How after that until the backdoor gets leaked? I’m writing this on a computer built around an EPYC chip. It will be my last.
Can yout check the Threadripper Pro, it looks like they have the same vendor lock.
Lockdown is a big thing these days, and the excuse is always “security”… in the world of homosapiens there are all kinds of viri which give govs reason to enforce lockdowns… and in the world of computers there are also all kinds of viri which giv vendors reason to enforce lockdowns…
…what many are beginning to realise is that ‘lockdown’ is a politically correct vessel to achieve control.
Back in the old days we understood the need to build natural resiliency against viri, and in the world of computers we understood that everyone on a level playing field benefited from proven technology, not like all the fiasco with custom firmwares making SSDs go pop after 32768 hours lmao.
Intel played a clean game keeping away from lockdowns.
AMD has opened the door to a very messy future, thank you AMD.
My next dog will be named, wait for it, drum roll…
Lockdown.
Easy just don’t buy from Dell, since they are the ones doing it.
Herm… I remember seeing a strange case of the same thing using Dell servers running Windows Server with Intel CPUs. Once I tried to repurpose the server to Unix, the security features kept from booting, and I had to disable all the BIOS security features to run my Unix OS on it. Dell was already practicing vendor lock-in with Microsoft… there was this big uproar about it too… ? So its not only CPU lock-in, it’s also OS lock-in.
Just as I was about to suggest R6525s and R7525s to my company I see this, looks like we will be going Intel and Supermicro instead who thankfully don’t exercise this “”security feature”” (vendor lock).
This only creates more ewaste and is an obvious attempt to kill off the second party and further markets so they can get them to buy new servers. This affects both the first party given that these servers and chips will devalue much faster as well as anyone further down the line who wants to buy them.
Anyone who is trying to make the security argument like above and doesn’t care about the next guy is brainwashed and should seriously start caring about the impact of computing as a whole on the environment. It is obvious this has nothing to do security on the premise alone that it isn’t a motherboard lock, I even expect to see in the future that the lock won’t even be burnt automatically into new CPUs either and you will be forced to buy your CPU from your motherboard/server vendor otherwise it won’t work. Don’t worry, it’s for security!
There is going to be many ways to improve AMD’s platform security that don’t intentionally create ewaste, maybe AMD should start by open sourcing their PSB code too. Or maybe let people completely disable it, like the US government does with Intel ME because “it is a security hole”.
This is not a feature we want, we do resell our hardware here to recoup initial investment costs and without a doubt this will devalue it significantly. Nor will it increase security given that PSB is not open sourced. I’d also really like to see how few instances of servers not having this specific feature has actually allowed attackers access into the system. Given that other Epyc board vendors are not implementing it, I doubt it is even a single one.
Well it turns out that researchers have managed to totally hack the PSP by just reducing the supply voltage to the processor. Even on the latest Zen 3 architecture. Nothing exotic required.
https://arxiv.org/abs/2108.04575
Some interesting stuff in that doc, I wonder why does the PSP need anything to do with WiFi (p. 49)?
0x3d WLAN Umac
0x3e WLAN Imac
0x3f WLAN Bluetooth
0x88 WLAN firmware copied to MPM memory by MFD and then MPM will send this to WLAN
Just imagine if that WLAN firmware was remotely exploitable somehow..
Actually it’s a rebranded MediaTek MT7921K.
Basically:
Computer manufacturer: Hey AMD, lets help making our hardware as useless as possible on second hand market.
AMD: Hi PC manufacturers, lets help making our processors useless on second hand maket
AMD & Computer manufacturer: $$$$ as greed can get….
Everyone else: :-/