Bloomberg today came out with an industry shocker. The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies. In that article, Bloomberg reports that the PLA managed to infiltrate Supermicro’s supply chain and add small chips that allowed Chinese agencies to hack into 30 companies such as Apple and Amazon. The company also published, in a different article, statements from Amazon, Apple, and Supermicro strongly rebutting the story. See The Big Hack: Statements From Amazon, Apple, Supermicro, and the Chinese Government. Something is certainly strange here, and at STH, we review more server platforms than anywhere else on the Internet, including those from Supermicro. We also, by chance, started diving into the BMC security space more recently so it is clearly time to investigate.
What is this hack?
According to Bloomberg, the hack involves a small IC inserted into the Supermicro motherboard PCB. In previous generations, this would have been a surface mount component. The story claims current generations have these devices embedded in PCB.
There, of course, has to be much more than a simple chip. That chip needs to tap into electrical signals both for power and for data transfer. That means that not only must a component be inserted, but also PCB wires. Bloomberg says it is in line with memory to CPUs to intercept some password validation code. By changing this code in Linux, it allows remote attackers to access the servers and potentially phone home.
That is a little strange frankly from a technical standpoint. Where could these chips be located?
DRAM memory traces are very complex. 288 pins per DDR4 DRAM module (not all are data of course), times 8 modules per CPU times two CPUs and that is a lot of pins to monitor from such a small IC. Even in the older 240-pin DDR3 generation, with 16 modules there is no way a small IC can monitor that many wires. Also, memory traces in motherboards are often an area where PCB designers spend a lot of time to get correct lengths and timings on the wires. Inserting a small IC would not be the easiest feat there.
The other candidates are more probable. The first is using the onboard SATADOM wires. SATADOMs are small flash memory devices used to load base operating systems. SATA cables are 7-pin designs with three ground wires and two A/B +/- pairs. Supermicro SATADOM connectors have an extra power capability.
This would be a lower pin count option to exploit. The problem, of course, is that most large shops encrypt data on the SATADOMs. Most SATADOMs do not have self-encrypting capabilities which means it is host encryption. The Bloomberg article said that the hardware would intercept storage to CPU transfers. If the data is encrypted when transferred, it would be nearly impossible in that IC footprint to crack reasonable encryption and change the OS in-line.
The final, and perhaps most likely vector would be the BMC. We have a piece Explaining the Baseboard Management Controller or BMC in Servers. A hardware chip that could impact the BMC firmware is more probable.
Each BMC has local storage ever since the 1998 IPMI 1.0 spec was announced.
This is generally a very small flash module for storage, often a few MB in size. The BMC usually runs a flavor of Linux. Getting root access to the BMC is bad, but it is not the same as getting full access to the main server OS.
The BMC has root console access to the server. It is on before the server boots. It can mount media and has network access. Think of it as an administrator sitting at the machine, but bringing that functionality anywhere in the world.
BMCs are amazingly hacked devices. The Bloomberg story’s comments from Amazon and Apple both point to the BMC and IPMI firmware/ management interfaces. We think this is the most likely vector.
The bad news is that BMC’s are extremely dangerous. They are also pervasive with a few points under 100% of servers having them these days. The Bloomberg article cites the well-known Supermicro BMC/ IPMI vulnerabilities. Supermicro is not alone. Every Dell EMC PowerEdge server (edit: 13th generation and older, the new 14th generation has a fix to prevent this) has a local and remote exploit available that the company can mitigate with patches, but cannot fix. We broke this story with iDRACula. If you think you are safe with HPE or Lenovo servers, here are BMC vulnerabilities for other vendors.
The security community, as a whole, knows that BMCs are both useful if not mandatory in today’s infrastructure. As a result, the security community, and major hyper-scale vendors are putting a lot of effort in researching security solutions.
One of the more interesting bits is that if it is a BMC vulnerability or anything that “phones home” over a network interface, one would expect that security researchers would have seen it. There are companies that put boxes on networks just to see what network traffic they create. Supermicro tends to build common designs that it ships to multiple customers. It would be slightly interesting if only some Supermicro servers, e.g. for certain customers were impacted. If China did not do this, it would have been caught earlier. If China did limit to a few customers, it would be difficult to target them at PCB. As we will show shortly, Supermicro PCBs are used across products.
Bottom line, if this Supermicro attack vector is to the BMC, then the Bloomberg story is no bigger than the Dell EMC PowerEdge iDRACula story or any others. Saying there is a vulnerability in a BMC is like saying the sun is hot.
Some higher-resolution areas of MicroBlade BMCs
We had some similar generation Supermicro MicroBlades where we could provide higher-resolution photos of their BMC areas. This is where the hacked chips are located on the board that Bloomberg depicts. This also shows that a Supermicro PCB is spun for multiple products. That makes it extremely difficult to target a specific customer at the time of PCB construction. Here we have two different products built on the same underlying PCB.
For our less technical readers, this is what the actual PCB looks like. For our more technical readers, you may want to see for yourself.
Here are two MicroBlades of that era the Supermicro B1SD1-TF and the B1SD2-TF. The “2” represents that the PCB houses two complete server nodes. We highlight this because if the attack is present on this platform, presumably it would require a second inserted chip which would not be required on the B1SD1-TF.
There are a ton of ICs there. I know we have STH readers who will want to look. Have at it.
The Counterpoint Published Outside of the Main Story
Bloomberg posts statements from companies, not in their main article, but linked in a separate article.
Amazon, Apple, and Supermicro all deny that this is happening.
Just for a taste, here is an excerpt from Apple’s statement:
“We are deeply disappointed that in their dealings with us, Bloomberg’s reporters have not been open to the possibility that they or their sources might be wrong or misinformed. Our best guess is that they are confusing their story with a previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs. That one-time event was determined to be accidental and not a targeted attack against Apple.” (Source: Bloomberg/ Apple)
This is a little strange. All three are public companies. A simple “no comment” would have sufficed. Or a “we would not be allowed to comment on your classified source story” perhaps. Supermicro one can dismiss their lack of knowledge to perhaps the intelligence community not wanting to alert anyone there. Apple and Amazon went beyond a simple “no recollection” or “no comment” type response. They should not be allowed to make these types of responses if they are untrue since they would be potentially misleading investors. Even if they could not speak about the issues, they did not have to go into the depth that they did.
Indeed, when we broke iDRACula the persistent (with mitigations) non-fixable Dell EMC PowerEdge issue impaction tens of millions of their servers, we held the entire story while Dell EMC’s confirmation went through legal and management approvals. Having just broken a similar story, the responses from parties are in an absolutely sharp contrast.
Where the Bloomberg Piece Makes No Sense
There is one area where the Bloomberg piece makes no sense. Supermicro servers are procured for US Military contracts and use to this day. Supermicro’s government business is nowhere near a large as some other vendors, but there are solutions providers who sell Supermicro platforms into highly sensitive government programs.
If the FBI, or other intelligence officials, had reason to believe Supermicro hardware was compromised, then we would expect it would have taken less than a few years for this procurement to stop.
Assuming the Bloomberg story is accurate, that means that the US intelligence community, during a period spanning two administrations, saw a foreign threat and allowed that threat to infiltrate the US military. If the story is untrue, or incorrect on its technical merits, then it would make sense that Supermicro gear is being used by the US military.
Final Words
First and foremost, I think we need to call for an immediate SEC investigation around anyone who has recently taken short positions or sold shares in Supermicro. With the accompanying Supermicro stock price hit that was foreseeable prior to the story, if anyone knew the story would be published, and acted on that non-public or classified information, the SEC needs to take action. There seems to have been over 20 people that knew about this.
Further, with public companies making statements on the impact, unless there is a valid national security/ classified reason that they gave the responses they did, there is a mismatch. Apple and Amazon did not say “no comment” they called Bloomberg’s account false. The SEC needs to investigate here as well to see if these were publicly misleading statements.
Second, we need a formal investigation into why, if this is a true and serious threat, why it was not flagged in military procurement years ago.
There are parts of the Bloomberg story, the rebuttals from Amazon, Apple, and Supermicro, and logical reasoning which point to one key takeaway: server security is a big deal. Perhaps the bigger takeaway is that this is a 21st-century battleground that is active every day. Government agencies from China, the US, Russia, Israel, and others all have ways to impact servers and more broadly computing devices. We know the Intel management engine has been compromised. There are reports of Lenovo laptops phoning home data. It would be naive to think that any major world power is not working to get information from compute devices whether they are from Supermicro or another vendor. It is probably better to assume your server is compromised and start with that.
Holy fuck. You’re right. This has to be a BMC exploit to make Apple and Amazon line up with the Bloomberg story and the size of that chip.
This now makes sense. It’s gotta be the BMC hack. They played it up well. It’s also bad that China can get stuff implanted. Supermicro will have to shut down China manufacturing and only make in Taiwan.
Bloomberg article says that the rouge chip was connected to the BMC:
“The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.”
I think you might have placed the B1SD1 board photo twice
https://www.servethehome.com/wp-content/uploads/2018/10/Supermicro-B1SD1-TF-BMC-Area.jpg
https://www.servethehome.com/wp-content/uploads/2018/10/Supermicro-B1SD2-TF-BMC-Area.jpg
After the recent hit pieces against AMD I am also curious if the author or Bloomberg have a short against Supermicro or own stock in Dell, HPE, etc.
https://www.supermicro.com/newsroom/pressreleases/2018/press181004_Bloomberg.cfm
Great job STH. Bloomberg did say it was a BMC hack but your analysis is spot on.
Questionable Bloomberg journalism
“This happened at a crucial moment, as small bits of the operating system were being stored in the board’s temporary memory en route to the server’s central processor, the CPU. The implant was placed on the board in a way that allowed it to effectively edit this information queue, injecting its own code or altering the order of the instructions the CPU was meant to follow. Deviously small changes could create disastrous effects.”
CPU on a server is the Xeon. BMC is a SOC. They played it up and lied to make the story.
The US Intelligence orgs would let it go forward. IE let the Chinese think they are getting away with while secretly hacking the Chinese hack. Spy craft is very complex, but the US has done that many times before.
It’s all Trumps fault… if he would have just let China keep taxing our exports and not taxing theirs, they would have left us alone.
It’s interesting that Supermicro was delisted from NASDAQ 2 months ago and this happened afterwards.
Or perhaps the current administration wants another boogieman to scare voters into voting for him, to distract from the russia boogieman. And to further hurt china’s manufacturing industry while encouraging companies to start producing more in the US, in a way that tariffs just by the US would fail to do.
SuperMicro stock today
12.60 -8.80 -41.12%
so down 41percent says someone believes the article to be true.
What’s going on with these comments? I read the article. It’s showing how and why it’s gotta be the BMC. There’s many attacks for everyone’s BMCs. It’s showing that there are denials by companies who cannot legally deny this in the manner they did unless it is indeed untrue. It’s saying that there may be others with gains. Seeing supermicro’s stock that’s for sure someone profited.
The assumption that ” that security researchers would have seen it” does no stand long if the chip is designed to be remotely triggered, like by some targeted incoming network traffic waking up the chip.
What about sneaking things in with the RDIMMs or LRDIMMs on the board? They buffer all the Dram signals right, RDIMM registers the signals as well so timing skew won’t be an issue with the hack.
“so down 41percent says someone believes the article to be true.”
With the stock market it only takes one or two big sellers to panic, especially with automated trading, to trigger a panic sell off, which is what this is. Panic sell offs on the stock markets are pretty much the norm rather than the exception. This is why there are rules and laws to prevent manipulating the market with false or insider information.
I’m more concerned about Bloomberg’s poor quality reporting on highly technical topics they aren’t entirely qualified to expound upon. Having read through the report, this is obviously a person that’s got enough knowledge to appear credible to the Average Joe, perhaps even the Above Average Joe, but not good enough to answer the real questions to be posed, and some were objectively raised by the rebuttals from Super Micro, et al, and STH.
It doesn’t appear to me that Bloomberg’s reporter actually did any real checking of their information, only asking for corroborating quote solicitation from the industry players and when they couldn’t get them, ignored any claims from those involved the reporter’s understanding of the events is probably faulty.
This reads more like a politically motivated hack piece with superficial hand waving of purported unsubstantiated facts to support an Anti-PRC stance from the government.
I very much agree with the STH’s author’s caution here. There’s much in this article that doesn’t add up, and it seems to me that the Bloomberg reporter is trying to hype sensationalism over an already known problem in cybersecurity. Caution is advised in taking mainstream media outlet reporting on technology and science (and law) at face value. They usually get it wrong where it counts.
That said, we do need to be cautious about doing business with the PRC even if this turns out to be a mole hill, or worse, never happened at all in the way the article purports. Transparency of design, IC layout, open specifications, open firmware, and open software are the only ways to reasonably assure the reliability and security of computing platforms. This situation illustrates this simple principle even if the article proves substantially false, and even more so if it’s substantially true. We’re long past the point where we should remain tolerant of business as usual from the likes of Intel, Nvidia, Qualcomm, ARM, TI, and other corporations that are hostile to the openness required to verify the security of the platforms modern society, government, and commerce depend upon.
Good work. I appreciate it.
Great article, I really like your post keep. Thank you for sharing information.
Thanks for sharing keep it up we supported you.
STH and Patrick are the world’s server gurus.
The great post you have written well I will try to visit there once thanks a lot.
Thank you for the sharing. Nice
” Transparency of design, IC layout, open specifications, open firmware, and open software are the only ways to reasonably assure the reliability and security of computing platforms.”
I agree with Ross above.
But don’t you know all the non-standard IP they’re pushing is the cash cow they rely on for business?
The only way to displace that is to make replacements, as quoted, that are superior in every way and still check the box of profit.
Unfortunately, up until very recently, security was just a PR stunt. And, based on this article by Bloomberg, the press still think it is.
And no, I don’t lump STH in with “the press.” Keep up the good work Patrick!
Why is everyone acting like Sherlock Holmes for pointing out “it’s gotta be a BMC hack”?
Bloomberg literally said in their original article it was a BMC hack. Zero points awarded for saying it again.
“The illicit chips could do all this because they were connected to the *baseboard management controller*…”
I understand the level of technical suspicion that can be generated by a general news article like this. Then again, had they written it to answer 100% of every question, nobody would read the thing because only EE’s would understand it. Nitpicking the thing to death, given it’s audience, is less than productive. However, asking hard questions about some points is something everyone should do.
Honestly, I give Bloomberg the benefit of the doubt (for the time being). Nobody levels wild accusations implicating not one, but TWO Trillion dollar companies without having some serious senior level management buy in. It goes without question to assume that there were some strong and tenaciously hard questions being asked of the author(s?) before this information ever even sniffed at publication.
Finally! A decent analysis of the situation.
The US goverment did this a lot easier they just asked the Intel, Asmedia, ARM to give them a backdoor (and of course the complied).
I would be shocked if the Chinese government hasn’t been doing this for years. Check out Trenton Systems for a US design and manufacturing computer company.
Why don’t they report that “Intel Inside” means “Spec Violation Inside” and “Security Risk Inside”?
According to Intel, the patch for the “Foreshadow” security problem can’t eliminate the risk but only reduces it!
By reading the comments here, it’s hard to believe this is a hardware forum or maybe nobody read the original article. “A tiny chip that is smaller than the size of a sharpened pencil tip”? Are you kidding? Please show me there even exists another packaged IC of that size and functionality.
1. US hacks and inserts backdoors. A lot. So if a hack indeed happened, this is not something extraordinary, just a usual government sandbox game. Its the thing happening since Adam and Eve.
2. Exact affected motherboard models with pictures of chip. There are none. So until now, its only Bloomberg article that hacks the minds.
This is a hit job on a Taiwanese American to manipulate stock prices. management controllers are vulnerable, everyone knows that.
What I want to know is how did STH get this up so fast? You’ve got more depth on the probable how and implications here than other articles that came out a day or two after.
Thanks for sharing keep it up we supported you.
This is a very awesome article. thank you for this
thanks for sharing such an beautiful article i liked this post so much and also it contains great knowledge
I have never felt so good… helps with so many ailments read the information on their website
I am very satisfying to read this article. Thank you for sharing it.
I am glad to read these topic, Thank you for sharing it.
some of the city’s top shawarmas. Even a perfectly grilled Chicken and Paneer Shawarma Pizza is available at this tiny establishment. Try everything here, we say!
A fantastic source of vitamin D is eggs. Eggs Help You Manage Your Weight and Are Filling. One of the best dietary sources of choline is eggs.
In search of the best pizza restaurants in Noida? Order your favourite pizza for delivery to your home from the closest city pizzerias.