Today, Bloomberg upped the stakes in its contentious story around Chinese tampering of Supermicro hardware, citing Yossi Appleboum CEO of Sepio Systems. Here is that story. I reached out to Mr. Appleboum for comment via telephone. Whereas the Bloomberg story singles out Supermicro servers, Mr. Appleboum’s sentiment is that this is an industrywide issue. Other very large server and networking manufacturers are certainly impacted, perhaps more so. He also stated that as an industry, or a society, we have two options: we go with the narrative that a US company, Supermicro, is the only one impacted as the Bloomberg reporting suggests, or we recognize that this is a persistent threat that impacts the entire hardware supply chain that underpins the lynchpin communications infrastructure of our global economy.
Background Perspective for Our Readers
Before we proceed, I wanted to present some personal bias. At STH, we review servers from every major server vendor. As a result, I speak with a lot of people in the industry. I have visited manufacturing operations for different companies and we also work with some major component suppliers. STH focuses on deeper technical content, but this needs to be an industry discussion. For our more technically minded readers, here is what an RJ-45 network port, at the heart of today’s story looks like:
For the record, I personally believe that there are hardware and software attack vectors against baseboard management controllers, as well as other components in modern systems. I believe that there have been attacks on BMCs and network hardware against every major server vendor in the US, China, and elsewhere. Positioning this industrywide issue as a Supermicro issue is irresponsible at best. Case in point, we unveiled iDRACula Vulnerability Impacts Millions of Legacy Dell EMC Servers which allows BMC firmware tampering on Dell EMC PowerEdge 13th generation and prior if a malicious actor has either physical access or remote access with valid credentials. This is not a single vendor issue. This is a global issue that participants in the industry must tackle.
My Discussion with Yossi Appleboum
In the 4th paragraph of the Bloomberg piece cited above, I noticed that he seems to refer to this as not just a Supermicro problem. I asked Mr. Appleboum about this:
We found it in different vendors, not just Supermicro. We found it not just in servers, in different variations, but hardware manipulation on different interfaces, mostly in network related. We found it in different devices connected to the network, even Ethernet switches. I am talking about really big what are considered to be major American brands, many compromised through the same method.
This is why I think that Supermicro has nothing to do with that. In many cases, by the way, it is not through manufacturing, it is after through the supply chain.
People think of the supply chain in a very narrow sense between the manufacturer and the customer. Supply chain never ends. There are technicians, there are integrators, there are people that work in your facilities. We have seen after installation, after the fact attacks where someone switched something already installed. This is why Supermicro would have no idea what happens later in the supply chain.
I asked Yossi if the supply chain can include those integrating racks or those moving boxes. He said:
Totally possible. We have been a witness, or have been involved in an investigation, not in the US by the way, but who cares. Eventually, the one who switched the box was a guy who got money to switch boxes during the shipment.
We have a problem. The problem is the hardware supply chain. All of us are dealing with what happened to Supermicro, and whether Amazon knew or did not know. That is not the main issue for me. The main issue is that we have a problem. It is global. This is why I think Supermicro is suffering from the big players. I am talking about the really big players who know that they have the same problem, and they are kind of using the story right now to throw Supermicro under the bus instead of coming out and saying that it is a global problem, let’s fix it and find a solution.
I asked Mr. Appleboum for his thoughts around the silicon root of trust or hardware root of trust that hyper-scale companies like Google (with Titan) or other vendors like HPE and Dell EMC are implementing in their newest servers. I asked if that is the answer to the supply chain problem.
It’s a good start but again since you do not really what is going on in your supply chain, you don’t know where it is going to happen in your supply chain. Having a root that will vouch for the hardware is a good start. The problem is that when you get the hardware how can you make sure the product was not compromised? Someone can replace modules that validate hardware with other modules that say it is okay. We have seen things like that in our past.
There are two things that we need to do. One as a society come back to the ones that manufacture for us and say that they cannot let this happen or it will be the end of business for you. That is politics. The second one is technology-wise, this is why we started Sepio. I am not going to do a marketing pitch around that. Companies should have a policy that says my hardware is an immediate suspect, just like my software. We are spending $100B on software related attacks, but near zero for hardware attacks. That is irresponsible and that is the problem that we need to fix.
I asked Mr. Appleboum if he had a sense of how much happens in the manufacturing process versus after.
I don’t have the numbers or statistics around how much it happens in the manufacturing process or how much it happens after. My educated guess is that most of it happens later because it is much easier and much harder to find the source. If you do this through the manufacturing, eventually there is a name and address to go back and ask questions. If it happens after, you go where? Who is going to answer your questions?
Summarizing Supermicro’s fault in this, he said:
I think they are innocent and someone is using them to dilute the story instead of mitigating the threat. Please help me, them, and everyone else to understand that the problem is bigger. Dealing with this as a Supermicro problem will ruin the opportunity to face the reality that we need to fix it.
Mr. Appleboum said that those that have been in Israeli intelligence, (where Mr. Appleboum has worked for) or the CIA, being in the news is a nightmare. From the discussion, I got the sense that Mr. Appleboum was taking my call because he wanted to clarify how the story was being told instead of for personal or professional gains. In our discussion, he came across as someone who shares the belief that this is an industry-wide problem, not a Supermicro problem.
The line that stuck with me during our discussion happened when I confirmed he agreed to be quoted in this. Mr. Appleboum replied:
I want to be quoted. I am angry and I am nervous and I hate what happened to the story. Everyone misses the main issue.
Final Words
This is extremely strange. I believe that these types of attacks are more common than we often assume. Bloomberg’s reporting has done an excellent job bringing this to top of mind. At the same time, it begs the question of whether we hang this solely around Supermicro, and angle Bloomberg seems to be advocating by not providing balanced reporting. The other option is that we recognize this is an industry problem. Further, it is going to be an industry problem for the foreseeable future that impacts all companies that provide the infrastructure we enjoy every day.
I strongly believe there needs to be an SEC investigation here. This is a dangerous precedent to set where equities are being impacted by selective reporting and strong denials from companies who have obligations to give truthful statements to shareholders. As a global market leader, the US SEC needs to ensure that companies are making truthful statements and that markets are not being manipulated for financial gain. This very well may not be the case here, but this is the case where a lot of parts are not aligning warranting some assurances from regulators that manipulation is not happening.
My takeaway from the conversation was fairly simple. First, I immediately re-read today’s Bloomberg piece and thought, this is very different than the impression I had after reading that piece. Second, for STH readers who are mainly in the IT industry, it is worth having a conversation with your teams about this. Everyone in the industry knows that there are multiple facets to security, and the hardware supply chain is one of them. Bloomberg did the industry a service by highlighting the need for greater hardware security and scrutiny.
We should take this as an opportunity to take simple steps such as ensuring that BMC IPMI interfaces are not publicly routable, egress from BMC networks is restricted, and default passwords are not left on production machines. If a BMC were compromised, as discussed in the original report, competent IT setups from large organizations would not give packets egress from BMC networks. There are thousands of publicly routable BMCs on the Internet right now across vendors. This should be the point where that number goes to zero.
It is worth noting that hardware tampering being an actual threat or not, propagating FUD on this topic is obviously a good thing for M. Appleboum’s company.
The previous STH article about the Bloomberg/Supermicro story was interesting: well documented and technical about what is technically possible and what is not. This one is only propagating the opinion of M. Appleboum that everything can be tampered when you don’t handle yourself every aspect from the manufacturing to the shipping and exploitation (and even then there could be “traitors”). To me, this is clearly FUD to promote his security company: obviously, with enough money, you can infiltrate any company with software, hardware or human means. This has always been the case, nothing new here.
Where is the damn evidence?
“We found it in different vendors, not just Supermicro. We found it not just in servers, in different variations, but hardware manipulation on different interfaces, mostly in network related. We found it in different devices connected to the network, even Ethernet switches.”
Really? ..so where are the pictures, xrays, microscope images… This is ludicrous. Somehow these things are all over the place yet impossible to find.
As has been pointed out, ethernet jacks don’t even have power wired to them (unless they’re combo ethernet-usb; and no, nobody runs POE internally) – the whole story is bullshit
Yes and no … think of Intel ME, VPro, etc.
An independent network card with an operating system hidden inside a processor that communicates with the outside without any possibility of being controlled….
https://en.wikipedia.org/wiki/Intel_Management_Engine
Someone who thought he was never discovered in his evil intentions lost control of his malicious tools that were used against him, and now he wants to close the huge black hole without admitting that he himself is responsible for this disaster…. Think about it
Herve Bloomberg’s source disagrees with the conclusions drawn. That’s big.
I don’t understand why there weren’t real images of the devices in the original story. In 2015 everyone had a phone in their pocket. You’ve gotta assume someone has taken a picture. Even Appleboum’s gotta have photos if they’re seeing this.
I’m not pretending it doesn’t happen. I don’t think it’s Supermicro only.
No Evidence whatsoever… The old China Boogieman propaganda… once the Russian Boogieman propaganda…
Bloomberg used fake photos presented zero evidence and published a story that Apple, Amazon and Super Micro DENIED BEFORE AND AFTER publishing. Apple, Amazon and Super Micro testified under oath at a Congressional hearing yesterday that the Bloomberg story was false. The origin of the story was a small gathering in McClean VA at the invitation of the NSA. Yossi Appleboum is an ex-Mossad! This story is a TAINT!
This is a three-year-old story, launched by Bloomberg at roughly 11AM on a trading morning Super Micro stock dropped 60% in two hours. The SEC most certainly should already be investigating.
FUD or not, I get the strong feeling that more and more individuals are just trying to profile themself and their business. It’s a bit of a letdown when you see otherwise respectable people claim first story or try to push a new buzzwords for old problems.
Even if there is such an implant chip, can any one prove it is not a part of the prism program?
I don’t doubt that governments can do this. I don’t think it’s that widespread or we’d have seen it more.
If it’s between the he factory and the customer there’s not a lot you can do.
Bloomberg is twisting this story even if I think Appleboum is overblown in scale. It isn’t Supermicro only. That’s total BS.
Is it just me or is it looking more like Bloomberg is trying to bring politics like half truth news to tech? We should boycott.
I think someone should at least publish the IP addresses of the control and command servers, so we can setup traps. There cant be that many.
one more, please
If this component really was sitting on the actual board, then it must have been implemented in the circuit board design. Traces for power and data must have gone to the chip in question. They also stated there was an embedded circuit in the actual fiber glass board. I doubt this would have been performed after manufacturing, it would have been too difficult to do. Not impossible, but very hard to do. I think we need to see some proof like pictures or part numbers – schematics – anything – oh yes, and IP addresses :)
Also, I really would like to know where said 7000 super micro servers from apple ended up at?
SMCI was targeted alone and its stock dropped more than 40% in one day. Has SEC been hibernating with paychecks being automatically and directly deposited?
To add legitimacy to this story there really needs to be absolute proof one way or the other. However lets not be foolish to think that something like this hasn’t already happened and that nation state actors aren’t using local groups to carry out such attacks. If it could happen with something as small as a USB drive then its not hard to replace a component during the supply chain handling. Though I really think that if its true that Supermicro had no idea that it was going on.
meanwhile..I just bought 1400 shares at <12$/share. time to HODL. As others have mentioned, where's the evidence? with all the damn leaks going around you'd think someone would have a grainy/blury cell phone shot of one of these so-called chips… yet nothing. Someone at bloomberg shorted Supermicro big time.
The 1mmx2mm chip embedded in the circuit board sounded so feasible.
Right beneath the SPI chip.
This was highly complex. Embedded a tiny chip in superhot laminated PCB layers.
I wonder how they get the alignment so precise. When pressed at high pressure the chip might wiggle around a little. There are not very wide traces these days.
I bet they have to make 10 boards to get one good one.
Then slip it in at Foxconn during a run of Amazon boards.
Totally feasible.
When I was in the Navy we had to buy fresh produce from ports we pulled into.
This was a huge risk for security,
So they went to the local farm market unannounced and bought up everything they needed,
No chance for sabotage.
I imagine an Amazon or Gooogle Motherboard run at FoxConn is a special run. These are custom boards.
That works highly against them. No way you could do this implant on every SuperMicro board.
So here is my guess how it went down. Due to rampant IP theft board design is coveted. So they use Taiwanese to do the PCB manufacturing and offshore the pick and place and wave soldering to the mainland.
Well one board slips out in transit, is reversed engineered layer by layer and rebuilt at special TAO lab.
Figure out placement for implant and make PCB’s and test. Send good ones on to factory.
Maybe they just flat out stole or hacked the PCB drawings and saved themselves some time.
Loyalty these days is waning.
Appleboum and Bloomberg have done everyone a great service. If what he says is true, and he has everything to lose by lying, Appleboum deserves special admiration for being able to snag an eavesdropping device. China has state controlled wafer fabs and design shops fully capable of orchestrating a program like this.
Bloomberg isn’t Fox News and Appleboum isn’t the only source for their story. Bloomberg is a politically conservative news organization, not prone to ignorant blunders relating to national security. On the other hand, China and Russia have everything to gain by discrediting the individuals and teams behind this story. One can safely expect a disinformation campaign targeting social media and web comments in an effort to obfuscate the truth.
In my opinion, given the risks and a growing menace from China, exemplified by their actions in the South China Sea, its time for the US to rethink the technology investment, training and jobs we continue to pour into China. That investment and tax money should be put to work building our own infrastructure and military, not China’s.
“They [capitalists] will furnish credits which will serve us for the support of the Communist Party in their countries and, by supplying us materials and technical equipment which we lack, will restore our military industry necessary for our future attacks against our suppliers. To put it in other words, they will work on the preparation of their own suicide.”
Vladimir Ilich (Ulyanov) Lenin, as reported by I. U. Annenkov in an article entitled, “Remembrances of Lenin,” Novyi Zhurnal / New Review, September 1961, p. 147. Cited by https://quoteinvestigator.com/2018/02/22/rope/
Who’s pushing Businessweek?
– Hedge fund?
– Politics?
– Domestic competitor threatened by Supermicro like HP or Dell?
– Taiwanese competitor that wants Supermicro’s business like Quanta or Tyan?
We checked over racks of servers and there’s no metal insides to ethernet ports on our Supermicro servers.
I see there are some Harry Potter styled conspiracy theorists that have joined the discussion. With that in mind, I would like to suggest that the new logic can best be summed as “If you can’t prove it doesn’t exist then it could exists” and another fallacy saw is “If it is not impossible then it must be possible.” Both are fallacy arguments and termed plausible denial schemes. They are populist schemes but they are indeed not plausible.
There is some mathematical probability that needs to be considered.
1) What does the spy want? Encryption keys.
2) Why would boards be targeted at Apple and Amazon? They are very careful with their networks and installations and surveillance. Possibly the worst two targets in America.
3) Amazon makes there own boards using proprietary software and in some cases ARM Alpine chips. The only real assumption that can be made is these server boards are for Intel processors. That’s interesting. The one company that missed the handheld industry entirely. AMD has some servers and could sell them as unmolested.
4) The Apple and Amazon boards use different operating systems unknow to any randomized intruder. An embedded chip would have to get access to the unknown kernel. Some are hardwired, some are booted. The actual board maker would have to know where the boards were going and what OS would be used.
5) Then there is the matter of the addressing system, a big matter indeed.
6) Then the expense of the project to collect encryption keys. Better off buying keys from an employee for a few hundred thousand. The cost to make a run of 10,000 servers would be millions and not cost effective. If the servers were randomized to task and location, there is no guarantee they would even function. The idea is stupid, dare I say really stupid.
There are even more variables than this but mathematically I am at a limit approaching infinity. To a common sense individual, not an engineer or mathematician, the more apt word is impossible.
An example is 128 bit encryption broken by brute force would take the fastest supercomputer known a mere 1 million years to break the code. I regard that task as a pragmatic impossibility if not completely utterly worthless.
Bloomberg is selling fiction.
eBay
Show SOME actual evidence or it didn’t happen. This nonsense about “metal cases on Ethernet jacks” is laughable, they ALL have metal cases for RF shielding and for the cable latch. There is no printed circuit board inside an Ethernet module, only the magnetics to match impedance. When pressed for details, the guy who claims to be the expert says “maybe it’s not just supermicro and maybe it’s not just servers and….”maybe he’s just making all this up. The photos showing the “pencil tip sized component” are also nonsense. Look at the circuit built into an Apple lightning connector sometime; it’s a circuit board with a dozen components all no wider than the lightning connector(I’ve studied these under a microscope personally). All this magic can’t be done with a single component with two leads; not enough real estate for a processor plus memory and not enough wires for data plus power plus ground. TLDR; there are no ghosts in the machine.
NSA has been doing this for years. It is nothing new, just getting more visibility now that other adversaries are doing the same thing.
NSA, GCHQ, etc… have been doing this since the invention of data communication. AMD and Intel are not allowed to make a chip (CPU, GPU, FPGA, etc..) without a backdoor for the NSA and the same is true for ARM with GCHQ. The extra Chinees mini-chip is a BS-story. The companies who do eavesdropping on consumers are among others, Google, Amazon, Apple, Samsung… some do this because their goverments forces them some as a business and most as both…
Thank you Patrick, great interview and article.