A few weeks ago, we broke a story about finding something in high-end Dell EMC switches and possibly HPE-Cray supercomputer nodes that should not belong. Specifically, we found stickers that said American “Megatrands” and not “Megatrends” as the supplier company is named. As part of that story, we had a small experiment running, and this highlights just how hard it is to catch even the easy-to-spot deltas in IT equipment. After 130,000 views on YouTube, we have some data to bring to that discussion.
The Easter Egg Experiment
As part of that piece, we showed this timeline that charts our efforts and interactions with Dell EMC and American Megatrends (AMI) during the investigation:
Here, the pertinent box is the first box “October 4-7” in the Dell swimlane. This says “Dell question on if the hardware could be a rogue reseller.” That is the version from the STH main site article. We also had an accompanying video, and that is where we hid the Easter Egg as a bit of an experiment.
What others may have seen is the version of the above chart that we snuck into the accompanying video:
The difference is a small one that most will subconsciously skip. This was a small experiment to see how many people would point out the “rogue” to “rouge” swap.
After two weeks, and around 130,000 views, we have an answer: only 17. I replied to all of those who pointed it out.
Of course, not everyone will have watched and found the Easter Egg. It was actually in several places throughout the video. Just based on where people were watching on the timeline, it seems like at around 130K views it had around 100K folks seeing the chart. Many even if they have the video up are multi-tasking while it is running. Those folks are not necessarily going to see something like this. Even those that spotted the swap may not have taken the time to comment.
Still, those working in factories have other things going on so they may similarly be distracted. They may also not be incentivized to speak up even if they see something out of place. The point is, for someone to find something and speak up seems to be quite rare.
Final Words
Really, the key takeaway of this unscientific experiment is just how hard it is to spot anything out of place in modern hardware. There are thousands of components and the sticker is perhaps the easiest to read.
Perhaps the biggest lesson learned here is just how vulnerable the supply chain for modern IT equipment really is. In 2022, we are going to see a bigger focus on hardware security. We recently discussed how Lenovo Vendor Locking AMD Ryzen-based Systems with AMD PSB at the consumer level after we saw it on the server and workstation side. Cloud providers are building their own hardware root of trust to help validate components and firmware. Even Intel and its partners are adding FPGAs for its PFR feature to Ice Lake Xeon generation systems.
What is clear is that this is an industry challenge and one that we are going to see taken up more this year. Relying on a supply chain to spot differences visually if it is anywhere near a 17 in 100,000 Easter Egg rate is not going to be sufficient to protect the world’s infrastructure.
That’s what I never understood with the Bloomberg story. They said AWS and Apple saw spy chips, but nobody’s actually shown them except for a proof of concept. Meanwhile, nobody notices this. I’d agree most won’t see. Those that do won’t say a thing.
I rarely leave a comment… I will say I noticed it. At the time I though it was a typo since it was done in a rush and left it at that.
Like Daniel said, not everyone will take the effort for a typo. Given how many people read STH articles and how many actually comment, I’d expect a similar ratio of (ten?) thousands to one.
It is amazing to see the difference between STH article page views to comments and YouTube views to comments. Then again, it is also a different demographic on YouTube.
No offense Patrick, but I am definitely not one of your YT fans even if I fit the demographic. I will stick to reading the articles.
Most people still read STH instead of watching our YT :-)
I am also one of those who wouldn’t post a comment just because of a typo in the article or a diagram generated by the writer. If I had noticed the typo on the Megatrands sticker, I would have commented, but I would not have commented on a typo on STH’s site or any other news site unless it changed the entire tone of the article.
On most sites I frequent, the writer doesn’t read more than the first few comments on their articles, so comments mentioning typos are unlikely to get back to the writer and I find them pointless. STH seems to be different in that regard, which I appreciate.
I also think it’s rude if the only thing you can say after reading an article is “you made a typo”. It’s the equivalent of commenting “First!” It’s not something that will stimulate discussion or make an interesting point that people will find valuable.
As Daniel pointed out, not everyone will go to the trouble of correcting a typo. Given the number of people who read stories on STH and the number of people who actually comment, I would anticipate a ratio of (tens of thousands?) to one.