Today we are going to share the result of a bit of investigation that started a few months ago on STH. The short version, it appears as though the Dell EMC S5200-ON series switches, the company’s high-end 25GbE-200GbE switches, have license/ royalty stickers that have a different company name on them than they should have. Instead of saying “American Megatrends”, they instead said “American Megatrands”. To give some perspective, this looks strange because it would be like buying a Dell notebook and getting a “Macrosoft Wandows” license sticker on it.
While we have been able to confirm these stickers on four different models in the Dell S5200-ON line, we have reason to believe they go beyond just Dell. If you happen to work on a certain US Army supercomputer by HPE Cray you may want to open up the system and have a look. More on that later since we do not have those HPE systems in the lab.
Through a fairly rough October, we validated that indeed these stickers are in the wild. Ultimately, after we brought their existence to American Megatrends (AMI) and Dell’s attention (HPE did not care enough to investigate), we now have an artifact that says that American Megatrends is honoring the license stickers and will not pursue legal action against Dell’s customers or those using them.
Since we have an official response from AMI and Dell, instead of going out there weaving untrue or bending facts as others have (I did the follow-up Yossi Appleboum interview on How Bloomberg is Positioning His Research Against Supermicro for reference, and that was eye-opening), all we can do is explain what we saw, what our process was, what the response is, and the impact now that we have all of this information. If you agree or disagree with this, all I can say is feel free to take this to whatever forum you would like online to comment on it.
Video Version
Since this was found by a STH YouTube viewer, we are also going to have a video version of this piece. You can find it here:
As always, we suggest opening this in a new browser, tab, or app for a better viewing experience.
For Context: The Only Technical Background You Need
At STH, we do a lot of in-depth technical articles. This particular article is going to be around the baseboard management controller or BMC. That is actually the main chip in the complex Bloomberg was trying to highlight in its story. We have an entire article on Explaining the Baseboard Management Controller or BMC in Servers. Effectively, think of a BMC as a small computer within a computer that operates independently. That independent operation allows features on servers, switches, storage, and other IT items to power on/ off, update firmware, provide remote access instead of needing to physically troubleshoot at a box, and even do things like aggregate, log, and report sensor readings. BMCs are fundamental to modern IT.
On the Dell EMC PowerEdge side, the company’s solution is called iDRAC and that is a custom hardware and software solution.
Still, Dell actually uses the ASPEED BMCs in many of its products. Even the old Dell PowerEdge C6100 utilized ASPEED BMCs because large hyper-scale customers want industry-standard management. While iDRAC has many great features, for the majority of the industry the ASPEED BMCs are the go-to option.
Since the BMC is a small computer that runs mostly independently from the main server, storage, or network device, it needs its own OS and applications. That is where American Megatrends steps in. This is a bit of an oversimplification, but AMI MegaRAC one can think of like the Microsoft Windows of the mainstream BMC world. AMI has a number of versions, and there are different feature levels in BMCs, but at a high level, that is what is going on.
Now that we have that basic technical context,
Discovering the Dell Megatrand Issue
In August 2021, I made the decision that we needed to do more hands-on switch reviews. We had done the Edgecore AS7712-32X, Celestica Seastone DX010, Arista DCS-7060CX-32S, Innovium Teralynx 7-based 32x 400GbE Switch, and more, but it seemed like it was time to look at other high-performance switches on STH.
After chatting with Dell about reviews at the end of August, I was told someone in a Dell meeting said something to the effect of “why should we care about STH?” For context, STH is the largest editorially independent server, storage, and networking review site in the world, by a wide margin. Most of the coverage of enterprise gear is done by sites that allow vendors to write or review articles before they are published in exchange for a fee or preferential treatment. STH has never allowed this, and while it craters revenue, I am happy to do so to ensure the site keeps growing. We have been doing reviews of Dell products for years, so the comment was more of a normal process they go through when evaluating activities, and not meant in a malicious tone. Still, as an A-type personality, that immediately triggered a response of “OK, I am just going to pick a product, and we are going to do a review and get more page views/ video views than Dell.” Since I was on network switches, that became the focus.
On September 5, 2021, we purchased a few switches. First up was the Dell S5148F-ON piece where Dell used Marvell instead of a Broadcom switch ASIC. Datacenter switches that were $10K+ new are a niche category, but it still did fine for STH since we are a niche site.
With that, it was time to look at other switches and from the newer series. We looked at the subsequent Dell S5232F-ON Hands-on A Vastly Improved 32-port 100GbE Switch.
As one would imagine, we did these switches including photography, testing, and all of the B-roll as a set, and then published over time. After publishing that video, we had a very interesting comment on YouTube:
To be fair, we had both the Dell S5232F-ON and the S5296F-ON pieces done, and never had noticed that there was a misspelled sticker. A quick post on Twitter and it did not seem like there was a lot on MegatrAnds versus the correct Megatrends.
Here is the photo in question from the original piece:
Here is a closer view:
Here it is even closer. I wanted to show the full zoom-in progression just to show how amazing it is that someone saw this on the YouTube video. Also, just how hard it is to miss. This sticker should say “AMERICAN MEGATRENDS” not “AMERICAN MEGATRANDS”.
Of course, we checked the Dell EMC S5296F-ON piece that the video was already back from the editor and just waiting for a publishing slot. We do try to space these out a bit, but that is a challenge when you have spent tens of thousands of dollars on switches and the content is just sitting there waiting to be published.
Sure enough, there was an “AMERICAN MEGATRANDS” sticker here as well. You can see them in the review, but we specifically edited out this photo as well as in the S5296F-ON video when it went live in mid-October. The investigation was ongoing at this point, but we had two S5200-ON data points when we pulled the images so as not to start a firestorm. Bloomberg’s story never showed the alleged hardware, yet here was something that even a non-technical person could see is not right. It was too dangerous to put online.
On October 4, I sent a note via the American Megatrends website’s legal form alerting them to finding “Megatrands.” I was contacted by AMI on October 5, 2021. I was asked not to share the conversation, so I will respect that. All I will share is that on October 5, there was no answer as to why this was, even after AMI had enough time to do research the next day. If it was something AMI headquarters knew about as a well-known typo, they did not tell me on that call. They needed time to do more research because, just like finding a “Wandows” license sticker on a PC, this looked off to both parties.
I let Dell know via a phone conversation that I had found something that did not look right in two of its switches. We had Dell Technologies Summit on October 13, and I did not want this to become a topic. Plus, everyone at Dell was busy with that event. Instead, I scheduled a lunch with Dell on October 19 to show them what I had found. I will get more into my thought process for disclosure in a bit. During the conversation though, I was asked a few questions. One was simply to the effect of, “how do you know the switch was not tampered with by the reseller?”
That was a great question. The S5148F-ON that used a different management solution and the S5232F-ON and S5296F-ON all came from the same reseller in the southeastern US. It could have been a case of supply chain tampering on the path to me. On the other hand, remember these are 2019 era (so still very current) $10,000+ switches so investigating further is not cheap, especially since we have humble budgets compared to the hardware we review.
Meanwhile, on October 7, 2021, we purchased a Dell S5248F-ON from a reseller in the midwest. We also purchased a S5224-ON from an international reseller. These two we did not do our reviews of because we were waiting to investigate Megatrands. Sure enough, though, we saw some things.
The following week, before Dell Technologies Summit, the switches arrived. Here is the S5248F-ON from the midwest reseller:
Here is the S5224F-ON from the international reseller:
We now had four models, from three resellers, in two countries all with “AMERICAN MEGATRANDS”. This was not just a rogue reseller. It seemed to be part of the high-end Dell switch platform. At this point, I had also gone through our archives and I found a few instances of these stickers with “Megatrands” that I had never noticed previously from the 2018-2019 era or so. This appeared to be more than just Dell being impacted, but I had four of these switches sitting in our studio.
I knew all of this by Friday, October 15, 2021. American Megatrends was investigating but did not have an answer over a week later. The following week, it was time for the sit-down with Dell.
John Etulain of Seattle Washington registered those 2 domain names, and it is being served HTTP using STH’s SSL cert.
Staffer of yours, Pat?
Amazon Route53.
What was HPE thinking?
MegaRAC PM is so old guys. You’re missing that. It’s almost 2 decades old and being open-sourced right?
You didn’t second source that it’s misspelled properly.
A few random remarks:
1. this whole thing seems to completely ignore the fact that latin characters are a second thought in the countries where those things are manufactured. So an honest mistake is still a very credible cause here. Would anyone in the US notice a misspelled name written in Chinese characters?
2. what would prevent a counterfeiter to print labels with correct spelling? If we’re to suspect foul play and consider a supply chain compromise, why blindly trust the correctly-spelled labels and be certain that they couldn’t possibly have been tampered with, just because the vendor name is spelled right?
3. and the fact that it hasn’t been noticed before, by the vendors and manufacturers themselves? Well, exec have probably never seen one of these products in real life, let alone the insides of it. And people on the manufacturing floor probably don’t care or know enough to report it…
I’m surprised that such a glaring defect would make it through the process that produces an incredibly precise and demanding piece of gear; but the really scary thing isn’t so much a bad batch of stickers; but the fact that evidence as weak as the sticker would even cross our minds when examining the question of whether the firmware wrapped tightly around our server or switch’s brainstem is what it ought to be or something similar but less benevolent.
I’m not aware of any really good alternatives at present, though it’s an urgent problem so the search continues; but it’s terrifying that looking at the sticker would be something other than a waste of time if you want to verify the BMC and its firmware.
I searched the web for American “Megatrands” and was quite surprised to find
mentions of the name as far back as this 2002 patent application:
https://patents.google.com/patent/US6739896B2/en
The search yielded many surprising hits which might merit more investigation.
Sorry, but the name “Serve The Home” doesn’t exactly scream enterprise content.
This response only concerns me more. Letters of this nature are extremely precise and gone over with a fine-toothed comb by management, PR, internal counsel, and often external councel for backup. Every single word is lawyered up to the nth degree.
They specifically say that “licensing is not a concern.” Fine. Wonderful.
They specifically do not say that “these systems were distributed with legitimate firmware.”
They specifically do not say that “these systems were distributed with legitimate hardware.”
If they do not specifically say these things then that means that they either do not know or that they do know and they do not want to tell you.
A suspect thit thas artacle wall wand up tranding.
As a long time reader, I love this site and the content. I do agree that ServeTheHome has always been an odd choice of name. I get that alot of homelab geeks hang on this site and it’s forums and the convos we have translate back to the enterprise. I cant tell you the amount of times someone has linked me an STH article or forum post and I have said, yeah I already seen that. I don’t think I realized until that happened a few times that this site has a pretty large following in the tech world. That said, the name has always struck me as weird. Keep up the content!
Regarding ‘ServeTheHome’, does it count if Patrick’s home probably is wired up with 100G fiber and he’s running half these servers there? :D
I still like the content because it’s kind of a preview of some of the gear that could magically filter its way down to my homelab in a decade or so.
These are still 25Gb and 100Gb switches that companies have deployed right now. For those that think it’s just a spelling error, there’s more that the article talks about. The letter says the license is still valid so that’s fine now, but would it be valid without the letter?
All I’ll say is the big one to me is that Dell and HPE told everyone how trusted supply chains for them were when the Bloomberg piece was out in 2018. Right as they’re doing that, these stickers are getting installed and they didn’t know. That’s the real problem. They’ve told everyone they’ve got supply chain checks so their gear is more valuable. This shows that they don’t really have good checks. This doesn’t take a microscope to find.
So will the youtuber who spotted it get one of these switches as a present. I mean, he deserves it, right?
@Pete Mitchel – I first read STH maybe 10 or 15 years ago, and then it was covering stuff that you might find in the home of a serious IT professional.
It’s evolved since then.
I must be missing something. Why is anybody concerned about a “license” and why do we need a written letter absolving us (somebody?) from a licence issue? Before we had the letter were we morally questioning our “license” and turning off all equipment with wrong stickers in it because it might be “ilegal”?
Come to it, why are there even stickers on the chips? I am sure you are not buying the switch because you want to be looking at stickers. Falsifying a sticker is probably 0.0001% of complexity when working on switch hw.
As long as the device functions exactly as it should, does not do less, does not do more, functions exactly as it should who cares what the sticker says? Why does the letter not mention anything about functionality and security guarantees?
My opinion is that new gear functionality (the thing that matters) is the thing that needs 100x more QA. It has become network industry practice to release alpha software/hardware and have customers act as beta testers.
At least these don’t have the Trump chip from American MAGAtrends. :-)
At a company, to remain nameless, a product manager and a vice president of legal got into a turf war about who was in charge of licensing stickers. To demonstrate their authority, they kept having the graphics arts depart make them mockups of the stickers with various color text.
The mockups went through endless meetings and reviews to ensure they had just the right color. When, after four months of bickering, the decision about what color text to use had to be elevated to the regional vice president. He approved and everyone went on their merry way.
Alas, after spending the last three weekends pulling all-nighters, (unpaid because he was salaried) doing emergency mockups of the stickers, a graphics artist said to his cubicle mate, “After pissing away the last four months of my life on these stickers, I’ll bet these morons won’t even notice that the name of the company is misspelled.”
So the name is spelled correctly, or it isn’t spelled correctly … either way, AMI has licensed the manufacturing out to a third-party fab who probably *does* have chinese spyware installed on the chip. If it was manufactured anywhere on that side of the world, the supply chain is compromised.
I think Patrick’s point was more along the lines of “Why didn’t Dell throw a fit when they received parts from a supplier which were incorrectly labeled, and therefore could have easily been counterfeit?”
Sounds like a business opportunity. Time to start up Armenian MangaTrends and start selling competing BMC chips lol.
Amelican Megatland
Incel Inside?
American Maga Trends sounds like a terrible 4 years.
Asian bribery of import and export officers are renown. Even for stuff made under contract. If a misspelled sticker can get past these “trusted supplier” OEM’s, then they are full of crap and the offering is just marketing BS.
I know of a US based company who went BK because an Asian supplier bribed the import officer, which caused sub par product to pollute the US supply chain. It got into commercial, military, aerospace, automotive industries and caused such a large liability, the company went under as a result.
The US has gotten so lazy and so dependent on contract work globally and don’t maintain any controls over it, that even the import inspection is contracted out which makes it highly susceptible to bribery.
Why if companies like Ford, Chevrolet, VW, Toyota, etc. Can chance a mechanical piece when It has a defect, I don’t get It that Dell, HPE and others can not chance a simple sticker on a enterprise piece of HW if the customers ask for It.
STH is an independent review site, yet bends over backwards *not* to publish an article that could hurt a (major) company whose products it reviews. Give a company 24 hours, rewrite based on the reply but get the article out within the next 24 hours. Corporate divestiture be damned. The article is important, even if the root cause is a typo. The “final” piece can always say, “our research is ongoing.”
I’d be very interested in seeing this chip be compared with a correctly-labelled chip, if any can be found, using something like ABI SENTRY counterfeit chip detector (disclosure: I worked on the software for that device).
If they’re made somewhere else, then even if it’s made to the same basic spec, there’ll likely be significantly different V/I curves on the pins.