This week I received an Amazon Prime Day purchase EVGA NVIDIA GeForce RTX 3090, but there was a catch. The GPU being sold as “new” by Amazon.com seemed to have been a customer return that had its security seals broken and had another card substituted in its place. While Amazon has been good about the return thus far, this experience has profound hardware security and hardware supply chain security implications.
Video Version
For this, we also have a video version that you can find here:
There are a few photos that are in that version that are not in this, just given the timing of both. As always, we suggest opening the video in its own tab, browser, or app for a better viewing experience.
How I Got Scammed on an EVGA NVIDIA GeForce RTX 3090
Followers of my personal account on Twitter this week may have seen the tweet that came out in disbelief of what happened. On Amazon Prime Day I ordered a new EVGA NVIDIA GeForce RTX 3090 FTW3 Ultra edition and things did not go as planned.
Scammed by Amazon. Bought a new @TEAMEVGA @NVIDIA RTX 3090 (Amazon seller.) It arrived. Box seals broken. Inside was a RTX 3070. :-( pic.twitter.com/ECMyn0eEfP
— Patrick J Kennedy (@Patrick1Kennedy) July 21, 2022
On July 12, 2022, I ordered the GPU. Although NVIDIA sent us our original GeForce RTX 3090 for review, we needed another one for a test platform for several of our other series on STH. The EVGA card was sold by Amazon.com Services LLC and was listed in New condition. This was purchased via the Add to Cart/ Buy Now box, not by looking for used units.
The unit shipped on July 13, 2022, and then things started to go awry.
First, the unit was shipped, requiring a signature delivery. I was heading to Zurich, Switzerland that weekend, but according to the estimated arrival date, I was to be around for the shipment. Instead, Amazon batched the card with other items so it instead would arrive after I left for Europe. That meant I had to spend $10 to get UPS to deliver the package at a later date when I could be there to sign for it. When the package arrived, it looked like it had a tough life.
It seems as though this package was open at some point during transit. When it was opened, it appeared that someone had folded the flaps over each other. Then a copious amount of clear plastic tape was used to seal the box. You can see it on the doorstep at delivery in the video, but the opposite side was opened to gain access to products just to preserve this packaging masterpiece.
The contents seem to have arrived, but the GeForce RTX 3090, destined for a test node, had a curious feature. EVGA uses security tape to seal its packaging. On this package, both EVGA seals were already broken.
In addition to the two broken seals, there was a LPN PM sticker. Often Amazon uses the LPN RR stickers for returns put back into stock, so this looked similar. Also, there is a sticker under that LPN sticker with the model information and a “NewItem” printed on the label.
Here is another look at the label on the end of the box, this time opened.
Inside the box was a normal inside box with the foam carrier.
Removing the top foam made us see the ICX3 cooler style that EVGA uses on a number of its cards. If someone processing a return saw this, they would see the card on the front of the box and assume the same card was sitting there.
Even the EVGA badge and the non-model specific EVGA GeForce installation guide was included.
EVGA uses branded ESD bags, and this card was using the EVGA bag with an ESD seal still on the bag.
The one view, other than tracing the serial numbers that would give this away was looking at the top of the card that says “RTX 3070” instead of “RTX 3090”. That single 7 versus 9 digit change is the only giveaway one would reasonably see when inspecting this package to tell that this was the wrong card.
That “3070” is often hidden by the foam carrier in the box. Being empathetic, it is not too hard to understand how this could happen. Someone that did not know the difference between a RTX 3070 and a RTX 3090 would miss this if they were under time pressure to validate a previous return. When opening the box and seeing the card that looked like the box cover, I originally assumed it was the right card, and I saw the ESD sticker sealing the card. It was only later that I realized this was a 3070.
My best guess is that someone purchased an EVGA RTX 3070 and a 3090. They then kept the 3090 and returned the 3090 packaging with the RTX 3070 version inside. Amazon processed the return of an expensive card and the person checking missed the 7 versus 9 on the top of the card, and did not have a way to validate serial numbers. They then saw the ESD bag seal and decided that it could go in new stock instead of a used queue.
I opened the return with Amazon and spoke to someone on the phone to explain what happened and this card is now on its way back to Amazon. I wanted to get on the phone because I was sending back a RTX 3070 in a 3090 box, and if someone does proper checks, they will see this is not the card that matches the packaging. I was a bit apprehensive about doing this type of return since it is a mismatch.
In the end, assuming the refund is processed, I will be out about a week putting together the system we will use to test the Project TinyMiniMicro nodes and for our STH Mini PC series. I also have the risk of having to do the return and having an issue because there is a mismatched product. Further, I had to stop by the UPS store and drop off the package, and that is a 15+ minute one-way trip. The positive is that pricing on the RTX 3090 has fallen since then, so I probably will end up a bit better off with the return other than all of the lost time.
The eye-opening impact is really the hardware security aspect of this story.
Why This is an Important Hardware Security Story
Bloomberg’s discredited spy chip article and a follow-up piece that I interviewed their primary source who disagreed with Bloomberg’s reporting, while fake news did something for the industry. It helped highlight the challenges with hardware security and supply chains. This story is a more practical and dangerous story from a hardware security perspective.
One of the big challenges with hardware security is that items can be intercepted during shipping. We assume this was just a package that Amazon did not properly fill and that got crushed and re-taped by UPS. Compared to some stories of systems being intercepted during shipment, this is a poor cover-up job. Still, it is not completely uncommon to see Amazon packages take a few bumps along the way.
The bigger concern is that it seems like a unit sold by Amazon.com and not a 3rd party seller was not new. Instead, even with security seals being broken, it was placed back into NewItem stock and sold as “New” per the invoice above.
Taking a step back for a moment, if the card was actually an EVGA GeForce RTX 3090 and with the ESD bag still sealed, I probably would have used it being in a time crunch to get the new system online. If the entire card can be substituted and go through Amazon’s return logistics process, ending up as “new” stock, then what if someone had more nefarious intentions? What if someone tampered with the GeForce RTX 3090 and got it accepted back as new stock by Amazon?
The downstream buyer might use the compromised RTX 3090 in their system. This is a card where the PCB is not exposed since there are fans, heatsinks, and shrouding covering the entire card. It would be impossible without disassembly to see if the card was tampered with before installation. That compromised RTX 3090 would be plugged into the PCIe slot in a motherboard. PCIe devices physically installed into motherboards have very low-level system access due to their natures. This is perhaps one of the scariest types of attacks since it would allow a huge amount of access to a system.
The key here is that by taking a previously sold computer product and reselling it, by Amazon.com as the seller as “new,” most buyers have the reasonable expectation that the unit went from the manufacturer to a distributor to Amazon or directly to Amazon. Having the potential of muddling returned computer products into this new item pool means that that chain of custody cannot be guaranteed.
Instead, we had a third party tamper with the GPU Amazon.com Services LLC sold us as new. Then, something happened in shipping leading to the package being accessible en route. This story, given Amazon’s large presence, should make computer hardware security experts very nervous as it has all of the elements to be disastrous for end users.
Final Words
Again, the chain of events is simple to understand. I am a bit disheartened by the inconvenience of this. I will say that Amazon has been easy to initiate the return on so I have no complaints at this point on the return process (we may have an update if that changes.)
Still, given Amazon’s market presence and the fact that it is letting highly tampered products into its supply chain and selling them as new, the hardware security aspects of this are mind-boggling. For Amazon, the challenge is that the alternative is that taking returns on these products and automatically not putting them into a used/ refurbished stock can be costly. However, if this practice continues, it becomes hard to trust one of the world’s largest computer component resellers. I doubt AWS would accept this in its hardware supply chain, so there is a bit of a “do unto others” that I cannot escape feeling here.
Originally, I was not going to write up this story, much less do a video. The Twitter post was all I planned on doing. At the same time, I think that our readers need to be aware that something like this can happen. From a hardware security and supply chain perspective, the box was bad, but there is little one can do to show tampered hardware can get through into a reseller’s “new” stock more than having a product that has been substituted in the package.
For the industry and our readers, Amazon’s computer component supply chain being this compromised and passing tampered packages as new and sold by Amazon, not a third party seller, should be a wake-up call. This is as bad, or worse, than what we found in Dude this should NOT be in a Dell Switch or HPE Supercomputer.
A few years ago I ordered four 16GB RAM modules from an online seller. One package clearly labelled as 16GB on the outside had a 4GB module inside. The other three were okay. I can share your nervousness about whether a return with the wrong device inside would be accepted. When the return was accepted I was relieved and just carried on without another thought.
Now that I think about it, this happened twice over the last couple years. The other time was with a single-board computer packed in a box sold and marked as the more expensive unit. Out of two units shipped directly from the manufacturer, one was wrong. That time the window to return the mispackaged item expired.
It never occurred to me that either of these incidents could have been some sort of supply-chain tampering.
I wonder how common this sort of thing really is?
Good to know that some of us aren’t the only ones being scammed here.
If the ASIN was reported in the article, I can’t find it.
Where multiple sellers are offering the same ASIN, there’s really no way for the customer to know who supplied the item pulled from the AMZN warehouse tote … entirely apart from risk of tampered return re-sale.
@Bob Niland
So you’re saying that if I buy a new “sold by” and “shipped by” Amazon item, it could be pulled from the stock of some third party as long as the ASINs are the same?
The moral of this story is, don’t buy anything important from Amazon. (Or anything at all, really).
I once bought a book from Amazon as part of a larger batch of books.
The book in question had been shipped separately from the batch as it came from a different warehouse.
The book arrived in a box with little packaging inside. The box was intact but nicely creased on 1 side. The book had a completely broken paper link between the spine and the endpaper that is pasted to the cardboard cover. Perhaps it was damaged in the warehouse or damaged when delivered to my home.
I found such damage extremely difficult to believe until I reviewed my security camera footage of that delivery. Yes, the Amazon driver had tossed the book onto my front door threshold from 6 feet away. I did not know that “social distancing” applied to delivering a box.
Amazon was very smooth in handling the return & replacement as I was an Amazon Prime customer at that time. I guess Prime membership is good for a few things, eh?
Pete, as I understand it:
“Sold by” denotes who gets the payment.
“Shipped by” indicates whose final warehouse handed off to the carrier.
So on a shared ASIN (which may or may not apply in the present case), sold-by means AMZN gets the $$ for the sale, and shipped-by means from an AMZN warehouse, yet could be from a tote full of the same ASIN, stocked-in from who-knows-where.
But it looks like even for an AMZN-specific ASIN, their returns process raises further concerns.
That looks like it was done in transit.
This type of fraud isn’t new. There have been reports about this happening on amazon.de for 5+ years, especially with CPUs.
What i find interesting is this seems new to the writer. Ive had amazon 15+ years. This is not isolated it happens all the time. Its happened to me. Its a catch me if you can and if you catch us we will take the return but if you dont we scammed another paying customer.
Yes amazon sells returns as new. That is why i stopped buying new hardware from them 5 years ago. They once were a trustworthy company they had ethics. This is about amazon not a isolated video card it applies to tvs etc.
If you want something new do not buy from them. Ebay used to be scam central but amazon morphed into that role yeah s ago just sorry this new user was suckered.
Its amazon not anyone else do not buy anything new from them and expect its really new and not someones return
That item had been processed as a return by Amazon (lpn tag visible) and then they tried to sell it as a new item. That’s the story here. Someone at Amazon warehouse done messed up. Even if it was internally sealed, this should never have happened. Should have been a warehouse used item for a discount.
I used to work in the shipping department. From the sound of your story it seems like this was a shipping problem. For the product to get to your house, it needs to cross many hands. Someone along the way must have gutted the package and replaced the GPU. I ordered a lot from Amazon and my orders nevered looked like that unless someone tampered with it.
I too had a similar experience ordering a new Dymo dual label printer through Amazon. Issues included:
Open box / used device
Serial Number on box did not match serial number of device
Open box / missing items (2x)
While cases were opened for each – this was a frustrating hassle. I think I must have cycled through a good portion of the return / sell as new until I actually got a brand new unit.
As noted above, even though a price may be good, it is not worth the aggravation and time. I tend to go to B&H, Provantage, or direct through mfg site.
What’s most frustrating about Amazon is how they (quite likely deliberately) obfuscate when you are and aren’t wandering into the flea-market zone, both in terms of their UI and on the backend with inventory commingling.
At least on ebay or aliexpress you know when you are dealing with murky provenance in the interests of a deal; but on Amazon it is still mostly made to look just as it did back prior to 3rd party sellers and inventory; while actually being all over the place in terms of reliability.
Broo atleast you got 30 series card and a good GPU(Damn 3070) back, here some people dont even have that privilege and are still stuck on Geforce GT 730 2 GB!
Leaving aside the joke, Amazon messing up a top notch and expensive piece of equipment is really a matter of concern
In all the years that I have been using Amazon and eBay to buy things from, I have not had a single issue with shipments I’ve received. Not from UPS, the postal service or Amazon’s own drivers. I have had times where the seller I tried buying from ended up proving a total flake-out, requiring me to file a claim and get my money back and I’ve had to return one item for refund that had been apparently damaged NOT by the USPS but by someone responsible for the stock at one of the companies I ordered from (in that case a used Sun Microsystems UltraSPARC T1 processor for my chip collection). But so far, I’ve not had any issues with the shipper or postal service with damaged packages, nor have I had things intercepted or arrive in a shoddily put together box that looked like it got riffled through in transit.
I would posit the following hypothesis, however: I’m betting that the person who got this at the returns department at Amazon didn’t bother actually taking the GPU card out of the box to inspect it and check to see if it was, in fact, the correct card. I’m guessing the original buyer might have mistakenly placed the RTX 3070 into the RTX 3090’s box, by accident, and shipped it back, not knowing the difference or realizing his/her mistake. Once the Amazon.com returns people received the card, they simply opened the box, pulled the foam inner tray out enough to verify that a card was in the box, stuffed it back in and sent the returned “RTX 3090” back to the warehouse to be re-sold. Once at the warehouse, the staff, also not knowing any better and probably under very high duress and very busy, simply tossed the card into the “New Item” bin without further ado. No verification of the contents, no check of the sealing tape used on the box from the OEM.
Of course, Patrick is in his right mind to be more than a bit worried, though I don’t advise jumping straight to the “nefarious activity” conclusion. Certainly, that’s now a very real and serious possibility, but I think more likely is the scenario I gave above – a chance occurrence of multiple mistakes that, in this case, managed to happen all at once and in succession, leading to this particular end result.
But absolutely Amazon needs be held to account for not positively checking returned items. They ALL need to be held accountable, be they Amazon, eBay or some other marketplace. And customers need to not be quite so rushed when sending things back. Be a bit more mindful of what you are doing and don’t be shy about alerting others to mistakes. Slow down, stop and think and stuff like this won’t happen. :-)
I had a similar experience with an EBay video card purchase the seller claimed was new. I videoed unpacking the GPU and narrated every indication of how the card was in fact used. The seller ended refunding my money because he had no choice… Always document!
One thing I neglected to mention in my earlier comment is, if you receive any package that looks like it may have been tampered with or shows obvious signs that someone at some point during transit opened it, your first reaction ought to be to report it first to the shipper, in this case UPS. If it comes back that UPS didn’t mess with the box, then report it to Amazon. Perhaps that is one thing Patrick should have done before filing claim with Amazon, just for thoroughness’ sake. That’s what I would have done.
I buy a huge amount of product from Amazon for work and also have had something similar happen.
In my case, I ordered a 4K monitor and had a 1080p monitor delivered instead.
My suspicion was that it was an inside job involving coordination between the shipper and a driver.
My reasons for this shipping are as follows:
Every single Amazon order I make is delivered by an Amazon driver in a grey van. The Amazon distribution centre is only ten minutes from my house.
The Amazon box had black Amazon tape over both cracks however on the bottom of the box the tape had been sliced open and clear packing tape was placed on top.
The monitor box had also been opened.
My suspicion was that someone in the distribution centre saw a monitor order on the list of orders to pick and then placed an order for the similarly sized 1080p monitor. In my area you can get same day delivery even if you order partway through the day so it’s not impossible for the similar monitor to be ordered and get on the same truck.
The driver would just need to know which two boxes needed to get switched and he could stop somewhere along his route to do this.
So was the purchase from Amazon.com or from the Amazon warehouse? Reason I ask is that I have always ordered from the .com and never the warehouse as I have been under the assumption that if you order from the warehouse you might be sent a used item.
Looks pretty obvious that the box and product were tampered with during shipping.
You didn’t get scammed. There was a mistake, and they immediately took action to help you out. If they denied a return/refund, then yes, you got scammed, but that didn’t happen.
This is a contender the biggest “nothingburger” story that I have seen published on STH. Not only were there no “nefarious intentions” (except presumably that someone stole a 3090), but to describe it as “dangerous” and “disastrous” is completely off the rails. So much hyperbole to describe a simple Amazon return. And to compare it to the Megatrands story… I don’t think the DoD orders their 3090s from Amazon :)
Patrick, your coverage of industry trends from Ice Lake & Epyc, to Broadcom & VMware, to TinyMiniMicro is always spot on. It is unfortunate to see content such as this which has slid down to the level of YouTube clickbait.
Amazon has run their internal numbers and have found it much easier to outsource the “part checking” to the end consumer. It’s much more cost efficient to quickly move the item out and let the end consumer identify bad returns than hiring more returns staff that have to be professionally trained.
This article should be titled how I got scammed by amazon and how terrible there quality control is … And how you should probably not trust there sellers
If I received a box in that abysmal of a shape, I would have refused delivery and complained to both Amazon and the carrier. I suspect foul play in the transit of the item as have some others. But even if that package was mangled that badly from employee abuse/misconduct, at the very least the card would have been damaged and needed to be returned anyway.
I don’t know why you guys still buy at Amazon. Not only are you enabling a slaver company, but you are also contributing to the decline of small businesses. You’ll have less hassle buying your hardware from your local computer store. The amount you save in not buying from Amazon is already worth it in gold.
Laurence, your comment is worth its weight in gold. Probably more.
@Laurence
I don’t know where you live, but where I live (New Orleans metro area, specifically Metairie) we kind of don’t have dedicated computer stores any more. We USED to have CompUSA, but that went under long, long ago. The only place we have that I know of that could even remotely be called a “computer store” is Best Buy. We USED to have small local shops, but I don’t think we have ’em any more, though it’s been a long time since I last went out shopping for a PC. We don’t have the luxury of a place like Micro Center in Metairie, we just don’t. Even if we did, it’s all too likely that I’d never be able to buy older used processors for my CPU collection. That kind of thing can only be gotten from either Amazon or eBay. Maybe I am supporting a “slaver company” that’s “contributing to the decline of small businesses”. It’s a necessary evil for an amateur chip collector on a extremely limited budget.
I’d like to see YOU come set up a store that deals in used server, mainframe and oddball architecture processors and computer systems and components RIGHT HERE in Metairie, LA and have a full stock or ready access to one at prices per unit of $50 or less for any architecture and any class of system. I’d be more than happy to buy from you.
You’re full of crap to say what you just said, whether its worth any amount of gold or not. I don’t lie when I say that southeast Louisiana is a supply chain DESERT as far as technology-related jobs and stores are concerned.
Newsflash: Its already too late to stop shopping at Amazon. They won. Bezos built his rocket and went to space and we all paid for it either via shopping there, using AWS or through govt handouts.
The computer stores are all gone. Whats the best in the USA? MicroCenter? Whats 2nd? There are no more Frys/CompUSA/TigerDirect. Other online retailers already do the same crap with “marketplaces” and selling returned items (NewEgg/etc).
I live in a town with a pop over 1M. There is no national computer chains here. There are some mom & pop computer repair places, but that’s it. These places have been on life support since the Computer Shopper days. All that’s left is retail electronics stores that don’t really specialize in anything people on these forums purchase.
Also, this is just false… “You’ll have less hassle buying your hardware from your local computer store. The amount you save in not buying from Amazon is already worth it in gold.”
Less hassle buying locally? The place in the strip mall that fixes computers? They’re really gonna have stock of the exact DIMM I want? Is it not a “hassle” to wait for them to order it online themselves, have it shipped in, and still be at any reasonable price compared to amazon/etc? Should I pay more just to keep them open?
Also, what are you saving not buying from Amazon? It isn’t money. Maybe its frustration if something goes sideways like the post above, but I think there are mechanisms to fix these scenarios either with Amazon or the purchaser’s card agreements.
I miss when local computer stores were relevant. I also miss all the chain computer stores that once existed.