SMT Virtualization Licensing Impact
As we were doing the Mapping Licensing for Virtualization is Cool Now piece, something stood out. SMT is currently “free” in virtualization. This is a major reason that hyper-threading is important in the enterprise virtualization scene. Just for some sense:
- VMware (now owned by Broadcom) has been using physical cores, not logical threads for at least a decade.
- Microsoft Windows Server also uses physical cores, not logical threads for licensing. Perhaps the clearest place to see this is on the HPE Microsoft Windows Server License Calculator tool that explicitly states this.
- Red Hat OpenShift uses physicla cores, not threads which is important especially as part of IBM.
In those three examples that are big in the server licensing space, the goal is to get the maximum performance per core. As a result, P-cores with full performance are strongly preferred over smaller E-cores with reduced performance. If SMT gets one more performance on that physical core, then that is worthwhile.
This might seem like a small detail, but the performance per core has largely kept Arm CPUs and the Intel Xeon 6700E series out of the enterprise data center, and caused the Xeon 6900E series to not be generally available to enterprises. To compete, the small integer cores without SMT have not been viable. VMware ESXi 64-bit Arm Support was announced in 2018. Tom Fenton and my book on running VMware ESXi on a Raspberry Pi (Amazon Affilate link) came out in 2021. It is 2025 and while the HPE ProLiant RL300 Gen11 attempted to break into the traditional enterprise data center, it did not sell particularly well and companies like Dell and Lenovo skipped similar products.
There are merits in some markets for non-SMT cores, but enterprise licensing prefers SMT and that is a big force in the industry.
You Can Use A SMT CPU and Disable SMT
At this point, you might be wondering what happens if you have a HPC or another workload where you want SMT off. Are you stuck with a SMT=2 CPU? The answer is no. You can change it quickly.
SMT off is easy. A few months ago, we had A Guide to Quickly Enable and Disable SMT and Hyper-Threading on Ubuntu and Debian. While that works for Linux within a few seconds, for other OSes, one can disable SMT in the BIOS. In case you were wondering, we also cover in that guide how to enable SMT in Linux so you do not have to reboot.
To turn SMT on, use:
echo on | sudo tee /sys/devices/system/cpu/smt/controlTo turn SMT off, use:
echo off | sudo tee /sys/devices/system/cpu/smt/controlPerhaps one has a cluster that is going to transition into an HPC role, and then a quick reboot to change the SMT setting might help performance. While you are in BIOS, you are also likely changing NPS settings on the system.
Final Words
At STH, we have probably not given this enough discussion as the vast majority of server CPUs we test have SMT. Today, the non-SMT parts are not hitting thread/ vCPU counts that are higher than the SMT parts. For example AmpereOne at 192 cores/ 192 threads versus the AMD EPYC 9965 at 192 cores/ 384 threads per socket. Even with cores designed to more, the EPYC 9965 has higher per-socket thread density. If you did not wis ant SMT, you can turn it off and you are at the same 192 core/ socket density. So at best, SMT is ahead, and at worst it is even.
This feels a bit like a critical moment for the SMT versus non-SMT debate in the server market. AMD is fully on the SMT bandwagon given the performance provided versus the small transistor count cost and miniscule power consumption cost (we often measure <3.5% delta between SMT on and off at a system level which is a margin of error.) Intel in 2025 is also mostly on the SMT bandwagon after it decided not to GA its Xeon 6900E series without SMT due to lack of demand. On the Arm side, there are two viable general-purpose Arm server vendors. Ampere, which was just bought by Softbank after achieving limited enterprise adoption does not currently support SMT. NVIDIA is the other Arm server vendor but is focused on the performance end of the market and just announced its next-gen will adopt SMT. At the high end IBM has been a big proponent of SMT. Cloud providers, of course, have their own chips, but for the enterprise market, it feels like after many years of debate, the answer seems to be that, outside of some specific applications (e.g. low-power network appliances) the P-core with SMT combination is continuing to be the go-to option.
Maybe the bigger fact is this: Despite having access to just about any type of technology we want, we are still deploying P-cores with SMT=2, and leaving SMT on into our production clusters. Seven years ago, I am not sure that I would have said that would be the case in 2025, but in 2025 that is the clear answer for us, and I think many other folks out there.
There were rumors that AMD was considering SMT=4 for Epyc a couple years ago. Would have been cool to see.
I’ve always understood Arm CPU guys trumpeting that they’ve got no SMT as a feature as it’s B.S. And that doesn’t stand for Blue Smoke
It should be pointed out that, while Cavium’s initial design for the ThunderX2 used a single-thread-only core design, the final production ThunderX2s that were released (CN99xx) were based on Broadcom’s Vulcan core and that core supports SMT4, four threads per core. Technically, this means that the ThunderX2 does, in fact, have SMT. Patrick isn’t wrong, but he isn’t entirely correct, either.
No Nutanix AHV? No idea how you can have Redhat virtualization as enterprise level, but forget the only real enterprise grade alternative to esx….
I am not so sure the idea is clear cut, and that is ignoring security issues with SMT. For cloud vendors with vCPU unit selling price that is very nice way to sell it. For x86 that might have make sense. but once you start comparing the core size of ARM without SMT is also smaller.
At the end of the day it is just about price / performance ratio. And COGS of CPU is only part of the calculation without also factoring their margin, R&D, support etc.
You’re missing one issue here *against* SMT — the shared state between threads makes spectre/meltdown type attacks much harder to defend against. It’s really hard to avoid leaking cache miss data between threads, as there’s just too much in common between them.
It’s not necessarily that big of a deal when you own both threads, but when someone else’s VM is running on the other half of “your” CPU then you’re at risk of them being able to extract your encryption keys, etc just by cache timing attacks. We *might* have closed all of the known attacks today, but I’d be shocked if we don’t continue finding more over time.
Spectre and Meltdown was over 7 years ago. I don’t think 2025’s gens of CPUs with Hyper Threading have Spectre and Meltdown vulns
Outside of small Linux DNS or ancient Server 2003 systems, almost nothing runs well on a single core anymore. Even a simple Win 10 Jumpbox needs 2t to be usable. Therefore there really isn’t any reason to not use SMT to double your vCPU.
Is that licensing per physical core a major reason why some places deploy IBM Power Systems? Those eight threads per core (also in Power 10) might be a strong incentive.
@eastcoast
People deploy Power because they went Power years ago and cannot get off that platform. On a performance, and especially cost, aspect Power cannot compete with x86 even with SMT 8. Some software, IIRC, like Oracle licenses per vCPU on a virtualized platform so SMT 8 doesn’t help you reduce cost.
Thanks for the response! Yes, it seems that the POWER user base keeps shrinking by the year. Was also disappointing that IBM abandoned Open Power after POWER 9.
And, I’m a bit amazed that Oracle is still charging by the thread.
I’m glad to see there are options if I “did not wis ant” to buy a server CPU with “cores designed to more!”
Zen 5 was specifically designed for SMT. It features dual instruction decoders – one for each thread. The second decoder is not being actively used for single threaded use cases, in contrast to Intel Gracemont’s clustered out-of-order 3+3 decoder design for example.
The anti-SMT sentiment is based on older designs, especially Intel, that did lower overall performance in some use cases. This is similar to how early AVX-512 implementations lowered performance vs. AVX2 due to heavy downclocking.
In both cases AMD’s designs seem to have the edge, but Intel’s later designs have mostly caught up.
@geoff
In order for Nutsnix AHV to be considered an esxi alternative it has to be comparable on price. I can tell you it isn’t close to the same price. Their “mid level” license is as expensive as the top esxi license. That makes it a hard no for a lot of companies.
I don’t think many people have an issue with SMT. But the original HT, man, that was a mess.