About a month ago we published a piece showing how trying to install VMware ESXi 6.5 on AMD Ryzen and getting a PSOD. Since then, we confirmed that AMD Ryzen plus Windows 10 will yield a usable VMware Workstation 12 Pro setup. Now we have the “fix” to both the VMware PSOD and many other Linux kernel issues: disabling SMT. This is frankly not a good fix. Disabling SMT will yield a 30% (or more) digression in performance. That means that your 8-core Ryzen 7 with SMT off will often be out-performed by a 6-core Ryzen 5 with SMT on. At our STH, our stance is this. If a fundamental feature (e.g. one of the top 5 specs every CPU vendor touts) cannot be used in conjunction with an OS from installation to stable operation, we do not call that working.
Alas, we are going to show how to get something at least running.
Setting Up AMD Ryzen with VMware ESXi 6.5
We took a fairly simple test system and got everything ready to install ESXi. Here is the basic configuration:
- CPU: AMD Ryzen 7 1700X
- Motherboard: ASUS Prime B350-Plus
- Cooler: Noctua NH-L9x65 SE-AM4
- 64GB G-Skill DDR4 (4x 16GB)
- SSDs: 16GB Kingston m.2 2242, Intel DC S3700 800GB
, Intel DC S3700 100GB
- PCIe Networking: Intel XL710-QDA2
(dual 40GbE) and Intel Desktop Adapter 82572EI (1GbE)
- Video: EVGA GeForce 710 2GB
- KVM: Lantronix Spider external KVM
We knew the ASUS motherboard’s Realtek NIC was not ideal for VMware so we added the XL710-QDA2 40GbE adapter as this setup in our DemoEval lab and various software companies have used it to patch their distributions at this point.
The Video
Here is the video where we get the AMD Ryzen 7 system to install VMware ESXi 6.5 past the PSOD we received in the previous iteration.
Again, as a major caveat, doing this will have significant detriments to the AMD Ryzen 7 system performance to the tune of 30%. While we expect RHEL, CentOS, Debian and Ubuntu systems to be patched soon for Ryzen, VMware is likely a longer timeframe until we expect it will work.
Major Caveats to Getting ESXi 6.5 Working with Ryzen
Here are some caveats you should be aware of if you want to try this.
- Not all BIOS versions have had options to turn SMT off. For example, the ASUS Prime B350-Plus we are using did not get a BIOS update with this option until mid-to-late March 2017.
- As of this writing, Ryzen platforms do not have iKVM or IPMI so remotely deploying them is still not advisable. We, therefore, do not recommend Ryzen systems as “servers” in remote data centers. We have a solution that has been working for the half dozen machines in DemoEval but it is far from ideal.
- Many motherboards will use unsupported NICs. Our test system above has an Intel XL710-QDA2 40GbE card installed. Otherwise, ESXi would not find a supported NIC and fail the installation process.
- Just because something can be installed that does not mean it is ready for stable operation. Ryzen outside of Windows is still not stable enough to be deployed in data centers.
- We know a number of folks are interested in Ryzen ECC support. Now that we have hosted almost three dozen different parties, and with ten test systems set up, we are confident in saying that outside of Windows, the stability impact of ECC memory is negligible. There are still unpatched software issues that will take your system down well before a potential memory error will.
- Given the above, consider this an AMD enthusiast only workaround, and we do not recommend it.
Hi There, All I want to know is will it work in My Lab.
I already have a bunch a PCI-E Intel NIC`s I just want the more Threads and RAM.
Hello Patrick,
I would like to try out to install an ESXi system with AMD Ryzen. Even if I loose 30 % of performance it seems me better then a I7 7700K. And in the future I think that VMWARE will update the Linux kernel and I could re-switch on the SMT on.
But I’m a bit confused about the problem with the M.2 SSD drive. Is that just a problem for the boot disk or is it impossible to use M.2 devices even as a datastore?
Here is how to make ESXi work on Ryzen, if your ESXi reside in a physical disk, (1) install a bootable with new Linux kernel 4.11 such as Ubuntu 17.04 (2) install VMware workstation. (3) run as adminstrator and create a virtual machine with physical disk where old ESXi was running before. (4) start the virtual machine. You will enjoy the power of Ryzen like never before!
There’s some good news over on VMware VMTN. Looks like the issue might be related to the number of cores / threads being greater than 15. Sounds like Ryzen 5 might actually work! Now if someone could just get the built in network card and SATA working too :)
See it here:
https://communities.vmware.com/thread/561795
@alt We cannot specifically comment on why for another few days.
The fix is incoming. It is not a 15 core issue.
It’s been a few more days… any news?
Bill – I expect we will be talking about AMD and servers next week https://www.servethehome.com/amd-epyc-june-20-2017-threadripper-vega-computex-2017/
How about some standard functionality on Ryzen 7 1700x? Any way to get VMware working properly yet? Kinda sucks that my laptop out performs my 16 thread desktop right now…
Any updates yet?
I really want to replace my E3-v3 and my 4690k with 1700x’s. I use ESXI on the xeon and Workstation on the i5 on a constant basis. I cannot upgrade until these are functional. Hurry up AMD!! I need more than 32GB of ram in my Xeon machine and would rather AMD than Skylake.
We expect 6.5u1 to support Zen due to EPYC.
Has anyone heard word of EPYC / ESXi?
Next ESXi update.
Now, since last Thuesday ESXi 6.5 U1 is out. In the update documentaton I find hints for a very old Linux kernel 2.18 so I’m really fear that even with the new release of ESXi Epyc will not run propely. I hope you test it soon.
Esxi 6.5 U1 is out and I can verify it takes care of this issue. We updated to that and enabled hyper-threading and it completes boot up and shows the 8 cores with the 16 logical processors and its working like a champ.
Great @seneca
@seneca, none of this NPT troubles that Ryzen gives for KVM/QEMU use then? It seems with KVM we have a choice of poor performance for CPU or for Graphics… Does ESXi work with full near native performance on CPU and GPU?
Hi all! I just installed ESXi 6.5.0 Update 1 (Build 5969303) on my Ryzen 7 minilab and it works fine with all cores enabled!! :-D
I managed to run Windows 7 VMs and it’s sooo sweeeeeet! I never had so smooth and fast VMs like this. Ryzen is the king of virtualization and heavy multitasking. I warmly recommend it for ESXi hosts. Unbeatable price/performance ratio and really great, fast and smooth experience. I have 16 CPU cores for less than 300$ and AMD’s Simultaneous Multi-Thread beats Intel’s HyperThreading without breaking a sweat!
The only issue I had during install was due to the NIC drivers as ESXi 6.5 doesn’t support Realtek NICs by default anymore. But I easily found the way to customize ESXi install iso with the Realtek drivers for my integrated NIC with ESXi Customizer and the .vib package for Realtek drivers available here:
https://www.v-front.de/p/esxi-customizer.html
https://vibsdepot.v-front.de/wiki/index.php/List_of_currently_available_ESXi_packages
So I can only recommend Ryzen CPUs for ESXi labs hosts in a small form factor (mine is a micro-ATX build). And for production hosts I think Threadripper or Epyc CPUs should be warhorses. The AMD’s Zen architecture is truly amazing!
My build (about 750$ at Newegg) :
MB : AsRock A320M Pro 4
RAM : G.Skill RipJaws 5 Series 16 Go (2x 8 Go) DDR4 3200 MHz
CPU : AMD Ryzen 7 1700 (3.0 – 3.7 Ghz)
GPU : XFX Radeon R5 230 2G DDR3
PSU : Enermax NAXN 450W Tomahawk II
Case : DIYPC MA01-G (micro-ATX mini-tower)
HDD : 1 SSD 120 GB + 1 TB SSHD
which drive was used?
driver**