Oftentimes people talk about passing a RAID controller/ HBA/ USB drive through to a guest OS in VMware ESXi (this guide is based on 4.1 and works for ESX too) using VMDirectPath I/O or disk through disk in Microsoft Hyper-V. This is especially useful when one either needs direct RAID controller/ HBA access for an operating system which is recommended for OpenSolaris/ FreeBSD solutions or USB access for an OS installation. It is a fairly simple process, so I figured I would provide a quick guide. Before embarking on this process, please note that the server will need to be restarted mid-process so this is not something that should be done on an ESX host with critical virtual machines running on the host.
Test Configuration
- CPU(s): Intel Xeon X3440
- Motherboard: Supermicro X8SI6-F
- Memory: 2x 4GB kits of Kingston ECC 1333MHz DDR3 KVR1333D3E9SK2/4G (Unbuffered)
- Case: Norco RPC-4220
- Drives: 8x Hitachi 7,200rpm 2TB, 2x Western Digital Green 2TB, 6x Seagate 1.5TB 7,200rpm
- OS(es): VMWare ESXi 4.1
Configuring VMDirectPath
First, make sure that your CPU and motherboard support Intel VT-d or AMD IOMMU. Once you are sure that the specs support VMDirectPath, you should double-check that the features are enabled in the BIOS. If the system is remote, you can still do this with IPMI 2.0 and KVM-over-IP. Once this check is complete, let the server boot.
Next log into the vSphere client and connect to your server. Select the appropriate server and go to the Configuration Tab.
From there, go to Advanced Settings where passthrough can be configured by clicking the button labeled “Configure Passthrough”
A window will then pop up asking to select a device to pass through (note actually associating the passthrough device with a VM occurs later.) Make sure that you select the correct device, a lot of USB controllers for example look very similar on this screen, so you may need to download and view the motherboard manual when figuring out what to pass through. Once the correct check box has been selected, click OK.
At this point, one needs to do a step I forgot my first time doing passthrough of a controller… and that is actually rebooting the ESXi host.
After the reboot occurs, the next important step is to make sure that the virtual machine where the PCIe or USB device is being passed through to is in an off state. This is important for machines where the target VM automatically starts up upon server boot.
Once you are sure the VM is off, go to the Summary tab -> Edit Settings and then add a PCI Device followed by clicking next. From the installation wizard that opens, pick the PCI/ PCIe device from the drop-down menu followed by clicking Next and Finish. After this is complete, wait a few seconds while the VM configuration is completed and then you can start the VM. Personally, I prefer doing the above configuration before installing an operating system to ensure the highest chance of working drivers being installed in the initial installation.
A word of caution: not every device will work with this feature, but I have had nothing but luck installing RAID controllers in this manner. Another note is that installing a RAID card as a passthrough device will mean that the attached disks are not available for other VMs. So best practice is to make sure that every drive attached to the RAID card or HBA is destined to be attached to the VM in question. I often use either onboard SATA or a dedicated controller to host ESXi and drives for other VMs while passing through a RAID card (often with a HP SAS Expander) to VMs.
Enjoy!
I’ve never played with passthrough – can vmware use an nfs datastore from a vm running on the same box in this model? or is there a possibility of priority inversion?
1. install vsphere on a usb drive
2. configure passthrough on a LSI 1068/2008 (it mode), or an AOC-SAT2-MV8
3. install a solaris 11 express VM using drives on the passthrough device and enable a nfs share
4. enable autostart on the sol11 install and configure a vmware datastore using the sol11 vm’s nfs share
5. create vm(s) using the sol11 vm as a datastore ( they all will need a delayed start )
will it work?
Handy tutorial as I will need to be doing this myself in the near future! Haven’t had much chance figuring it out, but does the i3-530 support VT-d? Considering that I will need to find non-ECC memory that the X8SIL-F can use, I wonder if it might be better to go Xeon :/
Sadly, the i3-530 does not support VT-d probably because it is meant to be a budget processor for consumer systems rather than running virtual machines beefy enough to have RAID controllers passed through :-) Intel does tweak features based on processor family which is why ark.intel.com is such a great resource.
Bummer, but I guess that just means I get to have a heftier server, thanks for the heads up and the excellent link Patrick!
good stuff Patrick.
Hi – can you please tell me exactly which make/model you used that worked well with VMDirectPath? I tried a cheap(er) model and it didn’t pass things through properly. Thanks!
It is also possible to Passthrough individual SATA disks to a VM in ESXi.
Please note that this proceedure is unsupported and will give the careless admin the chance to hose their systems in several interesting and creative ways.
Here’s a link to the the best Howto I’ve found.
http://www.vm-help.com/esx40i/SATA_RDMs.php
I’ve done this and am currently running OpenIndiana in a VM with ‘VIRTUAL’ RDM Passthrough to 4 x 1.5TB drives as a ZFS RAIDZ1 storage pool with good results in a Home environment.
I did have data corruption issues with ZFS when trying a ‘PHYSICAL’ RDM Passthrough so I can only recommend the ‘VIRTUAL’ passthrough when using ZFS on RDM SATA drives in this way.
Good point Dan. I actually have an ESXi RDM pass-through article that I just haven’t had the opportunity to finish.
Hello Patrick,
Nice article. I happened across it when trying to do this on a X8DTH-6F. In my case however I have the SAS2008 RAID key and access to chip based RAID-5. Unfortunatly, so far, my results are no where near as good as yours. It’s been a long and frustrating weekend.
With the RAID key installed (and properly configured with the iMR BIOS from Nov-2010) ESXi identified the controller as an ‘LSI Logic / Symbios Logic MegaRAID SKINNY Controller’. I am able to create the array using the MegaRAID BIOS so I know that the firmware is functional. Unfortunatly this is where the fun starts.. Though Windows 2008 R2 SP1 and Windows 7 SP1 see the card as an ‘LSI MegaRAID SAS PCI Express ROMB’ but the latest LSI drivers that I could find (4.32) do not seem to work with the device.. The drivers will not start (Code 10). Before you ask… I only ran one of the VMs at a time and reconfigured each to use (and the other to not use) the LSI PCI device.
I have also tried a HighPoint 2720 and had no luck with it either. With the HighPoint Windows can see the card and the drivers load but can only see an existing array as a random set of initialized and unintialized disks.
Similarly I tried an LSI SAS 8308ELP. In that case windows was able to see the card and the drivers worked but the existing NTFS file system on teh array appeard only as a single uninitialized volume.
I’m now considering whether bump the SAS 2008 controller back to IT mode and give nexentastore a try or try a differnet hardware RAID card.
Which ones have you had success using? Is there something special I should be configuring on the X8DTH-6F motherboard? Is there some reason, that you can think of why the SAS2008 in iMR mode or the HighPoint would only partially work?
Thanks, in advance, for the advice.
Chris
P.S… Nice site… lots of very interesting content in an area that interests me greatly. Keep up the good work!
Thanks Chris.
I haven’t unlocked RAID 5 on any of these boards yet. Without onboard cache, the performance is generally not too great and RAID 5 with large consumer SATA disks can be scary at best. I also have not tried the Highpoint controllers as I tend to get a lot of users with issues and do not want to become the primary Highpoint support hub. :-)
Feel free to post in the forums, maybe someone there has experience with a similar setup. One other thought is to e-mail Supermicro support and see if you can get an updated driver.
Regards,
Patrick
Thanks for the pointers.
I will give the forums a try but the more I think about it the more I like the idea of trying out nexentastor with the SAS2 2008 in IT mode (probibly a bit faster then the IR mode). What I’m not looking forward to though is moving 9TB from the old array to the new over a network connection. A windows soft RAID array might be intereating too.
I think what I’ll do is put 4 disks on a Windows array and 4 on a nexentastor virtual and stress test the two.
I don’t blame you re the Highpoint. I genuinely hate that piece of hardware and regret ever buying it.
Just out of curiosity… Have you successfully used any proper (with cache) RAID cards with vmDirectPath?
Thanks Again
Chris
I tried enabling passthrough on my Supermicro/LSI 9211-8i setup. However, after enabling passthrough I wasn’t able to install Nexentastor because it was complaining about missing HD drivers. (BTW, I also found that a bare-metal install fails with the same error on my hardware…)
I therefore had to disable passthrough again, install Nexentastor and then enable passthrough. Now it seems to work.
Cheers,
teq
yes.
[…] one can passthrough the device to a guest VM directly on platforms that support VT-d. STH had a VT-d ESXi PCIe passthrough guide some time […]
[…] followed instructions to enable passthrough – enabled VT/d for CPU and motherboard, assigned the PCI card to the […]