Tuesday, December 18, 2012

Passthrough PCIe devices from ESXi to FreeBSD - Leave MSI Enabled to Avoid Interrupt Storms With More than One vCPU

If you want to pass a PCIe device (such as a LSI SAS controller) through ESXi to a FreeBSD VM, you're probably aware that you may need to disable MSI/MSI-X interrupts to make it work.

However, I'm finding that while this setting will make the system boot, if you add a second vCPU, you will quickly stat to see Interrupt Storm messages on your console, and one of your CPU's will be nearly 100% used on interrupt tasks.

The easy fix for me? Only disable MSIX, not MSI.

Here are the relevant lines from my /boot/loader.conf:

hw.pci.enable_msix=0
hw.pci.enable_msi=1

All of my interrupt storm issues are now gone. 

2 comments:

  1. Just a heads up for FreeBSD 9/stable you currently need to add a 3rd line(hw.pci.honor_msi_blacklist=0). I've sent the maintainers a patch that should remove the need for all 3 lines, keeping my fingers crossed that it does fix the problem.

    BTW: This blog has helped me a ton with FreeBSD, ZFS, ESXi, and NFS.

    ReplyDelete
  2. pbucher: I'm glad my blog has been of help to you - That was my goal in creating it in the first place.

    I'm very interested in your patch, could you share it here?

    [Send a commend with your email (I won't make it a published comment) if it's too much of a hassle to fit into a comment box, and I'll give you my email. ]

    I'll look into the honor_msi line shortly, as I still have grief with MSI/MSIX for passthru and FreeBSD.

    ReplyDelete