Passthrough PCIe devices from ESXi to FreeBSD - Leave MSI Enabled to Avoid Interrupt Storms With More than One vCPU

If you want to pass a PCIe device (such as a LSI SAS controller) through ESXi to a FreeBSD VM, you're probably aware that you may need to disable MSI/MSI-X interrupts to make it work.

However, I'm finding that while this setting will make the system boot, if you add a second vCPU, you will quickly stat to see Interrupt Storm messages on your console, and one of your CPU's will be nearly 100% used on interrupt tasks.

The easy fix for me? Only disable MSIX, not MSI.

Here are the relevant lines from my /boot/loader.conf:

hw.pci.enable_msix=0
hw.pci.enable_msi=1

All of my interrupt storm issues are now gone. 

VMWare Enhanced vMotion Compatibility (EVC) Not Needed between AMD Bulldozer 6234 and AMD Piledriver 6348 on VMware ESXi 5.1

I recently upgraded one of our datacenter servers from an AMD Bulldozer 6234 to a AMD Piledriver 6348.

I did find the new chip was faster for memory/CPU operations in our VM's, but I'll post about that in a different article.

What surprised me was that I didn't need to turn on EVC to allow VM's to migrate between hosts running the different CPU's.

Either VMware 5.1 doesn't recognize the new instruction set, or they don't matter in terms of migration.

Either way, I can migrate to the newer chips without worry, and since the 63xx are socket compatible with the 62xx and 61xx series, I have a fairly easy upgrade path for our servers if I wish.

ESXi 5.1 running FreeBSD 9.0 / 9.1 and VMware Tools

Note - I have another article for ESXi 5.0. These instructions won't work for a 5.0 box.

ESXi 5.1 includes support for the official VMware Tools on FreeBSD 9.0 (yay!).

I'm somewhere between 9.0 and 9.1 RC3, but I feel these instructions will work for either. 

I'll be testing the throughput of the vmxnet3 driver under FreeBSD shorty. For now, I can just confirm that bothe vmxnet2 and vmxnet3 drivers show in FreeBSD, and can ping.

Here's a quick script to install it for you.

You will need to start a VMware Tools install from the guest before running this.


echo "Make sure you have started the vmware tools install"
mount -t cd9660 /dev/cd0 /media
cp /media/vmware-freebsd-tools.tar.gz ~
cd ~
tar xvf vmware-freebsd-tools.tar.gz
cd vmware-tools-distrib
./vmware-install.pl
echo # DONE. Remember that the vmxnet3 driver is called vmxnet3f0

ESXi 5.0 running FreeBSD 9.0 / 9.1 and VMware Tools

Note - I have another article for ESXi 5.1. If you are running 5.1, these instructions won't work for you.

Installing the proper VMware Tools for FreeBSD 9.0 / 9.1 can be a pain.

The hard work to figure this out has already been done by others (http://ogris.de/vmware/), so here is a quick script that will install the official VMware Tools using Dru's patch.

Make sure to start a vmtools install from your vsphere console so ESXi will make the CD available to you.

If you run into problems with a missing library when you try and execute /usr/local/bin/vmare-toolbox-cmd, try looking here (http://lists.freebsd.org/pipermail/freebsd-questions/2010-June/217718.html) it solved the problem for me. 

(Cut-n-Paste into a terminal window with root access)

echo "Make sure you have started the vmware tools install"
mount -t cd9660 /dev/cd0 /media
cp /media/vmware-freebsd-tools.tar.gz ~
cd ~
tar xvf vmware-freebsd-tools.tar.gz
cd vmware-tools-distrib
cd lib/modules/source
tar xvf vmblock.tar
tar xvf vmmemctl.tar
tar xvf vmxnet.tar
tar xvf vmxnet3.tar

fetch http://ogris.de/vmware/vmxnet.diff
fetch http://ogris.de/vmware/vmxnet3.diff

echo #
echo # If it's thinking you have a previously applied patch (-R) say yes.
echo #

patch -p1 < vmxnet.diff
patch -p0 < vmxnet3.diff


cd vmblock-only
make && make install
cd ..

cd vmmemctl-only
make && make install
cd ..

cd vmxnet-only
make && make install
cd ..

cd vmxnet3-only
make && make install
cd ..

cd ~/vmware-tools-distrib


./vmware-install.pl

echo # DONE. Remember that the vmxnet3 driver is called vmxnet3f0

Blogger Code Syntax Highlighting

I really like Wordpress.org's Syntax Highlighter.

http://wordpress.org/extend/plugins/syntaxhighlighter/screenshots/

Particularly the button to copy the raw code without the line numbers.

Unfortunately, this is not easy to implement on Blogger.

The closest I have managed is from this blog:

http://www.cyberack.com/2007/07/adding-syntax-highlighter-to-blogger.html

Which does work, but we're missing the important buttons. Here's a demo of it in action on my saturate.c code - Note Blogger still completely mangles my code after a few lines, and we're missing the important buttons that Wordpress has.

Time to switch to Wordpress, or does anyone have any bright ideas?

(Update - Pastebin seems to work well for me, but it's an external link, not quite the same thing.  http://pastebin.com/4SexdvLq )

Saturate.c



#define _FILE_OFFSET_BITS 64  /* enable large file support  */
#include 
#include 
#include 
#include 
#include 

float nCM=0; /* count of 100's of megs we've written */


void finish(const char *where) {
        printf("%s: wrote %.1f gig file\n",where,0.1*nCM);
        exit(0);
}

void hitError(const char *where) { perror(where); finish(where); }

void writer(int id)
{
        char c=0; /* The byte we're writing */
        int cm=100*1024*1024;
        FILE *f[50];

        int     r;
        int     block[262114];
        int     i;
        char    fName[40];
        int     numFiles=4;

        printf("This is the %i writer, going to create %i files.\nOpened : ", id, numFiles);

        for(i=0; i
(end Code) 

Fixing "Open File - Security Warning" prompts in Windows 7 / 2008

If you enable Folder Redirection from a GPO, or even if you are trying to run a program from a non-local source, you'll run into this dialogue box:

For me, it was particularly annoying as it would happen when accessing any item from the start menu on our remote desktop server, as we were redirecting the AppData folder as well.

The fix is through GPO. I assume you know how to make a new GPO, and so to save time, here is the location.

User Configuration - Administative Templates - Windows Components - Internet Explorer - Internet Control Panel - Security Page  ... then it's Site to Zone Assignment List

I set both values for my file server here. You could also use a wildcard like *.yourdomain.com if you liked.

2 is the value for Trusted Sites. Check the help in the GPO Management box for other options.

The only downside to this is that now the user can't add their own trusted domains, as once it's specified in GPO, it can't be altered by the user.

Running Windows 2012 Server RC on ESXi 5.0 U1

If you're wanting to play with the upcomming Windows 2012 RC trial under ESXi 5.0 U1, you may find that it hangs if you're running your Virtual Machine as "Windows 8 (64 Bit)".

You need to set your Virtual Machine to "Windows 2008 R2 (64 bit) not "Windows 8 (64 Bit)". Once you make this change, it boots, and looks to run smoothly.

Vmware tools looks to install correctly, I can power up and down, and I can use the VMXNET3 network adapter in Windows. I also have the PVSCSI adapter enabled, and it looks to be running well.

I'm curious to test out it's built in iSCSI Target / clone abilities for running diskless (iSCSI Boot) Windows 7 Pro workstations.

Oh, and if you want to be able to use 2012 / Windows 8, check this site for where to find simpe things like shutdown and control panel.  I'm not sure it's wise to change such a staple of Windows useage, but it does make for a cleaner interface.

So far, I like the new interface.. we'll see how useful it is after extended use.

ZFS raidz - Make Sure To Select the Correct Number of Disks

I've read that selecting the correct number of drives when building a raidz ZFS array is important if you are trying to maximize performance.

 http://mail.opensolaris.org/pipermail/zfs-discuss/2010-September/044701.html

Of course the best way to maximize performance is to use a straight stripe or a mirror instead of a raidz, but that isn't always an option when you have price, power, or physical space restrictions.

Since I'm perpetually curious, I set up tests to compare the write speed of a 3, 4, 5, 6, and 7 drive raidz array.

I used my saturate.c program to put the arrays under heavy write load, repeated 6 times, and took avg and std deviation.

My results are not clean enough to post, but they would suggest that the number of drives is important - Follow the rules below;

RAIDZ1 vdevs should have 3, 5, or 9 devices in each vdev
RAIDZ2 vdevs should have 4, 6, or 10 devices in each vdev
RAIDZ3 vdevs should have 5, 7, or 11 devices in each vdev

With some luck, I hope to have the time to revisit the tests and obtain results that are postable.

Supermicro USB Boot Issues, H8SGL

Thought I'd pass on a small tidbit;

A SuperMicro H8SGL(-F) won't boot from USB properly when there are more than 12 bootable items in the server.

In my case, filling my enclosure with drives surpassed this limit.  Quite the surprise, as it was working great with 8 drives in, but at 16 drives, a routine maintenance reboot left it unable to start again.

The symptom is : It will just act like there isn't a boot sector on the USB drive.

I'm unsure at this stage if it's just USB, or if it's any bootable device.

I'm guessing that the MB's BIOS keeps a very short table of bootable items, and as it discovers more, the first to be discovered scroll off and are no longer available, even when they still list on a F11 Boot Selection.

I fixed it by turning off the BIOS boot option inside my SAS adapter's BIOS, and everything was fine.

It goes to show that you _ALWAYS_ need to reboot a server after any hardware/software changes, just to make sure. If I had learned about this problem when we were in some sort of urgent situation where I needed to quickly power cycle this server, well, that would have been a bad thing.

 

ZFS Testing - Saturate.c

Benchmarking ZFS is hard.

ZFS is such a complex FS with multiple levels of cache that getting a good solid reading on it's performance is always difficult.

I was recently tasked with building a FreeBSD ZFS SAN for a client that consisted of 90 1TB Seagate ST1000DM003 6GBps HD's in 2 SuperMicro SC847 Enclosures. The heads were 2 Dell PowerEdge T710's, 96 Gig DDR-3, Dual Xeon 5620 CPU's. I connected the head to the enclosure with a LSI2008 based SAS card.

As always, budget was tight, but performance HAD to be there. I needed to find out for sure what the write speed of this array was going to be under ZFS.

The throttling-effect of using a single SAS card was one of the items I needed to check .

Since the SuperMicro SC847's use a LSI2x36 backplane, I don't have a full SAS channel available to all drives at all times.  I wanted to start collecting data on what effect this would have on the performance of the arrays, and did we need to use 2 SAS cards to achieve a higher performance? This wasn't a "performance at all costs" scenario, rather a real-world situation with real-world budgets and needs.

This client had a lot of users, and a lot of big databases that are very active during the day. I needed to know that the array would quickly write down as much data as possible to satisfy their needs.

Knowing that the average-write of the array wouldn't saturate the single SAS card or the drives, but a heavy-write could, I needed to put the array into heavy-writes for a sustained period of time to compare various hardware and software configurations.

Thus began my search for a benchmark program that could really load down this gear.
To simplify things, I decided that I would only concern myself with a write saturation event - When the data is flowing to the drives so quickly that they never catch up - As close to 100% utilization as possible.

 I didn't have much luck with the standard FreeBSD benchmarks.. bonnie, bonnie++, iozone, etc. Either they were too hard to lock into a saturate write, or they didn't spawn enough writers to really load down the system.

In the end I ended up coding a quick and dirty program called Saturate.c that I'm passing on here. It's not very pretty, but it works.

It's hard-coded to spawn 4 files to write to for each forked child process. Tweak it as you need to for your system to really drive the files.

You can simply execute the file using time to check how long it takes like this;

time ./saturate

I created a simple script file that then executed the saturate program in different zpool configurations so I could confirm what was working best for us.  I've run through various raidz configurations, compression, etc. I'm still sifting through all of the data.

Warning: This will create a very large set of test files - I believe in it's current state it writes 40,000 MegaBytes (~40 Gigs).

Oh, with FreeBSD 9.0, my best average time was 1.2 Minutes.  Not bad for a Free Operating System.

I'll post some of my results of 2 months of various ZFS tests over the next while as I have time.

Code on PasteBin: http://pastebin.com/4SexdvLq