VMWare Enhanced vMotion Compatibility (EVC) Not Needed between AMD Bulldozer 6234 and AMD Piledriver 6348 on VMware ESXi 5.1

I recently upgraded one of our datacenter servers from an AMD Bulldozer 6234 to a AMD Piledriver 6348.

I did find the new chip was faster for memory/CPU operations in our VM's, but I'll post about that in a different article.

What surprised me was that I didn't need to turn on EVC to allow VM's to migrate between hosts running the different CPU's.

Either VMware 5.1 doesn't recognize the new instruction set, or they don't matter in terms of migration.

Either way, I can migrate to the newer chips without worry, and since the 63xx are socket compatible with the 62xx and 61xx series, I have a fairly easy upgrade path for our servers if I wish.

Blank NFS Datastore under ESXi - Check your MTU

If you've used ESX/ESXi over NFS long enough, you have probably encountered a situation where you have a connected NFS datastore, but when you browse the datastore, it's blank, or only a few files are showing.



There are many reasons for this situation, and there is plenty of vmWare KB's about troubleshooting such a problem.

Turning on NFS logging is a good idea.

This is the standard vmWare Troubleshooting document.

One thing that you may not be thinking of is a MTU mismatch.

I had this exact problem tonight, and it unfortunately took a long time to figure out because I was suspecting FreeBSD as being the culprit due to some struggles I've had with it as of late.

I came across this by issuing this command;

vmkping -s 8500 san0 -s 8500

This tells the ping command to send a much larger payload than usual. My 10Gbe network is setup with a MTU of 9000, something I verified many times over, so I really didn't think this was the issue. I only entered it out of completeness after realizing this was a serious problem that wasn't going to be solved quickly, and I needed to start a proper documentation trail of what I was doing if I hoped to solve the issue.

I won't bore you with the details of my troubleshooting, and get to the point.

Turns out my new Dell M8024-k blade 10Gbe switch requires you to set the MTU on the LAG channel group separately from the ports. That's not uncommon, but as far as I can tell, this isn't in the M8024-k GUI anywhere, it can only be done via the CLI.

I guess I've been spoiled by LAG's that set their MTU based on the members MTU, and when I didn't see a separate LAG MTU setting in the GUI anywhere, I assumed that everything was working fine - After all, pings were working, and that didn't work in the past when I had the wrong MTU applied on different adapters (although that could also be an oddity with the FreeBSD lagg driver).

By default the Dell M8024-k makes LAG groups with a MTU of 1500. 

What's odd is that some connectivity is maintained with this mismatch. You can browse and list some directories, and even see some information - I believe the point where it really breaks is when it needs to transfer more than the 1500 limit allows.

Further oddness: FreeBSD didn't have problems mounting and browsing the directories over NFS that ESXi couldn't browse or list properly. This could be to FreeBSD connecting via UDP instead of TCP, at this point I'm not sure.

I thought I'd pass this along in case anyone else is forgetting about each point in your data transmission chain that a MTU mismatch could be affecting.

Speeding up FreeBSD's NFS on ZFS for ESX clients

My life revolves around four 3-letter acronyms: BSD, ESX, ZFS, and NFS.

However, these four do not get along well, at least not on FreeBSD.

The problem is between ESX's NFSv3 client and ZFS's ZIL.

You can read a bit about this from one of the ZFS programmers here - Although I don't agree that it's as much of a non-issue as this writer found.

ESX uses a NFSv3 client, and when it connects to the server, it always asks for a sync connection. It doesn't matter what you set your server to, it will be forced by the O_SYNC command from ESX to sync all writes.

By itself, this isn't a bad thing, but when you add ZFS to the equation,we now have an unnecessary NFS sync due to ZFS's ZIL. It's best to leave ZFS alone, and let it write to disk when it's ready, instead of instructing it to flush the ZIL all the time. Once ZFS has it, you can forget about it (assuming you haven't turned off the ZIL).

Even if your ZIL is on hardware RAM-drives, you're going to notice a slow-down. The effect is magnified on a HD based ZIL (which is what you have if you don't have a separate log device on SSD/RAM). For my tests, I was using a hardware RAM device for my ZIL.

Some ZFS instances can disable the ZIL. We can't in FreeBSD if you're running ZFS v28.

Here's two quick iometer tests to show the difference between a standard FreeBSD NFS server, and my modified FreeBSD NFS server.

Test Steup: Running iometer 1.1 devel,  on a Windows 7 SP1 machine, connected to the test drives via NFS. iometer has 128 writers, full random, 4k size, 50% write 50% read, 100% sequential access, 8GB file, 15 run-time. Reboot after each test, ran each test twice to make sure we were receiving a sane result. Using FreeBSD 9-CURRENT as of 2011.05.28.15.00.00

Standard NFS
Test1    1086 IOPS    4.45 MBs    117 AvgIO (ms)
Test2    1020 IOPS    4.18 MBs    125 AvgIO (ms)

Modified NFS
Test 3   2309 IOPS    9.45 MBs    55 AvgIO (ms)
Test 4   2243 IOPS    9.19 MBs    57 AvgIO (ms)

I feel the results speak for themselves, but in case they don't - We're looking at an increase in IOPS, MB/per sec, and a decrease in the time to access the information when we use the modified NFS server code. For this particular test, we're looking at nearly a doubling in performance. Other tests are close to a 10% increase in speed, but that's still a wanted increase.

These test results will be apparent if you're using the old NFS server (v2 and v3 only) or the new NFS server (v2-3-4) that is now the default in FreeBSD 9 as of a month ago.

I've used this hack for over 6 months now on my SANs without any issue or corruption, on both 8.1 and various 9-Current builds, so I believe it's fairly safe to use.

I'm too lazy to make a proper patch, but manually editing the source is very easy:

- The file is /usr/src/sys/fs/nfsserver/nfs_nfsdport.c
- Go to line 704, you'll see code like this;
(Edit: Now line 727 in FreeBSD-9.0-RC3)

if (stable == NFSWRITE_UNSTABLE)
  ioflags = IO_NODELOCKED;
else
  ioflags = (IO_SYNC IO_NODELOCKED);
uiop->uio_resid = retlen;
uiop->uio_rw = UIO_WRITE;

- Change the code to look like this below. We're commenting out the logic that decides to allow this to be an IO_SYNC write.

// if (stable == NFSWRITE_UNSTABLE)
ioflags = IO_NODELOCKED;
// else
// ioflags = (IO_SYNC | IO_NODELOCKED);
uiop->uio_resid = retlen;
uiop->uio_rw = UIO_WRITE;

- Recompile your kernel, install it, and reboot - You're now free from NFS O_SYNC's under ESX.


If you are running the older NFS server (which is the default for 8.2 or older), the file to modify is /usr/src/sys/nfsserver/nfs_serv.c  - Go to line 1162 and comment out these lines as shown in this example:

// if (stable == NFSV3WRITE_UNSTABLE)
ioflags = IO_NODELOCKED;
// else if (stable == NFSV3WRITE_DATASYNC)
// ioflags = (IO_SYNC IO_NODELOCKED);
// else
// ioflags = (IO_METASYNC | IO_SYNC | IO_NODELOCKED);

If you try this, and let me know what type of before and after speed results you're receiving.

Partition Alignment with ESX and Windows 2003. Just do it.

You really want to make sure your partitions are aligned.

Partition Alignment, in a nutshell, is making sure your partitions start on Sector # 2048 (or some multiple of 512) instead of the standard Sector # 63.

There are some great blog entries that define the issue, so I won't get into it here.

While you don't need to do this for 2008 partitions (they start properly aligned), I've always known I should align my older 2003 partitions, but I haven't bothered. Today I decided to do a simple test to see the results of running un-aligned.


Test Details:
OS: Windows 2003 32bit
Test Package: Performance Test 6.1, running Webserver disk role benchmark
VM: ESXi 4.1
Drives: NFS share to a Solaris 11 ZFS box.

I chose the webserver role because it's a heavy reader (writes are harder to benchmark on ZFS), and because it's my worst performer. The other roles are hitting 900-1200 MB/sec, which is close to the max I can drive through my 10Gbe card. 

Aligned Partition: 609 MB/sec

Non-Aligned Partition: 436 MB/sec

That's a 30% increase in read speed. Wow, that's a lot of speed to throw away for not aligning a partition. 

Here's a link to a quick little tool that will let you know within Windows if your partition is aligned: http://ctxadmtools.musumeci.com.ar/VMCheckAlign/VMCheckAlignment10.html

Of course, it can be a pain in the ass to align a partition. You can use a livecd of gparted, or create a new disk and partition it with Windows 2008, copying the data to the new partition from a seperate VM.

Some partition tools are now including the ability to quickly and easily specify a aligned-partition start, so hopefully this will be an easy task in the near future.  If you know the name of any easy booting GUI type partition tools that can move a Windows partition to sector 2048 without much fuss, please let me know and I'll post it here.

I personally do it with gparted live, but I'm also not affraid of the command line. Something easier would be nice.

ESXi 4.0.0 vs 4.1 Speed - 4.1 Not as fast

I've just upgraded my 4.0.0 ESXi box to 4.1.

...Well, lets not call it upgrading, the process I used was to pull my existing boot drive, install a new 8 gig  USB stick inside my Dell T710, and install a fresh copy of the Dell Installable 4.1 ESXi. This results in a fresh copy, and all I have to do is reattach my storage, import my .VMX files, and I'm back in business.

I once had a bit of a mess with the upgrade process from 3.0 to 3.5, so I settled on this method as being he safest. I even pull my drives when I do the upgrade, so there is no chance the installer will become confused and format my main datastore. Am I paranoid? It's happened before, and you only have to burn me once..

The process isn't that much more time consuming than an in-place upgrade. When your ESXi files are all on one drive/flash stick, and your datastores are all on another, you've got plenty of flexibility. I was able to do it within 30 minutes, because I really don't edit my ESXi configuration that much from default.

 I'm very interested in 4.1 because it has some neat power management interfaces. Look here, I can now track my wattage being burned on this server:

This is for a Dell T710 with 24 gig DDR3 memory, 2 Xeon 5660's,  4 600Gig SAS drives, and 4 1TB nSAS drives. I've yet to check how accurate this is with a KilloWatt or similar meter, but this sounds about right.


I'm also interested in memory compression - When you building redundancy by having two ESX servers, you need to have very similar configurations for processor / memory on your backup server, or you take a large performance hit. That's not always easy to budget for. If I can get away with my Exchange servers still running for a couple of hours on a server that is only 1/2 or 1/4 the memory of my main ESX, then I'll be happy. It's got to be quicker to compress memory than to swap it to disk - We'll see. I'll be testing that later on.

Service Console is now no longer an unsupported hack.

But really, the bit that gets me very interested the most is that vmware is now putting it's full weight behind ESXi - There won't be any more ESX! And to make this transition easier, you can now access vMotion from an ESXi 4.1 server. You still need licenses, but now it's nearly $10k cheaper to access this technology.

With my Ghetto-SAN comming online any week now, I'm very excited about this development.

BUT - There seems to be a but of a trade-off for the new things that 4.1 brings - It's just a tad slower than 4.0.0.

Once again, I do quick and dirty benchmarks to get a feel for things - Performance Test 6.1 isn't the best tool, but it's quick and makes for nice easy graphs to compare. After pouring through reams of iozone stats in Excel, I sometimes like quick, easy, and pretty.

My process was simple: Take a Performance Meter tests from a running 2008 R2 server before the upgrade to 4.1, and one after (with the new vmware tools loaded).

While 4.1 had better graphic performance (who cares!), it was around 2% slower for memory and CPU performance.

That's a small price to pay for new features, and I'm hoping it's just a result of new technology being focused on stability first, performance second.

Anyone else run benchmarks that can confirm or deny this?

RAID Alignment and VMware ESX 3i 3.5

Make sure to create your datastores from within the Virtual Infrastructure Client GUI, don't take what the ESX install CD creates (this means deleting any data stores created by the install CD).

It's because the Install CD creates the partitions on the standard Sector 63 starting position (31.5k into the drive. This isn't a proper multiple of any common RAID stripe size (2k, 4k, 8k, 16k, 32k, 64k, etc) so the partition ends up being out of alignment with the RAID stripe elements, and READ/WRITE performance will suffer.

This is all documented in a number of VMware documents.

Creating the partitions in VCI makes them start on sector 128 (64k into the drive), which lines up properly.

Questions on what RAID stripe size and what Windows Allocation Unit size to use will be in another post.

VMWare ESX 3.5 and 3i on ASUS DSEB-D16/SAS Motherboard with SATA drives

Okay, the board arrived yesterday. So far both VMWare's ESX 3.5 and 3.5i are able to boot, install, and stay stable.

I'm not using the LSI SAS controller as of yet - Just the Intel Matrix controller, with a plain old SATA drive. I haven't created any RAID, so both versions of 3.5 are able to spot the Intel controller and work just fine.

This means that people wanting to save a few bucks could use the Intel 6321 controller only board (no SAS).

Watch out - Make sure you use SATA ports 1-4. Ports 5 and 6 are not spotted by the install scripts.