Showing posts with label ESXi. Show all posts
Showing posts with label ESXi. Show all posts

Sunday, March 1, 2015

Quick iperf test on FreeBSD 10.1 running as a VM in ESXi 5.5

Take this with a grain of salt, as I really haven't done enough testing to state anything conclusive.

However, under ESXi 5.1u3, I was only able to get my iperf scores up to 2/3Gb/s, with the occasional 5Gb/s bust on my 10Gb/s network.

With ESXi 5.5u2 (patches as of today's date) I am able to get scores like this fairly consistently:




[  3]  0.0- 5.0 sec  5.70 GBytes  9.79 Gbits/sec
[  3]  5.0-10.0 sec  5.69 GBytes  9.77 Gbits/sec
[  3] 10.0-15.0 sec  5.69 GBytes  9.77 Gbits/sec
[  3] 15.0-20.0 sec  5.70 GBytes  9.80 Gbits/sec
[  3]  0.0-20.0 sec  22.8 GBytes  9.78 Gbits/sec


Not bad, not bad.

Even if the scores are not accurate, it's at least showing a dramatic improvement over my 5.1 configuration.

This is from a FreeBSD 10.1 VM under ESXi 5.5 to a bare-metal FreeBSD 10.1. VM-to-VM performance is a bit lower, but still quite good:



[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec  4.89 GBytes  8.41 Gbits/sec
[  3]  5.0-10.0 sec  5.46 GBytes  9.37 Gbits/sec
[  3] 10.0-15.0 sec  4.32 GBytes  7.42 Gbits/sec
[  3] 15.0-20.0 sec  4.87 GBytes  8.37 Gbits/sec
[  3]  0.0-20.0 sec  19.5 GBytes  8.39 Gbits/sec



The FreeBSD VM's are still vmx-08, and I'm using the built-in vmx driver in FreeBSD 10.1

This is also with the default configuration for sysctl values - I was planning on building a small script to check different tcp settings, but with speed like this, there are more pressing things to worry about.

Thursday, April 17, 2014

Alternate Method for Converting VMware vCenter Server from Physical to Virtual (VM)

At some point you are going to figure out that the advantages of having your vCenter Server as a VM outweigh the risks, and you're going to want to convert your existing physical vCenter into a virtual one.

There are a few ways of going about this, including just migrating vCenter to a fresh VM.

However, if you're stuck using only the VMware Converter standalone, you may find it frustratingly hard to make the conversion.

If you throw caution to the wind and just convert the vCenter Server on the fly while it's running (using the sync command to try an clean up after), you may get lucky, but more than likely you will have database corruption and other fun.

If you follow VMware's procedures for converting the machine and shut down all the services, the SQL databases, etc. then the VMware Converter can't connect to your vCenter to add the machine to the cluster.

One way around that would be converting to a seperate ESXi host that isn't in your cluster, and later join the host to the cluster, bringing vCenter with it.

However, that's a bit difficult when you are remote to a network (like I was) so there is no oppourtunity to provision physical hardware.

The answer? Double-Conversion.

Shut down all of the necessary services on your vCenter server, and run the Standalone VMware Converter.

Once there, simply convert your vCenter server to a VMware Workstation/Fusion .vmdk image.

Once that is complete, reboot vCenter so all the services are functional again, and convert that .vmdk into a VM on your cluster with the Standalone VMware Converter. vCenter treats this like any other vanilla VM, nothing special is needed.

Yes, you have a period of time where the old vCenter is running and controlling the network, and you're going to be 'going back in time' so to speak to the time of the conversion, but it's no worse than restoring vCenter from a backup.

I connected my new vCenter VM to a dummy network, and did all the final cleanup on it (installing vmxnet3, pvscsi, etc).

Make sure vCenter is running. If you have 5.1 with the Single Sign On (SSO) service, it will probably keep you from running vCenter.

It's not an error in the conversion process. When enough hardware changes, it breaks the SSO service, which still starts normally. However, vCenter won't start. The full article from VMware about this condition is here:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2036170

The set of commands I had to use was:

set JAVA_HOME=C:\Program Files\VMware\Infrastructure\jre
cd C:\Program Files\VMware\Infrastructure\SSOServer\Utils
rsautil manage-secrets -a recover -m MASTER_PASSWORD


(Where MASTER_PASSWORD is your master password when you installed vCenter/SSO)


When you have everything on your vCenter VM running smoothly, shut down your physical vCenter server, and connect your vSphere Client to the raw ESXi host holding your vCenter VM. Once you start it, you'll be able to connect to vCenter as per normal.

Make sure you've read the various forums on how to protect yourself from a bad situation with a virtual vCenter server.

Oh, and if you can keep the old physical server around for a bit, you have a quick emergency backup in case you run into a problem cold-starting your datacenter. 


Tuesday, December 18, 2012

Passthrough PCIe devices from ESXi to FreeBSD - Leave MSI Enabled to Avoid Interrupt Storms With More than One vCPU

If you want to pass a PCIe device (such as a LSI SAS controller) through ESXi to a FreeBSD VM, you're probably aware that you may need to disable MSI/MSI-X interrupts to make it work.

However, I'm finding that while this setting will make the system boot, if you add a second vCPU, you will quickly stat to see Interrupt Storm messages on your console, and one of your CPU's will be nearly 100% used on interrupt tasks.

The easy fix for me? Only disable MSIX, not MSI.

Here are the relevant lines from my /boot/loader.conf:

hw.pci.enable_msix=0
hw.pci.enable_msi=1

All of my interrupt storm issues are now gone. 

Wednesday, December 12, 2012

VMWare Enhanced vMotion Compatibility (EVC) Not Needed between AMD Bulldozer 6234 and AMD Piledriver 6348 on VMware ESXi 5.1

I recently upgraded one of our datacenter servers from an AMD Bulldozer 6234 to a AMD Piledriver 6348.

I did find the new chip was faster for memory/CPU operations in our VM's, but I'll post about that in a different article.

What surprised me was that I didn't need to turn on EVC to allow VM's to migrate between hosts running the different CPU's.

Either VMware 5.1 doesn't recognize the new instruction set, or they don't matter in terms of migration.

Either way, I can migrate to the newer chips without worry, and since the 63xx are socket compatible with the 62xx and 61xx series, I have a fairly easy upgrade path for our servers if I wish.

Thursday, November 22, 2012

ESXi 5.0 running FreeBSD 9.0 / 9.1 and VMware Tools

Note - I have another article for ESXi 5.1. If you are running 5.1, these instructions won't work for you.

Installing the proper VMware Tools for FreeBSD 9.0 / 9.1 can be a pain.

The hard work to figure this out has already been done by others (http://ogris.de/vmware/), so here is a quick script that will install the official VMware Tools using Dru's patch.

Make sure to start a vmtools install from your vsphere console so ESXi will make the CD available to you.

If you run into problems with a missing library when you try and execute /usr/local/bin/vmare-toolbox-cmd, try looking here (http://lists.freebsd.org/pipermail/freebsd-questions/2010-June/217718.html) it solved the problem for me. 

(Cut-n-Paste into a terminal window with root access)

echo "Make sure you have started the vmware tools install"
mount -t cd9660 /dev/cd0 /media
cp /media/vmware-freebsd-tools.tar.gz ~
cd ~
tar xvf vmware-freebsd-tools.tar.gz
cd vmware-tools-distrib
cd lib/modules/source
tar xvf vmblock.tar
tar xvf vmmemctl.tar
tar xvf vmxnet.tar
tar xvf vmxnet3.tar

fetch http://ogris.de/vmware/vmxnet.diff
fetch http://ogris.de/vmware/vmxnet3.diff

echo #
echo # If it's thinking you have a previously applied patch (-R) say yes.
echo #

patch -p1 < vmxnet.diff
patch -p0 < vmxnet3.diff


cd vmblock-only
make && make install
cd ..

cd vmmemctl-only
make && make install
cd ..

cd vmxnet-only
make && make install
cd ..

cd vmxnet3-only
make && make install
cd ..

cd ~/vmware-tools-distrib


./vmware-install.pl

echo # DONE. Remember that the vmxnet3 driver is called vmxnet3f0

Wednesday, June 6, 2012

Running Windows 2012 Server RC on ESXi 5.0 U1

If you're wanting to play with the upcomming Windows 2012 RC trial under ESXi 5.0 U1, you may find that it hangs if you're running your Virtual Machine as "Windows 8 (64 Bit)".

You need to set your Virtual Machine to "Windows 2008 R2 (64 bit) not "Windows 8 (64 Bit)". Once you make this change, it boots, and looks to run smoothly.

Vmware tools looks to install correctly, I can power up and down, and I can use the VMXNET3 network adapter in Windows. I also have the PVSCSI adapter enabled, and it looks to be running well.

I'm curious to test out it's built in iSCSI Target / clone abilities for running diskless (iSCSI Boot) Windows 7 Pro workstations.

Oh, and if you want to be able to use 2012 / Windows 8, check this site for where to find simpe things like shutdown and control panel.  I'm not sure it's wise to change such a staple of Windows useage, but it does make for a cleaner interface.

So far, I like the new interface.. we'll see how useful it is after extended use.

Tuesday, March 13, 2012

Blank NFS Datastore under ESXi - Check your MTU


If you've used ESX/ESXi over NFS long enough, you have probably encountered a situation where you have a connected NFS datastore, but when you browse the datastore, it's blank, or only a few files are showing.



There are many reasons for this situation, and there is plenty of vmWare KB's about troubleshooting such a problem.

Turning on NFS logging is a good idea.

This is the standard vmWare Troubleshooting document.

One thing that you may not be thinking of is a MTU mismatch.

I had this exact problem tonight, and it unfortunately took a long time to figure out because I was suspecting FreeBSD as being the culprit due to some struggles I've had with it as of late.

I came across this by issuing this command;

vmkping -s 8500 san0 -s 8500

This tells the ping command to send a much larger payload than usual. My 10Gbe network is setup with a MTU of 9000, something I verified many times over, so I really didn't think this was the issue. I only entered it out of completeness after realizing this was a serious problem that wasn't going to be solved quickly, and I needed to start a proper documentation trail of what I was doing if I hoped to solve the issue.

I won't bore you with the details of my troubleshooting, and get to the point.

Turns out my new Dell M8024-k blade 10Gbe switch requires you to set the MTU on the LAG channel group separately from the ports. That's not uncommon, but as far as I can tell, this isn't in the M8024-k GUI anywhere, it can only be done via the CLI.

I guess I've been spoiled by LAG's that set their MTU based on the members MTU, and when I didn't see a separate LAG MTU setting in the GUI anywhere, I assumed that everything was working fine - After all, pings were working, and that didn't work in the past when I had the wrong MTU applied on different adapters (although that could also be an oddity with the FreeBSD lagg driver).

By default the Dell M8024-k makes LAG groups with a MTU of 1500. 

What's odd is that some connectivity is maintained with this mismatch. You can browse and list some directories, and even see some information - I believe the point where it really breaks is when it needs to transfer more than the 1500 limit allows.

Further oddness: FreeBSD didn't have problems mounting and browsing the directories over NFS that ESXi couldn't browse or list properly. This could be to FreeBSD connecting via UDP instead of TCP, at this point I'm not sure.

I thought I'd pass this along in case anyone else is forgetting about each point in your data transmission chain that a MTU mismatch could be affecting.