LSI 2008 SAS Chipset (mps0) problems on recent FreeBSD versions

I've encountered a problem with never versions of FreeBSD and my LSI SAS 2008.

I have a number of LSI 2008 based HBAs in service in my FreeBSD storage devices. Most are re-flashed Dell PERC, some are IBM, some are branded LSI.

This particular card is flashed to MPT2BIOS 7.11.10.00 (2011.06.02) in IT mode.

Under FreeBSD 10.2 (and many earlier versions) I have no problems with it.

Attempting to install FreeBSD 11.1, I couldn't get FreeBSD to detect the HBA. I could see my two SES devices (SAS Expanders), but no mps0, no drives.

I did a bit of troubleshooting, and found this across the FreeBSD versions:

10.2  - No Problems
10.3  - Not Tested
10.4  - mps0 shows up, all drives show up, but I have screens and screens of SCSI SENSE errors, can't import zpool's, etc.
11.0 - mps0 dosen't show up, no drives show up, only ses0 ses1
11.1 - same as 11.0

I unfortunately wasted too much time troubleshooting my inactive server. Once I knew what I needed to do to get it online again, I couldn't spend more time to file a bug report and help track this down.

I did encounter this thread, sounds very similar to my problem:

https://forums.freebsd.org/threads/63800/

For now, I'm back to 10.2 on this server until I have the time to fire up a parts machine that I have in storage.

Let me know if you're running into something similar, or have any suggestions.

FreeBSD 11.x Guest Keyboard Stops Working in VMware

If you're firing up a 11.x instance of FreeBSD in VMware, you may find that the keyboard works until you shift focus to a different window. Shifting focus back to the FreeBSD window will not give you keyboard input again.

For me the fix was this:

In the .vmx file (or in the GUI under Edit Settings -> Options -> General, then Configuration Parameters...) add this line:

Let me know if you needed to do something different to make it work.

keyboard.typematicMinDelay = "2000000"

Let me know if you needed to do something different to make it work. 

Installing vCenter 6.0.0 gives: Installation of component VCSServiceManager failed with error code '1603'. Check the logs for more details.

I'm rolling out a new vCenter 6.0.0 install from scratch to test our environment on it, and quickly ran into a show stopper:

Installation of component VCSServiceManager failed with error code '1603'. Check the logs for more details.

I was installing on a fresh new Windows 2012R2 server with all the updates, but I hadn't installed the .NET 3.5 feature. 

The release notes from VMware mention this exact error, but in the context of an ipv4 address issue, so that wasn't much help, as all of my ipv4 addressing was correct. 

After a bit of poking around, I noticed that a 1603 error in a MSI is often due to .NET being missing.

After installing .NET 3.5 and updates, the install worked properly.

There is no mention of .NET being required in the Release Notes, the Windows Requirements, etc. 

Hopefully that saves someone a bit of time. :-)

Now let's see what that new Web Client performs like. 

Quick iperf test on FreeBSD 10.1 running as a VM in ESXi 5.5

Take this with a grain of salt, as I really haven't done enough testing to state anything conclusive.

However, under ESXi 5.1u3, I was only able to get my iperf scores up to 2/3Gb/s, with the occasional 5Gb/s bust on my 10Gb/s network.

With ESXi 5.5u2 (patches as of today's date) I am able to get scores like this fairly consistently:

[  3]  0.0- 5.0 sec  5.70 GBytes  9.79 Gbits/sec

[  3]  5.0-10.0 sec  5.69 GBytes  9.77 Gbits/sec

[  3] 10.0-15.0 sec  5.69 GBytes  9.77 Gbits/sec

[  3] 15.0-20.0 sec  5.70 GBytes  9.80 Gbits/sec

[  3]  0.0-20.0 sec  22.8 GBytes  9.78 Gbits/sec

Not bad, not bad.

Even if the scores are not accurate, it's at least showing a dramatic improvement over my 5.1 configuration.

This is from a FreeBSD 10.1 VM under ESXi 5.5 to a bare-metal FreeBSD 10.1. VM-to-VM performance is a bit lower, but still quite good:

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0- 5.0 sec  4.89 GBytes  8.41 Gbits/sec

[  3]  5.0-10.0 sec  5.46 GBytes  9.37 Gbits/sec

[  3] 10.0-15.0 sec  4.32 GBytes  7.42 Gbits/sec

[  3] 15.0-20.0 sec  4.87 GBytes  8.37 Gbits/sec

[  3]  0.0-20.0 sec  19.5 GBytes  8.39 Gbits/sec

The FreeBSD VM's are still vmx-08, and I'm using the built-in vmx driver in FreeBSD 10.1

This is also with the default configuration for sysctl values - I was planning on building a small script to check different tcp settings, but with speed like this, there are more pressing things to worry about.

ESXI 5.5u2 Hangs during upgrade from 5.1

Quick post:

 If you're upgrading from ESXi 5.1 to 5.5 Update 2, and you don't have all of the network ports hooked up on your host, you may hang the upgrade process in an infinite loop.

 After upgrading two other hosts from 5.1 to 5.5U2 within ~30 minutes, I was concerned that the third server was still at it after 2 hours.

 Watching the console carefully from KVM showed it complain for about 5 seconds about a missing MAC address.  Looking at the server's physical connections, it did indeed have one network port not connected.  Most of my servers are Dell Blades, so all network ports are always hooked up, however, I don't completely trust putting all of my computing in one box, so I still retain a fairly beefy external server in case my blade enclosure ever bites it.

 This particular server had been running for years with that port unplugged, but it only now posed a problem.

 Connecting the port solved the issue.

 Now if the vCenter Web Client didn't suck so badly, I'd be having a good night. :-)

Alternate Method for Converting VMware vCenter Server from Physical to Virtual (VM)

At some point you are going to figure out that the advantages of having your vCenter Server as a VM outweigh the risks, and you're going to want to convert your existing physical vCenter into a virtual one.

There are a few ways of going about this, including just migrating vCenter to a fresh VM.

However, if you're stuck using only the VMware Converter standalone, you may find it frustratingly hard to make the conversion.

If you throw caution to the wind and just convert the vCenter Server on the fly while it's running (using the sync command to try an clean up after), you may get lucky, but more than likely you will have database corruption and other fun.

If you follow VMware's procedures for converting the machine and shut down all the services, the SQL databases, etc. then the VMware Converter can't connect to your vCenter to add the machine to the cluster.

One way around that would be converting to a seperate ESXi host that isn't in your cluster, and later join the host to the cluster, bringing vCenter with it.

However, that's a bit difficult when you are remote to a network (like I was) so there is no oppourtunity to provision physical hardware.

The answer? Double-Conversion.

Shut down all of the necessary services on your vCenter server, and run the Standalone VMware Converter.

Once there, simply convert your vCenter server to a VMware Workstation/Fusion .vmdk image.

Once that is complete, reboot vCenter so all the services are functional again, and convert that .vmdk into a VM on your cluster with the Standalone VMware Converter. vCenter treats this like any other vanilla VM, nothing special is needed.

Yes, you have a period of time where the old vCenter is running and controlling the network, and you're going to be 'going back in time' so to speak to the time of the conversion, but it's no worse than restoring vCenter from a backup.

I connected my new vCenter VM to a dummy network, and did all the final cleanup on it (installing vmxnet3, pvscsi, etc).

Make sure vCenter is running. If you have 5.1 with the Single Sign On (SSO) service, it will probably keep you from running vCenter.

It's not an error in the conversion process. When enough hardware changes, it breaks the SSO service, which still starts normally. However, vCenter won't start. The full article from VMware about this condition is here:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2036170

The set of commands I had to use was:

set JAVA_HOME=C:\Program Files\VMware\Infrastructure\jre
cd C:\Program Files\VMware\Infrastructure\SSOServer\Utils
rsautil manage-secrets -a recover -m MASTER_PASSWORD

(Where MASTER_PASSWORD is your master password when you installed vCenter/SSO)


When you have everything on your vCenter VM running smoothly, shut down your physical vCenter server, and connect your vSphere Client to the raw ESXi host holding your vCenter VM. Once you start it, you'll be able to connect to vCenter as per normal.

Make sure you've read the various forums on how to protect yourself from a bad situation with a virtual vCenter server.

Oh, and if you can keep the old physical server around for a bit, you have a quick emergency backup in case you run into a problem cold-starting your datacenter. 

FreeBSD 10 ZFS Boot Hang with USB Boot Drive and Non-USB Keyboard

Here's an interesting one.

If you create a FreeBSD 10 ZFS system that boots from a USB drive, you may hang if you don't have a USB keyboard.

I've noticed this on a few different hardware systems, and it may even be present in FreeBSD 9.x - I haven't had much chance to look into it.

The hangs are always the same ; The zfs boot loader can't find a bootable disk.

When I manually type in zfs:rpool (in the case of the manual ZFS on root setup here) or zfs:zroot/ROOT/default (in the case of the new FreeBSD10 ZFS installer) I'll get one of two things;

1) kernel panic and crash. Note that this is often from pressing 'return' at the boot loader prompt, but typing 'zfs:rpool' won't cause an issue.

2) a small move forward to set the host UUID and hostid, then nothing - just hung.

It looks like when the USB keyboard is not present, then da0 (my USB Flash Stick) doesn't exist.

At some point I'll look into this further, as you shouldn't need any keyboard (USB or other) connected to boot FreeBSD, but for now just make sure you have a USB keyboard hooked up for boot.

Additionally, the order of the USB devices seems to be important. If I have a USB Keyboard on my USB0 Root, and the boot device on my USB1 Root, then I have this symptom as well. If I boot with the boot device on USB0, and keyboard on USB1, then I'm okay. 

Using MessagePack as Transport Beween a Python Tornado Server, and a JavaScript Browser Client

First of all: Good Luck. :-)

MessagePack is a great binary serialization format, allowing you to transport globs of variables across whatever transport you like. It's very similar to JSON, but leaning towards a tighter packet size , as shown in the graphic I took from the msgpack.org site above.

MessagePack also has a great variety of languages supported. If you code it, you can probably find a msgpack.X file on their site.

Now the problems:

1) MessagePack has evolved over the years, and updated the specifications for the protocol. This isn't well documented, nor is there a "Version 2.1" or similar tag on the specification documents. If you look around, you can see that some links refer to this as v5, but how do I know that we're not at v8 and that linking site is stale? This makes it less obvious about what version a msgpack library supports, which means it's more difficult to choose compatible msgpack libraries when you're using different languages. 

2) No CHANGELOG, because there are no revisions/versions. Good luck seeing what msgpack was 3 years ago without detective work.

3) Because the specification has changed, some of the MessagePack libraries are up-to-date, others are years behind. The pure Python u-msgpack.py is up-to-date as of Dec 2013, but the JavaScript msgpack.js (v1.05) is not.

Through trial and error, I was able to get the JavaScript version working with the u-msgpack.py version, but not the standard msgpack.py (which is a wrapper for the C library).

However, since u-msgpack.py is following the new specification, it can create msgpacks that the JavaScript version can't decode, particularly when Python is sending a long string.

I've patched msgpack.js, and have contacted the author (uupaa) to see if he has an updated version. If that dosen't pan out, I'll fork and update the code so others can have access to it.  I'm also interested in patching it to work with JavaScript Blobs, so I can send binary data to it.

There is one small error in the u-msgpack.py file:

The problem is in _pack_string. It's calculating the length of the string before it's encoded to UTF-8.

I think you must encode the string before you find the length of it, as some characters need to encode as double-byte or longer.

An example would be the French name Allagbe, or the French word precedent , where the 'e' is with Acute (http://www.fileformat.info/info/unicode/char/e9/index.htm)

  

Python's encoder makes this b'Allagb\xc3\xa9', which is one byte longer than than the original string.

u-msgpack encodes this as b'\xa7Allagb\xc3\xa9'  - notice how it's only 7 bytes long - it's trimming the \xa9 char from the msgpack.

When you feed this trimmed string through a .decode('utf-8') method, you'll crash with a python error : 'utf-8' codec can't decode byte 0xc3 in position 6: unexpected end of data

The solution is to encode to UTF-8 before calculating the string length, as detailed below:

def _pack_string(x):

x = x.encode('utf-8')

    if len(x) <= 31:

        return struct.pack("B", 0xa0 | len(x)) + x

    elif len(x) <= 2**8-1:

        return b"\xd9" + struct.pack("B", len(x)) + x

    elif len(x) <= 2**16-1:

        return b"\xda" + struct.pack(">H", len(x)) + x

    elif len(x) <= 2**32-1:

        return b"\xdb" + struct.pack(">I", len(x)) + x

    else:

        raise UnsupportedTypeException("huge string")

With this patch in place, I'm able to pass plenty of French and Italian names through u-msgpack.py to msgpack.js

Now.. if you're trying to make all of this work across Tornado's WebSockets, keep in mind that what you get from Tornado will probably not fit into u-msgpack.py's .unpackb() function properly, but I'm out of time to detail that fun today.

In the end: I'm happy with MessagePack, but I wish it was a little easier to get into.