Alternate Method for Converting VMware vCenter Server from Physical to Virtual (VM)

At some point you are going to figure out that the advantages of having your vCenter Server as a VM outweigh the risks, and you're going to want to convert your existing physical vCenter into a virtual one.

There are a few ways of going about this, including just migrating vCenter to a fresh VM.

However, if you're stuck using only the VMware Converter standalone, you may find it frustratingly hard to make the conversion.

If you throw caution to the wind and just convert the vCenter Server on the fly while it's running (using the sync command to try an clean up after), you may get lucky, but more than likely you will have database corruption and other fun.

If you follow VMware's procedures for converting the machine and shut down all the services, the SQL databases, etc. then the VMware Converter can't connect to your vCenter to add the machine to the cluster.

One way around that would be converting to a seperate ESXi host that isn't in your cluster, and later join the host to the cluster, bringing vCenter with it.

However, that's a bit difficult when you are remote to a network (like I was) so there is no oppourtunity to provision physical hardware.

The answer? Double-Conversion.

Shut down all of the necessary services on your vCenter server, and run the Standalone VMware Converter.

Once there, simply convert your vCenter server to a VMware Workstation/Fusion .vmdk image.

Once that is complete, reboot vCenter so all the services are functional again, and convert that .vmdk into a VM on your cluster with the Standalone VMware Converter. vCenter treats this like any other vanilla VM, nothing special is needed.

Yes, you have a period of time where the old vCenter is running and controlling the network, and you're going to be 'going back in time' so to speak to the time of the conversion, but it's no worse than restoring vCenter from a backup.

I connected my new vCenter VM to a dummy network, and did all the final cleanup on it (installing vmxnet3, pvscsi, etc).

Make sure vCenter is running. If you have 5.1 with the Single Sign On (SSO) service, it will probably keep you from running vCenter.

It's not an error in the conversion process. When enough hardware changes, it breaks the SSO service, which still starts normally. However, vCenter won't start. The full article from VMware about this condition is here:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2036170

The set of commands I had to use was:

set JAVA_HOME=C:\Program Files\VMware\Infrastructure\jre
cd C:\Program Files\VMware\Infrastructure\SSOServer\Utils
rsautil manage-secrets -a recover -m MASTER_PASSWORD

(Where MASTER_PASSWORD is your master password when you installed vCenter/SSO)


When you have everything on your vCenter VM running smoothly, shut down your physical vCenter server, and connect your vSphere Client to the raw ESXi host holding your vCenter VM. Once you start it, you'll be able to connect to vCenter as per normal.

Make sure you've read the various forums on how to protect yourself from a bad situation with a virtual vCenter server.

Oh, and if you can keep the old physical server around for a bit, you have a quick emergency backup in case you run into a problem cold-starting your datacenter. 

FreeBSD 10 ZFS Boot Hang with USB Boot Drive and Non-USB Keyboard

Here's an interesting one.

If you create a FreeBSD 10 ZFS system that boots from a USB drive, you may hang if you don't have a USB keyboard.

I've noticed this on a few different hardware systems, and it may even be present in FreeBSD 9.x - I haven't had much chance to look into it.

The hangs are always the same ; The zfs boot loader can't find a bootable disk.

When I manually type in zfs:rpool (in the case of the manual ZFS on root setup here) or zfs:zroot/ROOT/default (in the case of the new FreeBSD10 ZFS installer) I'll get one of two things;

1) kernel panic and crash. Note that this is often from pressing 'return' at the boot loader prompt, but typing 'zfs:rpool' won't cause an issue.

2) a small move forward to set the host UUID and hostid, then nothing - just hung.

It looks like when the USB keyboard is not present, then da0 (my USB Flash Stick) doesn't exist.

At some point I'll look into this further, as you shouldn't need any keyboard (USB or other) connected to boot FreeBSD, but for now just make sure you have a USB keyboard hooked up for boot.

Additionally, the order of the USB devices seems to be important. If I have a USB Keyboard on my USB0 Root, and the boot device on my USB1 Root, then I have this symptom as well. If I boot with the boot device on USB0, and keyboard on USB1, then I'm okay. 

Using MessagePack as Transport Beween a Python Tornado Server, and a JavaScript Browser Client

First of all: Good Luck. :-)

MessagePack is a great binary serialization format, allowing you to transport globs of variables across whatever transport you like. It's very similar to JSON, but leaning towards a tighter packet size , as shown in the graphic I took from the msgpack.org site above.

MessagePack also has a great variety of languages supported. If you code it, you can probably find a msgpack.X file on their site.

Now the problems:

1) MessagePack has evolved over the years, and updated the specifications for the protocol. This isn't well documented, nor is there a "Version 2.1" or similar tag on the specification documents. If you look around, you can see that some links refer to this as v5, but how do I know that we're not at v8 and that linking site is stale? This makes it less obvious about what version a msgpack library supports, which means it's more difficult to choose compatible msgpack libraries when you're using different languages. 

2) No CHANGELOG, because there are no revisions/versions. Good luck seeing what msgpack was 3 years ago without detective work.

3) Because the specification has changed, some of the MessagePack libraries are up-to-date, others are years behind. The pure Python u-msgpack.py is up-to-date as of Dec 2013, but the JavaScript msgpack.js (v1.05) is not.

Through trial and error, I was able to get the JavaScript version working with the u-msgpack.py version, but not the standard msgpack.py (which is a wrapper for the C library).

However, since u-msgpack.py is following the new specification, it can create msgpacks that the JavaScript version can't decode, particularly when Python is sending a long string.

I've patched msgpack.js, and have contacted the author (uupaa) to see if he has an updated version. If that dosen't pan out, I'll fork and update the code so others can have access to it.  I'm also interested in patching it to work with JavaScript Blobs, so I can send binary data to it.

There is one small error in the u-msgpack.py file:

The problem is in _pack_string. It's calculating the length of the string before it's encoded to UTF-8.

I think you must encode the string before you find the length of it, as some characters need to encode as double-byte or longer.

An example would be the French name Allagbe, or the French word precedent , where the 'e' is with Acute (http://www.fileformat.info/info/unicode/char/e9/index.htm)

  

Python's encoder makes this b'Allagb\xc3\xa9', which is one byte longer than than the original string.

u-msgpack encodes this as b'\xa7Allagb\xc3\xa9'  - notice how it's only 7 bytes long - it's trimming the \xa9 char from the msgpack.

When you feed this trimmed string through a .decode('utf-8') method, you'll crash with a python error : 'utf-8' codec can't decode byte 0xc3 in position 6: unexpected end of data

The solution is to encode to UTF-8 before calculating the string length, as detailed below:

def _pack_string(x):

x = x.encode('utf-8')

    if len(x) <= 31:

        return struct.pack("B", 0xa0 | len(x)) + x

    elif len(x) <= 2**8-1:

        return b"\xd9" + struct.pack("B", len(x)) + x

    elif len(x) <= 2**16-1:

        return b"\xda" + struct.pack(">H", len(x)) + x

    elif len(x) <= 2**32-1:

        return b"\xdb" + struct.pack(">I", len(x)) + x

    else:

        raise UnsupportedTypeException("huge string")

With this patch in place, I'm able to pass plenty of French and Italian names through u-msgpack.py to msgpack.js

Now.. if you're trying to make all of this work across Tornado's WebSockets, keep in mind that what you get from Tornado will probably not fit into u-msgpack.py's .unpackb() function properly, but I'm out of time to detail that fun today.

In the end: I'm happy with MessagePack, but I wish it was a little easier to get into.

Python - The One True Language

I find it interesting when people bad-mouth Python - To the point where I felt the need to write this little blog article.

First, I need to state the obvious - Python is the 'One True Language'.  :-)

I say that as someone who boot-strapped himself from Commodore BASIC (PET, not that fancy C64 thing), to MS-Basic, to Turbo C. Many an hour was logged in front of this little window..

Sexy IDE for 1987

C was clearly my first true love (well perhaps Angeline who sat in front of me in Gr. 7, but she never responded to me like an 16550 UART call in C did). Compared to the other languages at the time, it gave you incredible power, without the difficulty of ASM.

I've had the (mis?)fortune to have programmed in just about every popular language at one time of another in my 30+ year career, and I must say Python is the first language that usurped C's place in my heart.

My only regret is that I didn't try Python earlier.

A few years ago I had a choice to make between Perl and Python as my next scripting language for Unix tasks that were too complex for shells or PHP, and I made the choice to go for Perl, as evidenced by the 2006 receipt still in my O'Riley Programming Perl book on my shelf.

At the time it was a close call; Perl had more popularity, and looked to be continuing to grow. This was an important point when you want to invest time in a new language (anyone remember the D Language?).  I also think I liked the camel on the Perl book cover more than the python as well... who was into snakes? (I did not realize the Monty Python reference at the time). These things do matter when you're down to brass tacks.

So the books were purchased and well read, and a few years were spent in Perl. 

I enjoyed the C-like syntax, and relaxed feel. However, it wasn't everything that I wanted.. and I'm not sure exactly how I stumbled on Python again, nor what the light bulb moment was - I worked a lot back then, so memories are fuzzy. Something was lacking, something kept me driving forward into new languages.

Whatever made me stop and reconsider, I must say I'm very happy to be in Python now - and nearly 100% of what I write today is Python.

I need a bit of Javascript for DOM work, and a bit of C for some very heavy transactional stuff, but even some of my stress tests and message passing routines are now Python.

Why do I bring this up?

I feel a number of programmers looking at Perl/Puby/Python may fall into the same situation, even today when the Python community continues to gather strength.  No one is really out there bashing Perl or Ruby, but Python has it's haters.

Some people love to badmouth Python.. I don't really understand them - I think it must be a machismo thing. The fact that it's an interpreter, and that it doesn't thread well (you shouldn't use threads boys.. that' 1990's tech) seem to enrage some.

If you're considering investing time into Python or XYZ, then my suggestion is to take the time to try Python first. I'm not going to get into a pointed discussion on the pros and cons of the various languages - That's covered in detail across the web.

Instead I'll state: They are all good languages, but if you take the time to learn Python and really 'drink the kool-aid', I think you will be rewarded.

Python brings a joy back into programming that I haven't felt since I used to sit in front of that little Turbo C IDE back in 1988. I've hit a productivity level that is a career milestone thanks to Python.

Next: PyCharm - The Turbo C IDE for Python. :-)

PS - I predict an OS entirely in Python soon - Obviously with a special interpreter, but it will be here.