FreeBSD 10 ZFS Boot Hang with USB Boot Drive and Non-USB Keyboard

Here's an interesting one.

If you create a FreeBSD 10 ZFS system that boots from a USB drive, you may hang if you don't have a USB keyboard.

I've noticed this on a few different hardware systems, and it may even be present in FreeBSD 9.x - I haven't had much chance to look into it.

The hangs are always the same ; The zfs boot loader can't find a bootable disk.

When I manually type in zfs:rpool (in the case of the manual ZFS on root setup here) or zfs:zroot/ROOT/default (in the case of the new FreeBSD10 ZFS installer) I'll get one of two things;

1) kernel panic and crash. Note that this is often from pressing 'return' at the boot loader prompt, but typing 'zfs:rpool' won't cause an issue.

2) a small move forward to set the host UUID and hostid, then nothing - just hung.

It looks like when the USB keyboard is not present, then da0 (my USB Flash Stick) doesn't exist.

At some point I'll look into this further, as you shouldn't need any keyboard (USB or other) connected to boot FreeBSD, but for now just make sure you have a USB keyboard hooked up for boot.

Additionally, the order of the USB devices seems to be important. If I have a USB Keyboard on my USB0 Root, and the boot device on my USB1 Root, then I have this symptom as well. If I boot with the boot device on USB0, and keyboard on USB1, then I'm okay. 

Boot from ZFS root with FreeBSD 9.2

(Update: The newly released FreeBSD 10 makes a ZFS setup in the menu system easy.. give it a try)

Booting from a ZFS root has always been a dream of mine, but I've never managed to make it work.

Since I'm deploying new FreeBSD 9.2 images for testing in my lab, I thought this would be a good time to give it a shot again.

I based this off the FreeBSD install page, with some simplification and a few small additions.

I took a freshly installed FreeBSD 9.2 system to act as my workstation for this, and plugged in a 8 gig USB Flash drive to be my ZFS root.

In my setup, da0 is the booted system, da1 is the blank USB Flash Drive.

I often do not use swap partitions on USB Flash, as it can prematurely wear the drive.  I make my USB Flash drives as disposable as possible, so keeping crash dumps isn't a concern.

NOTE: Solaris always called it's root ZFS pool 'rpool' and I recommend you do the same, as it's a existing standard. My use of 'zroot' here is because I was only fooling around... and seem to be too lazy to change the text below without testing for typos.

Here is what I have executed to make it work:


gpart create -s gpt da1
gpart add -b 34 -s 64k -t freebsd-boot -l boot0 da1
gpart add -t freebsd-zfs -l root0 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

#This may fail with an "operation not permitted" error message, since the kernel likes to protect critical parts of the #disk. If this happens for you, run:
#
# sysctl kern.geom.debugflags=0x10


kldload /boot/kernel/opensolaris.ko
kldload /boot/kernel/zfs.ko

zpool create zroot da1p2
# DO NOT FORGET TO MAKE IT P2! Specify da1, and zfs will kill your gpt tabes
zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot


#Make sure your FreeBSD 9.2 CD is in your drive

mount -t cd9660 /dev/cd0 /mnt

cd /mnt/usr/freebsd-dist

cat base.txz | tar --unlink -xpJf - -C /zroot
cat kernel.txz | tar --unlink -xpJf - -C /zroot
cat doc.txz | tar --unlink -xpJf - -C /zroot
cat lib32.txz | tar --unlink -xpJf - -C /zroot
cat ports.txz | tar --unlink -xpJf - -C /zroot
cat src.txz | tar --unlink -xpJf - -C /zroot

chroot /zroot


echo 'zfs_enable="YES"' > /etc/rc.conf
echo 'sshd_enable="YES"' >> /etc/rc.conf
echo 'hostname="freebsd92zfs"' >> /etc/rc.conf

echo 'zfs_load="YES"' > /boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /boot/loader.conf

tzsetup

cd /etc/mail
make aliases

exit
cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache
zpool export zroot




I then pull the USB Flash Drive, pop it into a spare machine, and presto, it's booting.


I'll fill in more about this as I play with ZFS on root for my USB Flash Drive systems.