Wednesday, March 11, 2009

ZFS on FreeBSD 7.1

I've been flirting with ZFS on FreeBSD 7.1-RELEASE AMD64, and I must say I like it so far.

I'm hoping and dreaming of using it as my main filestore in a 'raidz' array of 1.5 TB drives to give me a massive temporary file store for my ever increasing Video and Music files.

Why ZFS?

I'm excited about ZFS mostly for being transactional, and for the raidz support - This is an enhanced version of RAID-5. I could have ran a GENOM RAID-5 array, but when you add in the other features of ZFS (and it has many - go check) it's very tempting to go with ZFS instead.

Unfortunately it's still experimental in FreeBSD, which may have some sticking with more tested technologies.

I should give a few tips if you're trying out ZFS

1) Don't bother with i386

You can run ZFS under i386 with some tweaks, but why? It's going to be limited, and I think you're asking for trouble. ZFS needs a lot of memory address space. Give it 64 Bits and you just have to put

zfs_enable="YES"

in /etc/rc.conf to make it start up.

2) Give it some power

I love using old hardware to run FreeBSD servers, but ZFS needs some oomph to run proprly. Sure people are running it in i386 with 384 Meg, but why? You can grab a decent Socket 775 board and 64 bit chip for $120, throw in 2 gigs of RAM for $60, and you're still under $200

3) Give it respect

This isn't UFS. It's still experimental, don't put your only copy of critical data on it. I'm going to put things that I really don't want to lose - MP3's and Videos, but not accounting data or similar. Will the world end if I lost it? No, but I'd be pretty pissed, so you know I'm doing a lot of testing on my hardware so I know how to recover from faults. Learn how to use it if you're going to want any dependability from it.

4) Don't try and boot from it

It won't. There are tricks to make the /usr and other partitions ZFS and the main boot on UFS, but why frig around and play with fire? We all have an old 80 or 40 gig HD around. Make that your UFS boot, and setup your other drives as dedicated ZFS.

5) Speed

In a very slap-dash test, my GEOM stripe was delivering data via samba at the same speed as my ZFS raidz pool. In some cases, even faster. Of course I'd have to do proper tests to make the playing field level, so this is just a quickie test to see if a ZFS raidz could hold up in speed to my GEOM stripe on my box for samba network use, and it does that just fine. It may suck for a MySQL db, but I haven't tested that yet.

6) Moving to different hardware

- If you build your pool on /dev/ad0, /dev/ad2, ad4 (etc) and then move to a new controller that is discovered as da0, da1, etc. You simply do this;

zpool tank export
zpool tank import

presto. My ad* based zpool is now showing up on my da* based controller.

7) Failure Redundancy

I setup my raidz, transferred a few gig of data to it, then pulled a drive while it was running. A few complaints on the console, but my data was still there. Speed seemed okay, and data was intact without a lot of rebuilding necessary. I later on rebooted and reconnect the drive, added it back into the pool, and life went on.

GEOM is annoying after a crash as it has to check the entire array before it lets you back in. On a 1.5 TB array, that takes some time. With ZFS, you can still fire up and access data.

You can also run a background check by typing

zfs scrub

No more off-line fscks! Schedule a low-priority scrub to run in the background, and you're ensuring your data is always clean and matching.

This simple crash test proved ZFS can do two main things for me ; 1) Notify me when something fails, and 2) accept the drive back after it's been fixed.

I now need to see if I pull a drive, format it, and put it back if it accepts it properly and rebuilds the array on it. One would hope, but I'd like to manually test it before putting all that media on it.


Conclusions:

Get FreeBSD 7.1-RELEASE AMD64 and grab 2 or more drives and play with it. Check the links below for setup info - it's really quite simple. I'm loving it, and I guess I will until it screws me over. :-)


Links for more info:

http://blog.thefrog.net/2008/04/zfs-on-freebsd.html

(Top link references the two bottoms ones which I used to setup my ZFS)

http://wiki.freebsd.org/ZFSQuickStartGuide

http://wiki.freebsd.org/ZFSTuningGuide

I also recommend

http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

http://kerneltrap.org/FreeBSD/BSDCan_2008_ZFS_Internals
http://opensolaris.org/os/project/sdosug/past_meetings/ZFS-general.20070919s.pdf
http://www.tech-recipes.com/rx/1405/zfs-how-to-fsck-or-check-filesystem-integrity-with-scrub/

3 comments:

  1. I was very excited when I heard that one of the people from Google's Summer of Code ported ZFS to FreeBSD. It seems decent in 7.1. I've been using ZFS a lot on Solaris 10 recently, and it's a great thing. I'm using it as a cheap NAS with iSCSI and samba.

    Because of my love of ZFS's snapshots, I've been putting a ZFS partition on FreeBSD 7.1 i386 virtual machines (with only 1GB RAM) to be used for data (databases, etc.) Performance is reasonable. I'm not doing raidz, so it's not so memory intensive.

    ReplyDelete
  2. One option for using older machines with ZFS is to use them as a drive chassis and let ZFS access them via iSCSI. Run ZFS itself on a 64 bit machine somewhere else on the network. In this case, performance will likely be limited by the fastest network card you can put into the machines.

    One advantage of this approach is you can gain additional redundancy depending on how you structure your ZFS pool. In the strongest protection case, you would create a 3 way mirror between drives hosted in 3 different machines. Any two of these machines could completely fail without data loss.

    The remaining single points of failure are the machine running ZFS itself (which if virtual could be quickly migrated to another host), and potentially the LAN.

    ReplyDelete
  3. Haudy: That's interesting. The only things against old machines for me these days is power usage. They gobble up more, and produce more heat, otherwise I'd keep them all going running redundant tasks like you mentioned there..

    With iSCSI or NFS, I'd think I'd run a sync util to keep the drives in sync, not a full ZFS mirror, as I think it would degrad performance too much.. a slow sync set to keep network util low, would eventually catch up enough if you wern't doing DB work.

    ReplyDelete