Benchmarks? FreeBSD ZFS vs GEOM vs Hardware Speed

I'm currently running through some disk benchmarks to determine the speed of various ZFS arrays in FreeBSD 7.1 compared to GEOM or Hardware arrays.

I'm selecting bonnie and dbench for my testing at the moment - any other benchmarks come to mind that would be useful?

ZFS on FreeBSD 7.1

I've been flirting with ZFS on FreeBSD 7.1-RELEASE AMD64, and I must say I like it so far.

I'm hoping and dreaming of using it as my main filestore in a 'raidz' array of 1.5 TB drives to give me a massive temporary file store for my ever increasing Video and Music files.

Why ZFS?

I'm excited about ZFS mostly for being transactional, and for the raidz support - This is an enhanced version of RAID-5. I could have ran a GENOM RAID-5 array, but when you add in the other features of ZFS (and it has many - go check) it's very tempting to go with ZFS instead.

Unfortunately it's still experimental in FreeBSD, which may have some sticking with more tested technologies.

I should give a few tips if you're trying out ZFS

1) Don't bother with i386

You can run ZFS under i386 with some tweaks, but why? It's going to be limited, and I think you're asking for trouble. ZFS needs a lot of memory address space. Give it 64 Bits and you just have to put

zfs_enable="YES"

in /etc/rc.conf to make it start up.

2) Give it some power

I love using old hardware to run FreeBSD servers, but ZFS needs some oomph to run proprly. Sure people are running it in i386 with 384 Meg, but why? You can grab a decent Socket 775 board and 64 bit chip for $120, throw in 2 gigs of RAM for $60, and you're still under $200

3) Give it respect

This isn't UFS. It's still experimental, don't put your only copy of critical data on it. I'm going to put things that I really don't want to lose - MP3's and Videos, but not accounting data or similar. Will the world end if I lost it? No, but I'd be pretty pissed, so you know I'm doing a lot of testing on my hardware so I know how to recover from faults. Learn how to use it if you're going to want any dependability from it.

4) Don't try and boot from it

It won't. There are tricks to make the /usr and other partitions ZFS and the main boot on UFS, but why frig around and play with fire? We all have an old 80 or 40 gig HD around. Make that your UFS boot, and setup your other drives as dedicated ZFS.

5) Speed

In a very slap-dash test, my GEOM stripe was delivering data via samba at the same speed as my ZFS raidz pool. In some cases, even faster. Of course I'd have to do proper tests to make the playing field level, so this is just a quickie test to see if a ZFS raidz could hold up in speed to my GEOM stripe on my box for samba network use, and it does that just fine. It may suck for a MySQL db, but I haven't tested that yet.

6) Moving to different hardware

- If you build your pool on /dev/ad0, /dev/ad2, ad4 (etc) and then move to a new controller that is discovered as da0, da1, etc. You simply do this;

zpool tank export
zpool tank import

presto. My ad* based zpool is now showing up on my da* based controller.

7) Failure Redundancy

I setup my raidz, transferred a few gig of data to it, then pulled a drive while it was running. A few complaints on the console, but my data was still there. Speed seemed okay, and data was intact without a lot of rebuilding necessary. I later on rebooted and reconnect the drive, added it back into the pool, and life went on.

GEOM is annoying after a crash as it has to check the entire array before it lets you back in. On a 1.5 TB array, that takes some time. With ZFS, you can still fire up and access data.

You can also run a background check by typing

zfs scrub

No more off-line fscks! Schedule a low-priority scrub to run in the background, and you're ensuring your data is always clean and matching.

This simple crash test proved ZFS can do two main things for me ; 1) Notify me when something fails, and 2) accept the drive back after it's been fixed.

I now need to see if I pull a drive, format it, and put it back if it accepts it properly and rebuilds the array on it. One would hope, but I'd like to manually test it before putting all that media on it.


Conclusions:

Get FreeBSD 7.1-RELEASE AMD64 and grab 2 or more drives and play with it. Check the links below for setup info - it's really quite simple. I'm loving it, and I guess I will until it screws me over. :-)


Links for more info:

http://blog.thefrog.net/2008/04/zfs-on-freebsd.html

(Top link references the two bottoms ones which I used to setup my ZFS)

http://wiki.freebsd.org/ZFSQuickStartGuide
http://wiki.freebsd.org/ZFSTuningGuide

I also recommend

http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
http://kerneltrap.org/FreeBSD/BSDCan_2008_ZFS_Internals
http://opensolaris.org/os/project/sdosug/past_meetings/ZFS-general.20070919s.pdf
http://www.tech-recipes.com/rx/1405/zfs-how-to-fsck-or-check-filesystem-integrity-with-scrub/

MSP Offerings: Zenith Infotech in more detail

I signed up for Zenith's trial over a month ago, and only had a few hours to play with it before running out of research time. My life is pretty tightly scheduled so I had to shelve my MSP demo for almost a month.

I'm back to it now, but my trial expired. No big deal. Zenith is dirt cheap to use. Check these prices:

http://www.saaztraining.com/zenith deliverables 2007.pdf

So I just signed up and am now paying a few bucks a month to continue to try out Zenith. I'm also now seeing Zenith from the side of a "real user", not a "prospective user" so I should be seeing the true system, not the salesman-enhanced side of things.

I like the fact that I'm not pressured to buy anything big upfront. It's all SAAS, so there is next to no requirement on my end, other than using IE (Firefox doesn't work) for their management console.

True, with SAAS you're always paying for it. But remember; software is always evolving. I have a few products I can keep running for a few years without upgrading, but not too many. Usually you're always in need of the new features or the bug fixes, or the new platform support that the new version brings.

I'm starting to become very aware of the internal costs we pay for maintaining our our software - The time we spend frigging with our servers, setting them up, maintaining them, etc. That time is far more productive being spent on customer's needs, which can be billed. Part of me is a bit apprehensive about the lack of control, but most of me is happy that I don't have to deal with putting it all back together if it crashes.

So what do I like so far about Zenith?

Well, I like the reports - Very similar to what I generate for my clients, except it's all there. I can output nearly the same thing with 5 clicks, and I'm done. This task is simple enough that it can now be given to the admin staff, not needing a technician to build the reports.

The dashboard/overview of what is going on with my servers is rather simple, but you know, it tells me the main points I need. I'm sure this is customizable, I'm still in the process of figuring that out.




I will keep learning this package and then write a better report on it soon. Once that's done, I'm off to check out Kaseya, Bigfix, and probably Level in the same detail. It's going to be a lot of time, but this is a big decision. These are complex packages, and they need proper attention to compare.

Switching Firewalls from ipf to pf on FreeBSD

It's time to deploy a new FreeBSD firewall, and I thought I'd check out pf instead of my standard ipf package.

pf is written by the OpenBSD team, and was designed to replace ipf because of the licensing issues with that code not being a proper BSD license. There's also some political issues, but I'll leave those along.

The Background:

Since the early days my office firewalls have been FreeBSD platforms. I've always loved the BSD platform, back from the days where I built an ISP around the BSDi package in the early 90's.

For as long as I can remember, I've been using ipf (ipfilter) by Darren Reed. It became so standard that FreeBSD started including it in their releases which makes life much easier.

Unix based firewalls always had my vote - They are infinitely configurable and extensible. They are not overly hard to setup, and they don't need a lot of hardware to run. In the day my options were a very expensive Cisco-class router, or Unix.

But as NAT routers kept dropping in price and improving in quality, my Unix based router/firewalls started slowly disappearing from sites. Most small offices don't need much - basic NAT with one port for remote access from us, and they are happy. A simple D-Link DI-604 goes for $50. That's a far cry cheaper than a decent computer and the charge for my time to setup a custom firewall.

So the Unix-Based firewalls only exist in some of my more complex, larger, or secure customer sites. The rest have Linksys or D-Link routers.

A few months ago my aged FreeBSD firewall finally died. Heck, it was a Pentium-233Mhz - I think I got my money's worth out of it. I knew it's days were numbered so I had a spare DI-604 ready to go, and popped it into place when needed.

I didn't really notice much of a difference in speed of the network (my main concern against the little routers) and it was handy to be able to use a web interface to configure it. I know you can do this with Unix, but it was one of the things I didn't bother updating as I upgraded the boxes over the years, relying on SSH instead. I don't pay a lot of attention to my firewall, and I didn't want to worry about security issues in the web interface screwing me up.

Life's been fine on the little D-Link until I started getting heavier into uTorrent. You put a few Windows 2003 machines on a network running uTorrent, and you can bring a little NAT router to it's knees. They just don't have enough memory and processor power.

The answer was to go back into some sort of Unix based firewall. Wanting something easier and FreeBSD based, I found pfSense. That didn't work out well for me, but it's another blog.

I made some compromises on how I use uTorrent, and my little DI-604 was still useful...

..until I had some heavy Spammer hacking going on. Talk about dragging your network into the ground. The DI-604 can't handle large amounts of traffic, even if it's rejecting it, and I was getting a lot of traffic. One old Windows 2000 IIS server (ticking time bomb really) was compromised, and the scum of the world was lining up at my router to use my bandwidth.

Back to Today - Trying pf over ipf:

So I have to build a new FreeBSD firewall. A quick download of 7.1-RELEASE i386 Disc1 (no need for 64 bit on a fw as far as I can see), and I'm on my way.

(btw msk support in 7.1 seems to be much better than 7.0 - no watchdog timeouts for me so far)

I decided to make this one a pf test platform as pf is also included by default with FreeBSD 7.1. I've heard mixed reviews of which is better, but the thing that intrigued me the most was that pf was a superset of ipf - my knowledge of rules for ipf wouldn't' be wasted, and I'd have more things to play with. ipf bugs me at times, as it doesn't seem to be evolving (like qmail). There is something to be said for a stable codebase, but I also don't think we've reached the pinacle of these packages.

Setup is simple;

in rc.conf;

pf_enable="YES"
pflog_enable="YES"

and your config file is /etc/pf.conf

pfctl is your control program for start, stop, and status.

If you want to be lazy, I'm quite sure a standard ipf.conf would work if you wanted to play with this.

So far I like pf. It's got a bit of a learning curve, but it has additional features that ipf lacks.

I like that the NAT portion is in the pf.conf - just looks cleaner to me to have one file for it. I like that I have more NAT rules. I have a "no nat" rule that blocks any outgoing SMTP traffic that doesn't go through one of our main mailservers. Just a different way of doing it, but nice.

pf also has the ability to do QoS routing via the ALTQ interface.

I'm currently checking out fwanalog so I can get a better look at pf's logs (also works for ipf) and I think I'm going to have to move into a Snort setup so I can detect these attacks and prevent them in the future.

This link has some simple info for monitoring pf firewalls;

http://prefetch.net/articles/monitoringpf.html

I'll post more as I find them.