Tuesday, August 24, 2010

OpenSolaris: So close...

I posted earlier of my love for ZFS Dedup, and how I was experimenting with OpenSolaris as a new base-unix instead of FreeBSD, because FreeBSD is so far behind in the ZFS ports.

Well, I'm switching back. It's been a few months, and we still don't see any new OpenSolaris code. Will Oracle release a new OpenSolaris? There's talk about it being dumped. There's too many questions, so I have decided to side-line and wait - so I'm back to FreeBSD until I see an OpenSolaris 2010 - But it's quite possible we'll have FreeBSD Dedup before we have another solid OpenSolaris release.

Quick and Dirty: FreeBSD 8.1 AHCI vs ATA (CAM)

While building and testing what will be a 24 drive SAN running FreeBSD 8.1, ZFS, and NFS/iSCSI I discovered a problem with hot-swapping the SATA drives.

It seems FreeBSD 8.1 AMD64 was running the SATA drives in ATA mode, so they really didn't hot-swap. If I pulled a drive and reinserted, I couldn't make it understand that the drive was back, even if I fooled with atacontrol's attach and detach. I had to reboot, which is no way to run a SAN, no mater how ghetto it may be.

A bit of poking showed that while my BIOS was set to AHCI (Intel ICH7R chipset), FreeBSD was still running in ATA mode. (type 'atacontrol list' and if you see drives, you are too). camcontrol is the program you use once in CAM mode.

The answer was to put ahci_enable="YES" in /boot/loader.conf.

This changes your drives to ada's from ad's which causes a boot problem, but that's easily fixed.

I did a quick a dirty test with 'raidtest' to show the speed difference. Here's my raidtest creation command:

raidtest genfile -s 128 -S 512 -n 50000

Here's the results:

With CAM: (ahci_enable="YES" in /boot/loader.conf)

iscsi# raidtest test -d /dev/zvol/tank/vol
Read 50000 requests from raidtest.data.
Number of READ requests: 24831.
Number of WRITE requests: 25169.
Number of bytes to transmit: 3286713344.
Number of processes: 1.
Bytes per second: 37162958
Requests per second: 565


Without CAM

iscsi# raidtest test -d /dev/zvol/tank/vol
Read 50000 requests from raidtest.data.
Number of READ requests: 24831.
Number of WRITE requests: 25169.
Number of bytes to transmit: 3286713344.
Number of processes: 1.
Bytes per second: 6069384
Requests per second: 92



Big difference eh? It's night and day. It then lead to some ZFS issues when I turned CAM on, because ZFS likes to hit write/read stalls when you move a lot of data, but I cleared that up with some further ZFS tweaks that I will detail another day.

If you're not running with ahci_enable="yes" in your loader.conf, you may want to look at enabling it.

I"ll be doing more tests over the new few days on a few different FreeBSD machines, and we'll see what the general results are.

Monday, August 2, 2010

ZFS on FreeBSD 8.1

I've been using ZFS on my FreeBSD box for some time now, and it's been serving out gigs and gigs of data faithfully for some time.

I am looking into OpenSolaris, but with the slow (or never) release of 2010.x I'm still fully a FreeBSD shop at this stage for Unix.

I've just upgraded to 8.1 Release a few days ago, and I'm liking the new release - I took the opportunity to also upgrade my Samba to 3.4.8 at the same time, so the system seems perkier. I didn't have the time to do any real speed tests, so take that for what it's worth.

BUT - I've been getting some "kmem_map to small" panics when I tried to copy down a 160 gig file. I can't recall if I've ever moved such a large file before, so I don't think this is an 8.1 issue.

Some digging around and I found this information on the FreeBSD mailing list that I wanted to pass along.


From: Steve Polyack
Subject: Re: Freebsd 8.0 kmem map too small
Date: Wednesday, May 5, 2010 - 7:46 am


On a system here with 8GB of RAM and a very large zpool consisting of
multiple zdevs, little tuning was needed. The system is running
8-STABLE(amd64) as of a month or two ago. The only things I have set in
/boot/loader.conf are:
vm.kmem_size="12G"
vfs.zfs.arc_max="4G"

Setting kmem_size to 12G came from a combination of recommendations I
saw in various mailing list posts (you can probably dig them up).
Setting it to the physical memory size was the initial recommendation,
while others recommended 1.5x physical memory size to help prevent
fragmentation/wasted space in kmem.

Regardless, this has served us quite well for the ~6 months the system
has been in use. It has never crashed, even under intensive
multi-threaded benchmarking.


Other people recommend vm.kmem_size to be 1/2 of available RAM, and then vsf.zfs.arc_max to be 512M less than that, but that doesn't make sense to me - What's the point of RAM if you can't use it? This box only runs ZFS and Samba, let these things have some memory.

I've now set those same values as Steve on my FreeBSD system in question is running 8 gigs as well. I'm going to start my 160 gig copy again (which BTW, I get ~34 Meg/Sec according to Windows 7)

If you don't hear back from me, it's working well for me. :-)