FreeBSD 10 ZFS Boot Hang with USB Boot Drive and Non-USB Keyboard

Here's an interesting one.

If you create a FreeBSD 10 ZFS system that boots from a USB drive, you may hang if you don't have a USB keyboard.

I've noticed this on a few different hardware systems, and it may even be present in FreeBSD 9.x - I haven't had much chance to look into it.

The hangs are always the same ; The zfs boot loader can't find a bootable disk.

When I manually type in zfs:rpool (in the case of the manual ZFS on root setup here) or zfs:zroot/ROOT/default (in the case of the new FreeBSD10 ZFS installer) I'll get one of two things;

1) kernel panic and crash. Note that this is often from pressing 'return' at the boot loader prompt, but typing 'zfs:rpool' won't cause an issue.

2) a small move forward to set the host UUID and hostid, then nothing - just hung.

It looks like when the USB keyboard is not present, then da0 (my USB Flash Stick) doesn't exist.

At some point I'll look into this further, as you shouldn't need any keyboard (USB or other) connected to boot FreeBSD, but for now just make sure you have a USB keyboard hooked up for boot.

Additionally, the order of the USB devices seems to be important. If I have a USB Keyboard on my USB0 Root, and the boot device on my USB1 Root, then I have this symptom as well. If I boot with the boot device on USB0, and keyboard on USB1, then I'm okay. 

Boot from ZFS root with FreeBSD 9.2

(Update: The newly released FreeBSD 10 makes a ZFS setup in the menu system easy.. give it a try)

Booting from a ZFS root has always been a dream of mine, but I've never managed to make it work.

Since I'm deploying new FreeBSD 9.2 images for testing in my lab, I thought this would be a good time to give it a shot again.

I based this off the FreeBSD install page, with some simplification and a few small additions.

I took a freshly installed FreeBSD 9.2 system to act as my workstation for this, and plugged in a 8 gig USB Flash drive to be my ZFS root.

In my setup, da0 is the booted system, da1 is the blank USB Flash Drive.

I often do not use swap partitions on USB Flash, as it can prematurely wear the drive.  I make my USB Flash drives as disposable as possible, so keeping crash dumps isn't a concern.

NOTE: Solaris always called it's root ZFS pool 'rpool' and I recommend you do the same, as it's a existing standard. My use of 'zroot' here is because I was only fooling around... and seem to be too lazy to change the text below without testing for typos.

Here is what I have executed to make it work:


gpart create -s gpt da1
gpart add -b 34 -s 64k -t freebsd-boot -l boot0 da1
gpart add -t freebsd-zfs -l root0 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

#This may fail with an "operation not permitted" error message, since the kernel likes to protect critical parts of the #disk. If this happens for you, run:
#
# sysctl kern.geom.debugflags=0x10


kldload /boot/kernel/opensolaris.ko
kldload /boot/kernel/zfs.ko

zpool create zroot da1p2
# DO NOT FORGET TO MAKE IT P2! Specify da1, and zfs will kill your gpt tabes
zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot


#Make sure your FreeBSD 9.2 CD is in your drive

mount -t cd9660 /dev/cd0 /mnt

cd /mnt/usr/freebsd-dist

cat base.txz | tar --unlink -xpJf - -C /zroot
cat kernel.txz | tar --unlink -xpJf - -C /zroot
cat doc.txz | tar --unlink -xpJf - -C /zroot
cat lib32.txz | tar --unlink -xpJf - -C /zroot
cat ports.txz | tar --unlink -xpJf - -C /zroot
cat src.txz | tar --unlink -xpJf - -C /zroot

chroot /zroot


echo 'zfs_enable="YES"' > /etc/rc.conf
echo 'sshd_enable="YES"' >> /etc/rc.conf
echo 'hostname="freebsd92zfs"' >> /etc/rc.conf

echo 'zfs_load="YES"' > /boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /boot/loader.conf

tzsetup

cd /etc/mail
make aliases

exit
cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache
zpool export zroot




I then pull the USB Flash Drive, pop it into a spare machine, and presto, it's booting.


I'll fill in more about this as I play with ZFS on root for my USB Flash Drive systems.

ZFS raidz - Make Sure To Select the Correct Number of Disks

I've read that selecting the correct number of drives when building a raidz ZFS array is important if you are trying to maximize performance.

 http://mail.opensolaris.org/pipermail/zfs-discuss/2010-September/044701.html

Of course the best way to maximize performance is to use a straight stripe or a mirror instead of a raidz, but that isn't always an option when you have price, power, or physical space restrictions.

Since I'm perpetually curious, I set up tests to compare the write speed of a 3, 4, 5, 6, and 7 drive raidz array.

I used my saturate.c program to put the arrays under heavy write load, repeated 6 times, and took avg and std deviation.

My results are not clean enough to post, but they would suggest that the number of drives is important - Follow the rules below;

RAIDZ1 vdevs should have 3, 5, or 9 devices in each vdev
RAIDZ2 vdevs should have 4, 6, or 10 devices in each vdev
RAIDZ3 vdevs should have 5, 7, or 11 devices in each vdev

With some luck, I hope to have the time to revisit the tests and obtain results that are postable.

ZFS Testing - Saturate.c

Benchmarking ZFS is hard.

ZFS is such a complex FS with multiple levels of cache that getting a good solid reading on it's performance is always difficult.

I was recently tasked with building a FreeBSD ZFS SAN for a client that consisted of 90 1TB Seagate ST1000DM003 6GBps HD's in 2 SuperMicro SC847 Enclosures. The heads were 2 Dell PowerEdge T710's, 96 Gig DDR-3, Dual Xeon 5620 CPU's. I connected the head to the enclosure with a LSI2008 based SAS card.

As always, budget was tight, but performance HAD to be there. I needed to find out for sure what the write speed of this array was going to be under ZFS.

The throttling-effect of using a single SAS card was one of the items I needed to check .

Since the SuperMicro SC847's use a LSI2x36 backplane, I don't have a full SAS channel available to all drives at all times.  I wanted to start collecting data on what effect this would have on the performance of the arrays, and did we need to use 2 SAS cards to achieve a higher performance? This wasn't a "performance at all costs" scenario, rather a real-world situation with real-world budgets and needs.

This client had a lot of users, and a lot of big databases that are very active during the day. I needed to know that the array would quickly write down as much data as possible to satisfy their needs.

Knowing that the average-write of the array wouldn't saturate the single SAS card or the drives, but a heavy-write could, I needed to put the array into heavy-writes for a sustained period of time to compare various hardware and software configurations.

Thus began my search for a benchmark program that could really load down this gear.
To simplify things, I decided that I would only concern myself with a write saturation event - When the data is flowing to the drives so quickly that they never catch up - As close to 100% utilization as possible.

 I didn't have much luck with the standard FreeBSD benchmarks.. bonnie, bonnie++, iozone, etc. Either they were too hard to lock into a saturate write, or they didn't spawn enough writers to really load down the system.

In the end I ended up coding a quick and dirty program called Saturate.c that I'm passing on here. It's not very pretty, but it works.

It's hard-coded to spawn 4 files to write to for each forked child process. Tweak it as you need to for your system to really drive the files.

You can simply execute the file using time to check how long it takes like this;

time ./saturate

I created a simple script file that then executed the saturate program in different zpool configurations so I could confirm what was working best for us.  I've run through various raidz configurations, compression, etc. I'm still sifting through all of the data.

Warning: This will create a very large set of test files - I believe in it's current state it writes 40,000 MegaBytes (~40 Gigs).

Oh, with FreeBSD 9.0, my best average time was 1.2 Minutes.  Not bad for a Free Operating System.

I'll post some of my results of 2 months of various ZFS tests over the next while as I have time.

Code on PasteBin: http://pastebin.com/4SexdvLq

FreeBSD SCSI Sense Errors - Did You Check Your Cables?

I've been suffering from a lot of SCSI Sense errors under FreeBSD 9.0-STABLE with a recent ZFS based SAN build that have been driving me mad.

They are mostly present when ZFS does a wide scan of the available drives - via a 'zpool import' or 'zpool scan' or other operation. 

My drives are Seagate ST1000DM0003 SATA3 1TB drives, contained in a SuperMicro SC847 Chassis (LSI SAS2x36 SAS2 Expander so the whole thing can run 6gbps).

These errors also show after a few hours of running a custom script to saturate writes to my ZFS pool. SCSI Sense errors build until we end up in a flurry of errors on the console, and a hang of the SAS expanders, sometimes taking ZFS with it. 

Mar  7 13:50:06 Test kernel: (da33:mps1:0:27:0): CAM status: SCSI Status Error
Mar  7 13:50:06 Test kernel: (da33:mps1:0:27:0): SCSI status: Check Condition
Mar  7 13:50:06 Test kernel: (da33:mps1:0:27:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Mar  7 13:50:11 Test kernel: (da1:mps1:0:11:0): WRITE(10). CDB: 2a 0 0 6e 45 bf 0 1 0 0 length 131072 SMID 303 terminated ioc 804b s

Doing a bit of research, I find that I'm not the only one with SCSI Sense error problems, and naturally I started suspecting the recent mps driver committed from LSI, or some sort of firmware interaction issue with it (Check my recent post on the ixbge driver issue with LACP).

I've used the exact same hardware in the past for FreeBSD 9.0-BETA builds without this trouble, so I was sure it was software or expander related.

My HBA's are mostly Supermicro AOC-USAS2-L8i internal SAS2 cards for my ZFS builds. It's a good value card, and gives me what I want at a decent price.

The firmware on these cards was v7.21 and I knew Supermicro had newer BIOS and firmware, so I flashed them up to BIOS v 11 and FW v7.23. The documentation with the flash files shows a lot of advancement in the FW and BIOS that corrects errors, and I thought it would be a good try - Maybe the new mps driver made better use of the card, and needed newer firmware.

However, after the upgrade, my problems continued. System did seem to boot a little faster however.

I then applied the LSI direct FW (v7.23) and BIOS (v12) that you can find on LSI support site for the 9211-8i card (same thing as SuperMicro's). The flashing process was the same (although the SuperMicro download was a bit more automated with it's batch file).

Same problems.

I then started testing one item at a time - ending up with replacing my brand new Amphenol SAS2 external Mini-SAS cables for some older ones I had in use on a different server.

And it fixed it!

Damn! I then confirmed by swapping between the new and old cables that it was the two new cables that I had purchased that were causing all of this. Very frustrating, as it's a high quality, expensive Amphenol cable, not a no-name thin OEM thingie.

What was more puzzling was that a nearly identical hardware array was using the same cables, (from the same order as the bad cables), and it wasn't having issues.

However, that in-use array is using SATA2 1.5 Gbps drives, not 6Gbps SATA3.  When I checked the logs carefully, I saw we were running into a few SCSI Sense errors, but nothing was really wrong with the servers, and ZFS was generally happy.

I'm fairly confident that if I put that array under the strain of my saturation script, it would throw errors as well, possibly not as fast as the 6Gbps array does.

The Amphenol supplier has been most kind, and is offering to RMA the 4 cables for me. I'll get a new batch of 4, and will check them out carefully under heavy load before I deploy them.

It makes me wonder what other cables I have that will fail under heavy load. Data speeds are quickly increasing, and I'm sure it puts a strain on the cable manufacturers to keep up with the latest spec'ed cable, connectors, and techniques to ensure that the cables they make are indeed reliable at any speed, not to mention having to buy new cable testing gear. 

ZFS Dedup Performance - Real World Experience After One Year

I've been running ZFS with deduplication from Open Solaris or FreeBSD for about a year now in a production environment.  I manage ESX vm's, mostly with various flavours of Windows server.


In that year, my thoughts on where to use dedup have changed.

My big ZFS pool is 28.5 TB in a basic mirror, currently showing 11.1TB used, and running a dedup ratio of 1.25x (25% of my data is deduplicated), a savings of around 2.75 TB. This machine runs 24 Gig of RAM, and has a L2ARC comprised of 4 120 Gig SSD's, in additional to 2 ZIL RAM devices.

I have two smaller production units, holding 20.4 TB in a raidz with 12 Gig of RAM, and 16.4 TB in a raidz with 24 gig of RAM.  These secondary systems may or may not have hardware RAM ZIL's and L2ARC SSD caches as I continually move around components in my secondary equipment to fins a sweet-spot between $$ for hardware and performance.


Back to the thought at hand..


Dedup obviously has to have some sort of performance penalty. It simply takes more time to check if you're writing a unique block compared to writing without the check.

Checking the uniqueness of the block involves looking at every block's hash in the pool (even if the fs inside the pool isn't participating in dedup) - A task that increases write execution time as your pool increases in size.


In a perfect world, your system will have enough RAM that the Dedup Table (DDT) will reside entirely in RAM. This means that it only requires traversing the DDT in RAM to find the uniqueness of the block being written.

In the real world, this is nearly impossible. The way the ARC/L2ARC is designed, there isn't a preference for the DDT data - it's simply thought of as metadata, and that competes with user data in the cache system of ZFS. You may think that setting primarycache and/or secondarycache to metadata will fix the problem, but once you switch the primary cache to metadata, you also doom the secondary cache to only hold metadata - Nothing can get into the L2ARC unless it exists in the ARC.

In my experience with a busy pool, enough of your DDT is eroded away from user data writes that dedup performance starts to become a serious issue. These issues don't show up on a fresh system without a lot of data in pool. You need an older, loaded, very fragmented pool to really notice the drop in performance.


How bad? Here's a quick and dirty iometer check on a fs with dedup turned on, and one with it off. This is on my big pool, with plenty of L2ARC and RAM.

Dedup On:  197 IOPS, 0.81 MB/s, 647 ms AvgIO

Dedup Off: 4851 IOPS, 19.87 MB/s, 28 ms AvgIO.

See the difference? This is not a 2x or 4x slowdown of my writes, we're in the 20x slower category to use dedup than with it off.

So we know it's going to be hard to keep the DDT in RAM, so there will be a performance hit. What about the large savings to disk writes, (you don't write the block if it already exists) and thus the performance increase there?

It's simply not there in my systems, and probably not in yours.

Additionally the idea that these dedup blocks could reside in memory cache, allowing multiple systems to take advantage of the same blocks is a good one, but it looks like there is still enough erosion of the caches that this doesn't make a difference in my systems.


If I had a higher dedup ratio, then maybe this would start working out better for me - But even with 40+ VM's of mostly the same stuff, I'm only running a 1.25x dedup ratio on my big pool - I have enough dissimilar data to make dedup impractical on this system.

I'm now going to turn dedup off for nearly all of my production boxes. 

However, turning dedup off doesn't de-dedup your data - it stays deduplicated until fresh data overwites it.

This provides a 'poor-man's offline dedup' trick. Leave dedup off during the day when you need performance, then enable via a script, and copy all of the data in place to *.tmp and then rename to drop the *.tmp when done - You just deduplicated all your data, without the performance hit during the writes in the daytime when I assume your systems would be the busiest.

I'm going to leave dedup enabled on my storage and backup archives - places where I have a lot of duplication, and not a lot of need for speed.Everywhere else, I'm just going to run compression (I find compression=gzip to be a good blend of storage and speed on my systems). I"m averaging close to 2x on my compression alone.

For dedup to work well in a production environment, you're going to need an extreme amount of RAM and L2ARC compared to your drive storage size. I don't have any figures for you, but it's probably cheaper to provision more storage than it would be to try and dedup the data.

What ZFS needs is an offline dedup ability, priority settings for the DDT in ARC/L2ARC, and some general tuning of the ARC/L2ARC to allow for situations like "DDT and Metadata only in RAM, everything else allowed in L2ARC".

..at least for it to be viable in my environment, which is the problem - We all have different environments with different needs.

Followup: ZFS Data Gone

In Feburary I blogged about a nasty data loss event with ZFS

http://christopher-technicalmusings.blogspot.com/2011/02/zfs-data-gone-data-gone-love-is-gone.html

I've been quite busy since then and haven't followed up on the results. As a few people have been asking if I was able to get the data back, here is my answer:

Yes, I did get most of it back, thanks to a lot of effort from George Wilson (great guy, and I'm very indebted to him). However, any data that was in play at the time of the fault was irreversibly damaged and couldn't be restored. Any data that wasn't active at the time of the crash was perfectly fine, it just needed to be copied out of the pool into a new pool. George had to mount my pool for me, as it was beyond non-ZFS-programmer skills to mount. Unfortunately Solaris would dump after about 24 hours, requiring a second mounting by George. It was also slower than cold molasses to copy anything in it's faulted state. If I was getting 1 Meg/Sec, I was lucky. You can imagine that creates an issue when you're trying to evacuate a few TB of data through a slow pipe.

After it dumped again, I didn’t bother George for a third remounting (or I tried very half-heartedly, the guy was already into this for a lot of time, and we all have our day jobs), and abandoned the data that was still stranded on the faulted pool. I copied my most wanted data first, so what I abandoned was a personal collection of movies that I could always re-rip.


I was still experimenting with ZFS at the time, so I wasn't using snapshots for backup, just conventional image backups of the VM's that were running. Snapshots would have had a good chance of protecting my data from the fault that I ran into.


I was originally blaming my Areca 1880 card, as I was working with Areca tech support on a more stable driver for Solaris, and was on the 3rd revision of a driver with them. However, in the end it wasn't the Areca, as I was very familiar with it's tricks - The Areca would hang (about once every day or two), but it wouldn't take out the pool. After removing the Arcea and going with just LSI 2008 based controllers, I had one final fault about 3 weeks later that corrupted another pool (luckily it was just a backup pool). At that point, the swearing in the server room reached a peak, I booted back into FreeBSD, and haven't looked back.

Originally when I used the Areca controller with FreeBSD, I didn't have any problems with it during the 2 month trial period.

I've had only small FreeBSD issues since then, nothing else has changed on my hardware. So the only claim I can make is that in my environment, on my hardware, I've had better stability with FreeBSD than I did with Solaris.

Interesting Note: One of the speed slow-downs with FreeBSD compared to Solaris from my tests was the O_SYNC method that ESX uses to mount a NFS store. I edited the FreeBSD NFS source to always do a async write, regardless of the O_SYNC from the client, and that perked FreeBSD up a lot for speed, making it fairly close to what I was getting on Solaris.

I'm not sure why this makes such a difference, as I'm sure Solaris is also obeying the O_SYNC command. I do know that the NFS 3 code from FreeBSD is very old, and a bit cluttered - It could just be issues there, and the async hack gives it back speed it loses in other areas.

FreeBSD is now using a 4.1 NFS server by default as of the last month, and I'm just starting my stability tests with using a new FreeBSD-9 build to see if I can run newer code. I'll do speed tests again, and will probably make the same hack to the 4.1 NFS code to force async writes. I'll post to an update when I get this far.

I do like Solaris - After some initial discomfort about the different way things were being done, I do see the overall design and idea, and I now have a wish list of features I'd like see ported to FreeBSD. I think I'll have a Solaris based box setup again for testing. We'll see what time allows.

ZFS: Data gone, data gone, the love is gone...

Holy %*^&$!

My  entire 21 TB array is now "FAULTED" and basically dead to the world.

It's listing the worst type of corrupt data - pool metadata.

How did I do this? I was just rebooting the box, as always. I was working on integrating Solaris 11 Express with my active directory so CIFS would allow for seamless browsing from my Windows boxes. I was partially through it, when I decided to reboot the box and try the commands again - When I did, my pool was no longer intact.

I was reslivering a drive, but I've rebooted many times in that situation, it shouldn't matter. I also was starting to change the hostname from a generic "solaris" to something that made sense in my network, and it was being cranky about that, but nothing odd was going on.

The official response from the ZFS troubleshooting guides is to restore your data from backup after destroying and making a new pool. Sounds like Microsoft's advice for when your exchange store tanks.

Of course, you know all of my data isn't backed up. Let's not even go there. The most critical stuff is, things that I'd get my ass kicked for - but other data on my second SAN is months old, and then there's the gigs of junk that I've collected over the years that I don't bother backing up because I don't need it, but I want it. Most importantly, I did some pretty decent work this week, and it's all in that SAN ZFS pool, and I really don't want to recreate it.

I KNOW my data is all there. At last count, I had around 12-16 TB of data on these drives, that can't disappear that quickly.

The question is: Which path is faster? Recreating my data, or trying to recover this beast?

Well, you know I'm going to have to try and recover it.

My first day was spent in a light daze (shock maybe) as I researched more about the inner workings of ZFS. Luckily I had already spent a few weekends messing around with trying to port PJD's v28 ZFS patch on FreeBSD to a newer build of FreeBSD, so I had some idea of the code, the vdevs, uberblocks, etc.

My research is leading me to think that my most recent uberblocks were somehow corrupted, leading ZFS to think that I have a destroyed pool.

This is a big issue for ZFS - when those uberblocks don't make sense, the pool won't auto-recover, you've got to dig into things and really mess with the internals of ZFS to get anywhere.

Currently, I'm working with this command;

zpool import -d /mytempdev -fFX -o ro -o failmode=continue -R /mnt 13666181038508963033
..which takes 21 minutes to complete, so it gives you some idea of the slowness of recovering this type of data.

I'll post back if/when I get this recovered.

(May 20th 2011 Followup: http://christopher-technicalmusings.blogspot.com/2011/05/followup-zfs-data-gone.html )

SSD Fade. It's real, and why you may not want SSDs for your ZIL

SSD's fade - and they fade quicker than you'd expect.

When I say "fade" I mean that their performance drops quickly with write use.

The owner of ddrdrive is rather active on the forums, and was posting some very interesting numbers about how quickly a SSD can drop in performance. He sells a RAM-drive based hardware device that is actually very suitable for a ZIL, it's just very expensive.

Since my ZFS implementation makes use of 8 OCZ Vertex 2 SSD drives (4x60GB ZIL, 4x120GB L2ARC), I thought I should check into this.

First off, when you test SSD's you have to use one of the beta versions of  iometer, so you can enable "Full Random" on the test data stream. OCZ's SSD's (like all others I believe)  perform compression/dedup to limit their writes, resulting in inflated numbers if you use easily compressed data.

How inflated? Try write speeds of 4700 IOPS / 2 MBs for random data, then 7000 IOPS / 3.6 MBs for repeating bytes. Nearly double the performance for easy-to-compress data.

With iometer generating random data, I tested out one of my OCZ Vertex 2 60 Gig SSDs that was in use for 4 months as a ZIL device. That's not exactly an ideal use of SSD's, and you'll see why shortly.

My test was iometer 1.1 on a Windows 7 32 bit box, driving 100% random, 100% write to the SSD, with a queue depth of 128. The writes were not 4k aligned, which does make a difference in speed, but for purposes of illustrating SSD fade, it doesn't matter as a 512b or 4k write is affected equally.Currently ZFS uses 512b as a drive sector size, so you're getting this type of speed unless you've tweaked your ashift value to 12 with a tool like zfs guru.

Back to the tests:

Fresh from my active ZFS system, I was getting 4300 IOPS / 17.5 MBs

After a secure erase with the OCZ Toolbox 2.22, I was getting 6000 IOPS / 24.5 MBs - That's a 40% increase in IOPS just with a secure erase.  My SSD performance has dropped 40% in 4 months.

Curious to see how quickly the SSD will fade, I set iometer up to run the random write for 8 hours. When I came back and ran the test again, I was shocked to see speed down to 910 IOPS / 3.75 MBs. Wow, we're getting close to the territory of a 15k 2.5" SAS drive (around 677 IOPS / 2.77 MBs).

I then did a secure erase of the drive and tested again - Speed was back up, but not as good as what I originally had. I was now obtaining 4000 IOPS / 16 MBs - Worse than when I first started. 8 hours of hard writes used up my SSD a bit more.

Why did 8 hours of writing break down my SSD faster than 4 months of life as a ZIL? Well, I was using 4 of these 60 gig SSDs as one big ZIL device, and the ZIL doesn't consume much space, so the # of GB written over these 4 SSDs in the course of 4 months wasn't nearly as bad as my 8 hour test on one device.

Interestingly, I also have 4 120 Gig Vertex 2 drives for my L2ARC cache. When I pulled and tested these drives, they were far less degraded - The L2ARC is designed to be very conservative on writes, and is mostly for reads. A MLC SSD here works well.

Should you use SSDs for ZIL? You can, but understand how they will degrade.

Your best bet is a RAM based device like a ddrdrive, or similar. There are other slower, cheaper RAM drives out there you could use. These will not degrade with use, and the ZIL gets a lot of writes - Nearly everything written to your pool is written to the ZIL.

If you don't have one of these, your other option is to use lots of devices like I did to ballance the wear/fade. Pull them every 6 months for 15 min of maintenance, and you'll maintain decent speeds.

For now, my 4 60 gig SSDs are going back into ZIL duty until I can decide if a ddrdrive (or a bunch of cheaper ddr devices) is in my future.

ZFS and NFS performance, with ZIL disable and/or flushcache

I'm building my new FreeBSD 8.1 SAN, and one of the first tasks is trying to pull decent NFS performance from the box.

As you may be aware, NFS and ZFS don't mix well. It's because NFS is asking for a flush of the ZFS ZIL after each write, which is incredibly slow. It destroys all caching or other speed enhancements that ZFS can bring to the table.

The quick and dirty response is to disable the ZIL. While this won't lead to corruption, it's removing one of ZFS's protection mechanisms, and as I'm building a very large SAN, I really don't want to hamper ZFS's ability to save my ass on data corruption issues.

This is basically opening NFS in async mode, which is what iSCSI does. I want iSCSI performance with the stability of NFS sync.

I thought I'd play around with disabling the ZIL (makes me nervous), using  loander.conf commands, and adding a SSD as the ZIL.

NFS Tests on a ZFS RAID10

Notes:
All numbers are MB/sec, tests run twice (thus 2 numbers) then rebooted.
Tests are from PerformanceTest 6.1, which is easy for quick-n-dirty testing.
Tests are on a Windows Server 2003 32 Bit, going to FreeBSD 8.1, but who cares? They are only valid as a comparison on my own machine.  

Without ZIL
Fileserver: 64.25, 65.97 MB/sec
Workstation:  9.52, 12.99
Database: 56.31, 56.98



Decent speed for a RAID-10 ZFS on SATA drives without any tweaking. Beats my C: drive which is a SAS RAID-10 under ESXi, and is around the same speed as I was getting from iSCSI

With ZIL
Fileserver:  8.37, 6.53  
Workstation: 2.51
Database:

Basically: Much, Much slower.  I gave up after a few tests, as it was so tedious I didn’t want to continue. I did these tests before, so I know it’s slower across the board. At this speed iSCSI kicks NFS's ass. 

NOTE: I did try with ZIL and vfs.zfs.cache_flush_disable=1 , but the speed is basically just as bad. Besides, why wouln't you want your ZIL on a SSD?


With ZIL on Intel X25-M SSD (32 Gig)
Fileserver: 61.38, 62.08
Workstation: 8.05, 7.66
Database:  23.07, 23.05

Hmm,this is faster. I wouldn't be too unhappy with this type of performance. Database still suffers though.
 

With ZIL on Intel X25-M SSD (32 Gig), vfs.zfs.cache_flush_disable=1
Fileserver 54.69, 62.57
Workstation 12.43, 9.54
Database  54.2, 54.69

Hey - That's pretty good. Just a tiny tad under ZIL-less operation. 

Notes for SSD tests: The SSD as ZIL stayed around 50% busy. The ZIL does work.

So all we have to do is make ZFS lie and say it flushes the cache when it doesn't. Editer your /boot/loder.conf to include the vfs.zfs.cache_flush_disable=1 command, and you're off and running.

I believe this is an enhancement in newer ZFS pools anyway, so I'm really not too worried about it. If it's on the ZIL, why do we need to flush it to the drive? A crash at this point will still have the transactions recorded on the ZIL, so we're not losing anything.

BTW - It looks like ZFS v23 is comming to FreeBSD sooner than we expected - So this may all be moot, as it's included around v18 I seem to recall. 

Final thoughts: Never, never run a ZIL that isn't mirrored. It dies, lots of bad things happen... although I was able to shut down the system, turn off my ZIL in loader.conf, and boot without the SSD, so I think you could recover.. I'll be testing how nasty things are with a destroyed ZIL during transactions tomorrow.

OpenSolaris: So close...

I posted earlier of my love for ZFS Dedup, and how I was experimenting with OpenSolaris as a new base-unix instead of FreeBSD, because FreeBSD is so far behind in the ZFS ports.

Well, I'm switching back. It's been a few months, and we still don't see any new OpenSolaris code. Will Oracle release a new OpenSolaris? There's talk about it being dumped. There's too many questions, so I have decided to side-line and wait - so I'm back to FreeBSD until I see an OpenSolaris 2010 - But it's quite possible we'll have FreeBSD Dedup before we have another solid OpenSolaris release.

ZFS on FreeBSD 8.1

I've been using ZFS on my FreeBSD box for some time now, and it's been serving out gigs and gigs of data faithfully for some time.

I am looking into OpenSolaris, but with the slow (or never) release of 2010.x I'm still fully a FreeBSD shop at this stage for Unix.

I've just upgraded to 8.1 Release a few days ago, and I'm liking the new release - I took the opportunity to also upgrade my Samba to 3.4.8 at the same time, so the system seems perkier. I didn't have the time to do any real speed tests, so take that for what it's worth.

BUT - I've been getting some "kmem_map to small" panics when I tried to copy down a 160 gig file. I can't recall if I've ever moved such a large file before, so I don't think this is an 8.1 issue.

Some digging around and I found this information on the FreeBSD mailing list that I wanted to pass along.


From: Steve Polyack
Subject: Re: Freebsd 8.0 kmem map too small
Date: Wednesday, May 5, 2010 - 7:46 am


On a system here with 8GB of RAM and a very large zpool consisting of
multiple zdevs, little tuning was needed. The system is running
8-STABLE(amd64) as of a month or two ago. The only things I have set in
/boot/loader.conf are:
vm.kmem_size="12G"
vfs.zfs.arc_max="4G"

Setting kmem_size to 12G came from a combination of recommendations I
saw in various mailing list posts (you can probably dig them up).
Setting it to the physical memory size was the initial recommendation,
while others recommended 1.5x physical memory size to help prevent
fragmentation/wasted space in kmem.

Regardless, this has served us quite well for the ~6 months the system
has been in use. It has never crashed, even under intensive
multi-threaded benchmarking.

Other people recommend vm.kmem_size to be 1/2 of available RAM, and then vsf.zfs.arc_max to be 512M less than that, but that doesn't make sense to me - What's the point of RAM if you can't use it? This box only runs ZFS and Samba, let these things have some memory.

I've now set those same values as Steve on my FreeBSD system in question is running 8 gigs as well. I'm going to start my 160 gig copy again (which BTW, I get ~34 Meg/Sec according to Windows 7)

If you don't hear back from me, it's working well for me. :-)

ZFS Dedeuplication

Oh my god - Is anyone else as excited about this as I am?

Are you thinking of a SAN running a massive ZFS raidz2 in deduplication mode?

I've already built a 6tb raidz2 array last year, and I'm finding I need more space - next shot will be for 12 or 16 tb, and I'd love to use dedup to help me out.

I'm so curious about it, I'm considering switching to Solaris as my disk host. Normally I'd use FreeBSD, but their ZFS is too far behind (version 13, Sun's got Version 22) to support dedup in the next 2 years.

I'll be posting some test results soon.. just experimenting with some 4 gig .OST files under a ZFS dedup scenario to get a feel for compression.

Early tests with straight MP3's showed 23% compression. These are very, very promising times.

Here's a few links you can drool over until you get the time to do something, or I post my results:

http://blogs.sun.com/jsavit/entry/deduplication_now_in_zfs

http://blogs.sun.com/bonwick/entry/zfs_dedup

http://en.wikipedia.org/wiki/Data_deduplication

Benchmarks? FreeBSD ZFS vs GEOM vs Hardware Speed

I'm currently running through some disk benchmarks to determine the speed of various ZFS arrays in FreeBSD 7.1 compared to GEOM or Hardware arrays.

I'm selecting bonnie and dbench for my testing at the moment - any other benchmarks come to mind that would be useful?