As you may be aware, NFS and ZFS don't mix well. It's because NFS is asking for a flush of the ZFS ZIL after each write, which is incredibly slow. It destroys all caching or other speed enhancements that ZFS can bring to the table.
The quick and dirty response is to disable the ZIL. While this won't lead to corruption, it's removing one of ZFS's protection mechanisms, and as I'm building a very large SAN, I really don't want to hamper ZFS's ability to save my ass on data corruption issues.
This is basically opening NFS in async mode, which is what iSCSI does. I want iSCSI performance with the stability of NFS sync.
I thought I'd play around with disabling the ZIL (makes me nervous), using loander.conf commands, and adding a SSD as the ZIL.
Notes:
All numbers are MB/sec, tests run twice (thus 2 numbers) then rebooted.
Tests are from PerformanceTest 6.1, which is easy for quick-n-dirty testing.
Tests are on a Windows Server 2003 32 Bit, going to FreeBSD 8.1, but who cares? They are only valid as a comparison on my own machine.
Without ZIL
Fileserver: 64.25, 65.97 MB/sec
Workstation: 9.52, 12.99
Database: 56.31, 56.98
Decent speed for a RAID-10 ZFS on SATA drives without any tweaking. Beats my C: drive which is a SAS RAID-10 under ESXi, and is around the same speed as I was getting from iSCSI
With ZIL
Fileserver: 8.37, 6.53
Workstation: 2.51
Database:
Basically: Much, Much slower. I gave up after a few tests, as it was so tedious I didn’t want to continue. I did these tests before, so I know it’s slower across the board. At this speed iSCSI kicks NFS's ass.
NOTE: I did try with ZIL and vfs.zfs.cache_flush_disable=1 , but the speed is basically just as bad. Besides, why wouln't you want your ZIL on a SSD?
With ZIL on Intel X25-M SSD (32 Gig)
Fileserver: 61.38, 62.08
Workstation: 8.05, 7.66
Database: 23.07, 23.05
Hmm,this is faster. I wouldn't be too unhappy with this type of performance. Database still suffers though.
With ZIL on Intel X25-M SSD (32 Gig), vfs.zfs.cache_flush_disable=1
Fileserver 54.69, 62.57
Workstation 12.43, 9.54
Database 54.2, 54.69
Hey - That's pretty good. Just a tiny tad under ZIL-less operation.
Notes for SSD tests: The SSD as ZIL stayed around 50% busy. The ZIL does work.
So all we have to do is make ZFS lie and say it flushes the cache when it doesn't. Editer your /boot/loder.conf to include the vfs.zfs.cache_flush_disable=1 command, and you're off and running.
I believe this is an enhancement in newer ZFS pools anyway, so I'm really not too worried about it. If it's on the ZIL, why do we need to flush it to the drive? A crash at this point will still have the transactions recorded on the ZIL, so we're not losing anything.
BTW - It looks like ZFS v23 is comming to FreeBSD sooner than we expected - So this may all be moot, as it's included around v18 I seem to recall.
Final thoughts: Never, never run a ZIL that isn't mirrored. It dies, lots of bad things happen... although I was able to shut down the system, turn off my ZIL in loader.conf, and boot without the SSD, so I think you could recover.. I'll be testing how nasty things are with a destroyed ZIL during transactions tomorrow.
So, did you test what happens when the ZIL is killed in a bad way?
ReplyDeleteHi ioquatix;
ReplyDeleteIf the ZIL is corrupted when you need it, you can lose the pool. That's always going to be a risk, which is why you want to make sure your ZIL is write safe.
Have you corrupted a pool due to issues with a ZIL?
I have tested pulling a ZIL from a running pool - No issues from FreeBSD 8.1 upwards. Older versions could kernel panic on a ZIL removal, but that's ancient history now.
The ZIL is only important if you're recovering from a system crash, and you have data corruption in the pool to fix. If it's not needed, it can disappear at any time without affecting your pool.