Wednesday, June 9, 2010

ZFS Dedeuplication

Oh my god - Is anyone else as excited about this as I am?

Are you thinking of a SAN running a massive ZFS raidz2 in deduplication mode?

I've already built a 6tb raidz2 array last year, and I'm finding I need more space - next shot will be for 12 or 16 tb, and I'd love to use dedup to help me out.

I'm so curious about it, I'm considering switching to Solaris as my disk host. Normally I'd use FreeBSD, but their ZFS is too far behind (version 13, Sun's got Version 22) to support dedup in the next 2 years.

I'll be posting some test results soon.. just experimenting with some 4 gig .OST files under a ZFS dedup scenario to get a feel for compression.

Early tests with straight MP3's showed 23% compression. These are very, very promising times.

Here's a few links you can drool over until you get the time to do something, or I post my results:

http://blogs.sun.com/jsavit/entry/deduplication_now_in_zfs

http://blogs.sun.com/bonwick/entry/zfs_dedup

http://en.wikipedia.org/wiki/Data_deduplication