Quick iperf test on FreeBSD 10.1 running as a VM in ESXi 5.5

Take this with a grain of salt, as I really haven't done enough testing to state anything conclusive.

However, under ESXi 5.1u3, I was only able to get my iperf scores up to 2/3Gb/s, with the occasional 5Gb/s bust on my 10Gb/s network.

With ESXi 5.5u2 (patches as of today's date) I am able to get scores like this fairly consistently:

[  3]  0.0- 5.0 sec  5.70 GBytes  9.79 Gbits/sec

[  3]  5.0-10.0 sec  5.69 GBytes  9.77 Gbits/sec

[  3] 10.0-15.0 sec  5.69 GBytes  9.77 Gbits/sec

[  3] 15.0-20.0 sec  5.70 GBytes  9.80 Gbits/sec

[  3]  0.0-20.0 sec  22.8 GBytes  9.78 Gbits/sec

Not bad, not bad.

Even if the scores are not accurate, it's at least showing a dramatic improvement over my 5.1 configuration.

This is from a FreeBSD 10.1 VM under ESXi 5.5 to a bare-metal FreeBSD 10.1. VM-to-VM performance is a bit lower, but still quite good:

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0- 5.0 sec  4.89 GBytes  8.41 Gbits/sec

[  3]  5.0-10.0 sec  5.46 GBytes  9.37 Gbits/sec

[  3] 10.0-15.0 sec  4.32 GBytes  7.42 Gbits/sec

[  3] 15.0-20.0 sec  4.87 GBytes  8.37 Gbits/sec

[  3]  0.0-20.0 sec  19.5 GBytes  8.39 Gbits/sec

The FreeBSD VM's are still vmx-08, and I'm using the built-in vmx driver in FreeBSD 10.1

This is also with the default configuration for sysctl values - I was planning on building a small script to check different tcp settings, but with speed like this, there are more pressing things to worry about.

ZFS Dedup Performance - Real World Experience After One Year

I've been running ZFS with deduplication from Open Solaris or FreeBSD for about a year now in a production environment.  I manage ESX vm's, mostly with various flavours of Windows server.


In that year, my thoughts on where to use dedup have changed.

My big ZFS pool is 28.5 TB in a basic mirror, currently showing 11.1TB used, and running a dedup ratio of 1.25x (25% of my data is deduplicated), a savings of around 2.75 TB. This machine runs 24 Gig of RAM, and has a L2ARC comprised of 4 120 Gig SSD's, in additional to 2 ZIL RAM devices.

I have two smaller production units, holding 20.4 TB in a raidz with 12 Gig of RAM, and 16.4 TB in a raidz with 24 gig of RAM.  These secondary systems may or may not have hardware RAM ZIL's and L2ARC SSD caches as I continually move around components in my secondary equipment to fins a sweet-spot between $$ for hardware and performance.


Back to the thought at hand..


Dedup obviously has to have some sort of performance penalty. It simply takes more time to check if you're writing a unique block compared to writing without the check.

Checking the uniqueness of the block involves looking at every block's hash in the pool (even if the fs inside the pool isn't participating in dedup) - A task that increases write execution time as your pool increases in size.


In a perfect world, your system will have enough RAM that the Dedup Table (DDT) will reside entirely in RAM. This means that it only requires traversing the DDT in RAM to find the uniqueness of the block being written.

In the real world, this is nearly impossible. The way the ARC/L2ARC is designed, there isn't a preference for the DDT data - it's simply thought of as metadata, and that competes with user data in the cache system of ZFS. You may think that setting primarycache and/or secondarycache to metadata will fix the problem, but once you switch the primary cache to metadata, you also doom the secondary cache to only hold metadata - Nothing can get into the L2ARC unless it exists in the ARC.

In my experience with a busy pool, enough of your DDT is eroded away from user data writes that dedup performance starts to become a serious issue. These issues don't show up on a fresh system without a lot of data in pool. You need an older, loaded, very fragmented pool to really notice the drop in performance.


How bad? Here's a quick and dirty iometer check on a fs with dedup turned on, and one with it off. This is on my big pool, with plenty of L2ARC and RAM.

Dedup On:  197 IOPS, 0.81 MB/s, 647 ms AvgIO

Dedup Off: 4851 IOPS, 19.87 MB/s, 28 ms AvgIO.

See the difference? This is not a 2x or 4x slowdown of my writes, we're in the 20x slower category to use dedup than with it off.

So we know it's going to be hard to keep the DDT in RAM, so there will be a performance hit. What about the large savings to disk writes, (you don't write the block if it already exists) and thus the performance increase there?

It's simply not there in my systems, and probably not in yours.

Additionally the idea that these dedup blocks could reside in memory cache, allowing multiple systems to take advantage of the same blocks is a good one, but it looks like there is still enough erosion of the caches that this doesn't make a difference in my systems.


If I had a higher dedup ratio, then maybe this would start working out better for me - But even with 40+ VM's of mostly the same stuff, I'm only running a 1.25x dedup ratio on my big pool - I have enough dissimilar data to make dedup impractical on this system.

I'm now going to turn dedup off for nearly all of my production boxes. 

However, turning dedup off doesn't de-dedup your data - it stays deduplicated until fresh data overwites it.

This provides a 'poor-man's offline dedup' trick. Leave dedup off during the day when you need performance, then enable via a script, and copy all of the data in place to *.tmp and then rename to drop the *.tmp when done - You just deduplicated all your data, without the performance hit during the writes in the daytime when I assume your systems would be the busiest.

I'm going to leave dedup enabled on my storage and backup archives - places where I have a lot of duplication, and not a lot of need for speed.Everywhere else, I'm just going to run compression (I find compression=gzip to be a good blend of storage and speed on my systems). I"m averaging close to 2x on my compression alone.

For dedup to work well in a production environment, you're going to need an extreme amount of RAM and L2ARC compared to your drive storage size. I don't have any figures for you, but it's probably cheaper to provision more storage than it would be to try and dedup the data.

What ZFS needs is an offline dedup ability, priority settings for the DDT in ARC/L2ARC, and some general tuning of the ARC/L2ARC to allow for situations like "DDT and Metadata only in RAM, everything else allowed in L2ARC".

..at least for it to be viable in my environment, which is the problem - We all have different environments with different needs.

SSD Fade. It's real, and why you may not want SSDs for your ZIL

SSD's fade - and they fade quicker than you'd expect.

When I say "fade" I mean that their performance drops quickly with write use.

The owner of ddrdrive is rather active on the forums, and was posting some very interesting numbers about how quickly a SSD can drop in performance. He sells a RAM-drive based hardware device that is actually very suitable for a ZIL, it's just very expensive.

Since my ZFS implementation makes use of 8 OCZ Vertex 2 SSD drives (4x60GB ZIL, 4x120GB L2ARC), I thought I should check into this.

First off, when you test SSD's you have to use one of the beta versions of  iometer, so you can enable "Full Random" on the test data stream. OCZ's SSD's (like all others I believe)  perform compression/dedup to limit their writes, resulting in inflated numbers if you use easily compressed data.

How inflated? Try write speeds of 4700 IOPS / 2 MBs for random data, then 7000 IOPS / 3.6 MBs for repeating bytes. Nearly double the performance for easy-to-compress data.

With iometer generating random data, I tested out one of my OCZ Vertex 2 60 Gig SSDs that was in use for 4 months as a ZIL device. That's not exactly an ideal use of SSD's, and you'll see why shortly.

My test was iometer 1.1 on a Windows 7 32 bit box, driving 100% random, 100% write to the SSD, with a queue depth of 128. The writes were not 4k aligned, which does make a difference in speed, but for purposes of illustrating SSD fade, it doesn't matter as a 512b or 4k write is affected equally.Currently ZFS uses 512b as a drive sector size, so you're getting this type of speed unless you've tweaked your ashift value to 12 with a tool like zfs guru.

Back to the tests:

Fresh from my active ZFS system, I was getting 4300 IOPS / 17.5 MBs

After a secure erase with the OCZ Toolbox 2.22, I was getting 6000 IOPS / 24.5 MBs - That's a 40% increase in IOPS just with a secure erase.  My SSD performance has dropped 40% in 4 months.

Curious to see how quickly the SSD will fade, I set iometer up to run the random write for 8 hours. When I came back and ran the test again, I was shocked to see speed down to 910 IOPS / 3.75 MBs. Wow, we're getting close to the territory of a 15k 2.5" SAS drive (around 677 IOPS / 2.77 MBs).

I then did a secure erase of the drive and tested again - Speed was back up, but not as good as what I originally had. I was now obtaining 4000 IOPS / 16 MBs - Worse than when I first started. 8 hours of hard writes used up my SSD a bit more.

Why did 8 hours of writing break down my SSD faster than 4 months of life as a ZIL? Well, I was using 4 of these 60 gig SSDs as one big ZIL device, and the ZIL doesn't consume much space, so the # of GB written over these 4 SSDs in the course of 4 months wasn't nearly as bad as my 8 hour test on one device.

Interestingly, I also have 4 120 Gig Vertex 2 drives for my L2ARC cache. When I pulled and tested these drives, they were far less degraded - The L2ARC is designed to be very conservative on writes, and is mostly for reads. A MLC SSD here works well.

Should you use SSDs for ZIL? You can, but understand how they will degrade.

Your best bet is a RAM based device like a ddrdrive, or similar. There are other slower, cheaper RAM drives out there you could use. These will not degrade with use, and the ZIL gets a lot of writes - Nearly everything written to your pool is written to the ZIL.

If you don't have one of these, your other option is to use lots of devices like I did to ballance the wear/fade. Pull them every 6 months for 15 min of maintenance, and you'll maintain decent speeds.

For now, my 4 60 gig SSDs are going back into ZIL duty until I can decide if a ddrdrive (or a bunch of cheaper ddr devices) is in my future.

ESXi 4.0.0 vs 4.1 Speed - 4.1 Not as fast

I've just upgraded my 4.0.0 ESXi box to 4.1.

...Well, lets not call it upgrading, the process I used was to pull my existing boot drive, install a new 8 gig  USB stick inside my Dell T710, and install a fresh copy of the Dell Installable 4.1 ESXi. This results in a fresh copy, and all I have to do is reattach my storage, import my .VMX files, and I'm back in business.

I once had a bit of a mess with the upgrade process from 3.0 to 3.5, so I settled on this method as being he safest. I even pull my drives when I do the upgrade, so there is no chance the installer will become confused and format my main datastore. Am I paranoid? It's happened before, and you only have to burn me once..

The process isn't that much more time consuming than an in-place upgrade. When your ESXi files are all on one drive/flash stick, and your datastores are all on another, you've got plenty of flexibility. I was able to do it within 30 minutes, because I really don't edit my ESXi configuration that much from default.

 I'm very interested in 4.1 because it has some neat power management interfaces. Look here, I can now track my wattage being burned on this server:

This is for a Dell T710 with 24 gig DDR3 memory, 2 Xeon 5660's,  4 600Gig SAS drives, and 4 1TB nSAS drives. I've yet to check how accurate this is with a KilloWatt or similar meter, but this sounds about right.


I'm also interested in memory compression - When you building redundancy by having two ESX servers, you need to have very similar configurations for processor / memory on your backup server, or you take a large performance hit. That's not always easy to budget for. If I can get away with my Exchange servers still running for a couple of hours on a server that is only 1/2 or 1/4 the memory of my main ESX, then I'll be happy. It's got to be quicker to compress memory than to swap it to disk - We'll see. I'll be testing that later on.

Service Console is now no longer an unsupported hack.

But really, the bit that gets me very interested the most is that vmware is now putting it's full weight behind ESXi - There won't be any more ESX! And to make this transition easier, you can now access vMotion from an ESXi 4.1 server. You still need licenses, but now it's nearly $10k cheaper to access this technology.

With my Ghetto-SAN comming online any week now, I'm very excited about this development.

BUT - There seems to be a but of a trade-off for the new things that 4.1 brings - It's just a tad slower than 4.0.0.

Once again, I do quick and dirty benchmarks to get a feel for things - Performance Test 6.1 isn't the best tool, but it's quick and makes for nice easy graphs to compare. After pouring through reams of iozone stats in Excel, I sometimes like quick, easy, and pretty.

My process was simple: Take a Performance Meter tests from a running 2008 R2 server before the upgrade to 4.1, and one after (with the new vmware tools loaded).

While 4.1 had better graphic performance (who cares!), it was around 2% slower for memory and CPU performance.

That's a small price to pay for new features, and I'm hoping it's just a result of new technology being focused on stability first, performance second.

Anyone else run benchmarks that can confirm or deny this?