Benchmarking ZFS is hard.
ZFS is such a complex FS with multiple levels of cache that getting a good solid reading on it's performance is always difficult.
I was recently tasked with building a FreeBSD ZFS SAN for a client that consisted of 90 1TB Seagate ST1000DM003 6GBps HD's in 2 SuperMicro SC847 Enclosures. The heads were 2 Dell PowerEdge T710's, 96 Gig DDR-3, Dual Xeon 5620 CPU's. I connected the head to the enclosure with a LSI2008 based SAS card.
As always, budget was tight, but performance HAD to be there. I needed to find out for sure what the write speed of this array was going to be under ZFS.
The throttling-effect of using a single SAS card was one of the items I needed to check .
Since the SuperMicro SC847's use a LSI2x36 backplane, I don't have a full SAS channel available to all drives at all times. I wanted to start collecting data on what effect this would have on the performance of the arrays, and did we need to use 2 SAS cards to achieve a higher performance? This wasn't a "performance at all costs" scenario, rather a real-world situation with real-world budgets and needs.
This client had a lot of users, and a lot of big databases that are very active during the day. I needed to know that the array would quickly write down as much data as possible to satisfy their needs.
Knowing that the average-write of the array wouldn't saturate the single SAS card or the drives, but a heavy-write could, I needed to put the array into heavy-writes for a sustained period of time to compare various hardware and software configurations.
Thus began my search for a benchmark program that could really load down this gear.
To simplify things, I decided that I would only concern myself with a write saturation event - When the data is flowing to the drives so quickly that they never catch up - As close to 100% utilization as possible.
I didn't have much luck with the standard FreeBSD benchmarks.. bonnie, bonnie++, iozone, etc. Either they were too hard to lock into a saturate write, or they didn't spawn enough writers to really load down the system.
In the end I ended up coding a quick and dirty program called Saturate.c that I'm passing on here. It's not very pretty, but it works.
It's hard-coded to spawn 4 files to write to for each forked child process. Tweak it as you need to for your system to really drive the files.
You can simply execute the file using time to check how long it takes like this;
time ./saturate
I created a simple script file that then executed the saturate program in different zpool configurations so I could confirm what was working best for us. I've run through various raidz configurations, compression, etc. I'm still sifting through all of the data.
Warning: This will create a very large set of test files - I believe in it's current state it writes 40,000 MegaBytes (~40 Gigs).
Oh, with FreeBSD 9.0, my best average time was 1.2 Minutes. Not bad for a Free Operating System.
I'll post some of my results of 2 months of various ZFS tests over the next while as I have time.
Code on PasteBin: http://pastebin.com/4SexdvLq
ZFS is such a complex FS with multiple levels of cache that getting a good solid reading on it's performance is always difficult.
I was recently tasked with building a FreeBSD ZFS SAN for a client that consisted of 90 1TB Seagate ST1000DM003 6GBps HD's in 2 SuperMicro SC847 Enclosures. The heads were 2 Dell PowerEdge T710's, 96 Gig DDR-3, Dual Xeon 5620 CPU's. I connected the head to the enclosure with a LSI2008 based SAS card.
As always, budget was tight, but performance HAD to be there. I needed to find out for sure what the write speed of this array was going to be under ZFS.
The throttling-effect of using a single SAS card was one of the items I needed to check .
Since the SuperMicro SC847's use a LSI2x36 backplane, I don't have a full SAS channel available to all drives at all times. I wanted to start collecting data on what effect this would have on the performance of the arrays, and did we need to use 2 SAS cards to achieve a higher performance? This wasn't a "performance at all costs" scenario, rather a real-world situation with real-world budgets and needs.
This client had a lot of users, and a lot of big databases that are very active during the day. I needed to know that the array would quickly write down as much data as possible to satisfy their needs.
Knowing that the average-write of the array wouldn't saturate the single SAS card or the drives, but a heavy-write could, I needed to put the array into heavy-writes for a sustained period of time to compare various hardware and software configurations.
Thus began my search for a benchmark program that could really load down this gear.
To simplify things, I decided that I would only concern myself with a write saturation event - When the data is flowing to the drives so quickly that they never catch up - As close to 100% utilization as possible.
I didn't have much luck with the standard FreeBSD benchmarks.. bonnie, bonnie++, iozone, etc. Either they were too hard to lock into a saturate write, or they didn't spawn enough writers to really load down the system.
In the end I ended up coding a quick and dirty program called Saturate.c that I'm passing on here. It's not very pretty, but it works.
It's hard-coded to spawn 4 files to write to for each forked child process. Tweak it as you need to for your system to really drive the files.
You can simply execute the file using time to check how long it takes like this;
time ./saturate
I created a simple script file that then executed the saturate program in different zpool configurations so I could confirm what was working best for us. I've run through various raidz configurations, compression, etc. I'm still sifting through all of the data.
Warning: This will create a very large set of test files - I believe in it's current state it writes 40,000 MegaBytes (~40 Gigs).
Oh, with FreeBSD 9.0, my best average time was 1.2 Minutes. Not bad for a Free Operating System.
I'll post some of my results of 2 months of various ZFS tests over the next while as I have time.
Code on PasteBin: http://pastebin.com/4SexdvLq
Can you provide a copy of saturate.c that is complete?
ReplyDeleteThat is complete, it's a very simple program. What isn't working for you?
Deleteits not complete, the #include lines are empty, the (for i=0; i is incomplete on line 33 and 79, and probably more stuff.
ReplyDeleteAh, google is reformatting my code. Hmm, I wonder what the trick is now to post code here?
ReplyDeleteCan you pastebin it somewhere? Whack it on github?
ReplyDeleteIs there any chance that you could post the code elsewhere? I've been getting SCSI sense errors with some of my cables, and I'd like to test them all to be sure.
ReplyDeletePastebin worked. It's added as a link at the bottom of the article now.
ReplyDeleteThanks for the file, however, I am getting an error while trying to compile it on Linux:
ReplyDelete$ gcc -o disk-saturate disk-saturate.c
disk-saturate.c: In function ‘main’:
disk-saturate.c:88:6: error: too few arguments to function ‘writer’
disk-saturate.c:18:6: note: declared here
I've got the copy from pastebin.
Sounds like a line is un-commented.. sanity-check what you pasted, as it may not be exactly what I originally posted some time ago.
ReplyDeleteI will soon be releasing a new toolset in Python (the one, true language :-) ) that will have a Python version of saturate.c.
It will be a decent exercise for me to see if I can thrash the disks in python as well as I can in c - I may need to cython it, or just wrap some c calls.