Showing posts with label FreeBSD. Show all posts
Showing posts with label FreeBSD. Show all posts

Monday, January 8, 2018

LSI 2008 SAS Chipset (mps0) problems on recent FreeBSD versions

I've encountered a problem with never versions of FreeBSD and my LSI SAS 2008.

I have a number of LSI 2008 based HBAs in service in my FreeBSD storage devices. Most are re-flashed Dell PERC, some are IBM, some are branded LSI.

This particular card is flashed to MPT2BIOS 7.11.10.00 (2011.06.02) in IT mode.

Under FreeBSD 10.2 (and many earlier versions) I have no problems with it.

Attempting to install FreeBSD 11.1, I couldn't get FreeBSD to detect the HBA. I could see my two SES devices (SAS Expanders), but no mps0, no drives.

I did a bit of troubleshooting, and found this across the FreeBSD versions:

10.2  - No Problems
10.3  - Not Tested
10.4  - mps0 shows up, all drives show up, but I have screens and screens of SCSI SENSE errors, can't import zpool's, etc.
11.0 - mps0 dosen't show up, no drives show up, only ses0 ses1
11.1 - same as 11.0

I unfortunately wasted too much time troubleshooting my inactive server. Once I knew what I needed to do to get it online again, I couldn't spend more time to file a bug report and help track this down.

I did encounter this thread, sounds very similar to my problem:

https://forums.freebsd.org/threads/63800/

For now, I'm back to 10.2 on this server until I have the time to fire up a parts machine that I have in storage.

Let me know if you're running into something similar, or have any suggestions.


Sunday, December 31, 2017

FreeBSD 11.x Guest Keyboard Stops Working in VMware

If you're firing up a 11.x instance of FreeBSD in VMware, you may find that the keyboard works until you shift focus to a different window. Shifting focus back to the FreeBSD window will not give you keyboard input again.

For me the fix was this:

In the .vmx file (or in the GUI under Edit Settings -> Options -> General, then Configuration Parameters...) add this line:

Let me know if you needed to do something different to make it work.

keyboard.typematicMinDelay = "2000000"

Let me know if you needed to do something different to make it work. 

Thursday, March 13, 2014

FreeBSD 10 ZFS Boot Hang with USB Boot Drive and Non-USB Keyboard

Here's an interesting one.

If you create a FreeBSD 10 ZFS system that boots from a USB drive, you may hang if you don't have a USB keyboard.

I've noticed this on a few different hardware systems, and it may even be present in FreeBSD 9.x - I haven't had much chance to look into it.

The hangs are always the same ; The zfs boot loader can't find a bootable disk.

When I manually type in zfs:rpool (in the case of the manual ZFS on root setup here) or zfs:zroot/ROOT/default (in the case of the new FreeBSD10 ZFS installer) I'll get one of two things;

1) kernel panic and crash. Note that this is often from pressing 'return' at the boot loader prompt, but typing 'zfs:rpool' won't cause an issue.

2) a small move forward to set the host UUID and hostid, then nothing - just hung.

It looks like when the USB keyboard is not present, then da0 (my USB Flash Stick) doesn't exist.

At some point I'll look into this further, as you shouldn't need any keyboard (USB or other) connected to boot FreeBSD, but for now just make sure you have a USB keyboard hooked up for boot.

Additionally, the order of the USB devices seems to be important. If I have a USB Keyboard on my USB0 Root, and the boot device on my USB1 Root, then I have this symptom as well. If I boot with the boot device on USB0, and keyboard on USB1, then I'm okay. 

Wednesday, December 11, 2013

FreeBSD 9.1 to 9.2 Upgrade with freebsd-update

FreeBSD 9.2 brings a very nice update to ZFS : LZ4 compression.  Upgrading from 9.1 to 9.2 is quite easy if you have a standard install (no customization of the code).


The basic process, which should work for other versions as well is this:

1) Get your system up-to-date with the latest patches for your version of FreeBSD. This isn't necessary, but I think may avoid some problems.

freebsd-update fetch
freebsd-update install
reboot

2) Upgrade the kernel

freebsd-update upgrade -r 9.2-RELEASE
freebsd-update install
reboot

3) Update userland (finishes off the 2 part install)

freebsd-update install
reboot

4) Update pkg if needed

pkg update

5) Enjoy

With my 9.1 -> 9.2 update, I didn't have any messy mergemaster conf files to pour through, nor did my packages notify me that I needed to upgrade them - One of the more painless upgrades I've made.

pkg - FreeBSD's new Package Manager is Finally Working

I was about to write a small entry about how to hook up FreeBSD's next-generation package manager (pkg) to PC-BSD's repository, as FreeBSD wasn't supplying package updates due to the security incident that occurred in Nov 2012.

However, pkg is now able to pull properly from FreeBSD sources, and it's a nice little program.

For all those still using the old pkg_* type tools, just type pkg and follow the prompts.

So far I am finding it easier to upgrade and maintain my packages with pkg, which is always a good thing.

Monday, October 7, 2013

Boot from ZFS root with FreeBSD 9.2

(Update: The newly released FreeBSD 10 makes a ZFS setup in the menu system easy.. give it a try)

Booting from a ZFS root has always been a dream of mine, but I've never managed to make it work.

Since I'm deploying new FreeBSD 9.2 images for testing in my lab, I thought this would be a good time to give it a shot again.

I based this off the FreeBSD install page, with some simplification and a few small additions.

I took a freshly installed FreeBSD 9.2 system to act as my workstation for this, and plugged in a 8 gig USB Flash drive to be my ZFS root.

In my setup, da0 is the booted system, da1 is the blank USB Flash Drive.

I often do not use swap partitions on USB Flash, as it can prematurely wear the drive.  I make my USB Flash drives as disposable as possible, so keeping crash dumps isn't a concern.

NOTE: Solaris always called it's root ZFS pool 'rpool' and I recommend you do the same, as it's a existing standard. My use of 'zroot' here is because I was only fooling around... and seem to be too lazy to change the text below without testing for typos.

Here is what I have executed to make it work:


gpart create -s gpt da1
gpart add -b 34 -s 64k -t freebsd-boot -l boot0 da1
gpart add -t freebsd-zfs -l root0 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

#This may fail with an "operation not permitted" error message, since the kernel likes to protect critical parts of the #disk. If this happens for you, run:
#
# sysctl kern.geom.debugflags=0x10


kldload /boot/kernel/opensolaris.ko
kldload /boot/kernel/zfs.ko

zpool create zroot da1p2
# DO NOT FORGET TO MAKE IT P2! Specify da1, and zfs will kill your gpt tabes
zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot


#Make sure your FreeBSD 9.2 CD is in your drive

mount -t cd9660 /dev/cd0 /mnt

cd /mnt/usr/freebsd-dist

cat base.txz | tar --unlink -xpJf - -C /zroot
cat kernel.txz | tar --unlink -xpJf - -C /zroot
cat doc.txz | tar --unlink -xpJf - -C /zroot
cat lib32.txz | tar --unlink -xpJf - -C /zroot
cat ports.txz | tar --unlink -xpJf - -C /zroot
cat src.txz | tar --unlink -xpJf - -C /zroot

chroot /zroot


echo 'zfs_enable="YES"' > /etc/rc.conf
echo 'sshd_enable="YES"' >> /etc/rc.conf
echo 'hostname="freebsd92zfs"' >> /etc/rc.conf

echo 'zfs_load="YES"' > /boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /boot/loader.conf

tzsetup

cd /etc/mail
make aliases

exit
cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache
zpool export zroot





I then pull the USB Flash Drive, pop it into a spare machine, and presto, it's booting.


I'll fill in more about this as I play with ZFS on root for my USB Flash Drive systems.

Tuesday, March 12, 2013

(FIXED) Network mbuf Leak / Exhaustion in FreeBSD 9.0 / 9.1

UPDATE: Patch included below

There is a rather annoying bug floating around in FreeBSD 9.0 / 9.1 regarding network mubf leaking to the point of mbuf exhaustion.

These is a problem report (PR) filed about this from last year, but it looks to be abandoned.

http://www.freebsd.org/cgi/query-pr.cgi?pr=165903&cat=



 I am experiencing the same mbuf leak on fresh 9.1-RELEASE machines (AMD64). Most of my machines are ESXi 5.1 VM's running the e1000 (em0) NIC. This VM is stock, just one freebsd-update done, nothing custom.

 I have also experienced this condition on an older 9.0-STABLE from Jul 1st 2012. I did not notice it much before that date, but I can't tell for sure. I have a few machines on that build that I still use, so confirmation was easy.

 I do not experience the error if I load up vmware tools and use the vmx3f0 adapter, it's just with em0.

 I have set the mbufs to very high numbers via sysctl kern.ipc.nmbclusters=322144 to buy more time between lockups/crashes. Most often the systems stay functional, they just need a reboot or more mbufs if I add them. Some times the servers lock up or crash as I ifconfig down/up the adapter or attempt to add more mbufs via sysctl.

Is anyone else able to reproduce this?

I have attempted to update the PR or post to the list, but the freebsd.org server and my mail server no longer seem to get along. I'll have to troubleshoot that later this week.

UPDATE Apr 19th 2013:

Gleb Smirnoff was kind enough to quickly forward me a patch that fixed the problem for me. You will need to apply this to usr/src/sys/netinet/if_ether.c

I've now run for 2 days, and my mbufs have not increased at all.  Thanks for the quick response from the FreeBSD-Stable list.

Index: if_ether.c
===================================================================
--- if_ether.c    (revision 249327)
+++ if_ether.c    (working copy)
@@ -558,13 +558,13 @@ in_arpinput(struct mbuf *m)
     if (ah->ar_pln != sizeof(struct in_addr)) {
         log(LOG_NOTICE, "in_arp: requested protocol length != %zu\n",
             sizeof(struct in_addr));
-        return;
+        goto drop;
     }

     if (allow_multicast == 0 && ETHER_IS_MULTICAST(ar_sha(ah))) {
         log(LOG_NOTICE, "arp: %*D is multicast\n",
             ifp->if_addrlen, (u_char *)ar_sha(ah), ":");
-        return;
+        goto drop;
     }

     op = ntohs(ah->ar_op);


Tuesday, December 18, 2012

Passthrough PCIe devices from ESXi to FreeBSD - Leave MSI Enabled to Avoid Interrupt Storms With More than One vCPU

If you want to pass a PCIe device (such as a LSI SAS controller) through ESXi to a FreeBSD VM, you're probably aware that you may need to disable MSI/MSI-X interrupts to make it work.

However, I'm finding that while this setting will make the system boot, if you add a second vCPU, you will quickly stat to see Interrupt Storm messages on your console, and one of your CPU's will be nearly 100% used on interrupt tasks.

The easy fix for me? Only disable MSIX, not MSI.

Here are the relevant lines from my /boot/loader.conf:

hw.pci.enable_msix=0
hw.pci.enable_msi=1

All of my interrupt storm issues are now gone. 

Thursday, November 22, 2012

ESXi 5.1 running FreeBSD 9.0 / 9.1 and VMware Tools

Note - I have another article for ESXi 5.0. These instructions won't work for a 5.0 box.

ESXi 5.1 includes support for the official VMware Tools on FreeBSD 9.0 (yay!).

I'm somewhere between 9.0 and 9.1 RC3, but I feel these instructions will work for either. 

I'll be testing the throughput of the vmxnet3 driver under FreeBSD shorty. For now, I can just confirm that bothe vmxnet2 and vmxnet3 drivers show in FreeBSD, and can ping.

Here's a quick script to install it for you.

You will need to start a VMware Tools install from the guest before running this.


echo "Make sure you have started the vmware tools install"
mount -t cd9660 /dev/cd0 /media
cp /media/vmware-freebsd-tools.tar.gz ~
cd ~
tar xvf vmware-freebsd-tools.tar.gz
cd vmware-tools-distrib
./vmware-install.pl
echo # DONE. Remember that the vmxnet3 driver is called vmxnet3f0


ESXi 5.0 running FreeBSD 9.0 / 9.1 and VMware Tools

Note - I have another article for ESXi 5.1. If you are running 5.1, these instructions won't work for you.

Installing the proper VMware Tools for FreeBSD 9.0 / 9.1 can be a pain.

The hard work to figure this out has already been done by others (http://ogris.de/vmware/), so here is a quick script that will install the official VMware Tools using Dru's patch.

Make sure to start a vmtools install from your vsphere console so ESXi will make the CD available to you.

If you run into problems with a missing library when you try and execute /usr/local/bin/vmare-toolbox-cmd, try looking here (http://lists.freebsd.org/pipermail/freebsd-questions/2010-June/217718.html) it solved the problem for me. 

(Cut-n-Paste into a terminal window with root access)

echo "Make sure you have started the vmware tools install"
mount -t cd9660 /dev/cd0 /media
cp /media/vmware-freebsd-tools.tar.gz ~
cd ~
tar xvf vmware-freebsd-tools.tar.gz
cd vmware-tools-distrib
cd lib/modules/source
tar xvf vmblock.tar
tar xvf vmmemctl.tar
tar xvf vmxnet.tar
tar xvf vmxnet3.tar

fetch http://ogris.de/vmware/vmxnet.diff
fetch http://ogris.de/vmware/vmxnet3.diff

echo #
echo # If it's thinking you have a previously applied patch (-R) say yes.
echo #

patch -p1 < vmxnet.diff
patch -p0 < vmxnet3.diff


cd vmblock-only
make && make install
cd ..

cd vmmemctl-only
make && make install
cd ..

cd vmxnet-only
make && make install
cd ..

cd vmxnet3-only
make && make install
cd ..

cd ~/vmware-tools-distrib


./vmware-install.pl

echo # DONE. Remember that the vmxnet3 driver is called vmxnet3f0

Thursday, May 3, 2012

ZFS Testing - Saturate.c

Benchmarking ZFS is hard.

ZFS is such a complex FS with multiple levels of cache that getting a good solid reading on it's performance is always difficult.

I was recently tasked with building a FreeBSD ZFS SAN for a client that consisted of 90 1TB Seagate ST1000DM003 6GBps HD's in 2 SuperMicro SC847 Enclosures. The heads were 2 Dell PowerEdge T710's, 96 Gig DDR-3, Dual Xeon 5620 CPU's. I connected the head to the enclosure with a LSI2008 based SAS card.

As always, budget was tight, but performance HAD to be there. I needed to find out for sure what the write speed of this array was going to be under ZFS.

The throttling-effect of using a single SAS card was one of the items I needed to check .

Since the SuperMicro SC847's use a LSI2x36 backplane, I don't have a full SAS channel available to all drives at all times.  I wanted to start collecting data on what effect this would have on the performance of the arrays, and did we need to use 2 SAS cards to achieve a higher performance? This wasn't a "performance at all costs" scenario, rather a real-world situation with real-world budgets and needs.

This client had a lot of users, and a lot of big databases that are very active during the day. I needed to know that the array would quickly write down as much data as possible to satisfy their needs.

Knowing that the average-write of the array wouldn't saturate the single SAS card or the drives, but a heavy-write could, I needed to put the array into heavy-writes for a sustained period of time to compare various hardware and software configurations.

Thus began my search for a benchmark program that could really load down this gear.
To simplify things, I decided that I would only concern myself with a write saturation event - When the data is flowing to the drives so quickly that they never catch up - As close to 100% utilization as possible.

 I didn't have much luck with the standard FreeBSD benchmarks.. bonnie, bonnie++, iozone, etc. Either they were too hard to lock into a saturate write, or they didn't spawn enough writers to really load down the system.

In the end I ended up coding a quick and dirty program called Saturate.c that I'm passing on here. It's not very pretty, but it works.

It's hard-coded to spawn 4 files to write to for each forked child process. Tweak it as you need to for your system to really drive the files.

You can simply execute the file using time to check how long it takes like this;

time ./saturate

I created a simple script file that then executed the saturate program in different zpool configurations so I could confirm what was working best for us.  I've run through various raidz configurations, compression, etc. I'm still sifting through all of the data.

Warning: This will create a very large set of test files - I believe in it's current state it writes 40,000 MegaBytes (~40 Gigs).

Oh, with FreeBSD 9.0, my best average time was 1.2 Minutes.  Not bad for a Free Operating System.

I'll post some of my results of 2 months of various ZFS tests over the next while as I have time.

Code on PasteBin: http://pastebin.com/4SexdvLq








Thursday, March 8, 2012

FreeBSD SCSI Sense Errors - Did You Check Your Cables?

I've been suffering from a lot of SCSI Sense errors under FreeBSD 9.0-STABLE with a recent ZFS based SAN build that have been driving me mad.

They are mostly present when ZFS does a wide scan of the available drives - via a 'zpool import' or 'zpool scan' or other operation. 

My drives are Seagate ST1000DM0003 SATA3 1TB drives, contained in a SuperMicro SC847 Chassis (LSI SAS2x36 SAS2 Expander so the whole thing can run 6gbps).

These errors also show after a few hours of running a custom script to saturate writes to my ZFS pool. SCSI Sense errors build until we end up in a flurry of errors on the console, and a hang of the SAS expanders, sometimes taking ZFS with it. 

Mar  7 13:50:06 Test kernel: (da33:mps1:0:27:0): CAM status: SCSI Status Error
Mar  7 13:50:06 Test kernel: (da33:mps1:0:27:0): SCSI status: Check Condition
Mar  7 13:50:06 Test kernel: (da33:mps1:0:27:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Mar  7 13:50:11 Test kernel: (da1:mps1:0:11:0): WRITE(10). CDB: 2a 0 0 6e 45 bf 0 1 0 0 length 131072 SMID 303 terminated ioc 804b s


Doing a bit of research, I find that I'm not the only one with SCSI Sense error problems, and naturally I started suspecting the recent mps driver committed from LSI, or some sort of firmware interaction issue with it (Check my recent post on the ixbge driver issue with LACP).

I've used the exact same hardware in the past for FreeBSD 9.0-BETA builds without this trouble, so I was sure it was software or expander related.

My HBA's are mostly Supermicro AOC-USAS2-L8i internal SAS2 cards for my ZFS builds. It's a good value card, and gives me what I want at a decent price.

The firmware on these cards was v7.21 and I knew Supermicro had newer BIOS and firmware, so I flashed them up to BIOS v 11 and FW v7.23. The documentation with the flash files shows a lot of advancement in the FW and BIOS that corrects errors, and I thought it would be a good try - Maybe the new mps driver made better use of the card, and needed newer firmware.

However, after the upgrade, my problems continued. System did seem to boot a little faster however.

I then applied the LSI direct FW (v7.23) and BIOS (v12) that you can find on LSI support site for the 9211-8i card (same thing as SuperMicro's). The flashing process was the same (although the SuperMicro download was a bit more automated with it's batch file).

Same problems.

I then started testing one item at a time - ending up with replacing my brand new Amphenol SAS2 external Mini-SAS cables for some older ones I had in use on a different server.

And it fixed it!

Damn! I then confirmed by swapping between the new and old cables that it was the two new cables that I had purchased that were causing all of this. Very frustrating, as it's a high quality, expensive Amphenol cable, not a no-name thin OEM thingie.

What was more puzzling was that a nearly identical hardware array was using the same cables, (from the same order as the bad cables), and it wasn't having issues.

However, that in-use array is using SATA2 1.5 Gbps drives, not 6Gbps SATA3.  When I checked the logs carefully, I saw we were running into a few SCSI Sense errors, but nothing was really wrong with the servers, and ZFS was generally happy.

I'm fairly confident that if I put that array under the strain of my saturation script, it would throw errors as well, possibly not as fast as the 6Gbps array does.

The Amphenol supplier has been most kind, and is offering to RMA the 4 cables for me. I'll get a new batch of 4, and will check them out carefully under heavy load before I deploy them.

It makes me wonder what other cables I have that will fail under heavy load. Data speeds are quickly increasing, and I'm sure it puts a strain on the cable manufacturers to keep up with the latest spec'ed cable, connectors, and techniques to ensure that the cables they make are indeed reliable at any speed, not to mention having to buy new cable testing gear. 


Wednesday, March 7, 2012

FreeBSD 9.0 and Intel X520's - ixgbe 2.3.11 won't allow LACP to function, but 2.4.4 does

Working with the Intel x520 adapters can be a little bit challenging if you're not used to them. I'll expand on what I've learned over the last year or so in a separate post.

For now, I hope to save someone a weekend of wasted time - If you're attempting to use a FreeBSD 9.0 system with an Intel x520 (ix/ixgbe driver), and you want to setup a LACP link, it won't work.

Here's what you'll see:

lagg1: flags=8843 metric 0 mtu 9000
        options=401bb
        ether d4:ae:52:5c:5d:af
        inet 172.16.0.30 netmask 0xffffff00 broadcast 172.16.0.255
        inet6 fe80::d6ae:52ff:fe5c:5daf%lagg1 prefixlen 64 scopeid 0xe
        nd6 options=29
        media: Ethernet autoselect
        status: active
        laggproto lacp
        laggport: ix1 flags=18 COLLECTING,DISTRIBUTING
        laggport: ix0 flags=18 COLLECTING,DISTRIBUTING


You'll notice it's just COLLECTING, DISTRIBUTING, where a properly functioning link should be ACTIVE, COLLECTING, DISTRIBUTING.

Check your driver version like so:

#sysctl dev.ix

dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.11
dev.ix.0.%driver: ix
dev.ix.0.%location: slot=0 function=0
dev.ix.0.%pnpinfo: vendor=0x8086 device=0x10f8 subvendor=0x8086 subdevice=0x000c class=0x020000



Older drivers seem to work on slightly older FreeBSD 9-Beta systems (like ixgbe 2.3.10).

The eventual fix was to download the latest FreeBSD driver from Intel's website (v2.4.4) compile it as a module, and then add it to my /boot/loader.conf file. Note, you can rename the produced ixgbe to to ixgbe2.4.4.ko before you copy it to /boot/kernel/, allowing you more control over which version of ixgbe you load. My kernel had ixgbe compiled in, but it still accepts the new 2.4.4 loaded via loader.conf:

ixgbe2.4.4_load="YES"

If you try loading a .ko over your existing ixgbe built into the kernel, make sure you check your driver version with a "sysctl dev.ix" command - I found that an older FreeBSD-9.0-RELEASE wouldn't load my new 2.4.4, but systems closer to the build time of my test FreeBSD-9-STABLE machine would.

This brings up questions of if we really want a lot of device drivers in the kernel, as it can make troubleshooting harder, but that's another post as well. 

You'll have to do a quick edit of the code to get it to compile,as it's complaining about the bool typedef.  Basically you need to remove the reference, and change the 4-6 instances of boolean_t to bool (you'll see the errors).

If that's too complex, hopefully Jack will have 2.4.4 ported to FreeBSD-9-STABLE shortly, I made a post to the stable list, and he's aware of it.


Monday, June 27, 2011

Speeding up FreeBSD's NFS on ZFS for ESX clients

My life revolves around four 3-letter acronyms: BSD, ESX, ZFS, and NFS.

However, these four do not get along well, at least not on FreeBSD.

The problem is between ESX's NFSv3 client and ZFS's ZIL.

You can read a bit about this from one of the ZFS programmers here - Although I don't agree that it's as much of a non-issue as this writer found.

ESX uses a NFSv3 client, and when it connects to the server, it always asks for a sync connection. It doesn't matter what you set your server to, it will be forced by the O_SYNC command from ESX to sync all writes.

By itself, this isn't a bad thing, but when you add ZFS to the equation,we now have an unnecessary NFS sync due to ZFS's ZIL. It's best to leave ZFS alone, and let it write to disk when it's ready, instead of instructing it to flush the ZIL all the time. Once ZFS has it, you can forget about it (assuming you haven't turned off the ZIL).

Even if your ZIL is on hardware RAM-drives, you're going to notice a slow-down. The effect is magnified on a HD based ZIL (which is what you have if you don't have a separate log device on SSD/RAM). For my tests, I was using a hardware RAM device for my ZIL.

Some ZFS instances can disable the ZIL. We can't in FreeBSD if you're running ZFS v28.

Here's two quick iometer tests to show the difference between a standard FreeBSD NFS server, and my modified FreeBSD NFS server.

Test Steup: Running iometer 1.1 devel,  on a Windows 7 SP1 machine, connected to the test drives via NFS. iometer has 128 writers, full random, 4k size, 50% write 50% read, 100% sequential access, 8GB file, 15 run-time. Reboot after each test, ran each test twice to make sure we were receiving a sane result. Using FreeBSD 9-CURRENT as of 2011.05.28.15.00.00

Standard NFS
Test1    1086 IOPS    4.45 MBs    117 AvgIO (ms)
Test2    1020 IOPS    4.18 MBs    125 AvgIO (ms)

Modified NFS
Test 3   2309 IOPS    9.45 MBs    55 AvgIO (ms)
Test 4   2243 IOPS    9.19 MBs    57 AvgIO (ms)

I feel the results speak for themselves, but in case they don't - We're looking at an increase in IOPS, MB/per sec, and a decrease in the time to access the information when we use the modified NFS server code. For this particular test, we're looking at nearly a doubling in performance. Other tests are close to a 10% increase in speed, but that's still a wanted increase.

These test results will be apparent if you're using the old NFS server (v2 and v3 only) or the new NFS server (v2-3-4) that is now the default in FreeBSD 9 as of a month ago.

I've used this hack for over 6 months now on my SANs without any issue or corruption, on both 8.1 and various 9-Current builds, so I believe it's fairly safe to use.

I'm too lazy to make a proper patch, but manually editing the source is very easy:

- The file is /usr/src/sys/fs/nfsserver/nfs_nfsdport.c
- Go to line 704, you'll see code like this;
(Edit: Now line 727 in FreeBSD-9.0-RC3)

if (stable == NFSWRITE_UNSTABLE)
  ioflags = IO_NODELOCKED;
else
  ioflags = (IO_SYNC IO_NODELOCKED);
uiop->uio_resid = retlen;
uiop->uio_rw = UIO_WRITE;

- Change the code to look like this below. We're commenting out the logic that decides to allow this to be an IO_SYNC write.

// if (stable == NFSWRITE_UNSTABLE)
ioflags = IO_NODELOCKED;
// else
// ioflags = (IO_SYNC | IO_NODELOCKED);
uiop->uio_resid = retlen;
uiop->uio_rw = UIO_WRITE;

- Recompile your kernel, install it, and reboot - You're now free from NFS O_SYNC's under ESX.


If you are running the older NFS server (which is the default for 8.2 or older), the file to modify is /usr/src/sys/nfsserver/nfs_serv.c  - Go to line 1162 and comment out these lines as shown in this example:

// if (stable == NFSV3WRITE_UNSTABLE)
ioflags = IO_NODELOCKED;
// else if (stable == NFSV3WRITE_DATASYNC)
// ioflags = (IO_SYNC IO_NODELOCKED);
// else
// ioflags = (IO_METASYNC | IO_SYNC | IO_NODELOCKED);

If you try this, and let me know what type of before and after speed results you're receiving.

Wednesday, June 22, 2011

FreeBSD 9.0 and EKOPath (Path64) Compiler

You may be aware that Pathscale announced they are making their EKOPath compiler suite open-source. This is great news, even if it's a GPLv3 license. I''d be happier with a BSD license so we could bundle it easier with FreeBSD, but regardless, having free access to this excellent compiler is just what we need to increase FreeBSD's performance, and start catching up with other OS'es for performance.

Phoronix released info about the story here

Compilers do make a difference in the speed of our OS, don't think for a second that you're getting full performance in FreeBSD from the old gcc 4.2.2 on modern hardware.

I've recently done some tests with clang/llvm vs gcc with various optimization switches, and there is a definite increase from the default generic builds of FreeBSD for my standard environment of NFS-ZFS-ESX. I'll post about that shortly when I've finished my tests.

For now, Martin Matuska has some info about compiling FreeBSD with a newer version of gcc, and also links to some statistically valid data showing the speed increase from doing this. You can start here, and find his performance data link at the bottom.  

clang/llvm is a great step forward, I wish it would beat gcc for compiled binary run speed, but it can't. In my tests, the only time clang/llvm was faster than gcc was when I was doing compression based tests, and I'm assuming this is because the newer clang/llvm can take advantage of more modern processor extensions than the older gcc 4.2.2

FreeBSD is committed to clang/llvm, as it's a BSD license, and we need that for making the entire FreeBSD distribution as GPL free.  It will get better, but FreeBSD isn't the focus of this project.

Anyway, I digress: This is about Path64. Start here:

Sources to download Path64 compiler:
You'll need to include two libraries for this to work;

pkg_add -r libdwarf
pkg_add -r cmake
rehash

Follow the instructions in the readme, or check Marcello's page for more info.

I followed his instructions, the only difference is that I'm on FreeBSD-9-CURRENT, a build from 2011.05.28.15.00.00 , and I used this cmake command:

set MYLIBPATH=/usr/lib

cmake ~/work/path64 \
-DPATH64_ENABLE_TARGETS=x86_64 \
-DPATH64_ENABLE_MATHLIBS=ON \
-DPATH64_ENABLE_HUGEPAGES=OFF \
-DPATH64_ENABLE_FORTRAN=OFF \
-DPSC_CRT_PATH_x86_64=$MYLIBPATH \
-DPSC_DYNAMIC_LINKER_x86_64=/libexec/ld-elf.so.1 \
-DPSC_LIBSUPCPP_PATH_x86_64=$MYLIBPATH \
-DPSC_LIBSTDCPP_PATH_x86_64=$MYLIBPATH \
-DPSC_LIBGCC_PATH_x86_64=$MYLIBPATH \
-DPSC_LIBGCC_EH_PATH_x86_64=$MYLIBPATH \
-DPSC_LIBGCC_S_PATH_x86_64=$MYLIBPATH \
-DPATH64_ENABLE_HUGEPAGES=OFF \
-DCMAKE_BUILD_TYPE=Debug

There is discussion about this on the FreeBSD mailing lists here.

That's as far as I've made it - The tests with the compiler start now. I dream of a buildworld with this, but I know that's not going to be easy. For now, I'm going to start with some "Hello World" type programs.

:-)

Friday, May 20, 2011

Followup: ZFS Data Gone

In Feburary I blogged about a nasty data loss event with ZFS

http://christopher-technicalmusings.blogspot.com/2011/02/zfs-data-gone-data-gone-love-is-gone.html

I've been quite busy since then and haven't followed up on the results. As a few people have been asking if I was able to get the data back, here is my answer:

Yes, I did get most of it back, thanks to a lot of effort from George Wilson (great guy, and I'm very indebted to him). However, any data that was in play at the time of the fault was irreversibly damaged and couldn't be restored. Any data that wasn't active at the time of the crash was perfectly fine, it just needed to be copied out of the pool into a new pool. George had to mount my pool for me, as it was beyond non-ZFS-programmer skills to mount. Unfortunately Solaris would dump after about 24 hours, requiring a second mounting by George. It was also slower than cold molasses to copy anything in it's faulted state. If I was getting 1 Meg/Sec, I was lucky. You can imagine that creates an issue when you're trying to evacuate a few TB of data through a slow pipe.

After it dumped again, I didn’t bother George for a third remounting (or I tried very half-heartedly, the guy was already into this for a lot of time, and we all have our day jobs), and abandoned the data that was still stranded on the faulted pool. I copied my most wanted data first, so what I abandoned was a personal collection of movies that I could always re-rip.


I was still experimenting with ZFS at the time, so I wasn't using snapshots for backup, just conventional image backups of the VM's that were running. Snapshots would have had a good chance of protecting my data from the fault that I ran into.


I was originally blaming my Areca 1880 card, as I was working with Areca tech support on a more stable driver for Solaris, and was on the 3rd revision of a driver with them. However, in the end it wasn't the Areca, as I was very familiar with it's tricks - The Areca would hang (about once every day or two), but it wouldn't take out the pool. After removing the Arcea and going with just LSI 2008 based controllers, I had one final fault about 3 weeks later that corrupted another pool (luckily it was just a backup pool). At that point, the swearing in the server room reached a peak, I booted back into FreeBSD, and haven't looked back.

Originally when I used the Areca controller with FreeBSD, I didn't have any problems with it during the 2 month trial period.

I've had only small FreeBSD issues since then, nothing else has changed on my hardware. So the only claim I can make is that in my environment, on my hardware, I've had better stability with FreeBSD than I did with Solaris.

Interesting Note: One of the speed slow-downs with FreeBSD compared to Solaris from my tests was the O_SYNC method that ESX uses to mount a NFS store. I edited the FreeBSD NFS source to always do a async write, regardless of the O_SYNC from the client, and that perked FreeBSD up a lot for speed, making it fairly close to what I was getting on Solaris.

I'm not sure why this makes such a difference, as I'm sure Solaris is also obeying the O_SYNC command. I do know that the NFS 3 code from FreeBSD is very old, and a bit cluttered - It could just be issues there, and the async hack gives it back speed it loses in other areas.

FreeBSD is now using a 4.1 NFS server by default as of the last month, and I'm just starting my stability tests with using a new FreeBSD-9 build to see if I can run newer code. I'll do speed tests again, and will probably make the same hack to the 4.1 NFS code to force async writes. I'll post to an update when I get this far.

I do like Solaris - After some initial discomfort about the different way things were being done, I do see the overall design and idea, and I now have a wish list of features I'd like see ported to FreeBSD. I think I'll have a Solaris based box setup again for testing. We'll see what time allows.

Sunday, January 9, 2011

Solaris 11 Express - Faster Speed for SAN-NFS-VM than FreeBSD

I love FreeBSD. It's been my go-to OS since I first switched to it from BSDi many many years ago. However, I hate to stick with something just because I've always used it. Technology changes too quickly to make lifelong bonds. While I prefer FreeBSD, I ultimately will choose what gets the job done the best.

ZFS under FreeBSD has always progressed nicely, but it is obviously behind Solaris, seeing how Sum/Oracle are the creators of ZFS. The ZFS v28 patch is currently being tested for FreeBSD-8.2 and 9, but not released yet.

While I realize these are pre-releases, I wanted to test the performance I could expect from FreeBSD vs Solaris for my SAN that I'm building. Once I put this SAN into regular use, I won't easily be able to switch to another OS, or do a lot of tweaking with it.

This is not just a test of ZFS implementations - It tests the hard disk subsystem, memory, NFS, ethernet drivers, and ZFS to give me a combined test result, which is what I really care about - How fast can my SAN accept and serve up data to my ESXi boxes.

I've seperately been benchmarking FreeBSD 8.2-PRE and 9.0-CURRENT with both the v28 ZFS patch, and with their native v15 ZFS implementations. I'll make that a seperate blog entry a bit later on, but for now, I'm choosing the fastest performer from the v28 portion of those tests, which was FreeBSD 9.0-CURENT.

My test environment is as follows:

ESX Box: Dell PowerEdge T710 w/96Gig
SAN Box: Dell PowerEdge Y710 w/24Gig
Network: Intel X510DA2 10Gbe Ethernet  in each box, direct attached
SAS Card: Areca 1880ix w/4G Cache
Enclosure: SuperMicro SC847-JBOD2 (Dual 6Gbps Backplanes)
Drives: Seagate 1.5 TB SATA, 5 disk raidz, no log or cache
Test Software: Performance Test 6.1 run on a Windows 2003 VM


Solaris 11 Express svn_151a

903 MBs - Fileserver
466 MBs - Webserver
53 MBs - Workstation
201 MBs - Database

FreeBSD-9.0 Current @ Dec 12th 2010 w/v28 Patch, all Debug off
95 MBs - Fileserver
60 MBs - Webserver
30 MBs - Workstation
32 MBs - Database


Wow. Notice the difference? I didn't believe it myself at first, but I tried it again and again, I watched the packet traffic accross the Intel card with iftop, and then I ran seperate benchmarks with iometer just to make sure it wasn't something silly in Performance Meter. I always received similar results - Solaris 11 Express can move some serious data in comparison to FreeBSD 9.0

At this stage, I don't know if it's a problem with the Intel driver holding back FreeBSD or not. You'll notice that the Workstation test is the only one that is compariable between FreeBSD and ZFS, which is made up of a lot of sync random read/writes that brings any ZFS implementation to it's knees.

The thing is: I don't care if it's the Intel driver, the memory sub-system, or whatever - In the end, I need the best performance delivered to my ESXi boxes, and Solaris 11 Express can deliver where FreeBSD isn't ready just yet.

I do fully expect FreeBSD to catch up, but for now, I'm strongly considering spendng the $$ on a Solaris 11 Express license so I can run it as my SAN OS. Solaris' easy CIFS implementation is another bonus for my SAN - it was delivering 78 MBs speeds when my FreeBSD/samba implementation was at 35MBs.

Friday, December 31, 2010

OpenBSD 4.8 vs FreeBSD 8.1 em0 Network Performance

As mentioned in one of my earlier postings, I have been looking into OpenBSD as a possible firewall OS.

I'll post a more opinion-based article shortly on what I think about OpenBSD vs FreeBSD, but for now, I thought I'd report this tidbit of info;

When benchmarking with netperf, OpenBSD 4.8 wasn't as fast as FreeBSD 8.1 until I tweaked some sysctls.

OpenBSD 4.8 - 520.49 Mbits/sec
FreeBSD 8.1 - 941.49 Mbits/sec

I then applied the speed tweaks for OpenBSD found here: https://calomel.org/network_performance.htm , and then was able to match FreeBSD's performance using NetPerf.

While this was with the em driver (Intel), I believe it would show on other drivers as well, as the sysctl's adjusted weren't specific to the em driver.

Conclusion? Make sure you're applying these tuning tips if you're running OpenBSD 4.8

Wednesday, December 15, 2010

lagg performance penalty between ESXi 4.1 and FreeBSD 8.1

While doing some benchmarks of my ZFS/NFS/ESX setups, I started fiddling with the lagg driver to create a fail-over connection between my ESX server and the ZFS server.

Both boxes have the Intel x520DA 10Gbe adapter, which has 2 10 Gig ports.

ESXi doesn't offer proper load-balance between two ports unless you get into the Cisco Nexus vswitch, which I currently don't have.


I ran a few quick tests with my usual 2003 32 Bit VM and Performance test 6.1, and here's what I found;



Type of Test        With lagg                Without lagg    % Diff
File Server:            478.16                     491                     2.6
Webserver:            84.19                       86.46                  2.6
Workstation:          0.52                         0.6                      13
Database:              127.88                     136.15                 6

Every test was slower with lagg being used.
This is something interesting to think about, as a lot of admins enable lagg or similar failover technology without thinking that there may be a performance impact. 

At this stage, I don't know if it's on the lagg side, or ESX's side - I'd have to run further tests. I'll add it to the list of things that I'd like to know if I had time.

But for now, I know that I'll take a small performance hit to have this fail-over setup.

Sunday, December 5, 2010

Netperf and SMP - Oddness (Part 2 of 2)

To continue my previous post.. http://christopher-technicalmusings.blogspot.com/2010/12/netperf-and-smp-oddness.html

I was concerned that the poor performance of netperf when it was run on a multi-processor system was due to SMP overhead.

The only way to know for sure would be to run the test with netperf and netserver on separate (but identical) machines. That way I could ensure data was transferring across the network cable, and not being accelerated by any buffer-copy process within the processors or system.

I once again setup my netperf and netserver test, using FreeBSD 8.1 AMD64 on two Dell PowerEge 1950's.

In one set of tests, I restricted both ends to use a single CPU. In the other I let the system choose what CPU to run on (which results in the process jumping CPU a number of times as it executes the threads on the most opportunistic CPU - as noted with top -P).


The command was;

netperf -H 192.168.88.1 -L 192.168.88.2 -t TCP_STREAM -l 300 -f  m

The results show no difference between multi-processor and single processor results. In either test, I saw 941.49 Megabits/sec, regardless of single or multiple CPU's.

Which means that there are some serious optimizations for working between 2 network cards in the same physical machine, as seen in my previous tests. The same tests run from one internal NIC to the other result in 4500 - 9000 MBits/sec depending on if it's all through one CPU or across multiple CPU's.

I obviously need to spend more time understanding this, as it would make a large difference when you're considering a firewall application, or anything that needs to move a lot of data between interfaces.

I welcome any insights..