Showing posts with label lagg. Show all posts
Showing posts with label lagg. Show all posts

Wednesday, March 7, 2012

FreeBSD 9.0 and Intel X520's - ixgbe 2.3.11 won't allow LACP to function, but 2.4.4 does

Working with the Intel x520 adapters can be a little bit challenging if you're not used to them. I'll expand on what I've learned over the last year or so in a separate post.

For now, I hope to save someone a weekend of wasted time - If you're attempting to use a FreeBSD 9.0 system with an Intel x520 (ix/ixgbe driver), and you want to setup a LACP link, it won't work.

Here's what you'll see:

lagg1: flags=8843 metric 0 mtu 9000
        options=401bb
        ether d4:ae:52:5c:5d:af
        inet 172.16.0.30 netmask 0xffffff00 broadcast 172.16.0.255
        inet6 fe80::d6ae:52ff:fe5c:5daf%lagg1 prefixlen 64 scopeid 0xe
        nd6 options=29
        media: Ethernet autoselect
        status: active
        laggproto lacp
        laggport: ix1 flags=18 COLLECTING,DISTRIBUTING
        laggport: ix0 flags=18 COLLECTING,DISTRIBUTING


You'll notice it's just COLLECTING, DISTRIBUTING, where a properly functioning link should be ACTIVE, COLLECTING, DISTRIBUTING.

Check your driver version like so:

#sysctl dev.ix

dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.11
dev.ix.0.%driver: ix
dev.ix.0.%location: slot=0 function=0
dev.ix.0.%pnpinfo: vendor=0x8086 device=0x10f8 subvendor=0x8086 subdevice=0x000c class=0x020000



Older drivers seem to work on slightly older FreeBSD 9-Beta systems (like ixgbe 2.3.10).

The eventual fix was to download the latest FreeBSD driver from Intel's website (v2.4.4) compile it as a module, and then add it to my /boot/loader.conf file. Note, you can rename the produced ixgbe to to ixgbe2.4.4.ko before you copy it to /boot/kernel/, allowing you more control over which version of ixgbe you load. My kernel had ixgbe compiled in, but it still accepts the new 2.4.4 loaded via loader.conf:

ixgbe2.4.4_load="YES"

If you try loading a .ko over your existing ixgbe built into the kernel, make sure you check your driver version with a "sysctl dev.ix" command - I found that an older FreeBSD-9.0-RELEASE wouldn't load my new 2.4.4, but systems closer to the build time of my test FreeBSD-9-STABLE machine would.

This brings up questions of if we really want a lot of device drivers in the kernel, as it can make troubleshooting harder, but that's another post as well. 

You'll have to do a quick edit of the code to get it to compile,as it's complaining about the bool typedef.  Basically you need to remove the reference, and change the 4-6 instances of boolean_t to bool (you'll see the errors).

If that's too complex, hopefully Jack will have 2.4.4 ported to FreeBSD-9-STABLE shortly, I made a post to the stable list, and he's aware of it.


Wednesday, December 15, 2010

lagg performance penalty between ESXi 4.1 and FreeBSD 8.1

While doing some benchmarks of my ZFS/NFS/ESX setups, I started fiddling with the lagg driver to create a fail-over connection between my ESX server and the ZFS server.

Both boxes have the Intel x520DA 10Gbe adapter, which has 2 10 Gig ports.

ESXi doesn't offer proper load-balance between two ports unless you get into the Cisco Nexus vswitch, which I currently don't have.


I ran a few quick tests with my usual 2003 32 Bit VM and Performance test 6.1, and here's what I found;



Type of Test        With lagg                Without lagg    % Diff
File Server:            478.16                     491                     2.6
Webserver:            84.19                       86.46                  2.6
Workstation:          0.52                         0.6                      13
Database:              127.88                     136.15                 6

Every test was slower with lagg being used.
This is something interesting to think about, as a lot of admins enable lagg or similar failover technology without thinking that there may be a performance impact. 

At this stage, I don't know if it's on the lagg side, or ESX's side - I'd have to run further tests. I'll add it to the list of things that I'd like to know if I had time.

But for now, I know that I'll take a small performance hit to have this fail-over setup.