Working with the Intel x520 adapters can be a little bit challenging if you're not used to them. I'll expand on what I've learned over the last year or so in a separate post.
For now, I hope to save someone a weekend of wasted time - If you're attempting to use a FreeBSD 9.0 system with an Intel x520 (ix/ixgbe driver), and you want to setup a LACP link, it won't work.
Here's what you'll see:
lagg1: flags=8843 metric 0 mtu 9000
options=401bb
ether d4:ae:52:5c:5d:af
inet 172.16.0.30 netmask 0xffffff00 broadcast 172.16.0.255
inet6 fe80::d6ae:52ff:fe5c:5daf%lagg1 prefixlen 64 scopeid 0xe
nd6 options=29
media: Ethernet autoselect
status: active
laggproto lacp
laggport: ix1 flags=18 COLLECTING,DISTRIBUTING
laggport: ix0 flags=18 COLLECTING,DISTRIBUTING
You'll notice it's just COLLECTING, DISTRIBUTING, where a properly functioning link should be ACTIVE, COLLECTING, DISTRIBUTING.
Check your driver version like so:
#sysctl dev.ix
dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.11
dev.ix.0.%driver: ix
dev.ix.0.%location: slot=0 function=0
dev.ix.0.%pnpinfo: vendor=0x8086 device=0x10f8 subvendor=0x8086 subdevice=0x000c class=0x020000
Older drivers seem to work on slightly older FreeBSD 9-Beta systems (like ixgbe 2.3.10).
The eventual fix was to download the latest FreeBSD driver from Intel's website (v2.4.4) compile it as a module, and then add it to my /boot/loader.conf file. Note, you can rename the produced ixgbe to to ixgbe2.4.4.ko before you copy it to /boot/kernel/, allowing you more control over which version of ixgbe you load. My kernel had ixgbe compiled in, but it still accepts the new 2.4.4 loaded via loader.conf:
If you try loading a .ko over your existing ixgbe built into the kernel, make sure you check your driver version with a "sysctl dev.ix" command - I found that an older FreeBSD-9.0-RELEASE wouldn't load my new 2.4.4, but systems closer to the build time of my test FreeBSD-9-STABLE machine would.
This brings up questions of if we really want a lot of device drivers in the kernel, as it can make troubleshooting harder, but that's another post as well.
You'll have to do a quick edit of the code to get it to compile,as it's complaining about the bool typedef. Basically you need to remove the reference, and change the 4-6 instances of boolean_t to bool (you'll see the errors).
If that's too complex, hopefully Jack will have 2.4.4 ported to FreeBSD-9-STABLE shortly, I made a post to the stable list, and he's aware of it.
For now, I hope to save someone a weekend of wasted time - If you're attempting to use a FreeBSD 9.0 system with an Intel x520 (ix/ixgbe driver), and you want to setup a LACP link, it won't work.
Here's what you'll see:
lagg1: flags=8843
options=401bb
ether d4:ae:52:5c:5d:af
inet 172.16.0.30 netmask 0xffffff00 broadcast 172.16.0.255
inet6 fe80::d6ae:52ff:fe5c:5daf%lagg1 prefixlen 64 scopeid 0xe
nd6 options=29
media: Ethernet autoselect
status: active
laggproto lacp
laggport: ix1 flags=18 COLLECTING,DISTRIBUTING
laggport: ix0 flags=18 COLLECTING,DISTRIBUTING
You'll notice it's just COLLECTING, DISTRIBUTING, where a properly functioning link should be ACTIVE, COLLECTING, DISTRIBUTING.
Check your driver version like so:
#sysctl dev.ix
dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.11
dev.ix.0.%driver: ix
dev.ix.0.%location: slot=0 function=0
dev.ix.0.%pnpinfo: vendor=0x8086 device=0x10f8 subvendor=0x8086 subdevice=0x000c class=0x020000
Older drivers seem to work on slightly older FreeBSD 9-Beta systems (like ixgbe 2.3.10).
The eventual fix was to download the latest FreeBSD driver from Intel's website (v2.4.4) compile it as a module, and then add it to my /boot/loader.conf file. Note, you can rename the produced ixgbe to to ixgbe2.4.4.ko before you copy it to /boot/kernel/, allowing you more control over which version of ixgbe you load. My kernel had ixgbe compiled in, but it still accepts the new 2.4.4 loaded via loader.conf:
ixgbe2.4.4_load="YES"
If you try loading a .ko over your existing ixgbe built into the kernel, make sure you check your driver version with a "sysctl dev.ix" command - I found that an older FreeBSD-9.0-RELEASE wouldn't load my new 2.4.4, but systems closer to the build time of my test FreeBSD-9-STABLE machine would.
This brings up questions of if we really want a lot of device drivers in the kernel, as it can make troubleshooting harder, but that's another post as well.
You'll have to do a quick edit of the code to get it to compile,as it's complaining about the bool typedef. Basically you need to remove the reference, and change the 4-6 instances of boolean_t to bool (you'll see the errors).
If that's too complex, hopefully Jack will have 2.4.4 ported to FreeBSD-9-STABLE shortly, I made a post to the stable list, and he's aware of it.
Hi Chistopher, Nice post. I'm trying to get some ideas on a problem I have with the intel driver on freebsd 10 when used with enhanced networking(Intel SR-IOV) on an AWS EC2 instance.
ReplyDeleteYou details above are just what is was looking for. Ie. How to get driver version and override the kernel one of you compile it youself.
Hopefully if I can get it to compile and load. Then it will function (pass traffic). Currently it detects the e intel card and shows a carrier as active. But no traffic
Hi Jarrod,
ReplyDeleteI find the 10gb cads to be a bit tricky with FreeBSD. Sometimes you need to start with an 'up' command - as in 'ifconfig ix0 up' then populate with IP info. If they are not in 'up' state, they won't work.
If you're using FreeBSD 10, you shouldn't have issues. It may be a good idea to try a boot into Windows 7/8 and see if it works there - I find Windows handles the card automatically with less fuss. Then once you've confirmed no hardware/cabling problems, you can try FreeBSD.
Good Luck!