Intel X520 DA 10 Gbe Network Card and FreeBSD 8.2-PRE and 9.0-Current

I love my Intel X520 DA. It's two SPF+ 10 Gbe Network ports on one PCIe 4x card. 

Out of the box, it works great with FreeBSD 8.1, even if you ignore Intel's optimization advice.

Currently for $380 a card (Retail close to $500), it's part of my Ghetto-SAN foundation. No need for an expensive 10 Gbe switch when you can afford to put two of these in each machine and make a simple point-to-point SAN network.

However, under FreeBSD 8.2-PRE and 9.0-CURRENT (both are not official releases yet) it won't work properly if you set the card to a MTU of 9000 or higher.

The reason is that it runs out of buffers to handle traffic of that size.

The quick fix is to put these settings into your /etc/sysctl.conf

hw.intr_storm_threshold=9000
kern.ipc.nmbclusters=262144
kern.ipc.nmbjumbop=262144

kern.ipc.nmbjumbo9=48000

The kicker is the last line - nmbjumbo9 - By default FreeBSD allocates 6400, which isn't enough to handle what this card can produce. This isn't some little 100 Mbit card.. that's 10,000 Mbit that it needs to push. Expect to require deep buffers.

Remember to check your netstat -m output to see if you are close to exhaustion with nmbjumbo9 at 48000 - Mine was using ~36000 after some load testing.

Here's some additional information on settings for the card, and drivers, etc.

http://www.intel.com/support/network/sb/CS-025829.htm

http://downloadcenter.intel.com/SearchResult.aspx?lang=eng&ProductFamily=Network+Connectivity&ProductLine=Intel%C2%AE+Server+Adapters&ProductProduct=Intel%C2%AE+Ethernet+Server+Adapter+X520+Series