Netperf and SMP - Oddness (Part 2 of 2)

To continue my previous post.. http://christopher-technicalmusings.blogspot.com/2010/12/netperf-and-smp-oddness.html

I was concerned that the poor performance of netperf when it was run on a multi-processor system was due to SMP overhead.

The only way to know for sure would be to run the test with netperf and netserver on separate (but identical) machines. That way I could ensure data was transferring across the network cable, and not being accelerated by any buffer-copy process within the processors or system.

I once again setup my netperf and netserver test, using FreeBSD 8.1 AMD64 on two Dell PowerEge 1950's.

In one set of tests, I restricted both ends to use a single CPU. In the other I let the system choose what CPU to run on (which results in the process jumping CPU a number of times as it executes the threads on the most opportunistic CPU - as noted with top -P).


The command was;

netperf -H 192.168.88.1 -L 192.168.88.2 -t TCP_STREAM -l 300 -f  m

The results show no difference between multi-processor and single processor results. In either test, I saw 941.49 Megabits/sec, regardless of single or multiple CPU's.

Which means that there are some serious optimizations for working between 2 network cards in the same physical machine, as seen in my previous tests. The same tests run from one internal NIC to the other result in 4500 - 9000 MBits/sec depending on if it's all through one CPU or across multiple CPU's.

I obviously need to spend more time understanding this, as it would make a large difference when you're considering a firewall application, or anything that needs to move a lot of data between interfaces.

I welcome any insights..