[Bro-Dev] snaplen and drops

[Lothar Braun]

Hi,

On Oct 27, 2011, at 6:15 PM, Martin Holste wrote:

> That's very weird.  It's abnormal to get packets > 1514, right?  Are
> you monitoring a link with a lot of jumbo packets?

On Linux you can observe packets > 1514 bytes, even if you monitor a link that does not carry a single jumbo frame. 

You can have large packets if your NIC supports RSC (Receive Side Coalescing). RSC is implemented in some network cards (e.g. Intel 10GE with a 82599 chipset), and reassembles subsequent TCP segments into larger packets in order to reduce the number of packets that need to be handled by the kernel.

Even if your network card does not implement RSC, you might also see large packets due to LRO/GRO (Large Receive Offload / Generic Receive Offload) done in software (more information: http://lwn.net/Articles/358910/).

However, this needs to be supported by your NIC driver and enabled via ethtool.

ethtool -k <dev>

will show you if you have LRO or GRO enabled. 

>> On a reasonable fast Linux box seeing (currently) <10M/bps, I'm
>> getting lots packet drops with current master, even though CPU is very
>> low. I did the usual sysctl tuning, but that didn't help. Then I
>> reduced the snaplen (which now defaults to 65K) down to 8K, and the
>> drops disappeared.

Which kernel and libpcap version do you use? 

Todays' Linux kernels support memory mapping for packet exchange between userland and kernel. Since version 1.0.0, libpcap uses this feature by default:

libpcap requests a two megabyte sized shared buffer (default size) from the kernel. The snaplen is passed to the kernel which will align captured packets according to the snaplen. The snaplen is therefore the only parameter that decides how many  packets fit into the buffer between kernel and application.

If you configure a snaplen of 64 KB, you will have space for only 32 packets in your buffer (2 MB / 64 KB).

You could try to make libpcap allocate a bigger buffer with pcap_set_buffer_size(). However, this must be called before pcap_activate(), which means that you cannot use pcap_open_live() but have to call pcap_create, pcap_set_snaplen, pcap_set_timeout, pcap_activate yourself..

Best regards,
  Lothar  

--
Lothar Braun
Chair for Network Architectures and Services (I8)
Department of Informatics
Technische Universität München
Boltzmannstr. 3, 85748 Garching bei München, Germany
Phone:  +49 89 289-18010       Fax: +49 89 289-18033
E-mail: braun at net.in.tum.de