[Bro] High packet drop rates
Azoff, Justin S
jazoff at illinois.edu
Fri Jan 15 12:31:45 PST 2016
> On Jan 15, 2016, at 3:14 PM, Donaldson, John <donaldson8 at llnl.gov> wrote:
> I've recently run into a problem, with Bro 2.4.1, where I have extremely high packet loss.
> I'm running on a server with dual quad-core Xeon processors, with no hyperthreading, and 64GB of RAM, monitoring a few small links, averaging, in aggregate, about 100Mbps of traffic. There isn't too much else running on this system, but we're seeing drop rates that average in the 70-99% range (via the capture-loss.bro policy), even though Bro's CPU utilization sits at around 20-30%.
A likely reason for this is the various NIC offloading features being enabled causing bro to not properly capture entire packets. Is your reporter.log complaining about this at all?
Can you try running
for i in rx tx sg tso ufo gso gro lro; do ethtool -K eth0 $i off; done
to disable all of those optional features (replace eth0 with appropriate interfaces)
Also, if you are using jumbo frames you may need to add to local.bro:
redef snaplen = 9000; #potentially as high as 9216
and ensure that the mtu on the nic is set appropriately as well.
Once you know you have capture loss it can be a good idea to look at the lower level records to see which connections are missing bytes:
cat conn.log | bro-cut id.orig_h id.resp_h id.resp_p orig_bytes resp_bytes missed_bytes
It's possible that all of your missed_bytes are coming from a small subset of hosts. We ran into that issue a while ago due to our MTU being just too small to properly capture traffic between 2 hosts. Since it was a large backup transfer our capture loss would shoot up to 75%+ even though it was only one flow being missed.
Since your traffic rate is relatively low one potential troubleshooting option is to dump the full traffic to a file using something like 'tcpdump -s 0 -w dump.cap -i eth0' and then run bro against that pcap file and see what that reports.
- Justin Azoff
More information about the Bro