[Bro] Issue with Bro reporting dropped packets

[Espresso Beanies]

Hi Drew,

I definitely did. I tried asking earlier if there was a difference between
adding more Bro workers via [worker-1],[worker-2],[worker-3],etc... vs
lb_procs 'N' but didn't receive a response. I tried both methods in the
node.cfg file with little to no noticeable performance impact. I'm
definitely using CPU affinity.

Yep, I tried disabling as many NIC offloading features as I could and I'd
like to mention I found a more comprehensive list of NIC offload disabling
for Suricata that may be applied to Bro as well:
http://pevma.blogspot.com/2014/03/suricata-prepearing-10gbps-network.html

Right now I'm working with ntop on an issue with PF_RING because their repo
and rpm packages are not correctly loading the pf_ring module into the
kernel and I get errors when attempting to run this following command to
validate my PF_RING install after a successful Bro installation:

bro -N Bro::PF_RING

I also have an issue where the pf_ring repo packages are interfering with a
Bro reinstall because Bro no longer recognizes the libpcap library. Still
working that one out with ntop.


> Have you reviewed everything located here? :
>
> https://www.bro.org/documentation/faq.html#how-can-i-reduce-
> the-amount-of-captureloss-or-dropped-packets-notices
>
> Specifically a few things come to mind...I know you mentioned NIC settings
> but are you sure you disabled all the NIC offloading features using
> ethtool?, more detail on that at this link:
>
> http://securityonion.blogspot.com/2011/10/when-is-full-
> packet-capture-not-full.html
>
> Also, wouldn't hurt to double check the the pf_ring kernel module is
> loaded/loading staying loaded?  If you patch the server and the kernel gets
> updated unless you have something automated to reload/reinstall the pf_ring
> module you will probably need to reload the pf_ring module for the new
> kernel...
>
> Also, did you configure the number of ring slots for PF_RING ?
>
> Check to be sure that /etc/modprobe.d/pf_ring.conf exists for your PF_RING
> installation...this is where you will configure the number of ring slots
> for PF_RING, the default is 4096 I believe but on busy networks this needs
> to be increased as appropriate (in increments of 4096)...the max value is
> 65534.  I would try that if you've tried everything else at the first link
> above to no avail...
>
> This is also a great resource re: PF_RING and number of ring slots:
>
> https://groups.google.com/forum/#!topic/security-onion/zu7U7U9pBT8
>
> Hope this helps,
>
> -Drew
>
> On Tue, Feb 28, 2017 at 3:20 PM, Espresso Beanies <
> espressobeanies at gmail.com> wrote:
>
>> Hi,
>>
>> I'm trying to troubleshoot a Bro IDS that is experiencing capture loss
>> with dropped packets. The machine I'm using has a 16-core Intel Xeon
>> processor, 96Gb RAM, and an Intel NIC. I have 3 Bro workers with CPU
>> affinity enabled and I'm using the pf_ring module on CentOS with no custom
>> Bro scripts running. All of my processors are running at 99% utilization.
>>
>> According to my operating system, I'm dropping about 8000 packets over
>> the course of a day on a 300-400Mbps network. According to Bro capstats, I
>> am dropping about the same number of packets I'm receiving, sometimes more
>> than I receive. My capture_loss.log shows my workers lose about 30-50%
>> packets and my manager and proxy, 70-90%. I can provide any configurations
>> or screenshots if necessary.
>>
>> I'm trying to troubleshoot where the issue lies. I initially installed
>> Bro with all the recommended packages (tcmalloc, etc...) and the pf_ring
>> module and I can see that Bro is using it. At this point, everything I see
>> is pointing to an application issue and I'm running Bro version 2.5. I had
>> the same issue with Bro v.2.4 as well.
>>
>> Short of tweaking OS kernel and NIC card settings, I'm not sure where
>> else I could try to reduce my packet drop count in Bro. Any recommendations?
>>
>> Thanks,
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170302/dbe70c9d/attachment.html