[Bro] Worker System Memory Exhaustion

Fri Apr 6 10:02:02 PDT 2018

I endorse commenting out "@load misc/scan", that really stabilized memory usage in our environment. We also commented out "@load misc/detect-traceroute" and that seemed to help as well. 

>From your description I don't know if this applies to your situation, but for what it's worth we use PF_RING and experienced high resource utilization initially due to PF_RING not loading correctly. The issue tracker for it is here: https://bro-tracker.atlassian.net/browse/BIT-1864 - TL;DR, change pfringclusterid in broctl.cfg to 21; if pfringclusterid is set to 0 PF_RING doesn't actually do anything even though everything will indicate it's running. Again, don't know if it applies but thought I'd throw it out there. :)

Thanks,
Carl
University of Idaho

-----Original Message-----
From: bro-bounces at bro.org <bro-bounces at bro.org> On Behalf Of Azoff, Justin S
Sent: Friday, April 6, 2018 05:25
To: Greg Grasmehr <greg.grasmehr at caltech.edu>
Cc: bro at bro.org
Subject: Re: [Bro] Worker System Memory Exhaustion

> On Apr 5, 2018, at 7:36 PM, Greg Grasmehr <greg.grasmehr at caltech.edu> wrote:
> 
> Hello List,
> 
> We're running Bro version 2.5-467 and shunting connections in an 
> Arista
> 5100 via the excellent Dumbno.  The system is essentially a 
> Bro-in-a-box with 251G RAM and a Myricom SNF+ card.  Bro spawns 32 
> workers to monitor a 10G link that averages roughly 5G and spikes to 7 
> on occasion.  Logs are rotated every 1/2 hour.

Cool :-)

> The current configuration has been running for about 2 weeks and thus 
> far the only problem I have encountered is that given enough time, the 
> workers will eventually exhaust all of the available system memory 
> including scratch.

Less cool :-(

> Currently I am running a watcher process which slowly restarts the 
> workers once available memory is < 4G and this has solved the problem, 
> but this seems an imperfect solution thus I am wondering if anyone 
> knows the source of the apparent memory leak?  Loaded scripts are 
> attached and thanks in advance for any informative illumination on this issue.

try commenting out

@load misc/scan

from local.bro.

If you have a lot of address space and bro is before any firewall as opposed to after it, this is likely the source of the problems.

If that fixes it there are other scan detection implementations that are a bit more efficient.

—
Justin Azoff

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro