[Bro] Worker System Memory Exhaustion

Greg Grasmehr greg.grasmehr at caltech.edu
Fri Apr 6 11:03:04 PDT 2018


Thanks Carl, will consider this avenue too.

Greg

On 04/06/18 17:02:02, Pearson, Carl (cpearson at uidaho.edu) wrote:
> I endorse commenting out "@load misc/scan", that really stabilized memory usage in our environment. We also commented out "@load misc/detect-traceroute" and that seemed to help as well. 
> 
> From your description I don't know if this applies to your situation, but for what it's worth we use PF_RING and experienced high resource utilization initially due to PF_RING not loading correctly. The issue tracker for it is here: https://bro-tracker.atlassian.net/browse/BIT-1864 - TL;DR, change pfringclusterid in broctl.cfg to 21; if pfringclusterid is set to 0 PF_RING doesn't actually do anything even though everything will indicate it's running. Again, don't know if it applies but thought I'd throw it out there. :)
> 
> Thanks,
> Carl
> University of Idaho
> 
> -----Original Message-----
> From: bro-bounces at bro.org <bro-bounces at bro.org> On Behalf Of Azoff, Justin S
> Sent: Friday, April 6, 2018 05:25
> To: Greg Grasmehr <greg.grasmehr at caltech.edu>
> Cc: bro at bro.org
> Subject: Re: [Bro] Worker System Memory Exhaustion
> 
> 
> > On Apr 5, 2018, at 7:36 PM, Greg Grasmehr <greg.grasmehr at caltech.edu> wrote:
> > 
> > Hello List,
> > 
> > We're running Bro version 2.5-467 and shunting connections in an 
> > Arista
> > 5100 via the excellent Dumbno.  The system is essentially a 
> > Bro-in-a-box with 251G RAM and a Myricom SNF+ card.  Bro spawns 32 
> > workers to monitor a 10G link that averages roughly 5G and spikes to 7 
> > on occasion.  Logs are rotated every 1/2 hour.
> 
> Cool :-)
> 
> > The current configuration has been running for about 2 weeks and thus 
> > far the only problem I have encountered is that given enough time, the 
> > workers will eventually exhaust all of the available system memory 
> > including scratch.
> 
> Less cool :-(
> 
> > Currently I am running a watcher process which slowly restarts the 
> > workers once available memory is < 4G and this has solved the problem, 
> > but this seems an imperfect solution thus I am wondering if anyone 
> > knows the source of the apparent memory leak?  Loaded scripts are 
> > attached and thanks in advance for any informative illumination on this issue.
> 
> try commenting out
> 
> @load misc/scan
> 
> from local.bro.
> 
> If you have a lot of address space and bro is before any firewall as opposed to after it, this is likely the source of the problems.
> 
> If that fixes it there are other scan detection implementations that are a bit more efficient.
> 
>> Justin Azoff
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


More information about the Bro mailing list