[Bro] Logger Child Memory Leak (logger crashing often)

[Azoff, Justin S]

> On Jan 13, 2017, at 2:15 PM, Ryan Leonard <rleonar7 at uoregon.edu> wrote:
> 
> Hey All,
>  
> Running Bro 2.5 on a single server with 20 cores and some 240 GB of memory. 
> node.cfg specifies 14 workers, 2 proxies, 1 manager and a 1 logger process. 
> We are running a custom build of bro built with tmalloc enabled and pfring enabled.
>  
> I’m working to get my bro cluster stable. As it stand, often the logger process will crash causing us to lose a period of log files. Looking at the output of broctl top, it seems that the system is likely killing the bro logger process when it sees the amount of memory resources it is consuming.
>  
> ==== stderr.log
> listening on p5p2
>  
> 1484325490.230681 received termination signal
>  
> # broctl top
> Name         Type    Host             Pid     Proc    VSize  Rss  Cpu   Cmd
> logger       logger  localhost        47880   parent    4G     3G  82%  bro
> logger       logger  localhost        47902   child    38G    37G  13%  bro
> 

Most likely this isn't a leak, but that the logger process isn't able to process the data fast enough.

What model CPUs does this server have?  Can you show what this command outputs after bro has been running for a bit:

    top -b -n 1 -H -o TIME | fgrep bro: | head -n 20

The last column will be truncated, don't worry about that.



-- 
- Justin Azoff