[Bro] Logger Child Memory Leak (logger crashing often)
Azoff, Justin S
jazoff at illinois.edu
Fri Jan 13 11:36:11 PST 2017
> On Jan 13, 2017, at 2:15 PM, Ryan Leonard <rleonar7 at uoregon.edu> wrote:
> Hey All,
> Running Bro 2.5 on a single server with 20 cores and some 240 GB of memory.
> node.cfg specifies 14 workers, 2 proxies, 1 manager and a 1 logger process.
> We are running a custom build of bro built with tmalloc enabled and pfring enabled.
> I’m working to get my bro cluster stable. As it stand, often the logger process will crash causing us to lose a period of log files. Looking at the output of broctl top, it seems that the system is likely killing the bro logger process when it sees the amount of memory resources it is consuming.
> ==== stderr.log
> listening on p5p2
> 1484325490.230681 received termination signal
> # broctl top
> Name Type Host Pid Proc VSize Rss Cpu Cmd
> logger logger localhost 47880 parent 4G 3G 82% bro
> logger logger localhost 47902 child 38G 37G 13% bro
Most likely this isn't a leak, but that the logger process isn't able to process the data fast enough.
What model CPUs does this server have? Can you show what this command outputs after bro has been running for a bit:
top -b -n 1 -H -o TIME | fgrep bro: | head -n 20
The last column will be truncated, don't worry about that.
- Justin Azoff
More information about the Bro