[Bro] Logging and memory leak

[Hovsep Levi]

Noticed this today and I believe it's related to a recent cluster crash.
The virtual memory size of the Bro manager continues to grow slowly (56G
right now) while the resident active memory remains around 4G.  To my
knowledge this suggests a memory leak.  The system is FreeBSD.

We are using a Kafka only based output with local file logging disabled and
a modified number of loggers.  I think a single logger works fine for the
most part but I used 4 here for testing after the recent cluster crash.

Regarding the crash, the cluster was running fine and then active memory
spiked sharply until reaching the ceiling and consuming swap.  I suspect it
was related to the VSIZE of the manager reaching a maximum of some sort and
triggering the failure.


Name         Type    Host             Pid     Proc    VSize  Rss  Cpu   Cmd
logger-1     logger  10.1.1.1   85959   parent  176M    67M   6%  bro
logger-1     logger  10.1.1.1   86115   child   194M    62M   0%  bro
logger-2     logger  10.1.1.1   85962   parent  709M   175M  19%  bro
logger-2     logger  10.1.1.1   86017   child   194M    65M   1%  bro
logger-3     logger  10.1.1.1   85965   parent  663M   164M  13%  bro
logger-3     logger  10.1.1.1   86114   child   202M    71M   0%  bro
logger-4     logger  10.1.1.1   85967   parent  663M   157M  17%  bro
logger-4     logger  10.1.1.1   86113   child   194M    63M   0%  bro
manager      manager 10.1.1.1   86204   child   878M   649M 100%  bro
manager      manager 10.1.1.1   86109   parent   56G     4G  16%  bro


last pid:  1482;  load averages:  4.73,  4.73,
4.58                                                          up
3+23:25:56  00:12:45
52 processes:  2 running, 50 sleeping
CPU:  3.4% user,  0.3% nice,  3.3% system,  0.0% interrupt, 92.9% idle
Mem: 538M Active, 6674M Inact, 17G Wired, 26M Cache, 101G Free
ARC: 14G Total, 2325M MFU, 11G MRU, 144K Anon, 57M Header, 548M Other
Swap: 12G Total, 17M Used, 12G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU
COMMAND
86204 bro           1 108    5   878M   649M CPU27  27  26.0H 100.00% bro
85962 bro         172  20    0   709M   175M select 25  22.4H  25.63% bro
86109 bro           7  20    0 57682M  4879M uwait  13 499:16  22.17% bro
85967 bro         162  20    0   663M   157M select 13  19.7H  20.80% bro
85965 bro         162  20    0   663M   164M select 23  18.2H  16.31% bro
85959 bro          21  20    0   176M 69352K select 40 193:32   6.30% bro
86017 bro           1  25    5   194M 66772K select 36  28:18   1.27% bro
86113 bro           1  25    5   194M 65068K select 42  23:28   0.98% bro
86114 bro           1  25    5   202M 73308K select 38  13:43   0.39% bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170201/f0737a5f/attachment-0001.html