[Bro] Bro cluster requirements and manager logging backlog bug

[Hovsep Levi]

Back from lunch and the cluster is essentially crashed.


@ Tue Dec 20 19:03:48 UTC 2016

[bro at mgr /opt/bro]$ bin/broctl top manager logger
Name         Type    Host             Pid     Proc    VSize  Rss  Cpu   Cmd
logger       logger  169.232.234.36   18832   parent   23G    22G 171%  bro
logger       logger  169.232.234.36   18874   child    11G    10G  58%  bro
manager      manager 169.232.234.36   18947   child   510M   255M  54%  bro
manager      manager 169.232.234.36   18905   parent   23G     2G  19%  bro

[bro at mgr /opt/bro_data/logs/current]$ du -ms;cat *|wc -l;sleep 60;du
-ms;cat *|wc -l
1096    .
 17483890
1393    .
 20633538

It's about ~55K EPS, but probably more since logs are buffering in memory.



@ Tue Dec 20 20:12:47 UTC 2016

[bro at mgr /opt/bro]$ bin/broctl top manager logger
Name         Type    Host             Pid     Proc    VSize  Rss  Cpu   Cmd
logger       logger  169.232.234.36   18832   parent   67G    20G   0%  bro
logger       logger  169.232.234.36   18874   child    44G    24G   0%  bro
manager      manager 169.232.234.36   18947   child   510M   249M  99%  bro
manager      manager 169.232.234.36   18905   parent   18G   798M   0%  bro


last pid: 42312;  load averages:  4.50,  5.40, 20.50
                                                                        up
0+02:27:46  20:12:58
63 processes:  5 running, 53 sleeping, 1 zombie, 4 waiting
CPU:  2.3% user,  0.2% nice, 13.7% system,  0.0% interrupt, 83.8% idle
Mem: 104G Active, 3328M Inact, 17G Wired, 340M Cache, 180M Free
ARC: 15G Total, 7820M MFU, 7921M MRU, 16K Anon, 46M Header, 44M Other
Swap: 12G Total, 12G Used, K Free, 100% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU
COMMAND
18947 bro           1 108    5   510M   249M CPU40  40  55:54 100.00% bro
18832 bro          36  20    0 69454M 21145M uwait  44 129:19   0.00% bro
18874 bro           1  52    5 45265M 25543M pfault  1  42:47   0.00% bro
18905 bro           7  20    0 19346M   798M uwait  13  16:27   0.00% bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161220/ca182891/attachment.html