[Bro-Dev] Broker port status

[Azoff, Justin S]

> On Mar 9, 2018, at 8:24 AM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> 
> Ah.. I grabbed the latest commit from the wrong checkout and was using a commit from a few weeks ago.
> 

Ok.. running the latest commit now (105fc386ef0f65a91839706641abae664c7f3e49)

Have noticed a problem where at the top of an hour, the logger runs into some issue while rotating the logs
and all logging stops but the logger buffers everything in memory until the box OOMs:

[root at bro-test ~]# broctl top manager logger proxy-{1,2,3}
Name         Type    Host             Pid     Proc    VSize  Rss  Cpu   Cmd
logger       logger  bro-test         14267   parent    5G     5G  93%  bro
manager      manager bro-test         15478   parent  649M   277M  25%  bro
proxy-1      proxy   bro-test         32061   parent  643M   309M   0%  bro
proxy-2      proxy   bro-test         32063   parent  642M   351M  12%  bro
proxy-3      proxy   bro-test         15732   parent  619M   322M  18%  bro
[root at bro-test ~]# cat /bro/logs/current/*.log|wc -l
2627
[root at bro-test ~]# broctl top manager logger proxy-{1,2,3}
Name         Type    Host             Pid     Proc    VSize  Rss  Cpu   Cmd
logger       logger  bro-test         14267   parent    6G     5G  66%  bro
manager      manager bro-test         15478   parent  649M   277M  33%  bro
proxy-1      proxy   bro-test         32061   parent  643M   309M  26%  bro
proxy-2      proxy   bro-test         32063   parent  642M   351M  20%  bro
proxy-3      proxy   bro-test         15732   parent  619M   322M  26%  bro
[root at bro-test ~]# cat /bro/logs/current/*.log|wc -l
2627
[root at bro-test ~]# broctl top manager logger proxy-{1,2,3}
Name         Type    Host             Pid     Proc    VSize  Rss  Cpu   Cmd
logger       logger  bro-test         14267   parent    7G     6G  75%  bro
manager      manager bro-test         15478   parent  649M   277M  25%  bro
proxy-1      proxy   bro-test         32061   parent  643M   309M  12%  bro
proxy-2      proxy   bro-test         32063   parent  642M   353M  25%  bro
proxy-3      proxy   bro-test         15732   parent  619M   322M  12%  bro
[root at bro-test ~]# cat /bro/logs/current/*.log|wc -l
2627
[root at bro-test ~]# broctl top manager logger proxy-{1,2,3}
Name         Type    Host             Pid     Proc    VSize  Rss  Cpu   Cmd
logger       logger  bro-test         14267   parent   12G    11G  68%  bro
manager      manager bro-test         15478   parent  649M   277M  31%  bro
proxy-1      proxy   bro-test         32061   parent  643M   313M  18%  bro
proxy-2      proxy   bro-test         32063   parent  642M   359M  12%  bro
proxy-3      proxy   bro-test         15732   parent  619M   322M   6%  bro
[root at bro-test ~]# ls -tl /bro/logs/current/|head
total 412
-rw-r--r--. 1 root root    682 Mar  9 11:00 stats.log
-rw-r--r--. 1 root root  14470 Mar  9 11:00 conn.log
[root at bro-test ~]# date
Fri Mar  9 11:06:59 CST 2018

up until log rotation is supposed to occur the log rate is normal.

— 
Justin Azoff