[Bro-Dev] Broker port status

Azoff, Justin S jazoff at illinois.edu
Thu Mar 8 16:11:20 PST 2018

> On Mar 8, 2018, at 5:46 PM, Jon Siwek <jsiwek at corelight.com> wrote:
> On Thu, Mar 8, 2018 at 5:07 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:
>> One thing I notice is that the traffic to/from the manager box and cpu has increased quite a bit.
>> Ignoring the large CPU spikes from building the new branch just before the switch at 15:30, the overall cpu load on the manager box is about 3x higher.
>> The bandwidth isn't terribly excessive, but it increased from about 16mbit to 40mbit (stupid graph is in mebibytes).
>> Based on some capstats runs, it's all going to the logger port 47761.  I have another graph that shows the total log volume being written to disk and that hasn't changed.
>> So it looks like this branch uses 2x the cpu and bandwidth to write the same volume of logs.
> Interesting, I would have thought maybe the manager could be utilized
> more, though not the logger.  Will have to think about that.

Yeah.. it's definitely worker -> logger:

logger=47761, manager=47762, proxy=47763

[root at bro-test ~]# capstats -i enp3s0f0 -I 1 -f 'port 47761' -n 5
1520553009.488934 pkts=1173 kpps=1.1 kbytes=2837 mbps=22.6 nic_pkts=1175 nic_drops=0 u=0 t=1173 i=0 o=0 nonip=0
1520553010.489053 pkts=1280 kpps=1.3 kbytes=3218 mbps=26.4 nic_pkts=2455 nic_drops=0 u=0 t=1280 i=0 o=0 nonip=0
1520553011.489177 pkts=1086 kpps=1.1 kbytes=2912 mbps=23.9 nic_pkts=3541 nic_drops=0 u=0 t=1086 i=0 o=0 nonip=0
1520553012.489310 pkts=1382 kpps=1.4 kbytes=3757 mbps=30.8 nic_pkts=4923 nic_drops=0 u=0 t=1382 i=0 o=0 nonip=0
1520553013.489471 pkts=1563 kpps=1.6 kbytes=3816 mbps=31.3 nic_pkts=6486 nic_drops=0 u=0 t=1563 i=0 o=0 nonip=0
[root at bro-test ~]# capstats -i enp3s0f0 -I 1 -f 'port 47762' -n 5
1520553018.874154 pkts=346 kpps=0.3 kbytes=92 mbps=0.7 nic_pkts=348 nic_drops=0 u=0 t=346 i=0 o=0 nonip=0
1520553019.875111 pkts=1044 kpps=1.0 kbytes=283 mbps=2.3 nic_pkts=1391 nic_drops=0 u=0 t=1044 i=0 o=0 nonip=0
1520553020.875448 pkts=759 kpps=0.8 kbytes=199 mbps=1.6 nic_pkts=2150 nic_drops=0 u=0 t=759 i=0 o=0 nonip=0
1520553021.875575 pkts=654 kpps=0.7 kbytes=196 mbps=1.6 nic_pkts=2804 nic_drops=0 u=0 t=654 i=0 o=0 nonip=0
1520553022.875920 pkts=470 kpps=0.5 kbytes=123 mbps=1.0 nic_pkts=3274 nic_drops=0 u=0 t=470 i=0 o=0 nonip=0
[root at bro-test ~]# capstats -i enp3s0f0 -I 1 -f 'port 47763' -n 5
1520553025.700919 pkts=26 kpps=0.0 kbytes=8 mbps=0.1 nic_pkts=31 nic_drops=0 u=0 t=26 i=0 o=0 nonip=0
1520553026.701047 pkts=27 kpps=0.0 kbytes=9 mbps=0.1 nic_pkts=58 nic_drops=0 u=0 t=27 i=0 o=0 nonip=0
1520553027.701185 pkts=24 kpps=0.0 kbytes=8 mbps=0.1 nic_pkts=82 nic_drops=0 u=0 t=24 i=0 o=0 nonip=0
1520553028.701306 pkts=55 kpps=0.1 kbytes=18 mbps=0.2 nic_pkts=137 nic_drops=0 u=0 t=55 i=0 o=0 nonip=0
1520553029.701434 pkts=9 kpps=0.0 kbytes=3 mbps=0.0 nic_pkts=146 nic_drops=0 u=0 t=9 i=0 o=0 nonip=0
[root at bro-test ~]#

On another cluster seeing the same traffic with 56 workers running 2.5.3ish shows this for the logger port at the same time:

1520553725.875230 pkts=681 kpps=0.7 kbytes=1946 mbps=15.5 nic_pkts=768 nic_drops=0 u=0 t=681 i=0 o=0 nonip=0
1520553726.875374 pkts=508 kpps=0.5 kbytes=1605 mbps=13.1 nic_pkts=1392 nic_drops=0 u=0 t=508 i=0 o=0 nonip=0
1520553727.875520 pkts=554 kpps=0.6 kbytes=1681 mbps=13.8 nic_pkts=2247 nic_drops=0 u=0 t=554 i=0 o=0 nonip=0
1520553728.875713 pkts=536 kpps=0.5 kbytes=1708 mbps=14.0 nic_pkts=2902 nic_drops=0 u=0 t=536 i=0 o=0 nonip=0
1520553729.875845 pkts=518 kpps=0.5 kbytes=1669 mbps=13.7 nic_pkts=3641 nic_drops=0 u=0 t=518 i=0 o=0 nonip=0

[root at bro-test ~]# broctl top manager logger
Name         Type    Host             Pid     Proc    VSize  Rss  Cpu   Cmd
logger       logger  bro-test         2016    parent  938M   283M  81%  bro
manager      manager bro-test         2107    parent  619M   244M  25%  bro
[root at bro-test ~]#

[root at bro-dev ~]# broctl top manager logger
Name         Type    Host             Pid     Proc    VSize  Rss  Cpu   Cmd
logger       logger  bro-dev    10704   child   492M   150M   6%  bro
logger       logger  bro-dev    10571   parent    1G   221M   0%  bro
manager      manager bro-dev    10734   parent  909M   497M   0%  bro
manager      manager bro-dev    10782   child   482M   110M   0%  bro

cpus are about the same, test has X5650  @ 2.67GHz and dev has E5-2420 @ 1.90GHz 
according to passmark, the older X5650 has a slightly faster single core performance.

> An interesting experiment you could try is switching to an alternate
> implementation of the Known scripts.  E.g. stick this in local.bro:
> redef Known::use_host_store = F;
> redef Known::use_cert_store = F;
> redef Known::use_device_store = F;
> redef Known::use_service_store = F;

I tried doing that for an unrelated reason.. but I'm not sure if it works right.

You can't redef Known::use_service_store before the script is loaded, because otherwise you get

Can't document redef of Known::use_service_store, identifier lookup failed

but if you load the script first, the

@if ( Known::use_service_store )

block is already evaluated before you change the value.

[jazoff at bro-test ~]$ cat foo.bro
@load protocols/conn/known-services
redef Known::use_service_store=F;

event bro_init()
        print Known::services;
[jazoff at bro-test ~]$ bro foo.bro
error in ./foo.bro, line 6: unknown identifier Known::services, at or near "Known::services"

[jazoff at bro-test ~]$ cat foo.bro
redef Known::use_service_store=F;
@load protocols/conn/known-services

event bro_init()
        print Known::services;
[jazoff at bro-test ~]$ bro foo.bro
error in ./foo.bro, line 1: "redef" used but not previously defined (Known::use_service_store)
internal warning in ./foo.bro, line 1: Can't document redef of Known::use_service_store, identifier lookup failed
error in ./foo.bro, line 1 and /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 34: already defined (Known::use_service_store)
error in /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 40: value used but not set (Known::use_service_store)
error in /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 40: invalid expression in @if (Known::use_service_store)
error in /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 93: value used but not set (Known::use_service_store)
error in /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 93: invalid expression in @if (Known::use_service_store)

Justin Azoff

More information about the bro-dev mailing list