[Bro] BRO Logger crashing due to large DNS log files

Azoff, Justin S jazoff at illinois.edu
Tue Aug 21 13:43:27 PDT 2018


> On Aug 21, 2018, at 4:22 PM, Ron McClellan <Ron_McClellan at ao.uscourts.gov> wrote:
> 
> Thanks Justin, here is the info.
> 
> Ron

ok so there's a lot of stuff wrong here, but it's all fixable.

> 
> 
> ######################Memory CPUI################################################################
> 
> [root@ ron]# free
>              total        used        free      shared  buff/cache   available
> Mem:      263620592   241469748      861800        4464    21289044    21038584
> Swap:       4194300       80508     4113792
> 
> [root@ current]# cat /proc/cpuinfo |grep 'model name'|sort|uniq  -c
>     72 model name      : Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
> 
> [worker-1]
> type=worker
> host=localhost
> interface=ens1f0
> lb_method=pf_ring
> lb_procs=48
> pin_cpus=12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60


This is an absolute beast of a CPU, but you only have 36 real cores.  I would run something like 30 workers, not 48.
You also have to ensure you don't pin workers to 'fake' hyperthreading cores.  Depending on how the CPU gets
enumerated you either need to use something like 6 through 35, or every other cpu.  6-35 is probably right for your CPU though.
This can be verified using hwloc-ps in the hwloc package.. I want to add this to bro-doctor but it's.. tricky :-)

You may also have 2 NUMA nodes in that case you would probably want to use 15 workers on each NUMA node, which would be something like
3-18 and 21-36... very tricky.  Ideally broctl could just support pin_cpus=auto as well :-)

> 
> ######################Perl Script OutputI################################################################
> 
> lag.value 500.000000

This isn't great.. but about what would be expected based on running out of ram.

> 
> [root@ ron]# broctl doctor.bro
> #################################################################
> # Checking if many recent connections have a SAD or had history #
> #################################################################
> error: No conn log files in the past day???

If you disabled the conn.log then most of the checks that bro-doctor does can't be ran :-(

> 
> ###############################################################################
> # Checking if bro is linked against a custom malloc like tcmalloc or jemalloc #
> ###############################################################################
> error: configured to use a custom malloc=False
> 

I would install gperftools-devel and rebuild bro


> ##################################
> # Checking pf_ring configuration #
> ##################################
> error: bro binary on node worker-1-1 is neither linked against pf_ring libpcap or using the bro pf_ring plugin
> error: bro binary on node worker-1-2 is neither linked against pf_ring libpcap or using the bro pf_ring plugin
> ..
> error: configured to use pf_ring=True pcap=False plugin=False

This is super super bad!  This means that every bro processes is seeing all of your traffic.. all 48 of them!

You can fix pf_ring, but on a recent Centos7 system AF_Packet works great out of the box, so I would try using that first
instead.  You just need to

bro-pkg install j-gras/bro-af_packet-plugin

and then in your node.cfg use

[worker-1]
type=worker
host=localhost
interface=af_packet::ens1f0
lb_method=custom
lb_procs=30
pin_cpus=6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35
af_packet_fanout_id=22
af_packet_fanout_mode=AF_Packet::FANOUT_HASH

— 
Justin Azoff



More information about the Bro mailing list