[Bro] BRO Logger crashing due to large DNS log files

[Azoff, Justin S]

> On Aug 21, 2018, at 2:53 PM, Ron McClellan <Ron_McClellan at ao.uscourts.gov> wrote:
> 
> Justin,
> 
> 	The first 5 lines are consistent, the last 2 lines the first time seen were today.  Crash report wasn't very useful (see below), diag was pretty much the same.  Hopefully the OOM message helps.
> 
> Ron
> 
> 
> Aug 21 09:45:18 aosoc kernel: Out of memory: Kill process 6610 (bro) score 507 or sacrifice child
> Aug 21 09:45:18 aosoc kernel: Killed process 6610 (bro) total-vm:139995144kB, anon-rss:137467264kB, file-rss:0kB, shmem-rss:0kB
> Aug 21 11:32:23 aosoc kernel: bro invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
> Aug 21 11:32:23 aosoc kernel: bro cpuset=/ mems_allowed=0-1
> Aug 21 11:32:23 aosoc kernel: CPU: 57 PID: 21655 Comm: bro Kdump: loaded Not tainted 3.10.0-862.11.6.el7.x86_64 #1
> Aug 21 11:32:23 aosoc kernel: Out of memory: Kill process 20158 (bro) score 544 or sacrifice child
> Aug 21 11:32:23 aosoc kernel: Killed process 20158 (bro) total-vm:150275592kB, anon-rss:147621508kB, file-rss:0kB, shmem-rss:0kB

Ah, this is great.. well, not great in that it is crashing but great in that now we know what is wrong: You're running out of ram.

So you said you had 256GB, which should normally be more than enough as long as everything is working properly, but I have a feeling some things are not working quite right though.

Have you had a chance to run that python program I posted?  If you have a high amount of log lag, something is not keeping up well.

Do you have any graphs of memory usage on your host?

What exactly does this output:

$ cat /proc/cpuinfo |grep 'model name'|sort|uniq  -c
     40 model name      : Intel(R) Xeon(R) CPU E5-2470 v2 @ 2.40GHz


The fact that you are seeing 

34264380 dns_unmatched_msg
16696030 dns_unmatched_reply
 62288 possible_split_routing
 59512 data_before_established

in your weird.log points to something being very very wrong with your traffic.  This can cause bro to work many times harder than it needs to.

How is your load balancing setup in your node.cfg?

Can you try running bro-doctor from bro-pkg: https://packages.bro.org/packages/view/74d45e8c-4fb7-11e8-88be-0a645a3f3086

If you can't run bro-pkg you just need to grab 

https://raw.githubusercontent.com/ncsa/bro-doctor/master/doctor.py

and drop it in

/usr/local/bro/lib/broctl/plugins/doctor.py

and run broctl doctor.bro



— 
Justin Azoff