[Bro] BRO Logger crashing due to large DNS log files

[Ron McClellan]

Justin,

	The first 5 lines are consistent, the last 2 lines the first time seen were today.  Crash report wasn't very useful (see below), diag was pretty much the same.  Hopefully the OOM message helps.

Ron


Aug 21 09:45:18 aosoc kernel: Out of memory: Kill process 6610 (bro) score 507 or sacrifice child
Aug 21 09:45:18 aosoc kernel: Killed process 6610 (bro) total-vm:139995144kB, anon-rss:137467264kB, file-rss:0kB, shmem-rss:0kB
Aug 21 11:32:23 aosoc kernel: bro invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Aug 21 11:32:23 aosoc kernel: bro cpuset=/ mems_allowed=0-1
Aug 21 11:32:23 aosoc kernel: CPU: 57 PID: 21655 Comm: bro Kdump: loaded Not tainted 3.10.0-862.11.6.el7.x86_64 #1
Aug 21 11:32:23 aosoc kernel: Out of memory: Kill process 20158 (bro) score 544 or sacrifice child
Aug 21 11:32:23 aosoc kernel: Killed process 20158 (bro) total-vm:150275592kB, anon-rss:147621508kB, file-rss:0kB, shmem-rss:0kB



===============Crash Report===================

This crash report does not include a backtrace.  In order for crash reports to be useful when Bro crashes, a backtrace is needed.

No core file found and gdb is not installed.  It is recommended to install gdb so that BroControl can output a backtrace if Bro crashes.

Bro 2.5.4
Linux 3.10.0-862.11.6.el7.x86_64

Bro plugins: (none found)

==== No reporter.log

==== stderr.log
received termination signal

==== stdout.log
max memory size         (kbytes, -m) unlimited
data seg size           (kbytes, -d) unlimited
virtual memory          (kbytes, -v) unlimited
core file size          (blocks, -c) unlimited

==== .cmdline
-U .status -p broctl -p broctl-live -p local -p logger local.bro broctl base/frameworks/cluster local-logger.bro broctl/auto

==== .env_vars
PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/bro/bin:/home/ron/.local/bin:/home/ron/bin:/usr/local/bro/bin
BROPATH=/logs/bro/spool/installed-scripts-do-not-touch/site::/logs/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site
CLUSTER_NODE=logger

==== .status
TERMINATED [atexit]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log

--
[Automatically generated.]



-----Original Message-----
From: Azoff, Justin S <jazoff at illinois.edu> 
Sent: Tuesday, August 21, 2018 2:36 PM
To: Ron McClellan <Ron_McClellan at ao.uscourts.gov>
Cc: bro at bro.org
Subject: Re: [Bro] BRO Logger crashing due to large DNS log files


> On Aug 21, 2018, at 8:21 AM, Ron McClellan <Ron_McClellan at ao.uscourts.gov> wrote:
> 
> Justin,
> 
> 	Nothing really on the system that would be killing logger, system is a base CENTOS 7 box, recently built just for BRO.   The .status file shows "TERMINATED[atexit]".  
> 
> Ron
> 
> [root@ ron]# sudo cat /logs/bro/spool/logger/.status TERMINATED 
> [atexit]
> 
> Name         Type    Host             Status    Pid    Started
> logger       logger  localhost        crashed

Ah.. well now that says 'crashed' which is what you'd expect if it was crashing (not 'terminating')

If it is crashing then something should say why...

Is broctl sending you a crash report when that happens? What does broctl diag say?

Are there any kernel OOM messages in dmesg or syslog?

Or any messages that look like

bro[60506]: segfault at 0 ip 00000000005fcf8d sp 00007fffaf9d2f40 error 6 in bro[400000+624000]

—
Justin Azoff