[Bro] BRO Logger crashing due to large DNS log files

Ron McClellan Ron_McClellan at ao.uscourts.gov
Tue Aug 21 05:21:39 PDT 2018


Justin,

	Nothing really on the system that would be killing logger, system is a base CENTOS 7 box, recently built just for BRO.   The .status file shows "TERMINATED[atexit]".  

Ron

[root@ ron]# sudo cat /logs/bro/spool/logger/.status
TERMINATED [atexit]

Name         Type    Host             Status    Pid    Started
logger       logger  localhost        crashed
manager      manager localhost        running   55680  20 Aug 15:25:47
proxy-1      proxy   localhost        running   55719  20 Aug 15:25:49
worker-1-1   worker  localhost        running   55893  20 Aug 15:25:50
worker-1-2   worker  localhost        running   55897  20 Aug 15:25:50
worker-1-40  worker  localhost        running   56411  20 Aug 15:25:50
worker-1-41  worker  localhost        crashed
worker-1-42  worker  localhost        running   56444  20 Aug 15:25:50
worker-1-43  worker  localhost        running   56446  20 Aug 15:25:50

-----Original Message-----
From: Azoff, Justin S <jazoff at illinois.edu> 
Sent: Monday, August 20, 2018 9:08 PM
To: Ron McClellan <Ron_McClellan at ao.uscourts.gov>
Cc: bro at bro.org
Subject: Re: [Bro] BRO Logger crashing due to large DNS log files


> On Aug 20, 2018, at 2:04 PM, Ron McClellan <Ron_McClellan at ao.uscourts.gov> wrote:
> 
> Update:
> 
> 	Worked for almost 3 hours, but then started failing again.  I even changed the log rotation to every 15 minutes and it still crashes .    Any other sugestions?  Has anyone ever tried to configured syslog-ng to handle the logging?
> 
> 
> Warning: broctl config has changed (run the broctl "deploy" command)
> Name         Type    Host             Status    Pid    Started
> logger       logger  localhost        terminating 28295  20 Aug 12:30:03
> manager      manager localhost        running   28336  20 Aug 12:30:05
> proxy-1      proxy   localhost        running   28375  20 Aug 12:30:06
> worker-1-1   worker  localhost        running   28565  20 Aug 12:30:08
> 

That's really interesting.. because "terminating" means something very specific, and not something you would see if it was crashing.

Unfortunately broctl throws away the 2nd part of the status file that would narrow that down further, but there are only a few reasons:

src/main.cc
275:	set_processing_status("TERMINATING", "done_with_network");
331:	set_processing_status("TERMINATING", "terminate_bro");
392:	set_processing_status("TERMINATING", "termination_signal");
413:	set_processing_status("TERMINATING", "sig_handler");

src/Net.cc
432:	set_processing_status("TERMINATING", "net_finish");
457:	set_processing_status("TERMINATING", "net_delete"); 

done_with_network, net_finish, and net_delete wouldn't apply to a logger node that has no network interfaces.

termination_signal and sig_handler happen when bro gets a SIGINT or SIGTERM, and terminate_bro happens when bro exits normally.

If it does happen again and stays like that if you could run

$ sudo cat /usr/local/bro/spool/logger/.status
RUNNING [net_run]

that should show

TERMINATING [one of those reasons]

which would definitively narrow it down.

Is there anything on your system that would be killing bro?  If it were the kernel OOM killer I'd expect that to show up as crashed and not terminating.

—
Justin Azoff




More information about the Bro mailing list