[Bro] BRO Logger crashing due to large DNS log files

Azoff, Justin S jazoff at illinois.edu
Mon Aug 20 18:08:11 PDT 2018


> On Aug 20, 2018, at 2:04 PM, Ron McClellan <Ron_McClellan at ao.uscourts.gov> wrote:
> 
> Update:
> 
> 	Worked for almost 3 hours, but then started failing again.  I even changed the log rotation to every 15 minutes and it still crashes .    Any other sugestions?  Has anyone ever tried to configured syslog-ng to handle the logging?
> 
> 
> Warning: broctl config has changed (run the broctl "deploy" command)
> Name         Type    Host             Status    Pid    Started
> logger       logger  localhost        terminating 28295  20 Aug 12:30:03
> manager      manager localhost        running   28336  20 Aug 12:30:05
> proxy-1      proxy   localhost        running   28375  20 Aug 12:30:06
> worker-1-1   worker  localhost        running   28565  20 Aug 12:30:08
> 

That's really interesting.. because "terminating" means something very specific, and not something you would see if it was crashing.

Unfortunately broctl throws away the 2nd part of the status file that would narrow that down further, but there are only a few reasons:

src/main.cc
275:	set_processing_status("TERMINATING", "done_with_network");
331:	set_processing_status("TERMINATING", "terminate_bro");
392:	set_processing_status("TERMINATING", "termination_signal");
413:	set_processing_status("TERMINATING", "sig_handler");

src/Net.cc
432:	set_processing_status("TERMINATING", "net_finish");
457:	set_processing_status("TERMINATING", "net_delete"); 

done_with_network, net_finish, and net_delete wouldn't apply to a logger node that has no network interfaces.

termination_signal and sig_handler happen when bro gets a SIGINT or SIGTERM, and terminate_bro happens
when bro exits normally.

If it does happen again and stays like that if you could run

$ sudo cat /usr/local/bro/spool/logger/.status
RUNNING [net_run]

that should show

TERMINATING [one of those reasons]

which would definitively narrow it down.

Is there anything on your system that would be killing bro?  If it were the kernel OOM killer I'd expect that to show up as crashed and not terminating.

— 
Justin Azoff




More information about the Bro mailing list