[Bro] BRO Logger crashing due to large DNS log files
Azoff, Justin S
jazoff at illinois.edu
Mon Aug 20 18:08:11 PDT 2018
> On Aug 20, 2018, at 2:04 PM, Ron McClellan <Ron_McClellan at ao.uscourts.gov> wrote:
>
> Update:
>
> Worked for almost 3 hours, but then started failing again. I even changed the log rotation to every 15 minutes and it still crashes . Any other sugestions? Has anyone ever tried to configured syslog-ng to handle the logging?
>
>
> Warning: broctl config has changed (run the broctl "deploy" command)
> Name Type Host Status Pid Started
> logger logger localhost terminating 28295 20 Aug 12:30:03
> manager manager localhost running 28336 20 Aug 12:30:05
> proxy-1 proxy localhost running 28375 20 Aug 12:30:06
> worker-1-1 worker localhost running 28565 20 Aug 12:30:08
>
That's really interesting.. because "terminating" means something very specific, and not something you would see if it was crashing.
Unfortunately broctl throws away the 2nd part of the status file that would narrow that down further, but there are only a few reasons:
src/main.cc
275: set_processing_status("TERMINATING", "done_with_network");
331: set_processing_status("TERMINATING", "terminate_bro");
392: set_processing_status("TERMINATING", "termination_signal");
413: set_processing_status("TERMINATING", "sig_handler");
src/Net.cc
432: set_processing_status("TERMINATING", "net_finish");
457: set_processing_status("TERMINATING", "net_delete");
done_with_network, net_finish, and net_delete wouldn't apply to a logger node that has no network interfaces.
termination_signal and sig_handler happen when bro gets a SIGINT or SIGTERM, and terminate_bro happens
when bro exits normally.
If it does happen again and stays like that if you could run
$ sudo cat /usr/local/bro/spool/logger/.status
RUNNING [net_run]
that should show
TERMINATING [one of those reasons]
which would definitively narrow it down.
Is there anything on your system that would be killing bro? If it were the kernel OOM killer I'd expect that to show up as crashed and not terminating.
—
Justin Azoff
More information about the Bro
mailing list