[Bro] BRO Logger crashing due to large DNS log files
Ron McClellan
Ron_McClellan at ao.uscourts.gov
Tue Aug 21 05:21:39 PDT 2018
Justin,
Nothing really on the system that would be killing logger, system is a base CENTOS 7 box, recently built just for BRO. The .status file shows "TERMINATED[atexit]".
Ron
[root@ ron]# sudo cat /logs/bro/spool/logger/.status
TERMINATED [atexit]
Name Type Host Status Pid Started
logger logger localhost crashed
manager manager localhost running 55680 20 Aug 15:25:47
proxy-1 proxy localhost running 55719 20 Aug 15:25:49
worker-1-1 worker localhost running 55893 20 Aug 15:25:50
worker-1-2 worker localhost running 55897 20 Aug 15:25:50
worker-1-40 worker localhost running 56411 20 Aug 15:25:50
worker-1-41 worker localhost crashed
worker-1-42 worker localhost running 56444 20 Aug 15:25:50
worker-1-43 worker localhost running 56446 20 Aug 15:25:50
-----Original Message-----
From: Azoff, Justin S <jazoff at illinois.edu>
Sent: Monday, August 20, 2018 9:08 PM
To: Ron McClellan <Ron_McClellan at ao.uscourts.gov>
Cc: bro at bro.org
Subject: Re: [Bro] BRO Logger crashing due to large DNS log files
> On Aug 20, 2018, at 2:04 PM, Ron McClellan <Ron_McClellan at ao.uscourts.gov> wrote:
>
> Update:
>
> Worked for almost 3 hours, but then started failing again. I even changed the log rotation to every 15 minutes and it still crashes . Any other sugestions? Has anyone ever tried to configured syslog-ng to handle the logging?
>
>
> Warning: broctl config has changed (run the broctl "deploy" command)
> Name Type Host Status Pid Started
> logger logger localhost terminating 28295 20 Aug 12:30:03
> manager manager localhost running 28336 20 Aug 12:30:05
> proxy-1 proxy localhost running 28375 20 Aug 12:30:06
> worker-1-1 worker localhost running 28565 20 Aug 12:30:08
>
That's really interesting.. because "terminating" means something very specific, and not something you would see if it was crashing.
Unfortunately broctl throws away the 2nd part of the status file that would narrow that down further, but there are only a few reasons:
src/main.cc
275: set_processing_status("TERMINATING", "done_with_network");
331: set_processing_status("TERMINATING", "terminate_bro");
392: set_processing_status("TERMINATING", "termination_signal");
413: set_processing_status("TERMINATING", "sig_handler");
src/Net.cc
432: set_processing_status("TERMINATING", "net_finish");
457: set_processing_status("TERMINATING", "net_delete");
done_with_network, net_finish, and net_delete wouldn't apply to a logger node that has no network interfaces.
termination_signal and sig_handler happen when bro gets a SIGINT or SIGTERM, and terminate_bro happens when bro exits normally.
If it does happen again and stays like that if you could run
$ sudo cat /usr/local/bro/spool/logger/.status
RUNNING [net_run]
that should show
TERMINATING [one of those reasons]
which would definitively narrow it down.
Is there anything on your system that would be killing bro? If it were the kernel OOM killer I'd expect that to show up as crashed and not terminating.
—
Justin Azoff
More information about the Bro
mailing list