[Bro] BRO Logger crashing due to large DNS log files

[Ron McClellan]

All,

                Having an issue with the bro logger crashing due to large volumes of DNS log traffic, 20-30GB an hour.  This is completely a local configuration, on a system with super-fast flash storage, 64 cores, 256GB RAM running BRO 2.5.4.  If I disable DNS logging, everything works fine without issue.  When I enable it, I get the results below.  I thought it might be an issue with gzipping the old logs, so I replaced the standard gzip with pigz and I can manually compress the 30+ gig files in seconds, so don't think that is the issue.  I also tried pinning dedicated cores to the logger, currently 6 cores, which should be way more than enough.   Any thoughts or suggestions.

Thanks,

Ron

current]# ll -h
total 43G
-rw-r--r--. 1 root root 3.2K Aug 18 12:00 capture_loss-18-08-18_11.00.00.log
-rw-r--r--. 1 root root 3.2K Aug 18 12:18 capture_loss-18-08-18_12.00.00.log
-rw-r--r--. 1 root root 2.3M Aug 18 12:00 communication-18-08-18_11.00.00.log
-rw-r--r--. 1 root root 1.4M Aug 18 12:18 communication-18-08-18_12.00.00.log
-rw-r--r--. 1 root root 4.8K Aug 18 12:18 communication.log
-rw-r--r--. 1 root root  19G Aug 18 11:39 dns-18-08-18_10.11.22.log
-rw-r--r--. 1 root root  16G Aug 18 12:26 dns-18-08-18_11.00.00.log
-rw-r--r--. 1 root root  12M Aug 18 12:00 files-18-08-18_11.00.00.log
-rw-r--r--. 1 root root 5.2M Aug 18 12:18 files-18-08-18_12.00.00.log
-rw-r--r--. 1 root root  15K Aug 18 12:00 known_certs-18-08-18_11.00.00.log
-rw-r--r--. 1 root root  15K Aug 18 12:18 known_certs-18-08-18_12.00.00.log
-rw-r--r--. 1 root root  98K Aug 18 12:00 known_hosts-18-08-18_11.00.00.log
-rw-r--r--. 1 root root  24K Aug 18 12:18 known_hosts-18-08-18_12.00.00.log
-rw-r--r--. 1 root root  71K Aug 18 12:00 known_services-18-08-18_11.00.00.log
-rw-r--r--. 1 root root 5.2K Aug 18 12:18 known_services-18-08-18_12.00.00.log
-rw-r--r--. 1 root root 1.6K Aug 18 12:00 notice-18-08-18_11.00.00.log
-rw-r--r--. 1 root root  954 Aug 18 12:18 notice-18-08-18_12.00.00.log
-rw-r--r--. 1 root root  262 Aug 18 12:18 reporter-18-08-18_12.00.00.log
-rw-r--r--. 1 root root  23M Aug 18 12:00 smtp-18-08-18_11.00.00.log
-rw-r--r--. 1 root root 9.2M Aug 18 12:18 smtp-18-08-18_12.00.00.log
-rw-r--r--. 1 root root 1.2M Aug 18 12:00 snmp-18-08-18_11.00.00.log
-rw-r--r--. 1 root root 415K Aug 18 12:18 snmp-18-08-18_12.00.00.log
-rw-r--r--. 1 root root  81K Aug 18 12:00 software-18-08-18_11.00.00.log
-rw-r--r--. 1 root root 8.4K Aug 18 12:18 software-18-08-18_12.00.00.log
-rw-r--r--. 1 root root  30K Aug 18 12:00 ssh-18-08-18_11.00.00.log
-rw-r--r--. 1 root root  13K Aug 18 12:18 ssh-18-08-18_12.00.00.log
-rw-r--r--. 1 root root 217K Aug 18 12:00 ssl-18-08-18_11.00.00.log
-rw-r--r--. 1 root root  78K Aug 18 12:18 ssl-18-08-18_12.00.00.log
-rw-r--r--. 1 root root  37K Aug 18 12:00 stats-18-08-18_11.00.00.log
-rw-r--r--. 1 root root  16K Aug 18 12:18 stats-18-08-18_12.00.00.log
-rw-r--r--. 1 root root   28 Aug 18 12:18 stderr.log
-rw-r--r--. 1 root root  188 Aug 18 10:11 stdout.log
-rw-r--r--. 1 root root 6.8G Aug 18 12:00 weird-18-08-18_11.00.00.log
-rw-r--r--. 1 root root 2.5G Aug 18 12:18 weird-18-08-18_12.00.00.log
-rw-r--r--. 1 root root 178K Aug 18 12:00 x509-18-08-18_11.00.00.log
-rw-r--r--. 1 root root  80K Aug 18 12:18 x509-18-08-18_12.00.00.log

# /usr/local/bro/bin/bro --version
/usr/local/bro/bin/bro version 2.5.4


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180819/378cf555/attachment.html