[Bro] OOM-killer & Bro

[Gary Faulkner]

Here it is just after a log rotation:
         14 app_stats.log
         32 capture_loss.log
       3075 communication.log
   10515588 conn.log
    1463723 dns.log
      13760 dpd.log
    1562035 files.log
       1527 ftp.log
    1771968 http.log
         74 irc.log
        127 known_certs.log
      21540 known_hosts.log
       2696 known_services.log
        325 notice.log
        242 reporter.log
      37892 smtp.log
         13 socks.log
      78387 software.log
       3247 ssh.log
     552563 ssl.log
          4 stderr.log
          3 stdout.log
     672817 syslog.log
        556 traceroute.log
       5790 tunnel.log
     472964 weird.log
   17180962 total

1 min later:
         14 app_stats.log
         32 capture_loss.log
       3470 communication.log
   11859982 conn.log
    1619893 dns.log
      15468 dpd.log
    1760513 files.log
       1679 ftp.log
    1993477 http.log
         86 irc.log
        139 known_certs.log
      23839 known_hosts.log
       2881 known_services.log
        352 notice.log
        259 reporter.log
      42941 smtp.log
         13 socks.log
      88544 software.log
       3581 ssh.log
     622256 ssl.log
          4 stderr.log
          3 stdout.log
     750444 syslog.log
        561 traceroute.log
       6567 tunnel.log
     530259 weird.log
   19327257 total

And the diff:

0 app_stats.log
0 capture_loss.log
395 communication.log
1344394 conn.log
156170 dns.log
1708 dpd.log
198478 files.log
152 ftp.log
221509 http.log
12 irc.log
12 known_certs.log
2299 known_hosts.log
185 known_services.log
27 notice.log
17 reporter.log
5049 smtp.log
0 socks.log
10157 software.log
334 ssh.log
69693 ssl.log
0 stderr.log
0 stdout.log
77627 syslog.log
5 traceroute.log
777 tunnel.log
57295 weird.log
2146295 total

Regards,

Gary Faulkner
UW Madison
Office of Campus Information Security
608-262-8591

On 2/4/2014 12:59 PM, Justin Azoff wrote:
> On Tue, Feb 04, 2014 at 12:43:14PM -0600, Gary Faulkner wrote:
>> 11:30AM
>> cat * | wc -l ; sleep 1m ; cat * | wc -l
>> 7618833
>> 9873332
>> diff=2,254,499/min
> That is quite a lot of logs... Can you do just a `wc -l *` a minute
> apart and diff that?  I'm particularly wondering what the rate of
> notices/sec you are getting.  I recently ran into and fixed an issue
> with notice supression using a lot of memory:
>
> https://bro-tracker.atlassian.net/browse/BIT-1115
> https://github.com/bro/bro/commit/ec3f684c610f084fdea8ed5cf85f9c4390eb58e6
>
> I wonder if that could be the issue you are running into..
>


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6257 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140204/8368fef8/attachment.bin