[Bro] Bro cluster requirements and manager logging backlog bug

Hovsep Levi hovsep.sanjay.levi at gmail.com
Tue Dec 20 13:09:18 PST 2016


> > In this condition all the workers are at 100% CPU and the worker nodes
> have all 128GB RAM used.  The manager node had to be rebooted as "killall
> -9 bro" had no effect.  This is what happens if Bro isn't restarted every
> 30 minutes.
>
> This output with the cpu at 0 is kind of odd, unless it was already
> swapping or something.


Yes, it was already swapping.




> Can you check the following:
>
> after bro has been running for a bit:
>
>     wc -l *.log | sort -n
>
> to show which log files are the largest
>
>

[bro at mgr /opt/bro_data/logs/current]$ wc -l *.log | sort -n
       1 stderr.log
       4 stdout.log
       8 packet_filter.log
       9 stats.log
      24 intel.log
      37 pe.log
      38 kerberos.log
      50 irc.log
      53 dce_rpc.log
      78 radius.log
     104 ftp.log
     332 mysql.log
     355 rfb.log
     416 snmp.log
     548 reporter.log
     841 notice.log
    1202 dpd.log
    2090 tunnel.log
    5515 known_certs.log
    7808 rdp.log
   15737 communication.log
   16393 smtp.log
   19713 known_services.log
   26794 ssh.log
   38487 sip.log
  101981 known_hosts.log
  106543 software.log
  492146 x509.log
  714576 ssl.log
  795818 dns.log
  886360 http.log
  985519 weird.log
 1936147 files.log
 5121874 conn.log
 11277601 total




> the output of this command:
>
>     top -b -n 1 -H -o TIME |grep bro:|head -n 20
>
> or just run top and press H.  That should show all the bro logging threads
> (it works on linux at least)  They may show up truncated but it's enough to
> tell them apart.
>
>

[bro at mgr /opt/bro_data/logs/current]$ top -n -H -o time | grep bro
 5672 bro        100   10 21076K  2796K RUN    19   6:35  62.79% gzip
 5858 bro         95    5   510M   257M RUN     8   3:26  61.38% bro
 5785 bro         95    5  2373M  2058M CPU11  11   3:20  59.08% bro
 5743 bro         87    0  7897M  7743M RUN    14   3:19  52.78% bro{bro}
 5743 bro         88    0  7897M  7743M CPU0    0   3:18  55.18% bro{bro}
 5816 bro         40    0  5298M  1158M nanslp 23   1:59  23.29% bro{bro}
 5743 bro         37    0  7897M  7743M uwait   4   1:32  23.58% bro{bro}




> What model/count CPU does your manager have?
>
>
Four of these with 32 GB per NUMA node.

Processor Information
        Socket Designation: CPU1
        Type: Central Processor
        Family: Opteron 6200
        Manufacturer: AMD
        ID: 12 0F 60 00 FF FB 8B 17
        Signature: Family 21, Model 1, Stepping 2
        Flags:
                 (...)
        Version: AMD Opteron(TM) Processor 6238
        Voltage: 1.2 V
        External Clock: 3200 MHz
        Max Speed: 3600 MHz
        Current Speed: 2600 MHz
        Status: Populated, Enabled
        Upgrade: Socket AM3
        L1 Cache Handle: 0x0700
        L2 Cache Handle: 0x0701
        L3 Cache Handle: 0x0702
        Serial Number: Not Specified
        Asset Tag: Not Specified
        Part Number: Not Specified
        Core Count: 12
        Core Enabled: 12
        Thread Count: 12




> Are you writing out logs as the default ascii or using json?
>
>

The default Ascii.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161220/9cb67c48/attachment.html 


More information about the Bro mailing list