[Bro] Bro cluster requirements and manager logging backlog bug
Hovsep Levi
hovsep.sanjay.levi at gmail.com
Tue Dec 20 13:09:18 PST 2016
> > In this condition all the workers are at 100% CPU and the worker nodes
> have all 128GB RAM used. The manager node had to be rebooted as "killall
> -9 bro" had no effect. This is what happens if Bro isn't restarted every
> 30 minutes.
>
> This output with the cpu at 0 is kind of odd, unless it was already
> swapping or something.
Yes, it was already swapping.
> Can you check the following:
>
> after bro has been running for a bit:
>
> wc -l *.log | sort -n
>
> to show which log files are the largest
>
>
[bro at mgr /opt/bro_data/logs/current]$ wc -l *.log | sort -n
1 stderr.log
4 stdout.log
8 packet_filter.log
9 stats.log
24 intel.log
37 pe.log
38 kerberos.log
50 irc.log
53 dce_rpc.log
78 radius.log
104 ftp.log
332 mysql.log
355 rfb.log
416 snmp.log
548 reporter.log
841 notice.log
1202 dpd.log
2090 tunnel.log
5515 known_certs.log
7808 rdp.log
15737 communication.log
16393 smtp.log
19713 known_services.log
26794 ssh.log
38487 sip.log
101981 known_hosts.log
106543 software.log
492146 x509.log
714576 ssl.log
795818 dns.log
886360 http.log
985519 weird.log
1936147 files.log
5121874 conn.log
11277601 total
> the output of this command:
>
> top -b -n 1 -H -o TIME |grep bro:|head -n 20
>
> or just run top and press H. That should show all the bro logging threads
> (it works on linux at least) They may show up truncated but it's enough to
> tell them apart.
>
>
[bro at mgr /opt/bro_data/logs/current]$ top -n -H -o time | grep bro
5672 bro 100 10 21076K 2796K RUN 19 6:35 62.79% gzip
5858 bro 95 5 510M 257M RUN 8 3:26 61.38% bro
5785 bro 95 5 2373M 2058M CPU11 11 3:20 59.08% bro
5743 bro 87 0 7897M 7743M RUN 14 3:19 52.78% bro{bro}
5743 bro 88 0 7897M 7743M CPU0 0 3:18 55.18% bro{bro}
5816 bro 40 0 5298M 1158M nanslp 23 1:59 23.29% bro{bro}
5743 bro 37 0 7897M 7743M uwait 4 1:32 23.58% bro{bro}
> What model/count CPU does your manager have?
>
>
Four of these with 32 GB per NUMA node.
Processor Information
Socket Designation: CPU1
Type: Central Processor
Family: Opteron 6200
Manufacturer: AMD
ID: 12 0F 60 00 FF FB 8B 17
Signature: Family 21, Model 1, Stepping 2
Flags:
(...)
Version: AMD Opteron(TM) Processor 6238
Voltage: 1.2 V
External Clock: 3200 MHz
Max Speed: 3600 MHz
Current Speed: 2600 MHz
Status: Populated, Enabled
Upgrade: Socket AM3
L1 Cache Handle: 0x0700
L2 Cache Handle: 0x0701
L3 Cache Handle: 0x0702
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Core Count: 12
Core Enabled: 12
Thread Count: 12
> Are you writing out logs as the default ascii or using json?
>
>
The default Ascii.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20161220/9cb67c48/attachment.html
More information about the Bro
mailing list