[Bro] Bro's limitations with high worker count and memory exhaustion
Baxter Milliwew
baxter.milliwew at gmail.com
Mon Jul 6 15:39:57 PDT 2015
Looks like you were right, after watching this closer I do see incidents of
pending counters over zero. I tried reducing logs by disabling files::log
but that didn't help.
If Bro is having problems writing to a ram backed FS I don't think adding
faster disks is going to help.
I still don't understand enough about the architecture from reading
src/RemoteSerializer.cc and src/input/readers/raw/Raw.cc. What else should
I look at if the child's buffering is not related to disk I/O ?
1436216314.304716 Triggers: total=0 pending=0
1436216314.304716 dns/Log::WRITER_ASCII in=1303 out=67 pending=1/0
(#queue r/w: in=1302/1303 out=67/67)
1436216314.304716 conn/Log::WRITER_ASCII in=4556 out=66 pending=11/0
(#queue r/w: in=4545/4556 out=66/66)
1436216350.384011 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0
cached_hosts=0 cached_addrs=0
1436216350.384011 Triggers: total=0 pending=0
1436216387.713486 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0
cached_hosts=0 cached_addrs=0
1436216387.713486 Triggers: total=0 pending=0
1436216387.713486 conn/Log::WRITER_ASCII in=5468 out=68 pending=10/0
(#queue r/w: in=5458/5468 out=68/68)
1436216463.909446 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0
cached_hosts=0 cached_addrs=0
1436216463.909446 Triggers: total=0 pending=0
1436216463.909446 loaded_scripts/Log::WRITER_ASCII in=78 out=76
pending=0/1 (#queue r/w: in=78/78 out=75/76)
1436216463.909446 communication/Log::WRITER_ASCII in=109 out=76
pending=0/1 (#queue r/w: in=109/109 out=75/76)
1436216463.909446 reporter/Log::WRITER_ASCII in=110 out=70 pending=1/0
(#queue r/w: in=109/110 out=70/70)
1436216463.909446 weird/Log::WRITER_ASCII in=533 out=70 pending=1/0
(#queue r/w: in=532/533 out=70/70)
1436216463.909446 packet_filter/Log::WRITER_ASCII in=72 out=71
pending=0/1 (#queue r/w: in=72/72 out=70/71)
1436216463.909446 software/Log::WRITER_ASCII in=305 out=70 pending=2/0
(#queue r/w: in=303/305 out=70/70)
1436216463.909446 dns/Log::WRITER_ASCII in=1697 out=70 pending=2/0
(#queue r/w: in=1695/1697 out=70/70)
1436216463.909446 ssl/Log::WRITER_ASCII in=925 out=70 pending=1/0
(#queue r/w: in=924/925 out=70/70)
1436216463.909446 x509/Log::WRITER_ASCII in=617 out=70 pending=2/0
(#queue r/w: in=615/617 out=70/70)
1436216463.909446 http/Log::WRITER_ASCII in=2468 out=70 pending=1/0
(#queue r/w: in=2467/2468 out=70/70)
1436216463.909446 known_hosts/Log::WRITER_ASCII in=248 out=70 pending=1/0
(#queue r/w: in=247/248 out=70/70)
1436216463.909446 known_services/Log::WRITER_ASCII in=144 out=70
pending=1/0 (#queue r/w: in=143/144 out=70/70)
1436216463.909446 notice/Log::WRITER_ASCII in=143 out=70 pending=2/0
(#queue r/w: in=141/143 out=70/70)
1436216463.909446 dpd/Log::WRITER_ASCII in=148 out=70 pending=1/0
(#queue r/w: in=147/148 out=70/70)
1436216463.909446 known_certs/Log::WRITER_ASCII in=141 out=70 pending=1/0
(#queue r/w: in=140/141 out=70/70)
1436216463.909446 tunnel/Log::WRITER_ASCII in=134 out=69 pending=2/0
(#queue r/w: in=132/134 out=69/69)
1436216463.909446 conn/Log::WRITER_ASCII in=6298 out=69 pending=1/0
(#queue r/w: in=6297/6298 out=69/69)
1436216463.909446 sip/Log::WRITER_ASCII in=250 out=68 pending=1/0
(#queue r/w: in=249/250 out=68/68)
1436216463.909446 ssh/Log::WRITER_ASCII in=130 out=68 pending=1/0
(#queue r/w: in=129/130 out=68/68)
1436216463.909446 smtp/Log::WRITER_ASCII in=147 out=62 pending=1/0
(#queue r/w: in=146/147 out=62/62)
1436216463.909446 radius/Log::WRITER_ASCII in=97 out=63 pending=0/1
(#queue r/w: in=97/97 out=62/63)
1436216463.909446 kerberos/Log::WRITER_ASCII in=74 out=59 pending=0/1
(#queue r/w: in=74/74 out=58/59)
1436216463.909446 rdp/Log::WRITER_ASCII in=104 out=49 pending=1/0
(#queue r/w: in=103/104 out=49/49)
1436216463.909446 ftp/Log::WRITER_ASCII in=63 out=44 pending=0/1
(#queue r/w: in=63/63 out=43/44)
1436216463.909446 pe/Log::WRITER_ASCII in=58 out=37 pending=0/1
(#queue r/w: in=58/58 out=36/37)
1436216463.909446 mysql/Log::WRITER_ASCII in=48 out=34 pending=0/1
(#queue r/w: in=48/48 out=33/34)
1436216463.909446 socks/Log::WRITER_ASCII in=51 out=31 pending=0/1
(#queue r/w: in=51/51 out=30/31)
1436216463.909446 snmp/Log::WRITER_ASCII in=49 out=23 pending=2/0
(#queue r/w: in=47/49 out=23/23)
1436216463.909446 traceroute/Log::WRITER_ASCII in=12 out=9 pending=0/1
(#queue r/w: in=12/12 out=8/9)
1436216463.909446 irc/Log::WRITER_ASCII in=3 out=1 pending=0/1
(#queue r/w: in=3/3 out=0/1)
1436216564.580684 DNS_Mgr: requests=0 succesful=0 failed=0 pending=0
cached_hosts=0 cached_addrs=0
2015-07-06/communication.00:12:27-00:15:01.log.gz:1436141693.018780
manager parent - - - info parent statistics:
pending=3403 bytes=554646435K/12548870K chunks=56936430/40277910
io=1017384/129431 bytes/io=545.17K/96.95K events=14526777/20138054
operations=0/0
2015-07-06/communication.00:12:27-00:15:01.log.gz:1436141696.242561
manager child - - - info selects=59700000
canwrites=58358970 pending=51013
2015-07-06/communication.00:15:01-00:23:49.log.gz:1436141795.153195
manager child - - - info selects=59800000
canwrites=58425751 pending=4263
2015-07-06/communication.00:15:01-00:23:49.log.gz:1436141795.153195
manager child - - - info selects=59900000
canwrites=58525751 pending=7069
2015-07-06/communication.00:15:01-00:23:49.log.gz:1436141795.153195
manager child - - - info selects=60000000
canwrites=58625751 pending=11809
2015-07-06/communication.00:23:49-00:45:54.log.gz:1436142229.139930
manager child - - - info selects=60100000
canwrites=58725751 pending=26653
2015-07-06/communication.00:23:49-00:45:54.log.gz:1436142229.139930
manager child - - - info selects=60200000
canwrites=58825751 pending=39675
2015-07-06/communication.00:23:49-00:45:54.log.gz:1436142229.139930
manager child - - - info selects=60300000
canwrites=58925751 pending=39679
2015-07-06/communication.00:23:49-00:45:54.log.gz:1436142229.139930
manager child - - - info selects=60400000
canwrites=59025751 pending=35003
2015-07-06/communication.00:23:49-00:45:54.log.gz:1436142229.139930
manager child - - - info selects=60500000
canwrites=59125751 pending=29945
2015-07-06/communication.00:23:49-00:45:54.log.gz:1436142229.139930
manager child - - - info selects=60600000
canwrites=59225751 pending=39657
2015-07-06/communication.00:23:49-00:45:54.log.gz:1436142229.139930
manager child - - - info selects=60700000
canwrites=59325751 pending=33067
2015-07-06/communication.00:23:49-00:45:54.log.gz:1436142229.139930
manager child - - - info selects=60800000
canwrites=59425751 pending=25391
2015-07-06/communication.00:45:54-00:53:29.log.gz:1436143554.526023
manager child - - - info selects=60900000
canwrites=59525751 pending=25487
2015-07-06/communication.00:45:54-00:53:29.log.gz:1436143554.526023
manager child - - - info selects=61000000
canwrites=59625751 pending=26713
2015-07-06/communication.00:45:54-00:53:29.log.gz:1436143554.526023
manager child - - - info selects=61100000
canwrites=59725751 pending=19103
2015-07-06/communication.00:53:29-00:59:29.log.gz:1436144009.086860
manager parent - - - info parent statistics:
pending=6767 bytes=570249604K/12864928K chunks=58094056/41111354
io=1051061/132100 bytes/io=542.55K/97.39K events=14754127/20554776
operations=0/0
2015-07-06/communication.00:53:29-00:59:29.log.gz:1436144050.534363
manager parent - - - info parent statistics:
pending=14551 bytes=570669549K/12872129K chunks=58132444/41134984
io=1052002/132120 bytes/io=542.46K/97.43K events=14764015/20566591
operations=0/0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150706/0e50b5c0/attachment-0001.html
More information about the Bro
mailing list