From thealy at bnl.gov Mon Dec 2 06:36:27 2013 From: thealy at bnl.gov (Terry Healy) Date: Mon, 02 Dec 2013 09:36:27 -0500 Subject: [Bro] Timestamp conversion code Message-ID: <529C9AEB.3050508@bnl.gov> Hello- New here an tasked with injesting some Bro log files. All is fine except for the timestamp. I have found that I can use bro-cut to see the desired (or usable) format, but I can't use bro-cut in my work flow and was unable to reverse engineer what it was doing. I just need to get from the (proprietary?) timestamp format to something standard like: 2013-12-02T09:30:34-0500 Thank. Terry From jwelcher at gmail.com Mon Dec 2 07:03:10 2013 From: jwelcher at gmail.com (James Welcher) Date: Mon, 2 Dec 2013 07:03:10 -0800 Subject: [Bro] Timestamp conversion code In-Reply-To: <529C9AEB.3050508@bnl.gov> References: <529C9AEB.3050508@bnl.gov> Message-ID: Add cf to your command pipeline http://www.bro.org/community/software.html On Monday, December 2, 2013, Terry Healy wrote: > Hello- New here an tasked with injesting some Bro log files. All is fine > except for the timestamp. I have found that I can use bro-cut to see the > desired (or usable) format, but I can't use bro-cut in my work flow and > was unable to reverse engineer what it was doing. > > I just need to get from the (proprietary?) timestamp format to something > standard like: > > 2013-12-02T09:30:34-0500 > > Thank. > > Terry > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- James Welcher -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131202/e36e11fa/attachment.html From thealy at bnl.gov Mon Dec 2 07:17:42 2013 From: thealy at bnl.gov (Terry Healy) Date: Mon, 02 Dec 2013 10:17:42 -0500 Subject: [Bro] Timestamp conversion code In-Reply-To: References: <529C9AEB.3050508@bnl.gov> Message-ID: <529CA496.2020909@bnl.gov> Perfect Jim. Thanks very much for the quick help. On 12/02/2013 10:03 AM, James Welcher wrote: > Add cf to your command pipeline > > http://www.bro.org/community/software.html > > On Monday, December 2, 2013, Terry Healy wrote: > > Hello- New here an tasked with injesting some Bro log files. All is fine > except for the timestamp. I have found that I can use bro-cut to see the > desired (or usable) format, but I can't use bro-cut in my work flow and > was unable to reverse engineer what it was doing. > > I just need to get from the (proprietary?) timestamp format to something > standard like: > > 2013-12-02T09:30:34-0500 > > Thank. > > Terry > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -- > James Welcher From jsiwek at illinois.edu Tue Dec 3 09:54:57 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Tue, 3 Dec 2013 17:54:57 +0000 Subject: [Bro] Some events not received by broccoli In-Reply-To: <1784413d504245c7ad9a32e9b60d22e1@AMSPR04MB147.eurprd04.prod.outlook.com> References: <1784413d504245c7ad9a32e9b60d22e1@AMSPR04MB147.eurprd04.prod.outlook.com> Message-ID: <32D600B4-4684-4C93-A632-A3EDDA41B7FC@illinois.edu> On Nov 22, 2013, at 9:04 AM, Bj?rn Samvik wrote: > Hello, > > I'm using broccoli to receive bro (2.2-5) events and are having some problems. Consider the following. > > The broccoli client is listening to 2 events. > bro_event_registry_add(m_bc, "file_new", (BroEventFunc)&Broccoli::newFile, this); > bro_event_registry_add(m_bc, "test_event", (BroEventFunc)&Broccoli::newFile, this); > > The following bro script is used. > > ... > global test_event: event(f: fa_file); > event file_new(f: fa_file) > { > event test_event(f); > } > > The file_new event is correctly received by my broccoli client however the test_event is not received. If I change the content of the test_event to something else it works. > > ... > global test_event: event(f: string); > event file_new(f: fa_file) > { > event test_event(f$mime_type); > } > > So, is this expected and in that case why and what is the proposed way of solving the issue? > > (Also noticed that the file_state_removed(f: fa_file) event is not received by the broccoli client.) It?s somewhat of a bug in broccoli: it doesn't support receiving events that have arguments containing vector values and silently discards those that do. The reason why file_new is received, but not test_event/file_state_removed is because the fa_file record argument starts off with some optional vector fields that aren?t initialized and they later become populated by some Bro scripts before test_event/file_state_removed make it through the event queue. I?ve patched broccoli [1] to be able to receive vectors, which should fix your problem if you want to try it. Otherwise, the workaround is to send the broccoli client simpler data types (Bro connection/fa_file/*::Info records can get complicated), perhaps picking out just pieces you need. - Jon [1] https://bro-tracker.atlassian.net/browse/BIT-1100 From jbabio at po-box.esu.edu Tue Dec 3 18:05:39 2013 From: jbabio at po-box.esu.edu (John Babio) Date: Wed, 4 Dec 2013 02:05:39 +0000 Subject: [Bro] scripting a notice based on software framework Message-ID: <6E7D7EC4661DB5438A60388F3ED1BBAA062FFC@msxmb2.admin.esu.edu> I was wondering if someone could help me out. I am looking to write a test script based on data logged to the software.log. I want to create a notice for a regex string match from the unparsed_version. I was going to use the intel framework but I have to match the entire string. From itxx00 at gmail.com Tue Dec 3 22:49:53 2013 From: itxx00 at gmail.com (Gao Yongwei) Date: Wed, 4 Dec 2013 14:49:53 +0800 Subject: [Bro] how to get bro to support utf-8 chars like Chinese in detect-webapps.sig ? Message-ID: Hello ,all: I am a newbie, I've enabled policy/protocols/http/detect-webapps in local.bro and prepared a redmine test site, bro can successful detect this type of web app,in software.log I can see: 1386139178.321119 192.168.1.2 80 HTTP::WEB_APPLICATION Redmine - - - - - Redmine http://redmine.example.com/ Now I want to detect web app by Chinese chars , I've tried to add utf-8 characters in detect-webapp.sig ,but could not get bro work. how can I achieve that goal ? thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131204/c2669575/attachment.html From Bjorn.Samvik at netclean.com Wed Dec 4 04:06:27 2013 From: Bjorn.Samvik at netclean.com (=?Windows-1252?Q?Bj=F6rn_Samvik?=) Date: Wed, 4 Dec 2013 12:06:27 +0000 Subject: [Bro] Some events not received by broccoli In-Reply-To: <32D600B4-4684-4C93-A632-A3EDDA41B7FC@illinois.edu> References: <1784413d504245c7ad9a32e9b60d22e1@AMSPR04MB147.eurprd04.prod.outlook.com>, <32D600B4-4684-4C93-A632-A3EDDA41B7FC@illinois.edu> Message-ID: <7dd86a3edc5c4e9b899af6e370df7e95@AM3PR04MB258.eurprd04.prod.outlook.com> Hello I have tested your patch and it works perfectly. Thanks for the information and the patch. /Bj?rn (DoNotAddDisclaimer) ________________________________________ From: Siwek, Jonathan Luke Sent: Tuesday, December 03, 2013 18:54 To: Bj?rn Samvik Cc: bro at bro.org Subject: Re: [Bro] Some events not received by broccoli On Nov 22, 2013, at 9:04 AM, Bj?rn Samvik wrote: > Hello, > > I'm using broccoli to receive bro (2.2-5) events and are having some problems. Consider the following. > > The broccoli client is listening to 2 events. > bro_event_registry_add(m_bc, "file_new", (BroEventFunc)&Broccoli::newFile, this); > bro_event_registry_add(m_bc, "test_event", (BroEventFunc)&Broccoli::newFile, this); > > The following bro script is used. > > ... > global test_event: event(f: fa_file); > event file_new(f: fa_file) > { > event test_event(f); > } > > The file_new event is correctly received by my broccoli client however the test_event is not received. If I change the content of the test_event to something else it works. > > ... > global test_event: event(f: string); > event file_new(f: fa_file) > { > event test_event(f$mime_type); > } > > So, is this expected and in that case why and what is the proposed way of solving the issue? > > (Also noticed that the file_state_removed(f: fa_file) event is not received by the broccoli client.) It?s somewhat of a bug in broccoli: it doesn't support receiving events that have arguments containing vector values and silently discards those that do. The reason why file_new is received, but not test_event/file_state_removed is because the fa_file record argument starts off with some optional vector fields that aren?t initialized and they later become populated by some Bro scripts before test_event/file_state_removed make it through the event queue. I?ve patched broccoli [1] to be able to receive vectors, which should fix your problem if you want to try it. Otherwise, the workaround is to send the broccoli client simpler data types (Bro connection/fa_file/*::Info records can get complicated), perhaps picking out just pieces you need. - Jon [1] https://bro-tracker.atlassian.net/browse/BIT-1100 From seth at icir.org Wed Dec 4 05:21:31 2013 From: seth at icir.org (Seth Hall) Date: Wed, 4 Dec 2013 08:21:31 -0500 Subject: [Bro] how to get bro to support utf-8 chars like Chinese in detect-webapps.sig ? In-Reply-To: References: Message-ID: <455461AF-18EE-4F0A-A7FC-8E284341054B@icir.org> On Dec 4, 2013, at 1:49 AM, Gao Yongwei wrote: > Now I want to detect web app by Chinese chars , I've tried to add utf-8 characters in detect-webapp.sig ,but could not get bro work. Unfortunately Bro doesn't have any form of unicode support yet (we've been talking about this for quite some time but haven't converged on anything quite yet). What you will have to do is expand the unicode characters you're hoping to match into it's constituent bytes. So your signature payload may look like this? payload /test\xF0\x9F\x92\xA9test/ It's certainly less than ideal, but does that work for you for now? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131204/04e564bc/attachment.bin From itxx00 at gmail.com Wed Dec 4 06:38:31 2013 From: itxx00 at gmail.com (Gao Yongwei) Date: Wed, 4 Dec 2013 22:38:31 +0800 Subject: [Bro] how to get bro to support utf-8 chars like Chinese in detect-webapps.sig ? In-Reply-To: <455461AF-18EE-4F0A-A7FC-8E284341054B@icir.org> References: <455461AF-18EE-4F0A-A7FC-8E284341054B@icir.org> Message-ID: > Unfortunately Bro doesn't have any form of unicode support yet (we've been > talking about this for quite some time but haven't converged on anything > quite yet). What you will have to do is expand the unicode characters > you're hoping to match into it's constituent bytes. So your signature > payload may look like this? > > payload /test\xF0\x9F\x92\xA9test/ > > It's certainly less than ideal, but does that work for you for now? > Hi ,Seth ,thanks for your quick reply , the constituent bytes format works for me now! but I get into another problem: when I turn off gzip in nginx.conf , bro works fine, but when turn on gzip, bro seems not work. how could bro unzip the html content before apply the payload ? thank. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131204/e999cf5f/attachment.html From seth at icir.org Wed Dec 4 08:06:01 2013 From: seth at icir.org (Seth Hall) Date: Wed, 4 Dec 2013 11:06:01 -0500 Subject: [Bro] how to get bro to support utf-8 chars like Chinese in detect-webapps.sig ? In-Reply-To: References: <455461AF-18EE-4F0A-A7FC-8E284341054B@icir.org> Message-ID: On Dec 4, 2013, at 9:38 AM, Gao Yongwei wrote: > when I turn off gzip in nginx.conf , bro works fine, but when turn on gzip, bro seems not work. > how could bro unzip the html content before apply the payload ? thank. We have a specialized keyword for checking patterns in HTTP payloads. It was decode the content prior to matching. http-reply-body /test\xF0\x9F\x92\xA9test/ .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131204/3e88344a/attachment.bin From jbabio at po-box.esu.edu Wed Dec 4 08:12:25 2013 From: jbabio at po-box.esu.edu (John Babio) Date: Wed, 4 Dec 2013 16:12:25 +0000 Subject: [Bro] software.log Message-ID: Here is what I have so far. It is working but I don?t know if it is written correctly. @load base/frameworks/notice @load base/frameworks/software module OLD_JAVA; module HTTP; export { redef enum Notice::Type += { OLD_JAVA::Java_seen, }; } event http_header(c: connection, is_orig: bool, name: string, value: string) &priority=2 { if ( Software::found(c$id, [$unparsed_version="Java/1.7.0_40", $host=c$id$orig_h])); { NOTICE([$note=OLD_JAVA::Java_seen, $msg=fmt("Old Java Seen")]); } } From vladg at cmu.edu Wed Dec 4 08:36:36 2013 From: vladg at cmu.edu (Vlad Grigorescu) Date: Wed, 4 Dec 2013 16:36:36 +0000 Subject: [Bro] software.log In-Reply-To: <31331_1386174345_rB4GPik9032241_CEC4BEBB.88AF%jbabio@po-box.esu.edu> References: <31331_1386174345_rB4GPik9032241_CEC4BEBB.88AF%jbabio@po-box.esu.edu> Message-ID: <9C6AF4A2-42BE-44FF-ADFC-C9EC9BCD8E0B@andrew.cmu.edu> John, Have you looked at policy/frameworks/software/vulnerable.bro[1]? It seems like it implements what you're looking to do. For example, here's what I have in my local.bro: > @load frameworks/software/vulnerable > global java_1_6_vuln: Software::VulnerableVersionRange = [$max=[$major=1,$minor=6,$minor2=0,$minor3=48]]; > global java_1_7_vuln: Software::VulnerableVersionRange = [$min=[$major=1,$minor=7], $max=[$major=1,$minor=7,$minor2=0,$minor3=22]]; > > redef Software::vulnerable_versions += { > ["Java"] = set(java_1_6_vuln, java_1_7_vuln) > }; See also: https://github.com/bro/bro/blob/master/NEWS#L313 What this does is define two ranges of vulnerable Java versions. The first is anything prior to 1.6.0.48 (including 1.5, 1.4, etc.). The second is anything between 1.7.0.0 and 1.7.0.22. Of course, if you only care about 1.7.0.40, you can just define that as the min/max. Does that help? Or was that not the functionality you were looking for? --Vlad [1] - On Dec 4, 2013, at 11:12 AM, John Babio wrote: > Here is what I have so far. It is working but I don?t know if it is written correctly. > > > @load base/frameworks/notice > > @load base/frameworks/software > > > module OLD_JAVA; > > > module HTTP; > > > export { > > redef enum Notice::Type += { > > OLD_JAVA::Java_seen, > > }; > > } > > > event http_header(c: connection, is_orig: bool, name: string, value: string) &priority=2 > > { > > > if ( Software::found(c$id, [$unparsed_version="Java/1.7.0_40", $host=c$id$orig_h])); > > { > > NOTICE([$note=OLD_JAVA::Java_seen, $msg=fmt("Old Java Seen")]); > > } > > } > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jbabio at po-box.esu.edu Wed Dec 4 08:40:23 2013 From: jbabio at po-box.esu.edu (John Babio) Date: Wed, 4 Dec 2013 16:40:23 +0000 Subject: [Bro] software.log In-Reply-To: <9C6AF4A2-42BE-44FF-ADFC-C9EC9BCD8E0B@andrew.cmu.edu> Message-ID: Yes This is exactly what I was looking for. I just didn?t know how to go about it. Thank you Vlad! From JAzoff at albany.edu Wed Dec 4 08:43:49 2013 From: JAzoff at albany.edu (Justin Azoff) Date: Wed, 4 Dec 2013 11:43:49 -0500 Subject: [Bro] software.log In-Reply-To: References: Message-ID: <20131204164349.GG12701@datacomm.albany.edu> On Wed, Dec 04, 2013 at 04:12:25PM +0000, John Babio wrote: > NOTICE([$note=OLD_JAVA::Java_seen, $msg=fmt("Old Java Seen")]); You want to add $conn=c to the notice, otherwise it won't contain the address information. -- -- Justin Azoff From jlay at slave-tothe-box.net Wed Dec 4 08:44:38 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Wed, 04 Dec 2013 09:44:38 -0700 Subject: [Bro] software.log In-Reply-To: References: Message-ID: On 2013-12-04 09:40, John Babio wrote: > Yes This is exactly what I was looking for. I just didn?t know how to > go > about it. Thank you Vlad! > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Care to send it to the list? I'd like to see it myself...thank you. James From jbabio at po-box.esu.edu Wed Dec 4 12:23:17 2013 From: jbabio at po-box.esu.edu (John Babio) Date: Wed, 4 Dec 2013 20:23:17 +0000 Subject: [Bro] Bro Digest, Vol 92, Issue 4 In-Reply-To: Message-ID: Add this to local.bro. Works awesome! Courtesy of Vlad. I wish I new it was as easy as this. :) @load frameworks/software/vulnerable global java_1_6_vuln: Software::VulnerableVersionRange = [$max=[$major=1,$minor=6,$minor2=0,$minor3=48]]; global java_1_7_vuln: Software::VulnerableVersionRange = [$min=[$major=1,$minor=7], $max=[$major=1,$minor=7,$minor2=0,$minor3=22]]; redef Software::vulnerable_versions += { ["Java"] = set(java_1_6_vuln, java_1_7_vuln) }; See also: https://github.com/bro/bro/blob/master/NEWS#L313 What this does is define two ranges of vulnerable Java versions. The first is anything prior to 1.6.0.48 (including 1.5, 1.4, etc.). The second is anything between 1.7.0.0 and 1.7.0.22. Of course, if you only care about 1.7.0.40, you can just define that as the min/max. Does that help? Or was that not the functionality you were looking for? On 12/4/13, 3:00 PM, "bro-request at bro.org" wrote: >Send Bro mailing list submissions to > bro at bro.org > >To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >or, via email, send a message with subject or body 'help' to > bro-request at bro.org > >You can reach the person managing the list at > bro-owner at bro.org > >When replying, please edit your Subject line so it is more specific >than "Re: Contents of Bro digest..." > > >Today's Topics: > > 1. Re: software.log (John Babio) > 2. Re: software.log (Justin Azoff) > 3. Re: software.log (James Lay) > > >---------------------------------------------------------------------- > >Message: 1 >Date: Wed, 4 Dec 2013 16:40:23 +0000 >From: John Babio >Subject: Re: [Bro] software.log >To: Vlad Grigorescu >Cc: "bro at bro.org" >Message-ID: >Content-Type: text/plain; charset="iso-8859-1" > >Yes This is exactly what I was looking for. I just didn?t know how to go >about it. Thank you Vlad! > > > > >------------------------------ > >Message: 2 >Date: Wed, 4 Dec 2013 11:43:49 -0500 >From: Justin Azoff >Subject: Re: [Bro] software.log >To: John Babio >Cc: "bro at bro.org" >Message-ID: <20131204164349.GG12701 at datacomm.albany.edu> >Content-Type: text/plain; charset=us-ascii > >On Wed, Dec 04, 2013 at 04:12:25PM +0000, John Babio wrote: >> NOTICE([$note=OLD_JAVA::Java_seen, $msg=fmt("Old Java >>Seen")]); > >You want to add $conn=c to the notice, otherwise it won't contain the >address information. > >-- >-- Justin Azoff > > >------------------------------ > >Message: 3 >Date: Wed, 04 Dec 2013 09:44:38 -0700 >From: James Lay >Subject: Re: [Bro] software.log >To: >Message-ID: >Content-Type: text/plain; charset=UTF-8; format=flowed > >On 2013-12-04 09:40, John Babio wrote: >> Yes This is exactly what I was looking for. I just didn?t know how to >> go >> about it. Thank you Vlad! >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > >Care to send it to the list? I'd like to see it myself...thank you. > >James > > >------------------------------ > >_______________________________________________ >Bro mailing list >Bro at bro.org >http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > >End of Bro Digest, Vol 92, Issue 4 >********************************** From gary at doit.wisc.edu Wed Dec 4 16:05:30 2013 From: gary at doit.wisc.edu (Gary Faulkner) Date: Wed, 04 Dec 2013 18:05:30 -0600 Subject: [Bro] out of memory after a couple days? Message-ID: <529FC34A.4010207@doit.wisc.edu> I've been running on Bro 2.2 for just under a month mostly without incident. After my most recent restart it ran for about two days since before I received crash reports for several workers and proxies across multiple hosts. Upon investigation it looks like I might have run out of memory. I found logs such as the following in /var/log/messages on all of my nodes (manager and worker nodes): bro invoked oom-killer ... Out of memory: Kill process 7152 (bro) score 151 or sacrifice child Has anyone seen this before? Is this just a sign I need more RAM or am I possibly running into a memory leak? I have run for up to a week without incident in the past before restarting of my own accord after making various changes to reporting, policy etc. The only thing I changed prior to the last restart was to disable an email notice I had previously set. Regards, -- Gary Faulkner UW Madison Office of Campus Information Security 608-262-8591 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6257 bytes Desc: S/MIME Cryptographic Signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131204/aaa77846/attachment.bin From sconzo at visiblerisk.com Wed Dec 4 16:36:49 2013 From: sconzo at visiblerisk.com (Mike Sconzo) Date: Wed, 4 Dec 2013 18:36:49 -0600 Subject: [Bro] out of memory after a couple days? In-Reply-To: <529FC34A.4010207@doit.wisc.edu> References: <529FC34A.4010207@doit.wisc.edu> Message-ID: I've seen this behavior too (in addition so some really weird and massive packet drops). I've been trying to gather some perf.log data and track it down to the time(s) in question. One of the strange parts is it's on a box that's only seeing 200mbps w/8 bro workers on 16 cores and 96GB of ram. On Wed, Dec 4, 2013 at 6:05 PM, Gary Faulkner wrote: > I've been running on Bro 2.2 for just under a month mostly without incident. > After my most recent restart it ran for about two days since before I > received crash reports for several workers and proxies across multiple > hosts. Upon investigation it looks like I might have run out of memory. I > found logs such as the following in /var/log/messages on all of my nodes > (manager and worker nodes): > > bro invoked oom-killer > ... > Out of memory: Kill process 7152 (bro) score 151 or sacrifice child > > Has anyone seen this before? Is this just a sign I need more RAM or am I > possibly running into a memory leak? I have run for up to a week without > incident in the past before restarting of my own accord after making various > changes to reporting, policy etc. The only thing I changed prior to the last > restart was to disable an email notice I had previously set. > > Regards, > > -- > Gary Faulkner > UW Madison > Office of Campus Information Security > 608-262-8591 > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- cat ~/.bash_history > documentation.txt From mike.patterson at uwaterloo.ca Wed Dec 4 19:34:01 2013 From: mike.patterson at uwaterloo.ca (Mike Patterson) Date: Wed, 4 Dec 2013 22:34:01 -0500 Subject: [Bro] out of memory after a couple days? In-Reply-To: <529FC34A.4010207@doit.wisc.edu> References: <529FC34A.4010207@doit.wisc.edu> Message-ID: <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> I think you?re definitely running into a memory leak. I?ve had 2.2 processes try to grab up to 100GB of RAM. 8 workers, 96GB of RAM, but the box splits time with another 8 snort workers. My late 2.1 release (september 21 IIRC) was quite a bit more stable. Mike On Dec 4, 2013, at 7:05 PM, Gary Faulkner wrote: > I've been running on Bro 2.2 for just under a month mostly without incident. After my most recent restart it ran for about two days since before I received crash reports for several workers and proxies across multiple hosts. Upon investigation it looks like I might have run out of memory. I found logs such as the following in /var/log/messages on all of my nodes (manager and worker nodes): > > bro invoked oom-killer > ... > Out of memory: Kill process 7152 (bro) score 151 or sacrifice child > > Has anyone seen this before? Is this just a sign I need more RAM or am I possibly running into a memory leak? I have run for up to a week without incident in the past before restarting of my own accord after making various changes to reporting, policy etc. The only thing I changed prior to the last restart was to disable an email notice I had previously set. > > Regards, > > -- > Gary Faulkner > UW Madison > Office of Campus Information Security > 608-262-8591 > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at icir.org Wed Dec 4 20:31:51 2013 From: seth at icir.org (Seth Hall) Date: Wed, 4 Dec 2013 23:31:51 -0500 Subject: [Bro] out of memory after a couple days? In-Reply-To: <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> Message-ID: <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> On Dec 4, 2013, at 10:34 PM, Mike Patterson wrote: > I think you?re definitely running into a memory leak. I?ve had 2.2 processes try to grab up to 100GB of RAM. 8 workers, 96GB of RAM, but the box splits time with another 8 snort workers. My late 2.1 release (september 21 IIRC) was quite a bit more stable. I think there is some particular traffic that you guys are running into that's causing it. A few other people have encountered that too but we haven't been able to nail down what it is yet. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131204/55df5afa/attachment.bin From gary at doit.wisc.edu Wed Dec 4 21:55:08 2013 From: gary at doit.wisc.edu (Gary Faulkner) Date: Wed, 04 Dec 2013 23:55:08 -0600 Subject: [Bro] out of memory after a couple days? In-Reply-To: <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> Message-ID: <52A0153C.3050804@doit.wisc.edu> It normally has a healthy diet of 7-8Gbps and 1Mpps of traffic to consume, and other than some initial growing pains (not enough proxies) it has mostly just worked, although sometimes when it has stopped receiving traffic, such as during network maintenance / code upgrades to the load-balancer it has complained and started killing processes. Regards, Gary On 12/4/2013 10:31 PM, Seth Hall wrote: > > On Dec 4, 2013, at 10:34 PM, Mike Patterson wrote: > >> I think you?re definitely running into a memory leak. I?ve had 2.2 processes try to grab up to 100GB of RAM. 8 workers, 96GB of RAM, but the box splits time with another 8 snort workers. My late 2.1 release (september 21 IIRC) was quite a bit more stable. > > > I think there is some particular traffic that you guys are running into that's causing it. A few other people have encountered that too but we haven't been able to nail down what it is yet. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > -- Gary Faulkner UW Madison Office of Campus Information Security 608-262-8591 From mike.patterson at uwaterloo.ca Thu Dec 5 06:12:03 2013 From: mike.patterson at uwaterloo.ca (Mike Patterson) Date: Thu, 5 Dec 2013 09:12:03 -0500 Subject: [Bro] out of memory after a couple days? In-Reply-To: <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> Message-ID: <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> On Dec 4, 2013, at 11:31 PM, Seth Hall wrote: > > On Dec 4, 2013, at 10:34 PM, Mike Patterson wrote: > >> I think you?re definitely running into a memory leak. I?ve had 2.2 processes try to grab up to 100GB of RAM. 8 workers, 96GB of RAM, but the box splits time with another 8 snort workers. My late 2.1 release (september 21 IIRC) was quite a bit more stable. > > > I think there is some particular traffic that you guys are running into that's causing it. A few other people have encountered that too but we haven't been able to nail down what it is yet. That was my assumption too. I upgraded on 8 November, leaked early AM 16th, and then again on the 29th. Traffic would have been at an ebb on the 16th, and rising on the 29th, so I don?t think it?s sheer volume - as you say, there must be something *in* the traffic. Or more likely, a sequence of things, otherwise I expect 2.2 would be vomiting all over my RAM far more often. Please let me know if there?s anything I can do to help; I got lucky with these, the first crash was the day before I started vacation (well, technically, my first day of) and the second crash was the day immediately after I returned. :) Unfortunately, when it does happen, it takes out my IDS entirely as I need to cold-boot the server, so if live diagnostics are required, it?ll have to be timed when people are around. Mike From sconzo at visiblerisk.com Thu Dec 5 07:14:13 2013 From: sconzo at visiblerisk.com (Mike Sconzo) Date: Thu, 5 Dec 2013 09:14:13 -0600 Subject: [Bro] out of memory after a couple days? In-Reply-To: <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> Message-ID: If we're able to get our hands on some of the traffic (pcaps spanning the time window of memory usage/massive drops) that causes these issues, what would be some good tests to run against it? Nothing suspicious or weird shows up in the perf.log for each worker (or manager). On Thu, Dec 5, 2013 at 8:12 AM, Mike Patterson wrote: > On Dec 4, 2013, at 11:31 PM, Seth Hall wrote: > >> >> On Dec 4, 2013, at 10:34 PM, Mike Patterson wrote: >> >>> I think you?re definitely running into a memory leak. I?ve had 2.2 processes try to grab up to 100GB of RAM. 8 workers, 96GB of RAM, but the box splits time with another 8 snort workers. My late 2.1 release (september 21 IIRC) was quite a bit more stable. >> >> >> I think there is some particular traffic that you guys are running into that's causing it. A few other people have encountered that too but we haven't been able to nail down what it is yet. > > That was my assumption too. I upgraded on 8 November, leaked early AM 16th, and then again on the 29th. Traffic would have been at an ebb on the 16th, and rising on the 29th, so I don?t think it?s sheer volume - as you say, there must be something *in* the traffic. Or more likely, a sequence of things, otherwise I expect 2.2 would be vomiting all over my RAM far more often. > > Please let me know if there?s anything I can do to help; I got lucky with these, the first crash was the day before I started vacation (well, technically, my first day of) and the second crash was the day immediately after I returned. :) Unfortunately, when it does happen, it takes out my IDS entirely as I need to cold-boot the server, so if live diagnostics are required, it?ll have to be timed when people are around. > > Mike > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- cat ~/.bash_history > documentation.txt From jessebowling at gmail.com Thu Dec 5 07:28:58 2013 From: jessebowling at gmail.com (Jesse Bowling) Date: Thu, 5 Dec 2013 10:28:58 -0500 Subject: [Bro] out of memory after a couple days? In-Reply-To: References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> Message-ID: I have little experience with it, but could running with valgrind be possible/advisable/useful? Perhaps best to be run with -01 on a box that is overpowered for the expected load... http://www.valgrind.org/ Cheers, Jesse On Thu, Dec 5, 2013 at 10:14 AM, Mike Sconzo wrote: > If we're able to get our hands on some of the traffic (pcaps spanning > the time window of memory usage/massive drops) that causes these > issues, what would be some good tests to run against it? > > Nothing suspicious or weird shows up in the perf.log for each worker > (or manager). > > On Thu, Dec 5, 2013 at 8:12 AM, Mike Patterson > wrote: > > On Dec 4, 2013, at 11:31 PM, Seth Hall wrote: > > > >> > >> On Dec 4, 2013, at 10:34 PM, Mike Patterson < > mike.patterson at uwaterloo.ca> wrote: > >> > >>> I think you?re definitely running into a memory leak. I?ve had 2.2 > processes try to grab up to 100GB of RAM. 8 workers, 96GB of RAM, but the > box splits time with another 8 snort workers. My late 2.1 release > (september 21 IIRC) was quite a bit more stable. > >> > >> > >> I think there is some particular traffic that you guys are running into > that's causing it. A few other people have encountered that too but we > haven't been able to nail down what it is yet. > > > > That was my assumption too. I upgraded on 8 November, leaked early AM > 16th, and then again on the 29th. Traffic would have been at an ebb on the > 16th, and rising on the 29th, so I don?t think it?s sheer volume - as you > say, there must be something *in* the traffic. Or more likely, a sequence > of things, otherwise I expect 2.2 would be vomiting all over my RAM far > more often. > > > > Please let me know if there?s anything I can do to help; I got lucky > with these, the first crash was the day before I started vacation (well, > technically, my first day of) and the second crash was the day immediately > after I returned. :) Unfortunately, when it does happen, it takes out my > IDS entirely as I need to cold-boot the server, so if live diagnostics are > required, it?ll have to be timed when people are around. > > > > Mike > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -- > cat ~/.bash_history > documentation.txt > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Jesse Bowling -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131205/795e6d75/attachment.html From seth at icir.org Thu Dec 5 08:20:08 2013 From: seth at icir.org (Seth Hall) Date: Thu, 5 Dec 2013 11:20:08 -0500 Subject: [Bro] out of memory after a couple days? In-Reply-To: <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> Message-ID: On Dec 5, 2013, at 9:12 AM, Mike Patterson wrote: > That was my assumption too. I upgraded on 8 November, leaked early AM 16th, and then again on the 29th. Traffic would have been at an ebb on the 16th, and rising on the 29th, so I don?t think it?s sheer volume - as you say, there must be something *in* the traffic. Or more likely, a sequence of things, otherwise I expect 2.2 would be vomiting all over my RAM far more often. Another question I had is if you're only seeing it on a couple of worker processes or if it's all of them? That might narrow it down a bit to tell us if it's just a single connection doing something weird that is causing it or if it's something larger. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131205/794f5278/attachment.bin From mike.patterson at uwaterloo.ca Thu Dec 5 08:28:14 2013 From: mike.patterson at uwaterloo.ca (Mike Patterson) Date: Thu, 5 Dec 2013 11:28:14 -0500 Subject: [Bro] out of memory after a couple days? In-Reply-To: References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> Message-ID: On Dec 5, 2013, at 11:20 AM, Seth Hall wrote: > > On Dec 5, 2013, at 9:12 AM, Mike Patterson wrote: > >> That was my assumption too. I upgraded on 8 November, leaked early AM 16th, and then again on the 29th. Traffic would have been at an ebb on the 16th, and rising on the 29th, so I don?t think it?s sheer volume - as you say, there must be something *in* the traffic. Or more likely, a sequence of things, otherwise I expect 2.2 would be vomiting all over my RAM far more often. > > > Another question I had is if you're only seeing it on a couple of worker processes or if it's all of them? > > That might narrow it down a bit to tell us if it's just a single connection doing something weird that is causing it or if it's something larger. It was not all of them, but it was more than one, both times. I?ll take a closer count next time. Maybe ?amount of RAM I?m currently using? could be added per-worker to some stats output plugin? Mike -- Beware of "the real world". A speaker's appeal to it is always an invitation not to challenge his tacit assumptions. - Edsger W. Dijkstra From sconzo at visiblerisk.com Thu Dec 5 08:32:57 2013 From: sconzo at visiblerisk.com (Mike Sconzo) Date: Thu, 5 Dec 2013 10:32:57 -0600 Subject: [Bro] out of memory after a couple days? In-Reply-To: References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> Message-ID: Only a couple of out my 8. Past 24hrs. Bro Netowrk Summary ------------------- worker-1-1: 1386256281.234408 recvd=129751113 dropped=15 link=129751113 worker-1-2: 1386256281.438440 recvd=140506954 dropped=539300 link=140506954 worker-1-3: 1386256281.638378 recvd=117420631 dropped=1043252 link=117420631 worker-1-4: 1386256281.838171 recvd=163357938 dropped=17 link=163357938 worker-1-5: 1386256282.038370 recvd=145517241 dropped=52855 link=145517241 worker-1-6: 1386256282.238350 recvd=144958714 dropped=18 link=144958714 worker-1-7: 1386256282.438315 recvd=185940362 dropped=31 link=185940362 worker-1-8: 1386256282.638694 recvd=158251689 dropped=33170 link=158251689 On Thu, Dec 5, 2013 at 10:20 AM, Seth Hall wrote: > > On Dec 5, 2013, at 9:12 AM, Mike Patterson wrote: > >> That was my assumption too. I upgraded on 8 November, leaked early AM 16th, and then again on the 29th. Traffic would have been at an ebb on the 16th, and rising on the 29th, so I don?t think it?s sheer volume - as you say, there must be something *in* the traffic. Or more likely, a sequence of things, otherwise I expect 2.2 would be vomiting all over my RAM far more often. > > > Another question I had is if you're only seeing it on a couple of worker processes or if it's all of them? > > That might narrow it down a bit to tell us if it's just a single connection doing something weird that is causing it or if it's something larger. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- cat ~/.bash_history > documentation.txt From sconzo at visiblerisk.com Thu Dec 5 08:39:57 2013 From: sconzo at visiblerisk.com (Mike Sconzo) Date: Thu, 5 Dec 2013 10:39:57 -0600 Subject: [Bro] out of memory after a couple days? In-Reply-To: References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> Message-ID: Ooops, that's drops, here's ram. The past 24 (time since last restart) have been pretty kind so far, but still some differences. Bro Top ------- Name Type Node Pid Proc VSize Rss Cpu Cmd manager manager 130.253.254.11 5075 child 153M 36M 25% bro manager manager 130.253.254.11 5070 parent 3G 1G 4% bro proxy-1 proxy 130.253.254.11 5179 child 165M 49M 25% bro proxy-1 proxy 130.253.254.11 5131 parent 260M 157M 0% bro worker-1-1 worker 130.253.254.11 5303 parent 444M 343M 16% bro worker-1-1 worker 130.253.254.11 5313 child 199M 90M 14% bro worker-1-2 worker 130.253.254.11 5307 parent 6G 6G 45% bro worker-1-2 worker 130.253.254.11 5314 child 199M 90M 16% bro worker-1-3 worker 130.253.254.11 5326 child 199M 90M 18% bro worker-1-3 worker 130.253.254.11 5306 parent 512M 410M 16% bro worker-1-4 worker 130.253.254.11 5308 parent 562M 463M 18% bro worker-1-4 worker 130.253.254.11 5329 child 199M 90M 18% bro worker-1-5 worker 130.253.254.11 5302 parent 2G 2G 20% bro worker-1-5 worker 130.253.254.11 5318 child 199M 90M 18% bro worker-1-6 worker 130.253.254.11 5305 parent 2G 2G 22% bro worker-1-6 worker 130.253.254.11 5315 child 199M 90M 16% bro worker-1-7 worker 130.253.254.11 5304 parent 1G 1G 18% bro worker-1-7 worker 130.253.254.11 5328 child 199M 90M 18% bro worker-1-8 worker 130.253.254.11 5327 child 199M 90M 18% bro worker-1-8 worker 130.253.254.11 5301 parent 2G 2G 16% bro On Thu, Dec 5, 2013 at 10:32 AM, Mike Sconzo wrote: > Only a couple of out my 8. > > Past 24hrs. > > Bro Netowrk Summary > ------------------- > worker-1-1: 1386256281.234408 recvd=129751113 dropped=15 link=129751113 > worker-1-2: 1386256281.438440 recvd=140506954 dropped=539300 link=140506954 > worker-1-3: 1386256281.638378 recvd=117420631 dropped=1043252 link=117420631 > worker-1-4: 1386256281.838171 recvd=163357938 dropped=17 link=163357938 > worker-1-5: 1386256282.038370 recvd=145517241 dropped=52855 link=145517241 > worker-1-6: 1386256282.238350 recvd=144958714 dropped=18 link=144958714 > worker-1-7: 1386256282.438315 recvd=185940362 dropped=31 link=185940362 > worker-1-8: 1386256282.638694 recvd=158251689 dropped=33170 link=158251689 > > On Thu, Dec 5, 2013 at 10:20 AM, Seth Hall wrote: >> >> On Dec 5, 2013, at 9:12 AM, Mike Patterson wrote: >> >>> That was my assumption too. I upgraded on 8 November, leaked early AM 16th, and then again on the 29th. Traffic would have been at an ebb on the 16th, and rising on the 29th, so I don?t think it?s sheer volume - as you say, there must be something *in* the traffic. Or more likely, a sequence of things, otherwise I expect 2.2 would be vomiting all over my RAM far more often. >> >> >> Another question I had is if you're only seeing it on a couple of worker processes or if it's all of them? >> >> That might narrow it down a bit to tell us if it's just a single connection doing something weird that is causing it or if it's something larger. >> >> .Seth >> >> -- >> Seth Hall >> International Computer Science Institute >> (Bro) because everyone has a network >> http://www.bro.org/ >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -- > cat ~/.bash_history > documentation.txt -- cat ~/.bash_history > documentation.txt From gary at doit.wisc.edu Thu Dec 5 15:53:26 2013 From: gary at doit.wisc.edu (Gary Faulkner) Date: Thu, 05 Dec 2013 17:53:26 -0600 Subject: [Bro] out of memory after a couple days? In-Reply-To: References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> Message-ID: <52A111F6.7010601@doit.wisc.edu> It seems like oom-killer got invoked on one host and then then 2 minutes later on another host in my cluster. The load-balancer should be sending each host unique traffic at any given moment. Only 3 workers out of around 40 got killed, but then 3 of my proxies crashed. Cooincidentally as I was writing this I just ran out of RAM again after 24 hours of operations. Regards, Gary Faulkner UW Madison Office of Campus Information Security 608-262-8591 On 12/5/2013 10:20 AM, Seth Hall wrote: > On Dec 5, 2013, at 9:12 AM, Mike Patterson wrote: > >> That was my assumption too. I upgraded on 8 November, leaked early AM 16th, and then again on the 29th. Traffic would have been at an ebb on the 16th, and rising on the 29th, so I don?t think it?s sheer volume - as you say, there must be something *in* the traffic. Or more likely, a sequence of things, otherwise I expect 2.2 would be vomiting all over my RAM far more often. > > Another question I had is if you're only seeing it on a couple of worker processes or if it's all of them? > > That might narrow it down a bit to tell us if it's just a single connection doing something weird that is causing it or if it's something larger. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6257 bytes Desc: S/MIME Cryptographic Signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131205/28164518/attachment.bin From itxx00 at gmail.com Thu Dec 5 16:49:27 2013 From: itxx00 at gmail.com (Gao Yongwei) Date: Fri, 6 Dec 2013 08:49:27 +0800 Subject: [Bro] how to get bro to support utf-8 chars like Chinese in detect-webapps.sig ? In-Reply-To: References: <455461AF-18EE-4F0A-A7FC-8E284341054B@icir.org> Message-ID: 2013/12/5 Seth Hall > > On Dec 4, 2013, at 9:38 AM, Gao Yongwei wrote: > > > when I turn off gzip in nginx.conf , bro works fine, but when turn on > gzip, bro seems not work. > > how could bro unzip the html content before apply the payload ? thank. > > We have a specialized keyword for checking patterns in HTTP payloads. It > was decode the content prior to matching. > > http-reply-body /test\xF0\x9F\x92\xA9test/ > > Thanks Seth,it works now ! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131206/00907948/attachment.html From jwillie4020 at gmail.com Thu Dec 5 20:29:07 2013 From: jwillie4020 at gmail.com (scottie) Date: Fri, 06 Dec 2013 15:29:07 +1100 Subject: [Bro] Help with http_entity_data script Message-ID: <52A15293.9090204@gmail.com> I have this script: event http_entity_data(c: connection, is_orig: bool, length: count, data: string) { if ( c$http$method == "GET" && /\.seek\.com\.au/ in c$http$host && /^\/Jobsearch\?=/ in c$http$uri ) { print fmt("%s", c$http$host); } } It doesn't print anything running this command: bro -r seek.pcap seek.bro All i get back is a whole bunch of these(bro 2.1): ERROR: 1386300192.418090 field value missing [c$http$method] (./seek.bro, line 4) If i run the same command on my local box(bro 2.2) i get a whole bunch of these: 1386301933.432853 expression error in ./seek.bro, line 4: field value missing [c$http$method] If i remove the first bit of the if statement "c$http$method == "GET" &&" i still get errors: ERROR: 1386300192.418090 field value missing [c$http$host] (./seek.bro, line 4) If i just run it through bro with: bro -r seek.pcap local.bro In the http.log its missing the method, host, uri, referrer, and user_agent, even though the method, host and uri are clearly in the pcap: 1386301933.281650 CJa1U94yRhXVPIdPde 10.1.40.137 44393 202.58.38.95 80 0 - - - - - 0 235 302 Found - - - (empty) - - -- - FYPqLs2ezZWfT428fi text/html Been stuck on this for a while, any ideas? I've been using Scott Runnels Youtube tutorial for reference: http://ryesecurity.blogspot.com.au/2013/01/logging-youtube-titles-with-bro-1-its.html Here's the pcap if you need to try to replicate it: https://www.dropbox.com/s/tndo4cahlp7lq66/seek.pcap Scott -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131206/67eda8c2/attachment.html From seth at icir.org Thu Dec 5 20:53:19 2013 From: seth at icir.org (Seth Hall) Date: Thu, 5 Dec 2013 23:53:19 -0500 Subject: [Bro] Help with http_entity_data script In-Reply-To: <52A15293.9090204@gmail.com> References: <52A15293.9090204@gmail.com> Message-ID: <42D7D7C2-69C1-4590-A6DC-22D7E4F701AF@icir.org> On Dec 5, 2013, at 11:29 PM, scottie wrote: > In the http.log its missing the method, host, uri, referrer, and user_agent, even though the method, host and uri are clearly in the pcap: > 1386301933.281650 CJa1U94yRhXVPIdPde 10.1.40.137 44393 202.58.38.95 80 0 - - - - - 0 235 302 Found - - - (empty) - - -- - FYPqLs2ezZWfT428fi text/html I didn't try your pcap but I don't think I need to. It looks like you have bad checksums. Run Bro with the -C flag. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131205/2b824466/attachment.bin From kmcmahon at mitre.org Fri Dec 6 09:32:57 2013 From: kmcmahon at mitre.org (McMahon, Kevin J) Date: Fri, 6 Dec 2013 17:32:57 +0000 Subject: [Bro] FTP MODE Z / zlib Message-ID: <00D3CD29F7C24A44B4D23450BB8E55B30AB3E2C7@IMCMBX03.MITRE.ORG> Hello, I'm new to the list and relatively new to Bro. I've got a problem with FTP file transfers using MODE Z. Has anyone come up with an easy way to deflate/uncompress these from within Bro? I can't seem to find an internal function to perform the equivalent of "openssl zlib -d". Thanks in advance, Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131206/5df2bdf9/attachment.html From gary at doit.wisc.edu Fri Dec 6 16:38:35 2013 From: gary at doit.wisc.edu (Gary Faulkner) Date: Fri, 06 Dec 2013 18:38:35 -0600 Subject: [Bro] Sizing a Bro Cluster [was out of memory after a couple days] In-Reply-To: <52A111F6.7010601@doit.wisc.edu> References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> <52A111F6.7010601@doit.wisc.edu> Message-ID: <52A26E0B.1000005@doit.wisc.edu> I've had some proxy crashes in the past and it was suggested that I increase my number of proxies -- which I did until my environment appeared stable for about a week. After being stable for about a week I started to run out of memory, and in subsequent restarts have been running out of memory after about 24 hours of operation, typically during non-peak times (50% of normal traffic). Naturally I'm wondering if I'm just doing it wrong and if my set-up is appropriately sized and configured to handle the load I'm asking it to deal with. I think I've seen folks on the list that were running Bro on similar hardware that might be able to tell me if my configuration is anything close to what works for them. I'm also curious how other folks determine how many proxies they need, how many workers per host etc. I'm mostly running Bro 2.2 stock with default scripts, and only minor edits to local.bro to test out email notices. I'm only using these systems for Bro, although they were originally from another project so they weren't necessarily ordered with Bro specs in mind. Here's how I've got things allocated currently: Bro -1 Host: 2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) 64G RAM manager 2 proxies 20 workers 2-4GB of Traffic Bro-2 Host: 2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) 64G RAM 2 proxies 20 workers 2-4 GB of traffic The following is a relatively light traffic load (late on a Friday) for my install (4Gbps vs 8Gbps): bro-1 $ ./broctl capstats Interface kpps mbps (10s average) ------------------------------ 192.168.0.10/dnacluster:21 338.6 2327.4 192.168.0.11/dnacluster:22 324.8 2264.7 Total 663.4 4592.1 bro-1 $ ./broctl top Name Type Node Pid Proc VSize Rss Cpu Cmd manager manager 192.168.0.10 14816 parent 2G 736M 88% bro manager manager 192.168.0.10 14817 child 169M 93M 44% bro proxy-1 proxy 192.168.0.10 14863 child 102M 26M 23% bro proxy-1 proxy 192.168.0.10 14860 parent 1G 1G 3% bro proxy-2 proxy 192.168.0.10 14862 child 102M 28M 27% bro proxy-2 proxy 192.168.0.10 14861 parent 1G 1G 3% bro proxy-3 proxy 192.168.0.11 28900 child 102M 46M 20% bro proxy-3 proxy 192.168.0.11 28898 parent 1G 1G 1% bro proxy-4 proxy 192.168.0.11 28899 child 102M 45M 21% bro proxy-4 proxy 192.168.0.11 28897 parent 1G 1G 1% bro worker-1-1 worker 192.168.0.10 15228 parent 2G 2G 65% bro worker-1-1 worker 192.168.0.10 15398 child 514M 11M 10% bro worker-1-10 worker 192.168.0.10 15230 parent 2G 2G 53% bro worker-1-10 worker 192.168.0.10 15407 child 514M 12M 8% bro worker-1-11 worker 192.168.0.10 15234 parent 2G 2G 78% bro worker-1-11 worker 192.168.0.10 15286 child 514M 9M 11% bro worker-1-12 worker 192.168.0.10 15235 parent 2G 2G 67% bro worker-1-12 worker 192.168.0.10 15267 child 514M 8M 12% bro worker-1-13 worker 192.168.0.10 15237 parent 2G 2G 82% bro worker-1-13 worker 192.168.0.10 15392 child 514M 9M 12% bro worker-1-14 worker 192.168.0.10 15238 parent 2G 2G 43% bro worker-1-14 worker 192.168.0.10 15264 child 514M 11M 8% bro worker-1-15 worker 192.168.0.10 15240 parent 2G 2G 76% bro worker-1-15 worker 192.168.0.10 15300 child 514M 7M 9% bro worker-1-16 worker 192.168.0.10 15243 parent 2G 2G 94% bro worker-1-16 worker 192.168.0.10 15404 child 514M 11M 9% bro worker-1-17 worker 192.168.0.10 15244 parent 2G 2G 67% bro worker-1-17 worker 192.168.0.10 15383 child 514M 8M 8% bro worker-1-18 worker 192.168.0.10 15246 parent 2G 2G 80% bro worker-1-18 worker 192.168.0.10 15372 child 514M 12M 11% bro worker-1-19 worker 192.168.0.10 15248 parent 2G 2G 76% bro worker-1-19 worker 192.168.0.10 15376 child 514M 8M 8% bro worker-1-2 worker 192.168.0.10 15251 parent 2G 2G 83% bro worker-1-2 worker 192.168.0.10 15414 child 514M 11M 10% bro worker-1-20 worker 192.168.0.10 15254 parent 2G 2G 86% bro worker-1-20 worker 192.168.0.10 15417 child 514M 12M 11% bro worker-1-3 worker 192.168.0.10 15253 parent 2G 2G 55% bro worker-1-3 worker 192.168.0.10 15375 child 514M 8M 12% bro worker-1-4 worker 192.168.0.10 15256 parent 2G 2G 87% bro worker-1-4 worker 192.168.0.10 15388 child 515M 8M 10% bro worker-1-5 worker 192.168.0.10 15257 parent 2G 2G 58% bro worker-1-5 worker 192.168.0.10 15395 child 515M 11M 10% bro worker-1-6 worker 192.168.0.10 15258 parent 2G 2G 96% bro worker-1-6 worker 192.168.0.10 15394 child 514M 11M 8% bro worker-1-7 worker 192.168.0.10 15259 parent 2G 2G 65% bro worker-1-7 worker 192.168.0.10 15413 child 514M 12M 6% bro worker-1-8 worker 192.168.0.10 15260 parent 2G 2G 99% bro worker-1-8 worker 192.168.0.10 15401 child 514M 11M 8% bro worker-1-9 worker 192.168.0.10 15261 parent 2G 2G 61% bro worker-1-9 worker 192.168.0.10 15408 child 514M 11M 8% bro worker-2-1 worker 192.168.0.11 29961 parent 2G 2G 85% bro worker-2-1 worker 192.168.0.11 29984 child 514M 31M 9% bro worker-2-10 worker 192.168.0.11 29959 parent 2G 2G 52% bro worker-2-10 worker 192.168.0.11 30085 child 515M 31M 8% bro worker-2-11 worker 192.168.0.11 29960 parent 2G 2G 96% bro worker-2-11 worker 192.168.0.11 30112 child 514M 31M 10% bro worker-2-12 worker 192.168.0.11 29973 parent 2G 2G 54% bro worker-2-12 worker 192.168.0.11 30082 child 514M 30M 8% bro worker-2-13 worker 192.168.0.11 29967 parent 2G 2G 93% bro worker-2-13 worker 192.168.0.11 30111 child 514M 31M 10% bro worker-2-14 worker 192.168.0.11 29962 parent 2G 2G 100% bro worker-2-14 worker 192.168.0.11 30076 child 514M 30M 8% bro worker-2-15 worker 192.168.0.11 29975 parent 2G 2G 55% bro worker-2-15 worker 192.168.0.11 30138 child 514M 31M 10% bro worker-2-16 worker 192.168.0.11 29965 parent 2G 2G 85% bro worker-2-16 worker 192.168.0.11 29994 child 514M 31M 8% bro worker-2-17 worker 192.168.0.11 29968 parent 2G 2G 76% bro worker-2-17 worker 192.168.0.11 30097 child 514M 31M 8% bro worker-2-18 worker 192.168.0.11 29972 parent 2G 2G 95% bro worker-2-18 worker 192.168.0.11 30115 child 514M 30M 10% bro worker-2-19 worker 192.168.0.11 29964 parent 2G 2G 68% bro worker-2-19 worker 192.168.0.11 30092 child 514M 31M 7% bro worker-2-2 worker 192.168.0.11 29974 parent 2G 2G 51% bro worker-2-2 worker 192.168.0.11 30133 child 514M 31M 7% bro worker-2-20 worker 192.168.0.11 29966 parent 2G 2G 59% bro worker-2-20 worker 192.168.0.11 29981 child 514M 30M 10% bro worker-2-3 worker 192.168.0.11 29969 parent 2G 2G 95% bro worker-2-3 worker 192.168.0.11 30095 child 514M 31M 8% bro worker-2-4 worker 192.168.0.11 29970 parent 2G 2G 95% bro worker-2-4 worker 192.168.0.11 30137 child 514M 30M 8% bro worker-2-5 worker 192.168.0.11 29977 parent 2G 2G 84% bro worker-2-5 worker 192.168.0.11 30100 child 514M 31M 10% bro worker-2-6 worker 192.168.0.11 29978 parent 2G 2G 73% bro worker-2-6 worker 192.168.0.11 29990 child 514M 30M 8% bro worker-2-7 worker 192.168.0.11 29976 parent 2G 2G 76% bro worker-2-7 worker 192.168.0.11 30081 child 514M 31M 10% bro worker-2-8 worker 192.168.0.11 29963 parent 2G 2G 57% bro worker-2-8 worker 192.168.0.11 29987 child 514M 30M 8% bro worker-2-9 worker 192.168.0.11 29971 parent 2G 2G 52% bro worker-2-9 worker 192.168.0.11 30096 child 514M 31M 10% bro bro-1 $ free -g total used free shared buffers cached Mem: 62 62 0 0 0 17 -/+ buffers/cache: 44 17 Swap: 0 0 0 bro-2 $ free -g total used free shared buffers cached Mem: 62 45 17 0 0 1 -/+ buffers/cache: 44 18 Swap: 0 0 0 What do you guys think? Regards, Gary PS ~ I've been reading the mailing list archives and it seems that folks with the older Xeons with higher clock rates (3.4Ghzish), but fewer cores were able to handle upwards of 400-500Mbps per worker process. I've also seen it hinted, I think by Vlad G., that he was fitting in 28 workers on boxes with similar core counts to my own, but slightly faster processors. Based on some of those remarks in previous threads I was thinking I should be able to handle a little over 300Mbps per process with these processors, but I've only had the traffic to push about 200Mbps per worker so far. I know some folks also like to put the manager and possibly the proxies on separate boxes from the workers, but I haven't gotten a good sense as to what kind of workload a proxy can handle. As far as proxies I've mostly seen comments such as "I probably have way more proxies than I need" or "Just keep adding proxies until they stop crashing". I don't currently have a spare box for the manager and proxy, but would be curious to know if folks feel it is a necessity. My observations on my own setup are that my Bro workers typically are using 99% of a logical core at peak network times, and my manager 150-175% (multi-threaded). My workers seem to use about 1-2G of memory normally. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6257 bytes Desc: S/MIME Cryptographic Signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131206/8f36f9ca/attachment.bin From bob at bobthesecurityguy.com Fri Dec 6 17:21:44 2013 From: bob at bobthesecurityguy.com (Bob) Date: Fri, 06 Dec 2013 19:21:44 -0600 Subject: [Bro] Sizing a Bro Cluster [was out of memory after a couple days] In-Reply-To: <52A26E0B.1000005@doit.wisc.edu> References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> <52A111F6.7010601@doit.wisc.edu> <52A26E0B.1000005@doit.wisc.edu> Message-ID: <5a9f5d75-83a3-4a61-9212-33413690ff38@email.android.com> Gary, Realistically, for that load I would recommend looking into a cluster. My personal sizing criteria is 4-5Gb/s max per box. That's for a box that's roughly the same as yours and that uses up about 40-50GB of RAM per box (although for as cheap as it is, I recommend to always guess high on the RAM). For this size of box, you can probably improve your performance by reducing the number of workers (one per real core is a good benchmark). I am conservative, so I like to keep a couple if cores free for system tasks to ensure reliable performance (setting the number of workers to something like 12 or 14), but that's up to you. As a caveat, the aforementioned recommendations assume that you're using a network card that's designed to do this work (like an Intel ixgbe card with pf_ring or a Myricom). If your card can't bypass the kernel's interfaces, then you're going to need a lot more hardware to get the same performance because you're spending CPU time shoving the packets through the kernel instead of just accessing them directly on the NIC That's my two cents. Bob Gary Faulkner wrote: >I've had some proxy crashes in the past and it was suggested that I >increase my number of proxies -- which I did until my environment >appeared stable for about a week. After being stable for about a week I > >started to run out of memory, and in subsequent restarts have been >running out of memory after about 24 hours of operation, typically >during non-peak times (50% of normal traffic). Naturally I'm wondering >if I'm just doing it wrong and if my set-up is appropriately sized and >configured to handle the load I'm asking it to deal with. > >I think I've seen folks on the list that were running Bro on similar >hardware that might be able to tell me if my configuration is anything >close to what works for them. I'm also curious how other folks >determine >how many proxies they need, how many workers per host etc. > >I'm mostly running Bro 2.2 stock with default scripts, and only minor >edits to local.bro to test out email notices. I'm only using these >systems for Bro, although they were originally from another project so >they weren't necessarily ordered with Bro specs in mind. > >Here's how I've got things allocated currently: > >Bro -1 Host: >2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) >64G RAM >manager >2 proxies >20 workers >2-4GB of Traffic > >Bro-2 Host: >2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) >64G RAM >2 proxies >20 workers >2-4 GB of traffic > >The following is a relatively light traffic load (late on a Friday) for > >my install (4Gbps vs 8Gbps): > >bro-1 $ ./broctl capstats > >Interface kpps mbps (10s average) >------------------------------ >192.168.0.10/dnacluster:21 338.6 2327.4 >192.168.0.11/dnacluster:22 324.8 2264.7 > >Total 663.4 4592.1 > >bro-1 $ ./broctl top >Name Type Node Pid Proc VSize Rss >Cpu Cmd >manager manager 192.168.0.10 14816 parent 2G 736M >88% bro >manager manager 192.168.0.10 14817 child 169M 93M >44% bro >proxy-1 proxy 192.168.0.10 14863 child 102M 26M >23% bro >proxy-1 proxy 192.168.0.10 14860 parent 1G 1G >3% bro >proxy-2 proxy 192.168.0.10 14862 child 102M 28M >27% bro >proxy-2 proxy 192.168.0.10 14861 parent 1G 1G >3% bro >proxy-3 proxy 192.168.0.11 28900 child 102M 46M >20% bro >proxy-3 proxy 192.168.0.11 28898 parent 1G 1G >1% bro >proxy-4 proxy 192.168.0.11 28899 child 102M 45M >21% bro >proxy-4 proxy 192.168.0.11 28897 parent 1G 1G >1% bro >worker-1-1 worker 192.168.0.10 15228 parent 2G 2G >65% bro >worker-1-1 worker 192.168.0.10 15398 child 514M 11M >10% bro >worker-1-10 worker 192.168.0.10 15230 parent 2G 2G >53% bro >worker-1-10 worker 192.168.0.10 15407 child 514M 12M >8% bro >worker-1-11 worker 192.168.0.10 15234 parent 2G 2G >78% bro >worker-1-11 worker 192.168.0.10 15286 child 514M 9M >11% bro >worker-1-12 worker 192.168.0.10 15235 parent 2G 2G >67% bro >worker-1-12 worker 192.168.0.10 15267 child 514M 8M >12% bro >worker-1-13 worker 192.168.0.10 15237 parent 2G 2G >82% bro >worker-1-13 worker 192.168.0.10 15392 child 514M 9M >12% bro >worker-1-14 worker 192.168.0.10 15238 parent 2G 2G >43% bro >worker-1-14 worker 192.168.0.10 15264 child 514M 11M >8% bro >worker-1-15 worker 192.168.0.10 15240 parent 2G 2G >76% bro >worker-1-15 worker 192.168.0.10 15300 child 514M 7M >9% bro >worker-1-16 worker 192.168.0.10 15243 parent 2G 2G >94% bro >worker-1-16 worker 192.168.0.10 15404 child 514M 11M >9% bro >worker-1-17 worker 192.168.0.10 15244 parent 2G 2G >67% bro >worker-1-17 worker 192.168.0.10 15383 child 514M 8M >8% bro >worker-1-18 worker 192.168.0.10 15246 parent 2G 2G >80% bro >worker-1-18 worker 192.168.0.10 15372 child 514M 12M >11% bro >worker-1-19 worker 192.168.0.10 15248 parent 2G 2G >76% bro >worker-1-19 worker 192.168.0.10 15376 child 514M 8M >8% bro >worker-1-2 worker 192.168.0.10 15251 parent 2G 2G >83% bro >worker-1-2 worker 192.168.0.10 15414 child 514M 11M >10% bro >worker-1-20 worker 192.168.0.10 15254 parent 2G 2G >86% bro >worker-1-20 worker 192.168.0.10 15417 child 514M 12M >11% bro >worker-1-3 worker 192.168.0.10 15253 parent 2G 2G >55% bro >worker-1-3 worker 192.168.0.10 15375 child 514M 8M >12% bro >worker-1-4 worker 192.168.0.10 15256 parent 2G 2G >87% bro >worker-1-4 worker 192.168.0.10 15388 child 515M 8M >10% bro >worker-1-5 worker 192.168.0.10 15257 parent 2G 2G >58% bro >worker-1-5 worker 192.168.0.10 15395 child 515M 11M >10% bro >worker-1-6 worker 192.168.0.10 15258 parent 2G 2G >96% bro >worker-1-6 worker 192.168.0.10 15394 child 514M 11M >8% bro >worker-1-7 worker 192.168.0.10 15259 parent 2G 2G >65% bro >worker-1-7 worker 192.168.0.10 15413 child 514M 12M >6% bro >worker-1-8 worker 192.168.0.10 15260 parent 2G 2G >99% bro >worker-1-8 worker 192.168.0.10 15401 child 514M 11M >8% bro >worker-1-9 worker 192.168.0.10 15261 parent 2G 2G >61% bro >worker-1-9 worker 192.168.0.10 15408 child 514M 11M >8% bro >worker-2-1 worker 192.168.0.11 29961 parent 2G 2G >85% bro >worker-2-1 worker 192.168.0.11 29984 child 514M 31M >9% bro >worker-2-10 worker 192.168.0.11 29959 parent 2G 2G >52% bro >worker-2-10 worker 192.168.0.11 30085 child 515M 31M >8% bro >worker-2-11 worker 192.168.0.11 29960 parent 2G 2G >96% bro >worker-2-11 worker 192.168.0.11 30112 child 514M 31M >10% bro >worker-2-12 worker 192.168.0.11 29973 parent 2G 2G >54% bro >worker-2-12 worker 192.168.0.11 30082 child 514M 30M >8% bro >worker-2-13 worker 192.168.0.11 29967 parent 2G 2G >93% bro >worker-2-13 worker 192.168.0.11 30111 child 514M 31M >10% bro >worker-2-14 worker 192.168.0.11 29962 parent 2G 2G >100% bro >worker-2-14 worker 192.168.0.11 30076 child 514M 30M >8% bro >worker-2-15 worker 192.168.0.11 29975 parent 2G 2G >55% bro >worker-2-15 worker 192.168.0.11 30138 child 514M 31M >10% bro >worker-2-16 worker 192.168.0.11 29965 parent 2G 2G >85% bro >worker-2-16 worker 192.168.0.11 29994 child 514M 31M >8% bro >worker-2-17 worker 192.168.0.11 29968 parent 2G 2G >76% bro >worker-2-17 worker 192.168.0.11 30097 child 514M 31M >8% bro >worker-2-18 worker 192.168.0.11 29972 parent 2G 2G >95% bro >worker-2-18 worker 192.168.0.11 30115 child 514M 30M >10% bro >worker-2-19 worker 192.168.0.11 29964 parent 2G 2G >68% bro >worker-2-19 worker 192.168.0.11 30092 child 514M 31M >7% bro >worker-2-2 worker 192.168.0.11 29974 parent 2G 2G >51% bro >worker-2-2 worker 192.168.0.11 30133 child 514M 31M >7% bro >worker-2-20 worker 192.168.0.11 29966 parent 2G 2G >59% bro >worker-2-20 worker 192.168.0.11 29981 child 514M 30M >10% bro >worker-2-3 worker 192.168.0.11 29969 parent 2G 2G >95% bro >worker-2-3 worker 192.168.0.11 30095 child 514M 31M >8% bro >worker-2-4 worker 192.168.0.11 29970 parent 2G 2G >95% bro >worker-2-4 worker 192.168.0.11 30137 child 514M 30M >8% bro >worker-2-5 worker 192.168.0.11 29977 parent 2G 2G >84% bro >worker-2-5 worker 192.168.0.11 30100 child 514M 31M >10% bro >worker-2-6 worker 192.168.0.11 29978 parent 2G 2G >73% bro >worker-2-6 worker 192.168.0.11 29990 child 514M 30M >8% bro >worker-2-7 worker 192.168.0.11 29976 parent 2G 2G >76% bro >worker-2-7 worker 192.168.0.11 30081 child 514M 31M >10% bro >worker-2-8 worker 192.168.0.11 29963 parent 2G 2G >57% bro >worker-2-8 worker 192.168.0.11 29987 child 514M 30M >8% bro >worker-2-9 worker 192.168.0.11 29971 parent 2G 2G >52% bro >worker-2-9 worker 192.168.0.11 30096 child 514M 31M >10% bro > >bro-1 $ free -g > total used free shared buffers cached >Mem: 62 62 0 0 0 17 >-/+ buffers/cache: 44 17 >Swap: 0 0 0 > >bro-2 $ free -g > total used free shared buffers cached >Mem: 62 45 17 0 0 1 >-/+ buffers/cache: 44 18 >Swap: 0 0 0 > >What do you guys think? > >Regards, >Gary > >PS ~ > >I've been reading the mailing list archives and it seems that folks >with >the older Xeons with higher clock rates (3.4Ghzish), but fewer cores >were able to handle upwards of 400-500Mbps per worker process. I've >also >seen it hinted, I think by Vlad G., that he was fitting in 28 workers >on >boxes with similar core counts to my own, but slightly faster >processors. Based on some of those remarks in previous threads I was >thinking I should be able to handle a little over 300Mbps per process >with these processors, but I've only had the traffic to push about >200Mbps per worker so far. > >I know some folks also like to put the manager and possibly the proxies > >on separate boxes from the workers, but I haven't gotten a good sense >as >to what kind of workload a proxy can handle. As far as proxies I've >mostly seen comments such as "I probably have way more proxies than I >need" or "Just keep adding proxies until they stop crashing". I don't >currently have a spare box for the manager and proxy, but would be >curious to know if folks feel it is a necessity. My observations on my >own setup are that my Bro workers typically are using 99% of a logical >core at peak network times, and my manager 150-175% (multi-threaded). >My >workers seem to use about 1-2G of memory normally. > > > >------------------------------------------------------------------------ > >_______________________________________________ >Bro mailing list >bro at bro-ids.org >http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From gary at doit.wisc.edu Fri Dec 6 17:37:12 2013 From: gary at doit.wisc.edu (Gary Faulkner) Date: Fri, 06 Dec 2013 19:37:12 -0600 Subject: [Bro] Sizing a Bro Cluster [was out of memory after a couple days] In-Reply-To: References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> <52A111F6.7010601@doit.wisc.edu> <52A26E0B.1000005@doit.wisc.edu> Message-ID: <52A27BC8.5030402@doit.wisc.edu> Bob, Thank you for the feedback. To clarify I am currently using two physical hosts clustered together (using broctl), so each box ends up with 64G of RAM and 16cores/32 threads and Intel IXGBE 10G Cards + pf_ring/DNA/libzero for distributing packets on the host. Each physical host then sees between 2-4Gbps and has 20 workers + 2 proxies. I recall reading a blog entry by Martin Holste where he mentioned allocating only half as many workers as you had logical cores/threads, but also seem to recall others (Vlad G.) pushing nearly as many workers as logical cores, but could have read to much into it. Are you referring to physical cores or logical cores/threads? If it is the former I think what you are saying is inline with what Martin suggested; although I had hoped I could push the worker count a bit higher based on what I thought I had read elsewhere. Regards, Gary On 12/6/2013 7:08 PM, Bob wrote: > Gary, > > Realistically, for that load I would recommend looking into a cluster. My personal sizing criteria is 4-5Gb/s max per box. That's for a box that's roughly the same as yours and that uses up about 40-50GB of RAM per box (although for as cheap as it is, I recommend to always guess high on the RAM). For this size of box, you can probably improve your performance by reducing the number of workers (one per real core is a good benchmark). I am conservative, so I like to keep a couple if cores free for system tasks to ensure reliable performance (setting the number of workers to something like 12 or 14), but that's up to you. > > As a caveat, the aforementioned recommendations assume that you're using a network card that's designed to do this work (like an Intel ixgbe card with pf_ring or a Myricom). If your card can't bypass the kernel's interfaces, then you're going to need a lot more hardware to get the same performance because you're spending CPU time shoving the packets through the kernel instead of just accessing them directly on the NIC > > That's my two cents. > > Bob > > Gary Faulkner wrote: >> I've had some proxy crashes in the past and it was suggested that I >> increase my number of proxies -- which I did until my environment >> appeared stable for about a week. After being stable for about a week I >> >> started to run out of memory, and in subsequent restarts have been >> running out of memory after about 24 hours of operation, typically >> during non-peak times (50% of normal traffic). Naturally I'm wondering >> if I'm just doing it wrong and if my set-up is appropriately sized and >> configured to handle the load I'm asking it to deal with. >> >> I think I've seen folks on the list that were running Bro on similar >> hardware that might be able to tell me if my configuration is anything >> close to what works for them. I'm also curious how other folks >> determine >> how many proxies they need, how many workers per host etc. >> >> I'm mostly running Bro 2.2 stock with default scripts, and only minor >> edits to local.bro to test out email notices. I'm only using these >> systems for Bro, although they were originally from another project so >> they weren't necessarily ordered with Bro specs in mind. >> >> Here's how I've got things allocated currently: >> >> Bro -1 Host: >> 2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) >> 64G RAM >> manager >> 2 proxies >> 20 workers >> 2-4GB of Traffic >> >> Bro-2 Host: >> 2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) >> 64G RAM >> 2 proxies >> 20 workers >> 2-4 GB of traffic >> >> The following is a relatively light traffic load (late on a Friday) for >> >> my install (4Gbps vs 8Gbps): >> >> bro-1 $ ./broctl capstats >> >> Interface kpps mbps (10s average) >> ------------------------------ >> 192.168.0.10/dnacluster:21 338.6 2327.4 >> 192.168.0.11/dnacluster:22 324.8 2264.7 >> >> Total 663.4 4592.1 >> >> bro-1 $ ./broctl top >> Name Type Node Pid Proc VSize Rss >> Cpu Cmd >> manager manager 192.168.0.10 14816 parent 2G 736M >> 88% bro >> manager manager 192.168.0.10 14817 child 169M 93M >> 44% bro >> proxy-1 proxy 192.168.0.10 14863 child 102M 26M >> 23% bro >> proxy-1 proxy 192.168.0.10 14860 parent 1G 1G >> 3% bro >> proxy-2 proxy 192.168.0.10 14862 child 102M 28M >> 27% bro >> proxy-2 proxy 192.168.0.10 14861 parent 1G 1G >> 3% bro >> proxy-3 proxy 192.168.0.11 28900 child 102M 46M >> 20% bro >> proxy-3 proxy 192.168.0.11 28898 parent 1G 1G >> 1% bro >> proxy-4 proxy 192.168.0.11 28899 child 102M 45M >> 21% bro >> proxy-4 proxy 192.168.0.11 28897 parent 1G 1G >> 1% bro >> worker-1-1 worker 192.168.0.10 15228 parent 2G 2G >> 65% bro >> worker-1-1 worker 192.168.0.10 15398 child 514M 11M >> 10% bro >> worker-1-10 worker 192.168.0.10 15230 parent 2G 2G >> 53% bro >> worker-1-10 worker 192.168.0.10 15407 child 514M 12M >> 8% bro >> worker-1-11 worker 192.168.0.10 15234 parent 2G 2G >> 78% bro >> worker-1-11 worker 192.168.0.10 15286 child 514M 9M >> 11% bro >> worker-1-12 worker 192.168.0.10 15235 parent 2G 2G >> 67% bro >> worker-1-12 worker 192.168.0.10 15267 child 514M 8M >> 12% bro >> worker-1-13 worker 192.168.0.10 15237 parent 2G 2G >> 82% bro >> worker-1-13 worker 192.168.0.10 15392 child 514M 9M >> 12% bro >> worker-1-14 worker 192.168.0.10 15238 parent 2G 2G >> 43% bro >> worker-1-14 worker 192.168.0.10 15264 child 514M 11M >> 8% bro >> worker-1-15 worker 192.168.0.10 15240 parent 2G 2G >> 76% bro >> worker-1-15 worker 192.168.0.10 15300 child 514M 7M >> 9% bro >> worker-1-16 worker 192.168.0.10 15243 parent 2G 2G >> 94% bro >> worker-1-16 worker 192.168.0.10 15404 child 514M 11M >> 9% bro >> worker-1-17 worker 192.168.0.10 15244 parent 2G 2G >> 67% bro >> worker-1-17 worker 192.168.0.10 15383 child 514M 8M >> 8% bro >> worker-1-18 worker 192.168.0.10 15246 parent 2G 2G >> 80% bro >> worker-1-18 worker 192.168.0.10 15372 child 514M 12M >> 11% bro >> worker-1-19 worker 192.168.0.10 15248 parent 2G 2G >> 76% bro >> worker-1-19 worker 192.168.0.10 15376 child 514M 8M >> 8% bro >> worker-1-2 worker 192.168.0.10 15251 parent 2G 2G >> 83% bro >> worker-1-2 worker 192.168.0.10 15414 child 514M 11M >> 10% bro >> worker-1-20 worker 192.168.0.10 15254 parent 2G 2G >> 86% bro >> worker-1-20 worker 192.168.0.10 15417 child 514M 12M >> 11% bro >> worker-1-3 worker 192.168.0.10 15253 parent 2G 2G >> 55% bro >> worker-1-3 worker 192.168.0.10 15375 child 514M 8M >> 12% bro >> worker-1-4 worker 192.168.0.10 15256 parent 2G 2G >> 87% bro >> worker-1-4 worker 192.168.0.10 15388 child 515M 8M >> 10% bro >> worker-1-5 worker 192.168.0.10 15257 parent 2G 2G >> 58% bro >> worker-1-5 worker 192.168.0.10 15395 child 515M 11M >> 10% bro >> worker-1-6 worker 192.168.0.10 15258 parent 2G 2G >> 96% bro >> worker-1-6 worker 192.168.0.10 15394 child 514M 11M >> 8% bro >> worker-1-7 worker 192.168.0.10 15259 parent 2G 2G >> 65% bro >> worker-1-7 worker 192.168.0.10 15413 child 514M 12M >> 6% bro >> worker-1-8 worker 192.168.0.10 15260 parent 2G 2G >> 99% bro >> worker-1-8 worker 192.168.0.10 15401 child 514M 11M >> 8% bro >> worker-1-9 worker 192.168.0.10 15261 parent 2G 2G >> 61% bro >> worker-1-9 worker 192.168.0.10 15408 child 514M 11M >> 8% bro >> worker-2-1 worker 192.168.0.11 29961 parent 2G 2G >> 85% bro >> worker-2-1 worker 192.168.0.11 29984 child 514M 31M >> 9% bro >> worker-2-10 worker 192.168.0.11 29959 parent 2G 2G >> 52% bro >> worker-2-10 worker 192.168.0.11 30085 child 515M 31M >> 8% bro >> worker-2-11 worker 192.168.0.11 29960 parent 2G 2G >> 96% bro >> worker-2-11 worker 192.168.0.11 30112 child 514M 31M >> 10% bro >> worker-2-12 worker 192.168.0.11 29973 parent 2G 2G >> 54% bro >> worker-2-12 worker 192.168.0.11 30082 child 514M 30M >> 8% bro >> worker-2-13 worker 192.168.0.11 29967 parent 2G 2G >> 93% bro >> worker-2-13 worker 192.168.0.11 30111 child 514M 31M >> 10% bro >> worker-2-14 worker 192.168.0.11 29962 parent 2G 2G >> 100% bro >> worker-2-14 worker 192.168.0.11 30076 child 514M 30M >> 8% bro >> worker-2-15 worker 192.168.0.11 29975 parent 2G 2G >> 55% bro >> worker-2-15 worker 192.168.0.11 30138 child 514M 31M >> 10% bro >> worker-2-16 worker 192.168.0.11 29965 parent 2G 2G >> 85% bro >> worker-2-16 worker 192.168.0.11 29994 child 514M 31M >> 8% bro >> worker-2-17 worker 192.168.0.11 29968 parent 2G 2G >> 76% bro >> worker-2-17 worker 192.168.0.11 30097 child 514M 31M >> 8% bro >> worker-2-18 worker 192.168.0.11 29972 parent 2G 2G >> 95% bro >> worker-2-18 worker 192.168.0.11 30115 child 514M 30M >> 10% bro >> worker-2-19 worker 192.168.0.11 29964 parent 2G 2G >> 68% bro >> worker-2-19 worker 192.168.0.11 30092 child 514M 31M >> 7% bro >> worker-2-2 worker 192.168.0.11 29974 parent 2G 2G >> 51% bro >> worker-2-2 worker 192.168.0.11 30133 child 514M 31M >> 7% bro >> worker-2-20 worker 192.168.0.11 29966 parent 2G 2G >> 59% bro >> worker-2-20 worker 192.168.0.11 29981 child 514M 30M >> 10% bro >> worker-2-3 worker 192.168.0.11 29969 parent 2G 2G >> 95% bro >> worker-2-3 worker 192.168.0.11 30095 child 514M 31M >> 8% bro >> worker-2-4 worker 192.168.0.11 29970 parent 2G 2G >> 95% bro >> worker-2-4 worker 192.168.0.11 30137 child 514M 30M >> 8% bro >> worker-2-5 worker 192.168.0.11 29977 parent 2G 2G >> 84% bro >> worker-2-5 worker 192.168.0.11 30100 child 514M 31M >> 10% bro >> worker-2-6 worker 192.168.0.11 29978 parent 2G 2G >> 73% bro >> worker-2-6 worker 192.168.0.11 29990 child 514M 30M >> 8% bro >> worker-2-7 worker 192.168.0.11 29976 parent 2G 2G >> 76% bro >> worker-2-7 worker 192.168.0.11 30081 child 514M 31M >> 10% bro >> worker-2-8 worker 192.168.0.11 29963 parent 2G 2G >> 57% bro >> worker-2-8 worker 192.168.0.11 29987 child 514M 30M >> 8% bro >> worker-2-9 worker 192.168.0.11 29971 parent 2G 2G >> 52% bro >> worker-2-9 worker 192.168.0.11 30096 child 514M 31M >> 10% bro >> >> bro-1 $ free -g >> total used free shared buffers cached >> Mem: 62 62 0 0 0 17 >> -/+ buffers/cache: 44 17 >> Swap: 0 0 0 >> >> bro-2 $ free -g >> total used free shared buffers cached >> Mem: 62 45 17 0 0 1 >> -/+ buffers/cache: 44 18 >> Swap: 0 0 0 >> >> What do you guys think? >> >> Regards, >> Gary >> >> PS ~ >> >> I've been reading the mailing list archives and it seems that folks >> with >> the older Xeons with higher clock rates (3.4Ghzish), but fewer cores >> were able to handle upwards of 400-500Mbps per worker process. I've >> also >> seen it hinted, I think by Vlad G., that he was fitting in 28 workers >> on >> boxes with similar core counts to my own, but slightly faster >> processors. Based on some of those remarks in previous threads I was >> thinking I should be able to handle a little over 300Mbps per process >> with these processors, but I've only had the traffic to push about >> 200Mbps per worker so far. >> >> I know some folks also like to put the manager and possibly the proxies >> >> on separate boxes from the workers, but I haven't gotten a good sense >> as >> to what kind of workload a proxy can handle. As far as proxies I've >> mostly seen comments such as "I probably have way more proxies than I >> need" or "Just keep adding proxies until they stop crashing". I don't >> currently have a spare box for the manager and proxy, but would be >> curious to know if folks feel it is a necessity. My observations on my >> own setup are that my Bro workers typically are using 99% of a logical >> core at peak network times, and my manager 150-175% (multi-threaded). >> My >> workers seem to use about 1-2G of memory normally. >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6257 bytes Desc: S/MIME Cryptographic Signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131206/ba74dc66/attachment.bin From bob at bobthesecurityguy.com Fri Dec 6 17:57:30 2013 From: bob at bobthesecurityguy.com (Bob) Date: Fri, 06 Dec 2013 19:57:30 -0600 Subject: [Bro] Sizing a Bro Cluster [was out of memory after a couple days] In-Reply-To: <52A27BC8.5030402@doit.wisc.edu> References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> <52A111F6.7010601@doit.wisc.edu> <52A26E0B.1000005@doit.wisc.edu> <52A27BC8.5030402@doit.wisc.edu> Message-ID: Gary, Ah, my apologies for my misreading of your specs. Two hosts seeing 2-4Gb/s each should be just fine. As far as my recommendation, I definitely meant real cores. I find, based on my own experimentation, that taking your number of workers higher than the number of physical cores in the box tends to hurt the performance instead of helping. In the case where a box was also doing manager/proxy duty, I'd subtract a physical core for each of those as well (I've broken those out into a separate box to avoid that issue on mine). I don't recall seeing anything from Vlad suggesting going higher, but I can't rule out the possibility that I missed it. Also to answer a question that I overlooked earlier, I'm running 3 proxies on that workload. I got to that number by just increasing by one until it stabilized. Bob Gary Faulkner wrote: >Bob, > >Thank you for the feedback. To clarify I am currently using two >physical >hosts clustered together (using broctl), so each box ends up with 64G >of >RAM and 16cores/32 threads and Intel IXGBE 10G Cards + >pf_ring/DNA/libzero for distributing packets on the host. Each physical > >host then sees between 2-4Gbps and has 20 workers + 2 proxies. I recall > >reading a blog entry by Martin Holste where he mentioned allocating >only >half as many workers as you had logical cores/threads, but also seem to > >recall others (Vlad G.) pushing nearly as many workers as logical >cores, >but could have read to much into it. Are you referring to physical >cores >or logical cores/threads? If it is the former I think what you are >saying is inline with what Martin suggested; although I had hoped I >could push the worker count a bit higher based on what I thought I had >read elsewhere. > >Regards, >Gary > > On 12/6/2013 7:08 PM, Bob wrote: >> Gary, >> >> Realistically, for that load I would recommend looking into a >cluster. My personal sizing criteria is 4-5Gb/s max per box. That's >for a box that's roughly the same as yours and that uses up about >40-50GB of RAM per box (although for as cheap as it is, I recommend to >always guess high on the RAM). For this size of box, you can probably >improve your performance by reducing the number of workers (one per >real core is a good benchmark). I am conservative, so I like to keep a >couple if cores free for system tasks to ensure reliable performance >(setting the number of workers to something like 12 or 14), but that's >up to you. >> >> As a caveat, the aforementioned recommendations assume that you're >using a network card that's designed to do this work (like an Intel >ixgbe card with pf_ring or a Myricom). If your card can't bypass the >kernel's interfaces, then you're going to need a lot more hardware to >get the same performance because you're spending CPU time shoving the >packets through the kernel instead of just accessing them directly on >the NIC >> >> That's my two cents. >> >> Bob >> >> Gary Faulkner wrote: >>> I've had some proxy crashes in the past and it was suggested that I >>> increase my number of proxies -- which I did until my environment >>> appeared stable for about a week. After being stable for about a >week I >>> >>> started to run out of memory, and in subsequent restarts have been >>> running out of memory after about 24 hours of operation, typically >>> during non-peak times (50% of normal traffic). Naturally I'm >wondering >>> if I'm just doing it wrong and if my set-up is appropriately sized >and >>> configured to handle the load I'm asking it to deal with. >>> >>> I think I've seen folks on the list that were running Bro on similar >>> hardware that might be able to tell me if my configuration is >anything >>> close to what works for them. I'm also curious how other folks >>> determine >>> how many proxies they need, how many workers per host etc. >>> >>> I'm mostly running Bro 2.2 stock with default scripts, and only >minor >>> edits to local.bro to test out email notices. I'm only using these >>> systems for Bro, although they were originally from another project >so >>> they weren't necessarily ordered with Bro specs in mind. >>> >>> Here's how I've got things allocated currently: >>> >>> Bro -1 Host: >>> 2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) >>> 64G RAM >>> manager >>> 2 proxies >>> 20 workers >>> 2-4GB of Traffic >>> >>> Bro-2 Host: >>> 2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) >>> 64G RAM >>> 2 proxies >>> 20 workers >>> 2-4 GB of traffic >>> >>> The following is a relatively light traffic load (late on a Friday) >for >>> >>> my install (4Gbps vs 8Gbps): >>> >>> bro-1 $ ./broctl capstats >>> >>> Interface kpps mbps (10s average) >>> ------------------------------ >>> 192.168.0.10/dnacluster:21 338.6 2327.4 >>> 192.168.0.11/dnacluster:22 324.8 2264.7 >>> >>> Total 663.4 4592.1 >>> >>> bro-1 $ ./broctl top >>> Name Type Node Pid Proc VSize Rss >>> Cpu Cmd >>> manager manager 192.168.0.10 14816 parent 2G 736M >>> 88% bro >>> manager manager 192.168.0.10 14817 child 169M 93M >>> 44% bro >>> proxy-1 proxy 192.168.0.10 14863 child 102M 26M >>> 23% bro >>> proxy-1 proxy 192.168.0.10 14860 parent 1G 1G >>> 3% bro >>> proxy-2 proxy 192.168.0.10 14862 child 102M 28M >>> 27% bro >>> proxy-2 proxy 192.168.0.10 14861 parent 1G 1G >>> 3% bro >>> proxy-3 proxy 192.168.0.11 28900 child 102M 46M >>> 20% bro >>> proxy-3 proxy 192.168.0.11 28898 parent 1G 1G >>> 1% bro >>> proxy-4 proxy 192.168.0.11 28899 child 102M 45M >>> 21% bro >>> proxy-4 proxy 192.168.0.11 28897 parent 1G 1G >>> 1% bro >>> worker-1-1 worker 192.168.0.10 15228 parent 2G 2G >>> 65% bro >>> worker-1-1 worker 192.168.0.10 15398 child 514M 11M >>> 10% bro >>> worker-1-10 worker 192.168.0.10 15230 parent 2G 2G >>> 53% bro >>> worker-1-10 worker 192.168.0.10 15407 child 514M 12M >>> 8% bro >>> worker-1-11 worker 192.168.0.10 15234 parent 2G 2G >>> 78% bro >>> worker-1-11 worker 192.168.0.10 15286 child 514M 9M >>> 11% bro >>> worker-1-12 worker 192.168.0.10 15235 parent 2G 2G >>> 67% bro >>> worker-1-12 worker 192.168.0.10 15267 child 514M 8M >>> 12% bro >>> worker-1-13 worker 192.168.0.10 15237 parent 2G 2G >>> 82% bro >>> worker-1-13 worker 192.168.0.10 15392 child 514M 9M >>> 12% bro >>> worker-1-14 worker 192.168.0.10 15238 parent 2G 2G >>> 43% bro >>> worker-1-14 worker 192.168.0.10 15264 child 514M 11M >>> 8% bro >>> worker-1-15 worker 192.168.0.10 15240 parent 2G 2G >>> 76% bro >>> worker-1-15 worker 192.168.0.10 15300 child 514M 7M >>> 9% bro >>> worker-1-16 worker 192.168.0.10 15243 parent 2G 2G >>> 94% bro >>> worker-1-16 worker 192.168.0.10 15404 child 514M 11M >>> 9% bro >>> worker-1-17 worker 192.168.0.10 15244 parent 2G 2G >>> 67% bro >>> worker-1-17 worker 192.168.0.10 15383 child 514M 8M >>> 8% bro >>> worker-1-18 worker 192.168.0.10 15246 parent 2G 2G >>> 80% bro >>> worker-1-18 worker 192.168.0.10 15372 child 514M 12M >>> 11% bro >>> worker-1-19 worker 192.168.0.10 15248 parent 2G 2G >>> 76% bro >>> worker-1-19 worker 192.168.0.10 15376 child 514M 8M >>> 8% bro >>> worker-1-2 worker 192.168.0.10 15251 parent 2G 2G >>> 83% bro >>> worker-1-2 worker 192.168.0.10 15414 child 514M 11M >>> 10% bro >>> worker-1-20 worker 192.168.0.10 15254 parent 2G 2G >>> 86% bro >>> worker-1-20 worker 192.168.0.10 15417 child 514M 12M >>> 11% bro >>> worker-1-3 worker 192.168.0.10 15253 parent 2G 2G >>> 55% bro >>> worker-1-3 worker 192.168.0.10 15375 child 514M 8M >>> 12% bro >>> worker-1-4 worker 192.168.0.10 15256 parent 2G 2G >>> 87% bro >>> worker-1-4 worker 192.168.0.10 15388 child 515M 8M >>> 10% bro >>> worker-1-5 worker 192.168.0.10 15257 parent 2G 2G >>> 58% bro >>> worker-1-5 worker 192.168.0.10 15395 child 515M 11M >>> 10% bro >>> worker-1-6 worker 192.168.0.10 15258 parent 2G 2G >>> 96% bro >>> worker-1-6 worker 192.168.0.10 15394 child 514M 11M >>> 8% bro >>> worker-1-7 worker 192.168.0.10 15259 parent 2G 2G >>> 65% bro >>> worker-1-7 worker 192.168.0.10 15413 child 514M 12M >>> 6% bro >>> worker-1-8 worker 192.168.0.10 15260 parent 2G 2G >>> 99% bro >>> worker-1-8 worker 192.168.0.10 15401 child 514M 11M >>> 8% bro >>> worker-1-9 worker 192.168.0.10 15261 parent 2G 2G >>> 61% bro >>> worker-1-9 worker 192.168.0.10 15408 child 514M 11M >>> 8% bro >>> worker-2-1 worker 192.168.0.11 29961 parent 2G 2G >>> 85% bro >>> worker-2-1 worker 192.168.0.11 29984 child 514M 31M >>> 9% bro >>> worker-2-10 worker 192.168.0.11 29959 parent 2G 2G >>> 52% bro >>> worker-2-10 worker 192.168.0.11 30085 child 515M 31M >>> 8% bro >>> worker-2-11 worker 192.168.0.11 29960 parent 2G 2G >>> 96% bro >>> worker-2-11 worker 192.168.0.11 30112 child 514M 31M >>> 10% bro >>> worker-2-12 worker 192.168.0.11 29973 parent 2G 2G >>> 54% bro >>> worker-2-12 worker 192.168.0.11 30082 child 514M 30M >>> 8% bro >>> worker-2-13 worker 192.168.0.11 29967 parent 2G 2G >>> 93% bro >>> worker-2-13 worker 192.168.0.11 30111 child 514M 31M >>> 10% bro >>> worker-2-14 worker 192.168.0.11 29962 parent 2G 2G >>> 100% bro >>> worker-2-14 worker 192.168.0.11 30076 child 514M 30M >>> 8% bro >>> worker-2-15 worker 192.168.0.11 29975 parent 2G 2G >>> 55% bro >>> worker-2-15 worker 192.168.0.11 30138 child 514M 31M >>> 10% bro >>> worker-2-16 worker 192.168.0.11 29965 parent 2G 2G >>> 85% bro >>> worker-2-16 worker 192.168.0.11 29994 child 514M 31M >>> 8% bro >>> worker-2-17 worker 192.168.0.11 29968 parent 2G 2G >>> 76% bro >>> worker-2-17 worker 192.168.0.11 30097 child 514M 31M >>> 8% bro >>> worker-2-18 worker 192.168.0.11 29972 parent 2G 2G >>> 95% bro >>> worker-2-18 worker 192.168.0.11 30115 child 514M 30M >>> 10% bro >>> worker-2-19 worker 192.168.0.11 29964 parent 2G 2G >>> 68% bro >>> worker-2-19 worker 192.168.0.11 30092 child 514M 31M >>> 7% bro >>> worker-2-2 worker 192.168.0.11 29974 parent 2G 2G >>> 51% bro >>> worker-2-2 worker 192.168.0.11 30133 child 514M 31M >>> 7% bro >>> worker-2-20 worker 192.168.0.11 29966 parent 2G 2G >>> 59% bro >>> worker-2-20 worker 192.168.0.11 29981 child 514M 30M >>> 10% bro >>> worker-2-3 worker 192.168.0.11 29969 parent 2G 2G >>> 95% bro >>> worker-2-3 worker 192.168.0.11 30095 child 514M 31M >>> 8% bro >>> worker-2-4 worker 192.168.0.11 29970 parent 2G 2G >>> 95% bro >>> worker-2-4 worker 192.168.0.11 30137 child 514M 30M >>> 8% bro >>> worker-2-5 worker 192.168.0.11 29977 parent 2G 2G >>> 84% bro >>> worker-2-5 worker 192.168.0.11 30100 child 514M 31M >>> 10% bro >>> worker-2-6 worker 192.168.0.11 29978 parent 2G 2G >>> 73% bro >>> worker-2-6 worker 192.168.0.11 29990 child 514M 30M >>> 8% bro >>> worker-2-7 worker 192.168.0.11 29976 parent 2G 2G >>> 76% bro >>> worker-2-7 worker 192.168.0.11 30081 child 514M 31M >>> 10% bro >>> worker-2-8 worker 192.168.0.11 29963 parent 2G 2G >>> 57% bro >>> worker-2-8 worker 192.168.0.11 29987 child 514M 30M >>> 8% bro >>> worker-2-9 worker 192.168.0.11 29971 parent 2G 2G >>> 52% bro >>> worker-2-9 worker 192.168.0.11 30096 child 514M 31M >>> 10% bro >>> >>> bro-1 $ free -g >>> total used free shared buffers >cached >>> Mem: 62 62 0 0 0 17 >>> -/+ buffers/cache: 44 17 >>> Swap: 0 0 0 >>> >>> bro-2 $ free -g >>> total used free shared buffers >cached >>> Mem: 62 45 17 0 0 1 >>> -/+ buffers/cache: 44 18 >>> Swap: 0 0 0 >>> >>> What do you guys think? >>> >>> Regards, >>> Gary >>> >>> PS ~ >>> >>> I've been reading the mailing list archives and it seems that folks >>> with >>> the older Xeons with higher clock rates (3.4Ghzish), but fewer cores >>> were able to handle upwards of 400-500Mbps per worker process. I've >>> also >>> seen it hinted, I think by Vlad G., that he was fitting in 28 >workers >>> on >>> boxes with similar core counts to my own, but slightly faster >>> processors. Based on some of those remarks in previous threads I was >>> thinking I should be able to handle a little over 300Mbps per >process >>> with these processors, but I've only had the traffic to push about >>> 200Mbps per worker so far. >>> >>> I know some folks also like to put the manager and possibly the >proxies >>> >>> on separate boxes from the workers, but I haven't gotten a good >sense >>> as >>> to what kind of workload a proxy can handle. As far as proxies I've >>> mostly seen comments such as "I probably have way more proxies than >I >>> need" or "Just keep adding proxies until they stop crashing". I >don't >>> currently have a spare box for the manager and proxy, but would be >>> curious to know if folks feel it is a necessity. My observations on >my >>> own setup are that my Bro workers typically are using 99% of a >logical >>> core at peak network times, and my manager 150-175% >(multi-threaded). >>> My >>> workers seem to use about 1-2G of memory normally. >>> >>> >>> >>> >------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From gary at doit.wisc.edu Fri Dec 6 18:40:33 2013 From: gary at doit.wisc.edu (Gary Faulkner) Date: Fri, 06 Dec 2013 20:40:33 -0600 Subject: [Bro] Sizing a Bro Cluster [was out of memory after a couple days] In-Reply-To: References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> <52A111F6.7010601@doit.wisc.edu> <52A26E0B.1000005@doit.wisc.edu> <52A27BC8.5030402@doit.wisc.edu> Message-ID: <52A28AA1.4090103@doit.wisc.edu> Bob, So, probably closer to 10-12 workers per host in my case since the proxies and manager are there? It was suggested to me early on to try to acquire another box to separate from manager and proxies, but I don't have one quite yet so I've been trying to make it work as is. It didn't seem like the worker child processes needed much CPU time, so I thought I could push the worker count higher and it also seemed like I got less loss per broctl netstat, but others have suggested that maybe the broctl netstat command wasn't the most reliable way to judge that. I ended up at 4 proxies mostly because two didn't seem stable and I like symmetry so I jumped straight to 4. Thanks again, Gary On 12/6/2013 7:57 PM, Bob wrote: > Gary, > > Ah, my apologies for my misreading of your specs. Two hosts seeing 2-4Gb/s each should be just fine. > > As far as my recommendation, I definitely meant real cores. I find, based on my own experimentation, that taking your number of workers higher than the number of physical cores in the box tends to hurt the performance instead of helping. In the case where a box was also doing manager/proxy duty, I'd subtract a physical core for each of those as well (I've broken those out into a separate box to avoid that issue on mine). I don't recall seeing anything from Vlad suggesting going higher, but I can't rule out the possibility that I missed it. > > Also to answer a question that I overlooked earlier, I'm running 3 proxies on that workload. I got to that number by just increasing by one until it stabilized. > > Bob > > Gary Faulkner wrote: >> Bob, >> >> Thank you for the feedback. To clarify I am currently using two >> physical >> hosts clustered together (using broctl), so each box ends up with 64G >> of >> RAM and 16cores/32 threads and Intel IXGBE 10G Cards + >> pf_ring/DNA/libzero for distributing packets on the host. Each physical >> >> host then sees between 2-4Gbps and has 20 workers + 2 proxies. I recall >> >> reading a blog entry by Martin Holste where he mentioned allocating >> only >> half as many workers as you had logical cores/threads, but also seem to >> >> recall others (Vlad G.) pushing nearly as many workers as logical >> cores, >> but could have read to much into it. Are you referring to physical >> cores >> or logical cores/threads? If it is the former I think what you are >> saying is inline with what Martin suggested; although I had hoped I >> could push the worker count a bit higher based on what I thought I had >> read elsewhere. >> >> Regards, >> Gary >> >> On 12/6/2013 7:08 PM, Bob wrote: >>> Gary, >>> >>> Realistically, for that load I would recommend looking into a >> cluster. My personal sizing criteria is 4-5Gb/s max per box. That's >> for a box that's roughly the same as yours and that uses up about >> 40-50GB of RAM per box (although for as cheap as it is, I recommend to >> always guess high on the RAM). For this size of box, you can probably >> improve your performance by reducing the number of workers (one per >> real core is a good benchmark). I am conservative, so I like to keep a >> couple if cores free for system tasks to ensure reliable performance >> (setting the number of workers to something like 12 or 14), but that's >> up to you. >>> >>> As a caveat, the aforementioned recommendations assume that you're >> using a network card that's designed to do this work (like an Intel >> ixgbe card with pf_ring or a Myricom). If your card can't bypass the >> kernel's interfaces, then you're going to need a lot more hardware to >> get the same performance because you're spending CPU time shoving the >> packets through the kernel instead of just accessing them directly on >> the NIC >>> >>> That's my two cents. >>> >>> Bob >>> >>> Gary Faulkner wrote: >>>> I've had some proxy crashes in the past and it was suggested that I >>>> increase my number of proxies -- which I did until my environment >>>> appeared stable for about a week. After being stable for about a >> week I >>>> >>>> started to run out of memory, and in subsequent restarts have been >>>> running out of memory after about 24 hours of operation, typically >>>> during non-peak times (50% of normal traffic). Naturally I'm >> wondering >>>> if I'm just doing it wrong and if my set-up is appropriately sized >> and >>>> configured to handle the load I'm asking it to deal with. >>>> >>>> I think I've seen folks on the list that were running Bro on similar >>>> hardware that might be able to tell me if my configuration is >> anything >>>> close to what works for them. I'm also curious how other folks >>>> determine >>>> how many proxies they need, how many workers per host etc. >>>> >>>> I'm mostly running Bro 2.2 stock with default scripts, and only >> minor >>>> edits to local.bro to test out email notices. I'm only using these >>>> systems for Bro, although they were originally from another project >> so >>>> they weren't necessarily ordered with Bro specs in mind. >>>> >>>> Here's how I've got things allocated currently: >>>> >>>> Bro -1 Host: >>>> 2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) >>>> 64G RAM >>>> manager >>>> 2 proxies >>>> 20 workers >>>> 2-4GB of Traffic >>>> >>>> Bro-2 Host: >>>> 2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) >>>> 64G RAM >>>> 2 proxies >>>> 20 workers >>>> 2-4 GB of traffic >>>> >>>> The following is a relatively light traffic load (late on a Friday) >> for >>>> >>>> my install (4Gbps vs 8Gbps): >>>> >>>> bro-1 $ ./broctl capstats >>>> >>>> Interface kpps mbps (10s average) >>>> ------------------------------ >>>> 192.168.0.10/dnacluster:21 338.6 2327.4 >>>> 192.168.0.11/dnacluster:22 324.8 2264.7 >>>> >>>> Total 663.4 4592.1 >>>> >>>> bro-1 $ ./broctl top >>>> Name Type Node Pid Proc VSize Rss >>>> Cpu Cmd >>>> manager manager 192.168.0.10 14816 parent 2G 736M >>>> 88% bro >>>> manager manager 192.168.0.10 14817 child 169M 93M >>>> 44% bro >>>> proxy-1 proxy 192.168.0.10 14863 child 102M 26M >>>> 23% bro >>>> proxy-1 proxy 192.168.0.10 14860 parent 1G 1G >>>> 3% bro >>>> proxy-2 proxy 192.168.0.10 14862 child 102M 28M >>>> 27% bro >>>> proxy-2 proxy 192.168.0.10 14861 parent 1G 1G >>>> 3% bro >>>> proxy-3 proxy 192.168.0.11 28900 child 102M 46M >>>> 20% bro >>>> proxy-3 proxy 192.168.0.11 28898 parent 1G 1G >>>> 1% bro >>>> proxy-4 proxy 192.168.0.11 28899 child 102M 45M >>>> 21% bro >>>> proxy-4 proxy 192.168.0.11 28897 parent 1G 1G >>>> 1% bro >>>> worker-1-1 worker 192.168.0.10 15228 parent 2G 2G >>>> 65% bro >>>> worker-1-1 worker 192.168.0.10 15398 child 514M 11M >>>> 10% bro >>>> worker-1-10 worker 192.168.0.10 15230 parent 2G 2G >>>> 53% bro >>>> worker-1-10 worker 192.168.0.10 15407 child 514M 12M >>>> 8% bro >>>> worker-1-11 worker 192.168.0.10 15234 parent 2G 2G >>>> 78% bro >>>> worker-1-11 worker 192.168.0.10 15286 child 514M 9M >>>> 11% bro >>>> worker-1-12 worker 192.168.0.10 15235 parent 2G 2G >>>> 67% bro >>>> worker-1-12 worker 192.168.0.10 15267 child 514M 8M >>>> 12% bro >>>> worker-1-13 worker 192.168.0.10 15237 parent 2G 2G >>>> 82% bro >>>> worker-1-13 worker 192.168.0.10 15392 child 514M 9M >>>> 12% bro >>>> worker-1-14 worker 192.168.0.10 15238 parent 2G 2G >>>> 43% bro >>>> worker-1-14 worker 192.168.0.10 15264 child 514M 11M >>>> 8% bro >>>> worker-1-15 worker 192.168.0.10 15240 parent 2G 2G >>>> 76% bro >>>> worker-1-15 worker 192.168.0.10 15300 child 514M 7M >>>> 9% bro >>>> worker-1-16 worker 192.168.0.10 15243 parent 2G 2G >>>> 94% bro >>>> worker-1-16 worker 192.168.0.10 15404 child 514M 11M >>>> 9% bro >>>> worker-1-17 worker 192.168.0.10 15244 parent 2G 2G >>>> 67% bro >>>> worker-1-17 worker 192.168.0.10 15383 child 514M 8M >>>> 8% bro >>>> worker-1-18 worker 192.168.0.10 15246 parent 2G 2G >>>> 80% bro >>>> worker-1-18 worker 192.168.0.10 15372 child 514M 12M >>>> 11% bro >>>> worker-1-19 worker 192.168.0.10 15248 parent 2G 2G >>>> 76% bro >>>> worker-1-19 worker 192.168.0.10 15376 child 514M 8M >>>> 8% bro >>>> worker-1-2 worker 192.168.0.10 15251 parent 2G 2G >>>> 83% bro >>>> worker-1-2 worker 192.168.0.10 15414 child 514M 11M >>>> 10% bro >>>> worker-1-20 worker 192.168.0.10 15254 parent 2G 2G >>>> 86% bro >>>> worker-1-20 worker 192.168.0.10 15417 child 514M 12M >>>> 11% bro >>>> worker-1-3 worker 192.168.0.10 15253 parent 2G 2G >>>> 55% bro >>>> worker-1-3 worker 192.168.0.10 15375 child 514M 8M >>>> 12% bro >>>> worker-1-4 worker 192.168.0.10 15256 parent 2G 2G >>>> 87% bro >>>> worker-1-4 worker 192.168.0.10 15388 child 515M 8M >>>> 10% bro >>>> worker-1-5 worker 192.168.0.10 15257 parent 2G 2G >>>> 58% bro >>>> worker-1-5 worker 192.168.0.10 15395 child 515M 11M >>>> 10% bro >>>> worker-1-6 worker 192.168.0.10 15258 parent 2G 2G >>>> 96% bro >>>> worker-1-6 worker 192.168.0.10 15394 child 514M 11M >>>> 8% bro >>>> worker-1-7 worker 192.168.0.10 15259 parent 2G 2G >>>> 65% bro >>>> worker-1-7 worker 192.168.0.10 15413 child 514M 12M >>>> 6% bro >>>> worker-1-8 worker 192.168.0.10 15260 parent 2G 2G >>>> 99% bro >>>> worker-1-8 worker 192.168.0.10 15401 child 514M 11M >>>> 8% bro >>>> worker-1-9 worker 192.168.0.10 15261 parent 2G 2G >>>> 61% bro >>>> worker-1-9 worker 192.168.0.10 15408 child 514M 11M >>>> 8% bro >>>> worker-2-1 worker 192.168.0.11 29961 parent 2G 2G >>>> 85% bro >>>> worker-2-1 worker 192.168.0.11 29984 child 514M 31M >>>> 9% bro >>>> worker-2-10 worker 192.168.0.11 29959 parent 2G 2G >>>> 52% bro >>>> worker-2-10 worker 192.168.0.11 30085 child 515M 31M >>>> 8% bro >>>> worker-2-11 worker 192.168.0.11 29960 parent 2G 2G >>>> 96% bro >>>> worker-2-11 worker 192.168.0.11 30112 child 514M 31M >>>> 10% bro >>>> worker-2-12 worker 192.168.0.11 29973 parent 2G 2G >>>> 54% bro >>>> worker-2-12 worker 192.168.0.11 30082 child 514M 30M >>>> 8% bro >>>> worker-2-13 worker 192.168.0.11 29967 parent 2G 2G >>>> 93% bro >>>> worker-2-13 worker 192.168.0.11 30111 child 514M 31M >>>> 10% bro >>>> worker-2-14 worker 192.168.0.11 29962 parent 2G 2G >>>> 100% bro >>>> worker-2-14 worker 192.168.0.11 30076 child 514M 30M >>>> 8% bro >>>> worker-2-15 worker 192.168.0.11 29975 parent 2G 2G >>>> 55% bro >>>> worker-2-15 worker 192.168.0.11 30138 child 514M 31M >>>> 10% bro >>>> worker-2-16 worker 192.168.0.11 29965 parent 2G 2G >>>> 85% bro >>>> worker-2-16 worker 192.168.0.11 29994 child 514M 31M >>>> 8% bro >>>> worker-2-17 worker 192.168.0.11 29968 parent 2G 2G >>>> 76% bro >>>> worker-2-17 worker 192.168.0.11 30097 child 514M 31M >>>> 8% bro >>>> worker-2-18 worker 192.168.0.11 29972 parent 2G 2G >>>> 95% bro >>>> worker-2-18 worker 192.168.0.11 30115 child 514M 30M >>>> 10% bro >>>> worker-2-19 worker 192.168.0.11 29964 parent 2G 2G >>>> 68% bro >>>> worker-2-19 worker 192.168.0.11 30092 child 514M 31M >>>> 7% bro >>>> worker-2-2 worker 192.168.0.11 29974 parent 2G 2G >>>> 51% bro >>>> worker-2-2 worker 192.168.0.11 30133 child 514M 31M >>>> 7% bro >>>> worker-2-20 worker 192.168.0.11 29966 parent 2G 2G >>>> 59% bro >>>> worker-2-20 worker 192.168.0.11 29981 child 514M 30M >>>> 10% bro >>>> worker-2-3 worker 192.168.0.11 29969 parent 2G 2G >>>> 95% bro >>>> worker-2-3 worker 192.168.0.11 30095 child 514M 31M >>>> 8% bro >>>> worker-2-4 worker 192.168.0.11 29970 parent 2G 2G >>>> 95% bro >>>> worker-2-4 worker 192.168.0.11 30137 child 514M 30M >>>> 8% bro >>>> worker-2-5 worker 192.168.0.11 29977 parent 2G 2G >>>> 84% bro >>>> worker-2-5 worker 192.168.0.11 30100 child 514M 31M >>>> 10% bro >>>> worker-2-6 worker 192.168.0.11 29978 parent 2G 2G >>>> 73% bro >>>> worker-2-6 worker 192.168.0.11 29990 child 514M 30M >>>> 8% bro >>>> worker-2-7 worker 192.168.0.11 29976 parent 2G 2G >>>> 76% bro >>>> worker-2-7 worker 192.168.0.11 30081 child 514M 31M >>>> 10% bro >>>> worker-2-8 worker 192.168.0.11 29963 parent 2G 2G >>>> 57% bro >>>> worker-2-8 worker 192.168.0.11 29987 child 514M 30M >>>> 8% bro >>>> worker-2-9 worker 192.168.0.11 29971 parent 2G 2G >>>> 52% bro >>>> worker-2-9 worker 192.168.0.11 30096 child 514M 31M >>>> 10% bro >>>> >>>> bro-1 $ free -g >>>> total used free shared buffers >> cached >>>> Mem: 62 62 0 0 0 17 >>>> -/+ buffers/cache: 44 17 >>>> Swap: 0 0 0 >>>> >>>> bro-2 $ free -g >>>> total used free shared buffers >> cached >>>> Mem: 62 45 17 0 0 1 >>>> -/+ buffers/cache: 44 18 >>>> Swap: 0 0 0 >>>> >>>> What do you guys think? >>>> >>>> Regards, >>>> Gary >>>> >>>> PS ~ >>>> >>>> I've been reading the mailing list archives and it seems that folks >>>> with >>>> the older Xeons with higher clock rates (3.4Ghzish), but fewer cores >>>> were able to handle upwards of 400-500Mbps per worker process. I've >>>> also >>>> seen it hinted, I think by Vlad G., that he was fitting in 28 >> workers >>>> on >>>> boxes with similar core counts to my own, but slightly faster >>>> processors. Based on some of those remarks in previous threads I was >>>> thinking I should be able to handle a little over 300Mbps per >> process >>>> with these processors, but I've only had the traffic to push about >>>> 200Mbps per worker so far. >>>> >>>> I know some folks also like to put the manager and possibly the >> proxies >>>> >>>> on separate boxes from the workers, but I haven't gotten a good >> sense >>>> as >>>> to what kind of workload a proxy can handle. As far as proxies I've >>>> mostly seen comments such as "I probably have way more proxies than >> I >>>> need" or "Just keep adding proxies until they stop crashing". I >> don't >>>> currently have a spare box for the manager and proxy, but would be >>>> curious to know if folks feel it is a necessity. My observations on >> my >>>> own setup are that my Bro workers typically are using 99% of a >> logical >>>> core at peak network times, and my manager 150-175% >> (multi-threaded). >>>> My >>>> workers seem to use about 1-2G of memory normally. >>>> >>>> >>>> >>>> >> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> Bro mailing list >>>> bro at bro-ids.org >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From bob at bobthesecurityguy.com Fri Dec 6 19:12:42 2013 From: bob at bobthesecurityguy.com (Bob) Date: Fri, 06 Dec 2013 21:12:42 -0600 Subject: [Bro] Sizing a Bro Cluster [was out of memory after a couple days] In-Reply-To: <52A28AA1.4090103@doit.wisc.edu> References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> <52A111F6.7010601@doit.wisc.edu> <52A26E0B.1000005@doit.wisc.edu> <52A27BC8.5030402@doit.wisc.edu> <52A28AA1.4090103@doit.wisc.edu> Message-ID: Twelve per box would be my plan. And to be honest, once you lower the number of workers, you could probably reduce the number of proxies as well. I'm running 3 for a cluster with 72 active workers, so you could probably do 24 with just one (and you could even put it on the non-manager box to maintain some symmetry). As always though, feel free to experiment with that to find what works best for you. Bob Gary Faulkner wrote: >Bob, > >So, probably closer to 10-12 workers per host in my case since the >proxies and manager are there? It was suggested to me early on to try >to >acquire another box to separate from manager and proxies, but I don't >have one quite yet so I've been trying to make it work as is. It didn't > >seem like the worker child processes needed much CPU time, so I thought > >I could push the worker count higher and it also seemed like I got less > >loss per broctl netstat, but others have suggested that maybe the >broctl >netstat command wasn't the most reliable way to judge that. I ended up >at 4 proxies mostly because two didn't seem stable and I like symmetry >so I jumped straight to 4. > >Thanks again, >Gary > >On 12/6/2013 7:57 PM, Bob wrote: >> Gary, >> >> Ah, my apologies for my misreading of your specs. Two hosts seeing >2-4Gb/s each should be just fine. >> >> As far as my recommendation, I definitely meant real cores. I find, >based on my own experimentation, that taking your number of workers >higher than the number of physical cores in the box tends to hurt the >performance instead of helping. In the case where a box was also doing >manager/proxy duty, I'd subtract a physical core for each of those as >well (I've broken those out into a separate box to avoid that issue on >mine). I don't recall seeing anything from Vlad suggesting going >higher, but I can't rule out the possibility that I missed it. >> >> Also to answer a question that I overlooked earlier, I'm running 3 >proxies on that workload. I got to that number by just increasing by >one until it stabilized. >> >> Bob >> >> Gary Faulkner wrote: >>> Bob, >>> >>> Thank you for the feedback. To clarify I am currently using two >>> physical >>> hosts clustered together (using broctl), so each box ends up with >64G >>> of >>> RAM and 16cores/32 threads and Intel IXGBE 10G Cards + >>> pf_ring/DNA/libzero for distributing packets on the host. Each >physical >>> >>> host then sees between 2-4Gbps and has 20 workers + 2 proxies. I >recall >>> >>> reading a blog entry by Martin Holste where he mentioned allocating >>> only >>> half as many workers as you had logical cores/threads, but also seem >to >>> >>> recall others (Vlad G.) pushing nearly as many workers as logical >>> cores, >>> but could have read to much into it. Are you referring to physical >>> cores >>> or logical cores/threads? If it is the former I think what you are >>> saying is inline with what Martin suggested; although I had hoped I >>> could push the worker count a bit higher based on what I thought I >had >>> read elsewhere. >>> >>> Regards, >>> Gary >>> >>> On 12/6/2013 7:08 PM, Bob wrote: >>>> Gary, >>>> >>>> Realistically, for that load I would recommend looking into a >>> cluster. My personal sizing criteria is 4-5Gb/s max per box. >That's >>> for a box that's roughly the same as yours and that uses up about >>> 40-50GB of RAM per box (although for as cheap as it is, I recommend >to >>> always guess high on the RAM). For this size of box, you can >probably >>> improve your performance by reducing the number of workers (one per >>> real core is a good benchmark). I am conservative, so I like to >keep a >>> couple if cores free for system tasks to ensure reliable performance >>> (setting the number of workers to something like 12 or 14), but >that's >>> up to you. >>>> >>>> As a caveat, the aforementioned recommendations assume that you're >>> using a network card that's designed to do this work (like an Intel >>> ixgbe card with pf_ring or a Myricom). If your card can't bypass >the >>> kernel's interfaces, then you're going to need a lot more hardware >to >>> get the same performance because you're spending CPU time shoving >the >>> packets through the kernel instead of just accessing them directly >on >>> the NIC >>>> >>>> That's my two cents. >>>> >>>> Bob >>>> >>>> Gary Faulkner wrote: >>>>> I've had some proxy crashes in the past and it was suggested that >I >>>>> increase my number of proxies -- which I did until my environment >>>>> appeared stable for about a week. After being stable for about a >>> week I >>>>> >>>>> started to run out of memory, and in subsequent restarts have been >>>>> running out of memory after about 24 hours of operation, typically >>>>> during non-peak times (50% of normal traffic). Naturally I'm >>> wondering >>>>> if I'm just doing it wrong and if my set-up is appropriately sized >>> and >>>>> configured to handle the load I'm asking it to deal with. >>>>> >>>>> I think I've seen folks on the list that were running Bro on >similar >>>>> hardware that might be able to tell me if my configuration is >>> anything >>>>> close to what works for them. I'm also curious how other folks >>>>> determine >>>>> how many proxies they need, how many workers per host etc. >>>>> >>>>> I'm mostly running Bro 2.2 stock with default scripts, and only >>> minor >>>>> edits to local.bro to test out email notices. I'm only using these >>>>> systems for Bro, although they were originally from another >project >>> so >>>>> they weren't necessarily ordered with Bro specs in mind. >>>>> >>>>> Here's how I've got things allocated currently: >>>>> >>>>> Bro -1 Host: >>>>> 2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) >>>>> 64G RAM >>>>> manager >>>>> 2 proxies >>>>> 20 workers >>>>> 2-4GB of Traffic >>>>> >>>>> Bro-2 Host: >>>>> 2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) >>>>> 64G RAM >>>>> 2 proxies >>>>> 20 workers >>>>> 2-4 GB of traffic >>>>> >>>>> The following is a relatively light traffic load (late on a >Friday) >>> for >>>>> >>>>> my install (4Gbps vs 8Gbps): >>>>> >>>>> bro-1 $ ./broctl capstats >>>>> >>>>> Interface kpps mbps (10s average) >>>>> ------------------------------ >>>>> 192.168.0.10/dnacluster:21 338.6 2327.4 >>>>> 192.168.0.11/dnacluster:22 324.8 2264.7 >>>>> >>>>> Total 663.4 4592.1 >>>>> >>>>> bro-1 $ ./broctl top >>>>> Name Type Node Pid Proc VSize Rss >>>>> Cpu Cmd >>>>> manager manager 192.168.0.10 14816 parent 2G 736M >>>>> 88% bro >>>>> manager manager 192.168.0.10 14817 child 169M 93M >>>>> 44% bro >>>>> proxy-1 proxy 192.168.0.10 14863 child 102M 26M >>>>> 23% bro >>>>> proxy-1 proxy 192.168.0.10 14860 parent 1G 1G >>>>> 3% bro >>>>> proxy-2 proxy 192.168.0.10 14862 child 102M 28M >>>>> 27% bro >>>>> proxy-2 proxy 192.168.0.10 14861 parent 1G 1G >>>>> 3% bro >>>>> proxy-3 proxy 192.168.0.11 28900 child 102M 46M >>>>> 20% bro >>>>> proxy-3 proxy 192.168.0.11 28898 parent 1G 1G >>>>> 1% bro >>>>> proxy-4 proxy 192.168.0.11 28899 child 102M 45M >>>>> 21% bro >>>>> proxy-4 proxy 192.168.0.11 28897 parent 1G 1G >>>>> 1% bro >>>>> worker-1-1 worker 192.168.0.10 15228 parent 2G 2G >>>>> 65% bro >>>>> worker-1-1 worker 192.168.0.10 15398 child 514M 11M >>>>> 10% bro >>>>> worker-1-10 worker 192.168.0.10 15230 parent 2G 2G >>>>> 53% bro >>>>> worker-1-10 worker 192.168.0.10 15407 child 514M 12M >>>>> 8% bro >>>>> worker-1-11 worker 192.168.0.10 15234 parent 2G 2G >>>>> 78% bro >>>>> worker-1-11 worker 192.168.0.10 15286 child 514M 9M >>>>> 11% bro >>>>> worker-1-12 worker 192.168.0.10 15235 parent 2G 2G >>>>> 67% bro >>>>> worker-1-12 worker 192.168.0.10 15267 child 514M 8M >>>>> 12% bro >>>>> worker-1-13 worker 192.168.0.10 15237 parent 2G 2G >>>>> 82% bro >>>>> worker-1-13 worker 192.168.0.10 15392 child 514M 9M >>>>> 12% bro >>>>> worker-1-14 worker 192.168.0.10 15238 parent 2G 2G >>>>> 43% bro >>>>> worker-1-14 worker 192.168.0.10 15264 child 514M 11M >>>>> 8% bro >>>>> worker-1-15 worker 192.168.0.10 15240 parent 2G 2G >>>>> 76% bro >>>>> worker-1-15 worker 192.168.0.10 15300 child 514M 7M >>>>> 9% bro >>>>> worker-1-16 worker 192.168.0.10 15243 parent 2G 2G >>>>> 94% bro >>>>> worker-1-16 worker 192.168.0.10 15404 child 514M 11M >>>>> 9% bro >>>>> worker-1-17 worker 192.168.0.10 15244 parent 2G 2G >>>>> 67% bro >>>>> worker-1-17 worker 192.168.0.10 15383 child 514M 8M >>>>> 8% bro >>>>> worker-1-18 worker 192.168.0.10 15246 parent 2G 2G >>>>> 80% bro >>>>> worker-1-18 worker 192.168.0.10 15372 child 514M 12M >>>>> 11% bro >>>>> worker-1-19 worker 192.168.0.10 15248 parent 2G 2G >>>>> 76% bro >>>>> worker-1-19 worker 192.168.0.10 15376 child 514M 8M >>>>> 8% bro >>>>> worker-1-2 worker 192.168.0.10 15251 parent 2G 2G >>>>> 83% bro >>>>> worker-1-2 worker 192.168.0.10 15414 child 514M 11M >>>>> 10% bro >>>>> worker-1-20 worker 192.168.0.10 15254 parent 2G 2G >>>>> 86% bro >>>>> worker-1-20 worker 192.168.0.10 15417 child 514M 12M >>>>> 11% bro >>>>> worker-1-3 worker 192.168.0.10 15253 parent 2G 2G >>>>> 55% bro >>>>> worker-1-3 worker 192.168.0.10 15375 child 514M 8M >>>>> 12% bro >>>>> worker-1-4 worker 192.168.0.10 15256 parent 2G 2G >>>>> 87% bro >>>>> worker-1-4 worker 192.168.0.10 15388 child 515M 8M >>>>> 10% bro >>>>> worker-1-5 worker 192.168.0.10 15257 parent 2G 2G >>>>> 58% bro >>>>> worker-1-5 worker 192.168.0.10 15395 child 515M 11M >>>>> 10% bro >>>>> worker-1-6 worker 192.168.0.10 15258 parent 2G 2G >>>>> 96% bro >>>>> worker-1-6 worker 192.168.0.10 15394 child 514M 11M >>>>> 8% bro >>>>> worker-1-7 worker 192.168.0.10 15259 parent 2G 2G >>>>> 65% bro >>>>> worker-1-7 worker 192.168.0.10 15413 child 514M 12M >>>>> 6% bro >>>>> worker-1-8 worker 192.168.0.10 15260 parent 2G 2G >>>>> 99% bro >>>>> worker-1-8 worker 192.168.0.10 15401 child 514M 11M >>>>> 8% bro >>>>> worker-1-9 worker 192.168.0.10 15261 parent 2G 2G >>>>> 61% bro >>>>> worker-1-9 worker 192.168.0.10 15408 child 514M 11M >>>>> 8% bro >>>>> worker-2-1 worker 192.168.0.11 29961 parent 2G 2G >>>>> 85% bro >>>>> worker-2-1 worker 192.168.0.11 29984 child 514M 31M >>>>> 9% bro >>>>> worker-2-10 worker 192.168.0.11 29959 parent 2G 2G >>>>> 52% bro >>>>> worker-2-10 worker 192.168.0.11 30085 child 515M 31M >>>>> 8% bro >>>>> worker-2-11 worker 192.168.0.11 29960 parent 2G 2G >>>>> 96% bro >>>>> worker-2-11 worker 192.168.0.11 30112 child 514M 31M >>>>> 10% bro >>>>> worker-2-12 worker 192.168.0.11 29973 parent 2G 2G >>>>> 54% bro >>>>> worker-2-12 worker 192.168.0.11 30082 child 514M 30M >>>>> 8% bro >>>>> worker-2-13 worker 192.168.0.11 29967 parent 2G 2G >>>>> 93% bro >>>>> worker-2-13 worker 192.168.0.11 30111 child 514M 31M >>>>> 10% bro >>>>> worker-2-14 worker 192.168.0.11 29962 parent 2G 2G >>>>> 100% bro >>>>> worker-2-14 worker 192.168.0.11 30076 child 514M 30M >>>>> 8% bro >>>>> worker-2-15 worker 192.168.0.11 29975 parent 2G 2G >>>>> 55% bro >>>>> worker-2-15 worker 192.168.0.11 30138 child 514M 31M >>>>> 10% bro >>>>> worker-2-16 worker 192.168.0.11 29965 parent 2G 2G >>>>> 85% bro >>>>> worker-2-16 worker 192.168.0.11 29994 child 514M 31M >>>>> 8% bro >>>>> worker-2-17 worker 192.168.0.11 29968 parent 2G 2G >>>>> 76% bro >>>>> worker-2-17 worker 192.168.0.11 30097 child 514M 31M >>>>> 8% bro >>>>> worker-2-18 worker 192.168.0.11 29972 parent 2G 2G >>>>> 95% bro >>>>> worker-2-18 worker 192.168.0.11 30115 child 514M 30M >>>>> 10% bro >>>>> worker-2-19 worker 192.168.0.11 29964 parent 2G 2G >>>>> 68% bro >>>>> worker-2-19 worker 192.168.0.11 30092 child 514M 31M >>>>> 7% bro >>>>> worker-2-2 worker 192.168.0.11 29974 parent 2G 2G >>>>> 51% bro >>>>> worker-2-2 worker 192.168.0.11 30133 child 514M 31M >>>>> 7% bro >>>>> worker-2-20 worker 192.168.0.11 29966 parent 2G 2G >>>>> 59% bro >>>>> worker-2-20 worker 192.168.0.11 29981 child 514M 30M >>>>> 10% bro >>>>> worker-2-3 worker 192.168.0.11 29969 parent 2G 2G >>>>> 95% bro >>>>> worker-2-3 worker 192.168.0.11 30095 child 514M 31M >>>>> 8% bro >>>>> worker-2-4 worker 192.168.0.11 29970 parent 2G 2G >>>>> 95% bro >>>>> worker-2-4 worker 192.168.0.11 30137 child 514M 30M >>>>> 8% bro >>>>> worker-2-5 worker 192.168.0.11 29977 parent 2G 2G >>>>> 84% bro >>>>> worker-2-5 worker 192.168.0.11 30100 child 514M 31M >>>>> 10% bro >>>>> worker-2-6 worker 192.168.0.11 29978 parent 2G 2G >>>>> 73% bro >>>>> worker-2-6 worker 192.168.0.11 29990 child 514M 30M >>>>> 8% bro >>>>> worker-2-7 worker 192.168.0.11 29976 parent 2G 2G >>>>> 76% bro >>>>> worker-2-7 worker 192.168.0.11 30081 child 514M 31M >>>>> 10% bro >>>>> worker-2-8 worker 192.168.0.11 29963 parent 2G 2G >>>>> 57% bro >>>>> worker-2-8 worker 192.168.0.11 29987 child 514M 30M >>>>> 8% bro >>>>> worker-2-9 worker 192.168.0.11 29971 parent 2G 2G >>>>> 52% bro >>>>> worker-2-9 worker 192.168.0.11 30096 child 514M 31M >>>>> 10% bro >>>>> >>>>> bro-1 $ free -g >>>>> total used free shared buffers >>> cached >>>>> Mem: 62 62 0 0 0 17 >>>>> -/+ buffers/cache: 44 17 >>>>> Swap: 0 0 0 >>>>> >>>>> bro-2 $ free -g >>>>> total used free shared buffers >>> cached >>>>> Mem: 62 45 17 0 0 1 >>>>> -/+ buffers/cache: 44 18 >>>>> Swap: 0 0 0 >>>>> >>>>> What do you guys think? >>>>> >>>>> Regards, >>>>> Gary >>>>> >>>>> PS ~ >>>>> >>>>> I've been reading the mailing list archives and it seems that >folks >>>>> with >>>>> the older Xeons with higher clock rates (3.4Ghzish), but fewer >cores >>>>> were able to handle upwards of 400-500Mbps per worker process. >I've >>>>> also >>>>> seen it hinted, I think by Vlad G., that he was fitting in 28 >>> workers >>>>> on >>>>> boxes with similar core counts to my own, but slightly faster >>>>> processors. Based on some of those remarks in previous threads I >was >>>>> thinking I should be able to handle a little over 300Mbps per >>> process >>>>> with these processors, but I've only had the traffic to push about >>>>> 200Mbps per worker so far. >>>>> >>>>> I know some folks also like to put the manager and possibly the >>> proxies >>>>> >>>>> on separate boxes from the workers, but I haven't gotten a good >>> sense >>>>> as >>>>> to what kind of workload a proxy can handle. As far as proxies >I've >>>>> mostly seen comments such as "I probably have way more proxies >than >>> I >>>>> need" or "Just keep adding proxies until they stop crashing". I >>> don't >>>>> currently have a spare box for the manager and proxy, but would be >>>>> curious to know if folks feel it is a necessity. My observations >on >>> my >>>>> own setup are that my Bro workers typically are using 99% of a >>> logical >>>>> core at peak network times, and my manager 150-175% >>> (multi-threaded). >>>>> My >>>>> workers seem to use about 1-2G of memory normally. >>>>> >>>>> >>>>> >>>>> >>> >------------------------------------------------------------------------ >>>>> >>>>> _______________________________________________ >>>>> Bro mailing list >>>>> bro at bro-ids.org >>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> From gary at doit.wisc.edu Sat Dec 7 11:08:32 2013 From: gary at doit.wisc.edu (Gary Faulkner) Date: Sat, 07 Dec 2013 13:08:32 -0600 Subject: [Bro] Sizing a Bro Cluster [was out of memory after a couple days] In-Reply-To: References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <73C8E240-60EC-4D2F-9058-0BD6BA8F2EBD@uwaterloo.ca> <52A111F6.7010601@doit.wisc.edu> <52A26E0B.1000005@doit.wisc.edu> <52A27BC8.5030402@doit.wisc.edu> <52A28AA1.4090103@doit.wisc.edu> Message-ID: <52A37230.90800@doit.wisc.edu> Bob, Thank you. I'll try out your suggestions and monitor performance. Regards, Gary On 12/6/2013 9:12 PM, Bob wrote: > Twelve per box would be my plan. And to be honest, once you lower the number of workers, you could probably reduce the number of proxies as well. I'm running 3 for a cluster with 72 active workers, so you could probably do 24 with just one (and you could even put it on the non-manager box to maintain some symmetry). As always though, feel free to experiment with that to find what works best for you. > > Bob > > Gary Faulkner wrote: >> Bob, >> >> So, probably closer to 10-12 workers per host in my case since the >> proxies and manager are there? It was suggested to me early on to try >> to >> acquire another box to separate from manager and proxies, but I don't >> have one quite yet so I've been trying to make it work as is. It didn't >> >> seem like the worker child processes needed much CPU time, so I thought >> >> I could push the worker count higher and it also seemed like I got less >> >> loss per broctl netstat, but others have suggested that maybe the >> broctl >> netstat command wasn't the most reliable way to judge that. I ended up >> at 4 proxies mostly because two didn't seem stable and I like symmetry >> so I jumped straight to 4. >> >> Thanks again, >> Gary >> >> On 12/6/2013 7:57 PM, Bob wrote: >>> Gary, >>> >>> Ah, my apologies for my misreading of your specs. Two hosts seeing >> 2-4Gb/s each should be just fine. >>> >>> As far as my recommendation, I definitely meant real cores. I find, >> based on my own experimentation, that taking your number of workers >> higher than the number of physical cores in the box tends to hurt the >> performance instead of helping. In the case where a box was also doing >> manager/proxy duty, I'd subtract a physical core for each of those as >> well (I've broken those out into a separate box to avoid that issue on >> mine). I don't recall seeing anything from Vlad suggesting going >> higher, but I can't rule out the possibility that I missed it. >>> >>> Also to answer a question that I overlooked earlier, I'm running 3 >> proxies on that workload. I got to that number by just increasing by >> one until it stabilized. >>> >>> Bob >>> >>> Gary Faulkner wrote: >>>> Bob, >>>> >>>> Thank you for the feedback. To clarify I am currently using two >>>> physical >>>> hosts clustered together (using broctl), so each box ends up with >> 64G >>>> of >>>> RAM and 16cores/32 threads and Intel IXGBE 10G Cards + >>>> pf_ring/DNA/libzero for distributing packets on the host. Each >> physical >>>> >>>> host then sees between 2-4Gbps and has 20 workers + 2 proxies. I >> recall >>>> >>>> reading a blog entry by Martin Holste where he mentioned allocating >>>> only >>>> half as many workers as you had logical cores/threads, but also seem >> to >>>> >>>> recall others (Vlad G.) pushing nearly as many workers as logical >>>> cores, >>>> but could have read to much into it. Are you referring to physical >>>> cores >>>> or logical cores/threads? If it is the former I think what you are >>>> saying is inline with what Martin suggested; although I had hoped I >>>> could push the worker count a bit higher based on what I thought I >> had >>>> read elsewhere. >>>> >>>> Regards, >>>> Gary >>>> >>>> On 12/6/2013 7:08 PM, Bob wrote: >>>>> Gary, >>>>> >>>>> Realistically, for that load I would recommend looking into a >>>> cluster. My personal sizing criteria is 4-5Gb/s max per box. >> That's >>>> for a box that's roughly the same as yours and that uses up about >>>> 40-50GB of RAM per box (although for as cheap as it is, I recommend >> to >>>> always guess high on the RAM). For this size of box, you can >> probably >>>> improve your performance by reducing the number of workers (one per >>>> real core is a good benchmark). I am conservative, so I like to >> keep a >>>> couple if cores free for system tasks to ensure reliable performance >>>> (setting the number of workers to something like 12 or 14), but >> that's >>>> up to you. >>>>> >>>>> As a caveat, the aforementioned recommendations assume that you're >>>> using a network card that's designed to do this work (like an Intel >>>> ixgbe card with pf_ring or a Myricom). If your card can't bypass >> the >>>> kernel's interfaces, then you're going to need a lot more hardware >> to >>>> get the same performance because you're spending CPU time shoving >> the >>>> packets through the kernel instead of just accessing them directly >> on >>>> the NIC >>>>> >>>>> That's my two cents. >>>>> >>>>> Bob >>>>> >>>>> Gary Faulkner wrote: >>>>>> I've had some proxy crashes in the past and it was suggested that >> I >>>>>> increase my number of proxies -- which I did until my environment >>>>>> appeared stable for about a week. After being stable for about a >>>> week I >>>>>> >>>>>> started to run out of memory, and in subsequent restarts have been >>>>>> running out of memory after about 24 hours of operation, typically >>>>>> during non-peak times (50% of normal traffic). Naturally I'm >>>> wondering >>>>>> if I'm just doing it wrong and if my set-up is appropriately sized >>>> and >>>>>> configured to handle the load I'm asking it to deal with. >>>>>> >>>>>> I think I've seen folks on the list that were running Bro on >> similar >>>>>> hardware that might be able to tell me if my configuration is >>>> anything >>>>>> close to what works for them. I'm also curious how other folks >>>>>> determine >>>>>> how many proxies they need, how many workers per host etc. >>>>>> >>>>>> I'm mostly running Bro 2.2 stock with default scripts, and only >>>> minor >>>>>> edits to local.bro to test out email notices. I'm only using these >>>>>> systems for Bro, although they were originally from another >> project >>>> so >>>>>> they weren't necessarily ordered with Bro specs in mind. >>>>>> >>>>>> Here's how I've got things allocated currently: >>>>>> >>>>>> Bro -1 Host: >>>>>> 2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) >>>>>> 64G RAM >>>>>> manager >>>>>> 2 proxies >>>>>> 20 workers >>>>>> 2-4GB of Traffic >>>>>> >>>>>> Bro-2 Host: >>>>>> 2ea Xeon E5-2670 at 2.6Ghz (32 combined Logical Cores / 16 Physical) >>>>>> 64G RAM >>>>>> 2 proxies >>>>>> 20 workers >>>>>> 2-4 GB of traffic >>>>>> >>>>>> The following is a relatively light traffic load (late on a >> Friday) >>>> for >>>>>> >>>>>> my install (4Gbps vs 8Gbps): >>>>>> >>>>>> bro-1 $ ./broctl capstats >>>>>> >>>>>> Interface kpps mbps (10s average) >>>>>> ------------------------------ >>>>>> 192.168.0.10/dnacluster:21 338.6 2327.4 >>>>>> 192.168.0.11/dnacluster:22 324.8 2264.7 >>>>>> >>>>>> Total 663.4 4592.1 >>>>>> >>>>>> bro-1 $ ./broctl top >>>>>> Name Type Node Pid Proc VSize Rss >>>>>> Cpu Cmd >>>>>> manager manager 192.168.0.10 14816 parent 2G 736M >>>>>> 88% bro >>>>>> manager manager 192.168.0.10 14817 child 169M 93M >>>>>> 44% bro >>>>>> proxy-1 proxy 192.168.0.10 14863 child 102M 26M >>>>>> 23% bro >>>>>> proxy-1 proxy 192.168.0.10 14860 parent 1G 1G >>>>>> 3% bro >>>>>> proxy-2 proxy 192.168.0.10 14862 child 102M 28M >>>>>> 27% bro >>>>>> proxy-2 proxy 192.168.0.10 14861 parent 1G 1G >>>>>> 3% bro >>>>>> proxy-3 proxy 192.168.0.11 28900 child 102M 46M >>>>>> 20% bro >>>>>> proxy-3 proxy 192.168.0.11 28898 parent 1G 1G >>>>>> 1% bro >>>>>> proxy-4 proxy 192.168.0.11 28899 child 102M 45M >>>>>> 21% bro >>>>>> proxy-4 proxy 192.168.0.11 28897 parent 1G 1G >>>>>> 1% bro >>>>>> worker-1-1 worker 192.168.0.10 15228 parent 2G 2G >>>>>> 65% bro >>>>>> worker-1-1 worker 192.168.0.10 15398 child 514M 11M >>>>>> 10% bro >>>>>> worker-1-10 worker 192.168.0.10 15230 parent 2G 2G >>>>>> 53% bro >>>>>> worker-1-10 worker 192.168.0.10 15407 child 514M 12M >>>>>> 8% bro >>>>>> worker-1-11 worker 192.168.0.10 15234 parent 2G 2G >>>>>> 78% bro >>>>>> worker-1-11 worker 192.168.0.10 15286 child 514M 9M >>>>>> 11% bro >>>>>> worker-1-12 worker 192.168.0.10 15235 parent 2G 2G >>>>>> 67% bro >>>>>> worker-1-12 worker 192.168.0.10 15267 child 514M 8M >>>>>> 12% bro >>>>>> worker-1-13 worker 192.168.0.10 15237 parent 2G 2G >>>>>> 82% bro >>>>>> worker-1-13 worker 192.168.0.10 15392 child 514M 9M >>>>>> 12% bro >>>>>> worker-1-14 worker 192.168.0.10 15238 parent 2G 2G >>>>>> 43% bro >>>>>> worker-1-14 worker 192.168.0.10 15264 child 514M 11M >>>>>> 8% bro >>>>>> worker-1-15 worker 192.168.0.10 15240 parent 2G 2G >>>>>> 76% bro >>>>>> worker-1-15 worker 192.168.0.10 15300 child 514M 7M >>>>>> 9% bro >>>>>> worker-1-16 worker 192.168.0.10 15243 parent 2G 2G >>>>>> 94% bro >>>>>> worker-1-16 worker 192.168.0.10 15404 child 514M 11M >>>>>> 9% bro >>>>>> worker-1-17 worker 192.168.0.10 15244 parent 2G 2G >>>>>> 67% bro >>>>>> worker-1-17 worker 192.168.0.10 15383 child 514M 8M >>>>>> 8% bro >>>>>> worker-1-18 worker 192.168.0.10 15246 parent 2G 2G >>>>>> 80% bro >>>>>> worker-1-18 worker 192.168.0.10 15372 child 514M 12M >>>>>> 11% bro >>>>>> worker-1-19 worker 192.168.0.10 15248 parent 2G 2G >>>>>> 76% bro >>>>>> worker-1-19 worker 192.168.0.10 15376 child 514M 8M >>>>>> 8% bro >>>>>> worker-1-2 worker 192.168.0.10 15251 parent 2G 2G >>>>>> 83% bro >>>>>> worker-1-2 worker 192.168.0.10 15414 child 514M 11M >>>>>> 10% bro >>>>>> worker-1-20 worker 192.168.0.10 15254 parent 2G 2G >>>>>> 86% bro >>>>>> worker-1-20 worker 192.168.0.10 15417 child 514M 12M >>>>>> 11% bro >>>>>> worker-1-3 worker 192.168.0.10 15253 parent 2G 2G >>>>>> 55% bro >>>>>> worker-1-3 worker 192.168.0.10 15375 child 514M 8M >>>>>> 12% bro >>>>>> worker-1-4 worker 192.168.0.10 15256 parent 2G 2G >>>>>> 87% bro >>>>>> worker-1-4 worker 192.168.0.10 15388 child 515M 8M >>>>>> 10% bro >>>>>> worker-1-5 worker 192.168.0.10 15257 parent 2G 2G >>>>>> 58% bro >>>>>> worker-1-5 worker 192.168.0.10 15395 child 515M 11M >>>>>> 10% bro >>>>>> worker-1-6 worker 192.168.0.10 15258 parent 2G 2G >>>>>> 96% bro >>>>>> worker-1-6 worker 192.168.0.10 15394 child 514M 11M >>>>>> 8% bro >>>>>> worker-1-7 worker 192.168.0.10 15259 parent 2G 2G >>>>>> 65% bro >>>>>> worker-1-7 worker 192.168.0.10 15413 child 514M 12M >>>>>> 6% bro >>>>>> worker-1-8 worker 192.168.0.10 15260 parent 2G 2G >>>>>> 99% bro >>>>>> worker-1-8 worker 192.168.0.10 15401 child 514M 11M >>>>>> 8% bro >>>>>> worker-1-9 worker 192.168.0.10 15261 parent 2G 2G >>>>>> 61% bro >>>>>> worker-1-9 worker 192.168.0.10 15408 child 514M 11M >>>>>> 8% bro >>>>>> worker-2-1 worker 192.168.0.11 29961 parent 2G 2G >>>>>> 85% bro >>>>>> worker-2-1 worker 192.168.0.11 29984 child 514M 31M >>>>>> 9% bro >>>>>> worker-2-10 worker 192.168.0.11 29959 parent 2G 2G >>>>>> 52% bro >>>>>> worker-2-10 worker 192.168.0.11 30085 child 515M 31M >>>>>> 8% bro >>>>>> worker-2-11 worker 192.168.0.11 29960 parent 2G 2G >>>>>> 96% bro >>>>>> worker-2-11 worker 192.168.0.11 30112 child 514M 31M >>>>>> 10% bro >>>>>> worker-2-12 worker 192.168.0.11 29973 parent 2G 2G >>>>>> 54% bro >>>>>> worker-2-12 worker 192.168.0.11 30082 child 514M 30M >>>>>> 8% bro >>>>>> worker-2-13 worker 192.168.0.11 29967 parent 2G 2G >>>>>> 93% bro >>>>>> worker-2-13 worker 192.168.0.11 30111 child 514M 31M >>>>>> 10% bro >>>>>> worker-2-14 worker 192.168.0.11 29962 parent 2G 2G >>>>>> 100% bro >>>>>> worker-2-14 worker 192.168.0.11 30076 child 514M 30M >>>>>> 8% bro >>>>>> worker-2-15 worker 192.168.0.11 29975 parent 2G 2G >>>>>> 55% bro >>>>>> worker-2-15 worker 192.168.0.11 30138 child 514M 31M >>>>>> 10% bro >>>>>> worker-2-16 worker 192.168.0.11 29965 parent 2G 2G >>>>>> 85% bro >>>>>> worker-2-16 worker 192.168.0.11 29994 child 514M 31M >>>>>> 8% bro >>>>>> worker-2-17 worker 192.168.0.11 29968 parent 2G 2G >>>>>> 76% bro >>>>>> worker-2-17 worker 192.168.0.11 30097 child 514M 31M >>>>>> 8% bro >>>>>> worker-2-18 worker 192.168.0.11 29972 parent 2G 2G >>>>>> 95% bro >>>>>> worker-2-18 worker 192.168.0.11 30115 child 514M 30M >>>>>> 10% bro >>>>>> worker-2-19 worker 192.168.0.11 29964 parent 2G 2G >>>>>> 68% bro >>>>>> worker-2-19 worker 192.168.0.11 30092 child 514M 31M >>>>>> 7% bro >>>>>> worker-2-2 worker 192.168.0.11 29974 parent 2G 2G >>>>>> 51% bro >>>>>> worker-2-2 worker 192.168.0.11 30133 child 514M 31M >>>>>> 7% bro >>>>>> worker-2-20 worker 192.168.0.11 29966 parent 2G 2G >>>>>> 59% bro >>>>>> worker-2-20 worker 192.168.0.11 29981 child 514M 30M >>>>>> 10% bro >>>>>> worker-2-3 worker 192.168.0.11 29969 parent 2G 2G >>>>>> 95% bro >>>>>> worker-2-3 worker 192.168.0.11 30095 child 514M 31M >>>>>> 8% bro >>>>>> worker-2-4 worker 192.168.0.11 29970 parent 2G 2G >>>>>> 95% bro >>>>>> worker-2-4 worker 192.168.0.11 30137 child 514M 30M >>>>>> 8% bro >>>>>> worker-2-5 worker 192.168.0.11 29977 parent 2G 2G >>>>>> 84% bro >>>>>> worker-2-5 worker 192.168.0.11 30100 child 514M 31M >>>>>> 10% bro >>>>>> worker-2-6 worker 192.168.0.11 29978 parent 2G 2G >>>>>> 73% bro >>>>>> worker-2-6 worker 192.168.0.11 29990 child 514M 30M >>>>>> 8% bro >>>>>> worker-2-7 worker 192.168.0.11 29976 parent 2G 2G >>>>>> 76% bro >>>>>> worker-2-7 worker 192.168.0.11 30081 child 514M 31M >>>>>> 10% bro >>>>>> worker-2-8 worker 192.168.0.11 29963 parent 2G 2G >>>>>> 57% bro >>>>>> worker-2-8 worker 192.168.0.11 29987 child 514M 30M >>>>>> 8% bro >>>>>> worker-2-9 worker 192.168.0.11 29971 parent 2G 2G >>>>>> 52% bro >>>>>> worker-2-9 worker 192.168.0.11 30096 child 514M 31M >>>>>> 10% bro >>>>>> >>>>>> bro-1 $ free -g >>>>>> total used free shared buffers >>>> cached >>>>>> Mem: 62 62 0 0 0 17 >>>>>> -/+ buffers/cache: 44 17 >>>>>> Swap: 0 0 0 >>>>>> >>>>>> bro-2 $ free -g >>>>>> total used free shared buffers >>>> cached >>>>>> Mem: 62 45 17 0 0 1 >>>>>> -/+ buffers/cache: 44 18 >>>>>> Swap: 0 0 0 >>>>>> >>>>>> What do you guys think? >>>>>> >>>>>> Regards, >>>>>> Gary >>>>>> >>>>>> PS ~ >>>>>> >>>>>> I've been reading the mailing list archives and it seems that >> folks >>>>>> with >>>>>> the older Xeons with higher clock rates (3.4Ghzish), but fewer >> cores >>>>>> were able to handle upwards of 400-500Mbps per worker process. >> I've >>>>>> also >>>>>> seen it hinted, I think by Vlad G., that he was fitting in 28 >>>> workers >>>>>> on >>>>>> boxes with similar core counts to my own, but slightly faster >>>>>> processors. Based on some of those remarks in previous threads I >> was >>>>>> thinking I should be able to handle a little over 300Mbps per >>>> process >>>>>> with these processors, but I've only had the traffic to push about >>>>>> 200Mbps per worker so far. >>>>>> >>>>>> I know some folks also like to put the manager and possibly the >>>> proxies >>>>>> >>>>>> on separate boxes from the workers, but I haven't gotten a good >>>> sense >>>>>> as >>>>>> to what kind of workload a proxy can handle. As far as proxies >> I've >>>>>> mostly seen comments such as "I probably have way more proxies >> than >>>> I >>>>>> need" or "Just keep adding proxies until they stop crashing". I >>>> don't >>>>>> currently have a spare box for the manager and proxy, but would be >>>>>> curious to know if folks feel it is a necessity. My observations >> on >>>> my >>>>>> own setup are that my Bro workers typically are using 99% of a >>>> logical >>>>>> core at peak network times, and my manager 150-175% >>>> (multi-threaded). >>>>>> My >>>>>> workers seem to use about 1-2G of memory normally. >>>>>> >>>>>> >>>>>> >>>>>> >>>> >> ------------------------------------------------------------------------ >>>>>> >>>>>> _______________________________________________ >>>>>> Bro mailing list >>>>>> bro at bro-ids.org >>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> > From jbabio at po-box.esu.edu Sat Dec 7 13:45:13 2013 From: jbabio at po-box.esu.edu (John Babio) Date: Sat, 7 Dec 2013 21:45:13 +0000 Subject: [Bro] plans for parsing other software in software.log Message-ID: <6E7D7EC4661DB5438A60388F3ED1BBAA064582@msxmb2.admin.esu.edu> Hello, are there any plans to include flash, silverlight, or software like these in the software.log file or is this already possible? I personally would find it valuable to know if someone was running an ancient version of flash or other vulnerable addons. From daniel at justbeck.com Mon Dec 9 16:40:15 2013 From: daniel at justbeck.com (Daniel Beck) Date: Mon, 9 Dec 2013 19:40:15 -0500 Subject: [Bro] Bro script to detect XOR'd executables Message-ID: Hello list, I've been working on my first real bro script and I had a couple questions. It uses the File API to detect the transfer of XOR'd Windows executables, the code is at https://github.com/justbeck/bro-xorpe. My questions are: - Is the file object in the file_new event guaranteed to have the beginning of the captured file? If not, is there a better location to hook the analysis into? - What's the performance impact of running a script like this on a large pipe? The script runs several (quasi) loops for each file_new event and I only have my home network to test it on. - Following on the last question, Is there a better way to do bitwise operations in Bro scripts besides creating a huge lookup table? Thanks, Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131209/84bf7ffe/attachment.html From seth at icir.org Mon Dec 9 18:07:08 2013 From: seth at icir.org (Seth Hall) Date: Mon, 9 Dec 2013 21:07:08 -0500 Subject: [Bro] Bro script to detect XOR'd executables In-Reply-To: References: Message-ID: On Dec 9, 2013, at 7:40 PM, Daniel Beck wrote: > - Is the file object in the file_new event guaranteed to have the beginning of the captured file? If not, is there a better location to hook the analysis into? No, you aren't guaranteed that it's the beginning of the file. You should be able to inspect the file record though to see if you have gotten the begging of the file although I'm blanking on how exactly you'd do that at the moment. > - What's the performance impact of running a script like this on a large pipe? The script runs several (quasi) loops for each file_new event and I only have my home network to test it on. Not sure, but likely to have a lot of overhead. There is quite a bit of code there that runs for each file. The best way to find out is to run it on a larger network though. > - Following on the last question, Is there a better way to do bitwise operations in Bro scripts besides creating a huge lookup table? Unfortunately not at the moment. There have been a number of discussions where we've talked about adding bitwise operators to Bro but we've never come to any firm conclusion. Anyway, overall it's a really neat script. Nice job! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131209/c65f2557/attachment.bin From wsladekjr at hotmail.com Tue Dec 10 06:32:09 2013 From: wsladekjr at hotmail.com (Ward Sladek) Date: Tue, 10 Dec 2013 08:32:09 -0600 Subject: [Bro] PF_RING Tuple config Message-ID: Is there a way to change PF_RING's cluster/tuple type within Bro? Such as a parameter I can include in node.cfg? I'm running Bro 2.2. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131210/00d6ebc2/attachment.html From konrad.weglowski at gmail.com Tue Dec 10 07:14:57 2013 From: konrad.weglowski at gmail.com (Konrad Weglowski) Date: Tue, 10 Dec 2013 10:14:57 -0500 Subject: [Bro] Connection summary and alarm summary emails frequency Message-ID: Hello, Where can I adjust the frequency of the Connection Summary/Alarm Summary emails that Bro sends via email? They come in every hour by default. Thank you, Konrad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131210/c6d1f113/attachment.html From jlay at slave-tothe-box.net Tue Dec 10 07:35:46 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Tue, 10 Dec 2013 08:35:46 -0700 Subject: [Bro] Connection summary and alarm summary emails frequency In-Reply-To: References: Message-ID: On 2013-12-10 08:14, Konrad Weglowski wrote: > Hello, > > Where can I adjust the frequency of the Connection Summary/Alarm > Summary emails that Bro sends via email? They come in every hour by > default. > > Thank you, > > Konrad I think this coincides with when broctl archives. I set mine within broctl.cfg to once a day with: # Rotation interval in seconds for log files on manager/standalone node. LogRotationInterval = 86400 Hope that helps. James From dnthayer at illinois.edu Tue Dec 10 07:39:24 2013 From: dnthayer at illinois.edu (Daniel Thayer) Date: Tue, 10 Dec 2013 09:39:24 -0600 Subject: [Bro] Connection summary and alarm summary emails frequency In-Reply-To: References: Message-ID: <52A735AC.50209@illinois.edu> On 12/10/2013 09:14 AM, Konrad Weglowski wrote: > > Hello, > > Where can I adjust the frequency of the Connection Summary/Alarm Summary > emails that Bro sends via email? They come in every hour by default. > > Thank you, > > Konrad > The LogRotationInterval and MailAlarmsInterval options are documented here: http://bro.org/sphinx/components/broctl/README.html#user-options The connection summary email interval is the same as the log rotation interval. If you don't want to receive connection summary emails, then you can set MailConnectionSummary=0. From konrad.weglowski at gmail.com Tue Dec 10 08:25:40 2013 From: konrad.weglowski at gmail.com (Konrad Weglowski) Date: Tue, 10 Dec 2013 11:25:40 -0500 Subject: [Bro] Connection summary and alarm summary emails frequency In-Reply-To: <52A735AC.50209@illinois.edu> References: <52A735AC.50209@illinois.edu> Message-ID: Thank you Daniel. Do you know if there a way to keep the log rotation default (1hr) but modify the connection summary emails to be sent out every 24 hours for example? On Tue, Dec 10, 2013 at 10:39 AM, Daniel Thayer wrote: > On 12/10/2013 09:14 AM, Konrad Weglowski wrote: > >> >> Hello, >> >> Where can I adjust the frequency of the Connection Summary/Alarm Summary >> emails that Bro sends via email? They come in every hour by default. >> >> Thank you, >> >> Konrad >> >> > > The LogRotationInterval and MailAlarmsInterval options are > documented here: > http://bro.org/sphinx/components/broctl/README.html#user-options > > The connection summary email interval is the same as the log rotation > interval. If you don't want to receive connection summary emails, > then you can set MailConnectionSummary=0. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131210/5c1c7e61/attachment.html From robin at icir.org Tue Dec 10 09:06:25 2013 From: robin at icir.org (Robin Sommer) Date: Tue, 10 Dec 2013 09:06:25 -0800 Subject: [Bro] Connection summary and alarm summary emails frequency In-Reply-To: References: <52A735AC.50209@illinois.edu> Message-ID: <20131210170625.GI31321@icir.org> On Tue, Dec 10, 2013 at 11:25 -0500, Konrad Weglowski wrote: > Do you know if there a way to keep the log rotation default (1hr) but > modify the connection summary emails to be sent out every 24 hours for > example? No, sorry, the summaries are implemented as a log post-processor that crunches conn.log as it gets rotated. That's why their intervals are tied together. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From konrad.weglowski at gmail.com Tue Dec 10 09:17:43 2013 From: konrad.weglowski at gmail.com (Konrad Weglowski) Date: Tue, 10 Dec 2013 12:17:43 -0500 Subject: [Bro] Connection summary and alarm summary emails frequency In-Reply-To: <20131210170625.GI31321@icir.org> References: <52A735AC.50209@illinois.edu> <20131210170625.GI31321@icir.org> Message-ID: Thank you Konrad On Tue, Dec 10, 2013 at 12:06 PM, Robin Sommer wrote: > > > On Tue, Dec 10, 2013 at 11:25 -0500, Konrad Weglowski wrote: > > > Do you know if there a way to keep the log rotation default (1hr) but > > modify the connection summary emails to be sent out every 24 hours for > > example? > > No, sorry, the summaries are implemented as a log post-processor that > crunches conn.log as it gets rotated. That's why their intervals are > tied together. > > Robin > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131210/624c7d34/attachment.html From dnthayer at illinois.edu Tue Dec 10 09:24:13 2013 From: dnthayer at illinois.edu (Daniel Thayer) Date: Tue, 10 Dec 2013 11:24:13 -0600 Subject: [Bro] Connection summary and alarm summary emails frequency In-Reply-To: References: <52A735AC.50209@illinois.edu> <20131210170625.GI31321@icir.org> Message-ID: <52A74E3D.8070801@illinois.edu> If you set MailConnectionSummary=0 in broctl.cfg, then you won't receive any connection summary emails, but the connection summary reports will still be created (they are archived along with the other bro log files), so you can look at them when it is convenient for you. On 12/10/2013 11:17 AM, Konrad Weglowski wrote: > Thank you > > Konrad > > > > On Tue, Dec 10, 2013 at 12:06 PM, Robin Sommer > wrote: > > > > On Tue, Dec 10, 2013 at 11:25 -0500, Konrad Weglowski wrote: > > > Do you know if there a way to keep the log rotation default (1hr) but > > modify the connection summary emails to be sent out every 24 > hours for > > example? > > No, sorry, the summaries are implemented as a log post-processor that > crunches conn.log as it gets rotated. That's why their intervals are > tied together. > > Robin > > -- > Robin Sommer * Phone +1 (510) 722-6541 > * robin at icir.org > > ICSI/LBNL * Fax +1 (510) 666-2956 > * www.icir.org/robin > > > From jbabio at po-box.esu.edu Tue Dec 10 16:02:40 2013 From: jbabio at po-box.esu.edu (John Babio) Date: Wed, 11 Dec 2013 00:02:40 +0000 Subject: [Bro] software.log Message-ID: <6E7D7EC4661DB5438A60388F3ED1BBAA065E69@msxmb2.admin.esu.edu> 1386717221.171855 x.x.x.x - HTTP::BROWSER Adobe Flash Player - - - - - Adobe Flash Player Bro isn't displaying and version info. Is there a reason? From JAzoff at albany.edu Tue Dec 10 16:16:46 2013 From: JAzoff at albany.edu (Justin Azoff) Date: Tue, 10 Dec 2013 19:16:46 -0500 Subject: [Bro] software.log In-Reply-To: <6E7D7EC4661DB5438A60388F3ED1BBAA065E69@msxmb2.admin.esu.edu> References: <6E7D7EC4661DB5438A60388F3ED1BBAA065E69@msxmb2.admin.esu.edu> Message-ID: <20131211001646.GP12701@datacomm.albany.edu> On Wed, Dec 11, 2013 at 12:02:40AM +0000, John Babio wrote: > 1386717221.171855 x.x.x.x - HTTP::BROWSER Adobe Flash Player - - - - - Adobe Flash Player > > Bro isn't displaying and version info. Is there a reason? For those connections the client sent just "Adobe Flash Player" as the user-agent. It appears the flash updater does this when downloading the newest version. -- -- Justin Azoff -- Network Security & Performance Analyst From jbabio at po-box.esu.edu Tue Dec 10 16:23:24 2013 From: jbabio at po-box.esu.edu (John Babio) Date: Wed, 11 Dec 2013 00:23:24 +0000 Subject: [Bro] software.log In-Reply-To: <20131211001646.GP12701@datacomm.albany.edu> References: <6E7D7EC4661DB5438A60388F3ED1BBAA065E69@msxmb2.admin.esu.edu>, <20131211001646.GP12701@datacomm.albany.edu> Message-ID: <6E7D7EC4661DB5438A60388F3ED1BBAA065E90@msxmb2.admin.esu.edu> I can't seem to get the actual player to trigger. Is it supposed to show up in software.log as Flash not Adobe Flash Player? Security onion has a vulnerable config including flash but I have never seen it show up in software.log. ________________________________________ From: Justin Azoff [JAzoff at albany.edu] Sent: Tuesday, December 10, 2013 7:16 PM To: John Babio Cc: bro at bro.org Subject: Re: [Bro] software.log On Wed, Dec 11, 2013 at 12:02:40AM +0000, John Babio wrote: > 1386717221.171855 x.x.x.x - HTTP::BROWSER Adobe Flash Player - - - - - Adobe Flash Player > > Bro isn't displaying and version info. Is there a reason? For those connections the client sent just "Adobe Flash Player" as the user-agent. It appears the flash updater does this when downloading the newest version. -- -- Justin Azoff -- Network Security & Performance Analyst From doug.burks at gmail.com Tue Dec 10 17:24:06 2013 From: doug.burks at gmail.com (Doug Burks) Date: Tue, 10 Dec 2013 20:24:06 -0500 Subject: [Bro] software.log In-Reply-To: <6E7D7EC4661DB5438A60388F3ED1BBAA065E90@msxmb2.admin.esu.edu> References: <6E7D7EC4661DB5438A60388F3ED1BBAA065E69@msxmb2.admin.esu.edu> <20131211001646.GP12701@datacomm.albany.edu> <6E7D7EC4661DB5438A60388F3ED1BBAA065E90@msxmb2.admin.esu.edu> Message-ID: On Tuesday, December 10, 2013, John Babio wrote: > I can't seem to get the actual player to trigger. Is it supposed to show > up in software.log as Flash not Adobe Flash Player? Security onion has a > vulnerable config including flash but I have never seen it show up in > software.log. > I don't believe Security Onion includes flash by default. -- Doug Burks http://securityonion.net -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131210/443dfa20/attachment.html From dnthayer at illinois.edu Wed Dec 11 16:12:18 2013 From: dnthayer at illinois.edu (Daniel Thayer) Date: Wed, 11 Dec 2013 18:12:18 -0600 Subject: [Bro] PF_RING Tuple config In-Reply-To: References: Message-ID: <52A8FF62.4000008@illinois.edu> On 12/10/2013 08:32 AM, Ward Sladek wrote: > Is there a way to change PF_RING's cluster/tuple type within Bro? Such > as a parameter I can include in node.cfg? I'm running Bro 2.2. > There is no such option in Bro 2.2, but I've created a git branch "topic/dnthayer/ticket1108" in the broctl repository that you can test if you want. In that branch, I've added a new broctl option called "PFRINGClusterType" that you can set in broctl.cfg in order to choose a different PF_RING cluster type. -Daniel From gary at doit.wisc.edu Wed Dec 11 20:34:08 2013 From: gary at doit.wisc.edu (Gary Faulkner) Date: Wed, 11 Dec 2013 22:34:08 -0600 Subject: [Bro] out of memory after a couple days? In-Reply-To: <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> Message-ID: <52A93CC0.90204@doit.wisc.edu> I came across BIT-1103 regarding a memory leak affecting Bro 2.2 in JIRA which appears to have been resolved recently. Any chance that ticket and fix has relevance to the discussion from last week concerning gradual memory exhaustion in Bro 2.2? Regards, Gary On 12/4/2013 10:31 PM, Seth Hall wrote: > > On Dec 4, 2013, at 10:34 PM, Mike Patterson wrote: > >> I think you?re definitely running into a memory leak. I?ve had 2.2 processes try to grab up to 100GB of RAM. 8 workers, 96GB of RAM, but the box splits time with another 8 snort workers. My late 2.1 release (september 21 IIRC) was quite a bit more stable. > > > I think there is some particular traffic that you guys are running into that's causing it. A few other people have encountered that too but we haven't been able to nail down what it is yet. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > -- Gary Faulkner UW Madison Office of Campus Information Security 608-262-8591 From bernhard at ICSI.Berkeley.EDU Thu Dec 12 05:51:11 2013 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Thu, 12 Dec 2013 07:51:11 -0600 Subject: [Bro] out of memory after a couple days? In-Reply-To: <52A93CC0.90204@doit.wisc.edu> References: <529FC34A.4010207@doit.wisc.edu> <114A3AA2-0F5A-4BE6-B505-2AB754AF108C@uwaterloo.ca> <7CCF04B3-5689-4F11-8B82-E90A88F4B89C@icir.org> <52A93CC0.90204@doit.wisc.edu> Message-ID: <4857431D-E87D-417C-9822-D451FA06F7A8@icsi.berkeley.edu> Hi, the fix in BIT-1103 only applies to you if you are extensively using the input framework to read events ? e.g. by using the intelligence framework and updating it regularly. If you do not use the Input framework to import data into Bro, this fix will change nothing. It also should not change memory on any worker nodes (the input framework usually is only used on the master). Bernhard On Dec 11, 2013, at 10:34 PM, Gary Faulkner wrote: > I came across BIT-1103 regarding a memory leak affecting Bro 2.2 in JIRA > which appears to have been resolved recently. Any chance that ticket and > fix has relevance to the discussion from last week concerning gradual > memory exhaustion in Bro 2.2? > > Regards, > Gary > > On 12/4/2013 10:31 PM, Seth Hall wrote: >> >> On Dec 4, 2013, at 10:34 PM, Mike Patterson wrote: >> >>> I think you?re definitely running into a memory leak. I?ve had 2.2 processes try to grab up to 100GB of RAM. 8 workers, 96GB of RAM, but the box splits time with another 8 snort workers. My late 2.1 release (september 21 IIRC) was quite a bit more stable. >> >> >> I think there is some particular traffic that you guys are running into that's causing it. A few other people have encountered that too but we haven't been able to nail down what it is yet. >> >> .Seth >> >> -- >> Seth Hall >> International Computer Science Institute >> (Bro) because everyone has a network >> http://www.bro.org/ >> > > -- > Gary Faulkner > UW Madison > Office of Campus Information Security > 608-262-8591 > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From mattchess50 at gmail.com Thu Dec 12 06:41:20 2013 From: mattchess50 at gmail.com (Matt Stucky) Date: Thu, 12 Dec 2013 08:41:20 -0600 Subject: [Bro] Whitelist a file/hash Message-ID: What is the most appropriate way to whitelist a file/hash that keeps getting flagged by the MHR notice? Is there a "hash whitelist" or is a notice hook that overrides the default action needed? Thanks, -matt -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131212/638aef0c/attachment.html From konrad.weglowski at gmail.com Fri Dec 13 22:19:56 2013 From: konrad.weglowski at gmail.com (Konrad Weglowski) Date: Sat, 14 Dec 2013 01:19:56 -0500 Subject: [Bro] Bro manager crashing Message-ID: Hello, When I uncommented the following line in /opt/bro/share/bro/site/local.bro to enable email alerts on notice alarms and did install and restart on broctl Bro Manager is crashing. redef Notice::policy += { [$action = Notice::ACTION_ALARM, $priority = 0] }; I get the following error message via email notification. I am running BRO as a part of the Security Onion. When I comment the line back, Manager starts but I do not think emails on notices will. Thank you again Konrad *If you want to help us debug this problem, then please forward this mail to reports at bro.org Bro 2.2 Linux 3.2.0-57-generic ==== No reporter.log ==== stderr.log error in /opt/bro/share/bro/base/* *frameworks/notice/./main.bro, line 182 and /nsm/bro/spool/installed-* *scripts-do-not-touch/site/* *local.bro, line 35: already defined (Notice::policy)* * ==== stdout.log unlimited unlimited unlimited* * ==== .cmdline -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto* * ==== .env_vars PATH=/opt/bro/bin:/opt/bro/**share/broctl/scripts:/usr/* *local/sbin:/usr/local/bin:/**usr/sbin:/usr/bin:/sbin:/bin:/* *opt/bro/bin BROPATH=/nsm/bro/spool/**installed-scripts-do-not-* *touch/site::/nsm/bro/spool/**installed-scripts-do-not-* *touch/auto:/opt/bro/share/bro:**/opt/bro/share/bro/policy:/* *opt/bro/share/bro/site CLUSTER_NODE=manager* * ==== .status TERMINATED [atexit]* * ==== No prof.log* * ==== No packet_filter.log* * ==== No loaded_scripts.log* * -- [Automatically generated.]* -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131214/05ce9fdb/attachment.html From jsiwek at illinois.edu Mon Dec 16 07:26:06 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Mon, 16 Dec 2013 15:26:06 +0000 Subject: [Bro] Bro manager crashing In-Reply-To: References: Message-ID: <1E2D06E2-56EA-48F2-A220-5DA9B51130FF@illinois.edu> On Dec 14, 2013, at 12:19 AM, Konrad Weglowski wrote: > When I uncommented the following line in /opt/bro/share/bro/site/local.bro to enable email alerts on notice alarms and did install and restart on broctl Bro Manager is crashing. > > redef Notice::policy += { [$action = Notice::ACTION_ALARM, $priority = 0] }; Unfortunately, that commented-out example in local.bro doesn?t reflect how to actually use Notice::policy in Bro 2.2. Using code like the following should be equivalent: hook Notice::policy(n: Notice::Info) { add n$actions[Notice::ACTION_ALARM]; } - Jon From mattchess50 at gmail.com Mon Dec 16 08:15:00 2013 From: mattchess50 at gmail.com (Matt Stucky) Date: Mon, 16 Dec 2013 10:15:00 -0600 Subject: [Bro] Incorrect file type notices gone? Message-ID: Has the HTTP::Incorrect_File_Type notice been replaced with something different in 2.2? Thanks, Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131216/aca82383/attachment.html From TSMullins at mecc.edu Mon Dec 16 10:11:05 2013 From: TSMullins at mecc.edu (Thomas Mullins) Date: Mon, 16 Dec 2013 18:11:05 +0000 Subject: [Bro] window_recision Message-ID: Hello everyone, While searching our BRO logs, I came across a few hosts giving window_recision errors. A Google search did not shed any light on this subject. What does window_recision mean? Thanks Shane From jsiwek at illinois.edu Mon Dec 16 11:48:34 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Mon, 16 Dec 2013 19:48:34 +0000 Subject: [Bro] window_recision In-Reply-To: References: Message-ID: <8BFDCEF4-7076-45B6-AE88-B357BA6EFDB1@illinois.edu> On Dec 16, 2013, at 12:11 PM, Thomas Mullins wrote: > While searching our BRO logs, I came across a few hosts giving window_recision errors. A Google search did not shed any light on this subject. What does window_recision mean? I think it means that a TCP shrunk its recv-window by more than the amount of data its ACKing. i.e. in https://tools.ietf.org/html/rfc793#section-3.7 : The mechanisms provided allow a TCP to advertise a large window and to subsequently advertise a much smaller window without having accepted that much data. This, so called "shrinking the window," is strongly discouraged. The robustness principle dictates that TCPs will not shrink the window themselves, but will be prepared for such behavior on the part of other TCPs. - Jon From seth at icir.org Mon Dec 16 11:51:57 2013 From: seth at icir.org (Seth Hall) Date: Mon, 16 Dec 2013 14:51:57 -0500 Subject: [Bro] window_recision In-Reply-To: References: Message-ID: <3D54C1F7-0DCD-4102-9E1C-1AE7E0468FEC@icir.org> On Dec 16, 2013, at 1:11 PM, Thomas Mullins wrote: > While searching our BRO logs, I came across a few hosts giving window_recision errors. A Google search did not shed any light on this subject. What does window_recision mean? I've been curious about this myself at a few sites that see a surprisingly high number of window_recision weird. My suspicion is that it's due to some middle box on the network that is running out of buffer space. Are you monitoring at your border and do you have a border firewall? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131216/a4fb1fde/attachment.bin From seth at icir.org Mon Dec 16 12:10:52 2013 From: seth at icir.org (Seth Hall) Date: Mon, 16 Dec 2013 15:10:52 -0500 Subject: [Bro] Incorrect file type notices gone? In-Reply-To: References: Message-ID: <077BEEB1-5459-4C5C-808E-2BCC14E88546@icir.org> On Dec 16, 2013, at 11:15 AM, Matt Stucky wrote: > Has the HTTP::Incorrect_File_Type notice been replaced with something different in 2.2? Yes, it was removed. Here is a new version of it? https://github.com/sethhall/bro-junk-drawer/blob/master/incorrect-file-type.bro .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131216/2efa5829/attachment.bin From konrad.weglowski at gmail.com Tue Dec 17 01:31:03 2013 From: konrad.weglowski at gmail.com (Konrad Weglowski) Date: Tue, 17 Dec 2013 04:31:03 -0500 Subject: [Bro] Bro manager crashing In-Reply-To: <1E2D06E2-56EA-48F2-A220-5DA9B51130FF@illinois.edu> References: <1E2D06E2-56EA-48F2-A220-5DA9B51130FF@illinois.edu> Message-ID: Hello Jon, Thank you for very much. I put this directly inside local.bro. Is that the right way of doing it or should I be inserting this in another or a new file and then loading it inside local.bro? Thanks, Konrad On Mon, Dec 16, 2013 at 10:26 AM, Siwek, Jonathan Luke wrote: > > On Dec 14, 2013, at 12:19 AM, Konrad Weglowski > wrote: > > > When I uncommented the following line in > /opt/bro/share/bro/site/local.bro to enable email alerts on notice alarms > and did install and restart on broctl Bro Manager is crashing. > > > > redef Notice::policy += { [$action = Notice::ACTION_ALARM, $priority = > 0] }; > > Unfortunately, that commented-out example in local.bro doesn?t reflect how > to actually use Notice::policy in Bro 2.2. Using code like the following > should be equivalent: > > hook Notice::policy(n: Notice::Info) > { > add n$actions[Notice::ACTION_ALARM]; > } > > - Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131217/03605760/attachment.html From jsiwek at illinois.edu Tue Dec 17 07:18:48 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Tue, 17 Dec 2013 15:18:48 +0000 Subject: [Bro] Bro manager crashing In-Reply-To: References: <1E2D06E2-56EA-48F2-A220-5DA9B51130FF@illinois.edu> Message-ID: <6549EDD0-9506-499F-A8E9-333145FE388B@illinois.edu> On Dec 17, 2013, at 3:31 AM, Konrad Weglowski wrote: > Thank you for very much. I put this directly inside local.bro. Is that the right way of doing it or should I be inserting this in another or a new file and then loading it inside local.bro? Whichever is easiest for you, but the later is probably preferable to maintain and more clearly separate local code additions from the upstream version. - Jon From tberger at foregroundsecurity.com Tue Dec 17 12:19:48 2013 From: tberger at foregroundsecurity.com (Terrence Berger) Date: Tue, 17 Dec 2013 20:19:48 +0000 Subject: [Bro] Bro parser config for Nitro ESM Message-ID: <9F4418E4AA01A44996ED3CB77F003611A6C5B1@ORD2MBX05B.mex05.mlsrvr.com> Hello, I've recently integrated Bro logs into Nitro ESM and I'm wondering if there is an existing parser that's been developed for this purpose. Is this something you've come across? Thanks. Terrence Berger Security Engineer Foreground Security tberger at foregroundsecurity.com Office - 703.722.8714 Mobile - 301.806.8340 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131217/9d7368c1/attachment.html From logoyda at gmail.com Tue Dec 17 12:26:00 2013 From: logoyda at gmail.com (Michael Logoyda) Date: Tue, 17 Dec 2013 13:26:00 -0700 Subject: [Bro] Tab Character Variable Message-ID: I'm writing a script to read from a tsv file containing blacklisted IPs (Input::add_table, using reread mode) and then append new IPs to the end of the same file. I can read the file just fine, but any time I try to write out "\t" or "\x09", the output shows up as "^I". That goes for the system() function to write to the file as well as just using print within the script. Does anyone know how to send the tab character properly? Alternatively: A) I could change to csv, but I don't know how to change the input separator to allow for this. B) I am using a second column that's basically just filler in order to use add_table like the example on the blog. Is there a way to just import a one-column table? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131217/d5750a96/attachment.html From anthony.kasza at gmail.com Tue Dec 17 13:43:22 2013 From: anthony.kasza at gmail.com (anthony kasza) Date: Tue, 17 Dec 2013 13:43:22 -0800 Subject: [Bro] Tab Character Variable In-Reply-To: References: Message-ID: Trying using the logging framework instead of printing directly to a file. Grep for 'const separator' in base/frameworks/logging for changing the default column separator. -AK On Dec 17, 2013 12:45 PM, "Michael Logoyda" wrote: > I'm writing a script to read from a tsv file containing blacklisted IPs > (Input::add_table, using reread mode) and then append new IPs to the end of > the same file. I can read the file just fine, but any time I try to write > out "\t" or "\x09", the output shows up as "^I". That goes for the system() > function to write to the file as well as just using print within the > script. Does anyone know how to send the tab character properly? > > > Alternatively: > > A) I could change to csv, but I don't know how to change the input > separator to allow for this. > B) I am using a second column that's basically just filler in order to use > add_table like the example on the blog. Is there a way to just import a > one-column table? > > > Thanks > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131217/ef631cc1/attachment.html From bernhard at ICSI.Berkeley.EDU Tue Dec 17 13:52:50 2013 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Tue, 17 Dec 2013 22:52:50 +0100 Subject: [Bro] Tab Character Variable In-Reply-To: References: Message-ID: Yes, this sounds like it would be the better solution. But - just for reference - enable_raw_output should solve the print-problem. http://www.bro.org/sphinx/scripts/base/bif/bro.bif.html#id-enable_raw_output Bernhard On Dec 17, 2013, at 10:43 PM, anthony kasza wrote: > Trying using the logging framework instead of printing directly to a file. > Grep for 'const separator' in base/frameworks/logging for changing the default column separator. > > -AK > > On Dec 17, 2013 12:45 PM, "Michael Logoyda" wrote: > I'm writing a script to read from a tsv file containing blacklisted IPs (Input::add_table, using reread mode) and then append new IPs to the end of the same file. I can read the file just fine, but any time I try to write out "\t" or "\x09", the output shows up as "^I". That goes for the system() function to write to the file as well as just using print within the script. Does anyone know how to send the tab character properly? > > > Alternatively: > > A) I could change to csv, but I don't know how to change the input separator to allow for this. > B) I am using a second column that's basically just filler in order to use add_table like the example on the blog. Is there a way to just import a one-column table? > > > Thanks > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From konrad.weglowski at gmail.com Wed Dec 18 00:57:29 2013 From: konrad.weglowski at gmail.com (Konrad Weglowski) Date: Wed, 18 Dec 2013 03:57:29 -0500 Subject: [Bro] Bro manager crashing In-Reply-To: <6549EDD0-9506-499F-A8E9-333145FE388B@illinois.edu> References: <1E2D06E2-56EA-48F2-A220-5DA9B51130FF@illinois.edu> <6549EDD0-9506-499F-A8E9-333145FE388B@illinois.edu> Message-ID: Thanks again. On Tuesday, December 17, 2013, Siwek, Jonathan Luke wrote: > > On Dec 17, 2013, at 3:31 AM, Konrad Weglowski > > wrote: > > > Thank you for very much. I put this directly inside local.bro. Is that > the right way of doing it or should I be inserting this in another or a new > file and then loading it inside local.bro? > > Whichever is easiest for you, but the later is probably preferable to > maintain and more clearly separate local code additions from the upstream > version. > > - Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131218/aeefbd14/attachment.html From sangdrax8 at gmail.com Wed Dec 18 04:57:49 2013 From: sangdrax8 at gmail.com (sangdrax8) Date: Wed, 18 Dec 2013 07:57:49 -0500 Subject: [Bro] missed bytes without gaps Message-ID: I am confused on how I am getting "missed_bytes." When I read the documentation it says that these are due to content gaps, and is caused by packet loss. So I have tried to look for signs of this, but I have yet to find anything that shows content gaps or any significant packet loss. I see no mention of content gaps in my notices.log file, and my packet loss report from broctl shows almost no dropped packets (recvd=2770338 dropped=92 link=2770430) Perhaps I am looking at this wrong, so if someone could help out here it would be greatly appreciated. I am logging correct streams, vs streams with missed bytes, to see how often these missed bytes show up. To do this I am filtering my logs in the following way Missed: only looking at bro_conn, orig_bytes AND resp_bytes > 0, only TCP packets, missed_bytes > 0 Non-missed: only looking at bro_conn, orig_bytes AND resp_bytes > 0, only TCP packets, missed_bytes = 0 Using these two definitions, I see almost 40% of my packets fall into the "missed" streams, while around 60% fall into the non-missed. I was doing this to check my setup and see if I had everything working. From everything else (no gaps reported, and no almost no dropped packets) I thought everything was working. Now I question if something else is wrong, and so I am weary about using this to look at other data as it may not be complete. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131218/9829f795/attachment.html From seth at icir.org Wed Dec 18 05:22:48 2013 From: seth at icir.org (Seth Hall) Date: Wed, 18 Dec 2013 08:22:48 -0500 Subject: [Bro] missed bytes without gaps In-Reply-To: References: Message-ID: On Dec 18, 2013, at 7:57 AM, sangdrax8 wrote: > Using these two definitions, I see almost 40% of my packets fall into the "missed" streams, while around 60% fall into the non-missed. I was doing this to check my setup and see if I had everything working. From everything else (no gaps reported, and no almost no dropped packets) I thought everything was working. Now I question if something else is wrong, and so I am weary about using this to look at other data as it may not be complete. There are a lot of reasons that you could be missing traffic that have nothing to do with the packet drop statistics reported by your NIC. I have a guess about what's happening in your traffic though. Have you disabled the special features on your NIC? Refer to this blog post on how to do it on linux: http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html If you want a much better mechanism to see if you're receiving all of the traffic you should be I recommend loading the misc/capture-loss script. By default it will write out to capture_loss.log every 15 minutes and due to it taking measurements of TCP streams themselves it can even detect packet loss occurring before the packets arrive at your monitoring interface. A number of people have detected faulty packet distribution boxes and overloaded switch SPAN ports with it. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131218/ff83fcb7/attachment.bin From sangdrax8 at gmail.com Wed Dec 18 05:51:19 2013 From: sangdrax8 at gmail.com (sangdrax8) Date: Wed, 18 Dec 2013 08:51:19 -0500 Subject: [Bro] missed bytes without gaps In-Reply-To: References: Message-ID: On Wed, Dec 18, 2013 at 8:22 AM, Seth Hall wrote: > > On Dec 18, 2013, at 7:57 AM, sangdrax8 wrote: > > > Using these two definitions, I see almost 40% of my packets fall into > the "missed" streams, while around 60% fall into the non-missed. I was > doing this to check my setup and see if I had everything working. From > everything else (no gaps reported, and no almost no dropped packets) I > thought everything was working. Now I question if something else is wrong, > and so I am weary about using this to look at other data as it may not be > complete. > > There are a lot of reasons that you could be missing traffic that have > nothing to do with the packet drop statistics reported by your NIC. I have > a guess about what's happening in your traffic though. Have you disabled > the special features on your NIC? Refer to this blog post on how to do it > on linux: > > http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html > > If you want a much better mechanism to see if you're receiving all of the > traffic you should be I recommend loading the misc/capture-loss script. By > default it will write out to capture_loss.log every 15 minutes and due to > it taking measurements of TCP streams themselves it can even detect packet > loss occurring before the packets arrive at your monitoring interface. A > number of people have detected faulty packet distribution boxes and > overloaded switch SPAN ports with it. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > Wow thank you this was exactly my problem! I have turned these settings off and in the last 5 minute window my missed bytes went from ~40% to 0%. Perhaps a quick note in the doc's could point others to offloading features in NIC's as well because this fixed everything for me! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131218/fc893b73/attachment.html From jpl at didconcept.com Wed Dec 18 09:13:06 2013 From: jpl at didconcept.com (Jean-Philippe Luiggi) Date: Wed, 18 Dec 2013 12:13:06 -0500 Subject: [Bro] plixer.com and bro log reporting Message-ID: <52B1D7A2.6070907@didconcept.com> Hello everybody, I just came over this news today : http://www.plixer.com/blog/log-management/bro-log-reporting/ From the web page : `Are you looking for a solution for*Bro Log Reporting*? This is a post on how we ended up with a really great solution for reporting, trending and searching through Bro logs by converting them to IPFIX." Cheers. Jean-Philippe. PS: It is possible to have a free 30 day trial of Scrutinizer (the software). -- Ce message a ?t? v?rifi? par MailScanner pour des virus ou des polluriels et rien de suspect n'a ?t? trouv?. For all your IT requirements visit: http://www.transtec.co.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131218/f79c4842/attachment.html From gary at doit.wisc.edu Wed Dec 18 21:31:53 2013 From: gary at doit.wisc.edu (Gary Faulkner) Date: Wed, 18 Dec 2013 23:31:53 -0600 Subject: [Bro] broctl cron running, but some scheduled tasks seem to be failing? Message-ID: <52B284C9.1000907@doit.wisc.edu> I'm trying to troubleshoot some odd behavior. I stopped receiving hourly email summaries and logs stopped being moved and compressed at some point this afternoon; although new logs are still being started hourly and the old log being renamed. As far as I can tell from the cron log the broctl cron job is running as scheduled. I tried running broctl cron manually, but no dice. It didn't see any hung processes from earlier cron jobs or any emails in the bro user's mailbox indicating something went awry. Does broctl cron produce any log output if it has trouble? -- Gary Faulkner UW Madison Office of Campus Information Security 608-262-8591 From dnthayer at illinois.edu Thu Dec 19 06:26:06 2013 From: dnthayer at illinois.edu (Daniel Thayer) Date: Thu, 19 Dec 2013 08:26:06 -0600 Subject: [Bro] broctl cron running, but some scheduled tasks seem to be failing? In-Reply-To: <52B284C9.1000907@doit.wisc.edu> References: <52B284C9.1000907@doit.wisc.edu> Message-ID: <52B301FE.4090905@illinois.edu> On 12/18/2013 11:31 PM, Gary Faulkner wrote: > I'm trying to troubleshoot some odd behavior. I stopped receiving hourly > email summaries and logs stopped being moved and compressed at some > point this afternoon; although new logs are still being started hourly > and the old log being renamed. > > As far as I can tell from the cron log the broctl cron job is running as > scheduled. I tried running broctl cron manually, but no dice. It didn't > see any hung processes from earlier cron jobs or any emails in the bro > user's mailbox indicating something went awry. Does broctl cron produce > any log output if it has trouble? > Actually, broctl cron doesn't do log rotation or hourly email summaries. In fact, those happen even if broctl isn't running at all. When it's time to do a log rotation, Bro itself (on the manager host) executes a script /share/broctl/scripts/archive-log and that script then executes a script /share/broctl/scripts/postprocessors/summarize-connections that generates and emails the connection summary report. So, I'd suggest making sure those scripts exist on your manager host, check if you see any "archive-log" processes running in the background, and then check if you're running out of disk space. From gary at doit.wisc.edu Thu Dec 19 07:35:39 2013 From: gary at doit.wisc.edu (Gary Faulkner) Date: Thu, 19 Dec 2013 09:35:39 -0600 Subject: [Bro] broctl cron running, but some scheduled tasks seem to be failing? In-Reply-To: <52B301FE.4090905@illinois.edu> References: <52B284C9.1000907@doit.wisc.edu> <52B301FE.4090905@illinois.edu> Message-ID: <52B3124B.6090904@doit.wisc.edu> Thanks for clearing that up. It helps to be looking in the right place :-) Only using about 11% of the disk space on the manager node and only 1% on the worker node. I don't see any archive-log processes running, but I've believe I've seen them in the process list after stopping my bro instance, so I think I have an idea what I'd see if they were running. Gary Faulkner UW Madison Office of Campus Information Security 608-262-8591 On 12/19/2013 8:26 AM, Daniel Thayer wrote: > On 12/18/2013 11:31 PM, Gary Faulkner wrote: >> I'm trying to troubleshoot some odd behavior. I stopped receiving hourly >> email summaries and logs stopped being moved and compressed at some >> point this afternoon; although new logs are still being started hourly >> and the old log being renamed. >> >> As far as I can tell from the cron log the broctl cron job is running as >> scheduled. I tried running broctl cron manually, but no dice. It didn't >> see any hung processes from earlier cron jobs or any emails in the bro >> user's mailbox indicating something went awry. Does broctl cron produce >> any log output if it has trouble? >> > > Actually, broctl cron doesn't do log rotation or hourly email summaries. > In fact, those happen even if broctl isn't running at all. When it's > time to do a log rotation, Bro itself (on the manager host) executes > a script > /share/broctl/scripts/archive-log > and that script then executes a script > /share/broctl/scripts/postprocessors/summarize-connections > that generates and emails the connection summary report. > > So, I'd suggest making sure those scripts exist on your manager host, > check if you see any "archive-log" processes running in the background, > and then check if you're running out of disk space. > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6257 bytes Desc: S/MIME Cryptographic Signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131219/87b63a02/attachment.bin From gary at doit.wisc.edu Thu Dec 19 15:57:35 2013 From: gary at doit.wisc.edu (Gary Faulkner) Date: Thu, 19 Dec 2013 17:57:35 -0600 Subject: [Bro] Question on log rotation Message-ID: <52B387EF.6020606@doit.wisc.edu> I had a situation where log rotation and post-processing (summary emails) were not completing. New logs would get started and each previous hour's logs renamed, but not get compressed and moved, which means that many of the previous logs were still in /current (or are they really in /spool/manager?). In any case upon stopping bro via broctl it appears that only the most current log got processed and archived while all of the logs in between that never got processed seem to simply have gotten deleted. Are those logs simply lost or somewhere other than the dated archive folder and /current folder? If so, is this expected behavior, or is there normally something that would check to see if previous logs failed to rotate out? Regards, -- Gary Faulkner UW Madison Office of Campus Information Security 608-262-8591 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6257 bytes Desc: S/MIME Cryptographic Signature Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131219/a383b6f7/attachment.bin From ericooi at gmail.com Thu Dec 19 17:45:27 2013 From: ericooi at gmail.com (Eric Ooi) Date: Thu, 19 Dec 2013 19:45:27 -0600 Subject: [Bro] Question on log rotation In-Reply-To: <52B387EF.6020606@doit.wisc.edu> References: <52B387EF.6020606@doit.wisc.edu> Message-ID: <83EFBEAF-1397-4B1F-832B-86BDA66D1447@gmail.com> I?ve noticed this before on Bro 2.1. I ended up writing a quick python script and configured it as an hourly cron job to complete the compression and move. I?ve attached it here. Hope this helps. Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: bro_missed_rotate.py Type: text/x-python-script Size: 2101 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131219/2b1d2873/attachment.bin -------------- next part -------------- On Dec 19, 2013, at 5:57 PM, Gary Faulkner wrote: > I had a situation where log rotation and post-processing (summary emails) were not completing. New logs would get started and each previous hour's logs renamed, but not get compressed and moved, which means that many of the previous logs were still in /current (or are they really in /spool/manager?). In any case upon stopping bro via broctl it appears that only the most current log got processed and archived while all of the logs in between that never got processed seem to simply have gotten deleted. Are those logs simply lost or somewhere other than the dated archive folder and /current folder? If so, is this expected behavior, or is there normally something that would check to see if previous logs failed to rotate out? > > Regards, > > -- > Gary Faulkner > UW Madison > Office of Campus Information Security > 608-262-8591 > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From gary at doit.wisc.edu Thu Dec 19 18:33:25 2013 From: gary at doit.wisc.edu (Gary Faulkner) Date: Thu, 19 Dec 2013 20:33:25 -0600 Subject: [Bro] Question on log rotation In-Reply-To: <83EFBEAF-1397-4B1F-832B-86BDA66D1447@gmail.com> References: <52B387EF.6020606@doit.wisc.edu> <83EFBEAF-1397-4B1F-832B-86BDA66D1447@gmail.com> Message-ID: <52B3AC75.7010901@doit.wisc.edu> Thank you, that should actually be very helpful, and not just for this problem, but also because I have a couple python books on my reading list over winter break, along with Applied NSM. ~Gary On 12/19/2013 7:45 PM, Eric Ooi wrote: > I?ve noticed this before on Bro 2.1. I ended up writing a quick python script and configured it as an hourly cron job to complete the compression and move. I?ve attached it here. Hope this helps. > > Eric > > On Dec 19, 2013, at 5:57 PM, Gary Faulkner wrote: > >> I had a situation where log rotation and post-processing (summary emails) were not completing. New logs would get started and each previous hour's logs renamed, but not get compressed and moved, which means that many of the previous logs were still in /current (or are they really in /spool/manager?). In any case upon stopping bro via broctl it appears that only the most current log got processed and archived while all of the logs in between that never got processed seem to simply have gotten deleted. Are those logs simply lost or somewhere other than the dated archive folder and /current folder? If so, is this expected behavior, or is there normally something that would check to see if previous logs failed to rotate out? >> >> Regards, From itsecderek at gmail.com Fri Dec 20 07:27:58 2013 From: itsecderek at gmail.com (Derek Banks) Date: Fri, 20 Dec 2013 10:27:58 -0500 Subject: [Bro] Intel Framework, Notices, and sending out emails Message-ID: Hello all, I am standing up what will become my first production Bro server. I am most interested in the Intel Framework and alerting on Intel hits and sending those alerts to the team. So far in testing the Intel framework is working great and the hits aregoing to the notice.log per the exercise from the 2013 exchange ( I have about 28k domain names in there and it *seems* to be working great). The problem I am having is sending email alerts from the notices. If I append the following into local.bro (from the post http://mailman.icsi.berkeley.edu/pipermail/bro/2013-December/007185.html) then it seems to kill the intel hits (none to notices and no intel.log generated). hook Notice::policy(n: Notice::Info) { add n$actions[Notice::ACTION_ALARM]; } If this section of code comes before the intel framework section, then the intel hits function fine, but I get no emails. My broscripting talents are still in development (i.e. I am not very good at it), but it seems like I am overwriting a variable (n) since it is used for the notice script and in do_notice. Or I may be completely wrong - I admit I have much to learn. Has anyone else scripted emailing Intel hits out to the the team and could point me in the right direction? Ideally I want the intel hits to go to notice.log and email out anything that goes to notices. My section for the Intel Framework in local.bro is the following: @load policy/frameworks/intel/seen @load frameworks/intel/do_notice redef Intel::read_files += { "/usr/local/bro/share/bro/site/file1.dat", "/usr/local/bro/share/bro/site/file2.dat", "/usr/local/bro/share/bro/site/file3.dat", }; Best Regards, Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131220/2032c390/attachment.html From seth at icir.org Fri Dec 20 08:11:13 2013 From: seth at icir.org (Seth Hall) Date: Fri, 20 Dec 2013 11:11:13 -0500 Subject: [Bro] Intel Framework, Notices, and sending out emails In-Reply-To: References: Message-ID: <8FB0EEC2-CA45-4CFA-88A2-E49FB915B87F@icir.org> On Dec 20, 2013, at 10:27 AM, Derek Banks wrote: > hook Notice::policy(n: Notice::Info) > { > add n$actions[Notice::ACTION_ALARM]; > } Try.. add n$actions[Notice::ACTION_EMAIL]; The alarm action may be a little confusing. What's it doing is batching up notices and then sending them out on your log rotation interval in a single email. It's sort of the lower priority notices that you don't care about receiving the instant they occur but you'd still like to know about them soon. You also have the ability to do multiple actions per-notice so you don't need to worry about overwriting an action if you add multiple. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131220/746e90d3/attachment.bin From itsecderek at gmail.com Fri Dec 20 09:13:36 2013 From: itsecderek at gmail.com (Derek Banks) Date: Fri, 20 Dec 2013 12:13:36 -0500 Subject: [Bro] Intel Framework, Notices, and sending out emails In-Reply-To: <8FB0EEC2-CA45-4CFA-88A2-E49FB915B87F@icir.org> References: <8FB0EEC2-CA45-4CFA-88A2-E49FB915B87F@icir.org> Message-ID: Thanks Seth - that works. On Fri, Dec 20, 2013 at 11:11 AM, Seth Hall wrote: > > On Dec 20, 2013, at 10:27 AM, Derek Banks wrote: > > > hook Notice::policy(n: Notice::Info) > > { > > add n$actions[Notice::ACTION_ALARM]; > > } > > Try.. > > add n$actions[Notice::ACTION_EMAIL]; > > The alarm action may be a little confusing. What's it doing is batching > up notices and then sending them out on your log rotation interval in a > single email. It's sort of the lower priority notices that you don't care > about receiving the instant they occur but you'd still like to know about > them soon. > > You also have the ability to do multiple actions per-notice so you don't > need to worry about overwriting an action if you add multiple. :) > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131220/fcab1ad0/attachment.html From schworer at gmail.com Fri Dec 20 14:00:37 2013 From: schworer at gmail.com (Andy Schworer) Date: Fri, 20 Dec 2013 14:00:37 -0800 Subject: [Bro] Bro Silent Crash Message-ID: Possibly a newb question, but "broctl status" is reporting bro is running, but broctl restart reportings taht bro was not running because it crashed. Why the discrepancy? Also, log/current/stderr.log & srdout.log do not contain any information regarding the crash. Is this crash info stored else where? Thanks root at node4:/usr/local/bro# bin/broctl status > Name Type Host Status Pid Peers Started > > bro standalone localhost running 22880 0 06 Dec > 23:49:28 > root at node4:/usr/local/bro# bin/broctl restart > stopping ... > warning: cannot send mail > bro not running (was crashed) > starting ... > starting bro ... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131220/54cc71de/attachment.html From jlay at slave-tothe-box.net Fri Dec 20 14:12:21 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 20 Dec 2013 15:12:21 -0700 Subject: [Bro] Bro Silent Crash In-Reply-To: References: Message-ID: <66b329801613a3f5b475326fbe3f7645@localhost> On 2013-12-20 15:00, Andy Schworer wrote: > Possibly a newb question, but "broctl status" is reporting bro is > running, but broctl restart reportings taht bro was not running > because it crashed. ?Why the discrepancy? Also, > log/current/stderr.log & srdout.log do not contain any information > regarding the crash. ?Is this crash info stored else where? > > Thanks > >> root at node4:/usr/local/bro# bin/broctl status >> Name ? ? ? Type ? ? ? Host ? ? ? Status ? ? ? ?Pid ? >> ?Peers ?Started ? ? ? ? ? ? ? >> bro ? ? ? ?standalone localhost ?running ? ? ? 22880 ?0 ? >> ? ?06 Dec 23:49:28 >> root at node4:/usr/local/bro# bin/broctl restart >> stopping ... >> warning: cannot send mail >> bro not running (was crashed) >> starting ... >> starting bro ... How's your syslog lookin? Dec 19 14:09:26 gateway kernel: bro[16591]: segfault at 0 ip (null) sp 00007fffa051d1a8 error 14 in bro[400000+4e3000] James From schworer at gmail.com Sat Dec 21 13:58:37 2013 From: schworer at gmail.com (Andy Schworer) Date: Sat, 21 Dec 2013 13:58:37 -0800 Subject: [Bro] Bro Digest, Vol 92, Issue 24 In-Reply-To: References: Message-ID: I found a couple fresh segfaults in syslog. What is "error 4 in bro..."? Dec 20 21:37:41 node4 kernel: [13454949.834289] bro[19790]: segfault at 28 ip 00000000005effbc sp 00007fff438b7cf0 error 4 in bro[400000+4ab000] Dec 21 19:09:40 node4 kernel: [13532430.859123] bro[17657]: segfault at 74656e2e71 ip 000000000051c0b9 sp 00007fffe8e2a3c8 error 4 in bro[400000+4ab000] On Sat, Dec 21, 2013 at 12:00 PM, wrote: > Send Bro mailing list submissions to > bro at bro.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > or, via email, send a message with subject or body 'help' to > bro-request at bro.org > > You can reach the person managing the list at > bro-owner at bro.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Bro digest..." > > > Today's Topics: > > 1. Bro Silent Crash (Andy Schworer) > 2. Re: Bro Silent Crash (James Lay) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 20 Dec 2013 14:00:37 -0800 > From: Andy Schworer > Subject: [Bro] Bro Silent Crash > To: bro at bro.org > Message-ID: > < > CANQy5G0Ai6WGSGLC-+5QNSainuLb0vHmBDRG60AXt7bwhYwMtA at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Possibly a newb question, but "broctl status" is reporting bro is running, > but broctl restart reportings taht bro was not running because it crashed. > Why the discrepancy? Also, log/current/stderr.log & srdout.log do not > contain any information regarding the crash. Is this crash info stored > else where? > > Thanks > > root at node4:/usr/local/bro# bin/broctl status > > Name Type Host Status Pid Peers Started > > > > bro standalone localhost running 22880 0 06 Dec > > 23:49:28 > > root at node4:/usr/local/bro# bin/broctl restart > > stopping ... > > warning: cannot send mail > > bro not running (was crashed) > > starting ... > > starting bro ... > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131220/54cc71de/attachment-0001.html > > ------------------------------ > > Message: 2 > Date: Fri, 20 Dec 2013 15:12:21 -0700 > From: James Lay > Subject: Re: [Bro] Bro Silent Crash > To: > Message-ID: <66b329801613a3f5b475326fbe3f7645 at localhost> > Content-Type: text/plain; charset=UTF-8; format=flowed > > On 2013-12-20 15:00, Andy Schworer wrote: > > Possibly a newb question, but "broctl status" is reporting bro is > > running, but broctl restart reportings taht bro was not running > > because it crashed. ?Why the discrepancy? Also, > > log/current/stderr.log & srdout.log do not contain any information > > regarding the crash. ?Is this crash info stored else where? > > > > Thanks > > > >> root at node4:/usr/local/bro# bin/broctl status > >> Name ? ? ? Type ? ? ? Host ? ? ? Status ? ? ? ?Pid ? > >> ?Peers ?Started ? ? ? ? ? ? ? > >> bro ? ? ? ?standalone localhost ?running ? ? ? 22880 ?0 ? > >> ? ?06 Dec 23:49:28 > >> root at node4:/usr/local/bro# bin/broctl restart > >> stopping ... > >> warning: cannot send mail > >> bro not running (was crashed) > >> starting ... > >> starting bro ... > > How's your syslog lookin? > > Dec 19 14:09:26 gateway kernel: bro[16591]: segfault at 0 ip (null) sp > 00007fffa051d1a8 error 14 in bro[400000+4e3000] > > James > > > ------------------------------ > > _______________________________________________ > Bro mailing list > Bro at bro.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > End of Bro Digest, Vol 92, Issue 24 > *********************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131221/ebdc32f7/attachment.html From zainab.abaid at gmail.com Mon Dec 23 03:28:03 2013 From: zainab.abaid at gmail.com (Zainab Abaid) Date: Mon, 23 Dec 2013 16:28:03 +0500 Subject: [Bro] Empty mime type in http.log Message-ID: Hi, I'm trying to use Bro to identify content types in http traces - so I'm looking at the orig_mime_types and resp_mime_types columns in http.log. Now, I captured a trace where I blocked all applications except Google chrome, and then used Chrome to play a bunch of videos on Dailymotion. Then I ran Bro over this trace and then ran the following command over the output: cat http.log | bro-cut orig_mime_types resp_mime_types | sort | uniq -c This is the output I got: 88 - - 35 - application/octet-stream 12 - application/xml 25 - image/gif 188 - image/jpeg 1 - image/png 13 - text/html 59 - text/plain 24 text/plain text/plain I can't figure out why blanks are appearing in so many of the lines. There are 88 lines where the content type (in both resp and orig columns) is empty - what does that signify? I did see that Bro uses signature matching to correct missing or incorrect file types. Does a "-" mean Bro wasn't able to match any signature to the content being transferred? Or that it found nothing in the HTTP header's content type field? Thanks, Zainab -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131223/ae0fcff4/attachment.html From bkellogg at dresser-rand.com Mon Dec 23 13:26:53 2013 From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN)) Date: Mon, 23 Dec 2013 21:26:53 +0000 Subject: [Bro] count connection bytes Message-ID: I'm very new to Bro scripting so I thank you ahead of time for your patience. I'm trying to write a simple script that just prints the bytes for the Rx and Tx of a TCP session. Below is what I have, but it isn't giving me anything. redef use_conn_size_analyzer = T; event connection_finished(c:connection) { print c$orig$num_bytes_ip; print c$resp$num_bytes_ip; } I'm probably missing something obvious but it is escaping me. thanks Thank you, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131223/a066799a/attachment.html From jlay at slave-tothe-box.net Mon Dec 23 15:47:35 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 23 Dec 2013 16:47:35 -0700 Subject: [Bro] Netwrix Password Manager Message-ID: http://www.netwrix.com/free_password_manager.html Free up to 100 users...interesting. James From jlay at slave-tothe-box.net Mon Dec 23 15:51:20 2013 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 23 Dec 2013 16:51:20 -0700 Subject: [Bro] Netwrix Password Manager In-Reply-To: References: Message-ID: On 2013-12-23 16:47, James Lay wrote: > http://www.netwrix.com/free_password_manager.html > > Free up to 100 users...interesting. > > James Kindly disregard...sorry about that. James From renaud.luca at gmail.com Mon Dec 23 17:58:15 2013 From: renaud.luca at gmail.com (Luca Renaud) Date: Tue, 24 Dec 2013 01:58:15 +0000 Subject: [Bro] Standard Bro checks on tcpdump files.What that really means? Message-ID: When we do a : bro -r dumpfile ,on a previously recorded dump file what are the standard checks Bro really executes? ALL that come defined within the Bro scripts directory ( export BROPATH=/...........)? Backdoor.bro,etc.etc.? Or just A PART of it? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131224/19071271/attachment.html From init.conf at gmail.com Mon Dec 23 18:46:09 2013 From: init.conf at gmail.com (Aashish Sharma) Date: Mon, 23 Dec 2013 18:46:09 -0800 Subject: [Bro] Standard Bro checks on tcpdump files.What that really means? In-Reply-To: References: Message-ID: When running bro -r dumpfile ; you have to specify a policy file or a set of policy files which are used to process the dumpfile. The most default option is to use local.bro (found in /share/bro/site/ folder. So this becomes: bro -r dumpfile local.bro Once successful, looking in the log directory for a file called loaded_scripts.log. This log will list the policy files which were used for this specific bro run and analysis. (if you are running bro as a daemon after doing broctl start, then you can use broctl scripts all to get a listing of loaded_policies as well) Aashish On Dec 23, 2013, at 5:58 PM, Luca Renaud wrote: > When we do a : bro -r dumpfile ,on a previously recorded dump file what are the standard checks Bro really executes? ALL that come defined within the Bro scripts directory ( export BROPATH=/...........)? Backdoor.bro,etc.etc.? Or just A PART of it? > Thanks. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From wsladekjr at hotmail.com Tue Dec 24 10:50:42 2013 From: wsladekjr at hotmail.com (Ward Sladek) Date: Tue, 24 Dec 2013 12:50:42 -0600 Subject: [Bro] Disable Base script Message-ID: What is the best practice for disabling a Base script? For example, I would like to disable syslog monitoring all together. I have included the following in my local.bro: event bro_init() { Log::disable_stream(Syslog::LOG); } This disables the logging of syslog messages, but does it prevent Bro from loading the base/protocols/syslog scripts? If not, what is the best practice for doing so? I'm trying to tune/tweak bro for best performance. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131224/1e789707/attachment.html From bkellogg at dresser-rand.com Thu Dec 26 08:26:23 2013 From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN)) Date: Thu, 26 Dec 2013 16:26:23 +0000 Subject: [Bro] table of record type Message-ID: type host_type: record { bytes: int; stime: time; }; type hostTable: table[addr] of host_type; global hosts: hostTable; I've create the above structure, but I haven't a clue on the syntax to dynamically populate it. I've tried the below with no success. hosts[c$id$orig_h] = [[$bytes=0, $stime=network_time()]]; Thank you, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131226/dedacb2a/attachment.html From bernhard at ICSI.Berkeley.EDU Thu Dec 26 08:54:07 2013 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Thu, 26 Dec 2013 17:54:07 +0100 Subject: [Bro] table of record type In-Reply-To: References: Message-ID: <36BDB345-CD7D-4BDB-806D-DCA4563C5A52@icsi.berkeley.edu> Hi, you can populate the table by using? > hosts[c$id$orig_h] = [$bytes=+0, $stime=network_time()]; You used one set of [] to many (the table contains a plain record). Furthermore, positive numbers are counts by default - if you specify them explicitly you have to prefix them with + to tell Bro to use an integer type instead. Bernhard On Dec 26, 2013, at 5:26 PM, Kellogg, Brian D (OLN) wrote: > type host_type: record { > bytes: int; > stime: time; > }; > > type hostTable: table[addr] of host_type; > > global hosts: hostTable; > > I've create the above structure, but I haven't a clue on the syntax to dynamically populate it. I've tried the below with no success. > > hosts[c$id$orig_h] = [[$bytes=0, $stime=network_time()]]; > > > > Thank you, > Brian > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From drummachina at tdhack.com Sun Dec 29 11:25:41 2013 From: drummachina at tdhack.com (drum) Date: Sun, 29 Dec 2013 20:25:41 +0100 Subject: [Bro] extract jar files from HTTP stream Message-ID: Hello, Is there a tutorial for version 2.2 of BRO? I'd like to understand how can I write my own scripts to support extraction of verious files, like jar. So far I tried adding "application/jar" (it was logged to /nsm/bro/logs/current/files. log) as mime type to /opt/bro/share/bro/file-extraction/extract.bro file but it seems I have to do something else too as this change is not capturing files to /nsm/bro/extracted/ directory. I guess it's not simple as that ;) I should also mention that I am using BRO installed within SecurityOnion distro. I posted this question there too, but got reply with links to version 2.1 which is not compatible with 2.2. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131229/f6b4be4b/attachment.html From bkellogg at dresser-rand.com Sun Dec 29 13:20:33 2013 From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN)) Date: Sun, 29 Dec 2013 21:20:33 +0000 Subject: [Bro] reference table in a table Message-ID: Sorry for another syntax question again, but how do I add table entries to and reference a table within a table. Both tables are tables of record types. I've tried the below: Reference: hosts[c$id$orig_h]$TxDests[c$id$resp_h]$num += 1; Add entry: hosts[c$id$orig_h] = [$TxDests[c$id$resp_h]=[$num=1], $state=F]; Thank you, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131229/a93f6a85/attachment.html From bkellogg at dresser-rand.com Sun Dec 29 15:58:11 2013 From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN)) Date: Sun, 29 Dec 2013 23:58:11 +0000 Subject: [Bro] reference table in a table In-Reply-To: References: Message-ID: Found the answer here finally, thanks: http://article.gmane.org/gmane.comp.security.detection.bro/1547/match=table+record Thank you, Brian ________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Kellogg, Brian D (OLN) [bkellogg at dresser-rand.com] Sent: Sunday, December 29, 2013 4:20 PM To: bro at bro.org Subject: [Bro] reference table in a table Sorry for another syntax question again, but how do I add table entries to and reference a table within a table. Both tables are tables of record types. I've tried the below: Reference: hosts[c$id$orig_h]$TxDests[c$id$resp_h]$num += 1; Add entry: hosts[c$id$orig_h] = [$TxDests[c$id$resp_h]=[$num=1], $state=F]; Thank you, Brian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131229/0e015ff2/attachment.html From filus at psc.edu Mon Dec 30 05:34:44 2013 From: filus at psc.edu (Shane Filus) Date: Mon, 30 Dec 2013 08:34:44 -0500 Subject: [Bro] extract jar files from HTTP stream In-Reply-To: References: Message-ID: <52C17674.6040500@psc.edu> On 12/29/13 2:25 PM, drum wrote: > Hello, > > > Is there a tutorial for version 2.2 of BRO? I'd like to understand how > can I write my own scripts to support extraction of verious files, > like jar. So far I tried adding "application/jar" (it was logged to > /nsm/bro/logs/current/files. > log) as mime type to /opt/bro/share/bro/file-extraction/extract.bro > file but it seems I have to do something else too as this change is > not capturing files to /nsm/bro/extracted/ directory. Hello, Check the second example under 'Adding Analysis' for a start in file extraction. http://www.bro.org/sphinx/frameworks/file-analysis.html#adding-analysis Also, not sure how it maps to Sec Onion, but there is $PREFX/share/bro/base/files/extract/main.bro from a source install. Might be your 'extract.bro'? I don't that file name in either 2.1 or 2.2 source trees. Used the code below to do something similar. There's probably a more elegant, or efficient solution, but it seems to working as expected, given the limited testing I've done. # define file extraction filters const match_file_source = /HTTP/ | /IRC/ | /IRC_DATA/ | /FTP/ | /FTP_DATA/ &redef; const match_file_mime = /text\/x-perl/ | /text\/x-msdos-batch/ | /text\/x-java/ | /application\/x-gzip/ | /application\/x-bzip2/ | /application\/x-dosexec/ | /application\/zip/ | /application\/jar/ | /application\/x-tar/ | /application\/x-archive/ | /application\/mac-binhex40/ | /application\/x-java-keystore/ | /application\/x-java-jce-keystore/ | /application\/x-executable/ | /application\/javascript/ &redef; # add analyer to file_new event event file_new(f: fa_file) { if ( f?$mime_type && match_file_source in f$source && match_file_mime in f$mime_type ) Files::add_analyzer(f, Files::ANALYZER_EXTRACT); } Thanks, Shane From sconzo at visiblerisk.com Mon Dec 30 10:52:23 2013 From: sconzo at visiblerisk.com (Mike Sconzo) Date: Mon, 30 Dec 2013 12:52:23 -0600 Subject: [Bro] extract jar files from HTTP stream In-Reply-To: <52C17674.6040500@psc.edu> References: <52C17674.6040500@psc.edu> Message-ID: As an aside, you might also add application/zip to your file extract (Shane has it in their list) as JAR files are also Zip files. On Mon, Dec 30, 2013 at 7:34 AM, Shane Filus wrote: > On 12/29/13 2:25 PM, drum wrote: >> Hello, >> >> >> Is there a tutorial for version 2.2 of BRO? I'd like to understand how >> can I write my own scripts to support extraction of verious files, >> like jar. So far I tried adding "application/jar" (it was logged to >> /nsm/bro/logs/current/files. >> log) as mime type to /opt/bro/share/bro/file-extraction/extract.bro >> file but it seems I have to do something else too as this change is >> not capturing files to /nsm/bro/extracted/ directory. > Hello, > > > Check the second example under 'Adding Analysis' for a start in file > extraction. > http://www.bro.org/sphinx/frameworks/file-analysis.html#adding-analysis > > Also, not sure how it maps to Sec Onion, but there is > $PREFX/share/bro/base/files/extract/main.bro from a source install. > Might be your 'extract.bro'? I don't that file name in either 2.1 or 2.2 > source trees. > > Used the code below to do something similar. There's probably a more > elegant, or efficient solution, but it seems to working as expected, > given the limited testing I've done. > > > # define file extraction filters > const match_file_source = /HTTP/ | > /IRC/ | > /IRC_DATA/ | > /FTP/ | > /FTP_DATA/ &redef; > > const match_file_mime = /text\/x-perl/ | > /text\/x-msdos-batch/ | > /text\/x-java/ | > /application\/x-gzip/ | > /application\/x-bzip2/ | > /application\/x-dosexec/ | > /application\/zip/ | > /application\/jar/ | > /application\/x-tar/ | > /application\/x-archive/ | > /application\/mac-binhex40/ | > /application\/x-java-keystore/ | > /application\/x-java-jce-keystore/ | > /application\/x-executable/ | > /application\/javascript/ &redef; > > # add analyer to file_new event > event file_new(f: fa_file) > { > if ( f?$mime_type && > match_file_source in f$source && > match_file_mime in f$mime_type ) > Files::add_analyzer(f, Files::ANALYZER_EXTRACT); > } > > > Thanks, > > > Shane > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- cat ~/.bash_history > documentation.txt