From jeff at jeffhammett.com Sun Nov 2 21:00:22 2014 From: jeff at jeffhammett.com (Jeff Hammett) Date: Sun, 2 Nov 2014 21:00:22 -0800 Subject: [Bro] Invalid_Server_Cert entries in notice.log Message-ID: I am seeing a lot of entries in notice.log for invalid SSL certs; SSL::Invalid_Server_Cert SSL certificate validation failed with (unable to get local issuer certificate) These are for legitimate sites, that I think have valid SSL certs. When I go to the IP listed in a web browser they do indeed have valid certificates. Is there any way to further verify that nothing strange is going on. And if everything is ok, is there a way suppress these warnings for sites that do have valid certs, so that if any users visit sites with self signed or otherwise invalid certificates they?ll stand out in the notice.log? A few examples from notice.log: #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path notice #open 2014-11-02-20-28-34 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc proto note msg sub src dst p n peer_descr actions suppress_for dropped remote_location.country_code remote_location.region remote_location.city remote_location.latitude remote_location.longitude #types time string addr port addr port string string string enum enum string string addr addr port count string set[enum] interval bool string string string double double 1414989068.580505 CyZhPK15RzCUnN7ura 192.168.1.143 49285 134.170.165.251 443 - - - tcp SSL::Invalid_Server_Cert SSL certificate validation failed with (unable to get local issuer certificate) CN=fe2.update.microsoft.com,OU=WUPDS,O=Microsoft,L=Redmond,ST=Washington,C=US 192.168.1.143 134.170.165.251 443 - bro Notice::ACTION_LOG 3600.000000 F - - - - - 1414989315.341931 C1Ll1O381lfcKl4H3k 192.168.1.105 57151 17.158.52.16 443 - - - tcp SSL::Invalid_Server_Cert SSL certificate validation failed with (unable to get local issuer certificate) CN=*.icloud.com,O=Apple Inc.,L=Cupertino,ST=California,C=US 192.168.1.105 17.158.52.16 443 - bro Notice::ACTION_LOG 3600.000000 F - - - - - 1414989316.321356 CHwvguxImPT6pSiU7 192.168.1.105 57152 17.158.52.77 443 - - - tcp SSL::Invalid_Server_Cert SSL certificate validation failed with (unable to get local issuer certificate) CN=*.icloud.com,O=Apple Inc.,L=Cupertino,ST=California,C=US 192.168.1.105 17.158.52.77 443 - bro Notice::ACTION_LOG 3600.000000 F - - - - - 1414989495.154433 C6TtbD2IR6tOvyBEze 192.168.1.195 50506 72.32.45.19 443 - - - tcp SSL::Invalid_Server_Cert SSL certificate validation failed with (unable to get local issuer certificate) CN=giga.logs.roku.com,O=Roku\, Inc.,ST=California,C=US 192.168.1.195 72.32.45.19 443 bro Notice::ACTION_LOG 3600.000000 F - - - - - 1414989678.402401 C2uDCc4cE0Brc2GUV1 192.168.1.143 49387 184.180.124.10 443 - - - tcp SSL::Invalid_Server_Cert SSL certificate validation failed with (unable to get local issuer certificate) CN=a248.e.akamai.net,O=Akamai Technologies\, Inc.,L=Cambridge,ST=MA,C=US 192.168.1.143 184.180.124.10 443 - bro Notice::ACTION_LOG 3600.000000 F - - - - - 1414990083.832444 C4Z0274jeydu7rN8G1 192.168.1.105 57356 17.158.52.69 443 - - - tcp SSL::Invalid_Server_Cert SSL certificate validation failed with (unable to get local issuer certificate) CN=*.icloud.com,O=Apple Inc.,L=Cupertino,ST=California,C=US 192.168.1.105 17.158.52.69 443 - bro Notice::ACTION_LOG 3600.000000 F - - - - - 1414990161.080209 CKVmf6WV0KGxfT3T7 192.168.1.105 57369 17.158.52.68 443 - - - tcp SSL::Invalid_Server_Cert SSL certificate validation failed with (unable to get local issuer certificate) CN=*.icloud.com,O=Apple Inc.,L=Cupertino,ST=California,C=US 192.168.1.105 17.158.52.68 443 - bro Notice::ACTION_LOG 3600.000000 F - - - - - -- Jeff Hammett From johanna at icir.org Mon Nov 3 08:52:35 2014 From: johanna at icir.org (Johanna Amann) Date: Mon, 3 Nov 2014 08:52:35 -0800 Subject: [Bro] Invalid_Server_Cert entries in notice.log In-Reply-To: References: Message-ID: <20141103165235.GA42108@LadyMacbeth.local> Hello Jeff, > I am seeing a lot of entries in notice.log for invalid SSL certs; > SSL::Invalid_Server_Cert SSL certificate validation failed with > (unable to get local issuer certificate) > > These are for legitimate sites, that I think have valid SSL certs. When > I go to the IP listed in a web browser they do indeed have valid > certificates. You stumbled accross one of the slightly annoying parts of the current certificate ecosystem here. What happens is that those servers are not sending a complete certificate chain. Instead, they only send the end-host certificates without the intermediate CA certificates that are necessary for verification. Browsers tend to still be able to verify the end-host certificates, even when the intermediates are missing. For example, Firefox just keeps a cached list of all intermediate certificates it ever encounters and uses those to build the chain and Browsers like Chrome use an extension field present in the certificate to automatically download missing intermediate certs. > Is there any way to further verify that nothing strange is going on. And > if everything is ok, is there a way suppress these warnings for sites > that do have valid certs, so that if any users visit sites with self > signed or otherwise invalid certificates they?ll stand out in the > notice.log? There is nothing strange going on and, sadly, at the Moment there is nothing you can do about these notices. The sites are not sending complete chains that can easily be verified and it is not easy to replicate Browser behavior in those instances. You also can verify that if you use tools like wget or curl, they also will complain about certificate mismatches (they use similar code to Bro for certificate verification). I hope this clears things up a bit, Johanna From jeff at jeffhammett.com Tue Nov 4 15:24:02 2014 From: jeff at jeffhammett.com (Jeff Hammett) Date: Tue, 4 Nov 2014 15:24:02 -0800 Subject: [Bro] Bro Script to detect plain text passwords? Message-ID: <9B561755-0491-4D36-A8B7-83C2104152EC@jeffhammett.com> I recently demo?d Tenable?s Passive Vulnerability Scanner, but found that it wasn?t a good fit for my environment. However it did have one nice feature I liked, the ability to detect passwords sent in plain text. Does Bro have this functionality? Or would it be feasible to write a script to do so? (I haven?t written any scripts yet, but am interested). I think I would be most interested in detecting plain text passwords used for http logins, but wouldn?t mind monitoring for other protocols as well. Jeff From anthony.kasza at gmail.com Tue Nov 4 16:32:02 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Tue, 4 Nov 2014 16:32:02 -0800 Subject: [Bro] Bro Script to detect plain text passwords? In-Reply-To: <9B561755-0491-4D36-A8B7-83C2104152EC@jeffhammett.com> References: <9B561755-0491-4D36-A8B7-83C2104152EC@jeffhammett.com> Message-ID: Absolutely. This is something well suited for Bro's policy scripts. -AK On Nov 4, 2014 3:45 PM, "Jeff Hammett" wrote: > I recently demo?d Tenable?s Passive Vulnerability Scanner, but found that > it wasn?t a good fit for my environment. However it did have one nice > feature I liked, the ability to detect passwords sent in plain text. > > Does Bro have this functionality? Or would it be feasible to write a > script to do so? (I haven?t written any scripts yet, but am interested). > > I think I would be most interested in detecting plain text passwords used > for http logins, but wouldn?t mind monitoring for other protocols as well. > > Jeff > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141104/46a3a513/attachment.html From seth at icir.org Tue Nov 4 18:09:00 2014 From: seth at icir.org (Seth Hall) Date: Tue, 4 Nov 2014 21:09:00 -0500 Subject: [Bro] Bro Script to detect plain text passwords? In-Reply-To: <9B561755-0491-4D36-A8B7-83C2104152EC@jeffhammett.com> References: <9B561755-0491-4D36-A8B7-83C2104152EC@jeffhammett.com> Message-ID: > On Nov 4, 2014, at 6:24 PM, Jeff Hammett wrote: > > Does Bro have this functionality? Or would it be feasible to write a script to do so? (I haven?t written any scripts yet, but am interested). Even better, it's something that we ship with, it just needs to be enabled. We decided to have a default setting of not capturing passwords. If you run Bro through BroControl, add the following line to your local.bro and do the check/install/restart commands in broctl. redef HTTP::default_capture_password = T; It will be in a field in your http.log named "password". There will also be a field named "username". > I think I would be most interested in detecting plain text passwords used for http logins, but wouldn?t mind monitoring for other protocols as well. For FTP: redef FTP::default_capture_password = T; Channel passwords are logged by default for IRC too. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From npratley at redhat.com Tue Nov 4 18:25:51 2014 From: npratley at redhat.com (Nick Pratley) Date: Wed, 05 Nov 2014 12:25:51 +1000 Subject: [Bro] Bro Script to detect plain text passwords? In-Reply-To: References: <9B561755-0491-4D36-A8B7-83C2104152EC@jeffhammett.com> Message-ID: <54598AAF.5010304@redhat.com> An example of detecting HTTP basic authentication is given here: http://ryesecurity.blogspot.com.au/2012/05/learning-bro-scripting-language.html On 11/05/2014 10:32 AM, anthony kasza wrote: > Absolutely. This is something well suited for Bro's policy scripts. > > -AK > > On Nov 4, 2014 3:45 PM, "Jeff Hammett" > wrote: > > I recently demo?d Tenable?s Passive Vulnerability Scanner, but found that it wasn?t a good fit > for my environment. However it did have one nice feature I liked, the ability to detect > passwords sent in plain text. > > Does Bro have this functionality? Or would it be feasible to write a script to do so? (I haven?t > written any scripts yet, but am interested). > > I think I would be most interested in detecting plain text passwords used for http logins, but > wouldn?t mind monitoring for other protocols as well. > > Jeff > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From npratley at redhat.com Tue Nov 4 18:29:47 2014 From: npratley at redhat.com (Nick Pratley) Date: Wed, 05 Nov 2014 12:29:47 +1000 Subject: [Bro] Bro Script to detect plain text passwords? In-Reply-To: References: <9B561755-0491-4D36-A8B7-83C2104152EC@jeffhammett.com> Message-ID: <54598B9B.3080909@redhat.com> Oh, I hadn't seen this before I sent my reply. Good to know, thanks. On 11/05/2014 12:09 PM, Seth Hall wrote: > >> On Nov 4, 2014, at 6:24 PM, Jeff Hammett wrote: >> >> Does Bro have this functionality? Or would it be feasible to write a script to do so? (I haven?t written any scripts yet, but am interested). > > Even better, it's something that we ship with, it just needs to be enabled. We decided to have a default setting of not capturing passwords. If you run Bro through BroControl, add the following line to your local.bro and do the check/install/restart commands in broctl. > > redef HTTP::default_capture_password = T; > > It will be in a field in your http.log named "password". There will also be a field named "username". > >> I think I would be most interested in detecting plain text passwords used for http logins, but wouldn?t mind monitoring for other protocols as well. > > For FTP: > redef FTP::default_capture_password = T; > > Channel passwords are logged by default for IRC too. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From npratley at redhat.com Tue Nov 4 21:02:38 2014 From: npratley at redhat.com (Nick Pratley) Date: Wed, 05 Nov 2014 15:02:38 +1000 Subject: [Bro] Regular expression parenthesised groups Message-ID: <5459AF6E.80301@redhat.com> Hi, is there a way to capture groups as part of a regular expression using parentheses in a Bro script? For example, to extract the value of a query string variable in a URI - /foo=([^&]*)/ - I just want the value in the parentheses. If not I guess I can do this with the sub and split functions in a way similar to http://stackoverflow.com/questions/10126956/capture-value-out-of-query-string-with-regex but I just thought I'd ask. Thanks, -- Nick Pratley Information Security, Red Hat, Inc. +61 7 3514 8268 From seth at icir.org Wed Nov 5 05:48:04 2014 From: seth at icir.org (Seth Hall) Date: Wed, 5 Nov 2014 08:48:04 -0500 Subject: [Bro] Regular expression parenthesised groups In-Reply-To: <5459AF6E.80301@redhat.com> References: <5459AF6E.80301@redhat.com> Message-ID: > On Nov 5, 2014, at 12:02 AM, Nick Pratley wrote: > > If not I guess I can do this with the sub and split functions in a way similar to > http://stackoverflow.com/questions/10126956/capture-value-out-of-query-string-with-regex but I just > thought I'd ask. Sorry, we don't have group capturing in our regular expressions. Here's an example where I've used a split function to achieve a similar result... https://github.com/bro/bro/blob/master/scripts/base/utils/paths.bro#L34 .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jeff at jeffhammett.com Wed Nov 5 07:36:13 2014 From: jeff at jeffhammett.com (Jeff Hammett) Date: Wed, 5 Nov 2014 07:36:13 -0800 Subject: [Bro] Bro Script to detect plain text passwords? In-Reply-To: References: <9B561755-0491-4D36-A8B7-83C2104152EC@jeffhammett.com> Message-ID: <9C5B596A-D837-4A17-95EE-D57DBC8D7D55@jeffhammett.com> > On Nov 4, 2014, at 6:09 PM, Seth Hall wrote: > Even better, it's something that we ship with, it just needs to be enabled. We decided to have a default setting of not capturing passwords. If you run Bro through BroControl, add the following line to your local.bro and do the check/install/restart commands in broctl. > > redef HTTP::default_capture_password = T; > > It will be in a field in your http.log named "password". There will also be a field named "username". Thanks! This is what I was looking for. One more question, how would I go about logging an entry in the notice.log when plaintext passwords are discovered? -- Jeff Hammett https://www.jeffhammett.com From anthony.kasza at gmail.com Wed Nov 5 12:07:25 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Wed, 5 Nov 2014 12:07:25 -0800 Subject: [Bro] URI Parsing Function Message-ID: Is there a function in scriptland that will decompose a URI? For example, given "http://example.com/path" the function would return a record with protocol, domain, and path? -AK -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141105/dec5da38/attachment.html From seth at icir.org Wed Nov 5 12:44:30 2014 From: seth at icir.org (Seth Hall) Date: Wed, 5 Nov 2014 15:44:30 -0500 Subject: [Bro] URI Parsing Function In-Reply-To: References: Message-ID: > On Nov 5, 2014, at 3:07 PM, anthony kasza wrote: > > Is there a function in scriptland that will decompose a URI? For example, given "http://example.com/path" the function would return a record with protocol, domain, and path? Not yet. I hope this means you're going to be doing this?!? :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From npratley at redhat.com Wed Nov 5 16:26:37 2014 From: npratley at redhat.com (Nick Pratley) Date: Thu, 06 Nov 2014 10:26:37 +1000 Subject: [Bro] Regular expression parenthesised groups In-Reply-To: References: <5459AF6E.80301@redhat.com> Message-ID: <545AC03D.4090708@redhat.com> On 11/05/2014 11:48 PM, Seth Hall wrote: > Sorry, we don't have group capturing in our regular expressions. Here's an example where I've used a split function to achieve a similar result... > > https://github.com/bro/bro/blob/master/scripts/base/utils/paths.bro#L34 Thanks Seth. I did manage to get it working using split. From anthony.kasza at gmail.com Wed Nov 5 16:34:07 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Wed, 5 Nov 2014 16:34:07 -0800 Subject: [Bro] URI Parsing Function In-Reply-To: References: Message-ID: Uh, OK. On Nov 5, 2014 12:44 PM, "Seth Hall" wrote: > > > On Nov 5, 2014, at 3:07 PM, anthony kasza > wrote: > > > > Is there a function in scriptland that will decompose a URI? For > example, given "http://example.com/path" the function would return a > record with protocol, domain, and path? > > Not yet. I hope this means you're going to be doing this?!? :) > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141105/83dde635/attachment.html From tiburcesotohou at yahoo.fr Thu Nov 6 08:33:50 2014 From: tiburcesotohou at yahoo.fr (SOTOHOU Osince Tiburce) Date: Thu, 6 Nov 2014 16:33:50 +0000 (UTC) Subject: [Bro] communication between several Bro instances Message-ID: <69358787.235605.1415291630922.JavaMail.yahoo@jws11172.mail.ir2.yahoo.com> Hi;I'm working with Bro and in the documentation, it is possible to share events? between several Bro instances.Now, i would like to know how and the differents steps to do that.Thank you for your responses. regards -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141106/76981717/attachment.html From jsiwek at illinois.edu Thu Nov 6 09:04:29 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 6 Nov 2014 17:04:29 +0000 Subject: [Bro] communication between several Bro instances In-Reply-To: <69358787.235605.1415291630922.JavaMail.yahoo@jws11172.mail.ir2.yahoo.com> References: <69358787.235605.1415291630922.JavaMail.yahoo@jws11172.mail.ir2.yahoo.com> Message-ID: <07858F64-D3F5-47C3-854B-2449B3729B38@illinois.edu> > On Nov 6, 2014, at 10:33 AM, SOTOHOU Osince Tiburce wrote: > > I'm working with Bro and in the documentation, it is possible to share events between several Bro instances. > Now, i would like to know how and the differents steps to do that. As a first step, these example scripts may be helpful in pointing out what other specific documentation to search for in the script-reference docs: https://gist.github.com/jsiwek/9509541 https://gist.github.com/jsiwek/9509511 - Jon From blackhole.em at gmail.com Thu Nov 6 16:54:13 2014 From: blackhole.em at gmail.com (Joe Blow) Date: Thu, 6 Nov 2014 19:54:13 -0500 Subject: [Bro] elastic search / bro questions Message-ID: Hey all, Just going to throw this out there and hope some people are willing to potentially share some learning experiences if they have any. We have a system which generates around 15k-30k BRO events/sec and are trying to ingest these logs into a fairly beefy elasticsearch cluster. Total cluster memory ~300GB, storage ~300TB. Long story short, we're having some problems keeping up with this feed. Does anyone have any performance tuning with this module? I've played a lot with rsyslog batch sizes with elasticsearch and was hoping there would be some simple directive i could try and apply to BRO. Does anyone have this experience here? Does this module batch anything? Thanks in advance. Cheers, JB -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141106/75a47e43/attachment.html From mkhan04 at gmail.com Thu Nov 6 18:25:29 2014 From: mkhan04 at gmail.com (M K) Date: Thu, 6 Nov 2014 21:25:29 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: References: Message-ID: Unless it's changed within the past month or so, the ElasticSearch writer that comes with Bro is very alpha-level code. For the most part it fires and forgets and can be prone to losing messages if your cluster isn't able to keep up or some other situation causes it not to be able to ingest the data properly. Your best bet, as of now, is to write out the logs to disk and use some intermediary program to process the logs and ingest them into ES. Logstash can help, but with the default custom format Bro uses, it can't parse the data properly. If you're using Bro 2.3, you can modify the output format of the ascii writer to use json instead and then use logstash to feed the data relatively easily into ES. Further, I'd recommend using a rabbit river so ES can ingest the data at its leisure. If you're stuck with the non-json format, well your options are kinda limited. You can write a crazy custom logstash conf using grok (which is super inefficient) or figure out some other mechanism. As an aside, I've written a custom logstash filter that processes the custom bro format and is, to a limited extent, bro type aware so it can take old-style bro logs relatively easily and make it more usable (numbers are turned into numbers and sets, vectors and tables are turned into arrays -- same as how I've seen the ES writer output data). There are some caveats in its usage though. I'm putting the finishing touches on it and plan to release it when I get a chance (hopefully within the next week or two). On Thu, Nov 6, 2014 at 7:54 PM, Joe Blow wrote: > Hey all, > > Just going to throw this out there and hope some people are willing to > potentially share some learning experiences if they have any. > > We have a system which generates around 15k-30k BRO events/sec and are > trying to ingest these logs into a fairly beefy elasticsearch cluster. > Total cluster memory ~300GB, storage ~300TB. > > Long story short, we're having some problems keeping up with this feed. > Does anyone have any performance tuning with this module? I've played a > lot with rsyslog batch sizes with elasticsearch and was hoping there would > be some simple directive i could try and apply to BRO. > > Does anyone have this experience here? Does this module batch anything? > > Thanks in advance. > > Cheers, > > JB > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141106/2143aec4/attachment.html From blackhole.em at gmail.com Thu Nov 6 18:37:50 2014 From: blackhole.em at gmail.com (Joe Blow) Date: Thu, 6 Nov 2014 21:37:50 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: References: Message-ID: Yuck. I was really hoping this wasn't the way. From everything you said, the river is where i'm focusing. I really, really dislike logstash (i'd rather bend rsyslog + ES output plugin to my liking any day). I've written a few custom ES output/input parsers and many SOLR parsers that will parse bro logs, proxy logs, etc..., but would rather focus on something more native to output to ES if possible. I guess it might be time to dig into some src... Thanks for the feedback. Cheers, JB On Thu, Nov 6, 2014 at 9:25 PM, M K wrote: > Unless it's changed within the past month or so, the ElasticSearch writer > that comes with Bro is very alpha-level code. For the most part it fires > and forgets and can be prone to losing messages if your cluster isn't able > to keep up or some other situation causes it not to be able to ingest the > data properly. > > Your best bet, as of now, is to write out the logs to disk and use some > intermediary program to process the logs and ingest them into ES. Logstash > can help, but with the default custom format Bro uses, it can't parse the > data properly. If you're using Bro 2.3, you can modify the output format of > the ascii writer to use json instead and then use logstash to feed the data > relatively easily into ES. Further, I'd recommend using a rabbit river so > ES can ingest the data at its leisure. > > If you're stuck with the non-json format, well your options are kinda > limited. You can write a crazy custom logstash conf using grok (which is > super inefficient) or figure out some other mechanism. > > As an aside, I've written a custom logstash filter that processes the > custom bro format and is, to a limited extent, bro type aware so it can > take old-style bro logs relatively easily and make it more usable (numbers > are turned into numbers and sets, vectors and tables are turned into arrays > -- same as how I've seen the ES writer output data). There are some caveats > in its usage though. I'm putting the finishing touches on it and plan to > release it when I get a chance (hopefully within the next week or two). > > On Thu, Nov 6, 2014 at 7:54 PM, Joe Blow wrote: > >> Hey all, >> >> Just going to throw this out there and hope some people are willing to >> potentially share some learning experiences if they have any. >> >> We have a system which generates around 15k-30k BRO events/sec and are >> trying to ingest these logs into a fairly beefy elasticsearch cluster. >> Total cluster memory ~300GB, storage ~300TB. >> >> Long story short, we're having some problems keeping up with this feed. >> Does anyone have any performance tuning with this module? I've played a >> lot with rsyslog batch sizes with elasticsearch and was hoping there would >> be some simple directive i could try and apply to BRO. >> >> Does anyone have this experience here? Does this module batch anything? >> >> Thanks in advance. >> >> Cheers, >> >> JB >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141106/066e55df/attachment.html From seth at icir.org Thu Nov 6 18:49:54 2014 From: seth at icir.org (Seth Hall) Date: Thu, 6 Nov 2014 21:49:54 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: References: Message-ID: <1BD85275-2C90-4B56-9EAD-B76DA01070FC@icir.org> > On Nov 6, 2014, at 7:54 PM, Joe Blow wrote: > > Long story short, we're having some problems keeping up with this feed. Does anyone have any performance tuning with this module? I've played a lot with rsyslog batch sizes with elasticsearch and was hoping there would be some simple directive i could try and apply to BRO. > > Does anyone have this experience here? Does this module batch anything? There is a solution that has been in development for some time. We've done some work with having Bro write directly to NSQ (a disk backed http based queuing daemon) and there is another tool that pulls from NSQ and inserts into Elasticsearch. So far it seems that this can keep up with quite high volume networks. Thanks for reporting to the list. More people showing problems like this can certainly prompt development on features like this. ;) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From mkhan04 at gmail.com Thu Nov 6 18:50:41 2014 From: mkhan04 at gmail.com (M K) Date: Thu, 6 Nov 2014 21:50:41 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: References: Message-ID: If you're looking at the source for 2.3 it'll be in src/logging/writers/ElasticSearch.cc In master, the ES writer has been moved to the new plugin architecture although the code remains largely the same (as far as I can tell), so you can take a look at the bro-plugins repo on github. As a little preview, the writer uses the ES bulk interface to send data using libcurl and if there's any error, basically ignores it and continues on. Also, I wrote two custom writers a few months back, an AMQP writer and an ElasticSearch River writer, both of which wrote to an AMQP server (the latter of which made river compliant messages for direct ingestion into ES). They worked well under testing, but I didn't go any further with them since my pull request to the bro repo wasn't accepted. On Thu, Nov 6, 2014 at 9:37 PM, Joe Blow wrote: > Yuck. I was really hoping this wasn't the way. From everything you said, > the river is where i'm focusing. I really, really dislike logstash (i'd > rather bend rsyslog + ES output plugin to my liking any day). > I've written a few custom ES output/input parsers and many SOLR parsers > that will parse bro logs, proxy logs, etc..., but would rather focus on > something more native to output to ES if possible. > > I guess it might be time to dig into some src... > > Thanks for the feedback. > > Cheers, > > JB > > > > On Thu, Nov 6, 2014 at 9:25 PM, M K wrote: > >> Unless it's changed within the past month or so, the ElasticSearch writer >> that comes with Bro is very alpha-level code. For the most part it fires >> and forgets and can be prone to losing messages if your cluster isn't able >> to keep up or some other situation causes it not to be able to ingest the >> data properly. >> >> Your best bet, as of now, is to write out the logs to disk and use some >> intermediary program to process the logs and ingest them into ES. Logstash >> can help, but with the default custom format Bro uses, it can't parse the >> data properly. If you're using Bro 2.3, you can modify the output format of >> the ascii writer to use json instead and then use logstash to feed the data >> relatively easily into ES. Further, I'd recommend using a rabbit river so >> ES can ingest the data at its leisure. >> >> If you're stuck with the non-json format, well your options are kinda >> limited. You can write a crazy custom logstash conf using grok (which is >> super inefficient) or figure out some other mechanism. >> >> As an aside, I've written a custom logstash filter that processes the >> custom bro format and is, to a limited extent, bro type aware so it can >> take old-style bro logs relatively easily and make it more usable (numbers >> are turned into numbers and sets, vectors and tables are turned into arrays >> -- same as how I've seen the ES writer output data). There are some caveats >> in its usage though. I'm putting the finishing touches on it and plan to >> release it when I get a chance (hopefully within the next week or two). >> >> On Thu, Nov 6, 2014 at 7:54 PM, Joe Blow wrote: >> >>> Hey all, >>> >>> Just going to throw this out there and hope some people are willing to >>> potentially share some learning experiences if they have any. >>> >>> We have a system which generates around 15k-30k BRO events/sec and are >>> trying to ingest these logs into a fairly beefy elasticsearch cluster. >>> Total cluster memory ~300GB, storage ~300TB. >>> >>> Long story short, we're having some problems keeping up with this feed. >>> Does anyone have any performance tuning with this module? I've played a >>> lot with rsyslog batch sizes with elasticsearch and was hoping there would >>> be some simple directive i could try and apply to BRO. >>> >>> Does anyone have this experience here? Does this module batch anything? >>> >>> Thanks in advance. >>> >>> Cheers, >>> >>> JB >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141106/95b20411/attachment.html From nick.turley at ucr.edu Thu Nov 6 18:59:36 2014 From: nick.turley at ucr.edu (Nick Turley) Date: Fri, 7 Nov 2014 02:59:36 +0000 Subject: [Bro] elastic search / bro questions In-Reply-To: References: Message-ID: <329FC045-5BC6-4942-B1D2-64BF54F6CF68@ucr.edu> We tried the ElasticSearch writer with mixed results. Our Bro cluster (two workers, one manager/proxy) processes somewhere the same number of events you?re dealing with and we write all Bro data to a RAID10 array. We then have a Logstash shipper grab the logs as fast as they come in and ship those off to a couple REDIS systems. We then have a Logstash indexer pull the data from REDIS and mutate the data in various ways (rename attributes, geolocation) before being shipped off to our ElasticSearch cluster. We have limited hardware but we?ve been able to pump data fast enough for 10Gbps link which peaks around 3.5Gbps. The architecture is more complicated, but its scalable. Having REDIS in place makes it nice when you have traffic bursts and Logstash needs time to catch up. It acts as a nice buffer. Logstash can also be tuned to have multiple worker processes and filter threads which took us some time to tune. It?s a bit of a balancing act. .. And I just saw your response. Sounds like Logstash is not a good option for you. Nick From: M K > Date: Thursday, November 6, 2014 at 6:25 PM To: Joe Blow > Cc: "bro at bro-ids.org List" > Subject: Re: [Bro] elastic search / bro questions Unless it's changed within the past month or so, the ElasticSearch writer that comes with Bro is very alpha-level code. For the most part it fires and forgets and can be prone to losing messages if your cluster isn't able to keep up or some other situation causes it not to be able to ingest the data properly. Your best bet, as of now, is to write out the logs to disk and use some intermediary program to process the logs and ingest them into ES. Logstash can help, but with the default custom format Bro uses, it can't parse the data properly. If you're using Bro 2.3, you can modify the output format of the ascii writer to use json instead and then use logstash to feed the data relatively easily into ES. Further, I'd recommend using a rabbit river so ES can ingest the data at its leisure. If you're stuck with the non-json format, well your options are kinda limited. You can write a crazy custom logstash conf using grok (which is super inefficient) or figure out some other mechanism. As an aside, I've written a custom logstash filter that processes the custom bro format and is, to a limited extent, bro type aware so it can take old-style bro logs relatively easily and make it more usable (numbers are turned into numbers and sets, vectors and tables are turned into arrays -- same as how I've seen the ES writer output data). There are some caveats in its usage though. I'm putting the finishing touches on it and plan to release it when I get a chance (hopefully within the next week or two). On Thu, Nov 6, 2014 at 7:54 PM, Joe Blow > wrote: Hey all, Just going to throw this out there and hope some people are willing to potentially share some learning experiences if they have any. We have a system which generates around 15k-30k BRO events/sec and are trying to ingest these logs into a fairly beefy elasticsearch cluster. Total cluster memory ~300GB, storage ~300TB. Long story short, we're having some problems keeping up with this feed. Does anyone have any performance tuning with this module? I've played a lot with rsyslog batch sizes with elasticsearch and was hoping there would be some simple directive i could try and apply to BRO. Does anyone have this experience here? Does this module batch anything? Thanks in advance. Cheers, JB _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141107/46517a01/attachment.html From seth at icir.org Thu Nov 6 19:17:38 2014 From: seth at icir.org (Seth Hall) Date: Thu, 6 Nov 2014 22:17:38 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: References: Message-ID: <39108B1D-E2F3-48F2-965E-38879AC2CCA8@icir.org> > On Nov 6, 2014, at 9:50 PM, M K wrote: > > Also, I wrote two custom writers a few months back, an AMQP writer and an ElasticSearch River writer, both of which wrote to an AMQP server (the latter of which made river compliant messages for direct ingestion into ES). They worked well under testing, but I didn't go any further with them since my pull request to the bro repo wasn't accepted. Could you remind me of the ticket number? I don't recall that we rejected your patches, it's possible that we've just not had a motivator to drive the patches forward. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From anthony.kasza at gmail.com Thu Nov 6 20:19:20 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Thu, 6 Nov 2014 20:19:20 -0800 Subject: [Bro] Redirect Chain Script Message-ID: If anyone is interested I have the beginnings of a redirect/driveby analysis policy script here: . I've only tested it on pcaps but it seems to work nicely. I image the output is a little difficult to interpret if you don't understand what the script is doing but I think it may be a good foundation for something. Thoughts and feedback are welcome. -AK From jdopheid at illinois.edu Fri Nov 7 08:42:59 2014 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Fri, 7 Nov 2014 16:42:59 +0000 Subject: [Bro] Bro Blog: Using Bro to Build a Passive DNS Database Message-ID: http://blog.bro.org/2014/11/using-bro-to-build-passive-dns-database.html Searching DNS logs became a lot faster with the launch of our Passive DNS tool for Bro. It uses Bro's DNS logs to build a database that is more compact, and therefore a lot easier to search. See how we did it by checking it out on GitHub: https://github.com/JustinAzoff/bro-pdns ------ Jeannette M. Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141107/9f73cd06/attachment.html From jlay at slave-tothe-box.net Fri Nov 7 09:54:50 2014 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 07 Nov 2014 10:54:50 -0700 Subject: [Bro] Redirect Chain Script In-Reply-To: References: Message-ID: <37078312f763829a9b75ad3c0329710f@localhost> On 2014-11-06 21:19, anthony kasza wrote: > If anyone is interested I have the beginnings of a redirect/driveby > analysis policy script here: > > . > > I've only tested it on pcaps but it seems to work nicely. I image the > output is a little difficult to interpret if you don't understand > what > the script is doing but I think it may be a good foundation for > something. Thoughts and feedback are welcome. > > -AK > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Yea this is kind of cool....in a nutshell, this adds: dns_domain dns_uid http_uri http_domain http_uid to your conn.log...kind of handy for tracking...thanks for this Anthony...I'll try this out full on in dev and if good go into production. I'll let you know if I run into any snags or surprises. James From anthony.kasza at gmail.com Fri Nov 7 10:29:34 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Fri, 7 Nov 2014 10:29:34 -0800 Subject: [Bro] Redirect Chain Script In-Reply-To: <37078312f763829a9b75ad3c0329710f@localhost> References: <37078312f763829a9b75ad3c0329710f@localhost> Message-ID: I'm glad you like it. You shouldn't plug this script directly into a production sensor without modifying it a bit. I wrote it with a very specific use case in mind: highlighting important connections in pcaps recorded in VMs that visit drive by sites. -AK On Nov 7, 2014 10:02 AM, "James Lay" wrote: > On 2014-11-06 21:19, anthony kasza wrote: > > If anyone is interested I have the beginnings of a redirect/driveby > > analysis policy script here: > > > > . > > > > I've only tested it on pcaps but it seems to work nicely. I image the > > output is a little difficult to interpret if you don't understand > > what > > the script is doing but I think it may be a good foundation for > > something. Thoughts and feedback are welcome. > > > > -AK > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > Yea this is kind of cool....in a nutshell, this adds: > > dns_domain dns_uid http_uri http_domain http_uid > > to your conn.log...kind of handy for tracking...thanks for this > Anthony...I'll try this out full on in dev and if good go into > production. I'll let you know if I run into any snags or surprises. > > James > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141107/884ee2db/attachment.html From hhoffman at ip-solutions.net Fri Nov 7 13:00:54 2014 From: hhoffman at ip-solutions.net (Harry Hoffman) Date: Fri, 07 Nov 2014 16:00:54 -0500 Subject: [Bro] GIT error compiling Message-ID: <545D3306.4040206@ip-solutions.net> Hi All, Just did a git pull and run into this error /root/downloads/bro/src/main.cc: In function ?int main(int, char**)?: /root/downloads/bro/src/main.cc:779: error: ?init? is not a member of ?binpac? make[3]: *** [src/CMakeFiles/bro.dir/main.cc.o] Error 1 make[3]: Leaving directory `/root/downloads/bro/build' make[2]: *** [src/CMakeFiles/bro.dir/all] Error 2 make[2]: Leaving directory `/root/downloads/bro/build' make[1]: *** [all] Error 2 make[1]: Leaving directory `/root/downloads/bro/build' make: *** [all] Error 2 Thought I'd give a heads up. Cheers, Harry From jsiwek at illinois.edu Fri Nov 7 13:21:44 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 7 Nov 2014 21:21:44 +0000 Subject: [Bro] GIT error compiling In-Reply-To: <545D3306.4040206@ip-solutions.net> References: <545D3306.4040206@ip-solutions.net> Message-ID: > On Nov 7, 2014, at 3:00 PM, Harry Hoffman wrote: > > Just did a git pull and run into this error > > /root/downloads/bro/src/main.cc: In function ?int main(int, char**)?: > /root/downloads/bro/src/main.cc:779: error: ?init? is not a member of > ?binpac? Can you try running the command `submodule update --recursive --init` inside your bro clone before compiling again? When updating a git clone of bro, you usually need to do that after `git pull` in order to keep everything in sync. - Jon From jlay at slave-tothe-box.net Fri Nov 7 17:34:48 2014 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 07 Nov 2014 18:34:48 -0700 Subject: [Bro] Redirect Chain Script In-Reply-To: References: <37078312f763829a9b75ad3c0329710f@localhost> Message-ID: <1415410488.2980.0.camel@JamesiMac> On Fri, 2014-11-07 at 10:29 -0800, anthony kasza wrote: > I'm glad you like it. > You shouldn't plug this script directly into a production sensor > without modifying it a bit. I wrote it with a very specific use case > in mind: highlighting important connections in pcaps recorded in VMs > that visit drive by sites. > > -AK > > > On Nov 7, 2014 10:02 AM, "James Lay" wrote: > > On 2014-11-06 21:19, anthony kasza wrote: > > If anyone is interested I have the beginnings of a > redirect/driveby > > analysis policy script here: > > > > > . > > > > I've only tested it on pcaps but it seems to work nicely. I > image the > > output is a little difficult to interpret if you don't > understand > > what > > the script is doing but I think it may be a good foundation > for > > something. Thoughts and feedback are welcome. > > > > -AK > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > Yea this is kind of cool....in a nutshell, this adds: > > dns_domain dns_uid http_uri http_domain http_uid > > to your conn.log...kind of handy for tracking...thanks for > this > Anthony...I'll try this out full on in dev and if good go into > production. I'll let you know if I run into any snags or > surprises. > Thanks Anthony...I'll be careful :) James -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141107/9ebe5f95/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 925 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141107/9ebe5f95/attachment.bin From michal at rsbac.org Sat Nov 8 03:39:38 2014 From: michal at rsbac.org (Michal Purzynski) Date: Sat, 08 Nov 2014 12:39:38 +0100 Subject: [Bro] elastic search / bro questions In-Reply-To: References: Message-ID: <545E00FA.2040801@rsbac.org> How about using Heka to read and parse the logs, and MozDef to collect them? That's what we do here with I believ 7k eps, soon to be more. Or just Heka. I'd go for both, we're working on a plug and play configuration. One of the good things about Heka is - it's insane fast. Tests were showing 10Gbit/sec pipe saturated with logs. Heka http://blog.mozilla.org/services/2013/04/30/introducing-heka/ https://github.com/mozilla-services/heka https://hekad.readthedocs.org/en/v0.8.0/ MozDef https://github.com/jeffbryner/MozDef http://mozdef.readthedocs.org/en/latest/ > On Thu, Nov 6, 2014 at 7:54 PM, Joe Blow > wrote: > > Hey all, > > Just going to throw this out there and hope some people are > willing to potentially share some learning experiences if they > have any. > > We have a system which generates around 15k-30k BRO events/sec and > are trying to ingest these logs into a fairly beefy elasticsearch > cluster. Total cluster memory ~300GB, storage ~300TB. > > Long story short, we're having some problems keeping up with this > feed. Does anyone have any performance tuning with this module? > I've played a lot with rsyslog batch sizes with elasticsearch and > was hoping there would be some simple directive i could try and > apply to BRO. > > Does anyone have this experience here? Does this module batch > anything? > > Thanks in advance. > > Cheers, > > JB > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141108/ba3aaa2d/attachment.html From hhoffman at ip-solutions.net Sat Nov 8 17:55:02 2014 From: hhoffman at ip-solutions.net (Harry Hoffman) Date: Sat, 08 Nov 2014 20:55:02 -0500 Subject: [Bro] Intel hits not being emailed Message-ID: <545EC976.5090408@ip-solutions.net> Hi All, Using CIF with my bro install (checked out from git). I've got the following in my local.bro: @load frameworks/intel/seen @load frameworks/intel/do_notice @load policy/integration/collective-intel redef Intel::read_files += { "/usr/local/bro/share/bro/site/infrastructure_scan.intel", "/usr/local/bro/share/bro/site/domain_botnet.intel", }; redef Notice::emailed_types += { Intel::Notice, }; redef Notice::type_suppression_intervals += { [Intel::Notice] = 1day, }; I see hits in my intel.log files but I don't get emails about this. Am I missing something? I'd taken this directly from the bro blog. Cheers, Harry From just2 at arcor.de Mon Nov 10 04:52:22 2014 From: just2 at arcor.de (just2 at arcor.de) Date: Mon, 10 Nov 2014 13:52:22 +0100 (CET) Subject: [Bro] Infrastructure with Bro and SDN-capable switch Message-ID: <1555491873.265889.1415623942311.JavaMail.ngmail@webmail06.arcor-online.net> Hello, I am a newbie on Bro IDS. I am working on a project in my university. Goal is to reduce packet drops in IDS by installing a SDN-capable switch which filters and only redirects suspicious content to the IDS, thus reducing workload on the IDS and therefore packet drops. I started researching on Bro and as far as I understand, it is capable of running in high-speed networks without packet drops. Therefore my question: Is it reasonable to do the research project (Bro + SDN switch) or is it very unlikely to have packet drops no matter how much traffic/speed the network is running? Thank you for your answers, Mirjam From seth at icir.org Mon Nov 10 06:40:21 2014 From: seth at icir.org (Seth Hall) Date: Mon, 10 Nov 2014 09:40:21 -0500 Subject: [Bro] Intel hits not being emailed In-Reply-To: <545EC976.5090408@ip-solutions.net> References: <545EC976.5090408@ip-solutions.net> Message-ID: > On Nov 8, 2014, at 8:55 PM, Harry Hoffman wrote: > > I see hits in my intel.log files but I don't get emails about this. Am I > missing something? I'd taken this directly from the bro blog. By default, notices are not generated for intel hits. There is a script that we ship with Bro that gives you the ability to turn intel hits into notices based on a field in the intel data (more information can be found here: https://www.bro.org/bro-exchange-2013/exercises/intel.html). If you have a solid idea of how you'd like things to work best for you, please let me know. There are many ways we could make this work. ;) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From blackhole.em at gmail.com Mon Nov 10 06:46:37 2014 From: blackhole.em at gmail.com (Joe Blow) Date: Mon, 10 Nov 2014 09:46:37 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: <545E00FA.2040801@rsbac.org> References: <545E00FA.2040801@rsbac.org> Message-ID: Hey all, We're still fighting with certain troubles but have completed isolating all data nodes from our master nodes. That seemed to help with general availability of the cluster, and data throughput (we think some of the data nodes couldn't talk to the masters, creating a bunch of stability issues). Now all that is kind of an ES thing, though i thought it might be valuable i've added it. I now have a question for the BRO folks regarding indexing to the 'bro' index (as opposed to the 'bro-201410242100'). We have our 'bro' index up over 10B records. When the index needs to be brought back up after (routine) catastrophic failures, we find ourselves waiting for a really, really long time while the massive 'bro' primary shards initialize. My question is this. Many of these ES issues appear that they can be alleviated if we were shoving all of the bro logs into 'bro-YYYYmmddHHMM', instead of some there, and some in the giant 'bro' index. Is there any reason why we can't force all of the ES logging into the time based indicies instead of the one giant bro index? Would anyone know where to start hacking the BRO code to try and make this possible? By the way, thanks tons for the help everyone, i'll definitely be posting a full lessons learned once we get everything up the way we're expecting. Cheers, JB On Sat, Nov 8, 2014 at 6:39 AM, Michal Purzynski wrote: > How about using Heka to read and parse the logs, and MozDef to collect > them? That's what we do here with I believ 7k eps, soon to be more. Or just > Heka. I'd go for both, we're working on a plug and play configuration. > > One of the good things about Heka is - it's insane fast. Tests were > showing 10Gbit/sec pipe saturated with logs. > > Heka > > http://blog.mozilla.org/services/2013/04/30/introducing-heka/ > https://github.com/mozilla-services/heka > https://hekad.readthedocs.org/en/v0.8.0/ > > MozDef > > https://github.com/jeffbryner/MozDef > http://mozdef.readthedocs.org/en/latest/ > > On Thu, Nov 6, 2014 at 7:54 PM, Joe Blow wrote: > >> Hey all, >> >> Just going to throw this out there and hope some people are willing to >> potentially share some learning experiences if they have any. >> >> We have a system which generates around 15k-30k BRO events/sec and are >> trying to ingest these logs into a fairly beefy elasticsearch cluster. >> Total cluster memory ~300GB, storage ~300TB. >> >> Long story short, we're having some problems keeping up with this >> feed. Does anyone have any performance tuning with this module? I've >> played a lot with rsyslog batch sizes with elasticsearch and was hoping >> there would be some simple directive i could try and apply to BRO. >> >> Does anyone have this experience here? Does this module batch anything? >> >> Thanks in advance. >> >> Cheers, >> >> JB >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > > > _______________________________________________ > Bro mailing listbro at bro-ids.orghttp://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141110/d7b194ef/attachment.html From seth at icir.org Mon Nov 10 06:48:23 2014 From: seth at icir.org (Seth Hall) Date: Mon, 10 Nov 2014 09:48:23 -0500 Subject: [Bro] Infrastructure with Bro and SDN-capable switch In-Reply-To: <1555491873.265889.1415623942311.JavaMail.ngmail@webmail06.arcor-online.net> References: <1555491873.265889.1415623942311.JavaMail.ngmail@webmail06.arcor-online.net> Message-ID: <3F20EB6D-5FF7-4021-9700-09AF05F95CF4@icir.org> > On Nov 10, 2014, at 7:52 AM, just2 at arcor.de wrote: > > Goal is to reduce packet drops in IDS by installing a SDN-capable switch which filters and only redirects suspicious content to the IDS, thus reducing workload on the IDS and therefore packet drops. This is something that people in our community are already starting to do except that it's typically done backwards from what you are describing. All traffic is directed to the IDS until the IDS decides that it doesn't want to see it anymore and then it is "shunted" on the switch (or at other locations). In my opinion, doing the opposite isn't possible because what is deciding what's suspicious? That sounds like the job of an IDS. ;) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Mon Nov 10 07:05:16 2014 From: seth at icir.org (Seth Hall) Date: Mon, 10 Nov 2014 10:05:16 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: References: <545E00FA.2040801@rsbac.org> Message-ID: <42FC8AA6-84F5-4E32-81C1-4B882C4D7CC4@icir.org> > On Nov 10, 2014, at 9:46 AM, Joe Blow wrote: > > My question is this. Many of these ES issues appear that they can be alleviated if we were shoving all of the bro logs into 'bro-YYYYmmddHHMM', instead of some there, and some in the giant 'bro' index. Is there any reason why we can't force all of the ES logging into the time based indicies instead of the one giant bro index? Would anyone know where to start hacking the BRO code to try and make this possible? Are you processing tracefiles? If you are processing live traffic from an interface it should already be sharding into indexes like you want. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From blackhole.em at gmail.com Mon Nov 10 07:20:41 2014 From: blackhole.em at gmail.com (Joe Blow) Date: Mon, 10 Nov 2014 10:20:41 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: <42FC8AA6-84F5-4E32-81C1-4B882C4D7CC4@icir.org> References: <545E00FA.2040801@rsbac.org> <42FC8AA6-84F5-4E32-81C1-4B882C4D7CC4@icir.org> Message-ID: I'm not processing offline files, if that's what you mean (still a bit new to bro, feel free to expand on the tracefiles). I'm sniffing many interfaces, but it appears most (not all, but most) logs are going into the bro index, without the time. I was going to try and hack something around this in 'share/bro/base/frameworks/logging/writers/elasticsearch.bro' to change the index to be dynamic with the date: ## Name of the ES index. const index_prefix = "bro" &redef; Not sure if that would only get read on program instantiation though... I might also be way out in left field... Any shove in the right direction helps :). Cheers, JB On Mon, Nov 10, 2014 at 10:05 AM, Seth Hall wrote: > > > On Nov 10, 2014, at 9:46 AM, Joe Blow wrote: > > > > My question is this. Many of these ES issues appear that they can be > alleviated if we were shoving all of the bro logs into 'bro-YYYYmmddHHMM', > instead of some there, and some in the giant 'bro' index. Is there any > reason why we can't force all of the ES logging into the time based > indicies instead of the one giant bro index? Would anyone know where to > start hacking the BRO code to try and make this possible? > > Are you processing tracefiles? If you are processing live traffic from an > interface it should already be sharding into indexes like you want. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141110/f1722a38/attachment.html From seth at icir.org Mon Nov 10 07:57:54 2014 From: seth at icir.org (Seth Hall) Date: Mon, 10 Nov 2014 10:57:54 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: References: <545E00FA.2040801@rsbac.org> <42FC8AA6-84F5-4E32-81C1-4B882C4D7CC4@icir.org> Message-ID: <932D3876-33C1-4273-826B-4F62251AF0ED@icir.org> > On Nov 10, 2014, at 10:20 AM, Joe Blow wrote: > > I'm not processing offline files, if that's what you mean (still a bit new to bro, feel free to expand on the tracefiles). Ohh, I know what's happening. You're running Bro directly at the command line without using broctl aren't you? Bro doesn't have log rotation enabled by default and the index name rotation is based on log log rotation. Set this in a script you're loading... redef Log::default_rotation_interval = 1hr;  I haven't double checked and I not sure what that will do to the Ascii logs, but it should at least give you partitioned index names in ES. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From blackhole.em at gmail.com Mon Nov 10 08:11:59 2014 From: blackhole.em at gmail.com (Joe Blow) Date: Mon, 10 Nov 2014 11:11:59 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: <932D3876-33C1-4273-826B-4F62251AF0ED@icir.org> References: <545E00FA.2040801@rsbac.org> <42FC8AA6-84F5-4E32-81C1-4B882C4D7CC4@icir.org> <932D3876-33C1-4273-826B-4F62251AF0ED@icir.org> Message-ID: Nope, i invoke bro using broctl like this: su snort -c "export https_proxy='https://$PROXY:$PROXYPORT'; /opt/data/bro/bin/broctl restart --clean" Which usually shows things like this: cleaning up ... cleaning up nodes ... checking configurations... manager scripts are ok. proxy-0 scripts are ok. worker-0-1 scripts are ok. worker-0-2 scripts are ok. worker-0-3 scripts are ok. worker-0-4 scripts are ok. worker-1-1 scripts are ok. worker-1-2 scripts are ok. worker-1-3 scripts are ok. worker-2-1 scripts are ok. worker-2-2 scripts are ok. worker-2-3 scripts are ok. worker-3-1 scripts are ok. worker-3-10 scripts are ok. worker-3-11 scripts are ok. worker-3-12 scripts are ok. worker-3-2 scripts are ok. worker-3-3 scripts are ok. worker-3-4 scripts are ok. worker-3-5 scripts are ok. worker-3-6 scripts are ok. worker-3-7 scripts are ok. worker-3-8 scripts are ok. worker-3-9 scripts are ok. worker-4-1 scripts are ok. worker-4-2 scripts are ok. worker-4-3 scripts are ok. worker-5-1 scripts are ok. worker-5-2 scripts are ok. worker-5-3 scripts are ok. worker-5-4 scripts are ok. installing ... removing old policies in /opt/data/bro/spool/installed-scripts-do-not-touch/site ... done. removing old policies in /opt/data/bro/spool/installed-scripts-do-not-touch/auto ... done. creating policy directories ... done. installing site policies ... done. generating cluster-layout.bro ... done. generating local-networks.bro ... done. generating broctl-config.bro ... done. updating nodes ... done. starting ... starting manager ... starting proxy-0 ... starting worker-0-1 ... starting worker-0-2 ... starting worker-0-3 ... starting worker-0-4 ... starting worker-1-1 ... starting worker-1-2 ... starting worker-1-3 ... starting worker-2-1 ... starting worker-2-2 ... starting worker-2-3 ... starting worker-3-1 ... starting worker-3-10 ... starting worker-3-11 ... starting worker-3-12 ... starting worker-3-2 ... starting worker-3-3 ... starting worker-3-4 ... starting worker-3-5 ... starting worker-3-6 ... starting worker-3-7 ... starting worker-3-8 ... starting worker-3-9 ... starting worker-4-1 ... starting worker-4-2 ... starting worker-4-3 ... starting worker-5-1 ... starting worker-5-2 ... starting worker-5-3 ... starting worker-5-4 ... Our node looks like this: [manager] type=manager host=$IP [proxy-0] type=proxy host=$IP [worker-0] type=worker host=$IP interface=eth2 lb_method=pf_ring lb_procs=4 pin_cpus=0,1,2,3 [worker-1] type=worker host=$IP interface=eth3 lb_method=pf_ring lb_procs=3 pin_cpus=5,6,7 [worker-2] type=worker host=$IP interface=eth4 lb_method=pf_ring lb_procs=3 pin_cpus=4,8,9 [worker-3] type=worker host=$IP interface=eth5 lb_method=pf_ring lb_procs=12 pin_cpus=10,11,12,13,14,15,23,24,25,26,27,28 [worker-4] type=worker host=$IP interface=eth6 lb_method=pf_ring lb_procs=3 pin_cpus=16,17,18 [worker-5] type=worker host=$IP interface=eth7 lb_method=pf_ring lb_procs=4 pin_cpus=19,20,21,22 Logs-to-elasticsearch.bro has this: const rotation_interval = 24hr &redef; We add custom country logging doing stuff like this (this is smtp/savecountry.bro): redef record SMTP::Info += { orig_cc: string &log &optional; resp_cc: string &log &optional; }; event smtp_reply(c: connection, is_orig: bool, code: count, cmd: string, msg: string, cont_resp: bool) &priority=3 { local orig_loc = lookup_location(c$id$orig_h); if ( orig_loc?$country_code ) c$smtp$orig_cc = orig_loc$country_code; local resp_loc = lookup_location(c$id$resp_h); if ( resp_loc?$country_code ) c$smtp$resp_cc = resp_loc$country_code; This shouldn't need to have the redef for log rotation should it? The only non stock stuff we do is adding countries to conn and smtp. Everything else should be stock. Any ideas? Cheers, JB On Mon, Nov 10, 2014 at 10:57 AM, Seth Hall wrote: > > > On Nov 10, 2014, at 10:20 AM, Joe Blow wrote: > > > > I'm not processing offline files, if that's what you mean (still a bit > new to bro, feel free to expand on the tracefiles). > > Ohh, I know what's happening. You're running Bro directly at the command > line without using broctl aren't you? Bro doesn't have log rotation > enabled by default and the index name rotation is based on log log rotation. > > Set this in a script you're loading... > > redef Log::default_rotation_interval = 1hr; > > I haven't double checked and I not sure what that will do to the Ascii > logs, but it should at least give you partitioned index names in ES. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141110/4a7eef4f/attachment.html From mkhan04 at gmail.com Mon Nov 10 08:48:24 2014 From: mkhan04 at gmail.com (M K) Date: Mon, 10 Nov 2014 11:48:24 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: References: <545E00FA.2040801@rsbac.org> <42FC8AA6-84F5-4E32-81C1-4B882C4D7CC4@icir.org> <932D3876-33C1-4273-826B-4F62251AF0ED@icir.org> Message-ID: Weird... As Seth mentioned, the writer uses the time and the rotation interval to name the indexes. It should also create an @ index for metadata. I thought the time format was hard coded in the es writer, but it's been a while since I read the code ... Also, in regards to ES restart, there are some tunable elements. For one, optimizing indexes should help. Also if you have the bandwidth, you can increase the number of concurrent recoveries and the allowed network throughput. On Nov 10, 2014 11:20 AM, "Joe Blow" wrote: > Nope, i invoke bro using broctl like this: > > su snort -c "export https_proxy='https://$PROXY:$PROXYPORT'; > /opt/data/bro/bin/broctl restart --clean" > > Which usually shows things like this: > > cleaning up ... > cleaning up nodes ... > checking configurations... > manager scripts are ok. > proxy-0 scripts are ok. > worker-0-1 scripts are ok. > worker-0-2 scripts are ok. > worker-0-3 scripts are ok. > worker-0-4 scripts are ok. > worker-1-1 scripts are ok. > worker-1-2 scripts are ok. > worker-1-3 scripts are ok. > worker-2-1 scripts are ok. > worker-2-2 scripts are ok. > worker-2-3 scripts are ok. > worker-3-1 scripts are ok. > worker-3-10 scripts are ok. > worker-3-11 scripts are ok. > worker-3-12 scripts are ok. > worker-3-2 scripts are ok. > worker-3-3 scripts are ok. > worker-3-4 scripts are ok. > worker-3-5 scripts are ok. > worker-3-6 scripts are ok. > worker-3-7 scripts are ok. > worker-3-8 scripts are ok. > worker-3-9 scripts are ok. > worker-4-1 scripts are ok. > worker-4-2 scripts are ok. > worker-4-3 scripts are ok. > worker-5-1 scripts are ok. > worker-5-2 scripts are ok. > worker-5-3 scripts are ok. > worker-5-4 scripts are ok. > installing ... > removing old policies in > /opt/data/bro/spool/installed-scripts-do-not-touch/site ... done. > removing old policies in > /opt/data/bro/spool/installed-scripts-do-not-touch/auto ... done. > creating policy directories ... done. > installing site policies ... done. > generating cluster-layout.bro ... done. > generating local-networks.bro ... done. > generating broctl-config.bro ... done. > updating nodes ... done. > starting ... > starting manager ... > starting proxy-0 ... > starting worker-0-1 ... > starting worker-0-2 ... > starting worker-0-3 ... > starting worker-0-4 ... > starting worker-1-1 ... > starting worker-1-2 ... > starting worker-1-3 ... > starting worker-2-1 ... > starting worker-2-2 ... > starting worker-2-3 ... > starting worker-3-1 ... > starting worker-3-10 ... > starting worker-3-11 ... > starting worker-3-12 ... > starting worker-3-2 ... > starting worker-3-3 ... > starting worker-3-4 ... > starting worker-3-5 ... > starting worker-3-6 ... > starting worker-3-7 ... > starting worker-3-8 ... > starting worker-3-9 ... > starting worker-4-1 ... > starting worker-4-2 ... > starting worker-4-3 ... > starting worker-5-1 ... > starting worker-5-2 ... > starting worker-5-3 ... > starting worker-5-4 ... > > Our node looks like this: > > [manager] > type=manager > host=$IP > [proxy-0] > type=proxy > host=$IP > [worker-0] > type=worker > host=$IP > interface=eth2 > lb_method=pf_ring > lb_procs=4 > pin_cpus=0,1,2,3 > [worker-1] > type=worker > host=$IP > interface=eth3 > lb_method=pf_ring > lb_procs=3 > pin_cpus=5,6,7 > [worker-2] > type=worker > host=$IP > interface=eth4 > lb_method=pf_ring > lb_procs=3 > pin_cpus=4,8,9 > [worker-3] > type=worker > host=$IP > interface=eth5 > lb_method=pf_ring > lb_procs=12 > pin_cpus=10,11,12,13,14,15,23,24,25,26,27,28 > [worker-4] > type=worker > host=$IP > interface=eth6 > lb_method=pf_ring > lb_procs=3 > pin_cpus=16,17,18 > [worker-5] > type=worker > host=$IP > interface=eth7 > lb_method=pf_ring > lb_procs=4 > pin_cpus=19,20,21,22 > > > Logs-to-elasticsearch.bro has this: > > const rotation_interval = 24hr &redef; > > We add custom country logging doing stuff like this (this is > smtp/savecountry.bro): > > redef record SMTP::Info += { > orig_cc: string &log &optional; > resp_cc: string &log &optional; > }; > > event smtp_reply(c: connection, is_orig: bool, code: count, cmd: string, > msg: string, cont_resp: bool) &priority=3 > { > local orig_loc = > lookup_location(c$id$orig_h); > if ( orig_loc?$country_code ) > c$smtp$orig_cc = orig_loc$country_code; > local resp_loc = lookup_location(c$id$resp_h); > if ( resp_loc?$country_code ) > c$smtp$resp_cc = resp_loc$country_code; > > > > This shouldn't need to have the redef for log rotation should it? The > only non stock stuff we do is adding countries to conn and smtp. > Everything else should be stock. > > Any ideas? > > Cheers, > > JB > > > > On Mon, Nov 10, 2014 at 10:57 AM, Seth Hall wrote: > >> >> > On Nov 10, 2014, at 10:20 AM, Joe Blow wrote: >> > >> > I'm not processing offline files, if that's what you mean (still a bit >> new to bro, feel free to expand on the tracefiles). >> >> Ohh, I know what's happening. You're running Bro directly at the command >> line without using broctl aren't you? Bro doesn't have log rotation >> enabled by default and the index name rotation is based on log log rotation. >> >> Set this in a script you're loading... >> >> redef Log::default_rotation_interval = 1hr; >> >> I haven't double checked and I not sure what that will do to the Ascii >> logs, but it should at least give you partitioned index names in ES. >> >> .Seth >> >> -- >> Seth Hall >> International Computer Science Institute >> (Bro) because everyone has a network >> http://www.bro.org/ >> >> > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141110/e3063487/attachment.html From hhoffman at ip-solutions.net Mon Nov 10 16:49:56 2014 From: hhoffman at ip-solutions.net (Harry Hoffman) Date: Mon, 10 Nov 2014 19:49:56 -0500 Subject: [Bro] Intel hits not being emailed In-Reply-To: References: <545EC976.5090408@ip-solutions.net> Message-ID: <54615D34.10608@ip-solutions.net> Thanks, Seth. So, I think ideally I'd like to be able to alert not solely upon connection but upon a conversation as determined either at the app level or a combination of other flow criteria that might indicate more then just a scan is happening. In a large, open, network getting hits from scans is less useful then it might be in other environments. I'll have a read over the tutorial and see where that gets me... Likely more questions to follow ;-) Cheers, Harry On 11/10/14 9:40 AM, Seth Hall wrote: > >> On Nov 8, 2014, at 8:55 PM, Harry Hoffman wrote: >> >> I see hits in my intel.log files but I don't get emails about this. Am I >> missing something? I'd taken this directly from the bro blog. > > By default, notices are not generated for intel hits. There is a script that we ship with Bro that gives you the ability to turn intel hits into notices based on a field in the intel data (more information can be found here: https://www.bro.org/bro-exchange-2013/exercises/intel.html). If you have a solid idea of how you'd like things to work best for you, please let me know. There are many ways we could make this work. ;) > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From blackhole.em at gmail.com Mon Nov 10 17:47:44 2014 From: blackhole.em at gmail.com (Joe Blow) Date: Mon, 10 Nov 2014 20:47:44 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: References: <545E00FA.2040801@rsbac.org> <42FC8AA6-84F5-4E32-81C1-4B882C4D7CC4@icir.org> <932D3876-33C1-4273-826B-4F62251AF0ED@icir.org> Message-ID: So for the record, this is what happens when you configure bro to have a log rotate interval of 0 within broctl, and still send logs to elasticsearch. Most of the logs will end up in the 'bro' index, but some will still end up being sent to bro-$DATETIME index. This was the result of some legacy configs (no logrotate for rsyslog so as not to lose file handles) which sent the data to a homebrew ES plugin. I had forgotten to remove these configs when setting up bro for the more native ES. Thanks tons for the quick response in the IRC channel. Cheers, JB On Mon, Nov 10, 2014 at 11:48 AM, M K wrote: > Weird... As Seth mentioned, the writer uses the time and the rotation > interval to name the indexes. It should also create an @ index for > metadata. I thought the time format was hard coded in the es writer, but > it's been a while since I read the code ... > > Also, in regards to ES restart, there are some tunable elements. For one, > optimizing indexes should help. Also if you have the bandwidth, you can > increase the number of concurrent recoveries and the allowed network > throughput. > On Nov 10, 2014 11:20 AM, "Joe Blow" wrote: > >> Nope, i invoke bro using broctl like this: >> >> su snort -c "export https_proxy='https://$PROXY:$PROXYPORT'; >> /opt/data/bro/bin/broctl restart --clean" >> >> Which usually shows things like this: >> >> cleaning up ... >> cleaning up nodes ... >> checking configurations... >> manager scripts are ok. >> proxy-0 scripts are ok. >> worker-0-1 scripts are ok. >> worker-0-2 scripts are ok. >> worker-0-3 scripts are ok. >> worker-0-4 scripts are ok. >> worker-1-1 scripts are ok. >> worker-1-2 scripts are ok. >> worker-1-3 scripts are ok. >> worker-2-1 scripts are ok. >> worker-2-2 scripts are ok. >> worker-2-3 scripts are ok. >> worker-3-1 scripts are ok. >> worker-3-10 scripts are ok. >> worker-3-11 scripts are ok. >> worker-3-12 scripts are ok. >> worker-3-2 scripts are ok. >> worker-3-3 scripts are ok. >> worker-3-4 scripts are ok. >> worker-3-5 scripts are ok. >> worker-3-6 scripts are ok. >> worker-3-7 scripts are ok. >> worker-3-8 scripts are ok. >> worker-3-9 scripts are ok. >> worker-4-1 scripts are ok. >> worker-4-2 scripts are ok. >> worker-4-3 scripts are ok. >> worker-5-1 scripts are ok. >> worker-5-2 scripts are ok. >> worker-5-3 scripts are ok. >> worker-5-4 scripts are ok. >> installing ... >> removing old policies in >> /opt/data/bro/spool/installed-scripts-do-not-touch/site ... done. >> removing old policies in >> /opt/data/bro/spool/installed-scripts-do-not-touch/auto ... done. >> creating policy directories ... done. >> installing site policies ... done. >> generating cluster-layout.bro ... done. >> generating local-networks.bro ... done. >> generating broctl-config.bro ... done. >> updating nodes ... done. >> starting ... >> starting manager ... >> starting proxy-0 ... >> starting worker-0-1 ... >> starting worker-0-2 ... >> starting worker-0-3 ... >> starting worker-0-4 ... >> starting worker-1-1 ... >> starting worker-1-2 ... >> starting worker-1-3 ... >> starting worker-2-1 ... >> starting worker-2-2 ... >> starting worker-2-3 ... >> starting worker-3-1 ... >> starting worker-3-10 ... >> starting worker-3-11 ... >> starting worker-3-12 ... >> starting worker-3-2 ... >> starting worker-3-3 ... >> starting worker-3-4 ... >> starting worker-3-5 ... >> starting worker-3-6 ... >> starting worker-3-7 ... >> starting worker-3-8 ... >> starting worker-3-9 ... >> starting worker-4-1 ... >> starting worker-4-2 ... >> starting worker-4-3 ... >> starting worker-5-1 ... >> starting worker-5-2 ... >> starting worker-5-3 ... >> starting worker-5-4 ... >> >> Our node looks like this: >> >> [manager] >> type=manager >> host=$IP >> [proxy-0] >> type=proxy >> host=$IP >> [worker-0] >> type=worker >> host=$IP >> interface=eth2 >> lb_method=pf_ring >> lb_procs=4 >> pin_cpus=0,1,2,3 >> [worker-1] >> type=worker >> host=$IP >> interface=eth3 >> lb_method=pf_ring >> lb_procs=3 >> pin_cpus=5,6,7 >> [worker-2] >> type=worker >> host=$IP >> interface=eth4 >> lb_method=pf_ring >> lb_procs=3 >> pin_cpus=4,8,9 >> [worker-3] >> type=worker >> host=$IP >> interface=eth5 >> lb_method=pf_ring >> lb_procs=12 >> pin_cpus=10,11,12,13,14,15,23,24,25,26,27,28 >> [worker-4] >> type=worker >> host=$IP >> interface=eth6 >> lb_method=pf_ring >> lb_procs=3 >> pin_cpus=16,17,18 >> [worker-5] >> type=worker >> host=$IP >> interface=eth7 >> lb_method=pf_ring >> lb_procs=4 >> pin_cpus=19,20,21,22 >> >> >> Logs-to-elasticsearch.bro has this: >> >> const rotation_interval = 24hr &redef; >> >> We add custom country logging doing stuff like this (this is >> smtp/savecountry.bro): >> >> redef record SMTP::Info += { >> orig_cc: string &log &optional; >> resp_cc: string &log &optional; >> }; >> >> event smtp_reply(c: connection, is_orig: bool, code: count, cmd: string, >> msg: string, cont_resp: bool) &priority=3 >> { >> local orig_loc = >> lookup_location(c$id$orig_h); >> if ( orig_loc?$country_code ) >> c$smtp$orig_cc = orig_loc$country_code; >> local resp_loc = lookup_location(c$id$resp_h); >> if ( resp_loc?$country_code ) >> c$smtp$resp_cc = resp_loc$country_code; >> >> >> >> This shouldn't need to have the redef for log rotation should it? The >> only non stock stuff we do is adding countries to conn and smtp. >> Everything else should be stock. >> >> Any ideas? >> >> Cheers, >> >> JB >> >> >> >> On Mon, Nov 10, 2014 at 10:57 AM, Seth Hall wrote: >> >>> >>> > On Nov 10, 2014, at 10:20 AM, Joe Blow wrote: >>> > >>> > I'm not processing offline files, if that's what you mean (still a bit >>> new to bro, feel free to expand on the tracefiles). >>> >>> Ohh, I know what's happening. You're running Bro directly at the >>> command line without using broctl aren't you? Bro doesn't have log >>> rotation enabled by default and the index name rotation is based on log log >>> rotation. >>> >>> Set this in a script you're loading... >>> >>> redef Log::default_rotation_interval = 1hr; >>> >>> I haven't double checked and I not sure what that will do to the Ascii >>> logs, but it should at least give you partitioned index names in ES. >>> >>> .Seth >>> >>> -- >>> Seth Hall >>> International Computer Science Institute >>> (Bro) because everyone has a network >>> http://www.bro.org/ >>> >>> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141110/4a106125/attachment.html From blackhole.em at gmail.com Mon Nov 10 18:19:27 2014 From: blackhole.em at gmail.com (Joe Blow) Date: Mon, 10 Nov 2014 21:19:27 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: References: <545E00FA.2040801@rsbac.org> <42FC8AA6-84F5-4E32-81C1-4B882C4D7CC4@icir.org> <932D3876-33C1-4273-826B-4F62251AF0ED@icir.org> Message-ID: One more thing i wanted to share... In 'bro/share/bro/base/frameworks/logging/writers/elasticsearch.bro' it says: ##! There is one known memory issue. If your elasticsearch server is ##! running slowly and taking too long to return from bulk insert ##! requests, the message queue to the writer thread will continue ##! growing larger and larger giving the appearance of a memory leak. Interesting to see this queuing graphed out on a box with 96gb of ram.... It ran into swap pretty quickly... :) [image: Inline image 1] All in good fun i suppose... Cheers, JB On Mon, Nov 10, 2014 at 8:47 PM, Joe Blow wrote: > So for the record, this is what happens when you configure bro to have a > log rotate interval of 0 within broctl, and still send logs to > elasticsearch. Most of the logs will end up in the 'bro' index, but some > will still end up being sent to bro-$DATETIME index. This was the result > of some legacy configs (no logrotate for rsyslog so as not to lose file > handles) which sent the data to a homebrew ES plugin. I had forgotten to > remove these configs when setting up bro for the more native ES. > > Thanks tons for the quick response in the IRC channel. > > Cheers, > > JB > > On Mon, Nov 10, 2014 at 11:48 AM, M K wrote: > >> Weird... As Seth mentioned, the writer uses the time and the rotation >> interval to name the indexes. It should also create an @ index for >> metadata. I thought the time format was hard coded in the es writer, but >> it's been a while since I read the code ... >> >> Also, in regards to ES restart, there are some tunable elements. For one, >> optimizing indexes should help. Also if you have the bandwidth, you can >> increase the number of concurrent recoveries and the allowed network >> throughput. >> On Nov 10, 2014 11:20 AM, "Joe Blow" wrote: >> >>> Nope, i invoke bro using broctl like this: >>> >>> su snort -c "export https_proxy='https://$PROXY:$PROXYPORT'; >>> /opt/data/bro/bin/broctl restart --clean" >>> >>> Which usually shows things like this: >>> >>> cleaning up ... >>> cleaning up nodes ... >>> checking configurations... >>> manager scripts are ok. >>> proxy-0 scripts are ok. >>> worker-0-1 scripts are ok. >>> worker-0-2 scripts are ok. >>> worker-0-3 scripts are ok. >>> worker-0-4 scripts are ok. >>> worker-1-1 scripts are ok. >>> worker-1-2 scripts are ok. >>> worker-1-3 scripts are ok. >>> worker-2-1 scripts are ok. >>> worker-2-2 scripts are ok. >>> worker-2-3 scripts are ok. >>> worker-3-1 scripts are ok. >>> worker-3-10 scripts are ok. >>> worker-3-11 scripts are ok. >>> worker-3-12 scripts are ok. >>> worker-3-2 scripts are ok. >>> worker-3-3 scripts are ok. >>> worker-3-4 scripts are ok. >>> worker-3-5 scripts are ok. >>> worker-3-6 scripts are ok. >>> worker-3-7 scripts are ok. >>> worker-3-8 scripts are ok. >>> worker-3-9 scripts are ok. >>> worker-4-1 scripts are ok. >>> worker-4-2 scripts are ok. >>> worker-4-3 scripts are ok. >>> worker-5-1 scripts are ok. >>> worker-5-2 scripts are ok. >>> worker-5-3 scripts are ok. >>> worker-5-4 scripts are ok. >>> installing ... >>> removing old policies in >>> /opt/data/bro/spool/installed-scripts-do-not-touch/site ... done. >>> removing old policies in >>> /opt/data/bro/spool/installed-scripts-do-not-touch/auto ... done. >>> creating policy directories ... done. >>> installing site policies ... done. >>> generating cluster-layout.bro ... done. >>> generating local-networks.bro ... done. >>> generating broctl-config.bro ... done. >>> updating nodes ... done. >>> starting ... >>> starting manager ... >>> starting proxy-0 ... >>> starting worker-0-1 ... >>> starting worker-0-2 ... >>> starting worker-0-3 ... >>> starting worker-0-4 ... >>> starting worker-1-1 ... >>> starting worker-1-2 ... >>> starting worker-1-3 ... >>> starting worker-2-1 ... >>> starting worker-2-2 ... >>> starting worker-2-3 ... >>> starting worker-3-1 ... >>> starting worker-3-10 ... >>> starting worker-3-11 ... >>> starting worker-3-12 ... >>> starting worker-3-2 ... >>> starting worker-3-3 ... >>> starting worker-3-4 ... >>> starting worker-3-5 ... >>> starting worker-3-6 ... >>> starting worker-3-7 ... >>> starting worker-3-8 ... >>> starting worker-3-9 ... >>> starting worker-4-1 ... >>> starting worker-4-2 ... >>> starting worker-4-3 ... >>> starting worker-5-1 ... >>> starting worker-5-2 ... >>> starting worker-5-3 ... >>> starting worker-5-4 ... >>> >>> Our node looks like this: >>> >>> [manager] >>> type=manager >>> host=$IP >>> [proxy-0] >>> type=proxy >>> host=$IP >>> [worker-0] >>> type=worker >>> host=$IP >>> interface=eth2 >>> lb_method=pf_ring >>> lb_procs=4 >>> pin_cpus=0,1,2,3 >>> [worker-1] >>> type=worker >>> host=$IP >>> interface=eth3 >>> lb_method=pf_ring >>> lb_procs=3 >>> pin_cpus=5,6,7 >>> [worker-2] >>> type=worker >>> host=$IP >>> interface=eth4 >>> lb_method=pf_ring >>> lb_procs=3 >>> pin_cpus=4,8,9 >>> [worker-3] >>> type=worker >>> host=$IP >>> interface=eth5 >>> lb_method=pf_ring >>> lb_procs=12 >>> pin_cpus=10,11,12,13,14,15,23,24,25,26,27,28 >>> [worker-4] >>> type=worker >>> host=$IP >>> interface=eth6 >>> lb_method=pf_ring >>> lb_procs=3 >>> pin_cpus=16,17,18 >>> [worker-5] >>> type=worker >>> host=$IP >>> interface=eth7 >>> lb_method=pf_ring >>> lb_procs=4 >>> pin_cpus=19,20,21,22 >>> >>> >>> Logs-to-elasticsearch.bro has this: >>> >>> const rotation_interval = 24hr &redef; >>> >>> We add custom country logging doing stuff like this (this is >>> smtp/savecountry.bro): >>> >>> redef record SMTP::Info += { >>> orig_cc: string &log &optional; >>> resp_cc: string &log &optional; >>> }; >>> >>> event smtp_reply(c: connection, is_orig: bool, code: count, cmd: string, >>> msg: string, cont_resp: bool) &priority=3 >>> { >>> local orig_loc = >>> lookup_location(c$id$orig_h); >>> if ( orig_loc?$country_code ) >>> c$smtp$orig_cc = orig_loc$country_code; >>> local resp_loc = lookup_location(c$id$resp_h); >>> if ( resp_loc?$country_code ) >>> c$smtp$resp_cc = resp_loc$country_code; >>> >>> >>> >>> This shouldn't need to have the redef for log rotation should it? The >>> only non stock stuff we do is adding countries to conn and smtp. >>> Everything else should be stock. >>> >>> Any ideas? >>> >>> Cheers, >>> >>> JB >>> >>> >>> >>> On Mon, Nov 10, 2014 at 10:57 AM, Seth Hall wrote: >>> >>>> >>>> > On Nov 10, 2014, at 10:20 AM, Joe Blow >>>> wrote: >>>> > >>>> > I'm not processing offline files, if that's what you mean (still a >>>> bit new to bro, feel free to expand on the tracefiles). >>>> >>>> Ohh, I know what's happening. You're running Bro directly at the >>>> command line without using broctl aren't you? Bro doesn't have log >>>> rotation enabled by default and the index name rotation is based on log log >>>> rotation. >>>> >>>> Set this in a script you're loading... >>>> >>>> redef Log::default_rotation_interval = 1hr; >>>> >>>> I haven't double checked and I not sure what that will do to the Ascii >>>> logs, but it should at least give you partitioned index names in ES. >>>> >>>> .Seth >>>> >>>> -- >>>> Seth Hall >>>> International Computer Science Institute >>>> (Bro) because everyone has a network >>>> http://www.bro.org/ >>>> >>>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141110/b08ea457/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 62031 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141110/b08ea457/attachment.bin From seth at icir.org Mon Nov 10 19:00:46 2014 From: seth at icir.org (Seth Hall) Date: Mon, 10 Nov 2014 22:00:46 -0500 Subject: [Bro] elastic search / bro questions In-Reply-To: References: <545E00FA.2040801@rsbac.org> <42FC8AA6-84F5-4E32-81C1-4B882C4D7CC4@icir.org> <932D3876-33C1-4273-826B-4F62251AF0ED@icir.org> Message-ID: <79A8E9D5-9066-4703-A599-07FB2ADF206D@icir.org> > On Nov 10, 2014, at 9:19 PM, Joe Blow wrote: > > One more thing i wanted to share... In 'bro/share/bro/base/frameworks/logging/writers/elasticsearch.bro' it says: > ##! There is one known memory issue. If your elasticsearch server is > ##! running slowly and taking too long to return from bulk insert > ##! requests, the message queue to the writer thread will continue > ##! growing larger and larger giving the appearance of a memory leak. > > Interesting to see this queuing graphed out on a box with 96gb of ram.... It ran into swap pretty quickly... :) Yeah, unfortunately ES frequently is having a hard time keeping up for people. This is where having logs go to an external queueing system first can be beneficial. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From pachinko.tw at gmail.com Tue Nov 11 00:26:11 2014 From: pachinko.tw at gmail.com (Po-Ching Lin) Date: Tue, 11 Nov 2014 16:26:11 +0800 Subject: [Bro] Worker configuration in Bro cluster Message-ID: <5461C823.2050502@gmail.com> Dear all, We intend to deploy a Bro cluster, in which the workers will get packets from another program via pipe, instead of capturing packets directly from a network interface. In case of a standalone deployment, we know it is trivial to achieve by the command such as "prog | bro -r -", where prog outputs packets to stdout. However, what should we configure the workers (or the manager) if the workers are to be launched from the manager through broctl? Thanks a lot. Po-Ching From seth at icir.org Tue Nov 11 08:02:20 2014 From: seth at icir.org (Seth Hall) Date: Tue, 11 Nov 2014 11:02:20 -0500 Subject: [Bro] Worker configuration in Bro cluster In-Reply-To: <5461C823.2050502@gmail.com> References: <5461C823.2050502@gmail.com> Message-ID: <9EAC42B3-CD8C-4942-A002-BE5AC41128A6@icir.org> > On Nov 11, 2014, at 3:26 AM, Po-Ching Lin wrote: > > We intend to deploy a Bro cluster, in which the workers will get packets > from another program via pipe, instead of capturing packets directly from a > network interface. In case of a standalone deployment, we know it is trivial > to achieve by the command such as "prog | bro -r -", where prog outputs > packets to stdout. However, what should we configure the workers (or the > manager) if the workers are to be launched from the manager through broctl? > Thanks a lot. Hm, that is a new challenge. BroControl really isn't meant for that kind of behavior. It's possible you could dig into it and make changes that would make it work. What I would possibly look into instead is testing our upcoming packet-bricks[1] tool. It uses netmap to pass packets around very efficiently and there is a new netmap feature called "pipes" (works on Linux and it's including in the upcoming FreeBSD 10.1) that it uses to pass packets to userland applications as sort of virtual interfaces. I would recommend going this route because you should get significantly better performance passing packets through packet-bricks than if you are actually passing packets through pipes.  Here is a link to packet-bricks: https://github.com/bro/packet-bricks Let me know if you need help figuring out anything about it and I can either help you or point you in the right direction. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdonnelly at dyn.com Thu Nov 13 05:51:20 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Thu, 13 Nov 2014 08:51:20 -0500 Subject: [Bro] Adding addtional files to debian package Message-ID: Hi, Any suggestions how to add additional files to the Debian package Bro creates with "make-deb-package " step ? bro [] # cd pkg && ( ./make-deb-packages ) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141113/4caf6a0b/attachment.html From jsiwek at illinois.edu Thu Nov 13 07:39:50 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 13 Nov 2014 15:39:50 +0000 Subject: [Bro] Adding addtional files to debian package In-Reply-To: References: Message-ID: > On Nov 13, 2014, at 7:51 AM, John Donnelly wrote: > > Hi, > Any suggestions how to add additional files to the Debian package > Bro creates with "make-deb-package " step ? > > bro [] # cd pkg && ( ./make-deb-packages ) Those are created via CPack, which uses CMake?s installation mechanism, so the first idea that comes to mind is to edit CMakeList.txt and insert install*() commands. - Jon From tiburcesotohou at yahoo.fr Thu Nov 13 09:05:57 2014 From: tiburcesotohou at yahoo.fr (SOTOHOU Osince Tiburce) Date: Thu, 13 Nov 2014 17:05:57 +0000 (UTC) Subject: [Bro] instrumentation with broccoli Message-ID: <1971743421.452195.1415898357093.JavaMail.yahoo@jws11125.mail.ir2.yahoo.com> Hi;I want to instrument OpenSSH application with broccoli. I have learned broccoli language.Now, i would like to know where can i start and how can i do that.The different steps and tools for that are that i need. .Thank you for your response. regards -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141113/0994cab2/attachment.html From slagell at illinois.edu Thu Nov 13 09:22:56 2014 From: slagell at illinois.edu (Slagell, Adam J) Date: Thu, 13 Nov 2014 17:22:56 +0000 Subject: [Bro] instrumentation with broccoli In-Reply-To: <1971743421.452195.1415898357093.JavaMail.yahoo@jws11125.mail.ir2.yahoo.com> References: <1971743421.452195.1415898357093.JavaMail.yahoo@jws11125.mail.ir2.yahoo.com> Message-ID: Jon Siwek created a python version of the SSLLogMux script that uses broccoli that is basically a drop-in replacement for what Scott Campbell has created for iSSHD. Is that what you are looking for? On Nov 13, 2014, at 11:05 AM, SOTOHOU Osince Tiburce > wrote: Hi; I want to instrument OpenSSH application with broccoli. I have learned broccoli language. Now, i would like to know where can i start and how can i do that. The different steps and tools for that are that i need. . Thank you for your response. regards _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro ------ Adam J. Slagell Chief Information Security Officer Assistant Director, Cybersecurity Directorate National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141113/2194bf81/attachment.html From edthoma at sandia.gov Thu Nov 13 10:05:37 2014 From: edthoma at sandia.gov (Thomas, Eric D) Date: Thu, 13 Nov 2014 18:05:37 +0000 Subject: [Bro] How to use PF_RING multi? Message-ID: What is the configuration needed to run bro with PF_RING using its link aggregation (multi) capability? This below (a snippet of node.cfg) doesn't do it: [bro-pf] type=worker host=X.X.X.X interface=multi:em1;em2 lb_method=pf_ring lb_procs=8 -- Eric Thomas edthoma at sandia.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141113/92aaed0f/attachment.html From edthoma at sandia.gov Thu Nov 13 10:18:30 2014 From: edthoma at sandia.gov (Thomas, Eric D) Date: Thu, 13 Nov 2014 18:18:30 +0000 Subject: [Bro] How to use PF_RING multi? In-Reply-To: Message-ID: I found the answer. I missed the lb_interfaces option. -- Eric Thomas edthoma at sandia.gov From: , "Thomas, Eric D" > Date: Thursday, November 13, 2014 10:05 AM To: "bro at bro.org" > Subject: How to use PF_RING multi? What is the configuration needed to run bro with PF_RING using its link aggregation (multi) capability? This below (a snippet of node.cfg) doesn't do it: [bro-pf] type=worker host=X.X.X.X interface=multi:em1;em2 lb_method=pf_ring lb_procs=8 -- Eric Thomas edthoma at sandia.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141113/c8175ccf/attachment.html From blackhole.em at gmail.com Thu Nov 13 10:37:57 2014 From: blackhole.em at gmail.com (Joe Blow) Date: Thu, 13 Nov 2014 13:37:57 -0500 Subject: [Bro] How to use PF_RING multi? In-Reply-To: References: Message-ID: would you mind posting the full working config? I'm interested in this as well. Cheers, JB On Thu, Nov 13, 2014 at 1:18 PM, Thomas, Eric D wrote: > I found the answer. I missed the lb_interfaces option. > -- > Eric Thomas > edthoma at sandia.gov > > From: , "Thomas, Eric D" > Date: Thursday, November 13, 2014 10:05 AM > To: "bro at bro.org" > Subject: How to use PF_RING multi? > > What is the configuration needed to run bro with PF_RING using its link > aggregation (multi) capability? This below (a snippet of node.cfg) doesn't > do it: > > [bro-pf] > type=worker > host=X.X.X.X > interface=multi:em1;em2 > lb_method=pf_ring > lb_procs=8 > > -- > Eric Thomas > edthoma at sandia.gov > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141113/328be0d7/attachment.html From edthoma at sandia.gov Thu Nov 13 10:22:06 2014 From: edthoma at sandia.gov (Thomas, Eric D) Date: Thu, 13 Nov 2014 18:22:06 +0000 Subject: [Bro] How to use PF_RING multi? In-Reply-To: Message-ID: Wait, does lb_interfaces work with lb_method=pf_ring? My bro instance didn't complain when I started it, but I do not believe it is actually processing packets. -- Eric Thomas edthoma at sandia.gov From: , "Thomas, Eric D" > Date: Thursday, November 13, 2014 10:18 AM To: "bro at bro.org" > Subject: Re: How to use PF_RING multi? I found the answer. I missed the lb_interfaces option. -- Eric Thomas edthoma at sandia.gov From: , "Thomas, Eric D" > Date: Thursday, November 13, 2014 10:05 AM To: "bro at bro.org" > Subject: How to use PF_RING multi? What is the configuration needed to run bro with PF_RING using its link aggregation (multi) capability? This below (a snippet of node.cfg) doesn't do it: [bro-pf] type=worker host=X.X.X.X interface=multi:em1;em2 lb_method=pf_ring lb_procs=8 -- Eric Thomas edthoma at sandia.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141113/4ac2dec1/attachment.html From slagell at illinois.edu Thu Nov 13 10:52:26 2014 From: slagell at illinois.edu (Slagell, Adam J) Date: Thu, 13 Nov 2014 18:52:26 +0000 Subject: [Bro] How to use PF_RING multi? In-Reply-To: References: Message-ID: <61FF1B37-4805-470A-84A9-2D394928577A@illinois.edu> https://github.com/jsiwek/sshd_audit_mux On Nov 13, 2014, at 12:37 PM, Joe Blow > wrote: would you mind posting the full working config? I'm interested in this as well. Cheers, JB On Thu, Nov 13, 2014 at 1:18 PM, Thomas, Eric D > wrote: I found the answer. I missed the lb_interfaces option. -- Eric Thomas edthoma at sandia.gov From: , "Thomas, Eric D" > Date: Thursday, November 13, 2014 10:05 AM To: "bro at bro.org" > Subject: How to use PF_RING multi? What is the configuration needed to run bro with PF_RING using its link aggregation (multi) capability? This below (a snippet of node.cfg) doesn't do it: [bro-pf] type=worker host=X.X.X.X interface=multi:em1;em2 lb_method=pf_ring lb_procs=8 -- Eric Thomas edthoma at sandia.gov _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro ------ Adam J. Slagell Chief Information Security Officer Assistant Director, Cybersecurity Directorate National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141113/33a2d990/attachment.html From blackhole.em at gmail.com Thu Nov 13 11:22:32 2014 From: blackhole.em at gmail.com (blackhole.em at gmail.com) Date: Thu, 13 Nov 2014 19:22:32 +0000 Subject: [Bro] How to use PF_RING multi? In-Reply-To: References: Message-ID: <1679737808-1415906553-cardhu_decombobulator_blackberry.rim.net-1102956521-@b28.c4.bise6.blackberry> I'm wondering the same thing... It would make pooling resources for multi nic'd bro works a ton easier... Has anyone done this? Can it be done? Cheers, JB Sent from my Verizon Wireless BlackBerry -----Original Message----- From: "Thomas, Eric D" Sender: bro-bounces at bro.org Date: Thu, 13 Nov 2014 18:22:06 To: bro at bro.org Subject: Re: [Bro] How to use PF_RING multi? _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at icir.org Thu Nov 13 12:17:28 2014 From: seth at icir.org (Seth Hall) Date: Thu, 13 Nov 2014 15:17:28 -0500 Subject: [Bro] How to use PF_RING multi? In-Reply-To: References: Message-ID: <6F73C365-07E0-4FC4-AD3F-9C3E16C3EA27@icir.org> > On Nov 13, 2014, at 1:05 PM, Thomas, Eric D wrote: > > What is the configuration needed to run bro with PF_RING using its link aggregation (multi) capability? This below (a snippet of node.cfg) doesn't do it: I'm afraid we don't have a terribly elegant method to do that with PF_Ring right now. You could use their ZC module and do the load balancing in userspace with their zbalance_ipc tool (or whatever it's called). I think that can merge traffic and distribute it out and we support sniffing from ZC load balanced interfaces. This is yet another area where our upcoming packet-bricks tool will make life easier. I just wish it was ready for people to generally use. :/ .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdonnelly at dyn.com Thu Nov 13 14:52:37 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Thu, 13 Nov 2014 17:52:37 -0500 Subject: [Bro] Adding addtional files to debian package In-Reply-To: References: Message-ID: Thank you. The bro/CMakelist.txt file ? I don't see anything obvious. I would like to add a /etc/init.d/bro script that adds bro to start as a service and gets included with the debian package . ( and gets removed when dpkd -P bro ) is used. Everything appears to be installed under /opt/bro . On Thu, Nov 13, 2014 at 10:39 AM, Siwek, Jon wrote: > > > On Nov 13, 2014, at 7:51 AM, John Donnelly wrote: > > > > Hi, > > Any suggestions how to add additional files to the Debian package > > Bro creates with "make-deb-package " step ? > > > > bro [] # cd pkg && ( ./make-deb-packages ) > > Those are created via CPack, which uses CMake?s installation mechanism, so > the first idea that comes to mind is to edit CMakeList.txt and insert > install*() commands. > > - Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141113/061c2c80/attachment.html From scottie.Dawson at colostate.edu Mon Nov 17 12:04:34 2014 From: scottie.Dawson at colostate.edu (Dawson,Scottie) Date: Mon, 17 Nov 2014 20:04:34 +0000 Subject: [Bro] First time Bro Cluster Spin up Message-ID: <9B0294B635D5C541A9C14A01F49E189E384174EA@ex11.colostate.edu> Hi. I am attempting to get a Bro cluster working and I get the following error for all my workers, (full output of diag below). "fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied" Thoughts on what I am missing? Config: I have an Endace DAG8.1SX set up to run with 22 streams. 1 Server set up to have 22 instances of BRO workers on it 1 server set up as the manager and prox Using libpcap-1.6.2 dag-5.2.0. bro version 2.3.1 TS: 1. I have run the following command on both the worker and the manager/proxy servers. sudo setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro 2. I can launch bro manually on the worker if I use sudo acns-bro at endace:/usr/local/bro/bin$ sudo ./bro -i dag0:42 listening on dag0:42, capture length 8192 bytes ^C1416254260.140036 received termination signal 1416254260.140036 209 packets received on interface dag0:42, 0 dropped 3. Manipulated the user launching bro (acns-bro) group permissions to be in the adm group FULL OUTPUT of DIAG: BroControl] > diag worker-21 [worker-21] Bro 2.3.1 Linux 3.13.0-39-generic ==== No reporter.log ==== stderr.log [dag_open] dag_clone dagfd for dagiom: Permission denied fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dag0:42 -U .status -p broctl -p broctl-live -p local -p worker-21 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site CLUSTER_NODE=worker-21 ==== .status TERMINATED [atexit] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log Scott Dawson ACNS Network Security Colorado State University 970-297-3712 "chop wood carry water" -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141117/ab7d30a8/attachment.html From donaldson8 at llnl.gov Mon Nov 17 12:14:44 2014 From: donaldson8 at llnl.gov (Donaldson, John) Date: Mon, 17 Nov 2014 20:14:44 +0000 Subject: [Bro] First time Bro Cluster Spin up In-Reply-To: <9B0294B635D5C541A9C14A01F49E189E384174EA@ex11.colostate.edu> References: <9B0294B635D5C541A9C14A01F49E189E384174EA@ex11.colostate.edu> Message-ID: Dawson, Do you have any other processes listening to those streams? Only one process can attach to each stream, and we usually see those types of errors when a Bro worker tries to attach to a stream that is already in use. This also happens if a process doesn't cleanly release the stream, and, in our experience, requires a full reboot to clear. v/r John Donaldson From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Dawson,Scottie Sent: Monday, November 17, 2014 12:05 PM To: bro at bro-ids.org Subject: [Bro] First time Bro Cluster Spin up Hi. I am attempting to get a Bro cluster working and I get the following error for all my workers, (full output of diag below). "fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied" Thoughts on what I am missing? Config: I have an Endace DAG8.1SX set up to run with 22 streams. 1 Server set up to have 22 instances of BRO workers on it 1 server set up as the manager and prox Using libpcap-1.6.2 dag-5.2.0. bro version 2.3.1 TS: 1. I have run the following command on both the worker and the manager/proxy servers. sudo setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro 2. I can launch bro manually on the worker if I use sudo acns-bro at endace:/usr/local/bro/bin$ sudo ./bro -i dag0:42 listening on dag0:42, capture length 8192 bytes ^C1416254260.140036 received termination signal 1416254260.140036 209 packets received on interface dag0:42, 0 dropped 3. Manipulated the user launching bro (acns-bro) group permissions to be in the adm group FULL OUTPUT of DIAG: BroControl] > diag worker-21 [worker-21] Bro 2.3.1 Linux 3.13.0-39-generic ==== No reporter.log ==== stderr.log [dag_open] dag_clone dagfd for dagiom: Permission denied fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dag0:42 -U .status -p broctl -p broctl-live -p local -p worker-21 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site CLUSTER_NODE=worker-21 ==== .status TERMINATED [atexit] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log Scott Dawson ACNS Network Security Colorado State University 970-297-3712 "chop wood carry water" -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141117/49c2e0b4/attachment.html From Keith_Schoenefeld at baylor.edu Mon Nov 17 12:22:05 2014 From: Keith_Schoenefeld at baylor.edu (Schoenefeld, Keith P.) Date: Mon, 17 Nov 2014 20:22:05 +0000 Subject: [Bro] First time Bro Cluster Spin up In-Reply-To: <9B0294B635D5C541A9C14A01F49E189E384174EA@ex11.colostate.edu> References: <9B0294B635D5C541A9C14A01F49E189E384174EA@ex11.colostate.edu> Message-ID: You must be running this on Linux as a user other than root (good for you). Run this command on each of the cluster nodes: setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/bro && setcap cap_net_raw,cap_net_admin=eip /opt/bro/bin/capstats I'd give credit if I could recall where I found this, I found it on a website somewhere. I blame Seth Hall. Note that you'll have to do this every time you push cluster config changes as well. -- KS Keith Schoenefeld Information Security Analyst Baylor University 254-710-6667 -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Dawson,Scottie Sent: Monday, November 17, 2014 2:05 PM To: bro at bro-ids.org Subject: [Bro] First time Bro Cluster Spin up Hi. I am attempting to get a Bro cluster working and I get the following error for all my workers, (full output of diag below). "fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied" Thoughts on what I am missing? Config: I have an Endace DAG8.1SX set up to run with 22 streams. 1 Server set up to have 22 instances of BRO workers on it 1 server set up as the manager and prox Using libpcap-1.6.2 dag-5.2.0. bro version 2.3.1 TS: 1. I have run the following command on both the worker and the manager/proxy servers. sudo setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro 2. I can launch bro manually on the worker if I use sudo acns-bro at endace:/usr/local/bro/bin$ sudo ./bro -i dag0:42 listening on dag0:42, capture length 8192 bytes ^C1416254260.140036 received termination signal 1416254260.140036 209 packets received on interface dag0:42, 0 dropped 3. Manipulated the user launching bro (acns-bro) group permissions to be in the adm group FULL OUTPUT of DIAG: BroControl] > diag worker-21 [worker-21] Bro 2.3.1 Linux 3.13.0-39-generic ==== No reporter.log ==== stderr.log [dag_open] dag_clone dagfd for dagiom: Permission denied fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dag0:42 -U .status -p broctl -p broctl-live -p local -p worker-21 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site CLUSTER_NODE=worker-21 ==== .status TERMINATED [atexit] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log Scott Dawson ACNS Network Security Colorado State University 970-297-3712 "chop wood carry water" From edthoma at sandia.gov Mon Nov 17 15:04:46 2014 From: edthoma at sandia.gov (Thomas, Eric D) Date: Mon, 17 Nov 2014 23:04:46 +0000 Subject: [Bro] How to use PF_RING multi? Message-ID: I have not yet tried zbalance (I will someday) because I?m now told that the packet stream is flow balanced upstream of the sensor. Each interface (em1 and em2) will get bi-directional flows. Each will get half of the total flows. So, can I have separate Bro pf_ring configs for each interface? That is, will the below node.cfg work? It is starting 8 worker procs as expected, but I?m not sure whether bro is doing what I think it would with this config. [manager] type=manager host=10.0.0.1 [proxy-1] type=proxy host=10.0.0.1 [bro-em1] type=worker host=10.0.0.1 interface=zc:em1 lb_method=pf_ring lb_procs=4 [bro-em2] type=worker host=10.0.0.1 interface=zc:em2 lb_method=pf_ring lb_procs=4 -- Eric Thomas edthoma at sandia.gov On 11/13/14, 12:17 PM, "Seth Hall" wrote: > >I'm afraid we don't have a terribly elegant method to do that with >PF_Ring right now. You could use their ZC module and do the load >balancing in userspace with their zbalance_ipc tool (or whatever it's >called). I think that can merge traffic and distribute it out and we >support sniffing from ZC load balanced interfaces. > >This is yet another area where our upcoming packet-bricks tool will make >life easier. I just wish it was ready for people to generally use. :/ > > .Seth > >-- >Seth Hall >International Computer Science Institute >(Bro) because everyone has a network >http://www.bro.org/ > From scottie.Dawson at colostate.edu Tue Nov 18 06:17:04 2014 From: scottie.Dawson at colostate.edu (Dawson,Scottie) Date: Tue, 18 Nov 2014 14:17:04 +0000 Subject: [Bro] First time Bro Cluster Spin up In-Reply-To: <9B0294B635D5C541A9C14A01F49E189E384174EA@ex11.colostate.edu> References: <9B0294B635D5C541A9C14A01F49E189E384174EA@ex11.colostate.edu> Message-ID: <9B0294B635D5C541A9C14A01F49E189E38418FD8@ex11.colostate.edu> Thank you both John and Keith for the suggestions. I rebooted the server for good measure and re-ran the setcap command, with the same results in broctl. worker-21 terminated immediately after starting; check output with "diag" ..... Diag worker-21 ... .... ==== stderr.log fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied ....... Is anyone using an Endace card in there cluster? I am starting to suspect that the permissions issue has to do with how I am configuring and launching the capture card. I am going to reach out to their support organization and see if there is a way to launch the card as a non root user. I am sure I can get all this working with root but I am trying to avoid that. Thanks again scott "chop wood carry water" From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Dawson,Scottie Sent: Monday, November 17, 2014 1:05 PM To: bro at bro-ids.org Subject: [Bro] First time Bro Cluster Spin up Hi. I am attempting to get a Bro cluster working and I get the following error for all my workers, (full output of diag below). "fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied" Thoughts on what I am missing? Config: I have an Endace DAG8.1SX set up to run with 22 streams. 1 Server set up to have 22 instances of BRO workers on it 1 server set up as the manager and prox Using libpcap-1.6.2 dag-5.2.0. bro version 2.3.1 TS: 1. I have run the following command on both the worker and the manager/proxy servers. sudo setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro 2. I can launch bro manually on the worker if I use sudo acns-bro at endace:/usr/local/bro/bin$ sudo ./bro -i dag0:42 listening on dag0:42, capture length 8192 bytes ^C1416254260.140036 received termination signal 1416254260.140036 209 packets received on interface dag0:42, 0 dropped 3. Manipulated the user launching bro (acns-bro) group permissions to be in the adm group FULL OUTPUT of DIAG: BroControl] > diag worker-21 [worker-21] Bro 2.3.1 Linux 3.13.0-39-generic ==== No reporter.log ==== stderr.log [dag_open] dag_clone dagfd for dagiom: Permission denied fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dag0:42 -U .status -p broctl -p broctl-live -p local -p worker-21 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site CLUSTER_NODE=worker-21 ==== .status TERMINATED [atexit] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log Scott Dawson ACNS Network Security Colorado State University 970-297-3712 "chop wood carry water" -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141118/74e0f877/attachment.html From awells at digiumcloud.com Tue Nov 18 07:38:34 2014 From: awells at digiumcloud.com (Aubrey Wells) Date: Tue, 18 Nov 2014 10:38:34 -0500 Subject: [Bro] Checking for null/undefined variables Message-ID: Greetings all, Fairly new to Bro and feel a bit silly that I can't figure this out, but I'm working on a Bro script that hooks into the notice framework to call an external script when a Notice is fired. I'm wanting to pass all (well, most) of the variables in the Notice object to my script, but I'm having trouble. I think that what is happening is some notices do not have all the variables defined and if I try to use one of those variables, my script block skips execution. So, what I thought I would do was instead of using n$dst for example, I would do something like: local dst = (n$dst == NULL ? "" : n$dst); I tried various forms using if blocks and regular assignment instead of the ternary operator but the end result is the same, nothing seems to happen. Is there a way to check for null/undefined variables that I'm just missing, or do I need to rethink they way I'm doing this entirely? Bro version 2.3 CentOS 6.5 w/ kernel 2.6.32-431.23.3.el6.x86_64 My code: http://pastebin.com/KR5vkdXM --------------------- Aubrey Wells Manager, Network Operations Digium Cloud Services Main: 888.305.3850 Support: 877.344.4861 or http://www.digium.com/en/support -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141118/7b58b886/attachment.html From jsiwek at illinois.edu Tue Nov 18 07:51:12 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 18 Nov 2014 15:51:12 +0000 Subject: [Bro] Checking for null/undefined variables In-Reply-To: References: Message-ID: > On Nov 18, 2014, at 9:38 AM, Aubrey Wells wrote: > > Is there a way to check for null/undefined variables that I'm just missing, or do I need to rethink they way I'm doing this entirely? The ??$? operator can be used to check whether an optional field of a record actually has a value. Example: http://try.bro.org/#/trybro/saved/96c4ba9d-3d49-4372-a01b-aa31157e46e1 - Jon From awells at digiumcloud.com Tue Nov 18 07:52:54 2014 From: awells at digiumcloud.com (Aubrey Wells) Date: Tue, 18 Nov 2014 10:52:54 -0500 Subject: [Bro] Checking for null/undefined variables In-Reply-To: References: Message-ID: Beautiful, thank you! --------------------- Aubrey Wells Manager, Network Operations Digium Cloud Services Main: 888.305.3850 Support: 877.344.4861 or http://www.digium.com/en/support On Tue, Nov 18, 2014 at 10:51 AM, Siwek, Jon wrote: > > > On Nov 18, 2014, at 9:38 AM, Aubrey Wells > wrote: > > > > Is there a way to check for null/undefined variables that I'm just > missing, or do I need to rethink they way I'm doing this entirely? > > The ??$? operator can be used to check whether an optional field of a > record actually has a value. Example: > > http://try.bro.org/#/trybro/saved/96c4ba9d-3d49-4372-a01b-aa31157e46e1 > > - Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141118/9b28980d/attachment.html From Ioannis.PSAROUDAKIS at ec.europa.eu Tue Nov 18 09:06:46 2014 From: Ioannis.PSAROUDAKIS at ec.europa.eu (Ioannis.PSAROUDAKIS at ec.europa.eu) Date: Tue, 18 Nov 2014 17:06:46 +0000 Subject: [Bro] Exclude IPS Message-ID: Hi All, I am running the latest version of Bro and I would like to exclude (not at all log) events from specific IPs. Can someone provide me with a link/info on how to do this? Thnx for your time. Regards Ioannis -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141118/f1aecd2d/attachment.html From dhoelzer at sans.org Tue Nov 18 09:10:10 2014 From: dhoelzer at sans.org (David Hoelzer) Date: Tue, 18 Nov 2014 12:10:10 -0500 Subject: [Bro] Workers/Proxies & port forwarding Message-ID: <767ECB13-373F-4C4A-A344-03C5B13F722D@sans.org> I recently spent some time trying to work through an issue that arises when port forwarding is in use. I?m wondering if I?m missing something obvious. Bro workers, proxies and managers seem to be very averse to port forwarding. I have a few workers that I?m trying to deploy outside of a perimeter. I am loathe to simply create permit rules and the infrastructure does not lend itself to creating a dark LAN just for Bro workers out of band. The compromise is that there is no issue creating a port forwarding rule that will permit these systems to get back to the proxy. Unfortunately, the workers and the proxy are very unhappy with this arrangement. Aside from creating really weird local IPTables rules that allow me to essentially trick them into thinking that they are talking to the ?actual? addresses, is there a more elegant way to do this with Bro as it stands today? Thanks From kmcmahon at mitre.org Tue Nov 18 09:43:01 2014 From: kmcmahon at mitre.org (McMahon, Kevin J) Date: Tue, 18 Nov 2014 17:43:01 +0000 Subject: [Bro] Exclude IPS In-Reply-To: References: Message-ID: <00D3CD29F7C24A44B4D23450BB8E55B3106D4275@IMCMBX03.MITRE.ORG> redef restrict_filters += [["blockedIPs"] = "not net 192.168.1.0/24"]; I think you may need to also include: redef PacketFilter::all_packets = F; I have both of these statements in my config, but I put them in there a long time ago. From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Ioannis.PSAROUDAKIS at ec.europa.eu Sent: Tuesday, November 18, 2014 12:07 PM To: bro at bro.org Subject: [Bro] Exclude IPS Hi All, I am running the latest version of Bro and I would like to exclude (not at all log) events from specific IPs. Can someone provide me with a link/info on how to do this? Thnx for your time. Regards Ioannis -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141118/45b770fd/attachment.html From grant at grantstavely.com Tue Nov 18 11:14:12 2014 From: grant at grantstavely.com (Grant Stavely) Date: Tue, 18 Nov 2014 11:14:12 -0800 Subject: [Bro] Exclude IPS In-Reply-To: <00D3CD29F7C24A44B4D23450BB8E55B3106D4275@IMCMBX03.MITRE.ORG> References: <00D3CD29F7C24A44B4D23450BB8E55B3106D4275@IMCMBX03.MITRE.ORG> Message-ID: Hi Ioannis, Docs:?https://www.bro.org/sphinx/scripts/base/frameworks/packet-filter/main.bro.html? I use this in my local.bro. A redef of restrict_filters should work too: ################################################################################ # Capture filter ################################################################################ event bro_init() &priority=-12 { restrict_filters["Ioannis.PSAROUDAKIS at ec.europa.eu"] = "not host x.x.x.x or net x.x.x.x/x or..."; PacketFilter::install(); } Grant On November 18, 2014 at 9:58:28 AM, McMahon, Kevin J (kmcmahon at mitre.org) wrote: redef restrict_filters += [[?blockedIPs?] = ?not net 192.168.1.0/24?]; ? I think you may need to also include: redef PacketFilter::all_packets = F;? I have both of these statements in my config, but I put them in there a long time ago. ? ? From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Ioannis.PSAROUDAKIS at ec.europa.eu Sent: Tuesday, November 18, 2014 12:07 PM To: bro at bro.org Subject: [Bro] Exclude IPS ? Hi All, ? I am running the latest version of Bro and I would like to exclude (not at all log) events from specific IPs. Can someone provide me with a link/info on how to do this? ? Thnx for your time. ? Regards Ioannis _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141118/74f8457e/attachment.html From scottie.Dawson at colostate.edu Tue Nov 18 11:22:17 2014 From: scottie.Dawson at colostate.edu (Dawson,Scottie) Date: Tue, 18 Nov 2014 19:22:17 +0000 Subject: [Bro] First time Bro Cluster Spin up In-Reply-To: <9B0294B635D5C541A9C14A01F49E189E38418FD8@ex11.colostate.edu> References: <9B0294B635D5C541A9C14A01F49E189E384174EA@ex11.colostate.edu> <9B0294B635D5C541A9C14A01F49E189E38418FD8@ex11.colostate.edu> Message-ID: <9B0294B635D5C541A9C14A01F49E189E38419661@ex11.colostate.edu> It was the Endace Capture card. scott "chop wood carry water" From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Dawson,Scottie Sent: Tuesday, November 18, 2014 7:17 AM To: bro at bro-ids.org Subject: Re: [Bro] First time Bro Cluster Spin up Thank you both John and Keith for the suggestions. I rebooted the server for good measure and re-ran the setcap command, with the same results in broctl. worker-21 terminated immediately after starting; check output with "diag" ..... Diag worker-21 ... .... ==== stderr.log fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied ....... Is anyone using an Endace card in there cluster? I am starting to suspect that the permissions issue has to do with how I am configuring and launching the capture card. I am going to reach out to their support organization and see if there is a way to launch the card as a non root user. I am sure I can get all this working with root but I am trying to avoid that. Thanks again scott "chop wood carry water" From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Dawson,Scottie Sent: Monday, November 17, 2014 1:05 PM To: bro at bro-ids.org Subject: [Bro] First time Bro Cluster Spin up Hi. I am attempting to get a Bro cluster working and I get the following error for all my workers, (full output of diag below). "fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied" Thoughts on what I am missing? Config: I have an Endace DAG8.1SX set up to run with 22 streams. 1 Server set up to have 22 instances of BRO workers on it 1 server set up as the manager and prox Using libpcap-1.6.2 dag-5.2.0. bro version 2.3.1 TS: 1. I have run the following command on both the worker and the manager/proxy servers. sudo setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro 2. I can launch bro manually on the worker if I use sudo acns-bro at endace:/usr/local/bro/bin$ sudo ./bro -i dag0:42 listening on dag0:42, capture length 8192 bytes ^C1416254260.140036 received termination signal 1416254260.140036 209 packets received on interface dag0:42, 0 dropped 3. Manipulated the user launching bro (acns-bro) group permissions to be in the adm group FULL OUTPUT of DIAG: BroControl] > diag worker-21 [worker-21] Bro 2.3.1 Linux 3.13.0-39-generic ==== No reporter.log ==== stderr.log [dag_open] dag_clone dagfd for dagiom: Permission denied fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i dag0:42 -U .status -p broctl -p broctl-live -p local -p worker-21 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto ==== .env_vars PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site CLUSTER_NODE=worker-21 ==== .status TERMINATED [atexit] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log Scott Dawson ACNS Network Security Colorado State University 970-297-3712 "chop wood carry water" -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141118/1e295891/attachment.html From mike.patterson at uwaterloo.ca Tue Nov 18 12:34:34 2014 From: mike.patterson at uwaterloo.ca (Mike Patterson) Date: Tue, 18 Nov 2014 20:34:34 +0000 Subject: [Bro] First time Bro Cluster Spin up In-Reply-To: <9B0294B635D5C541A9C14A01F49E189E38419661@ex11.colostate.edu> References: <9B0294B635D5C541A9C14A01F49E189E384174EA@ex11.colostate.edu> <9B0294B635D5C541A9C14A01F49E189E38418FD8@ex11.colostate.edu> <9B0294B635D5C541A9C14A01F49E189E38419661@ex11.colostate.edu> Message-ID: Just FYI, I run a DAG 9.2 here - in the past, I?ve run into situations where a crashing worker (or Snort process) will ?lock? its stream. I usually do a cold power off in those situations, as a reboot didn?t always seem to clear it. I don?t know if that?s what you ran into, but it?s something to keep in mind anyway. It?s not clear to me - did you manage to get it working as non-root then? Mike -- Simple, clear purpose and principles give rise to complex and intelligent behavior. Complex rules and regulations give rise to simple and stupid behavior. - Dee Hock > On Nov 18, 2014, at 2:22 PM, Dawson,Scottie wrote: > > It was the Endace Capture card. > > > > scott > > > > "chop wood carry water" > > > > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Dawson,Scottie > Sent: Tuesday, November 18, 2014 7:17 AM > To: bro at bro-ids.org > Subject: Re: [Bro] First time Bro Cluster Spin up > > > > Thank you both John and Keith for the suggestions. I rebooted the server for good measure and re-ran the setcap command, with the same results in broctl. > > > > worker-21 terminated immediately after starting; check output with "diag" > > > > ?.. > > > > Diag worker-21 > > > > ? > > ?. > > ==== stderr.log > > fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied > > > > ??. > > > > > > Is anyone using an Endace card in there cluster? I am starting to suspect that the permissions issue has to do with how I am configuring and launching the capture card. I am going to reach out to their support organization and see if there is a way to launch the card as a non root user. > > > > I am sure I can get all this working with root but I am trying to avoid that. > > > > Thanks again > > > > scott > > > > "chop wood carry water" > > > > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Dawson,Scottie > Sent: Monday, November 17, 2014 1:05 PM > To: bro at bro-ids.org > Subject: [Bro] First time Bro Cluster Spin up > > > > Hi. > > > > I am attempting to get a Bro cluster working and I get the following error for all my workers, (full output of diag below). ?fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied? > > > > Thoughts on what I am missing? > > > > > > Config: > > I have an Endace DAG8.1SX set up to run with 22 streams. > > 1 Server set up to have 22 instances of BRO workers on it > > 1 server set up as the manager and prox > > Using libpcap-1.6.2 > > dag-5.2.0. > > bro version 2.3.1 > > > > > > > > TS: > > > > 1. I have run the following command on both the worker and the manager/proxy servers. > > > > sudo setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro > > > > 2. I can launch bro manually on the worker if I use sudo > > > > acns-bro at endace:/usr/local/bro/bin$ sudo ./bro -i dag0:42 > > listening on dag0:42, capture length 8192 bytes > > > > ^C1416254260.140036 received termination signal > > 1416254260.140036 209 packets received on interface dag0:42, 0 dropped > > > > 3. Manipulated the user launching bro (acns-bro) group permissions to be in the adm group > > > > > > > > FULL OUTPUT of DIAG: > > BroControl] > diag worker-21 > > [worker-21] > > > > Bro 2.3.1 > > Linux 3.13.0-39-generic > > > > > > ==== No reporter.log > > > > ==== stderr.log > > [dag_open] dag_clone dagfd for dagiom: Permission denied > > fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied > > > > > > ==== stdout.log > > max memory size (kbytes, -m) unlimited > > data seg size (kbytes, -d) unlimited > > virtual memory (kbytes, -v) unlimited > > core file size (blocks, -c) unlimited > > > > ==== .cmdline > > -i dag0:42 -U .status -p broctl -p broctl-live -p local -p worker-21 local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto > > > > ==== .env_vars > > PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games > > BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site > > CLUSTER_NODE=worker-21 > > > > ==== .status > > TERMINATED [atexit] > > > > ==== No prof.log > > > > ==== No packet_filter.log > > > > ==== No loaded_scripts.log > > > > > > > > Scott Dawson > ACNS Network Security > > Colorado State University > 970-297-3712 > > > > "chop wood carry water" > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From scottie.Dawson at colostate.edu Tue Nov 18 13:14:09 2014 From: scottie.Dawson at colostate.edu (Dawson,Scottie) Date: Tue, 18 Nov 2014 21:14:09 +0000 Subject: [Bro] First time Bro Cluster Spin up In-Reply-To: References: <9B0294B635D5C541A9C14A01F49E189E384174EA@ex11.colostate.edu> <9B0294B635D5C541A9C14A01F49E189E38418FD8@ex11.colostate.edu> <9B0294B635D5C541A9C14A01F49E189E38419661@ex11.colostate.edu> Message-ID: <9B0294B635D5C541A9C14A01F49E189E3841979A@ex11.colostate.edu> Mike, Yes I did, or at least I don?t have to type sudo in front of the commands to launch and configure the endace card or bro anymore :) . I had to edit the file dagload and change the "group" from root to the group I was using. I than had to change ownership on /dev/dag* and /usr/loca/bin/dag* files to the group also. scott "chop wood carry water" -----Original Message----- From: Mike Patterson [mailto:mike.patterson at uwaterloo.ca] Sent: Tuesday, November 18, 2014 1:35 PM To: Dawson,Scottie Cc: bro at bro-ids.org Subject: Re: [Bro] First time Bro Cluster Spin up Just FYI, I run a DAG 9.2 here - in the past, I?ve run into situations where a crashing worker (or Snort process) will ?lock? its stream. I usually do a cold power off in those situations, as a reboot didn?t always seem to clear it. I don?t know if that?s what you ran into, but it?s something to keep in mind anyway. It?s not clear to me - did you manage to get it working as non-root then? Mike -- Simple, clear purpose and principles give rise to complex and intelligent behavior. Complex rules and regulations give rise to simple and stupid behavior. - Dee Hock > On Nov 18, 2014, at 2:22 PM, Dawson,Scottie wrote: > > It was the Endace Capture card. > > > > scott > > > > "chop wood carry water" > > > > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of > Dawson,Scottie > Sent: Tuesday, November 18, 2014 7:17 AM > To: bro at bro-ids.org > Subject: Re: [Bro] First time Bro Cluster Spin up > > > > Thank you both John and Keith for the suggestions. I rebooted the server for good measure and re-ran the setcap command, with the same results in broctl. > > > > worker-21 terminated immediately after starting; check output with "diag" > > > > ?.. > > > > Diag worker-21 > > > > ? > > ?. > > ==== stderr.log > > fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - > pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied > > > > ??. > > > > > > Is anyone using an Endace card in there cluster? I am starting to suspect that the permissions issue has to do with how I am configuring and launching the capture card. I am going to reach out to their support organization and see if there is a way to launch the card as a non root user. > > > > I am sure I can get all this working with root but I am trying to avoid that. > > > > Thanks again > > > > scott > > > > "chop wood carry water" > > > > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of > Dawson,Scottie > Sent: Monday, November 17, 2014 1:05 PM > To: bro at bro-ids.org > Subject: [Bro] First time Bro Cluster Spin up > > > > Hi. > > > > I am attempting to get a Bro cluster working and I get the following error for all my workers, (full output of diag below). ?fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied? > > > > Thoughts on what I am missing? > > > > > > Config: > > I have an Endace DAG8.1SX set up to run with 22 streams. > > 1 Server set up to have 22 instances of BRO workers on it > > 1 server set up as the manager and prox > > Using libpcap-1.6.2 > > dag-5.2.0. > > bro version 2.3.1 > > > > > > > > TS: > > > > 1. I have run the following command on both the worker and the manager/proxy servers. > > > > sudo setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro > > > > 2. I can launch bro manually on the worker if I use sudo > > > > acns-bro at endace:/usr/local/bro/bin$ sudo ./bro -i dag0:42 > > listening on dag0:42, capture length 8192 bytes > > > > ^C1416254260.140036 received termination signal > > 1416254260.140036 209 packets received on interface dag0:42, 0 dropped > > > > 3. Manipulated the user launching bro (acns-bro) group permissions to be in the adm group > > > > > > > > FULL OUTPUT of DIAG: > > BroControl] > diag worker-21 > > [worker-21] > > > > Bro 2.3.1 > > Linux 3.13.0-39-generic > > > > > > ==== No reporter.log > > > > ==== stderr.log > > [dag_open] dag_clone dagfd for dagiom: Permission denied > > fatal error: /usr/local/bro/bin/bro: problem with interface dag0:42 - > pcap_open_live: dag0:42: dag_open /dev/dag0: Permission denied > > > > > > ==== stdout.log > > max memory size (kbytes, -m) unlimited > > data seg size (kbytes, -d) unlimited > > virtual memory (kbytes, -v) unlimited > > core file size (blocks, -c) unlimited > > > > ==== .cmdline > > -i dag0:42 -U .status -p broctl -p broctl-live -p local -p worker-21 > local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto > > > > ==== .env_vars > > PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local > /sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/loc > al/games > > BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr > /local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/sh > are/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site > > CLUSTER_NODE=worker-21 > > > > ==== .status > > TERMINATED [atexit] > > > > ==== No prof.log > > > > ==== No packet_filter.log > > > > ==== No loaded_scripts.log > > > > > > > > Scott Dawson > ACNS Network Security > > Colorado State University > 970-297-3712 > > > > "chop wood carry water" > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From hckim at narusec.com Tue Nov 18 16:54:52 2014 From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=) Date: Wed, 19 Nov 2014 09:54:52 +0900 Subject: [Bro] Exclude IPS Message-ID: Hi I have exclude IP using these command local.bro: redef PacketFilter::enable_auto_protocol_capture_filters = F; redef capture_filters = { ["all"] = "ip or not ip" }; local-worker.bro: redef restrict_filters = { ["not-hosts"] = "not host X.X.X.X" }; -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141119/b60beb06/attachment.html From seth at icir.org Wed Nov 19 06:09:09 2014 From: seth at icir.org (Seth Hall) Date: Wed, 19 Nov 2014 09:09:09 -0500 Subject: [Bro] Exclude IPS In-Reply-To: References: Message-ID: <9312CBC0-337A-4012-8878-3013031C8658@icir.org> > On Nov 18, 2014, at 7:54 PM, ??? wrote: > > redef PacketFilter::enable_auto_protocol_capture_filters = F; > > redef capture_filters = { ["all"] = "ip or not ip" }; > > > local-worker.bro: > > redef restrict_filters = { ["not-hosts"] = "not host X.X.X.X" }; Hi Hichul! You could actually simplify this all by just putting that last line in local.bro. The rest aren't necessary. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From hckim at narusec.com Wed Nov 19 21:55:20 2014 From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=) Date: Thu, 20 Nov 2014 14:55:20 +0900 Subject: [Bro] Exclude IPS In-Reply-To: <9312CBC0-337A-4012-8878-3013031C8658@icir.org> References: <9312CBC0-337A-4012-8878-3013031C8658@icir.org> Message-ID: Hi Seth Thank you I put redef restrict_filters = { ["not-hosts"] = "not host X.X.X.X" }; in a local.bro and it worked. very simple oneliner Thank's On Wed, Nov 19, 2014 at 11:09 PM, Seth Hall wrote: > > > On Nov 18, 2014, at 7:54 PM, ??? wrote: > > > > redef PacketFilter::enable_auto_protocol_capture_filters = F; > > > > redef capture_filters = { ["all"] = "ip or not ip" }; > > > > > > local-worker.bro: > > > > redef restrict_filters = { ["not-hosts"] = "not host X.X.X.X" }; > > Hi Hichul! > > You could actually simplify this all by just putting that last line in > local.bro. The rest aren't necessary. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141120/34104cee/attachment.html From gl89 at cornell.edu Thu Nov 20 13:31:15 2014 From: gl89 at cornell.edu (Glenn Forbes Fleming Larratt) Date: Thu, 20 Nov 2014 16:31:15 -0500 (EST) Subject: [Bro] two newb questions Message-ID: Folks, 1. Is this list archived? 2. Is there a FAQ/etiquette guide/standards guide for this list? Thanks, -- Glenn Forbes Fleming Larratt Cornell University IT Security Office From jdopheid at illinois.edu Thu Nov 20 13:42:46 2014 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Thu, 20 Nov 2014 21:42:46 +0000 Subject: [Bro] two newb questions In-Reply-To: Message-ID: Hi Glenn, Thanks for contacting us. Yes we have an archive, see the link to the archive and a search bar in our Community page: https://www.bro.org/community/index.html We don't have a specific standards for the mailing list, our description on the page covers the general idea of the purpose of the mailing list: The best place ask questions is the Bro user mailing list. Feel free to ask for help, post your thoughts on Bro, and announce related work and projects that may be of interest to other Bro users. Search the mailing list?s archive using the field below. Thanks, Jeannette ------ Jeannette M. Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign On 11/20/14, 3:31 PM, "Glenn Forbes Fleming Larratt" wrote: Folks, 1. Is this list archived? 2. Is there a FAQ/etiquette guide/standards guide for this list? Thanks, -- Glenn Forbes Fleming Larratt Cornell University IT Security Office _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From doris at bro.org Thu Nov 20 15:54:23 2014 From: doris at bro.org (Doris Schioberg) Date: Thu, 20 Nov 2014 15:54:23 -0800 Subject: [Bro] Bro Monthly #3 Message-ID: <546E7F2F.6000508@bro.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 A few days late: Bro Monthly #3 is here and covers Bro Meet-ups, a new category to inform you about Bro related gatherings, news on ISLET, two new Bro publications, and more. http://blog.bro.org/2014/11/bro-monthly-3.html - - The Bro Team - -- Doris Schioberg Bro Outreach, Training, and Education Coordinator International Computer Science Institute (ICSI Berkeley) Phone: +1 (510) 289-8406 * doris at bro.org -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJUbn8uAAoJEFIoZlhdVQyUb2QP/j+jLW+4ljI5n4oSw+fOZj7w N1oG1kzjezoCDsEEAZIfqdWyr1U5d1hFcjqqYHt9d0xYk5bsHNdVx/jmTYNn5j4N HkenLxbE1H32hrWs9LLWCND8TLikpDNrdsXCMewqIQZ2YVZ26ObrYjJoc4HAzy5w g8LzwNIVmlg9f5/uVz09ujNByCM2Bilygb5kLe6RfzLhzbulhwCiYQ6v8K6slEKs DC9Pn3npqculmG1ND6sa2tVDYO3Nni2qo1J0MuTR6+LMmd7CTghwFKfnApSQfHoO 2PMrRo82VLV76FFCUsXuMafpVhjsqeJJ2tO2SuytbBtWvobSd9XfbD29+yKAgTmN TyosKU3XKLCKI4VFEu8+msIzs6VTHwOv8daYNjF6d2kD9fDP7Zg7qjboN1FK6cHc xxdWbnO/XhugQZ9+d4ImQU46j/wzpyogwGTEsdDcnrD2Kbb7SJ/3F9YpbMfGb9Od EY8CUnkAL9ukQEV4MBI9DOHqoJmM/9eff/f7s4CXv8NkpyDTJ5HcOcxyIYeWbNMn vcq9er+FFomuC72uRFlyex8iiSlN7oDPhF8IH/1uZtevQNNZL4HvJBzbK6T116LB 3QThSrNgQ52KWPdQ5iGGxsjihmSkSoaTkucTiThrRfjeq16xobORatlMtx+alJx8 XcIktIaAYmk0/SfHBSq6 =p8zV -----END PGP SIGNATURE----- From Ioannis.PSAROUDAKIS at ec.europa.eu Fri Nov 21 03:21:06 2014 From: Ioannis.PSAROUDAKIS at ec.europa.eu (Ioannis.PSAROUDAKIS at ec.europa.eu) Date: Fri, 21 Nov 2014 11:21:06 +0000 Subject: [Bro] Exclude IPS In-Reply-To: References: <9312CBC0-337A-4012-8878-3013031C8658@icir.org> Message-ID: Hi all, Thank you for your answers. Indeed it works fine for Bro 2.3.1 running in Ubuntu 14.04. From: ??? [mailto:hckim at narusec.com] Sent: Thursday, November 20, 2014 6:55 AM To: Seth Hall Cc: PSAROUDAKIS Ioannis (CERT-EU); bro at bro.org Subject: Re: [Bro] Exclude IPS Hi Seth Thank you I put redef restrict_filters = { ["not-hosts"] = "not host X.X.X.X" }; in a local.bro and it worked. very simple oneliner Thank's On Wed, Nov 19, 2014 at 11:09 PM, Seth Hall > wrote: > On Nov 18, 2014, at 7:54 PM, ??? > wrote: > > redef PacketFilter::enable_auto_protocol_capture_filters = F; > > redef capture_filters = { ["all"] = "ip or not ip" }; > > > local-worker.bro: > > redef restrict_filters = { ["not-hosts"] = "not host X.X.X.X" }; Hi Hichul! You could actually simplify this all by just putting that last line in local.bro. The rest aren't necessary. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141121/e9254c75/attachment.html From Ioannis.PSAROUDAKIS at ec.europa.eu Fri Nov 21 03:23:43 2014 From: Ioannis.PSAROUDAKIS at ec.europa.eu (Ioannis.PSAROUDAKIS at ec.europa.eu) Date: Fri, 21 Nov 2014 11:23:43 +0000 Subject: [Bro] Exclude IPS In-Reply-To: References: <00D3CD29F7C24A44B4D23450BB8E55B3106D4275@IMCMBX03.MITRE.ORG> Message-ID: Hi Grant Thanks for your answer. I tested your proposal and it runs OK for Bro 2.3.1 running in Ubuntu 14.04 except for the "OR" operator. I had to add the second IP to an additional filter line. Regards, Ioannis From: Grant Stavely [mailto:grant at grantstavely.com] Sent: Tuesday, November 18, 2014 8:14 PM To: McMahon, Kevin J; bro at bro.org; PSAROUDAKIS Ioannis (CERT-EU) Subject: Re: [Bro] Exclude IPS Hi Ioannis, Docs: https://www.bro.org/sphinx/scripts/base/frameworks/packet-filter/main.bro.html I use this in my local.bro. A redef of restrict_filters should work too: ################################################################################ # Capture filter ################################################################################ event bro_init() &priority=-12 { restrict_filters["Ioannis.PSAROUDAKIS at ec.europa.eu"] = "not host x.x.x.x or net x.x.x.x/x or..."; PacketFilter::install(); } Grant On November 18, 2014 at 9:58:28 AM, McMahon, Kevin J (kmcmahon at mitre.org) wrote: redef restrict_filters += [[?blockedIPs?] = ?not net 192.168.1.0/24?]; I think you may need to also include: redef PacketFilter::all_packets = F; I have both of these statements in my config, but I put them in there a long time ago. From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Ioannis.PSAROUDAKIS at ec.europa.eu Sent: Tuesday, November 18, 2014 12:07 PM To: bro at bro.org Subject: [Bro] Exclude IPS Hi All, I am running the latest version of Bro and I would like to exclude (not at all log) events from specific IPs. Can someone provide me with a link/info on how to do this? Thnx for your time. Regards Ioannis _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141121/030f7062/attachment.html From gl89 at cornell.edu Fri Nov 21 05:27:24 2014 From: gl89 at cornell.edu (Glenn Forbes Fleming Larratt) Date: Fri, 21 Nov 2014 08:27:24 -0500 (EST) Subject: [Bro] "hash-all-files", er, doesn't? Message-ID: Folks, Fairly new bro user, still figuring things out. I recent changed my local.bro file to call hash-all-files, viz.: ====== #### Network File Handling #### # Enable MD5 and SHA1 hashing for all files. @load frameworks/files/hash-all-files ====== , and I've confirmed that it seems to be loading - "broctl check" seems to return OK, and errors out if I tweak the path so it's invalid. However, I'm not seeing any checksums in the logs/YYYY-MM-DD/file.* files or anywhere else. Is there another piece I need configure? Might I be looking in the wrong place? Is there any telemmetry I can bring to bear to debug this? Thanks for any info or assistance, -- Glenn Forbes Fleming Larratt Cornell University IT Security Office From seth at icir.org Fri Nov 21 07:24:44 2014 From: seth at icir.org (Seth Hall) Date: Fri, 21 Nov 2014 10:24:44 -0500 Subject: [Bro] "hash-all-files", er, doesn't? In-Reply-To: References: Message-ID: <845DB223-51BF-4DF6-B3AD-A2094DF877AC@icir.org> > On Nov 21, 2014, at 8:27 AM, Glenn Forbes Fleming Larratt wrote: > > Is there another piece I need configure? Might I be looking in the wrong > place? Is there any telemmetry I can bring to bear to debug this? Did you run the "install" command in BroControl? Running the install command will essentially stage any changes that you've made to scripts so that the next time things start up they are running the changes you've made. You can imagine if the install command wasn't there it could be bad in the event that something crashed and was restarted while you were changing a script and the one process started running your changes or your script had a syntax error and the process couldn't even start up. That script should already be loaded by local.bro too so I'm actually kind of surprised that it wasn't already working? The result is that out of the box, Bro should be doing MD5 and SHA1 hashes by default when run with BroControl. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From gl89 at cornell.edu Fri Nov 21 08:42:53 2014 From: gl89 at cornell.edu (Glenn Forbes Fleming Larratt) Date: Fri, 21 Nov 2014 11:42:53 -0500 (EST) Subject: [Bro] "hash-all-files", er, doesn't? In-Reply-To: <845DB223-51BF-4DF6-B3AD-A2094DF877AC@icir.org> References: <845DB223-51BF-4DF6-B3AD-A2094DF877AC@icir.org> Message-ID: -- Glenn Forbes Fleming Larratt Cornell University IT Security Office On Fri, 21 Nov 2014, Seth Hall wrote: > > Did you run the "install" command in BroControl? Running the install > command will essentially stage any changes that you've made to scripts > so that the next time things start up they are running the changes > you've made. You can imagine if the install command wasn't there it > could be bad in the event that something crashed and was restarted while > you were changing a script and the one process started running your > changes or your script had a syntax error and the process couldn't even > start up. I did - my process for rule changes goes: broctl check manager proxy{rnd} bro{rnd}-{rnd} broctl install broctl restart > That script should already be loaded by local.bro too so I'm actually > kind of surprised that it wasn't already working? The result is that > out of the box, Bro should be doing MD5 and SHA1 hashes by default when > run with BroControl. A previous maintainer had commented out hash-all-files for performance reasons. > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > From matt.clemons at gmail.com Fri Nov 21 09:06:26 2014 From: matt.clemons at gmail.com (Matt Clemons) Date: Fri, 21 Nov 2014 11:06:26 -0600 Subject: [Bro] Worker Identification Message-ID: Lo All, Is there a way to extend Bro to add a "worker" field in the files.log? I'd like to know where the packets are being processed. I'm doing file carving and the carved files are stored locally to each respective worker. Finding the interface the files crossed is pretty difficult in a large network. Also, it would be nice to extend other logs to see what traffic is crossing what workers in order to map the network. Maybe this is already possible, but i couldn't find much, and I'm pretty new at brogramming. -- Regards, Matt Clemons -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141121/fc088451/attachment.html From donaldson8 at llnl.gov Fri Nov 21 09:33:19 2014 From: donaldson8 at llnl.gov (Donaldson, John) Date: Fri, 21 Nov 2014 17:33:19 +0000 Subject: [Bro] Worker Identification In-Reply-To: References: Message-ID: Matt, We use something like the below to add worker names to our connection logs redef record Conn::Info += { peer_descr: string &default="unknown" &log; }; event connection_state_remove(c: connection){ c$conn$peer_descr = peer_description; } John Donaldson From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Matt Clemons Sent: Friday, November 21, 2014 9:06 AM To: bro at bro.org Subject: [Bro] Worker Identification Lo All, Is there a way to extend Bro to add a "worker" field in the files.log? I'd like to know where the packets are being processed. I'm doing file carving and the carved files are stored locally to each respective worker. Finding the interface the files crossed is pretty difficult in a large network. Also, it would be nice to extend other logs to see what traffic is crossing what workers in order to map the network. Maybe this is already possible, but i couldn't find much, and I'm pretty new at brogramming. -- Regards, Matt Clemons -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141121/4a0df92a/attachment.html From matt.clemons at gmail.com Fri Nov 21 10:49:13 2014 From: matt.clemons at gmail.com (Matt Clemons) Date: Fri, 21 Nov 2014 12:49:13 -0600 Subject: [Bro] Worker Identification In-Reply-To: References: Message-ID: Works like a charm. Thanks John. -Matt On Fri, Nov 21, 2014 at 11:33 AM, Donaldson, John wrote: > Matt, > > > > We use something like the below to add worker names to our connection logs > > > > > > redef record Conn::Info += { > > peer_descr: string &default="unknown" &log; > > }; > > > > event connection_state_remove(c: connection){ > > c$conn$peer_descr = peer_description; > > } > > > > > > John Donaldson > > > > *From:* bro-bounces at bro.org [mailto:bro-bounces at bro.org] *On Behalf Of *Matt > Clemons > *Sent:* Friday, November 21, 2014 9:06 AM > *To:* bro at bro.org > *Subject:* [Bro] Worker Identification > > > > Lo All, > > Is there a way to extend Bro to add a "worker" field in the files.log? > I'd like to know where the packets are being processed. > > I'm doing file carving and the carved files are stored locally to each > respective worker. Finding the interface the files crossed is pretty > difficult in a large network. > > > Also, it would be nice to extend other logs to see what traffic is > crossing what workers in order to map the network. > > > Maybe this is already possible, but i couldn't find much, and I'm pretty > new at brogramming. > > > -- > > Regards, > > Matt Clemons > -- Regards, Matt Clemons (816) 200-0789 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141121/7ca5b40f/attachment.html From seth at icir.org Fri Nov 21 11:08:46 2014 From: seth at icir.org (Seth Hall) Date: Fri, 21 Nov 2014 14:08:46 -0500 Subject: [Bro] "hash-all-files", er, doesn't? In-Reply-To: References: <845DB223-51BF-4DF6-B3AD-A2094DF877AC@icir.org> Message-ID: <7C8F3BB6-C454-4B97-A838-7A99EC4427DD@icir.org> > On Nov 21, 2014, at 11:42 AM, Glenn Forbes Fleming Larratt wrote: > > I did - my process for rule changes goes: > > broctl check manager proxy{rnd} bro{rnd}-{rnd} > broctl install > broctl restart Have you looked at your loaded_scripts.log to see if the script isn't being loaded for some reason? >> That script should already be loaded by local.bro too so I'm actually kind of surprised that it wasn't already working? The result is that out of the box, Bro should be doing MD5 and SHA1 hashes by default when run with BroControl. >> > A previous maintainer had commented out hash-all-files for performance reasons. Ah. For the record, turning that feature off really doesn't change performance all that much (in my informal testing). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From michalpurzynski1 at gmail.com Tue Nov 25 09:58:21 2014 From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=) Date: Tue, 25 Nov 2014 17:58:21 +0000 Subject: [Bro] (no subject) Message-ID: Hi. A script that is a slightly modified version of what's shipped with Bro, gives me interesting results The script source http://michal.pastebin.mozilla.org/7542181 Take a look at lines 1. local key_length = cert$key_length; 2. 3. if ( key_length < notify_minimal_key_length ) 4. NOTICE([$note=Weak_Key, I can see (in notice.log) warnings about host using 1024 bit certificate. Well, the minimal acceptable length is set to 1024 so I should not get any warnings. notice.log 1416937779.196106 CoZK6Z1Y61rsevYSCd 63.245.221.32 34715 10.22.72.139 13000 - - - tcp SSL::Weak_Key Host uses weak certificate with 1024 bit key - 63.245.221.32 10.22.72.139 13000 - nsm7-eth4-6 Notice::ACTION_LOG 86400.000000 F The ssl.log and x509.log show that the connection was over SSL, and the certificate is 1024 bit. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141125/ed614e20/attachment.html From zholt at andrew.cmu.edu Tue Nov 25 10:33:32 2014 From: zholt at andrew.cmu.edu (Zach Holt) Date: Tue, 25 Nov 2014 18:33:32 +0000 Subject: [Bro] (no subject) In-Reply-To: References: Message-ID: Hi Micha?, The standard set by the Certification Authority/Browser (CA/B) Forum required that SSL certificates issued after January 1, 2014 must have a key length of at least 2048-bits. So while some 1024-bit SSL certs may still be valid if they were issued before that date, they are not up to current standards and are quickly becoming deprecated. Additionally, the overlap with SHA-1 phaseout and browser security warnings in the upcoming months, I expect most 1024-bit SSL certs will be killed off quickly. Hope this helps, Zach Zachary Holt Information Security Office Carnegie Mellon University On Nov 25, 2014, at 12:58 PM, Micha? Purzy?ski > wrote: Hi. A script that is a slightly modified version of what's shipped with Bro, gives me interesting results The script source http://michal.pastebin.mozilla.org/7542181 Take a look at lines 1. local key_length = cert$key_length; 2. 3. if ( key_length < notify_minimal_key_length ) 4. NOTICE([$note=Weak_Key, I can see (in notice.log) warnings about host using 1024 bit certificate. Well, the minimal acceptable length is set to 1024 so I should not get any warnings. notice.log 1416937779.196106 CoZK6Z1Y61rsevYSCd 63.245.221.32 34715 10.22.72.139 13000 - - - tcp SSL::Weak_Key Host uses weak certificate with 1024 bit key - 63.245.221.32 10.22.72.139 13000 - nsm7-eth4-6 Notice::ACTION_LOG 86400.000000 F The ssl.log and x509.log show that the connection was over SSL, and the certificate is 1024 bit. _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141125/c1bcc07e/attachment.html From wclarie at gmail.com Tue Nov 25 11:11:05 2014 From: wclarie at gmail.com (Wouter Clarie) Date: Tue, 25 Nov 2014 20:11:05 +0100 Subject: [Bro] Sending UDP packets from Bro script Message-ID: Hi all I'm working on some Bro scripts to log events directly to graphite and/or statsd. I have a working setup which builds a command string to be passed to Exec::run (or just plain old system()) that looks something like: echo 'some.graphite.metric 123' | nc -u -w 1 graphitehost port echo 'some.statsd.metric:123|c' | nc -u w 1 statsdhost port So this has to go to the shell every time, and it depends on netcat. I'm looking for a more elegant way to send UDP packets directly from Bro scripts, but can't find anything so far. Any ideas? Thanks Wouter -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141125/27f87587/attachment.html From jdonnelly at dyn.com Tue Nov 25 11:35:04 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Tue, 25 Nov 2014 13:35:04 -0600 Subject: [Bro] Can bro scripts invoke other shells Message-ID: Hi, Can a bro script run an external (linux ) command from an event handler ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141125/3cc8e8a4/attachment.html From seth at icir.org Tue Nov 25 11:35:48 2014 From: seth at icir.org (Seth Hall) Date: Tue, 25 Nov 2014 14:35:48 -0500 Subject: [Bro] Sending UDP packets from Bro script In-Reply-To: References: Message-ID: <7D3EDB9A-1CFC-4CF9-B5D8-1DDF31405B3D@icir.org> > On Nov 25, 2014, at 2:11 PM, Wouter Clarie wrote: > > I'm looking for a more elegant way to send UDP packets directly from Bro scripts, but can't find anything so far. Any ideas? I've been rumbling for a while now about the need to create a sockets API in Bro. No movement on that front yet. :/ .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From michal at rsbac.org Tue Nov 25 11:49:57 2014 From: michal at rsbac.org (Michal Purzynski) Date: Tue, 25 Nov 2014 20:49:57 +0100 Subject: [Bro] (no subject) In-Reply-To: References: Message-ID: <5474DD65.4040008@rsbac.org> Well that's an interesting story, completely off topic ;-) Anyway, I found the bug, I had the constant redefined somewhere else. Const that you can redef are funny sometimes. On 25/11/14 19:33, Zach Holt wrote: > Hi Micha?, > > The standard set by the Certification Authority/Browser (CA/B) Forum > required that SSL certificates issued after January 1, 2014 must have > a key length of at least 2048-bits. So while some 1024-bit SSL certs > may still be valid if they were issued before that date, they are not > up to current standards and are quickly becoming deprecated. > Additionally, the overlap with SHA-1 phaseout and browser security > warnings in the upcoming months, I expect most 1024-bit SSL certs will > be killed off quickly. > > Hope this helps, > Zach > > Zachary Holt > Information Security Office > Carnegie Mellon University > > > > On Nov 25, 2014, at 12:58 PM, Micha? Purzy?ski > > wrote: > >> Hi. >> >> >> A script that is a slightly modified version of what's shipped with >> Bro, gives me interesting results >> >> The script source >> >> http://michal.pastebin.mozilla.org/7542181 >> >> Take a look at lines >> >> 1. >> local key_length = cert$key_length; >> 2. >> 3. >> if ( key_length < notify_minimal_key_length ) >> 4. >> NOTICE([$note=Weak_Key, >> >> >> >> I can see (in notice.log) warnings about host using 1024 bit >> certificate. Well, the minimal acceptable length is set to 1024 so I >> should not get any warnings. >> >> >> notice.log >> >> >> 1416937779.196106CoZK6Z1Y61rsevYSCd63.245.221.323471510.22.72.13913000---tcpSSL::Weak_KeyHost >> uses weak certificate with 1024 bit >> key-63.245.221.3210.22.72.13913000-nsm7-eth4-6Notice::ACTION_LOG86400.000000F >> >> >> >> The ssl.log and x509.log show that the connection was over SSL, and >> the certificate is 1024 bit. >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141125/d6b574e8/attachment.html From hosom at battelle.org Tue Nov 25 11:55:04 2014 From: hosom at battelle.org (Hosom, Stephen M) Date: Tue, 25 Nov 2014 19:55:04 +0000 Subject: [Bro] Can bro scripts invoke other shells In-Reply-To: References: Message-ID: I think you?re looking for exec. https://www.bro.org/sphinx/scripts/base/utils/exec.bro.html From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of John Donnelly Sent: Tuesday, November 25, 2014 2:35 PM To: bro at bro.org Subject: [Bro] Can bro scripts invoke other shells Hi, Can a bro script run an external (linux ) command from an event handler ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141125/c4bc63d9/attachment.html From sconzo at visiblerisk.com Tue Nov 25 14:40:47 2014 From: sconzo at visiblerisk.com (Mike Sconzo) Date: Tue, 25 Nov 2014 16:40:47 -0600 Subject: [Bro] Can bro scripts invoke other shells In-Reply-To: References: Message-ID: Here's a sample script (for better or worse) that I wrote a while back that looks up files in Virustotal. https://github.com/sooshie/bro-scripts/blob/master/2.2-scripts/vt_check.bro On Tue, Nov 25, 2014 at 1:55 PM, Hosom, Stephen M wrote: > I think you?re looking for exec. > > > > https://www.bro.org/sphinx/scripts/base/utils/exec.bro.html > > > > > > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of John > Donnelly > Sent: Tuesday, November 25, 2014 2:35 PM > To: bro at bro.org > Subject: [Bro] Can bro scripts invoke other shells > > > > Hi, > > Can a bro script run an external (linux ) command from an event handler ? > > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- cat ~/.bash_history > documentation.txt From anthony.kasza at gmail.com Tue Nov 25 15:39:50 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Tue, 25 Nov 2014 15:39:50 -0800 Subject: [Bro] Can bro scripts invoke other shells In-Reply-To: References: Message-ID: You can also check out the ActiveHTTP module to see how the Exec framework uses curl to make http connections from Bro. -AK On Nov 25, 2014 2:50 PM, "Mike Sconzo" wrote: > Here's a sample script (for better or worse) that I wrote a while back > that looks up files in Virustotal. > > https://github.com/sooshie/bro-scripts/blob/master/2.2-scripts/vt_check.bro > > > > On Tue, Nov 25, 2014 at 1:55 PM, Hosom, Stephen M > wrote: > > I think you?re looking for exec. > > > > > > > > https://www.bro.org/sphinx/scripts/base/utils/exec.bro.html > > > > > > > > > > > > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of John > > Donnelly > > Sent: Tuesday, November 25, 2014 2:35 PM > > To: bro at bro.org > > Subject: [Bro] Can bro scripts invoke other shells > > > > > > > > Hi, > > > > Can a bro script run an external (linux ) command from an event handler > ? > > > > > > > > > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -- > cat ~/.bash_history > documentation.txt > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141125/00fb90a8/attachment.html From hckim at narusec.com Thu Nov 27 21:41:40 2014 From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=) Date: Fri, 28 Nov 2014 14:41:40 +0900 Subject: [Bro] how to get manager.mbps.csv Message-ID: Hi all I am running bro 2.3.1 with bro cron enabled + pfring + broctl bro is running fine. in the logs/stats/www/ directory I get 591144 Nov 28 14:30 manager.cpu.csv 15 Jul 18 11:07 manager.in.csv 15 Jul 18 11:07 manager.mbps.csv 856410 Nov 28 14:30 manager.mem.csv 24 Jul 18 11:07 manager.pkts.csv these files I only get data for manager.cpu.csv and manager.mem.csv. rest of other files has only header if I want to populate rest of files, do I have to setup something? Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141128/f0d27e8b/attachment.html From dnthayer at illinois.edu Fri Nov 28 11:23:07 2014 From: dnthayer at illinois.edu (Daniel Thayer) Date: Fri, 28 Nov 2014 13:23:07 -0600 Subject: [Bro] how to get manager.mbps.csv In-Reply-To: References: Message-ID: <5478CB9B.7090006@illinois.edu> This is expected behavior. Those 3 files will always be empty (except for the header), because the manager does not monitor the network (only the workers do that). I should probably change the code so that it creates those files only when it has some data to write. On 11/27/2014 11:41 PM, ??? wrote: > Hi all > > I am running bro 2.3.1 with bro cron enabled + pfring + broctl > bro is running fine. > > in the logs/stats/www/ directory I get > > 591144 Nov 28 14:30 manager.cpu.csv > > 15 Jul 18 11:07 manager.in.csv > > 15 Jul 18 11:07 manager.mbps.csv > > 856410 Nov 28 14:30 manager.mem.csv > > 24 Jul 18 11:07 manager.pkts.csv > > these files > > > I only get data for manager.cpu.csv and manager.mem.csv. > > rest of other files has only header > > if I want to populate rest of files, do I have to setup something? > > > Thank you