From jan.grashofer at cern.ch Tue Sep 1 02:03:44 2015 From: jan.grashofer at cern.ch (Jan Grashofer) Date: Tue, 1 Sep 2015 09:03:44 +0000 Subject: [Bro] Email Notice Suppression In-Reply-To: <55E4E706.7090403@gmail.com> References: <55E4E706.7090403@gmail.com> Message-ID: Hi Scotty, have a look at automated suppression and the Notice::policy hook (https://www.bro.org/sphinx-git/frameworks/notice.html#automated-suppression and https://www.bro.org/sphinx-git/frameworks/notice.html#extending-notice-framework). If you use the do_notice script that comes with Bro, you want to add an identifier to the notice, to get automated suppression. Best regards, Jan ________________________________________ From: bro-bounces at bro.org [bro-bounces at bro.org] on behalf of Scotty Brown [scotty.b.brown at gmail.com] Sent: Tuesday, September 01, 2015 01:45 To: bro at bro.org Subject: [Bro] Email Notice Suppression Hi All, I'm running bro 2.4 and have just added a bunch of critical stack intel feeds. All is working well. One of the feeds I have is a list of TOR ips, and once I set notices to true for the critical stack intel I start getting emails (I've set up email alerting for notices). What I would like to do is suppress email alerts for a particular notice from a particular src host. ie (intel.log): 1441063489.889373 CEyDP6zbg6ngOFFa 10.10.10.10 45969 213.163.70.234 443 - - - 213.163.70.234 Intel::ADDR Conn::IN_RESP sensor-eth1-1 from https://www.dan.me.uk/torlist/ via intel.criticalstack.com So any notice that fires from src 10.10.10.10 for the torlist intel - I'd still like to see the notice in the intel file - but not get the email alert. Any pointers? Cheers, Scotty _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jxbatchelor at gmail.com Tue Sep 1 06:40:27 2015 From: jxbatchelor at gmail.com (Jason Batchelor) Date: Tue, 1 Sep 2015 08:40:27 -0500 Subject: [Bro] using bro for file extraction In-Reply-To: References: Message-ID: Hello Earl: Are you attempting to do post processing on the file after it is fully extracted with Bro via a third party script? If so, you may want to tap into the file_state_remove event. I have an example of what this looks like here if you scroll to the bottom. https://github.com/EmersonElectricCo/fsf/blob/master/docs/MODULES.md Hope that helps, Jason On Mon, Aug 31, 2015 at 2:17 PM, Earl Eiland wrote: > I want to use bro to extract files for external analysis. > Bro::FileDataEvent appears to be the proper approach. However, I?m not > finding the event to write a script for, nor do I know how to write to > anything other than a log file. > > > > Please advise! > > > > Best Regards, > > > > Earl Eiland, > > Sr. Cyber Security Engineer, > > Emerging Technologies, root9B, > > San Antonio, Texas > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150901/f664495a/attachment-0001.html From hosom at battelle.org Tue Sep 1 07:17:31 2015 From: hosom at battelle.org (Hosom, Stephen M) Date: Tue, 1 Sep 2015 14:17:31 +0000 Subject: [Bro] using bro for file extraction In-Reply-To: References: Message-ID: I have examples of this at: https://github.com/hosom/bro-file-extraction The plugins directory has examples of running external scripts on the extracted files. Check out the ones that store files by their hash names. From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Jason Batchelor Sent: Tuesday, September 01, 2015 9:40 AM To: Earl Eiland; bro at bro.org Subject: Re: [Bro] using bro for file extraction Hello Earl: Are you attempting to do post processing on the file after it is fully extracted with Bro via a third party script? If so, you may want to tap into the file_state_remove event. I have an example of what this looks like here if you scroll to the bottom. https://github.com/EmersonElectricCo/fsf/blob/master/docs/MODULES.md Hope that helps, Jason On Mon, Aug 31, 2015 at 2:17 PM, Earl Eiland > wrote: I want to use bro to extract files for external analysis. Bro::FileDataEvent appears to be the proper approach. However, I?m not finding the event to write a script for, nor do I know how to write to anything other than a log file. Please advise! Best Regards, Earl Eiland, Sr. Cyber Security Engineer, Emerging Technologies, root9B, San Antonio, Texas _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150901/b03a06b8/attachment.html From earl.eiland at root9b.com Wed Sep 2 06:37:28 2015 From: earl.eiland at root9b.com (Earl Eiland) Date: Wed, 2 Sep 2015 13:37:28 +0000 Subject: [Bro] using bro for file extraction In-Reply-To: References: Message-ID: Hello, Stephen. Your code will work with minimal tweaking. Thanks! Earl From: Hosom, Stephen M [mailto:hosom at battelle.org] Sent: Tuesday, September 1, 2015 9:18 AM To: Jason Batchelor ; Earl Eiland ; bro at bro.org Subject: RE: [Bro] using bro for file extraction I have examples of this at: https://github.com/hosom/bro-file-extraction The plugins directory has examples of running external scripts on the extracted files. Check out the ones that store files by their hash names. From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Jason Batchelor Sent: Tuesday, September 01, 2015 9:40 AM To: Earl Eiland; bro at bro.org Subject: Re: [Bro] using bro for file extraction Hello Earl: Are you attempting to do post processing on the file after it is fully extracted with Bro via a third party script? If so, you may want to tap into the file_state_remove event. I have an example of what this looks like here if you scroll to the bottom. https://github.com/EmersonElectricCo/fsf/blob/master/docs/MODULES.md Hope that helps, Jason On Mon, Aug 31, 2015 at 2:17 PM, Earl Eiland > wrote: I want to use bro to extract files for external analysis. Bro::FileDataEvent appears to be the proper approach. However, I?m not finding the event to write a script for, nor do I know how to write to anything other than a log file. Please advise! Best Regards, Earl Eiland, Sr. Cyber Security Engineer, Emerging Technologies, root9B, San Antonio, Texas _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150902/9d425ff7/attachment.html From louieamone at gmail.com Thu Sep 3 08:15:33 2015 From: louieamone at gmail.com (vinnie) Date: Thu, 3 Sep 2015 10:15:33 -0500 Subject: [Bro] Raw (eml) Email Extraction Bro 2.4 Message-ID: Hi Everyone, I would like to do full email extraction (eml) to file from STMP traffic; should this happen naturally with the new file extraction framework? I found this exchange from a while back, but haven't found anything more recent on the topic: http://mailman.icsi.berkeley.edu/pipermail/bro/2014-July/007224.html I'm currently using Bro 2.4 and a script pretty similar to this one for file extraction: https://github.com/Security-Onion-Solutions/securityonion-bro-scripts/blob/master/file-extraction/extract.bro It looks like I'm getting the message content and attachments, but apparently not the raw email. Thanks very much! -- Vinnie -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150903/baecc725/attachment.html From vgarramone at gmail.com Thu Sep 3 09:39:23 2015 From: vgarramone at gmail.com (V. Garramone) Date: Thu, 3 Sep 2015 11:39:23 -0500 Subject: [Bro] Raw (eml) Email Extraction Bro 2.4 Message-ID: Hi Everyone, I would like to do full email extraction (eml) to file from STMP traffic; should this happen naturally with the new file extraction framework? I found this exchange from a while back, but haven't found anything more recent on the topic: http://mailman.icsi.berkeley.edu/pipermail/bro/2014-July/007224.html I'm currently using Bro 2.4 and a script pretty similar to this one for file extraction: https://github.com/Security-Onion-Solutions/securityonion-bro-scripts/blob/master/file-extraction/extract.bro It looks like I'm getting the message content and attachments, but apparently not the raw email. Any tips would be greatly appreciated! Thanks very much, VG -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150903/9f311fbc/attachment.html From sven at dreyer-net.de Thu Sep 3 15:02:15 2015 From: sven at dreyer-net.de (Sven Dreyer) Date: Fri, 4 Sep 2015 00:02:15 +0200 Subject: [Bro] TCP retransmissions Message-ID: <55E8C367.5090007@dreyer-net.de> Dear list, I stumbled upon a few entries in conn.log that tells me there is an incoming connection from an IMAP mailserver (public IP) to my notebook computer (private IP, behind NAT). In fact, I only have outgoing connections from that notebook computer to the IMAP server. I can find these in conn.log as well. Of course I do not have any port forwarding to that notebook computer, so I took a tshark trace on the router and waited for another occurance. According to tshark on the router, there was no incoming connection from the IMAP server. But tshark on the router also revealed some TCP retransmissions from the IMAP server to my notebook. Every time tshark sees one of there TCP retransmissions, I get an incoming connections in conn.log. I think the retransmissions are due to a weak Wifi signal between router and notebook. Is it possible that TCP retransmissions are classified as new connections by bro? Or does anybody have a hint where else to search for the reason? Thanks! Sven From anthony.kasza at gmail.com Thu Sep 3 15:08:22 2015 From: anthony.kasza at gmail.com (anthony kasza) Date: Thu, 3 Sep 2015 15:08:22 -0700 Subject: [Bro] TCP retransmissions In-Reply-To: <55E8C367.5090007@dreyer-net.de> References: <55E8C367.5090007@dreyer-net.de> Message-ID: They might be considered new connections if your router and laptop have a longer connection timeout than Bro. This is a guess. -AK On Sep 3, 2015 3:04 PM, "Sven Dreyer" wrote: > Dear list, > > I stumbled upon a few entries in conn.log that tells me there is an > incoming connection from an IMAP mailserver (public IP) to my notebook > computer (private IP, behind NAT). > > In fact, I only have outgoing connections from that notebook computer to > the IMAP server. I can find these in conn.log as well. > > Of course I do not have any port forwarding to that notebook computer, > so I took a tshark trace on the router and waited for another occurance. > > According to tshark on the router, there was no incoming connection from > the IMAP server. > > But tshark on the router also revealed some TCP retransmissions from the > IMAP server to my notebook. Every time tshark sees one of there TCP > retransmissions, I get an incoming connections in conn.log. I think the > retransmissions are due to a weak Wifi signal between router and notebook. > > Is it possible that TCP retransmissions are classified as new > connections by bro? Or does anybody have a hint where else to search for > the reason? > > Thanks! > Sven > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150903/1f8ee5e8/attachment.html From sven at dreyer-net.de Thu Sep 3 15:08:56 2015 From: sven at dreyer-net.de (Sven Dreyer) Date: Fri, 4 Sep 2015 00:08:56 +0200 Subject: [Bro] long SSH connection in conn.log Message-ID: <55E8C4F8.3080009@dreyer-net.de> Dear list, I started an SSH connection in my LAN on 3:32pm which lasted until 07:04pm - so we're talking about an SSH session lasting 3 1/2 hours. In my conn.log files, I find this single SSH connection as 5 connections: 1) conn_state S1, service ssh 2-4) conn_state OTH, service - 5) conn_state SF, service - Bro was started before the SSH connection was initiated, so I'd expect a single conn.log entry to be written when I disconnect. Or did I get something wrong here? Thanks! Sven From anthony.kasza at gmail.com Thu Sep 3 19:41:38 2015 From: anthony.kasza at gmail.com (anthony kasza) Date: Thu, 3 Sep 2015 19:41:38 -0700 Subject: [Bro] Broadcast detection In-Reply-To: <20150827150321.GA2780@icir.org> References: <20150825174016.C5AE22C4063@rock.ICSI.Berkeley.EDU> <20150827150321.GA2780@icir.org> Message-ID: I believe I have some logic that solves this. I created an xor (^) operator for IPAddr types similar to the inclusive or (|) and am making use of it to calculate the broadcast address of a subnet. My BiF follows: function subnet_end%(snet: subnet%): addr %{ IPAddr broadcast; if (snet->Prefix().GetFamily() == IPv4) //ipv4 { broadcast = (IPAddr(string("255.255.255.255")) ^ snet->Mask()) | snet->Prefix(); } else if (snet->Prefix().GetFamily() == IPv6) //ipv6 { broadcast = (IPAddr(string("ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff")) ^ snet->Mask()) | snet->Prefix(); } else { reporter->InternalError("Unsupported address size. Not IPv4 or IPv6."); } return new AddrVal(broadcast); %} When calling this from scriptland, v6 addresses work properly. However, v4 addresses are represented as v6 addresses still. I am missing some concept around how IPAddrs can be either v4 or v6 and how scriptland knows the difference. How might I properly indicate the IPAddr in the returned AddrVal is meant to represent a v4 address instead of a v6 address? Thanks! -AK On Thu, Aug 27, 2015 at 8:03 AM, Robin Sommer wrote: > > > On Wed, Aug 26, 2015 at 18:12 -0700, anthony kasza wrote: > >> I'm looking to write a bif which does this. How can I access a subnet's >> prefix as an int? > > snet->Prefix() yields an IPAddr. You don't easily get that as an int, > but it has a method for getting it as a sequence of bytes: > > int GetBytes(const uint32_t** bytes) > > That works for both IPv4 and v6. > > That said, I think you can solve this more easily by combining some > other methods that IPAddr offers as well: > > /** > * Masks out lower bits of the address. > [...] > */ > void Mask(int top_bits_to_keep); > > /** > * Masks out top bits of the address. > [...] > */ > void ReverseMask(int top_bits_to_chop); > > /** > * Bitwise OR operator returns the IP address resulting from the bitwise > * OR operation on the raw bytes of this address with another. > */ > IPAddr operator|(const IPAddr& other) > > You'd mask out the lower bits of the prefix, mask out the upper bits > of 255.255.255.255 (for IPv4), and then "or" the two together. > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From dnthayer at illinois.edu Thu Sep 3 23:39:04 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Fri, 4 Sep 2015 01:39:04 -0500 Subject: [Bro] Broadcast detection In-Reply-To: References: <20150825174016.C5AE22C4063@rock.ICSI.Berkeley.EDU> <20150827150321.GA2780@icir.org> Message-ID: <55E93C88.9040201@illinois.edu> Here is a simpler implementation of this function (no other changes to Bro are needed): function subnet_end%(s: subnet%): addr %{ IPAddr mask; int offset = 0; if ( s->Prefix().GetFamily() == IPv4 ) { mask = IPAddr("255.255.255.255"); offset = 96; } else { mask = IPAddr("ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"); } mask.ReverseMask(offset + s->Width()); return new AddrVal(mask | s->Prefix()); %} On 09/03/2015 09:41 PM, anthony kasza wrote: > I believe I have some logic that solves this. I created an xor (^) > operator for IPAddr types similar to the inclusive or (|) and am > making use of it to calculate the broadcast address of a subnet. My > BiF follows: > > function subnet_end%(snet: subnet%): addr > %{ > IPAddr broadcast; > > if (snet->Prefix().GetFamily() == IPv4) //ipv4 > { > broadcast = (IPAddr(string("255.255.255.255")) ^ > snet->Mask()) | snet->Prefix(); > } > else if (snet->Prefix().GetFamily() == IPv6) //ipv6 > { > broadcast = > (IPAddr(string("ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff")) ^ > snet->Mask()) | snet->Prefix(); > } > else > { > reporter->InternalError("Unsupported address size. Not > IPv4 or IPv6."); > } > > return new AddrVal(broadcast); > %} > > > When calling this from scriptland, v6 addresses work properly. > However, v4 addresses are represented as v6 addresses still. I am > missing some concept around how IPAddrs can be either v4 or v6 and how > scriptland knows the difference. How might I properly indicate the > IPAddr in the returned AddrVal is meant to represent a v4 address > instead of a v6 address? > Thanks! > > -AK > > On Thu, Aug 27, 2015 at 8:03 AM, Robin Sommer wrote: >> >> >> On Wed, Aug 26, 2015 at 18:12 -0700, anthony kasza wrote: >> >>> I'm looking to write a bif which does this. How can I access a subnet's >>> prefix as an int? >> >> snet->Prefix() yields an IPAddr. You don't easily get that as an int, >> but it has a method for getting it as a sequence of bytes: >> >> int GetBytes(const uint32_t** bytes) >> >> That works for both IPv4 and v6. >> >> That said, I think you can solve this more easily by combining some >> other methods that IPAddr offers as well: >> >> /** >> * Masks out lower bits of the address. >> [...] >> */ >> void Mask(int top_bits_to_keep); >> >> /** >> * Masks out top bits of the address. >> [...] >> */ >> void ReverseMask(int top_bits_to_chop); >> >> /** >> * Bitwise OR operator returns the IP address resulting from the bitwise >> * OR operation on the raw bytes of this address with another. >> */ >> IPAddr operator|(const IPAddr& other) >> >> You'd mask out the lower bits of the prefix, mask out the upper bits >> of 255.255.255.255 (for IPv4), and then "or" the two together. >> >> Robin >> >> -- >> Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From chenhj at cn.fujitsu.com Fri Sep 4 00:14:31 2015 From: chenhj at cn.fujitsu.com (Chen, Huajun) Date: Fri, 4 Sep 2015 07:14:31 +0000 Subject: [Bro] =?gb2312?b?y806IFtQQVRDSF1idWdmaXggZm9yIGJpbnBhYyBmYWlsIHRv?= =?gb2312?b?IHByb2Nlc3Mgc3RyaW5nIGNvbnN0?= Message-ID: <17057F4300007C45B01591F9A01B1FAFA8A4662E@G08CNEXMBPEKD01.g08.fujitsu.local> Hi,all I found a bug in binpac and made a patch for it. Binpac generated c++ source will lost string consts used in pac source. For example: type My_Message = record { d1:uint8; }&let { d2 = $context.flow.foo("test"); } will become the following in generated c++ source: d2_ = t_context->flow()->foo(); The patch in attachment can solve this problem. -- Best Regards, Chen Huajun -------------- next part -------------- A non-text attachment was scrubbed... Name: binpac_cstr_bugfix.patch Type: application/octet-stream Size: 433 bytes Desc: binpac_cstr_bugfix.patch Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150904/d09a851a/attachment.obj From chenhj at cn.fujitsu.com Fri Sep 4 00:15:07 2015 From: chenhj at cn.fujitsu.com (Chen, Huajun) Date: Fri, 4 Sep 2015 07:15:07 +0000 Subject: [Bro] Two strange codes in RuleMatcher.cc Message-ID: <17057F4300007C45B01591F9A01B1FAFA8A46640@G08CNEXMBPEKD01.g08.fujitsu.local> Hi,all I found two strange codes in RuleMatcher.cc, they seem have problem. Problem 1 ----------------------------------------------------- void RuleMatcher::InsertRuleIntoTree(Rule* r, int testnr, RuleHdrTest* dest, int level) { // Initiliaze the preconditions loop_over_list(r->preconds, i)// The loop should be called for per rule(not for per rule's RuleHdrTest). { Rule::Precond* pc = r->preconds[i]; Rule* pc_rule = rules_by_id.Lookup(pc->id); if ( ! pc_rule ) { rules_error(r, "unknown rule referenced"); return; } pc->rule = pc_rule; pc_rule->dependents.append(r); } ... } ----------------------------------------------------- The same dependented rule should only be insert into rule.dependents once, so rule.dependents.append() should be called for per rule (not for per rule's RuleHdrTest). And in my test,it really insert the same dependented rule into rule.dependents many times. Problem 2 ----------------------------------------------------- void RuleMatcher::Match(RuleEndpointState* state, Rule::PatternType type, const u_char* data, int data_len, bool bol, bool eol, bool clear) { ... // Check which of the matching rules really belong to any of our nodes. for ( set::const_iterator it = rule_matches.begin(); it != rule_matches.end(); ++it ) { Rule* r = *it; DBG_LOG(DBG_RULES, "Accepted rule: %s", r->id); loop_over_list(state->hdr_tests, k)// the loop for every hdr_test seems useless. { RuleHdrTest* h = state->hdr_tests[k]; DBG_LOG(DBG_RULES, "Checking for accepted rule on HdrTest %d", h->id); // Skip if rule does not belong to this node. if ( ! h->ruleset->Contains(r->Index()) ) continue; DBG_LOG(DBG_RULES, "On current node"); ... } ----------------------------------------------------- I think there must be one and only one HdrTest nodes contains the rule, the loop for every hdr_test seems useless. The attachment is a patch for them,but may be my judge is wrong. -- Best Regards, Chen Huajun -------------- next part -------------- A non-text attachment was scrubbed... Name: bro_strange_code.patch Type: application/octet-stream Size: 2333 bytes Desc: bro_strange_code.patch Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150904/848aef53/attachment-0001.obj From benson.mathews at gmail.com Fri Sep 4 06:33:46 2015 From: benson.mathews at gmail.com (Benson Mathews) Date: Fri, 4 Sep 2015 09:33:46 -0400 Subject: [Bro] bro script looking for hacker keywords Message-ID: Hi, I'm trying to write a bro script that would alert me whenever certain hacker keywords are seen in the http traffic. I found a bro script that captures the POST content and modified it a bit to check for the keywords. module HTTP; export { ## The number of bytes that will be included in the http ## log from the client body. const post_body_limit = 1024; redef record Info += { post_body: string &log &optional; }; redef enum Notice::Type += { Hack_keyword_match }; } event http_entity_data(c: connection, is_orig: bool, length: count, data: string) { if ( is_orig && Site::is_local_addr(c$id$resp_h) && !(Site::is_local_addr(c$id$orig_h)) ) { if (/******KEYWORDS TO MATCH******/ in data ) { NOTICE([$note=Hack_keyword_match, $msg=fmt("%s maybe attempting to access/upload hack file on %s. data: %s", c$id$orig_h,c$id$resp_h , data), $src=c$id$orig_h, $sub="Hack keyword match", $identifier=c$uid]); if ( ! c$http?$post_body ) c$http$post_body = sub_bytes(data, 0, post_body_limit); else if ( |c$http$post_body| < post_body_limit ) c$http$post_body = string_cat(c$http$post_body, sub_bytes(data, 0, post_body_limit-|c$http$post_body|)); } } } I do see some positive alerts when hackers try to bruteforce a login with passwords that match the keywords list, but I'm getting some false positives when the http response is gzip encoded. Is there a function that would decode the data, or another event I could use to achieve this... Thanks, Benson -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150904/03792c3d/attachment.html From jdopheid at illinois.edu Fri Sep 4 07:25:32 2015 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Fri, 4 Sep 2015 14:25:32 +0000 Subject: [Bro] BroCon '15 videos are posted Message-ID: Bro Community, The BroCon videos are posted! You can find the playlist here: https://goo.gl/ETuJxs Thanks for your enthusiasm and support, The Bro Team ------ Jeannette Dopheide Bro Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From anthony.kasza at gmail.com Fri Sep 4 08:00:13 2015 From: anthony.kasza at gmail.com (anthony kasza) Date: Fri, 4 Sep 2015 08:00:13 -0700 Subject: [Bro] Broadcast detection In-Reply-To: <55E93C88.9040201@illinois.edu> References: <20150825174016.C5AE22C4063@rock.ICSI.Berkeley.EDU> <20150827150321.GA2780@icir.org> <55E93C88.9040201@illinois.edu> Message-ID: That's much cleaner and doesn't have the v4/v6 issue I introduced. Thanks Daniel. I understand now why Robin suggested using those methods. -AK On Sep 3, 2015 11:38 PM, "Daniel Thayer" wrote: > Here is a simpler implementation of this function (no other > changes to Bro are needed): > > function subnet_end%(s: subnet%): addr > %{ > IPAddr mask; > int offset = 0; > > if ( s->Prefix().GetFamily() == IPv4 ) > { > mask = IPAddr("255.255.255.255"); > offset = 96; > } > else > { > mask = IPAddr("ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"); > } > > mask.ReverseMask(offset + s->Width()); > > return new AddrVal(mask | s->Prefix()); > %} > > > On 09/03/2015 09:41 PM, anthony kasza wrote: > >> I believe I have some logic that solves this. I created an xor (^) >> operator for IPAddr types similar to the inclusive or (|) and am >> making use of it to calculate the broadcast address of a subnet. My >> BiF follows: >> >> function subnet_end%(snet: subnet%): addr >> %{ >> IPAddr broadcast; >> >> if (snet->Prefix().GetFamily() == IPv4) //ipv4 >> { >> broadcast = (IPAddr(string("255.255.255.255")) ^ >> snet->Mask()) | snet->Prefix(); >> } >> else if (snet->Prefix().GetFamily() == IPv6) //ipv6 >> { >> broadcast = >> (IPAddr(string("ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff")) ^ >> snet->Mask()) | snet->Prefix(); >> } >> else >> { >> reporter->InternalError("Unsupported address size. Not >> IPv4 or IPv6."); >> } >> >> return new AddrVal(broadcast); >> %} >> >> >> When calling this from scriptland, v6 addresses work properly. >> However, v4 addresses are represented as v6 addresses still. I am >> missing some concept around how IPAddrs can be either v4 or v6 and how >> scriptland knows the difference. How might I properly indicate the >> IPAddr in the returned AddrVal is meant to represent a v4 address >> instead of a v6 address? >> Thanks! >> >> -AK >> >> On Thu, Aug 27, 2015 at 8:03 AM, Robin Sommer wrote: >> >>> >>> >>> On Wed, Aug 26, 2015 at 18:12 -0700, anthony kasza wrote: >>> >>> I'm looking to write a bif which does this. How can I access a subnet's >>>> prefix as an int? >>>> >>> >>> snet->Prefix() yields an IPAddr. You don't easily get that as an int, >>> but it has a method for getting it as a sequence of bytes: >>> >>> int GetBytes(const uint32_t** bytes) >>> >>> That works for both IPv4 and v6. >>> >>> That said, I think you can solve this more easily by combining some >>> other methods that IPAddr offers as well: >>> >>> /** >>> * Masks out lower bits of the address. >>> [...] >>> */ >>> void Mask(int top_bits_to_keep); >>> >>> /** >>> * Masks out top bits of the address. >>> [...] >>> */ >>> void ReverseMask(int top_bits_to_chop); >>> >>> /** >>> * Bitwise OR operator returns the IP address resulting from >>> the bitwise >>> * OR operation on the raw bytes of this address with another. >>> */ >>> IPAddr operator|(const IPAddr& other) >>> >>> You'd mask out the lower bits of the prefix, mask out the upper bits >>> of 255.255.255.255 (for IPv4), and then "or" the two together. >>> >>> Robin >>> >>> -- >>> Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin >>> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150904/c22706f0/attachment.html From earl.eiland at root9b.com Fri Sep 4 12:04:48 2015 From: earl.eiland at root9b.com (Earl Eiland) Date: Fri, 4 Sep 2015 19:04:48 +0000 Subject: [Bro] can't find extracted files Message-ID: Hello. The bro exercises/faf Part 3 references the extract_files/ subdirectory. I've run the sample code, and examined files.log. Everything seems to be running correctly. However, there appears to be no extract_files directory. Where are the extracted files found? Best Regards, Earl Eiland, Sr. Cyber Security Engineer, Emerging Technologies, root9B, San Antonio, Texas -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150904/d552201c/attachment.html From pachinko.tw at gmail.com Fri Sep 4 22:50:08 2015 From: pachinko.tw at gmail.com (Po-Ching Lin) Date: Sat, 5 Sep 2015 13:50:08 +0800 Subject: [Bro] Client identification from bro logs Message-ID: <55EA8290.2010809@gmail.com> Hi all, It is well known that a client may be behind NAT or using DHCP, so identifying an individual client solely from the IP address is unreliable. To track a client's behavior from Bro logs, it is therefore important to separate the clients behind NAT or using DHCP. Some passive methods for client identification were presented long ago, such as https://www.cs.columbia.edu/~smb/papers/fnat.pdf, or http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1453529&tag=1 The features leveraged by the above two papers, IP identifier and TCP timestamp option, are unavailable from default Bro logs. I would like to know whether the existing Bro design has a solution to this issue. Many thanks. Po-Ching From johanna at icir.org Tue Sep 8 09:34:44 2015 From: johanna at icir.org (Johanna Amann) Date: Tue, 8 Sep 2015 09:34:44 -0700 Subject: [Bro] Bro 2.4.1 release Message-ID: <20150908163444.GA20386@Beezling.local> Bro 2.4.1 has been released. This release addresses a few potential DOS vectors using specially crafted connections. The release also contains minor updates to analyzers to reduce the number of messages in reporter.log. The source distribution is available on the download page at https://www.bro.org/download/index.html. Our binary packages will be updated later today - users should be able to automatically update the package using their system package manager. See https://www.bro.org/download/CHANGES.bro.txt for the full list of changes in the release. Since this is only a bug fix release, we encourage users to update at their earliest convenience. The Bro Team From cdaviso1 at vols.utk.edu Tue Sep 8 13:33:40 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Tue, 8 Sep 2015 20:33:40 +0000 Subject: [Bro] Deployment Customization Questions Message-ID: Good Afternoon, I have made it so far to Browsing Log Files in Bro via the documentation provided under the Quick Start Guide and Managing Bro with BroControl. I am now in the Redefining Script Option Variables section and am trying to understand the documentation regarding adding the statement to local.bro: [cid:05df7d6f-9edd-47ef-bd5a-55832eb738ee] I browsed out to the below location and tried to insert the above text but don't know where to insert it. Can this be anywhere in the document? Also, how would you know to look into main.bro at the module notice and derive what needs to be added to local.bro, in this example or others? [https://confluence.charter.com/download/attachments/71827976/image2015-9-8%2013%3A57%3A33.png?version=1&modificationDate=1441742253458&api=v2] Thank you, Charles -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150908/0213ebcb/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: pastedImage.png Type: image/png Size: 63856 bytes Desc: pastedImage.png Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150908/0213ebcb/attachment-0001.bin From cdaviso1 at vols.utk.edu Tue Sep 8 13:41:23 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Tue, 8 Sep 2015 20:41:23 +0000 Subject: [Bro] Deployment Customization Questions In-Reply-To: References: Message-ID: Good Afternoon, I have made it so far to Browsing Log Files in Bro via the documentation provided under the Quick Start Guide and Managing Bro with BroControl. I am now in the Redefining Script Option Variables section and am trying to understand the documentation regarding adding the statement to local.bro: redef Notice::ignored_types += { SSL::Invalid_Server_Cert }; I browsed out to the below location and tried to insert the above text but don't know where to insert it. Can this be anywhere in the document? Also, how would you know to look into main.bro at the module notice and derive what needs to be added to local.bro, in this example or others? /usr/local/bro/share/bro/site Thank you, Charles -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150908/281b873c/attachment.html From dnthayer at illinois.edu Tue Sep 8 14:08:40 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Tue, 8 Sep 2015 16:08:40 -0500 Subject: [Bro] Deployment Customization Questions In-Reply-To: References: Message-ID: <55EF4E58.1020707@illinois.edu> On 09/08/2015 03:41 PM, Davison, Charles Robert wrote: > Good Afternoon, > > I have made it so far to Browsing Log Files in Bro via the documentation > provided under the Quick Start Guide and Managing Bro with BroControl > . > I am now in the *R***edefining *Script Option Variables *section and am > trying to understand the documentation regarding adding the statement to > local.bro: > > redef Notice::ignored_types += { SSL::Invalid_Server_Cert }; > > I browsed out to the below location and tried to insert the above text > but don't know where to insert it. Can this be anywhere in the > document? Also, how would you know to look into main.bro at the module > notice and derive what needs to be added to local.bro, in this example > or others? > The Quick Start guide is intended to give a (very) quick tour of some things you can do with Bro, so don't worry if you don't understand why the examples are written the way they are. There is more complete documentation in the "Reference" section. For example, the "Notice Framework" document gives more background information that will help you understand the example in the Quick Start guide. For documentation about specific Bro scripts, the "Bro Package Index" or "Bro Script Index" pages are quite useful (these are listed on the table of contents of the Bro Manual). From franky.meier.1 at gmx.de Wed Sep 9 07:04:34 2015 From: franky.meier.1 at gmx.de (Frank Meier) Date: Wed, 09 Sep 2015 16:04:34 +0200 Subject: [Bro] how to merge rx and tx from different pcaps / slightly off-topic Message-ID: <1441807474.21786.0@mail.gmx.net> Hi! Sorry if this is off-topic, but I hope to find the right audience here. I want to create bro-logs of around 900 Gb of data in 20.000 pcaps. Capturing was done on different interfaces for upstream and downstream (rx/tx). Because of the large number of files I cannot merge them in one step ("to many open files"), so I merged them to one pcap per day with mergecap. After that Bro is called like this: # mergecap -F pcap -w - *.pcap | bro -r - foo.bro foo.bro reads: redef bits_per_uids = 128; redef ignore_checksums = T; redef Log::default_rotation_interval = 1day; No real service logs are written, except for a weird.log full of: connection_originator_SYN_ack data_after_reset data_before_established inappropriate_FIN possible_split_routing simultaneous_open SYN_after_close SYN_after_reset SYN_inside_connection SYN_seq_jump TCP_ack_underflow_or_misorder TCP_seq_underflow_or_misorder window_recision It looks like Bro not seeing the data in the correct order. But from what I read in mergecap source in merge_read_packet() this should work as intended: "Read the next packet, in chronological order, from the set of files to be merged." I am thankful for any ideas. Bye, Franky -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150909/21a045ce/attachment.html From carlopmart at gmail.com Wed Sep 9 07:46:11 2015 From: carlopmart at gmail.com (C.L. Martinez) Date: Wed, 09 Sep 2015 14:46:11 +0000 Subject: [Bro] Upgrading from 2.4 to 2.4.1 Message-ID: <55F04633.20307@gmail.com> Hi all, I have installed Bro IDS sensors some weeks ago in four hosts. Due to the new release 2.4.1, what is the correct procedure to upgrade them?? I have installed all of them via source package. Thanks From dnthayer at illinois.edu Wed Sep 9 08:02:39 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Wed, 9 Sep 2015 10:02:39 -0500 Subject: [Bro] Upgrading from 2.4 to 2.4.1 In-Reply-To: <55F04633.20307@gmail.com> References: <55F04633.20307@gmail.com> Message-ID: <55F04A0F.1010405@illinois.edu> In the Bro Manual (https://www.bro.org/sphinx/index.html), there is a section "Upgrading Bro" that should explain how to do this. On 09/09/2015 09:46 AM, C.L. Martinez wrote: > Hi all, > > I have installed Bro IDS sensors some weeks ago in four hosts. Due to > the new release 2.4.1, what is the correct procedure to upgrade them?? I > have installed all of them via source package. > > Thanks > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From carlopmart at gmail.com Wed Sep 9 08:24:58 2015 From: carlopmart at gmail.com (C.L. Martinez) Date: Wed, 09 Sep 2015 15:24:58 +0000 Subject: [Bro] Upgrading from 2.4 to 2.4.1 In-Reply-To: <55F04A0F.1010405@illinois.edu> References: <55F04633.20307@gmail.com> <55F04A0F.1010405@illinois.edu> Message-ID: <55F04F4A.20309@gmail.com> On 09/09/2015 03:02 PM, Daniel Thayer wrote: > In the Bro Manual (https://www.bro.org/sphinx/index.html), there is > a section "Upgrading Bro" that should explain how to do this. > > Oops ... Your are right. Sorry for the noise. From vallentin at icir.org Wed Sep 9 09:16:28 2015 From: vallentin at icir.org (Matthias Vallentin) Date: Wed, 9 Sep 2015 09:16:28 -0700 Subject: [Bro] how to merge rx and tx from different pcaps / slightly off-topic In-Reply-To: <1441807474.21786.0@mail.gmx.net> References: <1441807474.21786.0@mail.gmx.net> Message-ID: <20150909161628.GP759@shogun> > It looks like Bro not seeing the data in the correct order. But from what I > read in mergecap source in merge_read_packet() this should work as intended: > "Read the next packet, in chronological order, from the set of files to be > merged." You could give this a shot: ipsumdump --collate -r *.pcap -w merged.pcap Unlike mergecap, ipsumdump does not assume packets are sorted within the trace. Matthias From jbarber at computer.org Wed Sep 9 14:09:16 2015 From: jbarber at computer.org (Jeff Barber) Date: Wed, 9 Sep 2015 17:09:16 -0400 Subject: [Bro] how to merge rx and tx from different pcaps / slightly off-topic In-Reply-To: <1441807474.21786.0@mail.gmx.net> References: <1441807474.21786.0@mail.gmx.net> Message-ID: I ran into some problems trying to process pcaps. One is the checksums issue but I see you've already handled that. The other seems like it might possibly be related: If you don't specify --pseudo-realtime, BRO will apparently run connection timers based on the current wall clock time, comparing the wall clock with the start time recorded in conjunction with the packets in the pcap. This means it may see a connection start, then immediately expire it as having passed the session time limit. [What? That session is six months old!] (This seems fundamentally broken to me, but it's also quite likely that I didn't fully understand the code and/or that there's some good reason for it to work this way; in any case, the --pseudo-realtime switch seems to make it behave more sanely -- for this particular case anyway.) Cheers. On Wed, Sep 9, 2015 at 10:04 AM, Frank Meier wrote: > Hi! > > Sorry if this is off-topic, but I hope to find the right audience here. > > I want to create bro-logs of around 900 Gb of data in 20.000 pcaps. > Capturing was done on different interfaces for upstream and downstream > (rx/tx). > > Because of the large number of files I cannot merge them in one step ("to > many open files"), > so I merged them to one pcap per day with mergecap. After that Bro is > called like this: > # mergecap -F pcap -w - *.pcap | bro -r - foo.bro > > foo.bro reads: > > redef bits_per_uids = 128; > redef ignore_checksums = T; > redef Log::default_rotation_interval = 1day; > > No real service logs are written, except for a weird.log full of: > > connection_originator_SYN_ack > data_after_reset > data_before_established > inappropriate_FIN > possible_split_routing > simultaneous_open > SYN_after_close > SYN_after_reset > SYN_inside_connection > SYN_seq_jump > TCP_ack_underflow_or_misorder > TCP_seq_underflow_or_misorder > window_recision > > It looks like Bro not seeing the data in the correct order. But from what > I read in mergecap > source in merge_read_packet() this should work as intended: "Read the next > packet, > in chronological order, from the set of files to be merged." > > I am thankful for any ideas. > > Bye, > > Franky > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150909/ad20362b/attachment.html From seth at icir.org Wed Sep 9 18:55:40 2015 From: seth at icir.org (Seth Hall) Date: Wed, 9 Sep 2015 21:55:40 -0400 Subject: [Bro] how to merge rx and tx from different pcaps / slightly off-topic In-Reply-To: References: <1441807474.21786.0@mail.gmx.net> Message-ID: <2CC4FFFD-97AC-478C-9284-892414585B29@icir.org> > On Sep 9, 2015, at 5:09 PM, Jeff Barber wrote: > > If you don't specify --pseudo-realtime, BRO will apparently run connection timers based on the current wall clock time, comparing the wall clock with the start time recorded in conjunction with the packets in the pcap. This means it may see a connection start, then immediately expire it as having passed the session time limit. [What? That session is six months old!] That?s actually not how Bro works, it uses the timestamps in the packets to drive it?s packet clock forward. Could you show how you?re running Bro? It sounds to me like you?re replaying traffic to and interface and then sniffing it. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From franky.meier.1 at gmx.de Thu Sep 10 00:58:45 2015 From: franky.meier.1 at gmx.de (Frank Meier) Date: Thu, 10 Sep 2015 09:58:45 +0200 Subject: [Bro] how to merge rx and tx from different pcaps / slightly off-topic In-Reply-To: <20150909161628.GP759@shogun> References: <1441807474.21786.0@mail.gmx.net> <20150909161628.GP759@shogun> Message-ID: <1441871925.23915.0@mail.gmx.net> Hi, On Mi, Sep 9, 2015 at 6:16 , Matthias Vallentin wrote: >> It looks like Bro not seeing the data in the correct order. But >> from what I >> read in mergecap source in merge_read_packet() this should work as >> intended: >> "Read the next packet, in chronological order, from the set of >> files to be >> merged." > > You could give this a shot: > > ipsumdump --collate -r *.pcap -w merged.pcap > > Unlike mergecap, ipsumdump does not assume packets are sorted within > the > trace. > thanks, this is an idea, but with my first run of mergecap I made sure, the order is correct. (verfied with capinfos -o). Beside from that it looks better now: Only 3300 lines of weird.log with 115000 in conn.log. I will investigate further, if the data in the pcaps is wrong or if bro is to blame. Franky -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150910/879ec82d/attachment.html From jbarber at computer.org Thu Sep 10 04:34:43 2015 From: jbarber at computer.org (Jeff Barber) Date: Thu, 10 Sep 2015 07:34:43 -0400 Subject: [Bro] how to merge rx and tx from different pcaps / slightly off-topic In-Reply-To: <2CC4FFFD-97AC-478C-9284-892414585B29@icir.org> References: <1441807474.21786.0@mail.gmx.net> <2CC4FFFD-97AC-478C-9284-892414585B29@icir.org> Message-ID: Seth, Thanks for the clarification. Uggh... It appears that shady stuff my plugin is doing is responsible for my problem. I think the problem is that I have opened a live pkt src from within my plugin, but then also trying to read a pcap. Maybe I've seeded BRO with a later timestamp than those in the pcap? Having a hard time following the timer logic. Is it possible to instantiate a per-PktSrc timer? Anyway, sorry to be spewing misinformation. On Wed, Sep 9, 2015 at 9:55 PM, Seth Hall wrote: > > > On Sep 9, 2015, at 5:09 PM, Jeff Barber wrote: > > > > If you don't specify --pseudo-realtime, BRO will apparently run > connection timers based on the current wall clock time, comparing the wall > clock with the start time recorded in conjunction with the packets in the > pcap. This means it may see a connection start, then immediately expire it > as having passed the session time limit. [What? That session is six months > old!] > > That?s actually not how Bro works, it uses the timestamps in the packets > to drive it?s packet clock forward. Could you show how you?re running > Bro? It sounds to me like you?re replaying traffic to and interface and > then sniffing it. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150910/063a5282/attachment.html From seth at icir.org Thu Sep 10 09:37:55 2015 From: seth at icir.org (Seth Hall) Date: Thu, 10 Sep 2015 12:37:55 -0400 Subject: [Bro] how to merge rx and tx from different pcaps / slightly off-topic In-Reply-To: References: <1441807474.21786.0@mail.gmx.net> <2CC4FFFD-97AC-478C-9284-892414585B29@icir.org> Message-ID: <5499A998-A3D0-4471-8148-31F39AF315E0@icir.org> > On Sep 10, 2015, at 7:34 AM, Jeff Barber wrote: > > Uggh... It appears that shady stuff my plugin is doing is responsible for my problem. Is your plugin posted anywhere? > I think the problem is that I have opened a live pkt src from within my plugin, but then also trying to read a pcap. Maybe I've seeded BRO with a later timestamp than those in the pcap? Having a hard time following the timer logic. You?re doing both in your plugin? That definitely isn?t a supported model. > Is it possible to instantiate a per-PktSrc timer? I assume you mean a per-pktsrc clock? (since timers have a meaning and are something different in Bro). If you meant clock, then no, a Bro process has the notion of a singular clock. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From erik.hjelmvik at gmail.com Fri Sep 11 07:19:56 2015 From: erik.hjelmvik at gmail.com (Erik Hjelmvik) Date: Fri, 11 Sep 2015 16:19:56 +0200 Subject: [Bro] Missing protocol column (TCP/DUP) in weird.log Message-ID: Hi all, I was just about to automate some handling of Bro logs but ran into issues with weird.log since it has no protocol column. There is IP and port columns, so it would be nice to also know if it is TCP/UDP/SCTP etc. Is there any chance to get such an update into Bro? Best regards, Erik -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150911/22948ae0/attachment.html From nathanael.rayborn at gmail.com Fri Sep 11 11:38:42 2015 From: nathanael.rayborn at gmail.com (nathanael rayborn) Date: Fri, 11 Sep 2015 13:38:42 -0500 Subject: [Bro] PF_Ring and Bro - packet loss Message-ID: I'm experiencing high packet loss (15% -50%) with Bro 2.4 compiled with PF_Ring. PFcount (pfcount -i eth0 -e 1) shows 0% packet loss while /proc/net/pf_ring/PID shows the same number of dropped packets as broctl netstats. The github link contains all changes and performance steps I've taken so far along with output from PFcount, broctl, and ethtool. Has anyone else experienced similar performance issues or have recommendations to get my dropped packets as close to 0% as possible? Thanks Current config - https://gist.github.com/nate-ray/8b4d03eab49d11715398 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150911/272ff4ac/attachment.html From jwilliams at emergingthreats.net Fri Sep 11 12:07:24 2015 From: jwilliams at emergingthreats.net (Jason Williams) Date: Fri, 11 Sep 2015 14:07:24 -0500 Subject: [Bro] PF_Ring and Bro - packet loss In-Reply-To: References: Message-ID: Nathanael, Just from initial glance you may want to allocate more slots in pfring... Min Num Slots : 4096 Bucket Len : 8192 Slot Len : 8224 [bucket+header] Tot Memory : 33697792 Tot Packets : 153298629 Tot Pkt Lost : 60413245 Tot Insert : 92885384 Tot Read : 92829402 Insert Offset : 3522336 Remove Offset : 3537608 Num Free Slots : 0 <-------------- maybe modprobe pf_ring enable_tx_capture=0 min_num_slots=32768 On Fri, Sep 11, 2015 at 1:38 PM, nathanael rayborn < nathanael.rayborn at gmail.com> wrote: > I'm experiencing high packet loss (15% -50%) with Bro 2.4 compiled with > PF_Ring. PFcount (pfcount -i eth0 -e 1) shows 0% packet loss while > /proc/net/pf_ring/PID shows the same number of dropped packets as broctl > netstats. The github link contains all changes and performance steps I've > taken so far along with output from PFcount, broctl, and ethtool. Has > anyone else experienced similar performance issues or have recommendations > to get my dropped packets as close to 0% as possible? Thanks > > > Current config - https://gist.github.com/nate-ray/8b4d03eab49d11715398 > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150911/3bd0d964/attachment.html From apumphrey at ivsec.com Fri Sep 11 12:55:56 2015 From: apumphrey at ivsec.com (Adam Pumphrey) Date: Fri, 11 Sep 2015 15:55:56 -0400 Subject: [Bro] PF_Ring and Bro - packet loss In-Reply-To: References: Message-ID: Your broctl status output shows Bro is in standalone mode and not configured to take advantage of pf_ring. You'll need to configure a local cluster with the pf_ring specific options set for the monitoring interface/worker. https://www.bro.org/sphinx-git/configuration/index.html#using-pf-ring Something like this might work in your case; notice the lb_method and lb_procs settings for the worker: [manager] type=manager host=127.0.0.1 [proxy-1] type=proxy host=127.0.0.1 [worker-1] type=worker host=127.0.0.1 interface=eth0 lb_method=pf_ring lb_procs=2 I think you need at least 2 lb_procs for pf_ring to provide any performance gain. You can also set cpu affinity for the worker processes; this is recommended for better performance and cluster stability, but not required. If you do be sure to specify only physical cpu/core ID?s. I believe the general rule of thumb is to leave half of your physical cores freed up for the OS and other Bro processes. For example: pin_cpus=2,3 # assumes a single quad-core with core ID?s 0-3 There are a few more pf_ring specific options available in BroControl?s config file. You can check them out here https://www.bro.org/sphinx/components/broctl/README.html#pfringclusterid You might want to give ?broctl capstats? and the capture-loss Bro script a try also; both are really helpful with troubleshooting traffic capture issues. -Adam > On Sep 11, 2015, at 2:38 PM, nathanael rayborn wrote: > > I'm experiencing high packet loss (15% -50%) with Bro 2.4 compiled with PF_Ring. PFcount (pfcount -i eth0 -e 1) shows 0% packet loss while /proc/net/pf_ring/PID shows the same number of dropped packets as broctl netstats. The github link contains all changes and performance steps I've taken so far along with output from PFcount, broctl, and ethtool. Has anyone else experienced similar performance issues or have recommendations to get my dropped packets as close to 0% as possible? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150911/3f4d3691/attachment.html From ilasic at reversinglabs.com Fri Sep 11 13:02:01 2015 From: ilasic at reversinglabs.com (Igor Lasic) Date: Fri, 11 Sep 2015 16:02:01 -0400 Subject: [Bro] PF_Ring and Bro - packet loss Message-ID: I found the loss is very dependent on the NIC used and ETHTOOL flags used. We've found Intel NIC and settings found in below papers give us the best performance. http://dak1n1.com/blog/7-performance-tuning-intel-10gbe https://home.regit.org/2012/07/suricata-to-10gbps-and-beyond/ > 2. PF_Ring and Bro - packet loss (nathanael rayborn) > > > Message: 2 > Date: Fri, 11 Sep 2015 13:38:42 -0500 > From: nathanael rayborn > Subject: [Bro] PF_Ring and Bro - packet loss > To: bro at bro.org > Message-ID: > < > CAMKC1B16BSGSJcu_PAixFSV9+3Rz0+Tm68KwoB0GyygsPuYRcQ at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > I'm experiencing high packet loss (15% -50%) with Bro 2.4 compiled with > PF_Ring. PFcount (pfcount -i eth0 -e 1) shows 0% packet loss while > /proc/net/pf_ring/PID shows the same number of dropped packets as broctl > netstats. The github link contains all changes and performance steps I've > taken so far along with output from PFcount, broctl, and ethtool. Has > anyone else experienced similar performance issues or have recommendations > to get my dropped packets as close to 0% as possible? Thanks > > > Current config - https://gist.github.com/nate-ray/8b4d03eab49d11715398 > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150911/272ff4ac/attachment-0001.html > > ------------------------------ > > _______________________________________________ > Bro mailing list > Bro at bro.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > End of Bro Digest, Vol 113, Issue 12 > ************************************ > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150911/3b426819/attachment.html From boreham.smith at gmail.com Fri Sep 11 19:45:08 2015 From: boreham.smith at gmail.com (Boreham-Smith) Date: Sat, 12 Sep 2015 12:45:08 +1000 Subject: [Bro] Issue when adding a field to files.log Message-ID: Hi All, I have written a script that extracts filetypes of interest, submits the extracted file to the cuckoo sandbox, and records the cuckoo task_id. I currently store this information successfully in the notice log, but would like to add an optional field to the files.log to store this task_id. I have confirmed that I can add and populate the new files.log field with static values, but if I attempt to do this when calling an external program to handle the cuckoo submission (ie I use the 'when' block below), the value is not output in the log. The print statement within the when block, and notice.log output confirms the value is being populated, it is just not being written to files.log. Any suggestions on what I might be doing incorrectly? I have provided what I think are the relevant code extracts below, but am happy to provide more detail if that will assist: # Add the new field to the files.log redef record Files::Info += { cuckoo_id: int &optional &log; }; # Function that returns the cuckoo task_id function submit_cuckoo(f: fa_file): int { local command = Exec::Command($cmd=fmt("%s extract_files/%s",tool,f$info$extracted)); return when ( local result = Exec::run(command)){ local id: int = to_int(result$stdout[0]); return id; } } # Populate the new field event file_state_remove( f: fa_file ) { if (f$info?$extracted) { when ( local id = submit_cuckoo(f) ){ f$info$cuckoo_id = id; print fmt("Cuckoo ID value set: %d", f$info$cuckoo_id); NOTICE([$note=File::Cuckoo_Submission, $msg=fmt("https://cuckoo/analysis/%s", f$info$cuckoo_id), $f=f]); } } } # files.log extract #fields ts fuid tx_hosts rx_hosts conn_uids source depth analyzers mime_type filename duration local_orig is_orig seen_bytes total_bytes missing_bytes overflow_bytes timedout parent_fuid md5 sha1 sha256 extracted cuckoo_id #types time string set[addr] set[addr] set[string] string count set[string] string string interval bool bool count count countcount bool string string string string string int 1441526348.202595 FtBY2c3CsMMNsBdAil 192.168.1.xxx 192.168.1.yyy CKkqBYszNpSR6Bgaf HTTP 0 EXTRACT application/msword - 0.108599 -F 616960 616960 0 0 F - - - - HTTP-FtBY2c3CsMMNsBdAil.doc - # notice.log extract #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc proto note msg sub src dst pn peer_descr actions suppress_for dropped remote_location.country_code remote_location.region remote_location.city remote_location.latitude remote_location.longitude #types time string addr port addr port string string string enum enum string string addr addr port count string set[enum] interval bool string string string double double 1441526362.215942 CKkqBYszNpSR6Bgaf 192.168.1.yyy 33805 192.168.1.xxx 80 FtBY2c3CsMMNsBdAil application/msword http://192.168.1.xxx/files/test.doc tcp File::Cuckoo_Submission https://cuckoo/analysis/80 - 192.168.1.yyy 192.168.1.xxx 80 - bro Notice::ACTION_LOG 3600.000000 F- - - - - ------- regards, Boreham -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150912/d85fcb22/attachment-0001.html From dnthayer at illinois.edu Fri Sep 11 22:33:29 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Sat, 12 Sep 2015 00:33:29 -0500 Subject: [Bro] Issue when adding a field to files.log In-Reply-To: References: Message-ID: <55F3B929.1010909@illinois.edu> What is most likely happening is that by the time your external program returns its result, the log record has already been written (without the cuckoo_id value) to files.log. On 09/11/2015 09:45 PM, Boreham-Smith wrote: > Hi All, > > I have written a script that extracts filetypes of interest, submits the > extracted file to the cuckoo sandbox, and records the cuckoo task_id. I > currently store this information successfully in the notice log, but > would like to add an optional field to the files.log to store this task_id. > > I have confirmed that I can add and populate the new files.log field > with static values, but if I attempt to do this when calling an external > program to handle the cuckoo submission (ie I use the 'when' block > below), the value is not output in the log. The print statement within > the when block, and notice.log output confirms the value is being > populated, it is just not being written to files.log. > > Any suggestions on what I might be doing incorrectly? > > I have provided what I think are the relevant code extracts below, but > am happy to provide more detail if that will assist: > > # Add the new field to the files.log > > redef record Files::Info += { > cuckoo_id: int &optional &log; > }; > > # Function that returns the cuckoo task_id > function submit_cuckoo(f: fa_file): int > { > local command = Exec::Command($cmd=fmt("%s > extract_files/%s",tool,f$info$extracted)); > return when ( local result = Exec::run(command)){ > local id: int = to_int(result$stdout[0]); > return id; > } > } > > # Populate the new field > event file_state_remove( f: fa_file ) > { > if (f$info?$extracted) { > when ( local id = submit_cuckoo(f) ){ > f$info$cuckoo_id = id; > print fmt("Cuckoo ID value set: %d", f$info$cuckoo_id); > NOTICE([$note=File::Cuckoo_Submission, > $msg=fmt("https://cuckoo/analysis/%s > ", > f$info$cuckoo_id), > $f=f]); > } > } > } > > > # files.log extract > #fields ts fuid tx_hosts rx_hosts conn_uids > source depth analyzers mime_type filename > duration local_orig is_orig seen_bytes > total_bytes missing_bytes overflow_bytes timedout > parent_fuid md5 sha1 sha256 extracted cuckoo_id > #types time string set[addr] set[addr] set[string] > string count set[string] string string interval bool > bool count count countcount bool string string string > string string int > 1441526348.202595 FtBY2c3CsMMNsBdAil 192.168.1.xxx > 192.168.1.yyy CKkqBYszNpSR6Bgaf HTTP 0 EXTRACT > application/msword - 0.108599 -F 616960 616960 > 0 0 F - - - - > HTTP-FtBY2c3CsMMNsBdAil.doc - > > # notice.log extract > #fields ts uid id.orig_h id.orig_p id.resp_h > id.resp_p fuid file_mime_type file_desc proto note > msg sub src dst pn peer_descr actions > suppress_for dropped remote_location.country_code > remote_location.region remote_location.city > remote_location.latitude remote_location.longitude > #types time string addr port addr port string string > string enum enum string string addr addr port count > string set[enum] interval bool string string string > double double > 1441526362.215942 CKkqBYszNpSR6Bgaf 192.168.1.yyy 33805 > 192.168.1.xxx 80 FtBY2c3CsMMNsBdAil application/msword > http://192.168.1.xxx/files/test.doc > > tcp File::Cuckoo_Submission https://cuckoo/analysis/80 > > - 192.168.1.yyy 192.168.1.xxx 80 - bro > Notice::ACTION_LOG 3600.000000 F- - - - - > > ------- > regards, > > Boreham > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From boreham.smith at gmail.com Fri Sep 11 23:38:27 2015 From: boreham.smith at gmail.com (Boreham-Smith) Date: Sat, 12 Sep 2015 16:38:27 +1000 Subject: [Bro] Issue when adding a field to files.log In-Reply-To: <55F3B929.1010909@illinois.edu> References: <55F3B929.1010909@illinois.edu> Message-ID: Thanks Daniel, What you suggest makes sense and explains the behaviour I observed. I guess this leads me to the next thought - is there a way to delay the file getting written out, or an alternate File event that could be used to achive the outcome I am looking for? I am happy pulling the data form the notice logs I am generating, but it seemed tidy to have this information in the file.log too if possible. regards, Boreham On Sat, Sep 12, 2015 at 3:33 PM, Daniel Thayer wrote: > What is most likely happening is that by the time your > external program returns its result, the log record has > already been written (without the cuckoo_id value) to files.log. > > > > On 09/11/2015 09:45 PM, Boreham-Smith wrote: > >> Hi All, >> >> I have written a script that extracts filetypes of interest, submits the >> extracted file to the cuckoo sandbox, and records the cuckoo task_id. I >> currently store this information successfully in the notice log, but >> would like to add an optional field to the files.log to store this >> task_id. >> >> I have confirmed that I can add and populate the new files.log field >> with static values, but if I attempt to do this when calling an external >> program to handle the cuckoo submission (ie I use the 'when' block >> below), the value is not output in the log. The print statement within >> the when block, and notice.log output confirms the value is being >> populated, it is just not being written to files.log. >> >> Any suggestions on what I might be doing incorrectly? >> >> I have provided what I think are the relevant code extracts below, but >> am happy to provide more detail if that will assist: >> >> # Add the new field to the files.log >> >> redef record Files::Info += { >> cuckoo_id: int &optional &log; >> }; >> >> # Function that returns the cuckoo task_id >> function submit_cuckoo(f: fa_file): int >> { >> local command = Exec::Command($cmd=fmt("%s >> extract_files/%s",tool,f$info$extracted)); >> return when ( local result = Exec::run(command)){ >> local id: int = to_int(result$stdout[0]); >> return id; >> } >> } >> >> # Populate the new field >> event file_state_remove( f: fa_file ) >> { >> if (f$info?$extracted) { >> when ( local id = submit_cuckoo(f) ){ >> f$info$cuckoo_id = id; >> print fmt("Cuckoo ID value set: %d", f$info$cuckoo_id); >> NOTICE([$note=File::Cuckoo_Submission, >> $msg=fmt("https://cuckoo/analysis/%s >> < >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cuckoo_analysis_-25s&d=AwMGaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzpfFBcawv0&m=ae24aZX-BlkOV69c9Alc2GQYRcmqia6ijosigglu5aM&s=uTipFhJT472EtdFuf9enkoihzQS0Hvht3uFGYtii2Bw&e= >> >", >> >> f$info$cuckoo_id), >> $f=f]); >> } >> } >> } >> >> >> # files.log extract >> #fields ts fuid tx_hosts rx_hosts conn_uids >> source depth analyzers mime_type filename >> duration local_orig is_orig seen_bytes >> total_bytes missing_bytes overflow_bytes timedout >> parent_fuid md5 sha1 sha256 extracted cuckoo_id >> #types time string set[addr] set[addr] set[string] >> string count set[string] string string interval bool >> bool count count countcount bool string string string >> string string int >> 1441526348.202595 FtBY2c3CsMMNsBdAil 192.168.1.xxx >> 192.168.1.yyy CKkqBYszNpSR6Bgaf HTTP 0 EXTRACT >> application/msword - 0.108599 -F 616960 616960 >> 0 0 F - - - - >> HTTP-FtBY2c3CsMMNsBdAil.doc - >> >> # notice.log extract >> #fields ts uid id.orig_h id.orig_p id.resp_h >> id.resp_p fuid file_mime_type file_desc proto note >> msg sub src dst pn peer_descr actions >> suppress_for dropped remote_location.country_code >> remote_location.region remote_location.city >> remote_location.latitude remote_location.longitude >> #types time string addr port addr port string string >> string enum enum string string addr addr port count >> string set[enum] interval bool string string string >> double double >> 1441526362.215942 CKkqBYszNpSR6Bgaf 192.168.1.yyy 33805 >> 192.168.1.xxx 80 FtBY2c3CsMMNsBdAil application/msword >> http://192.168.1.xxx/files/test.doc >> < >> https://urldefense.proofpoint.com/v2/url?u=http-3A__192.168.1.xxx_files_test.doc&d=AwMGaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzpfFBcawv0&m=ae24aZX-BlkOV69c9Alc2GQYRcmqia6ijosigglu5aM&s=C_rjh_HibNWOcOyptdaUavr_Ktn6wRtFVNCaq_cYAW4&e= >> > >> tcp File::Cuckoo_Submission https://cuckoo/analysis/80 >> < >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cuckoo_analysis_80&d=AwMGaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Bi5qPBnY0NmYPqnRTPj_AfXQKpfQTZUpCzpfFBcawv0&m=ae24aZX-BlkOV69c9Alc2GQYRcmqia6ijosigglu5aM&s=mhE5lNaIUBpgobeJZf9rZ9XlwD8p_Bjky-V2i9eheD8&e= >> > >> - 192.168.1.yyy 192.168.1.xxx 80 - bro >> Notice::ACTION_LOG 3600.000000 F- - - - - >> >> ------- >> regards, >> >> Boreham >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150912/c674fbf6/attachment.html From liburdi.joshua at gmail.com Sat Sep 12 07:16:18 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Sat, 12 Sep 2015 10:16:18 -0400 Subject: [Bro] Issue when adding a field to files.log In-Reply-To: References: <55F3B929.1010909@illinois.edu> Message-ID: My suggestion is to generate a whole new log with the cuckoo_id value (cuckoo.log ?). The main advantage to doing it this way is that new log entries will be written whenever Cuckoo analysis finishes-- you won't need to delay files.log or continue to put cuckoo_id values in notice.log. Additionally, if each entry in the new log has a UID, then that's a very Brogrammatic way to correlate the cuckoo_id value to entries in files.log. Josh On Sat, Sep 12, 2015 at 2:38 AM, Boreham-Smith wrote: > Thanks Daniel, > > What you suggest makes sense and explains the behaviour I observed. I guess > this leads me to the next thought - is there a way to delay the file getting > written out, or an alternate File event that could be used to achive the > outcome I am looking for? > > I am happy pulling the data form the notice logs I am generating, but it > seemed tidy to have this information in the file.log too if possible. > > regards, > Boreham > > > On Sat, Sep 12, 2015 at 3:33 PM, Daniel Thayer > wrote: >> >> What is most likely happening is that by the time your >> external program returns its result, the log record has >> already been written (without the cuckoo_id value) to files.log. >> >> >> >> On 09/11/2015 09:45 PM, Boreham-Smith wrote: >>> >>> Hi All, >>> >>> I have written a script that extracts filetypes of interest, submits the >>> extracted file to the cuckoo sandbox, and records the cuckoo task_id. I >>> currently store this information successfully in the notice log, but >>> would like to add an optional field to the files.log to store this >>> task_id. >>> >>> I have confirmed that I can add and populate the new files.log field >>> with static values, but if I attempt to do this when calling an external >>> program to handle the cuckoo submission (ie I use the 'when' block >>> below), the value is not output in the log. The print statement within >>> the when block, and notice.log output confirms the value is being >>> populated, it is just not being written to files.log. >>> >>> Any suggestions on what I might be doing incorrectly? >>> >>> I have provided what I think are the relevant code extracts below, but >>> am happy to provide more detail if that will assist: >>> >>> # Add the new field to the files.log >>> >>> redef record Files::Info += { >>> cuckoo_id: int &optional &log; >>> }; >>> >>> # Function that returns the cuckoo task_id >>> function submit_cuckoo(f: fa_file): int >>> { >>> local command = Exec::Command($cmd=fmt("%s >>> extract_files/%s",tool,f$info$extracted)); >>> return when ( local result = Exec::run(command)){ >>> local id: int = to_int(result$stdout[0]); >>> return id; >>> } >>> } >>> >>> # Populate the new field >>> event file_state_remove( f: fa_file ) >>> { >>> if (f$info?$extracted) { >>> when ( local id = submit_cuckoo(f) ){ >>> f$info$cuckoo_id = id; >>> print fmt("Cuckoo ID value set: %d", f$info$cuckoo_id); >>> NOTICE([$note=File::Cuckoo_Submission, >>> $msg=fmt("https://cuckoo/analysis/%s >>> >>> ", >>> >>> f$info$cuckoo_id), >>> $f=f]); >>> } >>> } >>> } >>> >>> >>> # files.log extract >>> #fields ts fuid tx_hosts rx_hosts conn_uids >>> source depth analyzers mime_type filename >>> duration local_orig is_orig seen_bytes >>> total_bytes missing_bytes overflow_bytes timedout >>> parent_fuid md5 sha1 sha256 extracted cuckoo_id >>> #types time string set[addr] set[addr] set[string] >>> string count set[string] string string interval bool >>> bool count count countcount bool string string string >>> string string int >>> 1441526348.202595 FtBY2c3CsMMNsBdAil 192.168.1.xxx >>> 192.168.1.yyy CKkqBYszNpSR6Bgaf HTTP 0 EXTRACT >>> application/msword - 0.108599 -F 616960 616960 >>> 0 0 F - - - - >>> HTTP-FtBY2c3CsMMNsBdAil.doc - >>> >>> # notice.log extract >>> #fields ts uid id.orig_h id.orig_p id.resp_h >>> id.resp_p fuid file_mime_type file_desc proto note >>> msg sub src dst pn peer_descr actions >>> suppress_for dropped remote_location.country_code >>> remote_location.region remote_location.city >>> remote_location.latitude remote_location.longitude >>> #types time string addr port addr port string string >>> string enum enum string string addr addr port count >>> string set[enum] interval bool string string string >>> double double >>> 1441526362.215942 CKkqBYszNpSR6Bgaf 192.168.1.yyy 33805 >>> 192.168.1.xxx 80 FtBY2c3CsMMNsBdAil application/msword >>> http://192.168.1.xxx/files/test.doc >>> >>> >>> tcp File::Cuckoo_Submission https://cuckoo/analysis/80 >>> >>> >>> - 192.168.1.yyy 192.168.1.xxx 80 - bro >>> Notice::ACTION_LOG 3600.000000 F- - - - - >>> >>> ------- >>> regards, >>> >>> Boreham >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dnthayer at illinois.edu Sat Sep 12 11:12:26 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Sat, 12 Sep 2015 13:12:26 -0500 Subject: [Bro] Issue when adding a field to files.log In-Reply-To: References: <55F3B929.1010909@illinois.edu> Message-ID: <55F46B0A.7030103@illinois.edu> This sounds like a good idea. The "Logging Framework" document in the Bro Manual shows an example of how to create a new log stream (look at the first part of the "Streams" section): https://www.bro.org/sphinx/frameworks/logging.html On 09/12/2015 09:16 AM, Josh Liburdi wrote: > My suggestion is to generate a whole new log with the cuckoo_id value > (cuckoo.log ?). The main advantage to doing it this way is that new > log entries will be written whenever Cuckoo analysis finishes-- you > won't need to delay files.log or continue to put cuckoo_id values in > notice.log. Additionally, if each entry in the new log has a UID, then > that's a very Brogrammatic way to correlate the cuckoo_id value to > entries in files.log. > > Josh > > On Sat, Sep 12, 2015 at 2:38 AM, Boreham-Smith wrote: >> Thanks Daniel, >> >> What you suggest makes sense and explains the behaviour I observed. I guess >> this leads me to the next thought - is there a way to delay the file getting >> written out, or an alternate File event that could be used to achive the >> outcome I am looking for? >> >> I am happy pulling the data form the notice logs I am generating, but it >> seemed tidy to have this information in the file.log too if possible. >> >> regards, >> Boreham From boreham.smith at gmail.com Sat Sep 12 14:16:01 2015 From: boreham.smith at gmail.com (Boreham-Smith) Date: Sun, 13 Sep 2015 07:16:01 +1000 Subject: [Bro] Issue when adding a field to files.log In-Reply-To: <55F46B0A.7030103@illinois.edu> References: <55F3B929.1010909@illinois.edu> <55F46B0A.7030103@illinois.edu> Message-ID: Hi Josh, Yes - this would seem to be a sensible way to go. I'll look in to the examples in the logging framework. Best regards, Boreham On Sun, Sep 13, 2015 at 4:12 AM, Daniel Thayer wrote: > This sounds like a good idea. The "Logging Framework" document in > the Bro Manual shows an example of how to create a new log stream > (look at the first part of the "Streams" section): > https://www.bro.org/sphinx/frameworks/logging.html > > > > On 09/12/2015 09:16 AM, Josh Liburdi wrote: > >> My suggestion is to generate a whole new log with the cuckoo_id value >> (cuckoo.log ?). The main advantage to doing it this way is that new >> log entries will be written whenever Cuckoo analysis finishes-- you >> won't need to delay files.log or continue to put cuckoo_id values in >> notice.log. Additionally, if each entry in the new log has a UID, then >> that's a very Brogrammatic way to correlate the cuckoo_id value to >> entries in files.log. >> >> Josh >> >> On Sat, Sep 12, 2015 at 2:38 AM, Boreham-Smith >> wrote: >> >>> Thanks Daniel, >>> >>> What you suggest makes sense and explains the behaviour I observed. I >>> guess >>> this leads me to the next thought - is there a way to delay the file >>> getting >>> written out, or an alternate File event that could be used to achive the >>> outcome I am looking for? >>> >>> I am happy pulling the data form the notice logs I am generating, but it >>> seemed tidy to have this information in the file.log too if possible. >>> >>> regards, >>> Boreham >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150913/ba14c71b/attachment.html From cdaviso1 at vols.utk.edu Mon Sep 14 12:41:47 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Mon, 14 Sep 2015 19:41:47 +0000 Subject: [Bro] Bro Cluster Documentation Error In-Reply-To: References: Message-ID: Good Afternoon, I am trying to make documentation for installing a bro cluster configuration, and receive the attached error when trying to install via broctl. I can log into both of my worker nodes from the bro manager via ssh fine, and without a password... Thank you, Charles -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150914/e20b697e/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: Error.PNG Type: image/png Size: 57833 bytes Desc: Error.PNG Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150914/e20b697e/attachment-0001.bin From cdaviso1 at vols.utk.edu Mon Sep 14 12:56:41 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Mon, 14 Sep 2015 19:56:41 +0000 Subject: [Bro] Bro Cluster Documentation Error In-Reply-To: References: , Message-ID: I assume attachments don't work... here is the test output: root at ip-172-31-41-32:/home/ubuntu# export PATH=/usr/local/bro/bin:$PATH root at ip-172-31-41-32:/home/ubuntu# broctl Warning: broctl node config has changed (run the broctl "deploy" command) Warning: Bro node "bro" possibly still running on host "localhost" (PID 16564) Welcome to BroControl 1.4 Type "help" for help. [BroControl] > install removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/sit e ... removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/aut o ... creating policy directories ... installing site policies ... generating cluster-layout.bro ... generating local-networks.bro ... generating broctl-config.bro ... generating broctl-config.sh ... updating nodes ... Host key verification failed. Host key verification failed. Error: cannot create (some of the) directories /usr/local/bro,/usr/local/bro/log s,/usr/local/bro/spool,/usr/local/bro/spool/tmp on node worker-1 [BroControl] > Host key verification failed. Host key verification failed. Host key verification failed. Host key verification failed. Thank you, Charles ________________________________ From: Davison, Charles Robert Sent: Monday, September 14, 2015 1:41 PM To: bro at bro.org Subject: Bro Cluster Documentation Error Good Afternoon, I am trying to make documentation for installing a bro cluster configuration, and receive the attached error when trying to install via broctl. I can log into both of my worker nodes from the bro manager via ssh fine, and without a password... Thank you, Charles -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150914/a208b1dd/attachment.html From dnthayer at illinois.edu Mon Sep 14 13:17:59 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Mon, 14 Sep 2015 15:17:59 -0500 Subject: [Bro] Bro Cluster Documentation Error In-Reply-To: References: , Message-ID: <55F72B77.8030905@illinois.edu> When you check if you can ssh to the other machines in your cluster, you need to make sure you're running ssh as the same user that you're running broctl. Also, what did you specify for the "host=" entries in your node.cfg? On 09/14/2015 02:56 PM, Davison, Charles Robert wrote: > I assume attachments don't work... here is the test output: > > > root at ip-172-31-41-32:/home/ubuntu# export PATH=/usr/local/bro/bin:$PATH > > root at ip-172-31-41-32:/home/ubuntu# broctl > > Warning: broctl node config has changed (run the broctl "deploy" command) > > Warning: Bro node "bro" possibly still running on host "localhost" (PID > 16564) > > > Welcome to BroControl 1.4 > > > Type "help" for help. > > > [BroControl] > install > > removing old policies in > /usr/local/bro/spool/installed-scripts-do-not-touch/sit > e ... > > removing old policies in > /usr/local/bro/spool/installed-scripts-do-not-touch/aut > o ... > > creating policy directories ... > > installing site policies ... > > generating cluster-layout.bro ... > > generating local-networks.bro ... > > generating broctl-config.bro ... > > generating broctl-config.sh ... > > updating nodes ... > > Host key verification failed. > > Host key verification failed. > > Error: cannot create (some of the) directories > /usr/local/bro,/usr/local/bro/log > > s,/usr/local/bro/spool,/usr/local/bro/spool/tmp on node worker-1 > > [BroControl] > Host key verification failed. > > Host key verification failed. > > Host key verification failed. > > Host key verification failed. > > > > Thank you, > > Charles > > > ------------------------------------------------------------------------ > *From:* Davison, Charles Robert > *Sent:* Monday, September 14, 2015 1:41 PM > *To:* bro at bro.org > *Subject:* Bro Cluster Documentation Error > > > Good Afternoon, > > > I am trying to make documentation for installing a bro cluster > configuration, and receive the attached error when trying to install via > broctl. I can log into both of my worker nodes from the bro manager via > ssh fine, and without a password... > > > Thank you, > > Charles > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From cdaviso1 at vols.utk.edu Mon Sep 14 14:24:28 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Mon, 14 Sep 2015 21:24:28 +0000 Subject: [Bro] Bro Cluster Documentation Error In-Reply-To: <55F72B77.8030905@illinois.edu> References: , , <55F72B77.8030905@illinois.edu> Message-ID: Please see the attached document on how configured the host entries. I can ssh into the computers as my ubuntu user fine, but I copied over my keys as follows: scp -v ~/.ssh/id_rsa.pub root at 172.31.41.31:/home/ubuntu/.ssh/authorized_keys2 scp -v ~/.ssh/id_rsa.pub root at 172.31.41.33:/home/ubuntu/.ssh/authorized_keys2 Is this an issue? I tried using ubuntu as the user and it hangs: ubuntu at ip-172-31-41-32:~$ scp -v ~/.ssh/id_rsa.pub ubuntu at 172.31.41.33:/home/ubuntu/.ssh/authorized_keys2 Executing: program /usr/bin/ssh host 172.31.41.33, user ubuntu, command scp -v -t /home/ubuntu/.ssh/authorized_keys2 OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug1: Connecting to 172.31.41.33 [172.31.41.33] port 22. debug1: Connection established. debug1: identity file /home/ubuntu/.ssh/id_rsa type 1 debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1 debug1: identity file /home/ubuntu/.ssh/id_dsa type -1 debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1 debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1 debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1 debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1 debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 pat OpenSSH_6.6.1* compat 0x04000000 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5-etm at openssh.com none debug1: kex: client->server aes128-ctr hmac-md5-etm at openssh.com none debug1: sending SSH2_MSG_KEX_ECDH_INIT debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ECDSA d1:0a:e6:c3:bf:ee:23:5a:63:63:ce:c8:71:41:88:29 debug1: Host '172.31.41.33' is known and matches the ECDSA host key. debug1: Found key in /home/ubuntu/.ssh/known_hosts:2 debug1: ssh_ecdsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: Roaming not allowed by server debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,password debug1: Next authentication method: publickey debug1: Offering RSA public key: /home/ubuntu/.ssh/id_rsa debug1: Server accepts key: pkalg ssh-rsa blen 279 debug1: key_parse_private2: missing begin marker debug1: read PEM private key done: type RSA debug1: Authentication succeeded (publickey). Authenticated to 172.31.41.33 ([172.31.41.33]:22). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions at openssh.com debug1: Entering interactive session. debug1: Sending environment. debug1: Sending env LANG = en_US.UTF-8 debug1: Sending command: scp -v -t /home/ubuntu/.ssh/authorized_keys2 Sending file modes: C0644 404 id_rsa.pub scp: /home/ubuntu/.ssh/authorized_keys2: Permission denied ubuntu at ip-172-31-41-32:~$ Sink: C0644 404 id_rsa.pub debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug1: channel 0: free: client-session, nchannels 1 debug1: fd 0 clearing O_NONBLOCK debug1: fd 1 clearing O_NONBLOCK Transferred: sent 3472, received 2636 bytes, in 0.2 seconds Bytes per second: sent 18676.3, received 14179.4 debug1: Exit status 1 ________________________________________ From: Daniel Thayer Sent: Monday, September 14, 2015 2:17 PM To: Davison, Charles Robert; bro at bro.org Subject: Re: [Bro] Bro Cluster Documentation Error When you check if you can ssh to the other machines in your cluster, you need to make sure you're running ssh as the same user that you're running broctl. Also, what did you specify for the "host=" entries in your node.cfg? On 09/14/2015 02:56 PM, Davison, Charles Robert wrote: > I assume attachments don't work... here is the test output: > > > root at ip-172-31-41-32:/home/ubuntu# export PATH=/usr/local/bro/bin:$PATH > > root at ip-172-31-41-32:/home/ubuntu# broctl > > Warning: broctl node config has changed (run the broctl "deploy" command) > > Warning: Bro node "bro" possibly still running on host "localhost" (PID > 16564) > > > Welcome to BroControl 1.4 > > > Type "help" for help. > > > [BroControl] > install > > removing old policies in > /usr/local/bro/spool/installed-scripts-do-not-touch/sit > e ... > > removing old policies in > /usr/local/bro/spool/installed-scripts-do-not-touch/aut > o ... > > creating policy directories ... > > installing site policies ... > > generating cluster-layout.bro ... > > generating local-networks.bro ... > > generating broctl-config.bro ... > > generating broctl-config.sh ... > > updating nodes ... > > Host key verification failed. > > Host key verification failed. > > Error: cannot create (some of the) directories > /usr/local/bro,/usr/local/bro/log > > s,/usr/local/bro/spool,/usr/local/bro/spool/tmp on node worker-1 > > [BroControl] > Host key verification failed. > > Host key verification failed. > > Host key verification failed. > > Host key verification failed. > > > > Thank you, > > Charles > > > ------------------------------------------------------------------------ > *From:* Davison, Charles Robert > *Sent:* Monday, September 14, 2015 1:41 PM > *To:* bro at bro.org > *Subject:* Bro Cluster Documentation Error > > > Good Afternoon, > > > I am trying to make documentation for installing a bro cluster > configuration, and receive the attached error when trying to install via > broctl. I can log into both of my worker nodes from the bro manager via > ssh fine, and without a password... > > > Thank you, > > Charles > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- A non-text attachment was scrubbed... Name: image2015-9-14 9-57-23.png Type: image/png Size: 55257 bytes Desc: image2015-9-14 9-57-23.png Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150914/a0e13cde/attachment-0001.bin From dnthayer at illinois.edu Mon Sep 14 14:40:35 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Mon, 14 Sep 2015 16:40:35 -0500 Subject: [Bro] Bro Cluster Documentation Error In-Reply-To: References: , , <55F72B77.8030905@illinois.edu> Message-ID: <55F73ED3.3080007@illinois.edu> In screenshot in previous email, it appeared you were running broctl as the "root" user. If that's the case, then you need to be able to ssh to your worker machine as the "root" user. The home directory of the "root" user is probably "/root". On 09/14/2015 04:24 PM, Davison, Charles Robert wrote: > Please see the attached document on how configured the host entries. > > I can ssh into the computers as my ubuntu user fine, but I copied over my keys as follows: > > scp -v ~/.ssh/id_rsa.pub root at 172.31.41.31:/home/ubuntu/.ssh/authorized_keys2 > scp -v ~/.ssh/id_rsa.pub root at 172.31.41.33:/home/ubuntu/.ssh/authorized_keys2 > > Is this an issue? I tried using ubuntu as the user and it hangs: > > ubuntu at ip-172-31-41-32:~$ scp -v ~/.ssh/id_rsa.pub ubuntu at 172.31.41.33:/home/ubuntu/.ssh/authorized_keys2 > Executing: program /usr/bin/ssh host 172.31.41.33, user ubuntu, command scp -v -t /home/ubuntu/.ssh/authorized_keys2 > OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014 > debug1: Reading configuration data /etc/ssh/ssh_config > debug1: /etc/ssh/ssh_config line 19: Applying options for * > debug1: Connecting to 172.31.41.33 [172.31.41.33] port 22. > debug1: Connection established. > debug1: identity file /home/ubuntu/.ssh/id_rsa type 1 > debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1 > debug1: identity file /home/ubuntu/.ssh/id_dsa type -1 > debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1 > debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1 > debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1 > debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1 > debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1 > debug1: Enabling compatibility mode for protocol 2.0 > debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 > debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 > debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 pat OpenSSH_6.6.1* compat 0x04000000 > debug1: SSH2_MSG_KEXINIT sent > debug1: SSH2_MSG_KEXINIT received > debug1: kex: server->client aes128-ctr hmac-md5-etm at openssh.com none > debug1: kex: client->server aes128-ctr hmac-md5-etm at openssh.com none > debug1: sending SSH2_MSG_KEX_ECDH_INIT > debug1: expecting SSH2_MSG_KEX_ECDH_REPLY > debug1: Server host key: ECDSA d1:0a:e6:c3:bf:ee:23:5a:63:63:ce:c8:71:41:88:29 > debug1: Host '172.31.41.33' is known and matches the ECDSA host key. > debug1: Found key in /home/ubuntu/.ssh/known_hosts:2 > debug1: ssh_ecdsa_verify: signature correct > debug1: SSH2_MSG_NEWKEYS sent > debug1: expecting SSH2_MSG_NEWKEYS > debug1: SSH2_MSG_NEWKEYS received > debug1: Roaming not allowed by server > debug1: SSH2_MSG_SERVICE_REQUEST sent > debug1: SSH2_MSG_SERVICE_ACCEPT received > debug1: Authentications that can continue: publickey,password > debug1: Next authentication method: publickey > debug1: Offering RSA public key: /home/ubuntu/.ssh/id_rsa > debug1: Server accepts key: pkalg ssh-rsa blen 279 > debug1: key_parse_private2: missing begin marker > debug1: read PEM private key done: type RSA > debug1: Authentication succeeded (publickey). > Authenticated to 172.31.41.33 ([172.31.41.33]:22). > debug1: channel 0: new [client-session] > debug1: Requesting no-more-sessions at openssh.com > debug1: Entering interactive session. > debug1: Sending environment. > debug1: Sending env LANG = en_US.UTF-8 > debug1: Sending command: scp -v -t /home/ubuntu/.ssh/authorized_keys2 > Sending file modes: C0644 404 id_rsa.pub > scp: /home/ubuntu/.ssh/authorized_keys2: Permission denied > ubuntu at ip-172-31-41-32:~$ Sink: C0644 404 id_rsa.pub > debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 > debug1: channel 0: free: client-session, nchannels 1 > debug1: fd 0 clearing O_NONBLOCK > debug1: fd 1 clearing O_NONBLOCK > Transferred: sent 3472, received 2636 bytes, in 0.2 seconds > Bytes per second: sent 18676.3, received 14179.4 > debug1: Exit status 1 > > ________________________________________ > From: Daniel Thayer > Sent: Monday, September 14, 2015 2:17 PM > To: Davison, Charles Robert; bro at bro.org > Subject: Re: [Bro] Bro Cluster Documentation Error > > When you check if you can ssh to the other machines in your cluster, > you need to make sure you're running ssh as the same user that > you're running broctl. > > Also, what did you specify for the "host=" entries in your node.cfg? > > > > On 09/14/2015 02:56 PM, Davison, Charles Robert wrote: >> I assume attachments don't work... here is the test output: >> >> >> root at ip-172-31-41-32:/home/ubuntu# export PATH=/usr/local/bro/bin:$PATH >> >> root at ip-172-31-41-32:/home/ubuntu# broctl >> >> Warning: broctl node config has changed (run the broctl "deploy" command) >> >> Warning: Bro node "bro" possibly still running on host "localhost" (PID >> 16564) >> >> >> Welcome to BroControl 1.4 >> >> >> Type "help" for help. >> >> >> [BroControl] > install >> >> removing old policies in >> /usr/local/bro/spool/installed-scripts-do-not-touch/sit >> e ... >> >> removing old policies in >> /usr/local/bro/spool/installed-scripts-do-not-touch/aut >> o ... >> >> creating policy directories ... >> >> installing site policies ... >> >> generating cluster-layout.bro ... >> >> generating local-networks.bro ... >> >> generating broctl-config.bro ... >> >> generating broctl-config.sh ... >> >> updating nodes ... >> >> Host key verification failed. >> >> Host key verification failed. >> >> Error: cannot create (some of the) directories >> /usr/local/bro,/usr/local/bro/log >> >> s,/usr/local/bro/spool,/usr/local/bro/spool/tmp on node worker-1 >> >> [BroControl] > Host key verification failed. >> >> Host key verification failed. >> >> Host key verification failed. >> >> Host key verification failed. >> >> >> >> Thank you, >> >> Charles >> >> >> ------------------------------------------------------------------------ >> *From:* Davison, Charles Robert >> *Sent:* Monday, September 14, 2015 1:41 PM >> *To:* bro at bro.org >> *Subject:* Bro Cluster Documentation Error >> >> >> Good Afternoon, >> >> >> I am trying to make documentation for installing a bro cluster >> configuration, and receive the attached error when trying to install via >> broctl. I can log into both of my worker nodes from the bro manager via >> ssh fine, and without a password... >> >> >> Thank you, >> >> Charles >> >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> From cdaviso1 at vols.utk.edu Mon Sep 14 14:45:12 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Mon, 14 Sep 2015 21:45:12 +0000 Subject: [Bro] Bro Cluster Documentation Error In-Reply-To: <55F73ED3.3080007@illinois.edu> References: , , <55F72B77.8030905@illinois.edu> , <55F73ED3.3080007@illinois.edu> Message-ID: I tried running bro from by ubuntu account and recieve this: Also for whatever reason I have to constantly export my paths to run broctl. Not a big issue but if you know a fix that would be great. ubuntu at ip-172-31-41-32:~$ export PATH=/usr/local/bro/bin:$PATH ubuntu at ip-172-31-41-32:~$ broctl Warning: broctl node config has changed (run the broctl "deploy" command) Warning: Bro node "bro" possibly still running on host "localhost" (PID 16564) Welcome to BroControl 1.4 Type "help" for help. [BroControl] > install Error: cannot acquire lock: [Errno 13] Permission denied: '/usr/local/bro/spool/lock.27491' Error: Unable to get lock [BroControl] > ________________________________________ From: Daniel Thayer Sent: Monday, September 14, 2015 3:40 PM To: Davison, Charles Robert; bro at bro.org Subject: Re: [Bro] Bro Cluster Documentation Error In screenshot in previous email, it appeared you were running broctl as the "root" user. If that's the case, then you need to be able to ssh to your worker machine as the "root" user. The home directory of the "root" user is probably "/root". On 09/14/2015 04:24 PM, Davison, Charles Robert wrote: > Please see the attached document on how configured the host entries. > > I can ssh into the computers as my ubuntu user fine, but I copied over my keys as follows: > > scp -v ~/.ssh/id_rsa.pub root at 172.31.41.31:/home/ubuntu/.ssh/authorized_keys2 > scp -v ~/.ssh/id_rsa.pub root at 172.31.41.33:/home/ubuntu/.ssh/authorized_keys2 > > Is this an issue? I tried using ubuntu as the user and it hangs: > > ubuntu at ip-172-31-41-32:~$ scp -v ~/.ssh/id_rsa.pub ubuntu at 172.31.41.33:/home/ubuntu/.ssh/authorized_keys2 > Executing: program /usr/bin/ssh host 172.31.41.33, user ubuntu, command scp -v -t /home/ubuntu/.ssh/authorized_keys2 > OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014 > debug1: Reading configuration data /etc/ssh/ssh_config > debug1: /etc/ssh/ssh_config line 19: Applying options for * > debug1: Connecting to 172.31.41.33 [172.31.41.33] port 22. > debug1: Connection established. > debug1: identity file /home/ubuntu/.ssh/id_rsa type 1 > debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1 > debug1: identity file /home/ubuntu/.ssh/id_dsa type -1 > debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1 > debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1 > debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1 > debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1 > debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1 > debug1: Enabling compatibility mode for protocol 2.0 > debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 > debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 > debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 pat OpenSSH_6.6.1* compat 0x04000000 > debug1: SSH2_MSG_KEXINIT sent > debug1: SSH2_MSG_KEXINIT received > debug1: kex: server->client aes128-ctr hmac-md5-etm at openssh.com none > debug1: kex: client->server aes128-ctr hmac-md5-etm at openssh.com none > debug1: sending SSH2_MSG_KEX_ECDH_INIT > debug1: expecting SSH2_MSG_KEX_ECDH_REPLY > debug1: Server host key: ECDSA d1:0a:e6:c3:bf:ee:23:5a:63:63:ce:c8:71:41:88:29 > debug1: Host '172.31.41.33' is known and matches the ECDSA host key. > debug1: Found key in /home/ubuntu/.ssh/known_hosts:2 > debug1: ssh_ecdsa_verify: signature correct > debug1: SSH2_MSG_NEWKEYS sent > debug1: expecting SSH2_MSG_NEWKEYS > debug1: SSH2_MSG_NEWKEYS received > debug1: Roaming not allowed by server > debug1: SSH2_MSG_SERVICE_REQUEST sent > debug1: SSH2_MSG_SERVICE_ACCEPT received > debug1: Authentications that can continue: publickey,password > debug1: Next authentication method: publickey > debug1: Offering RSA public key: /home/ubuntu/.ssh/id_rsa > debug1: Server accepts key: pkalg ssh-rsa blen 279 > debug1: key_parse_private2: missing begin marker > debug1: read PEM private key done: type RSA > debug1: Authentication succeeded (publickey). > Authenticated to 172.31.41.33 ([172.31.41.33]:22). > debug1: channel 0: new [client-session] > debug1: Requesting no-more-sessions at openssh.com > debug1: Entering interactive session. > debug1: Sending environment. > debug1: Sending env LANG = en_US.UTF-8 > debug1: Sending command: scp -v -t /home/ubuntu/.ssh/authorized_keys2 > Sending file modes: C0644 404 id_rsa.pub > scp: /home/ubuntu/.ssh/authorized_keys2: Permission denied > ubuntu at ip-172-31-41-32:~$ Sink: C0644 404 id_rsa.pub > debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 > debug1: channel 0: free: client-session, nchannels 1 > debug1: fd 0 clearing O_NONBLOCK > debug1: fd 1 clearing O_NONBLOCK > Transferred: sent 3472, received 2636 bytes, in 0.2 seconds > Bytes per second: sent 18676.3, received 14179.4 > debug1: Exit status 1 > > ________________________________________ > From: Daniel Thayer > Sent: Monday, September 14, 2015 2:17 PM > To: Davison, Charles Robert; bro at bro.org > Subject: Re: [Bro] Bro Cluster Documentation Error > > When you check if you can ssh to the other machines in your cluster, > you need to make sure you're running ssh as the same user that > you're running broctl. > > Also, what did you specify for the "host=" entries in your node.cfg? > > > > On 09/14/2015 02:56 PM, Davison, Charles Robert wrote: >> I assume attachments don't work... here is the test output: >> >> >> root at ip-172-31-41-32:/home/ubuntu# export PATH=/usr/local/bro/bin:$PATH >> >> root at ip-172-31-41-32:/home/ubuntu# broctl >> >> Warning: broctl node config has changed (run the broctl "deploy" command) >> >> Warning: Bro node "bro" possibly still running on host "localhost" (PID >> 16564) >> >> >> Welcome to BroControl 1.4 >> >> >> Type "help" for help. >> >> >> [BroControl] > install >> >> removing old policies in >> /usr/local/bro/spool/installed-scripts-do-not-touch/sit >> e ... >> >> removing old policies in >> /usr/local/bro/spool/installed-scripts-do-not-touch/aut >> o ... >> >> creating policy directories ... >> >> installing site policies ... >> >> generating cluster-layout.bro ... >> >> generating local-networks.bro ... >> >> generating broctl-config.bro ... >> >> generating broctl-config.sh ... >> >> updating nodes ... >> >> Host key verification failed. >> >> Host key verification failed. >> >> Error: cannot create (some of the) directories >> /usr/local/bro,/usr/local/bro/log >> >> s,/usr/local/bro/spool,/usr/local/bro/spool/tmp on node worker-1 >> >> [BroControl] > Host key verification failed. >> >> Host key verification failed. >> >> Host key verification failed. >> >> Host key verification failed. >> >> >> >> Thank you, >> >> Charles >> >> >> ------------------------------------------------------------------------ >> *From:* Davison, Charles Robert >> *Sent:* Monday, September 14, 2015 1:41 PM >> *To:* bro at bro.org >> *Subject:* Bro Cluster Documentation Error >> >> >> Good Afternoon, >> >> >> I am trying to make documentation for installing a bro cluster >> configuration, and receive the attached error when trying to install via >> broctl. I can log into both of my worker nodes from the bro manager via >> ssh fine, and without a password... >> >> >> Thank you, >> >> Charles >> >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> From dnthayer at illinois.edu Mon Sep 14 14:56:45 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Mon, 14 Sep 2015 16:56:45 -0500 Subject: [Bro] Bro Cluster Documentation Error In-Reply-To: References: , , <55F72B77.8030905@illinois.edu> , <55F73ED3.3080007@illinois.edu> Message-ID: <55F7429D.5060101@illinois.edu> The error message for the lock issue is "Permission denied", so you will need to check whether your "ubuntu" user has permission to write to the /usr/local/bro/spool/ directory. On 09/14/2015 04:45 PM, Davison, Charles Robert wrote: > I tried running bro from by ubuntu account and recieve this: > > Also for whatever reason I have to constantly export my paths to run broctl. Not a big issue but if you know a fix that would be great. > > ubuntu at ip-172-31-41-32:~$ export PATH=/usr/local/bro/bin:$PATH > ubuntu at ip-172-31-41-32:~$ broctl > Warning: broctl node config has changed (run the broctl "deploy" command) > Warning: Bro node "bro" possibly still running on host "localhost" (PID 16564) > > Welcome to BroControl 1.4 > > Type "help" for help. > > [BroControl] > install > Error: cannot acquire lock: [Errno 13] Permission denied: '/usr/local/bro/spool/lock.27491' > Error: Unable to get lock > [BroControl] > > > > ________________________________________ > From: Daniel Thayer > Sent: Monday, September 14, 2015 3:40 PM > To: Davison, Charles Robert; bro at bro.org > Subject: Re: [Bro] Bro Cluster Documentation Error > > In screenshot in previous email, it appeared you were running broctl > as the "root" user. If that's the case, then you need to be able > to ssh to your worker machine as the "root" user. The home > directory of the "root" user is probably "/root". > > > > On 09/14/2015 04:24 PM, Davison, Charles Robert wrote: >> Please see the attached document on how configured the host entries. >> >> I can ssh into the computers as my ubuntu user fine, but I copied over my keys as follows: >> >> scp -v ~/.ssh/id_rsa.pub root at 172.31.41.31:/home/ubuntu/.ssh/authorized_keys2 >> scp -v ~/.ssh/id_rsa.pub root at 172.31.41.33:/home/ubuntu/.ssh/authorized_keys2 >> >> Is this an issue? I tried using ubuntu as the user and it hangs: >> >> ubuntu at ip-172-31-41-32:~$ scp -v ~/.ssh/id_rsa.pub ubuntu at 172.31.41.33:/home/ubuntu/.ssh/authorized_keys2 >> Executing: program /usr/bin/ssh host 172.31.41.33, user ubuntu, command scp -v -t /home/ubuntu/.ssh/authorized_keys2 >> OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014 >> debug1: Reading configuration data /etc/ssh/ssh_config >> debug1: /etc/ssh/ssh_config line 19: Applying options for * >> debug1: Connecting to 172.31.41.33 [172.31.41.33] port 22. >> debug1: Connection established. >> debug1: identity file /home/ubuntu/.ssh/id_rsa type 1 >> debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1 >> debug1: identity file /home/ubuntu/.ssh/id_dsa type -1 >> debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1 >> debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1 >> debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1 >> debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1 >> debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1 >> debug1: Enabling compatibility mode for protocol 2.0 >> debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 >> debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 >> debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 pat OpenSSH_6.6.1* compat 0x04000000 >> debug1: SSH2_MSG_KEXINIT sent >> debug1: SSH2_MSG_KEXINIT received >> debug1: kex: server->client aes128-ctr hmac-md5-etm at openssh.com none >> debug1: kex: client->server aes128-ctr hmac-md5-etm at openssh.com none >> debug1: sending SSH2_MSG_KEX_ECDH_INIT >> debug1: expecting SSH2_MSG_KEX_ECDH_REPLY >> debug1: Server host key: ECDSA d1:0a:e6:c3:bf:ee:23:5a:63:63:ce:c8:71:41:88:29 >> debug1: Host '172.31.41.33' is known and matches the ECDSA host key. >> debug1: Found key in /home/ubuntu/.ssh/known_hosts:2 >> debug1: ssh_ecdsa_verify: signature correct >> debug1: SSH2_MSG_NEWKEYS sent >> debug1: expecting SSH2_MSG_NEWKEYS >> debug1: SSH2_MSG_NEWKEYS received >> debug1: Roaming not allowed by server >> debug1: SSH2_MSG_SERVICE_REQUEST sent >> debug1: SSH2_MSG_SERVICE_ACCEPT received >> debug1: Authentications that can continue: publickey,password >> debug1: Next authentication method: publickey >> debug1: Offering RSA public key: /home/ubuntu/.ssh/id_rsa >> debug1: Server accepts key: pkalg ssh-rsa blen 279 >> debug1: key_parse_private2: missing begin marker >> debug1: read PEM private key done: type RSA >> debug1: Authentication succeeded (publickey). >> Authenticated to 172.31.41.33 ([172.31.41.33]:22). >> debug1: channel 0: new [client-session] >> debug1: Requesting no-more-sessions at openssh.com >> debug1: Entering interactive session. >> debug1: Sending environment. >> debug1: Sending env LANG = en_US.UTF-8 >> debug1: Sending command: scp -v -t /home/ubuntu/.ssh/authorized_keys2 >> Sending file modes: C0644 404 id_rsa.pub >> scp: /home/ubuntu/.ssh/authorized_keys2: Permission denied >> ubuntu at ip-172-31-41-32:~$ Sink: C0644 404 id_rsa.pub >> debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 >> debug1: channel 0: free: client-session, nchannels 1 >> debug1: fd 0 clearing O_NONBLOCK >> debug1: fd 1 clearing O_NONBLOCK >> Transferred: sent 3472, received 2636 bytes, in 0.2 seconds >> Bytes per second: sent 18676.3, received 14179.4 >> debug1: Exit status 1 >> >> ________________________________________ >> From: Daniel Thayer >> Sent: Monday, September 14, 2015 2:17 PM >> To: Davison, Charles Robert; bro at bro.org >> Subject: Re: [Bro] Bro Cluster Documentation Error >> >> When you check if you can ssh to the other machines in your cluster, >> you need to make sure you're running ssh as the same user that >> you're running broctl. >> >> Also, what did you specify for the "host=" entries in your node.cfg? >> >> >> >> On 09/14/2015 02:56 PM, Davison, Charles Robert wrote: >>> I assume attachments don't work... here is the test output: >>> >>> >>> root at ip-172-31-41-32:/home/ubuntu# export PATH=/usr/local/bro/bin:$PATH >>> >>> root at ip-172-31-41-32:/home/ubuntu# broctl >>> >>> Warning: broctl node config has changed (run the broctl "deploy" command) >>> >>> Warning: Bro node "bro" possibly still running on host "localhost" (PID >>> 16564) >>> >>> >>> Welcome to BroControl 1.4 >>> >>> >>> Type "help" for help. >>> >>> >>> [BroControl] > install >>> >>> removing old policies in >>> /usr/local/bro/spool/installed-scripts-do-not-touch/sit >>> e ... >>> >>> removing old policies in >>> /usr/local/bro/spool/installed-scripts-do-not-touch/aut >>> o ... >>> >>> creating policy directories ... >>> >>> installing site policies ... >>> >>> generating cluster-layout.bro ... >>> >>> generating local-networks.bro ... >>> >>> generating broctl-config.bro ... >>> >>> generating broctl-config.sh ... >>> >>> updating nodes ... >>> >>> Host key verification failed. >>> >>> Host key verification failed. >>> >>> Error: cannot create (some of the) directories >>> /usr/local/bro,/usr/local/bro/log >>> >>> s,/usr/local/bro/spool,/usr/local/bro/spool/tmp on node worker-1 >>> >>> [BroControl] > Host key verification failed. >>> >>> Host key verification failed. >>> >>> Host key verification failed. >>> >>> Host key verification failed. >>> >>> >>> >>> Thank you, >>> >>> Charles >>> >>> >>> ------------------------------------------------------------------------ >>> *From:* Davison, Charles Robert >>> *Sent:* Monday, September 14, 2015 1:41 PM >>> *To:* bro at bro.org >>> *Subject:* Bro Cluster Documentation Error >>> >>> >>> Good Afternoon, >>> >>> >>> I am trying to make documentation for installing a bro cluster >>> configuration, and receive the attached error when trying to install via >>> broctl. I can log into both of my worker nodes from the bro manager via >>> ssh fine, and without a password... >>> >>> >>> Thank you, >>> >>> Charles >>> >>> >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> From ascetik at gmail.com Mon Sep 14 23:27:30 2015 From: ascetik at gmail.com (josh summitt) Date: Tue, 15 Sep 2015 01:27:30 -0500 Subject: [Bro] Realtime File Extracting problem Message-ID: Hey i'm new to bro but have been attempting to use the file extracting features. I can generally get it to work but a lot of the time its just wrong when i attempt it in real time. For instance i'm downloading putty.exe and trying to extract it off the wire i get the below response when downloading it 5 times. It only successfully extracted and hashed it once: file_hash, FZKBS62fkHvKf36GTd, sha1, 91b21fffe934d856c43e35a388c78fccce7471ea The other times it completely misses it. If i attempt from a pcap file on the same machine it grabs it every time. Is there a threshold or something i need to set in bro for real time captures. /tmp$ sudo /usr/local/bro/bin/bro -i eth0 -C listening on eth0, capture length 8192 bytes new file, FB4np7nWhWIo8sOg5 file_hash, FB4np7nWhWIo8sOg5, sha1, 7788b3ba9a36112e0d429ecd358420d21ace7e68 new file, FxPYHc1et6sMSMY2jf <----- missed the file new file, FsONwVnUBjs2Fq0i5 file_hash, FsONwVnUBjs2Fq0i5, sha1, 7788b3ba9a36112e0d429ecd358420d21ace7e68 new file, FZKBS62fkHvKf36GTd <----- Yes it got the file file_hash, FZKBS62fkHvKf36GTd, sha1, 91b21fffe934d856c43e35a388c78fccce7471ea new file, Fp04jH3KL23Zx75OVf file_hash, Fp04jH3KL23Zx75OVf, sha1, 7788b3ba9a36112e0d429ecd358420d21ace7e68 new file, FK2LoX14jpBSyfpy67 <----- missed the file new file, FnJ7Mg1ymupibnvSW1 file_hash, FnJ7Mg1ymupibnvSW1, sha1, 7788b3ba9a36112e0d429ecd358420d21ace7e68 new file, FXriBu1tLEBhRVWTG3 <----- missed the file new file, FwByiJ30INM9Mk6DO9 file_hash, FwByiJ30INM9Mk6DO9, sha1, 7788b3ba9a36112e0d429ecd358420d21ace7e68 new file, Fn5DEA1WWvsykOA2Lh <----- missed the file ^C1442296477.139167 received termination signal 1442296477.139167 2260 packets received on interface eth0, 0 dropped Thanks Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150915/dd175cf2/attachment.html From vitologrillo at gmail.com Tue Sep 15 01:44:04 2015 From: vitologrillo at gmail.com (Vito Logrillo) Date: Tue, 15 Sep 2015 10:44:04 +0200 Subject: [Bro] PF_Ring and Bro - packet loss In-Reply-To: References: Message-ID: Which is the maximum slot number that can be handled by Bro? Thanks 2015-09-11 21:55 GMT+02:00 Adam Pumphrey : > Your broctl status output shows Bro is in standalone mode and not configured > to take advantage of pf_ring. You'll need to configure a local cluster with > the pf_ring specific options set for the monitoring interface/worker. > https://www.bro.org/sphinx-git/configuration/index.html#using-pf-ring > > Something like this might work in your case; notice the lb_method and > lb_procs settings for the worker: > > [manager] > type=manager > host=127.0.0.1 > > [proxy-1] > type=proxy > host=127.0.0.1 > > [worker-1] > type=worker > host=127.0.0.1 > interface=eth0 > lb_method=pf_ring > lb_procs=2 > > I think you need at least 2 lb_procs for pf_ring to provide any performance > gain. You can also set cpu affinity for the worker processes; this is > recommended for better performance and cluster stability, but not required. > If you do be sure to specify only physical cpu/core ID?s. I believe the > general rule of thumb is to leave half of your physical cores freed up for > the OS and other Bro processes. For example: > > pin_cpus=2,3 # assumes a single quad-core with core ID?s 0-3 > > There are a few more pf_ring specific options available in BroControl?s > config file. You can check them out here > https://www.bro.org/sphinx/components/broctl/README.html#pfringclusterid > > You might want to give ?broctl capstats? and the capture-loss Bro script a > try also; both are really helpful with troubleshooting traffic capture > issues. > > -Adam > > > On Sep 11, 2015, at 2:38 PM, nathanael rayborn > wrote: > > I'm experiencing high packet loss (15% -50%) with Bro 2.4 compiled with > PF_Ring. PFcount (pfcount -i eth0 -e 1) shows 0% packet loss while > /proc/net/pf_ring/PID shows the same number of dropped packets as broctl > netstats. The github link contains all changes and performance steps I've > taken so far along with output from PFcount, broctl, and ethtool. Has anyone > else experienced similar performance issues or have recommendations to get > my dropped packets as close to 0% as possible? Thanks > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From lagoon7 at gmail.com Tue Sep 15 03:57:00 2015 From: lagoon7 at gmail.com (Ludwig Goon) Date: Tue, 15 Sep 2015 06:57:00 -0400 Subject: [Bro] Compiling bro 2.4.1 on Ubuntu && ARM (HELP Please) Message-ID: Trying to compile bro 2.4.1 on linux. After setting up the packages for compile I run the ./configure scriipt and get the following error: -- Looking for include file pthread.h -- Looking for include file pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE CMake Error at doc/CMakeLists.txt:14 (message): Problem setting BROPATH -- Configuring incomplete, errors occurred! See also "/home/john/Projects/BRO-IDS/bro-2.4.1/build/CMakeFiles/CMakeOutput.log". See also "/home/john/Projects/BRO-IDS/bro-2.4.1/build/CMakeFiles/CMakeError.log". After looking at the CMakeError.log file it seems that it's looking for the pthreads library which is defined by -lpthreads Can't find that exact library for posix threads however here are the ones in ubuntu root at merovingian:/usr/lib/x86_64-linux-gnu# aptitude search pthread i libevent-pthreads-2.0-5 - Asynchronous event notification library (pthreads) p libevent-pthreads-2.0-5:i386 - Asynchronous event notification library (pthreads) i libpthread-stubs0-dev - pthread stubs not provided by native libc, development files p libpthread-stubs0-dev:i386 - pthread stubs not provided by native libc, development files p libpthread-workqueue-dev - thread pool library (development files) p libpthread-workqueue-dev:i386 - thread pool library (development files) p libpthread-workqueue0 - thread pool library p libpthread-workqueue0:i386 - thread pool library SO can I adjust CMake to use -lpthread or similar or tell CMake to skip this overall? HELP please!!! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150915/719f2dc6/attachment.html From doug.burks at gmail.com Tue Sep 15 04:29:09 2015 From: doug.burks at gmail.com (Doug Burks) Date: Tue, 15 Sep 2015 07:29:09 -0400 Subject: [Bro] Realtime File Extracting problem In-Reply-To: References: Message-ID: Hi Josh, Have you verified that all NIC offloading functions are disabled? http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html On Tue, Sep 15, 2015 at 2:27 AM, josh summitt wrote: > Hey i'm new to bro but have been attempting to use the file extracting > features. I can generally get it to work but a lot of the time its just > wrong when i attempt it in real time. > > For instance i'm downloading putty.exe and trying to extract it off the wire > i get the below response when downloading it 5 times. It only successfully > extracted and hashed it once: > file_hash, FZKBS62fkHvKf36GTd, sha1, > 91b21fffe934d856c43e35a388c78fccce7471ea > > The other times it completely misses it. If i attempt from a pcap file on > the same machine it grabs it every time. Is there a threshold or something i > need to set in bro for real time captures. > > /tmp$ sudo /usr/local/bro/bin/bro -i eth0 -C > > listening on eth0, capture length 8192 bytes > > > new file, FB4np7nWhWIo8sOg5 > > file_hash, FB4np7nWhWIo8sOg5, sha1, 7788b3ba9a36112e0d429ecd358420d21ace7e68 > > new file, FxPYHc1et6sMSMY2jf <----- missed the file > > new file, FsONwVnUBjs2Fq0i5 > > file_hash, FsONwVnUBjs2Fq0i5, sha1, 7788b3ba9a36112e0d429ecd358420d21ace7e68 > > new file, FZKBS62fkHvKf36GTd <----- Yes it got the file > > file_hash, FZKBS62fkHvKf36GTd, sha1, > 91b21fffe934d856c43e35a388c78fccce7471ea > > new file, Fp04jH3KL23Zx75OVf > > file_hash, Fp04jH3KL23Zx75OVf, sha1, > 7788b3ba9a36112e0d429ecd358420d21ace7e68 > > new file, FK2LoX14jpBSyfpy67 <----- missed the file > > new file, FnJ7Mg1ymupibnvSW1 > > file_hash, FnJ7Mg1ymupibnvSW1, sha1, > 7788b3ba9a36112e0d429ecd358420d21ace7e68 > > new file, FXriBu1tLEBhRVWTG3 <----- missed the file > > new file, FwByiJ30INM9Mk6DO9 > > file_hash, FwByiJ30INM9Mk6DO9, sha1, > 7788b3ba9a36112e0d429ecd358420d21ace7e68 > > new file, Fn5DEA1WWvsykOA2Lh <----- missed the file > > ^C1442296477.139167 received termination signal > > 1442296477.139167 2260 packets received on interface eth0, 0 dropped > > > > > Thanks > Josh > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Doug Burks Need Security Onion Training or Commercial Support? http://securityonionsolutions.com From cdaviso1 at vols.utk.edu Tue Sep 15 06:33:12 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Tue, 15 Sep 2015 13:33:12 +0000 Subject: [Bro] Bro Cluster Documentation Error In-Reply-To: <55F7429D.5060101@illinois.edu> References: , , <55F72B77.8030905@illinois.edu> , <55F73ED3.3080007@illinois.edu> , <55F7429D.5060101@illinois.edu> Message-ID: This fixed it after i applied it to the manager and all the nodes! sudo chown -R ubuntu:ubuntu /usr/local/bro Thanks for your help. ________________________________________ From: Daniel Thayer Sent: Monday, September 14, 2015 3:56 PM To: Davison, Charles Robert; bro at bro.org Subject: Re: [Bro] Bro Cluster Documentation Error The error message for the lock issue is "Permission denied", so you will need to check whether your "ubuntu" user has permission to write to the /usr/local/bro/spool/ directory. On 09/14/2015 04:45 PM, Davison, Charles Robert wrote: > I tried running bro from by ubuntu account and recieve this: > > Also for whatever reason I have to constantly export my paths to run broctl. Not a big issue but if you know a fix that would be great. > > ubuntu at ip-172-31-41-32:~$ export PATH=/usr/local/bro/bin:$PATH > ubuntu at ip-172-31-41-32:~$ broctl > Warning: broctl node config has changed (run the broctl "deploy" command) > Warning: Bro node "bro" possibly still running on host "localhost" (PID 16564) > > Welcome to BroControl 1.4 > > Type "help" for help. > > [BroControl] > install > Error: cannot acquire lock: [Errno 13] Permission denied: '/usr/local/bro/spool/lock.27491' > Error: Unable to get lock > [BroControl] > > > > ________________________________________ > From: Daniel Thayer > Sent: Monday, September 14, 2015 3:40 PM > To: Davison, Charles Robert; bro at bro.org > Subject: Re: [Bro] Bro Cluster Documentation Error > > In screenshot in previous email, it appeared you were running broctl > as the "root" user. If that's the case, then you need to be able > to ssh to your worker machine as the "root" user. The home > directory of the "root" user is probably "/root". > > > > On 09/14/2015 04:24 PM, Davison, Charles Robert wrote: >> Please see the attached document on how configured the host entries. >> >> I can ssh into the computers as my ubuntu user fine, but I copied over my keys as follows: >> >> scp -v ~/.ssh/id_rsa.pub root at 172.31.41.31:/home/ubuntu/.ssh/authorized_keys2 >> scp -v ~/.ssh/id_rsa.pub root at 172.31.41.33:/home/ubuntu/.ssh/authorized_keys2 >> >> Is this an issue? I tried using ubuntu as the user and it hangs: >> >> ubuntu at ip-172-31-41-32:~$ scp -v ~/.ssh/id_rsa.pub ubuntu at 172.31.41.33:/home/ubuntu/.ssh/authorized_keys2 >> Executing: program /usr/bin/ssh host 172.31.41.33, user ubuntu, command scp -v -t /home/ubuntu/.ssh/authorized_keys2 >> OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014 >> debug1: Reading configuration data /etc/ssh/ssh_config >> debug1: /etc/ssh/ssh_config line 19: Applying options for * >> debug1: Connecting to 172.31.41.33 [172.31.41.33] port 22. >> debug1: Connection established. >> debug1: identity file /home/ubuntu/.ssh/id_rsa type 1 >> debug1: identity file /home/ubuntu/.ssh/id_rsa-cert type -1 >> debug1: identity file /home/ubuntu/.ssh/id_dsa type -1 >> debug1: identity file /home/ubuntu/.ssh/id_dsa-cert type -1 >> debug1: identity file /home/ubuntu/.ssh/id_ecdsa type -1 >> debug1: identity file /home/ubuntu/.ssh/id_ecdsa-cert type -1 >> debug1: identity file /home/ubuntu/.ssh/id_ed25519 type -1 >> debug1: identity file /home/ubuntu/.ssh/id_ed25519-cert type -1 >> debug1: Enabling compatibility mode for protocol 2.0 >> debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 >> debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 >> debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 pat OpenSSH_6.6.1* compat 0x04000000 >> debug1: SSH2_MSG_KEXINIT sent >> debug1: SSH2_MSG_KEXINIT received >> debug1: kex: server->client aes128-ctr hmac-md5-etm at openssh.com none >> debug1: kex: client->server aes128-ctr hmac-md5-etm at openssh.com none >> debug1: sending SSH2_MSG_KEX_ECDH_INIT >> debug1: expecting SSH2_MSG_KEX_ECDH_REPLY >> debug1: Server host key: ECDSA d1:0a:e6:c3:bf:ee:23:5a:63:63:ce:c8:71:41:88:29 >> debug1: Host '172.31.41.33' is known and matches the ECDSA host key. >> debug1: Found key in /home/ubuntu/.ssh/known_hosts:2 >> debug1: ssh_ecdsa_verify: signature correct >> debug1: SSH2_MSG_NEWKEYS sent >> debug1: expecting SSH2_MSG_NEWKEYS >> debug1: SSH2_MSG_NEWKEYS received >> debug1: Roaming not allowed by server >> debug1: SSH2_MSG_SERVICE_REQUEST sent >> debug1: SSH2_MSG_SERVICE_ACCEPT received >> debug1: Authentications that can continue: publickey,password >> debug1: Next authentication method: publickey >> debug1: Offering RSA public key: /home/ubuntu/.ssh/id_rsa >> debug1: Server accepts key: pkalg ssh-rsa blen 279 >> debug1: key_parse_private2: missing begin marker >> debug1: read PEM private key done: type RSA >> debug1: Authentication succeeded (publickey). >> Authenticated to 172.31.41.33 ([172.31.41.33]:22). >> debug1: channel 0: new [client-session] >> debug1: Requesting no-more-sessions at openssh.com >> debug1: Entering interactive session. >> debug1: Sending environment. >> debug1: Sending env LANG = en_US.UTF-8 >> debug1: Sending command: scp -v -t /home/ubuntu/.ssh/authorized_keys2 >> Sending file modes: C0644 404 id_rsa.pub >> scp: /home/ubuntu/.ssh/authorized_keys2: Permission denied >> ubuntu at ip-172-31-41-32:~$ Sink: C0644 404 id_rsa.pub >> debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 >> debug1: channel 0: free: client-session, nchannels 1 >> debug1: fd 0 clearing O_NONBLOCK >> debug1: fd 1 clearing O_NONBLOCK >> Transferred: sent 3472, received 2636 bytes, in 0.2 seconds >> Bytes per second: sent 18676.3, received 14179.4 >> debug1: Exit status 1 >> >> ________________________________________ >> From: Daniel Thayer >> Sent: Monday, September 14, 2015 2:17 PM >> To: Davison, Charles Robert; bro at bro.org >> Subject: Re: [Bro] Bro Cluster Documentation Error >> >> When you check if you can ssh to the other machines in your cluster, >> you need to make sure you're running ssh as the same user that >> you're running broctl. >> >> Also, what did you specify for the "host=" entries in your node.cfg? >> >> >> >> On 09/14/2015 02:56 PM, Davison, Charles Robert wrote: >>> I assume attachments don't work... here is the test output: >>> >>> >>> root at ip-172-31-41-32:/home/ubuntu# export PATH=/usr/local/bro/bin:$PATH >>> >>> root at ip-172-31-41-32:/home/ubuntu# broctl >>> >>> Warning: broctl node config has changed (run the broctl "deploy" command) >>> >>> Warning: Bro node "bro" possibly still running on host "localhost" (PID >>> 16564) >>> >>> >>> Welcome to BroControl 1.4 >>> >>> >>> Type "help" for help. >>> >>> >>> [BroControl] > install >>> >>> removing old policies in >>> /usr/local/bro/spool/installed-scripts-do-not-touch/sit >>> e ... >>> >>> removing old policies in >>> /usr/local/bro/spool/installed-scripts-do-not-touch/aut >>> o ... >>> >>> creating policy directories ... >>> >>> installing site policies ... >>> >>> generating cluster-layout.bro ... >>> >>> generating local-networks.bro ... >>> >>> generating broctl-config.bro ... >>> >>> generating broctl-config.sh ... >>> >>> updating nodes ... >>> >>> Host key verification failed. >>> >>> Host key verification failed. >>> >>> Error: cannot create (some of the) directories >>> /usr/local/bro,/usr/local/bro/log >>> >>> s,/usr/local/bro/spool,/usr/local/bro/spool/tmp on node worker-1 >>> >>> [BroControl] > Host key verification failed. >>> >>> Host key verification failed. >>> >>> Host key verification failed. >>> >>> Host key verification failed. >>> >>> >>> >>> Thank you, >>> >>> Charles >>> >>> >>> ------------------------------------------------------------------------ >>> *From:* Davison, Charles Robert >>> *Sent:* Monday, September 14, 2015 1:41 PM >>> *To:* bro at bro.org >>> *Subject:* Bro Cluster Documentation Error >>> >>> >>> Good Afternoon, >>> >>> >>> I am trying to make documentation for installing a bro cluster >>> configuration, and receive the attached error when trying to install via >>> broctl. I can log into both of my worker nodes from the bro manager via >>> ssh fine, and without a password... >>> >>> >>> Thank you, >>> >>> Charles >>> >>> >>> >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> From cdaviso1 at vols.utk.edu Tue Sep 15 06:41:11 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Tue, 15 Sep 2015 13:41:11 +0000 Subject: [Bro] Broctl Worker Issues Message-ID: When I try and start broctl on all my workers I receive the following: ubuntu at ip-172-31-41-32:~$ /usr/local/bro/bin/broctl start starting manager ... starting proxy-1 ... starting worker-1 ... starting worker-2 ... worker-1 terminated immediately after starting; check output with "diag" worker-2 terminated immediately after starting; check output with "diag" This was my output from the diag: Bro 2.4.1 Linux 3.13.0-48-generic No gdb installed. ==== reporter.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path reporter #open 2015-09-15-13-38-43 #fields ts level message location #types time enum string string 0.000000 Reporter::WARNING SumStat key request for the J1pRzdrrLK8 SumStat uid took longer than 1 minute and was automatically cancelled. /usr/local/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line 218 ==== stderr.log ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto ==== .env_vars PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site CLUSTER_NODE=manager ==== .status RUNNING [net_run] ==== No prof.log ==== No packet_filter.log ==== loaded_scripts.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path loaded_scripts #open 2015-09-15-13-34-43 #fields name #types string /usr/local/bro/share/bro/base/init-bare.bro /usr/local/bro/share/bro/base/bif/const.bif.bro /usr/local/bro/share/bro/base/bif/types.bif.bro /usr/local/bro/share/bro/base/bif/strings.bif.bro /usr/local/bro/share/bro/base/bif/bro.bif.bro /usr/local/bro/share/bro/base/bif/reporter.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SNMP.types.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_KRB.types.bif.bro /usr/local/bro/share/bro/base/bif/event.bif.bro /usr/local/bro/share/bro/base/frameworks/broker/__load__.bro /usr/local/bro/share/bro/base/frameworks/broker/main.bro /usr/local/bro/share/bro/base/frameworks/logging/__load__.bro /usr/local/bro/share/bro/base/frameworks/logging/main.bro /usr/local/bro/share/bro/base/bif/logging.bif.bro /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/__load__.bro /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/scp.bro /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/sftp.bro /usr/local/bro/share/bro/base/frameworks/logging/writers/ascii.bro /usr/local/bro/share/bro/base/frameworks/logging/writers/sqlite.bro /usr/local/bro/share/bro/base/frameworks/logging/writers/none.bro /usr/local/bro/share/bro/base/frameworks/input/__load__.bro /usr/local/bro/share/bro/base/frameworks/input/main.bro /usr/local/bro/share/bro/base/bif/input.bif.bro /usr/local/bro/share/bro/base/frameworks/input/readers/ascii.bro /usr/local/bro/share/bro/base/frameworks/input/readers/raw.bro /usr/local/bro/share/bro/base/frameworks/input/readers/benchmark.bro /usr/local/bro/share/bro/base/frameworks/input/readers/binary.bro /usr/local/bro/share/bro/base/frameworks/input/readers/sqlite.bro /usr/local/bro/share/bro/base/frameworks/analyzer/__load__.bro /usr/local/bro/share/bro/base/frameworks/analyzer/main.bro /usr/local/bro/share/bro/base/frameworks/packet-filter/utils.bro /usr/local/bro/share/bro/base/bif/analyzer.bif.bro /usr/local/bro/share/bro/base/frameworks/files/__load__.bro /usr/local/bro/share/bro/base/frameworks/files/main.bro /usr/local/bro/share/bro/base/bif/file_analysis.bif.bro /usr/local/bro/share/bro/base/utils/site.bro /usr/local/bro/share/bro/base/utils/patterns.bro /usr/local/bro/share/bro/base/frameworks/files/magic/__load__.bro /usr/local/bro/share/bro/base/bif/__load__.bro /usr/local/bro/share/bro/base/bif/broxygen.bif.bro /usr/local/bro/share/bro/base/bif/pcap.bif.bro /usr/local/bro/share/bro/base/bif/bloom-filter.bif.bro /usr/local/bro/share/bro/base/bif/cardinality-counter.bif.bro /usr/local/bro/share/bro/base/bif/top-k.bif.bro /usr/local/bro/share/bro/base/bif/comm.bif.bro /usr/local/bro/share/bro/base/bif/data.bif.bro /usr/local/bro/share/bro/base/bif/messaging.bif.bro /usr/local/bro/share/bro/base/bif/store.bif.bro /usr/local/bro/share/bro/base/bif/plugins/__load__.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_ARP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_AYIYA.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_BackDoor.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_BitTorrent.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_ConnSize.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_ConnSize.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_DCE_RPC.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_DHCP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_DNP3.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_DNS.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_File.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Finger.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_FTP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_FTP.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Gnutella.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_GTPv1.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_HTTP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_HTTP.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_ICMP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Ident.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_InterConn.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_IRC.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_KRB.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Login.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Login.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_MIME.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Modbus.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_MySQL.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_NCP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_NetBIOS.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_NetBIOS.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_NTP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_PIA.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_POP3.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_RADIUS.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_RDP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_RDP.types.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_RPC.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SIP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SNMP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SMB.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SMTP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SMTP.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SOCKS.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SSH.types.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SSH.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SSL.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SteppingStone.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Syslog.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_TCP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_TCP.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Teredo.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_UDP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_ZIP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_FileExtract.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_FileExtract.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_FileHash.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_PE.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Unified2.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Unified2.types.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.types.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_AsciiReader.ascii.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_BenchmarkReader.benchmark.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_BinaryReader.binary.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_RawReader.raw.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SQLiteReader.sqlite.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_AsciiWriter.ascii.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_NoneWriter.none.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SQLiteWriter.sqlite.bif.bro /usr/local/bro/share/bro/base/init-default.bro /usr/local/bro/share/bro/base/utils/active-http.bro /usr/local/bro/share/bro/base/utils/exec.bro /usr/local/bro/share/bro/base/utils/addrs.bro /usr/local/bro/share/bro/base/utils/conn-ids.bro /usr/local/bro/share/bro/base/utils/dir.bro /usr/local/bro/share/bro/base/frameworks/reporter/__load__.bro /usr/local/bro/share/bro/base/frameworks/reporter/main.bro /usr/local/bro/share/bro/base/utils/paths.bro /usr/local/bro/share/bro/base/utils/directions-and-hosts.bro /usr/local/bro/share/bro/base/utils/files.bro /usr/local/bro/share/bro/base/utils/numbers.bro /usr/local/bro/share/bro/base/utils/queue.bro /usr/local/bro/share/bro/base/utils/strings.bro /usr/local/bro/share/bro/base/utils/thresholds.bro /usr/local/bro/share/bro/base/utils/time.bro /usr/local/bro/share/bro/base/utils/urls.bro /usr/local/bro/share/bro/base/frameworks/notice/__load__.bro /usr/local/bro/share/bro/base/frameworks/notice/main.bro /usr/local/bro/share/bro/base/frameworks/notice/weird.bro /usr/local/bro/share/bro/base/frameworks/notice/actions/drop.bro /usr/local/bro/share/bro/base/frameworks/notice/actions/email_admin.bro /usr/local/bro/share/bro/base/frameworks/notice/actions/page.bro /usr/local/bro/share/bro/base/frameworks/notice/actions/add-geodata.bro /usr/local/bro/share/bro/base/frameworks/notice/extend-email/hostnames.bro /usr/local/bro/share/bro/base/frameworks/cluster/__load__.bro /usr/local/bro/share/bro/base/frameworks/cluster/main.bro /usr/local/bro/share/bro/base/frameworks/control/__load__.bro /usr/local/bro/share/bro/base/frameworks/control/main.bro /usr/local/bro/spool/installed-scripts-do-not-touch/auto/cluster-layout.bro /usr/local/bro/share/bro/base/frameworks/cluster/setup-connections.bro /usr/local/bro/share/bro/base/frameworks/communication/__load__.bro /usr/local/bro/share/bro/base/frameworks/communication/main.bro /usr/local/bro/share/bro/base/frameworks/packet-filter/__load__.bro /usr/local/bro/share/bro/base/frameworks/packet-filter/main.bro /usr/local/bro/share/bro/base/frameworks/packet-filter/netstats.bro /usr/local/bro/share/bro/base/frameworks/packet-filter/cluster.bro /usr/local/bro/share/bro/policy/frameworks/communication/listen.bro /usr/local/bro/share/bro/base/frameworks/cluster/nodes/manager.bro /usr/local/bro/share/bro/base/frameworks/notice/cluster.bro /usr/local/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro /usr/local/bro/share/bro/base/frameworks/dpd/__load__.bro /usr/local/bro/share/bro/base/frameworks/dpd/main.bro /usr/local/bro/share/bro/base/frameworks/signatures/__load__.bro /usr/local/bro/share/bro/base/frameworks/signatures/main.bro /usr/local/bro/share/bro/base/frameworks/software/__load__.bro /usr/local/bro/share/bro/base/frameworks/software/main.bro /usr/local/bro/share/bro/base/frameworks/intel/__load__.bro /usr/local/bro/share/bro/base/frameworks/intel/main.bro /usr/local/bro/share/bro/base/frameworks/intel/cluster.bro /usr/local/bro/share/bro/base/frameworks/intel/input.bro /usr/local/bro/share/bro/base/frameworks/sumstats/__load__.bro /usr/local/bro/share/bro/base/frameworks/sumstats/main.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/__load__.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/average.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/hll_unique.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/last.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/max.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/min.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/sample.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/std-dev.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/variance.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/sum.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/topk.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/unique.bro /usr/local/bro/share/bro/base/frameworks/sumstats/cluster.bro /usr/local/bro/share/bro/base/frameworks/tunnels/__load__.bro /usr/local/bro/share/bro/base/frameworks/tunnels/main.bro /usr/local/bro/share/bro/base/protocols/conn/__load__.bro /usr/local/bro/share/bro/base/protocols/conn/main.bro /usr/local/bro/share/bro/base/protocols/conn/contents.bro /usr/local/bro/share/bro/base/protocols/conn/inactivity.bro /usr/local/bro/share/bro/base/protocols/conn/polling.bro /usr/local/bro/share/bro/base/protocols/conn/thresholds.bro /usr/local/bro/share/bro/base/protocols/dhcp/__load__.bro /usr/local/bro/share/bro/base/protocols/dhcp/consts.bro /usr/local/bro/share/bro/base/protocols/dhcp/main.bro /usr/local/bro/share/bro/base/protocols/dhcp/utils.bro /usr/local/bro/share/bro/base/protocols/dnp3/__load__.bro /usr/local/bro/share/bro/base/protocols/dnp3/main.bro /usr/local/bro/share/bro/base/protocols/dnp3/consts.bro /usr/local/bro/share/bro/base/protocols/dns/__load__.bro /usr/local/bro/share/bro/base/protocols/dns/consts.bro /usr/local/bro/share/bro/base/protocols/dns/main.bro /usr/local/bro/share/bro/base/protocols/ftp/__load__.bro /usr/local/bro/share/bro/base/protocols/ftp/utils-commands.bro /usr/local/bro/share/bro/base/protocols/ftp/info.bro /usr/local/bro/share/bro/base/protocols/ftp/main.bro /usr/local/bro/share/bro/base/protocols/ftp/utils.bro /usr/local/bro/share/bro/base/protocols/ftp/files.bro /usr/local/bro/share/bro/base/protocols/ftp/gridftp.bro /usr/local/bro/share/bro/base/protocols/ssl/__load__.bro /usr/local/bro/share/bro/base/protocols/ssl/consts.bro /usr/local/bro/share/bro/base/protocols/ssl/main.bro /usr/local/bro/share/bro/base/protocols/ssl/mozilla-ca-list.bro /usr/local/bro/share/bro/base/protocols/ssl/files.bro /usr/local/bro/share/bro/base/files/x509/__load__.bro /usr/local/bro/share/bro/base/files/x509/main.bro /usr/local/bro/share/bro/base/files/hash/__load__.bro /usr/local/bro/share/bro/base/files/hash/main.bro /usr/local/bro/share/bro/base/protocols/http/__load__.bro /usr/local/bro/share/bro/base/protocols/http/main.bro /usr/local/bro/share/bro/base/protocols/http/entities.bro /usr/local/bro/share/bro/base/protocols/http/utils.bro /usr/local/bro/share/bro/base/protocols/http/files.bro /usr/local/bro/share/bro/base/protocols/irc/__load__.bro /usr/local/bro/share/bro/base/protocols/irc/main.bro /usr/local/bro/share/bro/base/protocols/irc/dcc-send.bro /usr/local/bro/share/bro/base/protocols/irc/files.bro /usr/local/bro/share/bro/base/protocols/krb/__load__.bro /usr/local/bro/share/bro/base/protocols/krb/main.bro /usr/local/bro/share/bro/base/protocols/krb/consts.bro /usr/local/bro/share/bro/base/protocols/krb/files.bro /usr/local/bro/share/bro/base/protocols/modbus/__load__.bro /usr/local/bro/share/bro/base/protocols/modbus/consts.bro /usr/local/bro/share/bro/base/protocols/modbus/main.bro /usr/local/bro/share/bro/base/protocols/mysql/__load__.bro /usr/local/bro/share/bro/base/protocols/mysql/main.bro /usr/local/bro/share/bro/base/protocols/mysql/consts.bro /usr/local/bro/share/bro/base/protocols/pop3/__load__.bro /usr/local/bro/share/bro/base/protocols/radius/__load__.bro /usr/local/bro/share/bro/base/protocols/radius/main.bro /usr/local/bro/share/bro/base/protocols/radius/consts.bro /usr/local/bro/share/bro/base/protocols/rdp/__load__.bro /usr/local/bro/share/bro/base/protocols/rdp/consts.bro /usr/local/bro/share/bro/base/protocols/rdp/main.bro /usr/local/bro/share/bro/base/protocols/sip/__load__.bro /usr/local/bro/share/bro/base/protocols/sip/main.bro /usr/local/bro/share/bro/base/protocols/snmp/__load__.bro /usr/local/bro/share/bro/base/protocols/snmp/main.bro /usr/local/bro/share/bro/base/protocols/smtp/__load__.bro /usr/local/bro/share/bro/base/protocols/smtp/main.bro /usr/local/bro/share/bro/base/protocols/smtp/entities.bro /usr/local/bro/share/bro/base/protocols/smtp/files.bro /usr/local/bro/share/bro/base/protocols/socks/__load__.bro /usr/local/bro/share/bro/base/protocols/socks/consts.bro /usr/local/bro/share/bro/base/protocols/socks/main.bro /usr/local/bro/share/bro/base/protocols/ssh/__load__.bro /usr/local/bro/share/bro/base/protocols/ssh/main.bro /usr/local/bro/share/bro/base/protocols/syslog/__load__.bro /usr/local/bro/share/bro/base/protocols/syslog/consts.bro /usr/local/bro/share/bro/base/protocols/syslog/main.bro /usr/local/bro/share/bro/base/protocols/tunnels/__load__.bro /usr/local/bro/share/bro/base/files/pe/__load__.bro /usr/local/bro/share/bro/base/files/pe/consts.bro /usr/local/bro/share/bro/base/files/pe/main.bro /usr/local/bro/share/bro/base/files/extract/__load__.bro /usr/local/bro/share/bro/base/files/extract/main.bro /usr/local/bro/share/bro/base/files/unified2/__load__.bro /usr/local/bro/share/bro/base/files/unified2/main.bro /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro /usr/local/bro/share/bro/base/misc/find-filtered-trace.bro /usr/local/bro/spool/installed-scripts-do-not-touch/site/local.bro /usr/local/bro/share/bro/policy/misc/loaded-scripts.bro /usr/local/bro/share/bro/policy/tuning/defaults/__load__.bro /usr/local/bro/share/bro/policy/tuning/defaults/packet-fragments.bro /usr/local/bro/share/bro/policy/tuning/defaults/warnings.bro /usr/local/bro/share/bro/policy/tuning/defaults/extracted_file_limits.bro /usr/local/bro/share/bro/policy/misc/scan.bro /usr/local/bro/share/bro/policy/misc/app-stats/__load__.bro /usr/local/bro/share/bro/policy/misc/app-stats/main.bro /usr/local/bro/share/bro/policy/misc/app-stats/plugins/__load__.bro /usr/local/bro/share/bro/policy/misc/app-stats/plugins/facebook.bro /usr/local/bro/share/bro/policy/misc/detect-traceroute/__load__.bro /usr/local/bro/share/bro/policy/misc/detect-traceroute/main.bro /usr/local/bro/share/bro/policy/frameworks/software/vulnerable.bro /usr/local/bro/share/bro/policy/frameworks/software/version-changes.bro /usr/local/bro/share/bro/policy/protocols/ftp/software.bro /usr/local/bro/share/bro/policy/protocols/smtp/software.bro /usr/local/bro/share/bro/policy/protocols/ssh/software.bro /usr/local/bro/share/bro/policy/protocols/http/software.bro /usr/local/bro/share/bro/policy/protocols/dns/detect-external-names.bro /usr/local/bro/share/bro/policy/protocols/ftp/detect.bro /usr/local/bro/share/bro/policy/protocols/conn/known-hosts.bro /usr/local/bro/share/bro/policy/protocols/conn/known-services.bro /usr/local/bro/share/bro/policy/protocols/ssl/known-certs.bro /usr/local/bro/share/bro/policy/protocols/ssl/validate-certs.bro /usr/local/bro/share/bro/policy/protocols/ssl/log-hostcerts-only.bro /usr/local/bro/share/bro/policy/protocols/ssh/geo-data.bro /usr/local/bro/share/bro/policy/protocols/ssh/detect-bruteforcing.bro /usr/local/bro/share/bro/policy/protocols/ssh/interesting-hostnames.bro /usr/local/bro/share/bro/policy/protocols/http/detect-sqli.bro /usr/local/bro/share/bro/policy/frameworks/files/hash-all-files.bro /usr/local/bro/share/bro/policy/frameworks/files/detect-MHR.bro /usr/local/bro/share/bro/broctl/__load__.bro /usr/local/bro/share/bro/broctl/main.bro /usr/local/bro/share/bro/policy/frameworks/control/controllee.bro /usr/local/bro/spool/installed-scripts-do-not-touch/site/local-manager.bro /usr/local/bro/share/bro/broctl/auto.bro /usr/local/bro/spool/installed-scripts-do-not-touch/auto/local-networks.bro /usr/local/bro/spool/installed-scripts-do-not-touch/auto/broctl-config.bro [proxy-1] Bro 2.4.1 Linux 3.13.0-48-generic No gdb installed. ==== No reporter.log ==== stderr.log ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -U .status -p broctl -p broctl-live -p local -p proxy-1 local.bro broctl base/frameworks/cluster local-proxy broctl/auto ==== .env_vars PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site CLUSTER_NODE=proxy-1 ==== .status RUNNING [net_run] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log [worker-1] error running crash-diag for worker-1 Host 172.31.41.33 is not alive [worker-2] error running crash-diag for worker-2 Host 172.31.41.31 is not alive -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150915/819f49ea/attachment-0001.html From ascetik at gmail.com Tue Sep 15 06:49:15 2015 From: ascetik at gmail.com (josh summitt) Date: Tue, 15 Sep 2015 08:49:15 -0500 Subject: [Bro] Realtime File Extracting problem In-Reply-To: References: Message-ID: YES! this seems to fix my issue. I had partially turned off offloaded with ethtool -K p1p2 tx off rx off but i didn't have the other options turned off as discussed in that blog post... These options below fixed my issue: ethtool -K p1p2 rx off ethtool -K p1p2 tx off ethtool -K p1p2 sg off ethtool -K p1p2 tso off ethtool -K p1p2 ufo off ethtool -K p1p2 gso off ethtool -K p1p2 gro off ethtool -K p1p2 lro off Thanks Doug!!! Thanks Josh On Tue, Sep 15, 2015 at 6:29 AM, Doug Burks wrote: > Hi Josh, > > Have you verified that all NIC offloading functions are disabled? > > http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html > > On Tue, Sep 15, 2015 at 2:27 AM, josh summitt wrote: > > Hey i'm new to bro but have been attempting to use the file extracting > > features. I can generally get it to work but a lot of the time its just > > wrong when i attempt it in real time. > > > > For instance i'm downloading putty.exe and trying to extract it off the > wire > > i get the below response when downloading it 5 times. It only > successfully > > extracted and hashed it once: > > file_hash, FZKBS62fkHvKf36GTd, sha1, > > 91b21fffe934d856c43e35a388c78fccce7471ea > > > > The other times it completely misses it. If i attempt from a pcap file on > > the same machine it grabs it every time. Is there a threshold or > something i > > need to set in bro for real time captures. > > > > /tmp$ sudo /usr/local/bro/bin/bro -i eth0 -C > > > > listening on eth0, capture length 8192 bytes > > > > > > new file, FB4np7nWhWIo8sOg5 > > > > file_hash, FB4np7nWhWIo8sOg5, sha1, > 7788b3ba9a36112e0d429ecd358420d21ace7e68 > > > > new file, FxPYHc1et6sMSMY2jf <----- missed the file > > > > new file, FsONwVnUBjs2Fq0i5 > > > > file_hash, FsONwVnUBjs2Fq0i5, sha1, > 7788b3ba9a36112e0d429ecd358420d21ace7e68 > > > > new file, FZKBS62fkHvKf36GTd <----- Yes it got the file > > > > file_hash, FZKBS62fkHvKf36GTd, sha1, > > 91b21fffe934d856c43e35a388c78fccce7471ea > > > > new file, Fp04jH3KL23Zx75OVf > > > > file_hash, Fp04jH3KL23Zx75OVf, sha1, > > 7788b3ba9a36112e0d429ecd358420d21ace7e68 > > > > new file, FK2LoX14jpBSyfpy67 <----- missed the file > > > > new file, FnJ7Mg1ymupibnvSW1 > > > > file_hash, FnJ7Mg1ymupibnvSW1, sha1, > > 7788b3ba9a36112e0d429ecd358420d21ace7e68 > > > > new file, FXriBu1tLEBhRVWTG3 <----- missed the file > > > > new file, FwByiJ30INM9Mk6DO9 > > > > file_hash, FwByiJ30INM9Mk6DO9, sha1, > > 7788b3ba9a36112e0d429ecd358420d21ace7e68 > > > > new file, Fn5DEA1WWvsykOA2Lh <----- missed the file > > > > ^C1442296477.139167 received termination signal > > > > 1442296477.139167 2260 packets received on interface eth0, 0 dropped > > > > > > > > > > Thanks > > Josh > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > -- > Doug Burks > Need Security Onion Training or Commercial Support? > http://securityonionsolutions.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150915/1e636916/attachment.html From cdaviso1 at vols.utk.edu Tue Sep 15 06:54:07 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Tue, 15 Sep 2015 13:54:07 +0000 Subject: [Bro] Bro Digest, Vol 113, Issue 21 In-Reply-To: References: Message-ID: This was the status I received: [BroControl] > status Getting process status ... Getting peer status ... Name Type Host Status Pid Peers Started manager manager 172.31.41.32 running 2405 0 15 Sep 13:43:53 proxy-1 proxy 172.31.41.32 running 2444 1 15 Sep 13:43:54 worker-1 worker 172.31.41.33 crashed worker-2 worker 172.31.41.31 crashed I also tried performing the following, but no matter what whenever I start i recieve the termination error. broctl stop > broctl cleanup --all > broctl install > broctl check > broctl start [BroControl] > start manager still running proxy-1 still running starting worker-1 (was crashed) ... starting worker-2 (was crashed) ... worker-1 terminated immediately after starting; check output with "diag" worker-2 terminated immediately after starting; check output with "diag" CHARLES R. DAVISON (865)730-0078 cdaviso1 at vols.utk.edu ________________________________________ From: bro-bounces at bro.org on behalf of bro-request at bro.org Sent: Tuesday, September 15, 2015 7:41 AM To: bro at bro.org Subject: Bro Digest, Vol 113, Issue 21 Send Bro mailing list submissions to bro at bro.org To subscribe or unsubscribe via the World Wide Web, visit http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro or, via email, send a message with subject or body 'help' to bro-request at bro.org You can reach the person managing the list at bro-owner at bro.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Bro digest..." Today's Topics: 1. Broctl Worker Issues (Davison, Charles Robert) ---------------------------------------------------------------------- Message: 1 Date: Tue, 15 Sep 2015 13:41:11 +0000 From: "Davison, Charles Robert" Subject: [Bro] Broctl Worker Issues To: "bro at bro.org" Message-ID: Content-Type: text/plain; charset="iso-8859-1" When I try and start broctl on all my workers I receive the following: ubuntu at ip-172-31-41-32:~$ /usr/local/bro/bin/broctl start starting manager ... starting proxy-1 ... starting worker-1 ... starting worker-2 ... worker-1 terminated immediately after starting; check output with "diag" worker-2 terminated immediately after starting; check output with "diag" This was my output from the diag: Bro 2.4.1 Linux 3.13.0-48-generic No gdb installed. ==== reporter.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path reporter #open 2015-09-15-13-38-43 #fields ts level message location #types time enum string string 0.000000 Reporter::WARNING SumStat key request for the J1pRzdrrLK8 SumStat uid took longer than 1 minute and was automatically cancelled. /usr/local/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line 218 ==== stderr.log ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto ==== .env_vars PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site CLUSTER_NODE=manager ==== .status RUNNING [net_run] ==== No prof.log ==== No packet_filter.log ==== loaded_scripts.log #separator \x09 #set_separator , #empty_field (empty) #unset_field - #path loaded_scripts #open 2015-09-15-13-34-43 #fields name #types string /usr/local/bro/share/bro/base/init-bare.bro /usr/local/bro/share/bro/base/bif/const.bif.bro /usr/local/bro/share/bro/base/bif/types.bif.bro /usr/local/bro/share/bro/base/bif/strings.bif.bro /usr/local/bro/share/bro/base/bif/bro.bif.bro /usr/local/bro/share/bro/base/bif/reporter.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SNMP.types.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_KRB.types.bif.bro /usr/local/bro/share/bro/base/bif/event.bif.bro /usr/local/bro/share/bro/base/frameworks/broker/__load__.bro /usr/local/bro/share/bro/base/frameworks/broker/main.bro /usr/local/bro/share/bro/base/frameworks/logging/__load__.bro /usr/local/bro/share/bro/base/frameworks/logging/main.bro /usr/local/bro/share/bro/base/bif/logging.bif.bro /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/__load__.bro /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/scp.bro /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/sftp.bro /usr/local/bro/share/bro/base/frameworks/logging/writers/ascii.bro /usr/local/bro/share/bro/base/frameworks/logging/writers/sqlite.bro /usr/local/bro/share/bro/base/frameworks/logging/writers/none.bro /usr/local/bro/share/bro/base/frameworks/input/__load__.bro /usr/local/bro/share/bro/base/frameworks/input/main.bro /usr/local/bro/share/bro/base/bif/input.bif.bro /usr/local/bro/share/bro/base/frameworks/input/readers/ascii.bro /usr/local/bro/share/bro/base/frameworks/input/readers/raw.bro /usr/local/bro/share/bro/base/frameworks/input/readers/benchmark.bro /usr/local/bro/share/bro/base/frameworks/input/readers/binary.bro /usr/local/bro/share/bro/base/frameworks/input/readers/sqlite.bro /usr/local/bro/share/bro/base/frameworks/analyzer/__load__.bro /usr/local/bro/share/bro/base/frameworks/analyzer/main.bro /usr/local/bro/share/bro/base/frameworks/packet-filter/utils.bro /usr/local/bro/share/bro/base/bif/analyzer.bif.bro /usr/local/bro/share/bro/base/frameworks/files/__load__.bro /usr/local/bro/share/bro/base/frameworks/files/main.bro /usr/local/bro/share/bro/base/bif/file_analysis.bif.bro /usr/local/bro/share/bro/base/utils/site.bro /usr/local/bro/share/bro/base/utils/patterns.bro /usr/local/bro/share/bro/base/frameworks/files/magic/__load__.bro /usr/local/bro/share/bro/base/bif/__load__.bro /usr/local/bro/share/bro/base/bif/broxygen.bif.bro /usr/local/bro/share/bro/base/bif/pcap.bif.bro /usr/local/bro/share/bro/base/bif/bloom-filter.bif.bro /usr/local/bro/share/bro/base/bif/cardinality-counter.bif.bro /usr/local/bro/share/bro/base/bif/top-k.bif.bro /usr/local/bro/share/bro/base/bif/comm.bif.bro /usr/local/bro/share/bro/base/bif/data.bif.bro /usr/local/bro/share/bro/base/bif/messaging.bif.bro /usr/local/bro/share/bro/base/bif/store.bif.bro /usr/local/bro/share/bro/base/bif/plugins/__load__.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_ARP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_AYIYA.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_BackDoor.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_BitTorrent.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_ConnSize.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_ConnSize.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_DCE_RPC.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_DHCP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_DNP3.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_DNS.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_File.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Finger.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_FTP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_FTP.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Gnutella.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_GTPv1.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_HTTP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_HTTP.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_ICMP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Ident.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_InterConn.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_IRC.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_KRB.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Login.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Login.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_MIME.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Modbus.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_MySQL.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_NCP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_NetBIOS.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_NetBIOS.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_NTP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_PIA.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_POP3.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_RADIUS.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_RDP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_RDP.types.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_RPC.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SIP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SNMP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SMB.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SMTP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SMTP.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SOCKS.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SSH.types.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SSH.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SSL.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SteppingStone.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Syslog.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_TCP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_TCP.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Teredo.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_UDP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_ZIP.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_FileExtract.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_FileExtract.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_FileHash.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_PE.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Unified2.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_Unified2.types.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.events.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.types.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.functions.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_AsciiReader.ascii.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_BenchmarkReader.benchmark.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_BinaryReader.binary.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_RawReader.raw.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SQLiteReader.sqlite.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_AsciiWriter.ascii.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_NoneWriter.none.bif.bro /usr/local/bro/share/bro/base/bif/plugins/Bro_SQLiteWriter.sqlite.bif.bro /usr/local/bro/share/bro/base/init-default.bro /usr/local/bro/share/bro/base/utils/active-http.bro /usr/local/bro/share/bro/base/utils/exec.bro /usr/local/bro/share/bro/base/utils/addrs.bro /usr/local/bro/share/bro/base/utils/conn-ids.bro /usr/local/bro/share/bro/base/utils/dir.bro /usr/local/bro/share/bro/base/frameworks/reporter/__load__.bro /usr/local/bro/share/bro/base/frameworks/reporter/main.bro /usr/local/bro/share/bro/base/utils/paths.bro /usr/local/bro/share/bro/base/utils/directions-and-hosts.bro /usr/local/bro/share/bro/base/utils/files.bro /usr/local/bro/share/bro/base/utils/numbers.bro /usr/local/bro/share/bro/base/utils/queue.bro /usr/local/bro/share/bro/base/utils/strings.bro /usr/local/bro/share/bro/base/utils/thresholds.bro /usr/local/bro/share/bro/base/utils/time.bro /usr/local/bro/share/bro/base/utils/urls.bro /usr/local/bro/share/bro/base/frameworks/notice/__load__.bro /usr/local/bro/share/bro/base/frameworks/notice/main.bro /usr/local/bro/share/bro/base/frameworks/notice/weird.bro /usr/local/bro/share/bro/base/frameworks/notice/actions/drop.bro /usr/local/bro/share/bro/base/frameworks/notice/actions/email_admin.bro /usr/local/bro/share/bro/base/frameworks/notice/actions/page.bro /usr/local/bro/share/bro/base/frameworks/notice/actions/add-geodata.bro /usr/local/bro/share/bro/base/frameworks/notice/extend-email/hostnames.bro /usr/local/bro/share/bro/base/frameworks/cluster/__load__.bro /usr/local/bro/share/bro/base/frameworks/cluster/main.bro /usr/local/bro/share/bro/base/frameworks/control/__load__.bro /usr/local/bro/share/bro/base/frameworks/control/main.bro /usr/local/bro/spool/installed-scripts-do-not-touch/auto/cluster-layout.bro /usr/local/bro/share/bro/base/frameworks/cluster/setup-connections.bro /usr/local/bro/share/bro/base/frameworks/communication/__load__.bro /usr/local/bro/share/bro/base/frameworks/communication/main.bro /usr/local/bro/share/bro/base/frameworks/packet-filter/__load__.bro /usr/local/bro/share/bro/base/frameworks/packet-filter/main.bro /usr/local/bro/share/bro/base/frameworks/packet-filter/netstats.bro /usr/local/bro/share/bro/base/frameworks/packet-filter/cluster.bro /usr/local/bro/share/bro/policy/frameworks/communication/listen.bro /usr/local/bro/share/bro/base/frameworks/cluster/nodes/manager.bro /usr/local/bro/share/bro/base/frameworks/notice/cluster.bro /usr/local/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro /usr/local/bro/share/bro/base/frameworks/dpd/__load__.bro /usr/local/bro/share/bro/base/frameworks/dpd/main.bro /usr/local/bro/share/bro/base/frameworks/signatures/__load__.bro /usr/local/bro/share/bro/base/frameworks/signatures/main.bro /usr/local/bro/share/bro/base/frameworks/software/__load__.bro /usr/local/bro/share/bro/base/frameworks/software/main.bro /usr/local/bro/share/bro/base/frameworks/intel/__load__.bro /usr/local/bro/share/bro/base/frameworks/intel/main.bro /usr/local/bro/share/bro/base/frameworks/intel/cluster.bro /usr/local/bro/share/bro/base/frameworks/intel/input.bro /usr/local/bro/share/bro/base/frameworks/sumstats/__load__.bro /usr/local/bro/share/bro/base/frameworks/sumstats/main.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/__load__.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/average.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/hll_unique.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/last.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/max.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/min.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/sample.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/std-dev.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/variance.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/sum.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/topk.bro /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/unique.bro /usr/local/bro/share/bro/base/frameworks/sumstats/cluster.bro /usr/local/bro/share/bro/base/frameworks/tunnels/__load__.bro /usr/local/bro/share/bro/base/frameworks/tunnels/main.bro /usr/local/bro/share/bro/base/protocols/conn/__load__.bro /usr/local/bro/share/bro/base/protocols/conn/main.bro /usr/local/bro/share/bro/base/protocols/conn/contents.bro /usr/local/bro/share/bro/base/protocols/conn/inactivity.bro /usr/local/bro/share/bro/base/protocols/conn/polling.bro /usr/local/bro/share/bro/base/protocols/conn/thresholds.bro /usr/local/bro/share/bro/base/protocols/dhcp/__load__.bro /usr/local/bro/share/bro/base/protocols/dhcp/consts.bro /usr/local/bro/share/bro/base/protocols/dhcp/main.bro /usr/local/bro/share/bro/base/protocols/dhcp/utils.bro /usr/local/bro/share/bro/base/protocols/dnp3/__load__.bro /usr/local/bro/share/bro/base/protocols/dnp3/main.bro /usr/local/bro/share/bro/base/protocols/dnp3/consts.bro /usr/local/bro/share/bro/base/protocols/dns/__load__.bro /usr/local/bro/share/bro/base/protocols/dns/consts.bro /usr/local/bro/share/bro/base/protocols/dns/main.bro /usr/local/bro/share/bro/base/protocols/ftp/__load__.bro /usr/local/bro/share/bro/base/protocols/ftp/utils-commands.bro /usr/local/bro/share/bro/base/protocols/ftp/info.bro /usr/local/bro/share/bro/base/protocols/ftp/main.bro /usr/local/bro/share/bro/base/protocols/ftp/utils.bro /usr/local/bro/share/bro/base/protocols/ftp/files.bro /usr/local/bro/share/bro/base/protocols/ftp/gridftp.bro /usr/local/bro/share/bro/base/protocols/ssl/__load__.bro /usr/local/bro/share/bro/base/protocols/ssl/consts.bro /usr/local/bro/share/bro/base/protocols/ssl/main.bro /usr/local/bro/share/bro/base/protocols/ssl/mozilla-ca-list.bro /usr/local/bro/share/bro/base/protocols/ssl/files.bro /usr/local/bro/share/bro/base/files/x509/__load__.bro /usr/local/bro/share/bro/base/files/x509/main.bro /usr/local/bro/share/bro/base/files/hash/__load__.bro /usr/local/bro/share/bro/base/files/hash/main.bro /usr/local/bro/share/bro/base/protocols/http/__load__.bro /usr/local/bro/share/bro/base/protocols/http/main.bro /usr/local/bro/share/bro/base/protocols/http/entities.bro /usr/local/bro/share/bro/base/protocols/http/utils.bro /usr/local/bro/share/bro/base/protocols/http/files.bro /usr/local/bro/share/bro/base/protocols/irc/__load__.bro /usr/local/bro/share/bro/base/protocols/irc/main.bro /usr/local/bro/share/bro/base/protocols/irc/dcc-send.bro /usr/local/bro/share/bro/base/protocols/irc/files.bro /usr/local/bro/share/bro/base/protocols/krb/__load__.bro /usr/local/bro/share/bro/base/protocols/krb/main.bro /usr/local/bro/share/bro/base/protocols/krb/consts.bro /usr/local/bro/share/bro/base/protocols/krb/files.bro /usr/local/bro/share/bro/base/protocols/modbus/__load__.bro /usr/local/bro/share/bro/base/protocols/modbus/consts.bro /usr/local/bro/share/bro/base/protocols/modbus/main.bro /usr/local/bro/share/bro/base/protocols/mysql/__load__.bro /usr/local/bro/share/bro/base/protocols/mysql/main.bro /usr/local/bro/share/bro/base/protocols/mysql/consts.bro /usr/local/bro/share/bro/base/protocols/pop3/__load__.bro /usr/local/bro/share/bro/base/protocols/radius/__load__.bro /usr/local/bro/share/bro/base/protocols/radius/main.bro /usr/local/bro/share/bro/base/protocols/radius/consts.bro /usr/local/bro/share/bro/base/protocols/rdp/__load__.bro /usr/local/bro/share/bro/base/protocols/rdp/consts.bro /usr/local/bro/share/bro/base/protocols/rdp/main.bro /usr/local/bro/share/bro/base/protocols/sip/__load__.bro /usr/local/bro/share/bro/base/protocols/sip/main.bro /usr/local/bro/share/bro/base/protocols/snmp/__load__.bro /usr/local/bro/share/bro/base/protocols/snmp/main.bro /usr/local/bro/share/bro/base/protocols/smtp/__load__.bro /usr/local/bro/share/bro/base/protocols/smtp/main.bro /usr/local/bro/share/bro/base/protocols/smtp/entities.bro /usr/local/bro/share/bro/base/protocols/smtp/files.bro /usr/local/bro/share/bro/base/protocols/socks/__load__.bro /usr/local/bro/share/bro/base/protocols/socks/consts.bro /usr/local/bro/share/bro/base/protocols/socks/main.bro /usr/local/bro/share/bro/base/protocols/ssh/__load__.bro /usr/local/bro/share/bro/base/protocols/ssh/main.bro /usr/local/bro/share/bro/base/protocols/syslog/__load__.bro /usr/local/bro/share/bro/base/protocols/syslog/consts.bro /usr/local/bro/share/bro/base/protocols/syslog/main.bro /usr/local/bro/share/bro/base/protocols/tunnels/__load__.bro /usr/local/bro/share/bro/base/files/pe/__load__.bro /usr/local/bro/share/bro/base/files/pe/consts.bro /usr/local/bro/share/bro/base/files/pe/main.bro /usr/local/bro/share/bro/base/files/extract/__load__.bro /usr/local/bro/share/bro/base/files/extract/main.bro /usr/local/bro/share/bro/base/files/unified2/__load__.bro /usr/local/bro/share/bro/base/files/unified2/main.bro /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro /usr/local/bro/share/bro/base/misc/find-filtered-trace.bro /usr/local/bro/spool/installed-scripts-do-not-touch/site/local.bro /usr/local/bro/share/bro/policy/misc/loaded-scripts.bro /usr/local/bro/share/bro/policy/tuning/defaults/__load__.bro /usr/local/bro/share/bro/policy/tuning/defaults/packet-fragments.bro /usr/local/bro/share/bro/policy/tuning/defaults/warnings.bro /usr/local/bro/share/bro/policy/tuning/defaults/extracted_file_limits.bro /usr/local/bro/share/bro/policy/misc/scan.bro /usr/local/bro/share/bro/policy/misc/app-stats/__load__.bro /usr/local/bro/share/bro/policy/misc/app-stats/main.bro /usr/local/bro/share/bro/policy/misc/app-stats/plugins/__load__.bro /usr/local/bro/share/bro/policy/misc/app-stats/plugins/facebook.bro /usr/local/bro/share/bro/policy/misc/detect-traceroute/__load__.bro /usr/local/bro/share/bro/policy/misc/detect-traceroute/main.bro /usr/local/bro/share/bro/policy/frameworks/software/vulnerable.bro /usr/local/bro/share/bro/policy/frameworks/software/version-changes.bro /usr/local/bro/share/bro/policy/protocols/ftp/software.bro /usr/local/bro/share/bro/policy/protocols/smtp/software.bro /usr/local/bro/share/bro/policy/protocols/ssh/software.bro /usr/local/bro/share/bro/policy/protocols/http/software.bro /usr/local/bro/share/bro/policy/protocols/dns/detect-external-names.bro /usr/local/bro/share/bro/policy/protocols/ftp/detect.bro /usr/local/bro/share/bro/policy/protocols/conn/known-hosts.bro /usr/local/bro/share/bro/policy/protocols/conn/known-services.bro /usr/local/bro/share/bro/policy/protocols/ssl/known-certs.bro /usr/local/bro/share/bro/policy/protocols/ssl/validate-certs.bro /usr/local/bro/share/bro/policy/protocols/ssl/log-hostcerts-only.bro /usr/local/bro/share/bro/policy/protocols/ssh/geo-data.bro /usr/local/bro/share/bro/policy/protocols/ssh/detect-bruteforcing.bro /usr/local/bro/share/bro/policy/protocols/ssh/interesting-hostnames.bro /usr/local/bro/share/bro/policy/protocols/http/detect-sqli.bro /usr/local/bro/share/bro/policy/frameworks/files/hash-all-files.bro /usr/local/bro/share/bro/policy/frameworks/files/detect-MHR.bro /usr/local/bro/share/bro/broctl/__load__.bro /usr/local/bro/share/bro/broctl/main.bro /usr/local/bro/share/bro/policy/frameworks/control/controllee.bro /usr/local/bro/spool/installed-scripts-do-not-touch/site/local-manager.bro /usr/local/bro/share/bro/broctl/auto.bro /usr/local/bro/spool/installed-scripts-do-not-touch/auto/local-networks.bro /usr/local/bro/spool/installed-scripts-do-not-touch/auto/broctl-config.bro [proxy-1] Bro 2.4.1 Linux 3.13.0-48-generic No gdb installed. ==== No reporter.log ==== stderr.log ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -U .status -p broctl -p broctl-live -p local -p proxy-1 local.bro broctl base/frameworks/cluster local-proxy broctl/auto ==== .env_vars PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site CLUSTER_NODE=proxy-1 ==== .status RUNNING [net_run] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log [worker-1] error running crash-diag for worker-1 Host 172.31.41.33 is not alive [worker-2] error running crash-diag for worker-2 Host 172.31.41.31 is not alive -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150915/819f49ea/attachment.html ------------------------------ _______________________________________________ Bro mailing list Bro at bro.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro End of Bro Digest, Vol 113, Issue 21 ************************************ From dnthayer at illinois.edu Tue Sep 15 08:41:44 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Tue, 15 Sep 2015 10:41:44 -0500 Subject: [Bro] Broctl Worker Issues In-Reply-To: References: Message-ID: <55F83C38.20702@illinois.edu> 1) Make sure all Bro processes are stopped: a) broctl stop b) broctl ps.bro If you see any Bro processes, then kill them before proceeding to next step. If you see any error or warning messages, then these need to be addressed before proceeding. 2) Since you're not running broctl as the "root" user, you need to make sure bro workers have permission to capture packets: https://www.bro.org/documentation/faq.html#how-can-i-capture-packets-as-an-unprivileged-user 3) Start Bro a) broctl deploy b) There should not be any errors or warnings. On 09/15/2015 08:41 AM, Davison, Charles Robert wrote: > When I try and start broctl on all my workers I receive the following: > > > ubuntu at ip-172-31-41-32:~$ /usr/local/bro/bin/broctl start > > starting manager ... > > starting proxy-1 ... > > starting worker-1 ... > > starting worker-2 ... > > worker-1 terminated immediately after starting; check output with "diag" > > worker-2 terminated immediately after starting; check output with "diag" > > > > This was my output from the diag: > > > Bro 2.4.1 > > Linux 3.13.0-48-generic > > > No gdb installed. > > > ==== reporter.log > > #separator \x09 > > #set_separator , > > #empty_field (empty) > > #unset_field - > > #path reporter > > #open 2015-09-15-13-38-43 > > #fields ts level message location > > #types time enum string string > > 0.000000 Reporter::WARNING SumStat key request for the > J1pRzdrrLK8 SumStat uid took longer than 1 minute and was automatically > cancelled. > /usr/local/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line 218 > > > ==== stderr.log > > > ==== stdout.log > > max memory size (kbytes, -m) unlimited > > data seg size (kbytes, -d) unlimited > > virtual memory (kbytes, -v) unlimited > > core file size (blocks, -c) unlimited > > > ==== .cmdline > > -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl > base/frameworks/cluster local-manager.bro broctl/auto > > > ==== .env_vars > > PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games > > BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site > > CLUSTER_NODE=manager > > > ==== .status > > RUNNING [net_run] > > > ==== No prof.log > > > ==== No packet_filter.log > > > ==== loaded_scripts.log > > #separator \x09 > > #set_separator , > > #empty_field (empty) > > #unset_field - > > #path loaded_scripts > > #open 2015-09-15-13-34-43 > > #fields name > > #types string > > /usr/local/bro/share/bro/base/init-bare.bro > > /usr/local/bro/share/bro/base/bif/const.bif.bro > > /usr/local/bro/share/bro/base/bif/types.bif.bro > > /usr/local/bro/share/bro/base/bif/strings.bif.bro > > /usr/local/bro/share/bro/base/bif/bro.bif.bro > > /usr/local/bro/share/bro/base/bif/reporter.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SNMP.types.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_KRB.types.bif.bro > > /usr/local/bro/share/bro/base/bif/event.bif.bro > > /usr/local/bro/share/bro/base/frameworks/broker/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/broker/main.bro > > /usr/local/bro/share/bro/base/frameworks/logging/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/logging/main.bro > > /usr/local/bro/share/bro/base/bif/logging.bif.bro > > > /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/__load__.bro > > > /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/scp.bro > > > /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/sftp.bro > > /usr/local/bro/share/bro/base/frameworks/logging/writers/ascii.bro > > /usr/local/bro/share/bro/base/frameworks/logging/writers/sqlite.bro > > /usr/local/bro/share/bro/base/frameworks/logging/writers/none.bro > > /usr/local/bro/share/bro/base/frameworks/input/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/input/main.bro > > /usr/local/bro/share/bro/base/bif/input.bif.bro > > /usr/local/bro/share/bro/base/frameworks/input/readers/ascii.bro > > /usr/local/bro/share/bro/base/frameworks/input/readers/raw.bro > > /usr/local/bro/share/bro/base/frameworks/input/readers/benchmark.bro > > /usr/local/bro/share/bro/base/frameworks/input/readers/binary.bro > > /usr/local/bro/share/bro/base/frameworks/input/readers/sqlite.bro > > /usr/local/bro/share/bro/base/frameworks/analyzer/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/analyzer/main.bro > > /usr/local/bro/share/bro/base/frameworks/packet-filter/utils.bro > > /usr/local/bro/share/bro/base/bif/analyzer.bif.bro > > /usr/local/bro/share/bro/base/frameworks/files/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/files/main.bro > > /usr/local/bro/share/bro/base/bif/file_analysis.bif.bro > > /usr/local/bro/share/bro/base/utils/site.bro > > /usr/local/bro/share/bro/base/utils/patterns.bro > > /usr/local/bro/share/bro/base/frameworks/files/magic/__load__.bro > > /usr/local/bro/share/bro/base/bif/__load__.bro > > /usr/local/bro/share/bro/base/bif/broxygen.bif.bro > > /usr/local/bro/share/bro/base/bif/pcap.bif.bro > > /usr/local/bro/share/bro/base/bif/bloom-filter.bif.bro > > /usr/local/bro/share/bro/base/bif/cardinality-counter.bif.bro > > /usr/local/bro/share/bro/base/bif/top-k.bif.bro > > /usr/local/bro/share/bro/base/bif/comm.bif.bro > > /usr/local/bro/share/bro/base/bif/data.bif.bro > > /usr/local/bro/share/bro/base/bif/messaging.bif.bro > > /usr/local/bro/share/bro/base/bif/store.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/__load__.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_ARP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_AYIYA.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_BackDoor.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_BitTorrent.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_ConnSize.events.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_ConnSize.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_DCE_RPC.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_DHCP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_DNP3.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_DNS.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_File.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Finger.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_FTP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_FTP.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Gnutella.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_GTPv1.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_HTTP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_HTTP.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_ICMP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Ident.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_InterConn.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_IRC.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_KRB.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Login.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Login.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_MIME.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Modbus.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_MySQL.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_NCP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_NetBIOS.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_NetBIOS.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_NTP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_PIA.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_POP3.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_RADIUS.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_RDP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_RDP.types.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_RPC.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SIP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SNMP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SMB.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SMTP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SMTP.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SOCKS.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SSH.types.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SSH.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SSL.events.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SteppingStone.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Syslog.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_TCP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_TCP.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Teredo.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_UDP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_ZIP.events.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_FileExtract.events.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_FileExtract.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_FileHash.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_PE.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Unified2.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Unified2.types.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.types.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_AsciiReader.ascii.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_BenchmarkReader.benchmark.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_BinaryReader.binary.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_RawReader.raw.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SQLiteReader.sqlite.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_AsciiWriter.ascii.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_NoneWriter.none.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SQLiteWriter.sqlite.bif.bro > > /usr/local/bro/share/bro/base/init-default.bro > > /usr/local/bro/share/bro/base/utils/active-http.bro > > /usr/local/bro/share/bro/base/utils/exec.bro > > /usr/local/bro/share/bro/base/utils/addrs.bro > > /usr/local/bro/share/bro/base/utils/conn-ids.bro > > /usr/local/bro/share/bro/base/utils/dir.bro > > /usr/local/bro/share/bro/base/frameworks/reporter/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/reporter/main.bro > > /usr/local/bro/share/bro/base/utils/paths.bro > > /usr/local/bro/share/bro/base/utils/directions-and-hosts.bro > > /usr/local/bro/share/bro/base/utils/files.bro > > /usr/local/bro/share/bro/base/utils/numbers.bro > > /usr/local/bro/share/bro/base/utils/queue.bro > > /usr/local/bro/share/bro/base/utils/strings.bro > > /usr/local/bro/share/bro/base/utils/thresholds.bro > > /usr/local/bro/share/bro/base/utils/time.bro > > /usr/local/bro/share/bro/base/utils/urls.bro > > /usr/local/bro/share/bro/base/frameworks/notice/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/notice/main.bro > > /usr/local/bro/share/bro/base/frameworks/notice/weird.bro > > /usr/local/bro/share/bro/base/frameworks/notice/actions/drop.bro > > /usr/local/bro/share/bro/base/frameworks/notice/actions/email_admin.bro > > /usr/local/bro/share/bro/base/frameworks/notice/actions/page.bro > > /usr/local/bro/share/bro/base/frameworks/notice/actions/add-geodata.bro > > > /usr/local/bro/share/bro/base/frameworks/notice/extend-email/hostnames.bro > > /usr/local/bro/share/bro/base/frameworks/cluster/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/cluster/main.bro > > /usr/local/bro/share/bro/base/frameworks/control/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/control/main.bro > > > /usr/local/bro/spool/installed-scripts-do-not-touch/auto/cluster-layout.bro > > > /usr/local/bro/share/bro/base/frameworks/cluster/setup-connections.bro > > /usr/local/bro/share/bro/base/frameworks/communication/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/communication/main.bro > > > /usr/local/bro/share/bro/base/frameworks/packet-filter/__load__.bro > > > /usr/local/bro/share/bro/base/frameworks/packet-filter/main.bro > > > /usr/local/bro/share/bro/base/frameworks/packet-filter/netstats.bro > > > /usr/local/bro/share/bro/base/frameworks/packet-filter/cluster.bro > > /usr/local/bro/share/bro/policy/frameworks/communication/listen.bro > > /usr/local/bro/share/bro/base/frameworks/cluster/nodes/manager.bro > > /usr/local/bro/share/bro/base/frameworks/notice/cluster.bro > > /usr/local/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro > > /usr/local/bro/share/bro/base/frameworks/dpd/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/dpd/main.bro > > /usr/local/bro/share/bro/base/frameworks/signatures/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/signatures/main.bro > > /usr/local/bro/share/bro/base/frameworks/software/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/software/main.bro > > /usr/local/bro/share/bro/base/frameworks/intel/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/intel/main.bro > > /usr/local/bro/share/bro/base/frameworks/intel/cluster.bro > > /usr/local/bro/share/bro/base/frameworks/intel/input.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/main.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/average.bro > > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/hll_unique.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/last.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/max.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/min.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/sample.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/std-dev.bro > > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/variance.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/sum.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/topk.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/unique.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/cluster.bro > > /usr/local/bro/share/bro/base/frameworks/tunnels/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/tunnels/main.bro > > /usr/local/bro/share/bro/base/protocols/conn/__load__.bro > > /usr/local/bro/share/bro/base/protocols/conn/main.bro > > /usr/local/bro/share/bro/base/protocols/conn/contents.bro > > /usr/local/bro/share/bro/base/protocols/conn/inactivity.bro > > /usr/local/bro/share/bro/base/protocols/conn/polling.bro > > /usr/local/bro/share/bro/base/protocols/conn/thresholds.bro > > /usr/local/bro/share/bro/base/protocols/dhcp/__load__.bro > > /usr/local/bro/share/bro/base/protocols/dhcp/consts.bro > > /usr/local/bro/share/bro/base/protocols/dhcp/main.bro > > /usr/local/bro/share/bro/base/protocols/dhcp/utils.bro > > /usr/local/bro/share/bro/base/protocols/dnp3/__load__.bro > > /usr/local/bro/share/bro/base/protocols/dnp3/main.bro > > /usr/local/bro/share/bro/base/protocols/dnp3/consts.bro > > /usr/local/bro/share/bro/base/protocols/dns/__load__.bro > > /usr/local/bro/share/bro/base/protocols/dns/consts.bro > > /usr/local/bro/share/bro/base/protocols/dns/main.bro > > /usr/local/bro/share/bro/base/protocols/ftp/__load__.bro > > /usr/local/bro/share/bro/base/protocols/ftp/utils-commands.bro > > /usr/local/bro/share/bro/base/protocols/ftp/info.bro > > /usr/local/bro/share/bro/base/protocols/ftp/main.bro > > /usr/local/bro/share/bro/base/protocols/ftp/utils.bro > > /usr/local/bro/share/bro/base/protocols/ftp/files.bro > > /usr/local/bro/share/bro/base/protocols/ftp/gridftp.bro > > /usr/local/bro/share/bro/base/protocols/ssl/__load__.bro > > /usr/local/bro/share/bro/base/protocols/ssl/consts.bro > > /usr/local/bro/share/bro/base/protocols/ssl/main.bro > > /usr/local/bro/share/bro/base/protocols/ssl/mozilla-ca-list.bro > > /usr/local/bro/share/bro/base/protocols/ssl/files.bro > > /usr/local/bro/share/bro/base/files/x509/__load__.bro > > /usr/local/bro/share/bro/base/files/x509/main.bro > > /usr/local/bro/share/bro/base/files/hash/__load__.bro > > /usr/local/bro/share/bro/base/files/hash/main.bro > > /usr/local/bro/share/bro/base/protocols/http/__load__.bro > > /usr/local/bro/share/bro/base/protocols/http/main.bro > > /usr/local/bro/share/bro/base/protocols/http/entities.bro > > /usr/local/bro/share/bro/base/protocols/http/utils.bro > > /usr/local/bro/share/bro/base/protocols/http/files.bro > > /usr/local/bro/share/bro/base/protocols/irc/__load__.bro > > /usr/local/bro/share/bro/base/protocols/irc/main.bro > > /usr/local/bro/share/bro/base/protocols/irc/dcc-send.bro > > /usr/local/bro/share/bro/base/protocols/irc/files.bro > > /usr/local/bro/share/bro/base/protocols/krb/__load__.bro > > /usr/local/bro/share/bro/base/protocols/krb/main.bro > > /usr/local/bro/share/bro/base/protocols/krb/consts.bro > > /usr/local/bro/share/bro/base/protocols/krb/files.bro > > /usr/local/bro/share/bro/base/protocols/modbus/__load__.bro > > /usr/local/bro/share/bro/base/protocols/modbus/consts.bro > > /usr/local/bro/share/bro/base/protocols/modbus/main.bro > > /usr/local/bro/share/bro/base/protocols/mysql/__load__.bro > > /usr/local/bro/share/bro/base/protocols/mysql/main.bro > > /usr/local/bro/share/bro/base/protocols/mysql/consts.bro > > /usr/local/bro/share/bro/base/protocols/pop3/__load__.bro > > /usr/local/bro/share/bro/base/protocols/radius/__load__.bro > > /usr/local/bro/share/bro/base/protocols/radius/main.bro > > /usr/local/bro/share/bro/base/protocols/radius/consts.bro > > /usr/local/bro/share/bro/base/protocols/rdp/__load__.bro > > /usr/local/bro/share/bro/base/protocols/rdp/consts.bro > > /usr/local/bro/share/bro/base/protocols/rdp/main.bro > > /usr/local/bro/share/bro/base/protocols/sip/__load__.bro > > /usr/local/bro/share/bro/base/protocols/sip/main.bro > > /usr/local/bro/share/bro/base/protocols/snmp/__load__.bro > > /usr/local/bro/share/bro/base/protocols/snmp/main.bro > > /usr/local/bro/share/bro/base/protocols/smtp/__load__.bro > > /usr/local/bro/share/bro/base/protocols/smtp/main.bro > > /usr/local/bro/share/bro/base/protocols/smtp/entities.bro > > /usr/local/bro/share/bro/base/protocols/smtp/files.bro > > /usr/local/bro/share/bro/base/protocols/socks/__load__.bro > > /usr/local/bro/share/bro/base/protocols/socks/consts.bro > > /usr/local/bro/share/bro/base/protocols/socks/main.bro > > /usr/local/bro/share/bro/base/protocols/ssh/__load__.bro > > /usr/local/bro/share/bro/base/protocols/ssh/main.bro > > /usr/local/bro/share/bro/base/protocols/syslog/__load__.bro > > /usr/local/bro/share/bro/base/protocols/syslog/consts.bro > > /usr/local/bro/share/bro/base/protocols/syslog/main.bro > > /usr/local/bro/share/bro/base/protocols/tunnels/__load__.bro > > /usr/local/bro/share/bro/base/files/pe/__load__.bro > > /usr/local/bro/share/bro/base/files/pe/consts.bro > > /usr/local/bro/share/bro/base/files/pe/main.bro > > /usr/local/bro/share/bro/base/files/extract/__load__.bro > > /usr/local/bro/share/bro/base/files/extract/main.bro > > /usr/local/bro/share/bro/base/files/unified2/__load__.bro > > /usr/local/bro/share/bro/base/files/unified2/main.bro > > /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro > > /usr/local/bro/share/bro/base/misc/find-filtered-trace.bro > > /usr/local/bro/spool/installed-scripts-do-not-touch/site/local.bro > > /usr/local/bro/share/bro/policy/misc/loaded-scripts.bro > > /usr/local/bro/share/bro/policy/tuning/defaults/__load__.bro > > /usr/local/bro/share/bro/policy/tuning/defaults/packet-fragments.bro > > /usr/local/bro/share/bro/policy/tuning/defaults/warnings.bro > > > /usr/local/bro/share/bro/policy/tuning/defaults/extracted_file_limits.bro > > /usr/local/bro/share/bro/policy/misc/scan.bro > > /usr/local/bro/share/bro/policy/misc/app-stats/__load__.bro > > /usr/local/bro/share/bro/policy/misc/app-stats/main.bro > > /usr/local/bro/share/bro/policy/misc/app-stats/plugins/__load__.bro > > /usr/local/bro/share/bro/policy/misc/app-stats/plugins/facebook.bro > > /usr/local/bro/share/bro/policy/misc/detect-traceroute/__load__.bro > > /usr/local/bro/share/bro/policy/misc/detect-traceroute/main.bro > > /usr/local/bro/share/bro/policy/frameworks/software/vulnerable.bro > > /usr/local/bro/share/bro/policy/frameworks/software/version-changes.bro > > /usr/local/bro/share/bro/policy/protocols/ftp/software.bro > > /usr/local/bro/share/bro/policy/protocols/smtp/software.bro > > /usr/local/bro/share/bro/policy/protocols/ssh/software.bro > > /usr/local/bro/share/bro/policy/protocols/http/software.bro > > /usr/local/bro/share/bro/policy/protocols/dns/detect-external-names.bro > > /usr/local/bro/share/bro/policy/protocols/ftp/detect.bro > > /usr/local/bro/share/bro/policy/protocols/conn/known-hosts.bro > > /usr/local/bro/share/bro/policy/protocols/conn/known-services.bro > > /usr/local/bro/share/bro/policy/protocols/ssl/known-certs.bro > > /usr/local/bro/share/bro/policy/protocols/ssl/validate-certs.bro > > /usr/local/bro/share/bro/policy/protocols/ssl/log-hostcerts-only.bro > > /usr/local/bro/share/bro/policy/protocols/ssh/geo-data.bro > > /usr/local/bro/share/bro/policy/protocols/ssh/detect-bruteforcing.bro > > /usr/local/bro/share/bro/policy/protocols/ssh/interesting-hostnames.bro > > /usr/local/bro/share/bro/policy/protocols/http/detect-sqli.bro > > /usr/local/bro/share/bro/policy/frameworks/files/hash-all-files.bro > > /usr/local/bro/share/bro/policy/frameworks/files/detect-MHR.bro > > /usr/local/bro/share/bro/broctl/__load__.bro > > /usr/local/bro/share/bro/broctl/main.bro > > /usr/local/bro/share/bro/policy/frameworks/control/controllee.bro > > /usr/local/bro/spool/installed-scripts-do-not-touch/site/local-manager.bro > > /usr/local/bro/share/bro/broctl/auto.bro > > > /usr/local/bro/spool/installed-scripts-do-not-touch/auto/local-networks.bro > > > /usr/local/bro/spool/installed-scripts-do-not-touch/auto/broctl-config.bro > > [proxy-1] > > > Bro 2.4.1 > > Linux 3.13.0-48-generic > > > No gdb installed. > > > ==== No reporter.log > > > ==== stderr.log > > > ==== stdout.log > > max memory size (kbytes, -m) unlimited > > data seg size (kbytes, -d) unlimited > > virtual memory (kbytes, -v) unlimited > > core file size (blocks, -c) unlimited > > > ==== .cmdline > > -U .status -p broctl -p broctl-live -p local -p proxy-1 local.bro broctl > base/frameworks/cluster local-proxy broctl/auto > > > ==== .env_vars > > PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games > > BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site > > CLUSTER_NODE=proxy-1 > > > ==== .status > > RUNNING [net_run] > > > ==== No prof.log > > > ==== No packet_filter.log > > > ==== No loaded_scripts.log > > [worker-1] > > error running crash-diag for worker-1 > > Host 172.31.41.33 is not alive > > [worker-2] > > error running crash-diag for worker-2 > > Host 172.31.41.31 is not alive > > > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From johanna at icir.org Tue Sep 15 09:03:28 2015 From: johanna at icir.org (Johanna Amann) Date: Tue, 15 Sep 2015 09:03:28 -0700 Subject: [Bro] Compiling bro 2.4.1 on Ubuntu && ARM (HELP Please) In-Reply-To: References: Message-ID: <20150915160324.GA45171@Beezling.local> Could you potentially append or send me the full CMakeError.log? Johanna On Tue, Sep 15, 2015 at 06:57:00AM -0400, Ludwig Goon wrote: > Trying to compile bro 2.4.1 on linux. After setting up the packages for > compile I run the ./configure scriipt and get the following error: > > > -- Looking for include file pthread.h > -- Looking for include file pthread.h - found > -- Looking for pthread_create > -- Looking for pthread_create - not found > -- Looking for pthread_create in pthreads > -- Looking for pthread_create in pthreads - not found > -- Looking for pthread_create in pthread > -- Looking for pthread_create in pthread - found > -- Found Threads: TRUE > CMake Error at doc/CMakeLists.txt:14 (message): > Problem setting BROPATH > -- Configuring incomplete, errors occurred! > See also > "/home/john/Projects/BRO-IDS/bro-2.4.1/build/CMakeFiles/CMakeOutput.log". > See also > "/home/john/Projects/BRO-IDS/bro-2.4.1/build/CMakeFiles/CMakeError.log". > > > After looking at the CMakeError.log file it seems that it's looking for the > pthreads library which is defined by -lpthreads > > Can't find that exact library for posix threads however here are the ones > in ubuntu > root at merovingian:/usr/lib/x86_64-linux-gnu# aptitude search pthread > i libevent-pthreads-2.0-5 - > Asynchronous event notification library (pthreads) > p libevent-pthreads-2.0-5:i386 - > Asynchronous event notification library (pthreads) > i libpthread-stubs0-dev - pthread > stubs not provided by native libc, development files > p libpthread-stubs0-dev:i386 - pthread > stubs not provided by native libc, development files > p libpthread-workqueue-dev - thread > pool library (development files) > p libpthread-workqueue-dev:i386 - thread > pool library (development files) > p libpthread-workqueue0 - thread > pool library > p libpthread-workqueue0:i386 - thread > pool library > > SO can I adjust CMake to use -lpthread or similar or tell CMake to skip > this overall? > > HELP please!!! > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Tue Sep 15 09:18:54 2015 From: johanna at icir.org (Johanna Amann) Date: Tue, 15 Sep 2015 09:18:54 -0700 Subject: [Bro] long SSH connection in conn.log In-Reply-To: <55E8C4F8.3080009@dreyer-net.de> References: <55E8C4F8.3080009@dreyer-net.de> Message-ID: <20150915161854.GA47853@Beezling.local> Hello Sven, you are probably running into internal Bro timeouts here. Generally, since Bro can only use limited amounts of RAM, Bro automatically times out connections after it does not see any activity (exchanged packets) for a specified period of time. For TCP, this is generally 5 minutes, defined in tcp_inactivity_timeout. Since interactive protocols tend to have longer periods of time where we might not see any exchanged connections, Bro has special settings for them - in base/protocols/conn/inactivity.bro the timeout for identified SSH sessions is set to 1 hour. However, if your session does not exchange any packets for more than one hour, Bro will assume that it has been closed and just not seen the packets closing the connection. Everything following will be regarded as a new connection - and in your case fall under the default 5 minute timeout since it cannot reliably be identified as SSH. This is a generic problem - you have to just assume that connections are terminated after you did not see any exchanged data for a specified period of time. In case the current Bro settings do not work for you, you can redef them. I hope that helps, Johanna On Fri, Sep 04, 2015 at 12:08:56AM +0200, Sven Dreyer wrote: > Dear list, > > I started an SSH connection in my LAN on 3:32pm which lasted until > 07:04pm - so we're talking about an SSH session lasting 3 1/2 hours. > > In my conn.log files, I find this single SSH connection as 5 connections: > > 1) conn_state S1, service ssh > 2-4) conn_state OTH, service - > 5) conn_state SF, service - > > Bro was started before the SSH connection was initiated, so I'd expect a > single conn.log entry to be written when I disconnect. Or did I get > something wrong here? > > Thanks! > Sven > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > From cdaviso1 at vols.utk.edu Tue Sep 15 10:24:14 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Tue, 15 Sep 2015 17:24:14 +0000 Subject: [Bro] Broctl Worker Issues In-Reply-To: <55F83C38.20702@illinois.edu> References: , <55F83C38.20702@illinois.edu> Message-ID: 1. Complete 2. Complete: sudo setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro 3. Error: [BroControl] > ps.bro USER PID PPID %CPU %MEM VSZ RSS TT S STARTED TIME COMMAND >>> 172.31.41.32 >>> 172.31.41.33 >>> 172.31.41.31 [BroControl] > deploy checking configurations ... installing ... removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/site ... removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/auto ... creating policy directories ... installing site policies ... generating cluster-layout.bro ... generating local-networks.bro ... generating broctl-config.bro ... generating broctl-config.sh ... updating nodes ... stopping ... worker-1 not running (was crashed) worker-2 not running (was crashed) proxy-1 not running (was crashed) manager not running (was crashed) starting ... starting manager ... starting proxy-1 ... starting worker-1 ... starting worker-2 ... worker-1 terminated immediately after starting; check output with "diag" worker-2 terminated immediately after starting; check output with "diag" [BroControl] > ________________________________________ From: Daniel Thayer Sent: Tuesday, September 15, 2015 9:41 AM To: Davison, Charles Robert; bro at bro.org Subject: Re: [Bro] Broctl Worker Issues 1) Make sure all Bro processes are stopped: a) broctl stop b) broctl ps.bro If you see any Bro processes, then kill them before proceeding to next step. If you see any error or warning messages, then these need to be addressed before proceeding. 2) Since you're not running broctl as the "root" user, you need to make sure bro workers have permission to capture packets: https://www.bro.org/documentation/faq.html#how-can-i-capture-packets-as-an-unprivileged-user 3) Start Bro a) broctl deploy b) There should not be any errors or warnings. On 09/15/2015 08:41 AM, Davison, Charles Robert wrote: > When I try and start broctl on all my workers I receive the following: > > > ubuntu at ip-172-31-41-32:~$ /usr/local/bro/bin/broctl start > > starting manager ... > > starting proxy-1 ... > > starting worker-1 ... > > starting worker-2 ... > > worker-1 terminated immediately after starting; check output with "diag" > > worker-2 terminated immediately after starting; check output with "diag" > > > > This was my output from the diag: > > > Bro 2.4.1 > > Linux 3.13.0-48-generic > > > No gdb installed. > > > ==== reporter.log > > #separator \x09 > > #set_separator , > > #empty_field (empty) > > #unset_field - > > #path reporter > > #open 2015-09-15-13-38-43 > > #fields ts level message location > > #types time enum string string > > 0.000000 Reporter::WARNING SumStat key request for the > J1pRzdrrLK8 SumStat uid took longer than 1 minute and was automatically > cancelled. > /usr/local/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line 218 > > > ==== stderr.log > > > ==== stdout.log > > max memory size (kbytes, -m) unlimited > > data seg size (kbytes, -d) unlimited > > virtual memory (kbytes, -v) unlimited > > core file size (blocks, -c) unlimited > > > ==== .cmdline > > -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl > base/frameworks/cluster local-manager.bro broctl/auto > > > ==== .env_vars > > PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games > > BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site > > CLUSTER_NODE=manager > > > ==== .status > > RUNNING [net_run] > > > ==== No prof.log > > > ==== No packet_filter.log > > > ==== loaded_scripts.log > > #separator \x09 > > #set_separator , > > #empty_field (empty) > > #unset_field - > > #path loaded_scripts > > #open 2015-09-15-13-34-43 > > #fields name > > #types string > > /usr/local/bro/share/bro/base/init-bare.bro > > /usr/local/bro/share/bro/base/bif/const.bif.bro > > /usr/local/bro/share/bro/base/bif/types.bif.bro > > /usr/local/bro/share/bro/base/bif/strings.bif.bro > > /usr/local/bro/share/bro/base/bif/bro.bif.bro > > /usr/local/bro/share/bro/base/bif/reporter.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SNMP.types.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_KRB.types.bif.bro > > /usr/local/bro/share/bro/base/bif/event.bif.bro > > /usr/local/bro/share/bro/base/frameworks/broker/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/broker/main.bro > > /usr/local/bro/share/bro/base/frameworks/logging/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/logging/main.bro > > /usr/local/bro/share/bro/base/bif/logging.bif.bro > > > /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/__load__.bro > > > /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/scp.bro > > > /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/sftp.bro > > /usr/local/bro/share/bro/base/frameworks/logging/writers/ascii.bro > > /usr/local/bro/share/bro/base/frameworks/logging/writers/sqlite.bro > > /usr/local/bro/share/bro/base/frameworks/logging/writers/none.bro > > /usr/local/bro/share/bro/base/frameworks/input/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/input/main.bro > > /usr/local/bro/share/bro/base/bif/input.bif.bro > > /usr/local/bro/share/bro/base/frameworks/input/readers/ascii.bro > > /usr/local/bro/share/bro/base/frameworks/input/readers/raw.bro > > /usr/local/bro/share/bro/base/frameworks/input/readers/benchmark.bro > > /usr/local/bro/share/bro/base/frameworks/input/readers/binary.bro > > /usr/local/bro/share/bro/base/frameworks/input/readers/sqlite.bro > > /usr/local/bro/share/bro/base/frameworks/analyzer/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/analyzer/main.bro > > /usr/local/bro/share/bro/base/frameworks/packet-filter/utils.bro > > /usr/local/bro/share/bro/base/bif/analyzer.bif.bro > > /usr/local/bro/share/bro/base/frameworks/files/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/files/main.bro > > /usr/local/bro/share/bro/base/bif/file_analysis.bif.bro > > /usr/local/bro/share/bro/base/utils/site.bro > > /usr/local/bro/share/bro/base/utils/patterns.bro > > /usr/local/bro/share/bro/base/frameworks/files/magic/__load__.bro > > /usr/local/bro/share/bro/base/bif/__load__.bro > > /usr/local/bro/share/bro/base/bif/broxygen.bif.bro > > /usr/local/bro/share/bro/base/bif/pcap.bif.bro > > /usr/local/bro/share/bro/base/bif/bloom-filter.bif.bro > > /usr/local/bro/share/bro/base/bif/cardinality-counter.bif.bro > > /usr/local/bro/share/bro/base/bif/top-k.bif.bro > > /usr/local/bro/share/bro/base/bif/comm.bif.bro > > /usr/local/bro/share/bro/base/bif/data.bif.bro > > /usr/local/bro/share/bro/base/bif/messaging.bif.bro > > /usr/local/bro/share/bro/base/bif/store.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/__load__.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_ARP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_AYIYA.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_BackDoor.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_BitTorrent.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_ConnSize.events.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_ConnSize.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_DCE_RPC.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_DHCP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_DNP3.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_DNS.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_File.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Finger.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_FTP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_FTP.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Gnutella.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_GTPv1.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_HTTP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_HTTP.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_ICMP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Ident.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_InterConn.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_IRC.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_KRB.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Login.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Login.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_MIME.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Modbus.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_MySQL.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_NCP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_NetBIOS.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_NetBIOS.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_NTP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_PIA.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_POP3.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_RADIUS.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_RDP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_RDP.types.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_RPC.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SIP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SNMP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SMB.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SMTP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SMTP.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SOCKS.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SSH.types.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SSH.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SSL.events.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SteppingStone.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Syslog.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_TCP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_TCP.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Teredo.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_UDP.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_ZIP.events.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_FileExtract.events.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_FileExtract.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_FileHash.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_PE.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Unified2.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_Unified2.types.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.events.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.types.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.functions.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_AsciiReader.ascii.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_BenchmarkReader.benchmark.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_BinaryReader.binary.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_RawReader.raw.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SQLiteReader.sqlite.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_AsciiWriter.ascii.bif.bro > > /usr/local/bro/share/bro/base/bif/plugins/Bro_NoneWriter.none.bif.bro > > > /usr/local/bro/share/bro/base/bif/plugins/Bro_SQLiteWriter.sqlite.bif.bro > > /usr/local/bro/share/bro/base/init-default.bro > > /usr/local/bro/share/bro/base/utils/active-http.bro > > /usr/local/bro/share/bro/base/utils/exec.bro > > /usr/local/bro/share/bro/base/utils/addrs.bro > > /usr/local/bro/share/bro/base/utils/conn-ids.bro > > /usr/local/bro/share/bro/base/utils/dir.bro > > /usr/local/bro/share/bro/base/frameworks/reporter/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/reporter/main.bro > > /usr/local/bro/share/bro/base/utils/paths.bro > > /usr/local/bro/share/bro/base/utils/directions-and-hosts.bro > > /usr/local/bro/share/bro/base/utils/files.bro > > /usr/local/bro/share/bro/base/utils/numbers.bro > > /usr/local/bro/share/bro/base/utils/queue.bro > > /usr/local/bro/share/bro/base/utils/strings.bro > > /usr/local/bro/share/bro/base/utils/thresholds.bro > > /usr/local/bro/share/bro/base/utils/time.bro > > /usr/local/bro/share/bro/base/utils/urls.bro > > /usr/local/bro/share/bro/base/frameworks/notice/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/notice/main.bro > > /usr/local/bro/share/bro/base/frameworks/notice/weird.bro > > /usr/local/bro/share/bro/base/frameworks/notice/actions/drop.bro > > /usr/local/bro/share/bro/base/frameworks/notice/actions/email_admin.bro > > /usr/local/bro/share/bro/base/frameworks/notice/actions/page.bro > > /usr/local/bro/share/bro/base/frameworks/notice/actions/add-geodata.bro > > > /usr/local/bro/share/bro/base/frameworks/notice/extend-email/hostnames.bro > > /usr/local/bro/share/bro/base/frameworks/cluster/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/cluster/main.bro > > /usr/local/bro/share/bro/base/frameworks/control/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/control/main.bro > > > /usr/local/bro/spool/installed-scripts-do-not-touch/auto/cluster-layout.bro > > > /usr/local/bro/share/bro/base/frameworks/cluster/setup-connections.bro > > /usr/local/bro/share/bro/base/frameworks/communication/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/communication/main.bro > > > /usr/local/bro/share/bro/base/frameworks/packet-filter/__load__.bro > > > /usr/local/bro/share/bro/base/frameworks/packet-filter/main.bro > > > /usr/local/bro/share/bro/base/frameworks/packet-filter/netstats.bro > > > /usr/local/bro/share/bro/base/frameworks/packet-filter/cluster.bro > > /usr/local/bro/share/bro/policy/frameworks/communication/listen.bro > > /usr/local/bro/share/bro/base/frameworks/cluster/nodes/manager.bro > > /usr/local/bro/share/bro/base/frameworks/notice/cluster.bro > > /usr/local/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro > > /usr/local/bro/share/bro/base/frameworks/dpd/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/dpd/main.bro > > /usr/local/bro/share/bro/base/frameworks/signatures/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/signatures/main.bro > > /usr/local/bro/share/bro/base/frameworks/software/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/software/main.bro > > /usr/local/bro/share/bro/base/frameworks/intel/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/intel/main.bro > > /usr/local/bro/share/bro/base/frameworks/intel/cluster.bro > > /usr/local/bro/share/bro/base/frameworks/intel/input.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/main.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/average.bro > > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/hll_unique.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/last.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/max.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/min.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/sample.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/std-dev.bro > > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/variance.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/sum.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/topk.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/unique.bro > > /usr/local/bro/share/bro/base/frameworks/sumstats/cluster.bro > > /usr/local/bro/share/bro/base/frameworks/tunnels/__load__.bro > > /usr/local/bro/share/bro/base/frameworks/tunnels/main.bro > > /usr/local/bro/share/bro/base/protocols/conn/__load__.bro > > /usr/local/bro/share/bro/base/protocols/conn/main.bro > > /usr/local/bro/share/bro/base/protocols/conn/contents.bro > > /usr/local/bro/share/bro/base/protocols/conn/inactivity.bro > > /usr/local/bro/share/bro/base/protocols/conn/polling.bro > > /usr/local/bro/share/bro/base/protocols/conn/thresholds.bro > > /usr/local/bro/share/bro/base/protocols/dhcp/__load__.bro > > /usr/local/bro/share/bro/base/protocols/dhcp/consts.bro > > /usr/local/bro/share/bro/base/protocols/dhcp/main.bro > > /usr/local/bro/share/bro/base/protocols/dhcp/utils.bro > > /usr/local/bro/share/bro/base/protocols/dnp3/__load__.bro > > /usr/local/bro/share/bro/base/protocols/dnp3/main.bro > > /usr/local/bro/share/bro/base/protocols/dnp3/consts.bro > > /usr/local/bro/share/bro/base/protocols/dns/__load__.bro > > /usr/local/bro/share/bro/base/protocols/dns/consts.bro > > /usr/local/bro/share/bro/base/protocols/dns/main.bro > > /usr/local/bro/share/bro/base/protocols/ftp/__load__.bro > > /usr/local/bro/share/bro/base/protocols/ftp/utils-commands.bro > > /usr/local/bro/share/bro/base/protocols/ftp/info.bro > > /usr/local/bro/share/bro/base/protocols/ftp/main.bro > > /usr/local/bro/share/bro/base/protocols/ftp/utils.bro > > /usr/local/bro/share/bro/base/protocols/ftp/files.bro > > /usr/local/bro/share/bro/base/protocols/ftp/gridftp.bro > > /usr/local/bro/share/bro/base/protocols/ssl/__load__.bro > > /usr/local/bro/share/bro/base/protocols/ssl/consts.bro > > /usr/local/bro/share/bro/base/protocols/ssl/main.bro > > /usr/local/bro/share/bro/base/protocols/ssl/mozilla-ca-list.bro > > /usr/local/bro/share/bro/base/protocols/ssl/files.bro > > /usr/local/bro/share/bro/base/files/x509/__load__.bro > > /usr/local/bro/share/bro/base/files/x509/main.bro > > /usr/local/bro/share/bro/base/files/hash/__load__.bro > > /usr/local/bro/share/bro/base/files/hash/main.bro > > /usr/local/bro/share/bro/base/protocols/http/__load__.bro > > /usr/local/bro/share/bro/base/protocols/http/main.bro > > /usr/local/bro/share/bro/base/protocols/http/entities.bro > > /usr/local/bro/share/bro/base/protocols/http/utils.bro > > /usr/local/bro/share/bro/base/protocols/http/files.bro > > /usr/local/bro/share/bro/base/protocols/irc/__load__.bro > > /usr/local/bro/share/bro/base/protocols/irc/main.bro > > /usr/local/bro/share/bro/base/protocols/irc/dcc-send.bro > > /usr/local/bro/share/bro/base/protocols/irc/files.bro > > /usr/local/bro/share/bro/base/protocols/krb/__load__.bro > > /usr/local/bro/share/bro/base/protocols/krb/main.bro > > /usr/local/bro/share/bro/base/protocols/krb/consts.bro > > /usr/local/bro/share/bro/base/protocols/krb/files.bro > > /usr/local/bro/share/bro/base/protocols/modbus/__load__.bro > > /usr/local/bro/share/bro/base/protocols/modbus/consts.bro > > /usr/local/bro/share/bro/base/protocols/modbus/main.bro > > /usr/local/bro/share/bro/base/protocols/mysql/__load__.bro > > /usr/local/bro/share/bro/base/protocols/mysql/main.bro > > /usr/local/bro/share/bro/base/protocols/mysql/consts.bro > > /usr/local/bro/share/bro/base/protocols/pop3/__load__.bro > > /usr/local/bro/share/bro/base/protocols/radius/__load__.bro > > /usr/local/bro/share/bro/base/protocols/radius/main.bro > > /usr/local/bro/share/bro/base/protocols/radius/consts.bro > > /usr/local/bro/share/bro/base/protocols/rdp/__load__.bro > > /usr/local/bro/share/bro/base/protocols/rdp/consts.bro > > /usr/local/bro/share/bro/base/protocols/rdp/main.bro > > /usr/local/bro/share/bro/base/protocols/sip/__load__.bro > > /usr/local/bro/share/bro/base/protocols/sip/main.bro > > /usr/local/bro/share/bro/base/protocols/snmp/__load__.bro > > /usr/local/bro/share/bro/base/protocols/snmp/main.bro > > /usr/local/bro/share/bro/base/protocols/smtp/__load__.bro > > /usr/local/bro/share/bro/base/protocols/smtp/main.bro > > /usr/local/bro/share/bro/base/protocols/smtp/entities.bro > > /usr/local/bro/share/bro/base/protocols/smtp/files.bro > > /usr/local/bro/share/bro/base/protocols/socks/__load__.bro > > /usr/local/bro/share/bro/base/protocols/socks/consts.bro > > /usr/local/bro/share/bro/base/protocols/socks/main.bro > > /usr/local/bro/share/bro/base/protocols/ssh/__load__.bro > > /usr/local/bro/share/bro/base/protocols/ssh/main.bro > > /usr/local/bro/share/bro/base/protocols/syslog/__load__.bro > > /usr/local/bro/share/bro/base/protocols/syslog/consts.bro > > /usr/local/bro/share/bro/base/protocols/syslog/main.bro > > /usr/local/bro/share/bro/base/protocols/tunnels/__load__.bro > > /usr/local/bro/share/bro/base/files/pe/__load__.bro > > /usr/local/bro/share/bro/base/files/pe/consts.bro > > /usr/local/bro/share/bro/base/files/pe/main.bro > > /usr/local/bro/share/bro/base/files/extract/__load__.bro > > /usr/local/bro/share/bro/base/files/extract/main.bro > > /usr/local/bro/share/bro/base/files/unified2/__load__.bro > > /usr/local/bro/share/bro/base/files/unified2/main.bro > > /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro > > /usr/local/bro/share/bro/base/misc/find-filtered-trace.bro > > /usr/local/bro/spool/installed-scripts-do-not-touch/site/local.bro > > /usr/local/bro/share/bro/policy/misc/loaded-scripts.bro > > /usr/local/bro/share/bro/policy/tuning/defaults/__load__.bro > > /usr/local/bro/share/bro/policy/tuning/defaults/packet-fragments.bro > > /usr/local/bro/share/bro/policy/tuning/defaults/warnings.bro > > > /usr/local/bro/share/bro/policy/tuning/defaults/extracted_file_limits.bro > > /usr/local/bro/share/bro/policy/misc/scan.bro > > /usr/local/bro/share/bro/policy/misc/app-stats/__load__.bro > > /usr/local/bro/share/bro/policy/misc/app-stats/main.bro > > /usr/local/bro/share/bro/policy/misc/app-stats/plugins/__load__.bro > > /usr/local/bro/share/bro/policy/misc/app-stats/plugins/facebook.bro > > /usr/local/bro/share/bro/policy/misc/detect-traceroute/__load__.bro > > /usr/local/bro/share/bro/policy/misc/detect-traceroute/main.bro > > /usr/local/bro/share/bro/policy/frameworks/software/vulnerable.bro > > /usr/local/bro/share/bro/policy/frameworks/software/version-changes.bro > > /usr/local/bro/share/bro/policy/protocols/ftp/software.bro > > /usr/local/bro/share/bro/policy/protocols/smtp/software.bro > > /usr/local/bro/share/bro/policy/protocols/ssh/software.bro > > /usr/local/bro/share/bro/policy/protocols/http/software.bro > > /usr/local/bro/share/bro/policy/protocols/dns/detect-external-names.bro > > /usr/local/bro/share/bro/policy/protocols/ftp/detect.bro > > /usr/local/bro/share/bro/policy/protocols/conn/known-hosts.bro > > /usr/local/bro/share/bro/policy/protocols/conn/known-services.bro > > /usr/local/bro/share/bro/policy/protocols/ssl/known-certs.bro > > /usr/local/bro/share/bro/policy/protocols/ssl/validate-certs.bro > > /usr/local/bro/share/bro/policy/protocols/ssl/log-hostcerts-only.bro > > /usr/local/bro/share/bro/policy/protocols/ssh/geo-data.bro > > /usr/local/bro/share/bro/policy/protocols/ssh/detect-bruteforcing.bro > > /usr/local/bro/share/bro/policy/protocols/ssh/interesting-hostnames.bro > > /usr/local/bro/share/bro/policy/protocols/http/detect-sqli.bro > > /usr/local/bro/share/bro/policy/frameworks/files/hash-all-files.bro > > /usr/local/bro/share/bro/policy/frameworks/files/detect-MHR.bro > > /usr/local/bro/share/bro/broctl/__load__.bro > > /usr/local/bro/share/bro/broctl/main.bro > > /usr/local/bro/share/bro/policy/frameworks/control/controllee.bro > > /usr/local/bro/spool/installed-scripts-do-not-touch/site/local-manager.bro > > /usr/local/bro/share/bro/broctl/auto.bro > > > /usr/local/bro/spool/installed-scripts-do-not-touch/auto/local-networks.bro > > > /usr/local/bro/spool/installed-scripts-do-not-touch/auto/broctl-config.bro > > [proxy-1] > > > Bro 2.4.1 > > Linux 3.13.0-48-generic > > > No gdb installed. > > > ==== No reporter.log > > > ==== stderr.log > > > ==== stdout.log > > max memory size (kbytes, -m) unlimited > > data seg size (kbytes, -d) unlimited > > virtual memory (kbytes, -v) unlimited > > core file size (blocks, -c) unlimited > > > ==== .cmdline > > -U .status -p broctl -p broctl-live -p local -p proxy-1 local.bro broctl > base/frameworks/cluster local-proxy broctl/auto > > > ==== .env_vars > > PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games > > BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site > > CLUSTER_NODE=proxy-1 > > > ==== .status > > RUNNING [net_run] > > > ==== No prof.log > > > ==== No packet_filter.log > > > ==== No loaded_scripts.log > > [worker-1] > > error running crash-diag for worker-1 > > Host 172.31.41.33 is not alive > > [worker-2] > > error running crash-diag for worker-2 > > Host 172.31.41.31 is not alive > > > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From Brett.Hite at parsons.com Tue Sep 15 10:41:56 2015 From: Brett.Hite at parsons.com (Hite, Brett) Date: Tue, 15 Sep 2015 17:41:56 +0000 Subject: [Bro] Previous version documentation Message-ID: I'm not able to easily find documentation for prior versions of Bro. Specifically, I'm looking for 2.3.1. Is this available somewhere? Thanks, Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150915/ff10f9b3/attachment.html From dnthayer at illinois.edu Tue Sep 15 11:44:45 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Tue, 15 Sep 2015 13:44:45 -0500 Subject: [Bro] Previous version documentation In-Reply-To: References: Message-ID: <55F8671D.10509@illinois.edu> You can download any older version of Bro, build it (./configure && make), and then build the documentation (make doc). You can then browse the documentation with a web browser (it is located in "build/html/"). There is a README in the "doc/" directory with more info. On 09/15/2015 12:41 PM, Hite, Brett wrote: > I'm not able to easily find documentation for prior versions of Bro. > Specifically, I'm looking for 2.3.1. Is this available somewhere? > > Thanks, > > Brett > > From brett.hite at parsons.com Tue Sep 15 11:49:49 2015 From: brett.hite at parsons.com (Brett Hite) Date: Tue, 15 Sep 2015 14:49:49 -0400 Subject: [Bro] Previous version documentation In-Reply-To: D09036F1F9F5BD488CE108D34A13438203ACE2EA@HSV-MB001.huntsville.ads.sparta.com Message-ID: <1442342989.6127.1.camel@Barn> I'll give that a look. Thank you! From dnthayer at illinois.edu Tue Sep 15 12:10:29 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Tue, 15 Sep 2015 14:10:29 -0500 Subject: [Bro] Broctl Worker Issues In-Reply-To: References: , <55F83C38.20702@illinois.edu> Message-ID: <55F86D25.9080402@illinois.edu> I'm guessing you probably ran the setcap on the manager. Actually, it really only needs to be run on the workers. However, doing a "broctl install" or "broctl deploy" will overwrite the bro executable on the worker machines, and then you'd need to do the "setcap" again before starting bro. In any case, the output of "broctl diag" is often useful to see why a bro node crashed. On 09/15/2015 12:24 PM, Davison, Charles Robert wrote: > 1. Complete > 2. Complete: sudo setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro > 3. Error: > > [BroControl] > ps.bro > USER PID PPID %CPU %MEM VSZ RSS TT S STARTED TIME COMMAND >>>> 172.31.41.32 >>>> 172.31.41.33 >>>> 172.31.41.31 > [BroControl] > deploy > checking configurations ... > installing ... > removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/site ... > removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/auto ... > creating policy directories ... > installing site policies ... > generating cluster-layout.bro ... > generating local-networks.bro ... > generating broctl-config.bro ... > generating broctl-config.sh ... > updating nodes ... > stopping ... > worker-1 not running (was crashed) > worker-2 not running (was crashed) > proxy-1 not running (was crashed) > manager not running (was crashed) > starting ... > starting manager ... > starting proxy-1 ... > starting worker-1 ... > starting worker-2 ... > worker-1 terminated immediately after starting; check output with "diag" > worker-2 terminated immediately after starting; check output with "diag" > [BroControl] > > > > > ________________________________________ > From: Daniel Thayer > Sent: Tuesday, September 15, 2015 9:41 AM > To: Davison, Charles Robert; bro at bro.org > Subject: Re: [Bro] Broctl Worker Issues > > 1) Make sure all Bro processes are stopped: > a) broctl stop > b) broctl ps.bro > If you see any Bro processes, then kill them before proceeding to > next step. If you see any error or warning messages, then > these need to be addressed before proceeding. > > 2) Since you're not running broctl as the "root" user, you need to make > sure bro workers have permission to capture packets: > > https://www.bro.org/documentation/faq.html#how-can-i-capture-packets-as-an-unprivileged-user > > 3) Start Bro > a) broctl deploy > b) There should not be any errors or warnings. > > > On 09/15/2015 08:41 AM, Davison, Charles Robert wrote: >> When I try and start broctl on all my workers I receive the following: >> >> >> ubuntu at ip-172-31-41-32:~$ /usr/local/bro/bin/broctl start >> >> starting manager ... >> >> starting proxy-1 ... >> >> starting worker-1 ... >> >> starting worker-2 ... >> >> worker-1 terminated immediately after starting; check output with "diag" >> >> worker-2 terminated immediately after starting; check output with "diag" >> >> >> >> This was my output from the diag: >> >> >> Bro 2.4.1 >> >> Linux 3.13.0-48-generic >> >> >> No gdb installed. >> >> >> ==== reporter.log >> >> #separator \x09 >> >> #set_separator , >> >> #empty_field (empty) >> >> #unset_field - >> >> #path reporter >> >> #open 2015-09-15-13-38-43 >> >> #fields ts level message location >> >> #types time enum string string >> >> 0.000000 Reporter::WARNING SumStat key request for the >> J1pRzdrrLK8 SumStat uid took longer than 1 minute and was automatically >> cancelled. >> /usr/local/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line 218 >> >> >> ==== stderr.log >> >> >> ==== stdout.log >> >> max memory size (kbytes, -m) unlimited >> >> data seg size (kbytes, -d) unlimited >> >> virtual memory (kbytes, -v) unlimited >> >> core file size (blocks, -c) unlimited >> >> >> ==== .cmdline >> >> -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl >> base/frameworks/cluster local-manager.bro broctl/auto >> >> >> ==== .env_vars >> >> PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games >> >> BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site >> >> CLUSTER_NODE=manager >> >> >> ==== .status >> >> RUNNING [net_run] >> >> >> ==== No prof.log >> >> >> ==== No packet_filter.log >> >> >> ==== loaded_scripts.log >> >> #separator \x09 >> >> #set_separator , >> >> #empty_field (empty) >> >> #unset_field - >> >> #path loaded_scripts >> >> #open 2015-09-15-13-34-43 >> >> #fields name >> >> #types string >> >> /usr/local/bro/share/bro/base/init-bare.bro >> >> /usr/local/bro/share/bro/base/bif/const.bif.bro >> >> /usr/local/bro/share/bro/base/bif/types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/strings.bif.bro >> >> /usr/local/bro/share/bro/base/bif/bro.bif.bro >> >> /usr/local/bro/share/bro/base/bif/reporter.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SNMP.types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_KRB.types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/event.bif.bro >> >> /usr/local/bro/share/bro/base/frameworks/broker/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/broker/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/logging/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/logging/main.bro >> >> /usr/local/bro/share/bro/base/bif/logging.bif.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/__load__.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/scp.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/sftp.bro >> >> /usr/local/bro/share/bro/base/frameworks/logging/writers/ascii.bro >> >> /usr/local/bro/share/bro/base/frameworks/logging/writers/sqlite.bro >> >> /usr/local/bro/share/bro/base/frameworks/logging/writers/none.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/main.bro >> >> /usr/local/bro/share/bro/base/bif/input.bif.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/readers/ascii.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/readers/raw.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/readers/benchmark.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/readers/binary.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/readers/sqlite.bro >> >> /usr/local/bro/share/bro/base/frameworks/analyzer/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/analyzer/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/packet-filter/utils.bro >> >> /usr/local/bro/share/bro/base/bif/analyzer.bif.bro >> >> /usr/local/bro/share/bro/base/frameworks/files/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/files/main.bro >> >> /usr/local/bro/share/bro/base/bif/file_analysis.bif.bro >> >> /usr/local/bro/share/bro/base/utils/site.bro >> >> /usr/local/bro/share/bro/base/utils/patterns.bro >> >> /usr/local/bro/share/bro/base/frameworks/files/magic/__load__.bro >> >> /usr/local/bro/share/bro/base/bif/__load__.bro >> >> /usr/local/bro/share/bro/base/bif/broxygen.bif.bro >> >> /usr/local/bro/share/bro/base/bif/pcap.bif.bro >> >> /usr/local/bro/share/bro/base/bif/bloom-filter.bif.bro >> >> /usr/local/bro/share/bro/base/bif/cardinality-counter.bif.bro >> >> /usr/local/bro/share/bro/base/bif/top-k.bif.bro >> >> /usr/local/bro/share/bro/base/bif/comm.bif.bro >> >> /usr/local/bro/share/bro/base/bif/data.bif.bro >> >> /usr/local/bro/share/bro/base/bif/messaging.bif.bro >> >> /usr/local/bro/share/bro/base/bif/store.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/__load__.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_ARP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_AYIYA.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_BackDoor.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_BitTorrent.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_ConnSize.events.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_ConnSize.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_DCE_RPC.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_DHCP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_DNP3.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_DNS.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_File.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Finger.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_FTP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_FTP.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Gnutella.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_GTPv1.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_HTTP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_HTTP.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_ICMP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Ident.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_InterConn.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_IRC.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_KRB.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Login.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Login.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_MIME.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Modbus.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_MySQL.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_NCP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_NetBIOS.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_NetBIOS.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_NTP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_PIA.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_POP3.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_RADIUS.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_RDP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_RDP.types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_RPC.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SIP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SNMP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SMB.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SMTP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SMTP.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SOCKS.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SSH.types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SSH.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SSL.events.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SteppingStone.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Syslog.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_TCP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_TCP.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Teredo.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_UDP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_ZIP.events.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_FileExtract.events.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_FileExtract.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_FileHash.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_PE.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Unified2.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Unified2.types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_AsciiReader.ascii.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_BenchmarkReader.benchmark.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_BinaryReader.binary.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_RawReader.raw.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SQLiteReader.sqlite.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_AsciiWriter.ascii.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_NoneWriter.none.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SQLiteWriter.sqlite.bif.bro >> >> /usr/local/bro/share/bro/base/init-default.bro >> >> /usr/local/bro/share/bro/base/utils/active-http.bro >> >> /usr/local/bro/share/bro/base/utils/exec.bro >> >> /usr/local/bro/share/bro/base/utils/addrs.bro >> >> /usr/local/bro/share/bro/base/utils/conn-ids.bro >> >> /usr/local/bro/share/bro/base/utils/dir.bro >> >> /usr/local/bro/share/bro/base/frameworks/reporter/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/reporter/main.bro >> >> /usr/local/bro/share/bro/base/utils/paths.bro >> >> /usr/local/bro/share/bro/base/utils/directions-and-hosts.bro >> >> /usr/local/bro/share/bro/base/utils/files.bro >> >> /usr/local/bro/share/bro/base/utils/numbers.bro >> >> /usr/local/bro/share/bro/base/utils/queue.bro >> >> /usr/local/bro/share/bro/base/utils/strings.bro >> >> /usr/local/bro/share/bro/base/utils/thresholds.bro >> >> /usr/local/bro/share/bro/base/utils/time.bro >> >> /usr/local/bro/share/bro/base/utils/urls.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/weird.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/actions/drop.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/actions/email_admin.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/actions/page.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/actions/add-geodata.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/notice/extend-email/hostnames.bro >> >> /usr/local/bro/share/bro/base/frameworks/cluster/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/cluster/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/control/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/control/main.bro >> >> >> /usr/local/bro/spool/installed-scripts-do-not-touch/auto/cluster-layout.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/cluster/setup-connections.bro >> >> /usr/local/bro/share/bro/base/frameworks/communication/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/communication/main.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/packet-filter/__load__.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/packet-filter/main.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/packet-filter/netstats.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/packet-filter/cluster.bro >> >> /usr/local/bro/share/bro/policy/frameworks/communication/listen.bro >> >> /usr/local/bro/share/bro/base/frameworks/cluster/nodes/manager.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/cluster.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro >> >> /usr/local/bro/share/bro/base/frameworks/dpd/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/dpd/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/signatures/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/signatures/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/software/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/software/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/intel/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/intel/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/intel/cluster.bro >> >> /usr/local/bro/share/bro/base/frameworks/intel/input.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/average.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/hll_unique.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/last.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/max.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/min.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/sample.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/std-dev.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/variance.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/sum.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/topk.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/unique.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/cluster.bro >> >> /usr/local/bro/share/bro/base/frameworks/tunnels/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/tunnels/main.bro >> >> /usr/local/bro/share/bro/base/protocols/conn/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/conn/main.bro >> >> /usr/local/bro/share/bro/base/protocols/conn/contents.bro >> >> /usr/local/bro/share/bro/base/protocols/conn/inactivity.bro >> >> /usr/local/bro/share/bro/base/protocols/conn/polling.bro >> >> /usr/local/bro/share/bro/base/protocols/conn/thresholds.bro >> >> /usr/local/bro/share/bro/base/protocols/dhcp/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/dhcp/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/dhcp/main.bro >> >> /usr/local/bro/share/bro/base/protocols/dhcp/utils.bro >> >> /usr/local/bro/share/bro/base/protocols/dnp3/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/dnp3/main.bro >> >> /usr/local/bro/share/bro/base/protocols/dnp3/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/dns/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/dns/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/dns/main.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/utils-commands.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/info.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/main.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/utils.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/files.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/gridftp.bro >> >> /usr/local/bro/share/bro/base/protocols/ssl/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/ssl/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/ssl/main.bro >> >> /usr/local/bro/share/bro/base/protocols/ssl/mozilla-ca-list.bro >> >> /usr/local/bro/share/bro/base/protocols/ssl/files.bro >> >> /usr/local/bro/share/bro/base/files/x509/__load__.bro >> >> /usr/local/bro/share/bro/base/files/x509/main.bro >> >> /usr/local/bro/share/bro/base/files/hash/__load__.bro >> >> /usr/local/bro/share/bro/base/files/hash/main.bro >> >> /usr/local/bro/share/bro/base/protocols/http/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/http/main.bro >> >> /usr/local/bro/share/bro/base/protocols/http/entities.bro >> >> /usr/local/bro/share/bro/base/protocols/http/utils.bro >> >> /usr/local/bro/share/bro/base/protocols/http/files.bro >> >> /usr/local/bro/share/bro/base/protocols/irc/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/irc/main.bro >> >> /usr/local/bro/share/bro/base/protocols/irc/dcc-send.bro >> >> /usr/local/bro/share/bro/base/protocols/irc/files.bro >> >> /usr/local/bro/share/bro/base/protocols/krb/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/krb/main.bro >> >> /usr/local/bro/share/bro/base/protocols/krb/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/krb/files.bro >> >> /usr/local/bro/share/bro/base/protocols/modbus/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/modbus/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/modbus/main.bro >> >> /usr/local/bro/share/bro/base/protocols/mysql/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/mysql/main.bro >> >> /usr/local/bro/share/bro/base/protocols/mysql/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/pop3/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/radius/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/radius/main.bro >> >> /usr/local/bro/share/bro/base/protocols/radius/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/rdp/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/rdp/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/rdp/main.bro >> >> /usr/local/bro/share/bro/base/protocols/sip/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/sip/main.bro >> >> /usr/local/bro/share/bro/base/protocols/snmp/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/snmp/main.bro >> >> /usr/local/bro/share/bro/base/protocols/smtp/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/smtp/main.bro >> >> /usr/local/bro/share/bro/base/protocols/smtp/entities.bro >> >> /usr/local/bro/share/bro/base/protocols/smtp/files.bro >> >> /usr/local/bro/share/bro/base/protocols/socks/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/socks/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/socks/main.bro >> >> /usr/local/bro/share/bro/base/protocols/ssh/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/ssh/main.bro >> >> /usr/local/bro/share/bro/base/protocols/syslog/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/syslog/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/syslog/main.bro >> >> /usr/local/bro/share/bro/base/protocols/tunnels/__load__.bro >> >> /usr/local/bro/share/bro/base/files/pe/__load__.bro >> >> /usr/local/bro/share/bro/base/files/pe/consts.bro >> >> /usr/local/bro/share/bro/base/files/pe/main.bro >> >> /usr/local/bro/share/bro/base/files/extract/__load__.bro >> >> /usr/local/bro/share/bro/base/files/extract/main.bro >> >> /usr/local/bro/share/bro/base/files/unified2/__load__.bro >> >> /usr/local/bro/share/bro/base/files/unified2/main.bro >> >> /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro >> >> /usr/local/bro/share/bro/base/misc/find-filtered-trace.bro >> >> /usr/local/bro/spool/installed-scripts-do-not-touch/site/local.bro >> >> /usr/local/bro/share/bro/policy/misc/loaded-scripts.bro >> >> /usr/local/bro/share/bro/policy/tuning/defaults/__load__.bro >> >> /usr/local/bro/share/bro/policy/tuning/defaults/packet-fragments.bro >> >> /usr/local/bro/share/bro/policy/tuning/defaults/warnings.bro >> >> >> /usr/local/bro/share/bro/policy/tuning/defaults/extracted_file_limits.bro >> >> /usr/local/bro/share/bro/policy/misc/scan.bro >> >> /usr/local/bro/share/bro/policy/misc/app-stats/__load__.bro >> >> /usr/local/bro/share/bro/policy/misc/app-stats/main.bro >> >> /usr/local/bro/share/bro/policy/misc/app-stats/plugins/__load__.bro >> >> /usr/local/bro/share/bro/policy/misc/app-stats/plugins/facebook.bro >> >> /usr/local/bro/share/bro/policy/misc/detect-traceroute/__load__.bro >> >> /usr/local/bro/share/bro/policy/misc/detect-traceroute/main.bro >> >> /usr/local/bro/share/bro/policy/frameworks/software/vulnerable.bro >> >> /usr/local/bro/share/bro/policy/frameworks/software/version-changes.bro >> >> /usr/local/bro/share/bro/policy/protocols/ftp/software.bro >> >> /usr/local/bro/share/bro/policy/protocols/smtp/software.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssh/software.bro >> >> /usr/local/bro/share/bro/policy/protocols/http/software.bro >> >> /usr/local/bro/share/bro/policy/protocols/dns/detect-external-names.bro >> >> /usr/local/bro/share/bro/policy/protocols/ftp/detect.bro >> >> /usr/local/bro/share/bro/policy/protocols/conn/known-hosts.bro >> >> /usr/local/bro/share/bro/policy/protocols/conn/known-services.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssl/known-certs.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssl/validate-certs.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssl/log-hostcerts-only.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssh/geo-data.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssh/detect-bruteforcing.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssh/interesting-hostnames.bro >> >> /usr/local/bro/share/bro/policy/protocols/http/detect-sqli.bro >> >> /usr/local/bro/share/bro/policy/frameworks/files/hash-all-files.bro >> >> /usr/local/bro/share/bro/policy/frameworks/files/detect-MHR.bro >> >> /usr/local/bro/share/bro/broctl/__load__.bro >> >> /usr/local/bro/share/bro/broctl/main.bro >> >> /usr/local/bro/share/bro/policy/frameworks/control/controllee.bro >> >> /usr/local/bro/spool/installed-scripts-do-not-touch/site/local-manager.bro >> >> /usr/local/bro/share/bro/broctl/auto.bro >> >> >> /usr/local/bro/spool/installed-scripts-do-not-touch/auto/local-networks.bro >> >> >> /usr/local/bro/spool/installed-scripts-do-not-touch/auto/broctl-config.bro >> >> [proxy-1] >> >> >> Bro 2.4.1 >> >> Linux 3.13.0-48-generic >> >> >> No gdb installed. >> >> >> ==== No reporter.log >> >> >> ==== stderr.log >> >> >> ==== stdout.log >> >> max memory size (kbytes, -m) unlimited >> >> data seg size (kbytes, -d) unlimited >> >> virtual memory (kbytes, -v) unlimited >> >> core file size (blocks, -c) unlimited >> >> >> ==== .cmdline >> >> -U .status -p broctl -p broctl-live -p local -p proxy-1 local.bro broctl >> base/frameworks/cluster local-proxy broctl/auto >> >> >> ==== .env_vars >> >> PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games >> >> BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site >> >> CLUSTER_NODE=proxy-1 >> >> >> ==== .status >> >> RUNNING [net_run] >> >> >> ==== No prof.log >> >> >> ==== No packet_filter.log >> >> >> ==== No loaded_scripts.log >> >> [worker-1] >> >> error running crash-diag for worker-1 >> >> Host 172.31.41.33 is not alive >> >> [worker-2] >> >> error running crash-diag for worker-2 >> >> Host 172.31.41.31 is not alive >> >> >> >> >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> From rdump at river.com Tue Sep 15 15:07:29 2015 From: rdump at river.com (Richard Johnson) Date: Tue, 15 Sep 2015 16:07:29 -0600 Subject: [Bro] restrict_filters not preventing logging of selected IP addresses In-Reply-To: <5533F305.6010101@river.com> References: <5533F305.6010101@river.com> Message-ID: <55F896A1.40603@river.com> On 2015-04-19 12:25, Richard Johnson wrote: > I think I'm specifying restrict_filters correctly to stop some hosts from > being logged, but it's not working as I intend/expect. ... > Yet when the restrict_filter is OK and is seemingly recognized, the IP > addresses in the restrict_filters still appear in log entries. ... > [manager-host ~]$ grep capture_filters /raid/bro/share/bro/site/local.bro > redef capture_filters = { ["all"] = "ip or not ip" }; > [manager-host ~]$ grep restrict_filters /raid/bro/share/bro/site/local.bro > redef restrict_filters += { ["not-these-hosts"] = "not host 172.16.1.1 and not > host 172.16.22.22 and not host 172.16.39.39 and not host 172.16.88.88" }; ... > [manager-host current]$ grep 172.16.88.88 conn.log | tail -3 > 1429461245.805348 CpuepS3Ds2GYzABCtb xx.xx.xx.xx xxxxx > 172.16.88.88 443 tcp ssl 4192.655995 14660 16441 S1 > F 0ShADda 50 17268 49 19001 (empty) > 1429464730.699197 CqVMY53iVvTFSWclAi xx.xx.xx.xx xxxxx > 172.16.88.88 443 tcp ssl 1002.988461 5491 4481 SF > F 0ShADdaFf 21 6591 17 5377 (empty) > 1429464286.982078 CUl3Cl24bUWkgbhAGd xx.xx.xx.xx xxxxx > 172.16.88.88 443 tcp ssl 1447.315821 7095 5595 SF > F 0ShADdafF 25 8403 21 6699 (empty) For the record, this is solved, thanks to the distributed kibitzing of Adam Slagell, Vern Paxson, Seth Hall, and others in the hallway track at BroCon 2015. "Check for VLAN tags." "Try 'vlan ####' in capture_filters." Our upstream feed had been switched to a trunk, and began carrying other VLANs in addition to the main tap feed we were expecting. When that happened, Bro quietly stepped past the VLAN tags in policy processing. As a result, there was no Bro monitoring outage. We just had some duplicate and unintentionally monitored connections which we didn't spot due to low volume. Thus the change slipped past us. However, the pcap filter specification in restrict_filters would no longer match due to the VLAN tags. Specifying the VLAN(s) to watch in capture_filters clears the match for the IP addresses in restrict_filters. Our fix, in local.bro (where 321 is the VLAN number of the tap feed): ------- redef capture_filters = { ["all"] = "vlan 321 and (ip or not ip)" } redef restrict_filters += { ["not-these-hosts"] = "not host 172.16.1.1 and not host 172.16.22.22 and not host 172.16.39.39 and not host 172.16.88.88" }; ------- Richard From hckim at narusec.com Tue Sep 15 22:51:00 2015 From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=) Date: Wed, 16 Sep 2015 14:51:00 +0900 Subject: [Bro] Any plans to use p0f V3 signature? Message-ID: Hi I have been using p0f -v1.8.3 fingerprints but having some issue bro is printing out couple of OS from same IP module osfound; redef generate_OS_version_event: set[subnet]={172.16.0.0/16, 192.168.0.0/16 }; export { redef enum Log::ID += { LOG }; type Info: record { ts: time &log; uid: string &log; srcip: addr &log; ostype: string &log &optional; }; } event bro_init() { Log::create_stream(osfound::LOG, [$columns = Info]); } event OS_version_found(c:connection, host:addr, OS:OS_version) { local log: Info; log = [$ts = c$start_time, $uid = c$uid, $srcip = host, $ostype = fmt("%s",OS)]; Log::write(osfound::LOG, log); } ------------ 1442380383.955525 CKYeuj3FmWKkSkvqja 192.168.0.xx [genre=iOS, detail=3.x, 4.2, dist=0, match_type=direct_inference] 1442380384.611330 CMVD6fzeHEGS4Q7el 192.168.0.xx [genre=UNKNOWN, detail=, dist=0, match_type=direct_inference] 1442380805.630824 CBytoWhjC7bBWFlKj 192.168.0.aa [genre=Windows, detail=Vista SP0/SP2, 7 SP0+, 2008 SP0, dist=0, match_type=direct_inference] 1442380811.907225 COmECC4qM5njKIsncb 192.168.0.aa [genre=Windows, detail=2000 SP2+, XP SP1+ (seldom 98), Vista SP1, 7 SP1, 2008 SP2, dist=0, match_type=direct_inference] So I test p0f - v3, so far I did not have this issue.( just p0f -i eth1 -a os.log) am I having this issue because of my bro script ? if not do you have any plans to use p0f - v3 (or fingerprints )? -- ------------------------------------------------------ Hichul Kim ??? ?? ??? Naru Security (?)?????? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150916/882cb0b3/attachment.html From cdaviso1 at vols.utk.edu Wed Sep 16 09:54:07 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Wed, 16 Sep 2015 16:54:07 +0000 Subject: [Bro] Install Instructions for gperftools Message-ID: Does anyone have install instructions for gperftools on Ubuntu 14.04? I received the following output before installing bro and wanted to make sure it was included: Broccoli: true Broctl: true Aux. Tools: true GeoIP: true gperftools found: false tcmalloc: false debugging: false jemalloc: false -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150916/83ee69cb/attachment.html From hosom at battelle.org Wed Sep 16 10:36:35 2015 From: hosom at battelle.org (Hosom, Stephen M) Date: Wed, 16 Sep 2015 17:36:35 +0000 Subject: [Bro] Install Instructions for gperftools In-Reply-To: References: Message-ID: I believe this is: sudo apt-get install libgoogle-perftools-dev From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Davison, Charles Robert Sent: Wednesday, September 16, 2015 12:54 PM To: bro at bro.org Subject: [Bro] Install Instructions for gperftools Does anyone have install instructions for gperftools on Ubuntu 14.04? I received the following output before installing bro and wanted to make sure it was included: Broccoli: true Broctl: true Aux. Tools: true GeoIP: true gperftools found: false tcmalloc: false debugging: false jemalloc: false -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150916/42fd7904/attachment-0001.html From cdaviso1 at vols.utk.edu Wed Sep 16 10:45:54 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Wed, 16 Sep 2015 17:45:54 +0000 Subject: [Bro] Install Instructions for gperftools In-Reply-To: References: , Message-ID: Thank you, I added this to your suggestion and it is working! sudo apt-get install google-perftools libgoogle-perftools-dev ________________________________ From: Hosom, Stephen M Sent: Wednesday, September 16, 2015 11:36 AM To: Davison, Charles Robert; bro at bro.org Subject: RE: Install Instructions for gperftools I believe this is: sudo apt-get install libgoogle-perftools-dev From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Davison, Charles Robert Sent: Wednesday, September 16, 2015 12:54 PM To: bro at bro.org Subject: [Bro] Install Instructions for gperftools Does anyone have install instructions for gperftools on Ubuntu 14.04? I received the following output before installing bro and wanted to make sure it was included: Broccoli: true Broctl: true Aux. Tools: true GeoIP: true gperftools found: false tcmalloc: false debugging: false jemalloc: false -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150916/b7a5fd10/attachment.html From jazoff at illinois.edu Wed Sep 16 10:49:33 2015 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 16 Sep 2015 17:49:33 +0000 Subject: [Bro] Install Instructions for gperftools In-Reply-To: References: Message-ID: > On Sep 16, 2015, at 1:36 PM, Hosom, Stephen M wrote: > > I believe this is: > > sudo apt-get install libgoogle-perftools-dev > Yeah.. just keep in mind that if you build bro against perftools on the manager, the worker boxes will need the library installed as well. Otherwise the resulting bro binary will fail to start. -- - Justin Azoff From cdaviso1 at vols.utk.edu Wed Sep 16 11:48:26 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Wed, 16 Sep 2015 18:48:26 +0000 Subject: [Bro] Broctl Worker Issues In-Reply-To: <55F86D25.9080402@illinois.edu> References: , <55F83C38.20702@illinois.edu> , <55F86D25.9080402@illinois.edu> Message-ID: Finally working, I started from scratch and made sure my workers had all the same dependencies and optional dependencies as my manager did and everything is working now: [BroControl] > status Getting process status ... Getting peer status ... Name Type Host Status Pid Peers Started manager manager 172.31.38.121 running 1440 2 16 Sep 18:43:35 proxy-1 proxy 172.31.38.121 running 1479 2 16 Sep 18:43:36 worker-1 worker 172.31.38.122 running 5550 2 16 Sep 18:43:38 [BroControl] > ________________________________________ From: Daniel Thayer Sent: Tuesday, September 15, 2015 1:10 PM To: Davison, Charles Robert; bro at bro.org Subject: Re: [Bro] Broctl Worker Issues I'm guessing you probably ran the setcap on the manager. Actually, it really only needs to be run on the workers. However, doing a "broctl install" or "broctl deploy" will overwrite the bro executable on the worker machines, and then you'd need to do the "setcap" again before starting bro. In any case, the output of "broctl diag" is often useful to see why a bro node crashed. On 09/15/2015 12:24 PM, Davison, Charles Robert wrote: > 1. Complete > 2. Complete: sudo setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro > 3. Error: > > [BroControl] > ps.bro > USER PID PPID %CPU %MEM VSZ RSS TT S STARTED TIME COMMAND >>>> 172.31.41.32 >>>> 172.31.41.33 >>>> 172.31.41.31 > [BroControl] > deploy > checking configurations ... > installing ... > removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/site ... > removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/auto ... > creating policy directories ... > installing site policies ... > generating cluster-layout.bro ... > generating local-networks.bro ... > generating broctl-config.bro ... > generating broctl-config.sh ... > updating nodes ... > stopping ... > worker-1 not running (was crashed) > worker-2 not running (was crashed) > proxy-1 not running (was crashed) > manager not running (was crashed) > starting ... > starting manager ... > starting proxy-1 ... > starting worker-1 ... > starting worker-2 ... > worker-1 terminated immediately after starting; check output with "diag" > worker-2 terminated immediately after starting; check output with "diag" > [BroControl] > > > > > ________________________________________ > From: Daniel Thayer > Sent: Tuesday, September 15, 2015 9:41 AM > To: Davison, Charles Robert; bro at bro.org > Subject: Re: [Bro] Broctl Worker Issues > > 1) Make sure all Bro processes are stopped: > a) broctl stop > b) broctl ps.bro > If you see any Bro processes, then kill them before proceeding to > next step. If you see any error or warning messages, then > these need to be addressed before proceeding. > > 2) Since you're not running broctl as the "root" user, you need to make > sure bro workers have permission to capture packets: > > https://www.bro.org/documentation/faq.html#how-can-i-capture-packets-as-an-unprivileged-user > > 3) Start Bro > a) broctl deploy > b) There should not be any errors or warnings. > > > On 09/15/2015 08:41 AM, Davison, Charles Robert wrote: >> When I try and start broctl on all my workers I receive the following: >> >> >> ubuntu at ip-172-31-41-32:~$ /usr/local/bro/bin/broctl start >> >> starting manager ... >> >> starting proxy-1 ... >> >> starting worker-1 ... >> >> starting worker-2 ... >> >> worker-1 terminated immediately after starting; check output with "diag" >> >> worker-2 terminated immediately after starting; check output with "diag" >> >> >> >> This was my output from the diag: >> >> >> Bro 2.4.1 >> >> Linux 3.13.0-48-generic >> >> >> No gdb installed. >> >> >> ==== reporter.log >> >> #separator \x09 >> >> #set_separator , >> >> #empty_field (empty) >> >> #unset_field - >> >> #path reporter >> >> #open 2015-09-15-13-38-43 >> >> #fields ts level message location >> >> #types time enum string string >> >> 0.000000 Reporter::WARNING SumStat key request for the >> J1pRzdrrLK8 SumStat uid took longer than 1 minute and was automatically >> cancelled. >> /usr/local/bro/share/bro/base/frameworks/sumstats/./cluster.bro, line 218 >> >> >> ==== stderr.log >> >> >> ==== stdout.log >> >> max memory size (kbytes, -m) unlimited >> >> data seg size (kbytes, -d) unlimited >> >> virtual memory (kbytes, -v) unlimited >> >> core file size (blocks, -c) unlimited >> >> >> ==== .cmdline >> >> -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl >> base/frameworks/cluster local-manager.bro broctl/auto >> >> >> ==== .env_vars >> >> PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games >> >> BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site >> >> CLUSTER_NODE=manager >> >> >> ==== .status >> >> RUNNING [net_run] >> >> >> ==== No prof.log >> >> >> ==== No packet_filter.log >> >> >> ==== loaded_scripts.log >> >> #separator \x09 >> >> #set_separator , >> >> #empty_field (empty) >> >> #unset_field - >> >> #path loaded_scripts >> >> #open 2015-09-15-13-34-43 >> >> #fields name >> >> #types string >> >> /usr/local/bro/share/bro/base/init-bare.bro >> >> /usr/local/bro/share/bro/base/bif/const.bif.bro >> >> /usr/local/bro/share/bro/base/bif/types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/strings.bif.bro >> >> /usr/local/bro/share/bro/base/bif/bro.bif.bro >> >> /usr/local/bro/share/bro/base/bif/reporter.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SNMP.types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_KRB.types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/event.bif.bro >> >> /usr/local/bro/share/bro/base/frameworks/broker/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/broker/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/logging/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/logging/main.bro >> >> /usr/local/bro/share/bro/base/bif/logging.bif.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/__load__.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/scp.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/logging/postprocessors/sftp.bro >> >> /usr/local/bro/share/bro/base/frameworks/logging/writers/ascii.bro >> >> /usr/local/bro/share/bro/base/frameworks/logging/writers/sqlite.bro >> >> /usr/local/bro/share/bro/base/frameworks/logging/writers/none.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/main.bro >> >> /usr/local/bro/share/bro/base/bif/input.bif.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/readers/ascii.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/readers/raw.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/readers/benchmark.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/readers/binary.bro >> >> /usr/local/bro/share/bro/base/frameworks/input/readers/sqlite.bro >> >> /usr/local/bro/share/bro/base/frameworks/analyzer/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/analyzer/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/packet-filter/utils.bro >> >> /usr/local/bro/share/bro/base/bif/analyzer.bif.bro >> >> /usr/local/bro/share/bro/base/frameworks/files/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/files/main.bro >> >> /usr/local/bro/share/bro/base/bif/file_analysis.bif.bro >> >> /usr/local/bro/share/bro/base/utils/site.bro >> >> /usr/local/bro/share/bro/base/utils/patterns.bro >> >> /usr/local/bro/share/bro/base/frameworks/files/magic/__load__.bro >> >> /usr/local/bro/share/bro/base/bif/__load__.bro >> >> /usr/local/bro/share/bro/base/bif/broxygen.bif.bro >> >> /usr/local/bro/share/bro/base/bif/pcap.bif.bro >> >> /usr/local/bro/share/bro/base/bif/bloom-filter.bif.bro >> >> /usr/local/bro/share/bro/base/bif/cardinality-counter.bif.bro >> >> /usr/local/bro/share/bro/base/bif/top-k.bif.bro >> >> /usr/local/bro/share/bro/base/bif/comm.bif.bro >> >> /usr/local/bro/share/bro/base/bif/data.bif.bro >> >> /usr/local/bro/share/bro/base/bif/messaging.bif.bro >> >> /usr/local/bro/share/bro/base/bif/store.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/__load__.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_ARP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_AYIYA.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_BackDoor.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_BitTorrent.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_ConnSize.events.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_ConnSize.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_DCE_RPC.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_DHCP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_DNP3.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_DNS.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_File.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Finger.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_FTP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_FTP.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Gnutella.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_GTPv1.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_HTTP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_HTTP.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_ICMP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Ident.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_InterConn.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_IRC.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_KRB.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Login.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Login.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_MIME.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Modbus.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_MySQL.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_NCP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_NetBIOS.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_NetBIOS.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_NTP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_PIA.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_POP3.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_RADIUS.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_RDP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_RDP.types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_RPC.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SIP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SNMP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SMB.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SMTP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SMTP.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SOCKS.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SSH.types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SSH.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SSL.events.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SteppingStone.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Syslog.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_TCP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_TCP.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Teredo.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_UDP.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_ZIP.events.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_FileExtract.events.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_FileExtract.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_FileHash.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_PE.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Unified2.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_Unified2.types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.events.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.types.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_X509.functions.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_AsciiReader.ascii.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_BenchmarkReader.benchmark.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_BinaryReader.binary.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_RawReader.raw.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SQLiteReader.sqlite.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_AsciiWriter.ascii.bif.bro >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_NoneWriter.none.bif.bro >> >> >> /usr/local/bro/share/bro/base/bif/plugins/Bro_SQLiteWriter.sqlite.bif.bro >> >> /usr/local/bro/share/bro/base/init-default.bro >> >> /usr/local/bro/share/bro/base/utils/active-http.bro >> >> /usr/local/bro/share/bro/base/utils/exec.bro >> >> /usr/local/bro/share/bro/base/utils/addrs.bro >> >> /usr/local/bro/share/bro/base/utils/conn-ids.bro >> >> /usr/local/bro/share/bro/base/utils/dir.bro >> >> /usr/local/bro/share/bro/base/frameworks/reporter/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/reporter/main.bro >> >> /usr/local/bro/share/bro/base/utils/paths.bro >> >> /usr/local/bro/share/bro/base/utils/directions-and-hosts.bro >> >> /usr/local/bro/share/bro/base/utils/files.bro >> >> /usr/local/bro/share/bro/base/utils/numbers.bro >> >> /usr/local/bro/share/bro/base/utils/queue.bro >> >> /usr/local/bro/share/bro/base/utils/strings.bro >> >> /usr/local/bro/share/bro/base/utils/thresholds.bro >> >> /usr/local/bro/share/bro/base/utils/time.bro >> >> /usr/local/bro/share/bro/base/utils/urls.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/weird.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/actions/drop.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/actions/email_admin.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/actions/page.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/actions/add-geodata.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/notice/extend-email/hostnames.bro >> >> /usr/local/bro/share/bro/base/frameworks/cluster/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/cluster/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/control/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/control/main.bro >> >> >> /usr/local/bro/spool/installed-scripts-do-not-touch/auto/cluster-layout.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/cluster/setup-connections.bro >> >> /usr/local/bro/share/bro/base/frameworks/communication/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/communication/main.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/packet-filter/__load__.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/packet-filter/main.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/packet-filter/netstats.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/packet-filter/cluster.bro >> >> /usr/local/bro/share/bro/policy/frameworks/communication/listen.bro >> >> /usr/local/bro/share/bro/base/frameworks/cluster/nodes/manager.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/cluster.bro >> >> /usr/local/bro/share/bro/base/frameworks/notice/actions/pp-alarms.bro >> >> /usr/local/bro/share/bro/base/frameworks/dpd/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/dpd/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/signatures/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/signatures/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/software/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/software/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/intel/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/intel/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/intel/cluster.bro >> >> /usr/local/bro/share/bro/base/frameworks/intel/input.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/main.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/average.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/hll_unique.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/last.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/max.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/min.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/sample.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/std-dev.bro >> >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/variance.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/sum.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/topk.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/plugins/unique.bro >> >> /usr/local/bro/share/bro/base/frameworks/sumstats/cluster.bro >> >> /usr/local/bro/share/bro/base/frameworks/tunnels/__load__.bro >> >> /usr/local/bro/share/bro/base/frameworks/tunnels/main.bro >> >> /usr/local/bro/share/bro/base/protocols/conn/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/conn/main.bro >> >> /usr/local/bro/share/bro/base/protocols/conn/contents.bro >> >> /usr/local/bro/share/bro/base/protocols/conn/inactivity.bro >> >> /usr/local/bro/share/bro/base/protocols/conn/polling.bro >> >> /usr/local/bro/share/bro/base/protocols/conn/thresholds.bro >> >> /usr/local/bro/share/bro/base/protocols/dhcp/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/dhcp/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/dhcp/main.bro >> >> /usr/local/bro/share/bro/base/protocols/dhcp/utils.bro >> >> /usr/local/bro/share/bro/base/protocols/dnp3/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/dnp3/main.bro >> >> /usr/local/bro/share/bro/base/protocols/dnp3/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/dns/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/dns/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/dns/main.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/utils-commands.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/info.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/main.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/utils.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/files.bro >> >> /usr/local/bro/share/bro/base/protocols/ftp/gridftp.bro >> >> /usr/local/bro/share/bro/base/protocols/ssl/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/ssl/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/ssl/main.bro >> >> /usr/local/bro/share/bro/base/protocols/ssl/mozilla-ca-list.bro >> >> /usr/local/bro/share/bro/base/protocols/ssl/files.bro >> >> /usr/local/bro/share/bro/base/files/x509/__load__.bro >> >> /usr/local/bro/share/bro/base/files/x509/main.bro >> >> /usr/local/bro/share/bro/base/files/hash/__load__.bro >> >> /usr/local/bro/share/bro/base/files/hash/main.bro >> >> /usr/local/bro/share/bro/base/protocols/http/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/http/main.bro >> >> /usr/local/bro/share/bro/base/protocols/http/entities.bro >> >> /usr/local/bro/share/bro/base/protocols/http/utils.bro >> >> /usr/local/bro/share/bro/base/protocols/http/files.bro >> >> /usr/local/bro/share/bro/base/protocols/irc/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/irc/main.bro >> >> /usr/local/bro/share/bro/base/protocols/irc/dcc-send.bro >> >> /usr/local/bro/share/bro/base/protocols/irc/files.bro >> >> /usr/local/bro/share/bro/base/protocols/krb/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/krb/main.bro >> >> /usr/local/bro/share/bro/base/protocols/krb/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/krb/files.bro >> >> /usr/local/bro/share/bro/base/protocols/modbus/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/modbus/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/modbus/main.bro >> >> /usr/local/bro/share/bro/base/protocols/mysql/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/mysql/main.bro >> >> /usr/local/bro/share/bro/base/protocols/mysql/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/pop3/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/radius/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/radius/main.bro >> >> /usr/local/bro/share/bro/base/protocols/radius/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/rdp/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/rdp/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/rdp/main.bro >> >> /usr/local/bro/share/bro/base/protocols/sip/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/sip/main.bro >> >> /usr/local/bro/share/bro/base/protocols/snmp/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/snmp/main.bro >> >> /usr/local/bro/share/bro/base/protocols/smtp/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/smtp/main.bro >> >> /usr/local/bro/share/bro/base/protocols/smtp/entities.bro >> >> /usr/local/bro/share/bro/base/protocols/smtp/files.bro >> >> /usr/local/bro/share/bro/base/protocols/socks/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/socks/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/socks/main.bro >> >> /usr/local/bro/share/bro/base/protocols/ssh/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/ssh/main.bro >> >> /usr/local/bro/share/bro/base/protocols/syslog/__load__.bro >> >> /usr/local/bro/share/bro/base/protocols/syslog/consts.bro >> >> /usr/local/bro/share/bro/base/protocols/syslog/main.bro >> >> /usr/local/bro/share/bro/base/protocols/tunnels/__load__.bro >> >> /usr/local/bro/share/bro/base/files/pe/__load__.bro >> >> /usr/local/bro/share/bro/base/files/pe/consts.bro >> >> /usr/local/bro/share/bro/base/files/pe/main.bro >> >> /usr/local/bro/share/bro/base/files/extract/__load__.bro >> >> /usr/local/bro/share/bro/base/files/extract/main.bro >> >> /usr/local/bro/share/bro/base/files/unified2/__load__.bro >> >> /usr/local/bro/share/bro/base/files/unified2/main.bro >> >> /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro >> >> /usr/local/bro/share/bro/base/misc/find-filtered-trace.bro >> >> /usr/local/bro/spool/installed-scripts-do-not-touch/site/local.bro >> >> /usr/local/bro/share/bro/policy/misc/loaded-scripts.bro >> >> /usr/local/bro/share/bro/policy/tuning/defaults/__load__.bro >> >> /usr/local/bro/share/bro/policy/tuning/defaults/packet-fragments.bro >> >> /usr/local/bro/share/bro/policy/tuning/defaults/warnings.bro >> >> >> /usr/local/bro/share/bro/policy/tuning/defaults/extracted_file_limits.bro >> >> /usr/local/bro/share/bro/policy/misc/scan.bro >> >> /usr/local/bro/share/bro/policy/misc/app-stats/__load__.bro >> >> /usr/local/bro/share/bro/policy/misc/app-stats/main.bro >> >> /usr/local/bro/share/bro/policy/misc/app-stats/plugins/__load__.bro >> >> /usr/local/bro/share/bro/policy/misc/app-stats/plugins/facebook.bro >> >> /usr/local/bro/share/bro/policy/misc/detect-traceroute/__load__.bro >> >> /usr/local/bro/share/bro/policy/misc/detect-traceroute/main.bro >> >> /usr/local/bro/share/bro/policy/frameworks/software/vulnerable.bro >> >> /usr/local/bro/share/bro/policy/frameworks/software/version-changes.bro >> >> /usr/local/bro/share/bro/policy/protocols/ftp/software.bro >> >> /usr/local/bro/share/bro/policy/protocols/smtp/software.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssh/software.bro >> >> /usr/local/bro/share/bro/policy/protocols/http/software.bro >> >> /usr/local/bro/share/bro/policy/protocols/dns/detect-external-names.bro >> >> /usr/local/bro/share/bro/policy/protocols/ftp/detect.bro >> >> /usr/local/bro/share/bro/policy/protocols/conn/known-hosts.bro >> >> /usr/local/bro/share/bro/policy/protocols/conn/known-services.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssl/known-certs.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssl/validate-certs.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssl/log-hostcerts-only.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssh/geo-data.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssh/detect-bruteforcing.bro >> >> /usr/local/bro/share/bro/policy/protocols/ssh/interesting-hostnames.bro >> >> /usr/local/bro/share/bro/policy/protocols/http/detect-sqli.bro >> >> /usr/local/bro/share/bro/policy/frameworks/files/hash-all-files.bro >> >> /usr/local/bro/share/bro/policy/frameworks/files/detect-MHR.bro >> >> /usr/local/bro/share/bro/broctl/__load__.bro >> >> /usr/local/bro/share/bro/broctl/main.bro >> >> /usr/local/bro/share/bro/policy/frameworks/control/controllee.bro >> >> /usr/local/bro/spool/installed-scripts-do-not-touch/site/local-manager.bro >> >> /usr/local/bro/share/bro/broctl/auto.bro >> >> >> /usr/local/bro/spool/installed-scripts-do-not-touch/auto/local-networks.bro >> >> >> /usr/local/bro/spool/installed-scripts-do-not-touch/auto/broctl-config.bro >> >> [proxy-1] >> >> >> Bro 2.4.1 >> >> Linux 3.13.0-48-generic >> >> >> No gdb installed. >> >> >> ==== No reporter.log >> >> >> ==== stderr.log >> >> >> ==== stdout.log >> >> max memory size (kbytes, -m) unlimited >> >> data seg size (kbytes, -d) unlimited >> >> virtual memory (kbytes, -v) unlimited >> >> core file size (blocks, -c) unlimited >> >> >> ==== .cmdline >> >> -U .status -p broctl -p broctl-live -p local -p proxy-1 local.bro broctl >> base/frameworks/cluster local-proxy broctl/auto >> >> >> ==== .env_vars >> >> PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/local/bro/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games >> >> BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site >> >> CLUSTER_NODE=proxy-1 >> >> >> ==== .status >> >> RUNNING [net_run] >> >> >> ==== No prof.log >> >> >> ==== No packet_filter.log >> >> >> ==== No loaded_scripts.log >> >> [worker-1] >> >> error running crash-diag for worker-1 >> >> Host 172.31.41.33 is not alive >> >> [worker-2] >> >> error running crash-diag for worker-2 >> >> Host 172.31.41.31 is not alive >> >> >> >> >> >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> From cdaviso1 at vols.utk.edu Wed Sep 16 12:40:34 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Wed, 16 Sep 2015 19:40:34 +0000 Subject: [Bro] PF_Ring Cluster Install Message-ID: I have seen different documentation around building PF_Ring to integrate with bro. I have performed the below already. However, I installed bro from wget instead of from source so does that mean I have to start over? If so does anyone have detailed documentation for configuring bro from source along with all components? I have worked with PF_Ring before and know it can be challenging to get going. I also have many questions regarding how to properly configure the node.cfg and making sure my boxes are configured in the right manner, depending on the processor architecture being used. Would processor architecture affect setting the lb_method, lb_procs, and pin_cpus in AWS? Honestly, if someone well versed in PF_Ring could speak to all those points and some of the questions below we could set up a web-ex. Once I get it running properly I can give the documentation I have made to the bro team so they can update the site regarding all these questions so others who are new to this and might have the same questions can just read the further documentation on the site. sudo apt-get update sudo apt-get upgrade sudo apt-get install linux-headers-$(uname -r) sudo apt-get install libnuma-dev git clone https://github.com/ntop/PF_RING.git cd PF_RING/kernel ./configure make make install sudo insmod ./pf_ring.ko cd .. cd userland/ cd ../userland make >From here when I install based on the instructions from the bro website I perform the following and do not see anything in /usr/src to extract: cd /usr/src tar xvzf PF_RING-5.6.2.tar.gz cd PF_RING-5.6.2/userland/lib ./configure --prefix=/opt/pfring make install I also receive the following when attempting to perform any further configurations: ubuntu at ip-172-31-38-121:~$ cd ../libpcap -bash: cd: ../libpcap: No such file or directory ubuntu at ip-172-31-38-121:~$ cd ../tcpdump-4.1.1 -bash: cd: ../tcpdump-4.1.1: No such file or directory ubuntu at ip-172-31-38-121:~$ cd ../../kernel -bash: cd: ../../kernel: No such file or directory ubuntu at ip-172-31-38-121:~$ I also need information on how to load the pf_ring module at boot time for Ubuntu 14.04. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150916/e9063644/attachment.html From hckim at narusec.com Wed Sep 16 23:39:58 2015 From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=) Date: Thu, 17 Sep 2015 15:39:58 +0900 Subject: [Bro] FTP password Message-ID: Hi I have setup bro 2.3 did not change any setting but some of ftp.log has password printed out 1442471625.330839 CJtp9r1Ww7Nrjco5H4 x.x.x.x 511 y.y.y.y 561 ftpuser safepc RETR ftp://w.w.w.w/WINDOWS7/64/Setup.dat - 226 Transfer complete. - - - - - I checked ftp config but "default_capture_password = F" do I have to do something to not capture password ? Thank you -- ------------------------------------------------------ Hichul Kim ??? ?? ??? Naru Security (?)?????? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150917/3dcda6f4/attachment.html From liburdi.joshua at gmail.com Thu Sep 17 05:02:57 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Thu, 17 Sep 2015 08:02:57 -0400 Subject: [Bro] FTP password In-Reply-To: References: Message-ID: That user value is one of the default values that Bro will always log the password for. const guest_ids = { "anonymous", "ftp", "ftpuser", "guest" } &redef; If you redef guest_ids to be empty, then it shouldn't log any passwords. Josh On Thu, Sep 17, 2015 at 2:39 AM, ??? wrote: > Hi > I have setup bro 2.3 > did not change any setting but some of ftp.log has password printed out > > 1442471625.330839 CJtp9r1Ww7Nrjco5H4 x.x.x.x 511 y.y.y.y 561 ftpuser safepc > RETR ftp://w.w.w.w/WINDOWS7/64/Setup.dat - 226 Transfer complete. - - - - - > > I checked ftp config but "default_capture_password = F" > > do I have to do something to not capture password ? > > Thank you > -- > ------------------------------------------------------ > Hichul Kim ??? ?? ??? > > Naru Security (?)?????? > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jazoff at illinois.edu Thu Sep 17 06:06:09 2015 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 17 Sep 2015 13:06:09 +0000 Subject: [Bro] Compiling bro 2.4.1 on Ubuntu && ARM (HELP Please) In-Reply-To: References: Message-ID: > On Sep 15, 2015, at 6:57 AM, Ludwig Goon wrote: > > Trying to compile bro 2.4.1 on linux. After setting up the packages for compile I run the ./configure scriipt and get the following error: > > > -- Looking for include file pthread.h > -- Looking for include file pthread.h - found > -- Looking for pthread_create > -- Looking for pthread_create - not found > -- Looking for pthread_create in pthreads > -- Looking for pthread_create in pthreads - not found > -- Looking for pthread_create in pthread > -- Looking for pthread_create in pthread - found > -- Found Threads: TRUE This is not your problem.. > CMake Error at doc/CMakeLists.txt:14 (message): > Problem setting BROPATH ..this is your problem. That error comes from doc/CMakeLists.txt where it tries to run ./build/bro-path-dev What happens when you run cmake --version ./build/bro-path-dev It is just a one line shell script that echos a path, so it should not be failing. -- - Justin Azoff From lagoon7 at gmail.com Thu Sep 17 06:31:50 2015 From: lagoon7 at gmail.com (Ludwig Goon) Date: Thu, 17 Sep 2015 09:31:50 -0400 Subject: [Bro] Compiling bro 2.4.1 on Ubuntu && ARM (HELP Please) In-Reply-To: References: Message-ID: Thanks, Actually Johanna provide a solution to the issue. What you suggested may work as well. However there is something going on when generating a make file for docs. I have also closed out the case with the solution she provided. Thanks again for your response... On Thu, Sep 17, 2015 at 9:06 AM, Azoff, Justin S wrote: > > > On Sep 15, 2015, at 6:57 AM, Ludwig Goon wrote: > > > > Trying to compile bro 2.4.1 on linux. After setting up the packages for > compile I run the ./configure scriipt and get the following error: > > > > > > -- Looking for include file pthread.h > > -- Looking for include file pthread.h - found > > -- Looking for pthread_create > > -- Looking for pthread_create - not found > > -- Looking for pthread_create in pthreads > > -- Looking for pthread_create in pthreads - not found > > -- Looking for pthread_create in pthread > > -- Looking for pthread_create in pthread - found > > -- Found Threads: TRUE > > This is not your problem.. > > > CMake Error at doc/CMakeLists.txt:14 (message): > > Problem setting BROPATH > > ..this is your problem. > > > That error comes from doc/CMakeLists.txt where it tries to run > ./build/bro-path-dev > > What happens when you run > > cmake --version > ./build/bro-path-dev > > It is just a one line shell script that echos a path, so it should not be > failing. > > > > -- > - Justin Azoff > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150917/52b6d73d/attachment-0001.html From lagoon7 at gmail.com Fri Sep 18 01:28:53 2015 From: lagoon7 at gmail.com (Ludwig Goon) Date: Fri, 18 Sep 2015 04:28:53 -0400 Subject: [Bro] BPF Filter per log file or framework such as x509, SSL Message-ID: when activating the x509.log or bro script in local.bro, can I configure a BPF filter to only affect x509 framework? For example I only want to have events that the dst_host is our DMZ subnet. Can I configure that in the x509.bro file/framework or some other bro configuration file? If so is this a local variable called subnet or something? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150918/401f1a2c/attachment.html From cdaviso1 at vols.utk.edu Fri Sep 18 06:16:05 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Fri, 18 Sep 2015 13:16:05 +0000 Subject: [Bro] PF_Ring Cluster Install In-Reply-To: References: Message-ID: I have seen different documentation around building PF_Ring to integrate with bro. I have performed the below already. However, I installed bro from wget instead of from source so does that mean I have to start over? If so does anyone have detailed documentation for configuring bro from source along with all components? I have worked with PF_Ring before and know it can be challenging to get going. I also have many questions regarding how to properly configure the node.cfg and making sure my boxes are configured in the right manner, depending on the processor architecture being used. Would processor architecture affect setting the lb_method, lb_procs, and pin_cpus in AWS? Honestly, if someone well versed in PF_Ring could speak to all those points and some of the questions below we could set up a web-ex. Once I get it running properly I can give the documentation I have made to the bro team so they can update the site regarding all these questions so others who are new to this and might have the same questions can just read the further documentation on the site. sudo apt-get update sudo apt-get upgrade sudo apt-get install linux-headers-$(uname -r) sudo apt-get install libnuma-dev git clone https://github.com/ntop/PF_RING.git cd PF_RING/kernel ./configure make make install sudo insmod ./pf_ring.ko cd .. cd userland/ cd ../userland make >From here when I install based on the instructions from the bro website I perform the following and do not see anything in /usr/src to extract: cd /usr/src tar xvzf PF_RING-5.6.2.tar.gz cd PF_RING-5.6.2/userland/lib ./configure --prefix=/opt/pfring make install I also receive the following when attempting to perform any further configurations: ubuntu at ip-172-31-38-121:~$ cd ../libpcap -bash: cd: ../libpcap: No such file or directory ubuntu at ip-172-31-38-121:~$ cd ../tcpdump-4.1.1 -bash: cd: ../tcpdump-4.1.1: No such file or directory ubuntu at ip-172-31-38-121:~$ cd ../../kernel -bash: cd: ../../kernel: No such file or directory ubuntu at ip-172-31-38-121:~$ I also need information on how to load the pf_ring module at boot time for Ubuntu 14.04. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150918/2cf76316/attachment.html From johanna at icir.org Fri Sep 18 10:15:57 2015 From: johanna at icir.org (Johanna Amann) Date: Fri, 18 Sep 2015 10:15:57 -0700 Subject: [Bro] BPF Filter per log file or framework such as x509, SSL In-Reply-To: References: Message-ID: <20150918171548.GA74978@wifi79.sys.ICSI.Berkeley.EDU> Just to repeat my answer from the bug tracker: you can add bpf filters with the syntax described in https://www.bro.org/sphinx/scripts/base/frameworks/packet-filter/main.bro.html The thread at http://comments.gmane.org/gmane.comp.security.detection.bro/4759 also has a few examples. There is no easy way to tell Bro to just allow traffic containing x509 certificates - you have to build the filter yourself, only allowing the hosts and services that have traffic containing x509 certificates. If using broctl, typically you would add the filter commands to local.bro or to a script that you load from local.bro ? it is discouraged to edit any scripts in base/ or policy/ yourself. Do you need anything else, or does that perhaps fulfill your requirements? Johanna On Fri, Sep 18, 2015 at 04:28:53AM -0400, Ludwig Goon wrote: > when activating the x509.log or bro script in local.bro, can I configure a > BPF filter to only affect x509 framework? For example I only want to have > events that the dst_host is our DMZ subnet. Can I configure that in the > x509.bro file/framework or some other bro configuration file? If so is this > a local variable called subnet or something? > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From johanna at icir.org Fri Sep 18 10:25:16 2015 From: johanna at icir.org (Johanna Amann) Date: Fri, 18 Sep 2015 10:25:16 -0700 Subject: [Bro] Any plans to use p0f V3 signature? In-Reply-To: References: Message-ID: <20150918172516.GB74978@wifi79.sys.ICSI.Berkeley.EDU> On Wed, Sep 16, 2015 at 02:51:00PM +0900, ??? wrote: > Hi > > I have been using p0f -v1.8.3 fingerprints > but > having some issue > bro is printing out couple of OS from same IP > These signatures are quite out of date by now - so I guess it is not really to be too unexpected that they do not really give you good results anymore. That being said - just to ask the obvious question - there is no chance someone is using virtual machines or a NAT gateway there? > So I test p0f - v3, so far I did not have this issue.( just p0f -i eth1 -a > os.log) > > am I having this issue because of my bro script ? Probably no... > if not do you have any plans to use p0f - v3 (or fingerprints > )? p0f v3 is quite different from the earlier versions and uses information from e.g. HTTP headers for its operating system determination. One could probably try to re-implement something similar using Bro scripts -- there already are scripts that track information about hosts (like software.log) that could be used towards this end. As far as I am aware, no one currently has plans to add p0f v3 support to Bro. Johanna From aidaros.dev at gmail.com Sun Sep 20 04:54:39 2015 From: aidaros.dev at gmail.com (Hashem Alaidaros) Date: Sun, 20 Sep 2015 14:54:39 +0300 Subject: [Bro] I want to capture certain traffic using input framework Message-ID: Hi All I used input framework blacklist approach ( https://www.bro.org/sphinx/frameworks/input.html) that let Bro script read (IP's) from a file (log file) that is dynamically written from other bro instance. I managed to read blacklist IPs from blacklist file. My goal is to let bro to only capture and process live packets that match those blacklist IPs But there is an issue that the event captures all incoming packets. The following event capture and process all packets before it read and match with the file. For example, once the following bro run, all incoming traffic is processed in this event, regardless blacklist match: event signature_match(state: signature_state, msg: string, data: string) { if(state$conn$id$orig_h in blacklist) { do analysis } } 1. Is there any way to filter the incoming traffic in bro based on input framework blacklist? FYI: I can use BPF (bro -f file.log), but in this case the issue is that bro has to be restart many times since the file keep adding new IPs so that the file.log is to be updated. I also find exclude filter function but that exclude, I want to include certain traffic to captured. 2. Can an event be provoked when only it pass a condition. for example, in my case, can I say: if (state$conn$id$orig_h in blacklist) { event signature_match(state: signature_state, msg: string, data: string) { print fmt("IRC bot Match!!! in %s",state$conn$id$orig_h); } elso { "do nothing" } If not, is there any way to make an event run when only pass if statement? Bro version 2.3 Thanks in advance -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150920/7b8273c5/attachment.html From jan.grashofer at cern.ch Sun Sep 20 06:25:00 2015 From: jan.grashofer at cern.ch (Jan Grashoefer) Date: Sun, 20 Sep 2015 15:25:00 +0200 Subject: [Bro] I want to capture certain traffic using input framework In-Reply-To: References: Message-ID: <55FEB3AC.1040703@cern.ch> Hi, > FYI: I can use BPF (bro -f file.log), but in this case the issue is that > bro has to be restart many times since the file keep adding new IPs so that > the file.log is to be updated. I also find exclude filter function but that > exclude, I want to include certain traffic to captured. you can use the packet filter framework (see https://www.bro.org/sphinx/scripts/base/frameworks/packet-filter/main.bro.html) to install your filter live. Regards, Jan From aidaros.dev at gmail.com Sun Sep 20 15:39:25 2015 From: aidaros.dev at gmail.com (Hashem Alaidaros) Date: Mon, 21 Sep 2015 01:39:25 +0300 Subject: [Bro] Bro Digest, Vol 113, Issue 31 In-Reply-To: References: Message-ID: Thanks Jan for your reply. Actually I was trying with packet filter framework before, but I found it to let "exclude" traffic based on IP's, but in my case is opposite, I want to "include" only and let traffic on my Blacklist IP's through to Bro. On the other way, I want to tell Bro, if the incoming IP address is matching with the blacklist file, then capture that file and analyze it, otherwise ignore (or drop) it. Correct me if I'm wrong. I hope I can find the answer in this mailing list. On Sun, Sep 20, 2015 at 10:00 PM, wrote: > Send Bro mailing list submissions to > bro at bro.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > or, via email, send a message with subject or body 'help' to > bro-request at bro.org > > You can reach the person managing the list at > bro-owner at bro.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Bro digest..." > > > Today's Topics: > > 1. I want to capture certain traffic using input framework > (Hashem Alaidaros) > 2. Re: I want to capture certain traffic using input framework > (Jan Grashoefer) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 20 Sep 2015 14:54:39 +0300 > From: Hashem Alaidaros > Subject: [Bro] I want to capture certain traffic using input framework > To: bro at bro.org > Message-ID: > DW2wE5WuLr+1zvTXjHw at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi All > > I used input framework blacklist approach ( > https://www.bro.org/sphinx/frameworks/input.html) that let Bro script read > (IP's) from a file (log file) that is dynamically written from other bro > instance. > I managed to read blacklist IPs from blacklist file. > > My goal is to let bro to only capture and process live packets that match > those blacklist IPs But there is an issue that the event captures all > incoming packets. > The following event capture and process all packets before it read and > match with the file. For example, once the following bro run, all incoming > traffic is processed in this event, regardless blacklist match: > > event signature_match(state: signature_state, msg: string, data: string) > { > > if(state$conn$id$orig_h in blacklist) { do analysis } > } > > 1. Is there any way to filter the incoming traffic in bro based on input > framework blacklist? > > FYI: I can use BPF (bro -f file.log), but in this case the issue is that > bro has to be restart many times since the file keep adding new IPs so that > the file.log is to be updated. I also find exclude filter function but that > exclude, I want to include certain traffic to captured. > > 2. Can an event be provoked when only it pass a condition. for example, in > my case, can I say: > > if (state$conn$id$orig_h in blacklist) { > event signature_match(state: signature_state, msg: string, data: string) > { > print fmt("IRC bot Match!!! in %s",state$conn$id$orig_h); > } > > elso { "do nothing" } > > If not, is there any way to make an event run when only pass if statement? > > Bro version 2.3 > > Thanks in advance > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150920/7b8273c5/attachment-0001.html > > ------------------------------ > > Message: 2 > Date: Sun, 20 Sep 2015 15:25:00 +0200 > From: Jan Grashoefer > Subject: Re: [Bro] I want to capture certain traffic using input > framework > To: > Message-ID: <55FEB3AC.1040703 at cern.ch> > Content-Type: text/plain; charset="windows-1252" > > Hi, > > > FYI: I can use BPF (bro -f file.log), but in this case the issue is that > > bro has to be restart many times since the file keep adding new IPs so > that > > the file.log is to be updated. I also find exclude filter function but > that > > exclude, I want to include certain traffic to captured. > > you can use the packet filter framework (see > > https://www.bro.org/sphinx/scripts/base/frameworks/packet-filter/main.bro.html > ) > to install your filter live. > > Regards, > Jan > > > ------------------------------ > > _______________________________________________ > Bro mailing list > Bro at bro.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > End of Bro Digest, Vol 113, Issue 31 > ************************************ > -- A friend in need Is a friend indeed -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150921/da5b93df/attachment.html From seth at icir.org Sun Sep 20 19:04:41 2015 From: seth at icir.org (Seth Hall) Date: Sun, 20 Sep 2015 22:04:41 -0400 Subject: [Bro] Bro Digest, Vol 113, Issue 31 In-Reply-To: References: Message-ID: <4CB6BABA-714F-4187-8A6E-2ABD80E48E60@icir.org> > On Sep 20, 2015, at 6:39 PM, Hashem Alaidaros wrote: > > Thanks Jan for your reply. > Actually I was trying with packet filter framework before, but I found it to let "exclude" traffic based on IP's, but in my case is opposite, I want to "include" only and let traffic on my Blacklist IP's through to Bro. On the other way, I want to tell Bro, if the incoming IP address is matching with the blacklist file, then capture that file and analyze it, otherwise ignore (or drop) it. > Correct me if I'm wrong. redef capture_filters += { ["one-host"] = "host 1.2.3.4", ["two-hosts"] = "host 5.6.7.8", }; This will automatically give you a packet filter of: ?(host 1.2.3.4) or (host 5.6.7.8)? To explain this a bit more, Bro will automatically use ?ip or not ip? which is a fully open capture filter if you don?t provide a capture_filter which puts you in a position of filtering down from having everything open. If you provide your own capture filter(s), it will use those instead so you can build up the traffic you?re choosing to monitor. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From franky.meier.1 at gmx.de Mon Sep 21 07:09:06 2015 From: franky.meier.1 at gmx.de (Frank Meier) Date: Mon, 21 Sep 2015 16:09:06 +0200 Subject: [Bro] deterministic uids Message-ID: <1442844546.27935.0@mail.gmx.net> Hi! Is there any reason why uids in bro are partly random and not just a function of the meta information of the flow? When I restart Bro with the same pcap, I have to make sure to set the seed file to get the same uids. I would just compute a hash over time, source-host, source-port, destination host, destination port and protocol: event new_connection(c: connection) { c$uid = md5_hash(c$start_time, c$id$orig_h, c$id$orig_p, c$id$resp_h, c$id$resp_p); } A disadvantage would be, that the length of the hash is not configurable anymore. Any ideas why this is a bad idea? Thanks, Franky -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150921/f7e7cd93/attachment.html From seth at icir.org Mon Sep 21 08:57:29 2015 From: seth at icir.org (Seth Hall) Date: Mon, 21 Sep 2015 11:57:29 -0400 Subject: [Bro] deterministic uids In-Reply-To: <1442844546.27935.0@mail.gmx.net> References: <1442844546.27935.0@mail.gmx.net> Message-ID: <7CAB2E66-9118-48B4-B467-E8AB39DD7909@icir.org> > On Sep 21, 2015, at 10:09 AM, Frank Meier wrote: > > Is there any reason why uids in bro are partly random and not just a function > of the meta information of the flow? When I restart Bro with the same pcap, > I have to make sure to set the seed file to get the same uids. If there was no randomness in the uid creation, uids could be influenced by potential adversaries which could dramatically impact your analysis. As it is now, attackers shouldn?t be able to influence uids. If you need determinism in them you can seed the random generator with either the BRO_SEED_FILE environment variable or with the command line option... -J|--set-seed | set the random number seed .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From patrick.peace at arlut.utexas.edu Mon Sep 21 10:49:21 2015 From: patrick.peace at arlut.utexas.edu (Patrick Peace) Date: Mon, 21 Sep 2015 17:49:21 +0000 Subject: [Bro] Help With Version Information Message-ID: Good Afternoon, I have been handed a Bro IDS sensor that someone else has setup and I am not that familiar with it yet. I am needing to verify which version of sensor is currently running, but I am not having much luck on finding how to check that on the internet. Is there any handy commands or files I can check for the version information? Patrick Peace IT Security Applied Research Labs at the University of Texas 512-835-3673 patrick.peace at arlut.utexas.edu From dnthayer at illinois.edu Mon Sep 21 10:53:09 2015 From: dnthayer at illinois.edu (Daniel Thayer) Date: Mon, 21 Sep 2015 12:53:09 -0500 Subject: [Bro] Help With Version Information In-Reply-To: References: Message-ID: <56004405.1060605@illinois.edu> Have you tried doing this: bro -v On 09/21/2015 12:49 PM, Patrick Peace wrote: > Good Afternoon, > > I have been handed a Bro IDS sensor that someone else has setup and I am not that familiar with it yet. > > I am needing to verify which version of sensor is currently running, but I am not having much luck on finding how to check that on the internet. > > Is there any handy commands or files I can check for the version information? > > Patrick Peace > IT Security > Applied Research Labs at the University of Texas > 512-835-3673 patrick.peace at arlut.utexas.edu From sven at dreyer-net.de Tue Sep 22 05:06:03 2015 From: sven at dreyer-net.de (Sven Dreyer) Date: Tue, 22 Sep 2015 14:06:03 +0200 Subject: [Bro] TCP retransmissions In-Reply-To: References: <55E8C367.5090007@dreyer-net.de> Message-ID: <5601442B.6080004@dreyer-net.de> Hi Anthony, Am 04.09.2015 um 00:08 schrieb anthony kasza: > They might be considered new connections if your router and laptop have > a longer connection timeout than Bro. This is a guess. Thanks for the hint, I think all of them have 300 seconds TCP timeout, at least I didn't change the default configuration. Best regards, Sven From sven at dreyer-net.de Tue Sep 22 05:08:53 2015 From: sven at dreyer-net.de (Sven Dreyer) Date: Tue, 22 Sep 2015 14:08:53 +0200 Subject: [Bro] long SSH connection in conn.log In-Reply-To: <20150915161854.GA47853@Beezling.local> References: <55E8C4F8.3080009@dreyer-net.de> <20150915161854.GA47853@Beezling.local> Message-ID: <560144D5.8060705@dreyer-net.de> Hi Johanna, Am 15.09.2015 um 18:18 schrieb Johanna Amann: > This is a generic problem - you have to just assume that connections are > terminated after you did not see any exchanged data for a specified period > of time. In case the current Bro settings do not work for you, you can > redef them. Thank you very much for the detailed explanation, I will try to change the default bro settings and check for the result. Thanks! Sven From ajackso at us.ibm.com Tue Sep 22 08:50:52 2015 From: ajackso at us.ibm.com (Amina Jackson) Date: Tue, 22 Sep 2015 11:50:52 -0400 Subject: [Bro] Bro installation Message-ID: I have CentOS 6.7. I have attempted to install bro in all possible ways with no success. I have installed the dependencies and development tools but I keep getting errors. The git installation yields an error When I try installing the binaries, I get Any help will be very appreciated. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150922/167f04e6/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 9805 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150922/167f04e6/attachment-0002.gif -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 17051 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150922/167f04e6/attachment-0003.gif From johanna at icir.org Tue Sep 22 09:09:40 2015 From: johanna at icir.org (Johanna Amann) Date: Tue, 22 Sep 2015 09:09:40 -0700 Subject: [Bro] Bro installation In-Reply-To: References: Message-ID: <20150922160940.GA59603@Beezling.local> Hello Amina, the current bro master installation from git.bro.org, that I think you are trying to use, requires an installation of libcaf (http://www.actor-framework.org). You either have to install it and provide the location to configure using --with-libcaf=PATH, or tell configure to disable the parts of Bro that need the library using --disable-broker. If you are trying to compile Bro 2.4.1, this should not be necessary. As to the binary packages, I just tried an installation on a fresh CentOs installation that worked without any problems. If you want help in trouble-shooting that, please provide all steps that you followed until you arrived at the error-message. I did: [root at test ~]# cd /etc/yum.repos.d/ [root at test yum.repos.d]# wget http://download.opensuse.org/repositories/network:bro/CentOS_6/network:bro.repo [wget output] [root at test yum.repos.d]# yum install bro [yum output, succesful installation] [root at test yum.repos.d]# /opt/bro/bin/bro --version /opt/bro/bin/bro version 2.4.1 [root at test yum.repos.d]# cat /etc/centos-release CentOS release 6.7 (Final) I hope this helps, Johanna On Tue, Sep 22, 2015 at 11:50:52AM -0400, Amina Jackson wrote: > I have CentOS 6.7. I have attempted to install bro in all possible ways > with no success. I have installed the dependencies and development tools > but I keep getting errors. > > The git installation yields an error > > > > When I try installing the binaries, I get > > > Any help will be very appreciated. > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From cdaviso1 at vols.utk.edu Tue Sep 22 13:10:34 2015 From: cdaviso1 at vols.utk.edu (Davison, Charles Robert) Date: Tue, 22 Sep 2015 20:10:34 +0000 Subject: [Bro] Bro PF RING Message-ID: I am following the instructions on bro.org for the PF_Ring install and have completed the below steps so far. I have a question about the next few steps: How do i complete this? ...Refer to the documentation for your Linux distribution on how to load the pf_ring module at boot time. Does this basically mean i need to use the steps below on all worker nodes? ...You will need to install the PF_RING library files and kernel module on all of the workers in your cluster. I already downloaded bro and installed /configured it.... is there a way to reconfigure bro without performing the below steps. 1. Download the Bro source code. 2. Configure and install Bro using the following commands: Steps Completed Thus Far on Ubuntu 14.04 LTS cd /usr/src sudo wget http://sourceforge.net/projects/ntop/files/PF_RING/PF_RING-6.0.3.tar.gz sudo tar zxvf PF_RING-6.0.3.tar.gz cd PF_RING-6.0.3/userland/lib ./configure --prefix=/opt/pfring make sudo make install cd ../libpcap ./configure --prefix=/opt/pfring make sudo make install cd ../tcpdump-4.1.1 ./configure --prefix=/opt/pfring make sudo make install cd ../../kernel make sudo make install sudo modprobe pf_ring enable_tx_capture=0 min_num_slots=32768 CHARLES R. DAVISON (865)730-0078 cdaviso1 at vols.utk.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150922/b6d34b2d/attachment.html From jazoff at illinois.edu Tue Sep 22 13:23:03 2015 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 22 Sep 2015 20:23:03 +0000 Subject: [Bro] Bro PF RING In-Reply-To: References: Message-ID: > On Sep 22, 2015, at 4:10 PM, Davison, Charles Robert wrote: > > I am following the instructions on bro.org for the PF_Ring install and have completed the below steps so far. I have a question about the next few steps: Looking good so far :-) > How do i complete this? > ...Refer to the documentation for your Linux distribution on how to load the pf_ring module at boot time. For ubuntu this should work, place modprobe pf_ring enable_tx_capture=0 in /etc/modules-load.d/pfring.conf > Does this basically mean i need to use the steps below on all worker nodes? > ...You will need to install the PF_RING library files and kernel module on all of the workers in your cluster. Yes. If your manager does not have a capture interface you can skip the kernel steps on that machine, but you need to install all of the components on the workers. > I already downloaded bro and installed /configured it.... is there a way to reconfigure bro without performing the below steps. > ? Download the Bro source code. You will need to configure bro using ./configure --with-pcap=/opt/pfring in order for it to link against pf_ring. > ? Configure and install Bro using the following commands: > Steps Completed Thus Far on Ubuntu 14.04 LTS > cd /usr/src > sudo wget http://sourceforge.net/projects/ntop/files/PF_RING/PF_RING-6.0.3.tar.gz > sudo tar zxvf PF_RING-6.0.3.tar.gz > cd PF_RING-6.0.3/userland/lib > ./configure --prefix=/opt/pfring > make > sudo make install > > cd ../libpcap > ./configure --prefix=/opt/pfring > make > sudo make install > > cd ../tcpdump-4.1.1 > ./configure --prefix=/opt/pfring > make > sudo make install > > cd ../../kernel > make > sudo make install > > sudo modprobe pf_ring enable_tx_capture=0 min_num_slots=32768 > > -- - Justin Azoff From dirk at dirkleinenbach.de Wed Sep 23 09:56:06 2015 From: dirk at dirkleinenbach.de (Dirk Leinenbach) Date: Wed, 23 Sep 2015 18:56:06 +0200 Subject: [Bro] Is it possible to export pcap for a given event / connection? Message-ID: Hi there, does bro provide some mechanism to find the packets that are related to (have caused) a given event or connection? Background: I'd like to be able to export pcap files in some situations for specific events; in that context I'm still able to get to the connection object, but I'd like to be able to see the original data as well for further analysis with Wireshark. One possibility would be to reconstruct filters from the event to filter the original trace retrospectively. But I'm wondering if there is a more direct way to identify / extract the relevant packets. Thanks for your help, Dirk From hosom at battelle.org Wed Sep 23 10:27:37 2015 From: hosom at battelle.org (Hosom, Stephen M) Date: Wed, 23 Sep 2015 17:27:37 +0000 Subject: [Bro] Is it possible to export pcap for a given event / connection? In-Reply-To: References: Message-ID: Dirk, Bro doesn't really have a good way to export packet captures. You would be best off running something like time machine or stenographer (both open source packet capture projects) and then using Bro to export the small pcap related to the connection you want. If you'd like some pointers on how to do that, let me know. I've got some similar stuff going on in my environment. Thanks, Stephen -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Dirk Leinenbach Sent: Wednesday, September 23, 2015 12:56 PM To: bro at bro.org Subject: [Bro] Is it possible to export pcap for a given event / connection? Hi there, does bro provide some mechanism to find the packets that are related to (have caused) a given event or connection? Background: I'd like to be able to export pcap files in some situations for specific events; in that context I'm still able to get to the connection object, but I'd like to be able to see the original data as well for further analysis with Wireshark. One possibility would be to reconstruct filters from the event to filter the original trace retrospectively. But I'm wondering if there is a more direct way to identify / extract the relevant packets. Thanks for your help, Dirk _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jlay at slave-tothe-box.net Wed Sep 23 14:47:06 2015 From: jlay at slave-tothe-box.net (James Lay) Date: Wed, 23 Sep 2015 15:47:06 -0600 Subject: [Bro] Is it possible to export pcap for a given event / connection? In-Reply-To: References: Message-ID: <725be3ac4fdd8326321a1bb73c2f3cfc@localhost> On 2015-09-23 11:27 AM, Hosom, Stephen M wrote: > Dirk, > > Bro doesn't really have a good way to export packet captures. > > You would be best off running something like time machine or > stenographer (both open source packet capture projects) and then using > Bro to export the small pcap related to the connection you want. If > you'd like some pointers on how to do that, let me know. I've got some > similar stuff going on in my environment. > > Thanks, > > Stephen > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of > Dirk Leinenbach > Sent: Wednesday, September 23, 2015 12:56 PM > To: bro at bro.org > Subject: [Bro] Is it possible to export pcap for a given event / > connection? > > Hi there, > > does bro provide some mechanism to find the packets that are related > to (have caused) a given event or connection? > > Background: I'd like to be able to export pcap files in some > situations for specific events; in that context I'm still able to get > to the connection object, but I'd like to be able to see the original > data as well for further analysis with Wireshark. > > One possibility would be to reconstruct filters from the event to > filter the original trace retrospectively. But I'm wondering if there > is a more direct way to identify / extract the relevant packets. > > Thanks for your help, > > Dirk We use dumpcap from wireshark from source for packet capture...example below: /usr/local/bin/dumpcap -q -b filesize:409600 -b files:50 -Z none -f 'ip and port 25' -i eth2 -w /home/pcaps/mailcapture/mailtraffic.pcap which creates 50 400 meg files and will start to overwrite after 50...works well when run on the same box as bro-ids...very easy to correlate and pluck out what I want. James From daniel.guerra69 at gmail.com Wed Sep 23 15:35:19 2015 From: daniel.guerra69 at gmail.com (Daniel Guerra) Date: Thu, 24 Sep 2015 00:35:19 +0200 Subject: [Bro] Is it possible to export pcap for a given event / connection? In-Reply-To: References: Message-ID: <2AE20299-1049-4C07-AF70-7B1480EDBB9C@gmail.com> There is way to extract the application layer. Check /usr/local/bro/share/base/protocols/conn/contents.bro > On 23 Sep 2015, at 18:56, Dirk Leinenbach wrote: > > Hi there, > > does bro provide some mechanism to find the packets that are related to (have caused) a given event or connection? > > Background: I'd like to be able to export pcap files in some situations for specific events; in that context I'm still able to get to the connection object, but I'd like to be able to see the original data as well for further analysis with Wireshark. > > One possibility would be to reconstruct filters from the event to filter the original trace retrospectively. But I'm wondering if there is a more direct way to identify / extract the relevant packets. > > Thanks for your help, > > Dirk > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150924/aaa5a0bc/attachment.html From renaud.luca at gmail.com Wed Sep 23 19:23:05 2015 From: renaud.luca at gmail.com (Luca Renaud) Date: Thu, 24 Sep 2015 03:23:05 +0100 Subject: [Bro] How to configure-control snaplen bits captured on live capture. Message-ID: Capturing with tcpdump (for offline analysis) I generally use the following command: tcpdump -s 96 .......... ( -s 0 is not necessary for me most of the time) So,using BroControl to start/stop a realtime capture and analysis how can the captured bits be configured-controled to match our needs? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150924/0a1e38ca/attachment.html From doug.burks at gmail.com Thu Sep 24 03:29:15 2015 From: doug.burks at gmail.com (Doug Burks) Date: Thu, 24 Sep 2015 06:29:15 -0400 Subject: [Bro] Is it possible to export pcap for a given event / connection? In-Reply-To: References: Message-ID: Hi Dirk, Here's what we do in Security Onion [1]: - Bro logs go into ELSA [2] - for most Bro logs, you can use ELSA's getPcap plugin to pivot to CapMe [3] - CapMe will then search the full packet capture store provided by netsniff-ng [4] and provide you with an ASCII rendering of the stream or the raw pcap itself For more information and a screenshot of this in action, please see [5]. Hope that helps! [1] - http://securityonion.net [2] - https://github.com/mcholste/elsa [3] - https://github.com/int13h/capme [4] - http://netsniff-ng.org/ [5] - http://taosecurity.blogspot.com/2013/01/security-onion-elsa-or-snorby-capme.html On Wed, Sep 23, 2015 at 12:56 PM, Dirk Leinenbach wrote: > Hi there, > > does bro provide some mechanism to find the packets that are related to (have caused) a given event or connection? > > Background: I'd like to be able to export pcap files in some situations for specific events; in that context I'm still able to get to the connection object, but I'd like to be able to see the original data as well for further analysis with Wireshark. > > One possibility would be to reconstruct filters from the event to filter the original trace retrospectively. But I'm wondering if there is a more direct way to identify / extract the relevant packets. > > Thanks for your help, > > Dirk > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Doug Burks Need Security Onion Training or Commercial Support? http://securityonionsolutions.com From aidaros.dev at gmail.com Sun Sep 27 20:14:07 2015 From: aidaros.dev at gmail.com (Hashem Alaidaros) Date: Mon, 28 Sep 2015 06:14:07 +0300 Subject: [Bro] Issue with bro reading a file that capturing live traffic Message-ID: Hi All, I run tcpdump live to capture the traffic into a file using "-w". Then I run bro to read that file offline using "-r". Both instances are running continuously. First it works fine but then bro stop generating results although it keep running, this means bro didn't continue reading from the file. Is it because bro -r is faster than the live capturing? How to let bro keep reading the file (this file is continuously whitening? My bro version: 2.3 running on ubuntu platform. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150928/657206e8/attachment.html From gabe at punchcyber.com Mon Sep 28 11:47:44 2015 From: gabe at punchcyber.com (Gabriel Dinkins) Date: Mon, 28 Sep 2015 14:47:44 -0400 Subject: [Bro] Bro Continually Crashes Message-ID: When starting the Bro service, an error stating that Bro terminated immediately after start-up continually appears. The following is the info I am provided upon running the diag command. --------------------------------------------------------------------------------------------------------------------------------------- Bro 2.4 Linux 2.6.32-573.7.1.el6.x86_64 ==== No reporter.log ==== stderr.log fatal error: problem with interface p3p1 (p3p1: no IPv4 address assigned) ==== stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited ==== .cmdline -i p3p1 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro broctl broctl/standalone broctl/auto ==== .env_vars PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site CLUSTER_NODE= ==== .status TERMINATED [atexit] ==== No prof.log ==== No packet_filter.log ==== No loaded_scripts.log ------------------------------------------------------------------------------------------------------------------------------------------ -- Gabriel Dinkins Cyber Security Engineer PUNCH Cyber Analytics Group -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150928/8490dc0d/attachment.html From skathare at solarflare.com Mon Sep 28 12:59:24 2015 From: skathare at solarflare.com (Sampada Kathare) Date: Mon, 28 Sep 2015 19:59:24 +0000 Subject: [Bro] Memory Issue with Bro Message-ID: Hello, I have been using the Bro 2.4 to test the performance of SFC driver. I have observed the following issue because of which I am unable to proceed with any analysis - There seems to be a memory leak somewhere as there are times when Bro runs out of memory too soon. These are the instances when drops are also seen too soon even at very low packet rates. When Bro is started, the available free memory keeps going down till a point where the server is extremely sluggish and there are drops being seen - An instance of Bro running out of memory (with 16 workers, no cpu pinning and having sent 155K pps for 7-8 minutes)- [root at dellr620c skathare]# free -m total used free shared buffers cached Mem: 32129 31917 211 3 1 376 --> 211MB : that's very low, considering the system started with some 26GB free memory (and this drop happens just within the first 2 minutes of running the traffic). System becomes very slow at this point and, of course, it has started dropping packets already. -/+ buffers/cache: 31539 589 Swap: 1907 1687 219 [root at dellr620c skathare]# Swap: 1907 1764 142 [root at dellr620c skathare]# cat /proc/meminfo MemTotal: 32900200 kB MemFree: 193384 kB MemAvailable: 480956 kB Buffers: 2464 kB Cached: 471260 kB SwapCached: 74860 kB Active: 23439908 kB Inactive: 3120012 kB Active(anon): 23179296 kB Inactive(anon): 2914628 kB Active(file): 260612 kB Inactive(file): 205384 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 1953076 kB SwapFree: 152264 kB Dirty: 22548 kB Writeback: 8 kB AnonPages: 26017216 kB Mapped: 15200 kB Shmem: 4692 kB Slab: 190556 kB SReclaimable: 93648 kB SUnreclaim: 96908 kB KernelStack: 4288 kB PageTables: 71380 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 15638376 kB Committed_AS: 29119672 kB VmallocTotal: 34359738367 kB VmallocUsed: 374920 kB VmallocChunk: 34342144308 kB HardwareCorrupted: 0 kB AnonHugePages: 243712 kB HugePages_Total: 2700 HugePages_Free: 46 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 345024 kB DirectMap2M: 18483200 kB DirectMap1G: 16777216 kB [root at dellr620c skathare]#top top - 22:48:06 up 70 days, 19:43, 5 users, load average: 18.25, 13.49, 10.04 Tasks: 17 total, 1 running, 16 sleeping, 0 stopped, 0 zombie %Cpu(s): 31.5 us, 2.9 sy, 0.7 ni, 43.7 id, 21.1 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32900200 total, 32731868 used, 168332 free, 336 buffers KiB Swap: 1953076 total, 1262956 used, 690120 free. 14248 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 20504 root 20 0 404644 32724 1464 D 0.0 0.1 6:32.62 bro 31 20552 root 20 0 404692 50108 1496 D 52.1 0.2 6:37.18 bro 29 20564 root 20 0 404824 50960 1508 D 49.8 0.2 6:37.16 bro 27 20574 root 20 0 404652 48748 1476 D 52.0 0.1 6:36.52 bro 24 20567 root 20 0 404684 49948 1456 D 0.0 0.2 6:33.32 bro 23 20561 root 20 0 421440 66672 1412 D 51.9 0.2 6:37.14 bro 18 20569 root 20 0 404708 31904 1508 D 41.1 0.1 6:34.77 bro 16 20495 root 20 0 404620 49936 1408 D 27.1 0.2 6:34.91 bro 13 20515 root 20 0 404684 46324 1500 D 21.9 0.1 6:33.25 bro 13 20548 root 20 0 404704 50188 1504 D 43.6 0.2 6:35.16 bro 13 20474 root 20 0 404736 32704 1508 D 0.0 0.1 6:32.79 bro 12 20502 root 20 0 404636 29300 1464 D 52.0 0.1 6:36.13 bro 12 20539 root 20 0 404748 32784 1484 D 44.7 0.1 6:34.08 bro 11 20537 root 20 0 404668 29284 1464 D 0.0 0.1 6:32.03 bro 4 20559 root 20 0 404684 32644 1444 R 54.6 0.1 6:38.12 bro 3 20542 root 20 0 404728 32704 1504 D 25.1 0.1 6:33.84 bro 1 20289 root 20 0 196768 412 412 S 0.0 0.0 0:00.03 solar_clusterd 0 After stopping the BRO workers (especially after the manager is killed/stopped), memory recovers - top - 22:53:01 up 70 days, 19:48, 5 users, load average: 3.06, 8.83, 9.20 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.1 us, 0.3 sy, 2.2 ni, 96.6 id, 0.8 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32900200 total, 6702252 used, 26197948 free, 8308 buffers --> This is almost what the system originally started with - 26GB KiB Swap: 1953076 total, 237216 used, 1715860 free. 528032 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 20289 root 20 0 196768 1028 484 S 0.0 0.0 0:00.03 solar_clusterd At very high packet rates, the available free memory keeps going down very fast and starts dropping packets. At lower packet rates, the drop in available free memory is comparatively slower, but it is still there and packets are dropped eventually. When the BRO workers are stopped, the available free memory recovers. During the few successful times when I have been able to go till 150Kpps without seeing any packet drops, the available free memory remained a constant at ~23G. It remained at this for the entire duration of the test (more than an hour ) and no drops were seen. The above data is a few days old. When I tried running BRO again today, I saw the memory drop from 18G to 4G in just a matter of few seconds after starting BRO (16 workers, each pinned to a CPU). Is it possible that Bro is accumulating some per-flow state and not freeing it? If so, is there any tuning that should be done to avoid this? Appreciate any help on this! - Sampada The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150928/cb067ed0/attachment-0001.html From mabuchan at gmail.com Mon Sep 28 13:27:42 2015 From: mabuchan at gmail.com (Mark Buchanan) Date: Mon, 28 Sep 2015 15:27:42 -0500 Subject: [Bro] Memory Issue with Bro In-Reply-To: References: Message-ID: Sampada, I've observed Bro utilizing more memory when PF_RING modules haven't been recompiled for the newest kernel. Once I recompiled for the current kernel, my memory usage got more inline with the numbers I would expect to see. Could that be a problem for you or is PF_RING not in the equation? Mark On Mon, Sep 28, 2015 at 2:59 PM, Sampada Kathare wrote: > Hello, > > > > I have been using the Bro 2.4 to test the performance of SFC driver. I > have observed the following issue because of which I am unable to proceed > with any analysis - > > > > There seems to be a memory leak somewhere as there are times when Bro runs > out of memory too soon. These are the instances when drops are also seen > too soon even at very low packet rates. > > When Bro is started, the available free memory keeps going down till a > point where the server is extremely sluggish and there are drops being seen > ? > > > > *An instance of Bro running out of memory (with 16 workers, no cpu pinning > and having sent 155K pps for 7-8 minutes)?* > > [root at dellr620c skathare]# free -m > > total used *free* shared buffers > cached > > Mem: 32129 31917 *211* 3 1 > 376 *?** 211MB : that?s very low, considering the system started > with some 26GB free memory (and this drop happens just within the first 2 > minutes of running the traffic). System becomes very slow at this > point and, of course, it has started dropping packets already.* > > -/+ buffers/cache: 31539 589 > > Swap: 1907 1687 219 > > [root at dellr620c skathare]# > > > > Swap: 1907 1764 142 > > > > > > [root at dellr620c skathare]# cat /proc/meminfo > > MemTotal: 32900200 kB > > *MemFree: 193384 kB* > > *MemAvailable: 480956 kB* > > Buffers: 2464 kB > > Cached: 471260 kB > > SwapCached: 74860 kB > > Active: 23439908 kB > > Inactive: 3120012 kB > > Active(anon): 23179296 kB > > Inactive(anon): 2914628 kB > > Active(file): 260612 kB > > Inactive(file): 205384 kB > > Unevictable: 0 kB > > Mlocked: 0 kB > > SwapTotal: 1953076 kB > > SwapFree: 152264 kB > > Dirty: 22548 kB > > Writeback: 8 kB > > AnonPages: 26017216 kB > > Mapped: 15200 kB > > Shmem: 4692 kB > > Slab: 190556 kB > > SReclaimable: 93648 kB > > SUnreclaim: 96908 kB > > KernelStack: 4288 kB > > PageTables: 71380 kB > > NFS_Unstable: 0 kB > > Bounce: 0 kB > > WritebackTmp: 0 kB > > CommitLimit: 15638376 kB > > Committed_AS: 29119672 kB > > VmallocTotal: 34359738367 kB > > VmallocUsed: 374920 kB > > VmallocChunk: 34342144308 kB > > HardwareCorrupted: 0 kB > > AnonHugePages: 243712 kB > > HugePages_Total: 2700 > > HugePages_Free: 46 > > HugePages_Rsvd: 0 > > HugePages_Surp: 0 > > Hugepagesize: 2048 kB > > DirectMap4k: 345024 kB > > DirectMap2M: 18483200 kB > > DirectMap1G: 16777216 kB > > > > > > [root at dellr620c skathare]#top > > > > top - 22:48:06 up 70 days, 19:43, 5 users, load average: 18.25, 13.49, > 10.04 > > Tasks:* 17 *total,* 1 *running,* 16 *sleeping,* 0 *stopped,* 0 * > zombie > > %Cpu(s):* 31.5 *us,* 2.9 *sy,* 0.7 *ni,* 43.7 *id,* 21.1 *wa,* 0.0 * > hi,* 0.0 *si,* 0.0 *st > > KiB Mem: * 32900200 *total,* 32731868 *used,* 168332 free*,* 336 * > buffers > > KiB Swap:* 1953076 *total,* 1262956 *used,* 690120 *free.* 14248 > cached Mem* > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND P > > 20504 root 20 0 404644 32724 1464 D 0.0 0.1 6:32.62 > bro 31 > > 20552 root 20 0 404692 50108 1496 D 52.1 0.2 6:37.18 > bro 29 > > 20564 root 20 0 404824 50960 1508 D 49.8 0.2 6:37.16 > bro 27 > > 20574 root 20 0 404652 48748 1476 D 52.0 0.1 6:36.52 > bro 24 > > 20567 root 20 0 404684 49948 1456 D 0.0 0.2 6:33.32 > bro 23 > > 20561 root 20 0 421440 66672 1412 D 51.9 0.2 6:37.14 > bro 18 > > 20569 root 20 0 404708 31904 1508 D 41.1 0.1 6:34.77 > bro 16 > > 20495 root 20 0 404620 49936 1408 D 27.1 0.2 6:34.91 > bro 13 > > 20515 root 20 0 404684 46324 1500 D 21.9 0.1 6:33.25 > bro 13 > > 20548 root 20 0 404704 50188 1504 D 43.6 0.2 6:35.16 > bro 13 > > 20474 root 20 0 404736 32704 1508 D 0.0 0.1 6:32.79 > bro 12 > > 20502 root 20 0 404636 29300 1464 D 52.0 0.1 6:36.13 > bro 12 > > 20539 root 20 0 404748 32784 1484 D 44.7 0.1 6:34.08 > bro 11 > > 20537 root 20 0 404668 29284 1464 D 0.0 0.1 6:32.03 > bro 4 > > *20559 root 20 0 404684 32644 1444 R 54.6 0.1 6:38.12 > bro 3* > > 20542 root 20 0 404728 32704 1504 D 25.1 0.1 6:33.84 > bro 1 > > 20289 root 20 0 196768 412 412 S 0.0 0.0 0:00.03 > solar_clusterd 0 > > > > > > *After stopping the BRO workers (especially after the manager is > killed/stopped), memory recovers ?* > > > > top - 22:53:01 up 70 days, 19:48, 5 users, load average: 3.06, 8.83, 9.20 > > Tasks:* 1 *total,* 0 *running,* 1 *sleeping,* 0 *stopped,* 0 * > zombie > > %Cpu(s):* 0.1 *us,* 0.3 *sy,* 2.2 *ni,* 96.6 *id,* 0.8 *wa,* 0.0 * > hi,* 0.0 *si,* 0.0 *st > > KiB Mem: * 32900200 *total,* 6702252 *used,* 26197948 free*,* 8308 *buffers > *?** This is almost what the system originally started with ? 26GB* > > KiB Swap:* 1953076 *total,* 237216 *used,* 1715860 *free.* 528032 > cached Mem* > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND P > > 20289 root 20 0 196768 1028 484 S 0.0 0.0 0:00.03 > solar_clusterd > > > > > > > > At very high packet rates, the available free memory keeps going down very > fast and starts dropping packets. At lower packet rates, the drop in > available free memory is comparatively slower, but it is still there and > packets are dropped eventually. When the BRO workers are stopped, the > available free memory recovers. During the few successful times when I have > been able to go till 150Kpps without seeing any packet drops, the available > free memory remained a constant at ~23G. It remained at this for the > entire duration of the test (more than an hour ) and no drops were seen. > > > > The above data is a few days old. When I tried running BRO again today, I > saw the memory drop from 18G to 4G in just a matter of few seconds after > starting BRO (16 workers, each pinned to a CPU). Is it possible that Bro is > accumulating some per-flow state and not freeing it? If so, is there any > tuning that should be done to avoid this? > > > > Appreciate any help on this! > > > > - > > Sampada > The information contained in this message is confidential and is intended > for the addressee(s) only. If you have received this message in error, > please notify the sender immediately and delete the message. Unless you are > an addressee (or authorized to receive for an addressee), you may not use, > copy or disclose to anyone this message or any information contained in > this message. The unauthorized use, disclosure, copying or alteration of > this message is strictly prohibited. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Mark Buchanan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150928/ac8139d6/attachment-0001.html From mabuchan at gmail.com Mon Sep 28 13:29:15 2015 From: mabuchan at gmail.com (Mark Buchanan) Date: Mon, 28 Sep 2015 15:29:15 -0500 Subject: [Bro] Bro Continually Crashes In-Reply-To: References: Message-ID: It sounds like you don't have an IPv4 address assigned to the interface on your RHEL/CentOS/Oracle Linux box. I've solved this in the past by using "IPADDR=0.0.0.0" and "NETMASK=255.255.255.255" in the /etc/sysconfig/networking-scripts/ifcfg-eth file. Does that fix the issue? Best regards, Mark On Mon, Sep 28, 2015 at 1:47 PM, Gabriel Dinkins wrote: > When starting the Bro service, an error stating that Bro terminated > immediately after start-up continually appears. The following is the info I > am provided upon running the diag command. > > > --------------------------------------------------------------------------------------------------------------------------------------- > > Bro 2.4 > Linux 2.6.32-573.7.1.el6.x86_64 > > > ==== No reporter.log > > ==== stderr.log > fatal error: problem with interface p3p1 (p3p1: no IPv4 address assigned) > > ==== stdout.log > max memory size (kbytes, -m) unlimited > data seg size (kbytes, -d) unlimited > virtual memory (kbytes, -v) unlimited > core file size (blocks, -c) unlimited > > ==== .cmdline > -i p3p1 -U .status -p broctl -p broctl-live -p standalone -p local -p bro > local.bro broctl broctl/standalone broctl/auto > > ==== .env_vars > > PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin > > BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site > CLUSTER_NODE= > > ==== .status > TERMINATED [atexit] > > ==== No prof.log > > ==== No packet_filter.log > > ==== No loaded_scripts.log > > > ------------------------------------------------------------------------------------------------------------------------------------------ > > -- > Gabriel Dinkins > Cyber Security Engineer > PUNCH Cyber Analytics Group > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Mark Buchanan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150928/cdbc728c/attachment.html From skathare at solarflare.com Mon Sep 28 13:35:43 2015 From: skathare at solarflare.com (Sampada Kathare) Date: Mon, 28 Sep 2015 20:35:43 +0000 Subject: [Bro] Memory Issue with Bro In-Reply-To: References: Message-ID: Mark, I have tried Bro both, with and without PF-RING, and saw the memory issue in both cases. When I am not using PF-RING, I use Solar Capture libpcap and flow balancer. - Sampada From: Mark Buchanan [mailto:mabuchan at gmail.com] Sent: Monday, September 28, 2015 1:28 PM To: Sampada Kathare Cc: bro at bro.org Subject: Re: [Bro] Memory Issue with Bro Sampada, I've observed Bro utilizing more memory when PF_RING modules haven't been recompiled for the newest kernel. Once I recompiled for the current kernel, my memory usage got more inline with the numbers I would expect to see. Could that be a problem for you or is PF_RING not in the equation? Mark On Mon, Sep 28, 2015 at 2:59 PM, Sampada Kathare > wrote: Hello, I have been using the Bro 2.4 to test the performance of SFC driver. I have observed the following issue because of which I am unable to proceed with any analysis - There seems to be a memory leak somewhere as there are times when Bro runs out of memory too soon. These are the instances when drops are also seen too soon even at very low packet rates. When Bro is started, the available free memory keeps going down till a point where the server is extremely sluggish and there are drops being seen ? An instance of Bro running out of memory (with 16 workers, no cpu pinning and having sent 155K pps for 7-8 minutes)? [root at dellr620c skathare]# free -m total used free shared buffers cached Mem: 32129 31917 211 3 1 376 --> 211MB : that?s very low, considering the system started with some 26GB free memory (and this drop happens just within the first 2 minutes of running the traffic). System becomes very slow at this point and, of course, it has started dropping packets already. -/+ buffers/cache: 31539 589 Swap: 1907 1687 219 [root at dellr620c skathare]# Swap: 1907 1764 142 [root at dellr620c skathare]# cat /proc/meminfo MemTotal: 32900200 kB MemFree: 193384 kB MemAvailable: 480956 kB Buffers: 2464 kB Cached: 471260 kB SwapCached: 74860 kB Active: 23439908 kB Inactive: 3120012 kB Active(anon): 23179296 kB Inactive(anon): 2914628 kB Active(file): 260612 kB Inactive(file): 205384 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 1953076 kB SwapFree: 152264 kB Dirty: 22548 kB Writeback: 8 kB AnonPages: 26017216 kB Mapped: 15200 kB Shmem: 4692 kB Slab: 190556 kB SReclaimable: 93648 kB SUnreclaim: 96908 kB KernelStack: 4288 kB PageTables: 71380 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 15638376 kB Committed_AS: 29119672 kB VmallocTotal: 34359738367 kB VmallocUsed: 374920 kB VmallocChunk: 34342144308 kB HardwareCorrupted: 0 kB AnonHugePages: 243712 kB HugePages_Total: 2700 HugePages_Free: 46 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 345024 kB DirectMap2M: 18483200 kB DirectMap1G: 16777216 kB [root at dellr620c skathare]#top top - 22:48:06 up 70 days, 19:43, 5 users, load average: 18.25, 13.49, 10.04 Tasks: 17 total, 1 running, 16 sleeping, 0 stopped, 0 zombie %Cpu(s): 31.5 us, 2.9 sy, 0.7 ni, 43.7 id, 21.1 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32900200 total, 32731868 used, 168332 free, 336 buffers KiB Swap: 1953076 total, 1262956 used, 690120 free. 14248 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 20504 root 20 0 404644 32724 1464 D 0.0 0.1 6:32.62 bro 31 20552 root 20 0 404692 50108 1496 D 52.1 0.2 6:37.18 bro 29 20564 root 20 0 404824 50960 1508 D 49.8 0.2 6:37.16 bro 27 20574 root 20 0 404652 48748 1476 D 52.0 0.1 6:36.52 bro 24 20567 root 20 0 404684 49948 1456 D 0.0 0.2 6:33.32 bro 23 20561 root 20 0 421440 66672 1412 D 51.9 0.2 6:37.14 bro 18 20569 root 20 0 404708 31904 1508 D 41.1 0.1 6:34.77 bro 16 20495 root 20 0 404620 49936 1408 D 27.1 0.2 6:34.91 bro 13 20515 root 20 0 404684 46324 1500 D 21.9 0.1 6:33.25 bro 13 20548 root 20 0 404704 50188 1504 D 43.6 0.2 6:35.16 bro 13 20474 root 20 0 404736 32704 1508 D 0.0 0.1 6:32.79 bro 12 20502 root 20 0 404636 29300 1464 D 52.0 0.1 6:36.13 bro 12 20539 root 20 0 404748 32784 1484 D 44.7 0.1 6:34.08 bro 11 20537 root 20 0 404668 29284 1464 D 0.0 0.1 6:32.03 bro 4 20559 root 20 0 404684 32644 1444 R 54.6 0.1 6:38.12 bro 3 20542 root 20 0 404728 32704 1504 D 25.1 0.1 6:33.84 bro 1 20289 root 20 0 196768 412 412 S 0.0 0.0 0:00.03 solar_clusterd 0 After stopping the BRO workers (especially after the manager is killed/stopped), memory recovers ? top - 22:53:01 up 70 days, 19:48, 5 users, load average: 3.06, 8.83, 9.20 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.1 us, 0.3 sy, 2.2 ni, 96.6 id, 0.8 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32900200 total, 6702252 used, 26197948 free, 8308 buffers --> This is almost what the system originally started with ? 26GB KiB Swap: 1953076 total, 237216 used, 1715860 free. 528032 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 20289 root 20 0 196768 1028 484 S 0.0 0.0 0:00.03 solar_clusterd At very high packet rates, the available free memory keeps going down very fast and starts dropping packets. At lower packet rates, the drop in available free memory is comparatively slower, but it is still there and packets are dropped eventually. When the BRO workers are stopped, the available free memory recovers. During the few successful times when I have been able to go till 150Kpps without seeing any packet drops, the available free memory remained a constant at ~23G. It remained at this for the entire duration of the test (more than an hour ) and no drops were seen. The above data is a few days old. When I tried running BRO again today, I saw the memory drop from 18G to 4G in just a matter of few seconds after starting BRO (16 workers, each pinned to a CPU). Is it possible that Bro is accumulating some per-flow state and not freeing it? If so, is there any tuning that should be done to avoid this? Appreciate any help on this! - Sampada The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited. _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Mark Buchanan The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150928/9a90b807/attachment-0001.html From skathare at solarflare.com Mon Sep 28 13:39:24 2015 From: skathare at solarflare.com (Sampada Kathare) Date: Mon, 28 Sep 2015 20:39:24 +0000 Subject: [Bro] Memory Issue with Bro References: Message-ID: <8a5755db1cc04a6e80372d76174ce6d6@ocex03.SolarFlarecom.com> In any case, I think I can give it a shot at least for the PF-RING testcase. How do I re-compile PF-RING for the newest kernel? - Sampada From: Sampada Kathare Sent: Monday, September 28, 2015 1:37 PM To: 'Mark Buchanan' Cc: bro at bro.org Subject: RE: [Bro] Memory Issue with Bro Mark, I have tried Bro both, with and without PF-RING, and saw the memory issue in both cases. When I am not using PF-RING, I use Solar Capture libpcap and flow balancer. - Sampada From: Mark Buchanan [mailto:mabuchan at gmail.com] Sent: Monday, September 28, 2015 1:28 PM To: Sampada Kathare > Cc: bro at bro.org Subject: Re: [Bro] Memory Issue with Bro Sampada, I've observed Bro utilizing more memory when PF_RING modules haven't been recompiled for the newest kernel. Once I recompiled for the current kernel, my memory usage got more inline with the numbers I would expect to see. Could that be a problem for you or is PF_RING not in the equation? Mark On Mon, Sep 28, 2015 at 2:59 PM, Sampada Kathare > wrote: Hello, I have been using the Bro 2.4 to test the performance of SFC driver. I have observed the following issue because of which I am unable to proceed with any analysis - There seems to be a memory leak somewhere as there are times when Bro runs out of memory too soon. These are the instances when drops are also seen too soon even at very low packet rates. When Bro is started, the available free memory keeps going down till a point where the server is extremely sluggish and there are drops being seen ? An instance of Bro running out of memory (with 16 workers, no cpu pinning and having sent 155K pps for 7-8 minutes)? [root at dellr620c skathare]# free -m total used free shared buffers cached Mem: 32129 31917 211 3 1 376 --> 211MB : that?s very low, considering the system started with some 26GB free memory (and this drop happens just within the first 2 minutes of running the traffic). System becomes very slow at this point and, of course, it has started dropping packets already. -/+ buffers/cache: 31539 589 Swap: 1907 1687 219 [root at dellr620c skathare]# Swap: 1907 1764 142 [root at dellr620c skathare]# cat /proc/meminfo MemTotal: 32900200 kB MemFree: 193384 kB MemAvailable: 480956 kB Buffers: 2464 kB Cached: 471260 kB SwapCached: 74860 kB Active: 23439908 kB Inactive: 3120012 kB Active(anon): 23179296 kB Inactive(anon): 2914628 kB Active(file): 260612 kB Inactive(file): 205384 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 1953076 kB SwapFree: 152264 kB Dirty: 22548 kB Writeback: 8 kB AnonPages: 26017216 kB Mapped: 15200 kB Shmem: 4692 kB Slab: 190556 kB SReclaimable: 93648 kB SUnreclaim: 96908 kB KernelStack: 4288 kB PageTables: 71380 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 15638376 kB Committed_AS: 29119672 kB VmallocTotal: 34359738367 kB VmallocUsed: 374920 kB VmallocChunk: 34342144308 kB HardwareCorrupted: 0 kB AnonHugePages: 243712 kB HugePages_Total: 2700 HugePages_Free: 46 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 345024 kB DirectMap2M: 18483200 kB DirectMap1G: 16777216 kB [root at dellr620c skathare]#top top - 22:48:06 up 70 days, 19:43, 5 users, load average: 18.25, 13.49, 10.04 Tasks: 17 total, 1 running, 16 sleeping, 0 stopped, 0 zombie %Cpu(s): 31.5 us, 2.9 sy, 0.7 ni, 43.7 id, 21.1 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32900200 total, 32731868 used, 168332 free, 336 buffers KiB Swap: 1953076 total, 1262956 used, 690120 free. 14248 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 20504 root 20 0 404644 32724 1464 D 0.0 0.1 6:32.62 bro 31 20552 root 20 0 404692 50108 1496 D 52.1 0.2 6:37.18 bro 29 20564 root 20 0 404824 50960 1508 D 49.8 0.2 6:37.16 bro 27 20574 root 20 0 404652 48748 1476 D 52.0 0.1 6:36.52 bro 24 20567 root 20 0 404684 49948 1456 D 0.0 0.2 6:33.32 bro 23 20561 root 20 0 421440 66672 1412 D 51.9 0.2 6:37.14 bro 18 20569 root 20 0 404708 31904 1508 D 41.1 0.1 6:34.77 bro 16 20495 root 20 0 404620 49936 1408 D 27.1 0.2 6:34.91 bro 13 20515 root 20 0 404684 46324 1500 D 21.9 0.1 6:33.25 bro 13 20548 root 20 0 404704 50188 1504 D 43.6 0.2 6:35.16 bro 13 20474 root 20 0 404736 32704 1508 D 0.0 0.1 6:32.79 bro 12 20502 root 20 0 404636 29300 1464 D 52.0 0.1 6:36.13 bro 12 20539 root 20 0 404748 32784 1484 D 44.7 0.1 6:34.08 bro 11 20537 root 20 0 404668 29284 1464 D 0.0 0.1 6:32.03 bro 4 20559 root 20 0 404684 32644 1444 R 54.6 0.1 6:38.12 bro 3 20542 root 20 0 404728 32704 1504 D 25.1 0.1 6:33.84 bro 1 20289 root 20 0 196768 412 412 S 0.0 0.0 0:00.03 solar_clusterd 0 After stopping the BRO workers (especially after the manager is killed/stopped), memory recovers ? top - 22:53:01 up 70 days, 19:48, 5 users, load average: 3.06, 8.83, 9.20 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.1 us, 0.3 sy, 2.2 ni, 96.6 id, 0.8 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32900200 total, 6702252 used, 26197948 free, 8308 buffers --> This is almost what the system originally started with ? 26GB KiB Swap: 1953076 total, 237216 used, 1715860 free. 528032 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 20289 root 20 0 196768 1028 484 S 0.0 0.0 0:00.03 solar_clusterd At very high packet rates, the available free memory keeps going down very fast and starts dropping packets. At lower packet rates, the drop in available free memory is comparatively slower, but it is still there and packets are dropped eventually. When the BRO workers are stopped, the available free memory recovers. During the few successful times when I have been able to go till 150Kpps without seeing any packet drops, the available free memory remained a constant at ~23G. It remained at this for the entire duration of the test (more than an hour ) and no drops were seen. The above data is a few days old. When I tried running BRO again today, I saw the memory drop from 18G to 4G in just a matter of few seconds after starting BRO (16 workers, each pinned to a CPU). Is it possible that Bro is accumulating some per-flow state and not freeing it? If so, is there any tuning that should be done to avoid this? Appreciate any help on this! - Sampada The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited. _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Mark Buchanan The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150928/04089214/attachment-0001.html From vgarramone at gmail.com Mon Sep 28 14:27:46 2015 From: vgarramone at gmail.com (V. Garramone) Date: Mon, 28 Sep 2015 16:27:46 -0500 Subject: [Bro] Raw (eml) Email Extraction Bro 2.4 Message-ID: Hi Everyone, I would like to do full email extraction (eml) to file from STMP traffic; should this happen naturally with the new file extraction framework? I found this exchange from a while back, but haven't found anything more recent on the topic: http://mailman.icsi.berkeley.edu/pipermail/bro/2014-July/007224.html I'm currently using Bro 2.4 and a script pretty similar to this one for file extraction: https://github.com/Security-Onion-Solutions/securityonion-bro -scripts/blob/master/file-extraction/extract.bro It looks like I'm getting the message content and attachments, but apparently not the raw email. Any tips would be greatly appreciated! Thanks very much, VG -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150928/8525549b/attachment.html From damian.gerow at shopify.com Tue Sep 29 07:22:24 2015 From: damian.gerow at shopify.com (Damian Gerow) Date: Tue, 29 Sep 2015 10:22:24 -0400 Subject: [Bro] Reading data into a table Message-ID: We've run into a few problems with our scripts and the use of &persistent, so we're looking to do some home-grown persistence. These scripts are part of a module called ConnectionValidation; the applicable bits of the scripts are at https://gist.github.com/mutemule/6076cddce3ce8c9e7013. It's worth pointing out that the module as a whole is loaded in to all components, but the persistence layer is only loaded in to the proxy. What I'm seeing is the table being written to disk as expected during bro_done(), but seemingly not being read back in during bro_init(): after startup, the table remains blank in all cluster components. I'd previously tried this with a set instead of a table, but that didn't work. Then I tried using events to populate the set, but that also didn't work. So now I'm on a table, and following the input framework documentation[0] almost exactly, but it's still not doing what I expected. What am I doing wrong? How do I read a table in from disk during initialization/startup? [0] https://www.bro.org/sphinx/frameworks/input.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150929/229231d1/attachment.html From earl.eiland at root9b.com Tue Sep 29 08:14:39 2015 From: earl.eiland at root9b.com (Earl Eiland) Date: Tue, 29 Sep 2015 15:14:39 +0000 Subject: [Bro] Reading data into a table In-Reply-To: References: Message-ID: Hello, Damian. I ran into a similar situation. I had defined conversation end nodes in a set and tried to use the set as a table entry identifier ? bro does not support this. I got around the problem by generating a table entry for both end node permutations. Earl Eiland From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Damian Gerow Sent: Tuesday, September 29, 2015 9:22 AM To: bro at bro.org Subject: [Bro] Reading data into a table We've run into a few problems with our scripts and the use of &persistent, so we're looking to do some home-grown persistence. These scripts are part of a module called ConnectionValidation; the applicable bits of the scripts are at https://gist.github.com/mutemule/6076cddce3ce8c9e7013. It's worth pointing out that the module as a whole is loaded in to all components, but the persistence layer is only loaded in to the proxy. What I'm seeing is the table being written to disk as expected during bro_done(), but seemingly not being read back in during bro_init(): after startup, the table remains blank in all cluster components. I'd previously tried this with a set instead of a table, but that didn't work. Then I tried using events to populate the set, but that also didn't work. So now I'm on a table, and following the input framework documentation[0] almost exactly, but it's still not doing what I expected. What am I doing wrong? How do I read a table in from disk during initialization/startup? [0] https://www.bro.org/sphinx/frameworks/input.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150929/43b9f564/attachment.html From jazoff at illinois.edu Tue Sep 29 08:56:05 2015 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 29 Sep 2015 15:56:05 +0000 Subject: [Bro] Memory Issue with Bro In-Reply-To: References: Message-ID: <7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu> > > [root at dellr620c skathare]#top > > top - 22:48:06 up 70 days, 19:43, 5 users, load average: 18.25, 13.49, 10.04 > Tasks: 17 total, 1 running, 16 sleeping, 0 stopped, 0 zombie > %Cpu(s): 31.5 us, 2.9 sy, 0.7 ni, 43.7 id, 21.1 wa, 0.0 hi, 0.0 si, 0.0 st > KiB Mem: 32900200 total, 32731868 used, 168332 free, 336 buffers > KiB Swap: 1953076 total, 1262956 used, 690120 free. 14248 cached Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P > 20504 root 20 0 404644 32724 1464 D 0.0 0.1 6:32.62 bro 31 > 20552 root 20 0 404692 50108 1496 D 52.1 0.2 6:37.18 bro 29 > 20564 root 20 0 404824 50960 1508 D 49.8 0.2 6:37.16 bro 27 > 20574 root 20 0 404652 48748 1476 D 52.0 0.1 6:36.52 bro 24 > 20567 root 20 0 404684 49948 1456 D 0.0 0.2 6:33.32 bro 23 > 20561 root 20 0 421440 66672 1412 D 51.9 0.2 6:37.14 bro 18 > 20569 root 20 0 404708 31904 1508 D 41.1 0.1 6:34.77 bro 16 > 20495 root 20 0 404620 49936 1408 D 27.1 0.2 6:34.91 bro 13 > 20515 root 20 0 404684 46324 1500 D 21.9 0.1 6:33.25 bro 13 > 20548 root 20 0 404704 50188 1504 D 43.6 0.2 6:35.16 bro 13 > 20474 root 20 0 404736 32704 1508 D 0.0 0.1 6:32.79 bro 12 > 20502 root 20 0 404636 29300 1464 D 52.0 0.1 6:36.13 bro 12 > 20539 root 20 0 404748 32784 1484 D 44.7 0.1 6:34.08 bro 11 > 20537 root 20 0 404668 29284 1464 D 0.0 0.1 6:32.03 bro 4 > 20559 root 20 0 404684 32644 1444 R 54.6 0.1 6:38.12 bro 3 > 20542 root 20 0 404728 32704 1504 D 25.1 0.1 6:33.84 bro 1 > 20289 root 20 0 196768 412 412 S 0.0 0.0 0:00.03 solar_clusterd 0 > Are those the only bro processes you are running? the %MEM is only .1 or .2% for all of those process, and even if I round most of them up to .2% that only adds up to 3.2% Can you press the M key in top to sort by memory usage? -- - Justin Azoff From Nathan.Pigott at parsons.com Tue Sep 29 10:08:01 2015 From: Nathan.Pigott at parsons.com (Pigott, Nathan) Date: Tue, 29 Sep 2015 17:08:01 +0000 Subject: [Bro] File name from fa_file Message-ID: <098822A9728D1643818A1347FC9875D44F9B62BD@HSV-MB001.huntsville.ads.sparta.com> Hello, I'm having problems getting file names from fa_file - the field f$info$filename is showing up uninitialized on every single fa_file in all my tests. Is there a known reason why this would be happening? I'm using Bro 2.3, but I tested on 2.4 as well and got the same results. Are there any alternative ways to get file names? For now I'm parsing the URL returned by Files::describe(f), but this does not work if the URL doesn't contain the file name, or if the file was transferred with a protocol other than HTTP. Thanks, Nathan Pigott -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150929/26942aa8/attachment-0001.html From hosom at battelle.org Tue Sep 29 10:26:57 2015 From: hosom at battelle.org (Hosom, Stephen M) Date: Tue, 29 Sep 2015 17:26:57 +0000 Subject: [Bro] File name from fa_file In-Reply-To: <098822A9728D1643818A1347FC9875D44F9B62BD@HSV-MB001.huntsville.ads.sparta.com> References: <098822A9728D1643818A1347FC9875D44F9B62BD@HSV-MB001.huntsville.ads.sparta.com> Message-ID: Filename does not always exist. That field is only created under circumstances where the protocol has a portion that would tell the server or client receiving the file what the name should be-most commonly that applies to HTTP. What is it that you're trying to do with filenames, or what information are you attempting to derive from them? Generally it isn't wise to trust filenames that you see on the wire for a whole lot. From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Pigott, Nathan Sent: Tuesday, September 29, 2015 1:08 PM To: bro at bro.org Subject: [Bro] File name from fa_file Hello, I'm having problems getting file names from fa_file - the field f$info$filename is showing up uninitialized on every single fa_file in all my tests. Is there a known reason why this would be happening? I'm using Bro 2.3, but I tested on 2.4 as well and got the same results. Are there any alternative ways to get file names? For now I'm parsing the URL returned by Files::describe(f), but this does not work if the URL doesn't contain the file name, or if the file was transferred with a protocol other than HTTP. Thanks, Nathan Pigott -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150929/e638ccc7/attachment.html From skathare at solarflare.com Tue Sep 29 10:29:53 2015 From: skathare at solarflare.com (Sampada Kathare) Date: Tue, 29 Sep 2015 17:29:53 +0000 Subject: [Bro] Memory Issue with Bro In-Reply-To: <7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu> References: <7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu> Message-ID: <69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com> Hi, These are the bro workers. I haven't shown the memory usage of the Bro manager and proxy processes. I believe the manager is the one that takes up most of the memory as when I stop the manager, the available free memory goes up by almost 10G! I will send out that log shortly. Could the manager in anyway be accumulating per worker or per flow state and not freeing it? Thanks! - Sampada -----Original Message----- From: Azoff, Justin S [mailto:jazoff at illinois.edu] Sent: Tuesday, September 29, 2015 8:56 AM To: Sampada Kathare Cc: bro at bro.org Subject: Re: [Bro] Memory Issue with Bro Importance: High > > [root at dellr620c skathare]#top > > top - 22:48:06 up 70 days, 19:43, 5 users, load average: 18.25, 13.49, 10.04 > Tasks: 17 total, 1 running, 16 sleeping, 0 stopped, 0 zombie > %Cpu(s): 31.5 us, 2.9 sy, 0.7 ni, 43.7 id, 21.1 wa, 0.0 hi, 0.0 si, 0.0 st > KiB Mem: 32900200 total, 32731868 used, 168332 free, 336 buffers > KiB Swap: 1953076 total, 1262956 used, 690120 free. 14248 cached Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P > 20504 root 20 0 404644 32724 1464 D 0.0 0.1 6:32.62 bro 31 > 20552 root 20 0 404692 50108 1496 D 52.1 0.2 6:37.18 bro 29 > 20564 root 20 0 404824 50960 1508 D 49.8 0.2 6:37.16 bro 27 > 20574 root 20 0 404652 48748 1476 D 52.0 0.1 6:36.52 bro 24 > 20567 root 20 0 404684 49948 1456 D 0.0 0.2 6:33.32 bro 23 > 20561 root 20 0 421440 66672 1412 D 51.9 0.2 6:37.14 bro 18 > 20569 root 20 0 404708 31904 1508 D 41.1 0.1 6:34.77 bro 16 > 20495 root 20 0 404620 49936 1408 D 27.1 0.2 6:34.91 bro 13 > 20515 root 20 0 404684 46324 1500 D 21.9 0.1 6:33.25 bro 13 > 20548 root 20 0 404704 50188 1504 D 43.6 0.2 6:35.16 bro 13 > 20474 root 20 0 404736 32704 1508 D 0.0 0.1 6:32.79 bro 12 > 20502 root 20 0 404636 29300 1464 D 52.0 0.1 6:36.13 bro 12 > 20539 root 20 0 404748 32784 1484 D 44.7 0.1 6:34.08 bro 11 > 20537 root 20 0 404668 29284 1464 D 0.0 0.1 6:32.03 bro 4 > 20559 root 20 0 404684 32644 1444 R 54.6 0.1 6:38.12 bro 3 > 20542 root 20 0 404728 32704 1504 D 25.1 0.1 6:33.84 bro 1 > 20289 root 20 0 196768 412 412 S 0.0 0.0 0:00.03 solar_clusterd 0 > Are those the only bro processes you are running? the %MEM is only .1 or .2% for all of those process, and even if I round most of them up to .2% that only adds up to 3.2% Can you press the M key in top to sort by memory usage? -- - Justin Azoff The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited. From gabe at punchcyber.com Tue Sep 29 10:36:58 2015 From: gabe at punchcyber.com (Gabriel Dinkins) Date: Tue, 29 Sep 2015 13:36:58 -0400 Subject: [Bro] Bro Continually Crashes In-Reply-To: References: Message-ID: That doesn't seem to fix the issue I'm still getting the error. On Mon, Sep 28, 2015 at 4:29 PM, Mark Buchanan wrote: > It sounds like you don't have an IPv4 address assigned to the interface on > your RHEL/CentOS/Oracle Linux box. I've solved this in the past by using > "IPADDR=0.0.0.0" and "NETMASK=255.255.255.255" in the > /etc/sysconfig/networking-scripts/ifcfg-eth file. > > Does that fix the issue? > > Best regards, > Mark > > On Mon, Sep 28, 2015 at 1:47 PM, Gabriel Dinkins > wrote: > >> When starting the Bro service, an error stating that Bro terminated >> immediately after start-up continually appears. The following is the info I >> am provided upon running the diag command. >> >> >> --------------------------------------------------------------------------------------------------------------------------------------- >> >> Bro 2.4 >> Linux 2.6.32-573.7.1.el6.x86_64 >> >> >> ==== No reporter.log >> >> ==== stderr.log >> fatal error: problem with interface p3p1 (p3p1: no IPv4 address assigned) >> >> ==== stdout.log >> max memory size (kbytes, -m) unlimited >> data seg size (kbytes, -d) unlimited >> virtual memory (kbytes, -v) unlimited >> core file size (blocks, -c) unlimited >> >> ==== .cmdline >> -i p3p1 -U .status -p broctl -p broctl-live -p standalone -p local -p bro >> local.bro broctl broctl/standalone broctl/auto >> >> ==== .env_vars >> >> PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:/usr/local/bro/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin >> >> BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/site::/usr/local/bro/spool/installed-scripts-do-not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/bro/policy:/usr/local/bro/share/bro/site >> CLUSTER_NODE= >> >> ==== .status >> TERMINATED [atexit] >> >> ==== No prof.log >> >> ==== No packet_filter.log >> >> ==== No loaded_scripts.log >> >> >> ------------------------------------------------------------------------------------------------------------------------------------------ >> >> -- >> Gabriel Dinkins >> Cyber Security Engineer >> PUNCH Cyber Analytics Group >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > > > -- > Mark Buchanan > -- Gabriel Dinkins Cyber Security Engineer PUNCH Cyber Analytics Group -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150929/10e57a42/attachment.html From Nathan.Pigott at parsons.com Tue Sep 29 10:56:38 2015 From: Nathan.Pigott at parsons.com (Pigott, Nathan) Date: Tue, 29 Sep 2015 17:56:38 +0000 Subject: [Bro] File name from fa_file In-Reply-To: References: <098822A9728D1643818A1347FC9875D44F9B62BD@HSV-MB001.huntsville.ads.sparta.com>, Message-ID: <098822A9728D1643818A1347FC9875D44F9B62E0@HSV-MB001.huntsville.ads.sparta.com> I see. My goal is to check each file's filename against its given mime type to ensure they match. Since f$info$filename is frequently non-existent, is there any more reliable way to get filenames besides parsing them out of the URL? Is this a fruitless/unnecessary pursuit since mime type can also be spoofed? ________________________________ From: Hosom, Stephen M [hosom at battelle.org] Sent: Tuesday, September 29, 2015 1:26 PM To: Pigott, Nathan; bro at bro.org Subject: RE: File name from fa_file Filename does not always exist. That field is only created under circumstances where the protocol has a portion that would tell the server or client receiving the file what the name should be?most commonly that applies to HTTP. What is it that you?re trying to do with filenames, or what information are you attempting to derive from them? Generally it isn?t wise to trust filenames that you see on the wire for a whole lot. From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Pigott, Nathan Sent: Tuesday, September 29, 2015 1:08 PM To: bro at bro.org Subject: [Bro] File name from fa_file Hello, I'm having problems getting file names from fa_file - the field f$info$filename is showing up uninitialized on every single fa_file in all my tests. Is there a known reason why this would be happening? I'm using Bro 2.3, but I tested on 2.4 as well and got the same results. Are there any alternative ways to get file names? For now I'm parsing the URL returned by Files::describe(f), but this does not work if the URL doesn't contain the file name, or if the file was transferred with a protocol other than HTTP. Thanks, Nathan Pigott -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150929/eef110c4/attachment-0001.html From jazoff at illinois.edu Tue Sep 29 12:18:53 2015 From: jazoff at illinois.edu (Azoff, Justin S) Date: Tue, 29 Sep 2015 19:18:53 +0000 Subject: [Bro] Memory Issue with Bro In-Reply-To: <69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com> References: <7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu> <69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com> Message-ID: <454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu> > On Sep 29, 2015, at 12:29 PM, Sampada Kathare wrote: > > Hi, > > These are the bro workers. I haven't shown the memory usage of the Bro manager and proxy processes. I believe the manager is the one that takes up most of the memory as when I stop the manager, the available free memory goes up by almost 10G! I will send out that log shortly. There's actually a manager parent and child process, knowing which one is using the memory can help figure this out. > Could the manager in anyway be accumulating per worker or per flow state and not freeing it? > > Thanks! There is sort of a known issue if the manager can't keep up logging the amount of data it is being sent. What sort of data are you sending bro? Is it something like random data that will cause a LOT of logging? -- - Justin Azoff From skathare at solarflare.com Tue Sep 29 14:03:04 2015 From: skathare at solarflare.com (Sampada Kathare) Date: Tue, 29 Sep 2015 21:03:04 +0000 Subject: [Bro] Memory Issue with Bro In-Reply-To: <454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu> References: <7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu> <69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com> <454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu> Message-ID: <9b1a7b20e3c8406294411a357d8a3f8f@ocex03.SolarFlarecom.com> Hi, I re-ran the same test case just now to get the data for you - 16 bro workers, each pinned to a core, no cpu pinning for the manager and proxy processes, data sent at 155000 pps - Before running traffic - top - 21:48:24 up 2:11, 5 users, load average: 0.84, 1.14, 0.88 Tasks: 20 total, 0 running, 20 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.6 us, 2.1 sy, 0.0 ni, 96.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32900200 total, 8381556 used, 24518644 free, 55736 buffers KiB Swap: 1953076 total, 0 used, 1953076 free. 249556 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 18301 root 25 5 129980 46096 884 S 0.0 0.1 0:00.00 bro 1 18299 root 25 5 130020 46116 892 S 0.0 0.1 0:00.00 bro 11 18294 root 25 5 129988 46116 892 S 0.0 0.1 0:00.00 bro 12 18289 root 25 5 130012 46076 840 S 0.0 0.1 0:00.00 bro 15 18285 root 25 5 130024 46128 896 S 0.0 0.1 0:00.00 bro 10 18284 root 25 5 129960 46132 892 S 0.0 0.1 0:00.00 bro 14 18283 root 25 5 130048 46136 888 S 0.0 0.1 0:00.00 bro 13 17863 root 20 0 387408 51964 5548 S 9.7 0.2 0:31.44 bro 16 17860 root 20 0 387344 51968 5548 S 9.3 0.2 0:29.73 bro 15 17836 root 20 0 387416 51956 5548 S 8.7 0.2 0:29.62 bro 14 17835 root 20 0 387348 51960 5548 S 9.7 0.2 0:30.02 bro 13 17834 root 20 0 387348 51948 5552 S 9.0 0.2 0:29.31 bro 11 17833 root 20 0 387292 51952 5548 S 9.3 0.2 0:29.26 bro 12 17832 root 20 0 387340 51960 5548 S 9.3 0.2 0:29.85 bro 10 17738 root 20 0 387296 51920 5548 S 10.0 0.2 0:30.59 bro 1 17651 root 25 5 145956 75268 960 S 0.0 0.2 0:00.00 bro 2 17650 root 20 0 109988 43080 5096 S 1.3 0.1 0:04.50 bro 0 17613 root 25 5 146096 75396 944 S 0.0 0.2 0:00.01 bro 9 17604 root 20 0 405392 45428 5116 S 0.7 0.1 0:03.71 bro 26 17457 root 20 0 196772 9016 3448 S 0.0 0.0 0:00.02 solar_clusterd 0 The highlighted rows are Bro manager processes (17613 being the child and 17604 being the parent) After running traffic at 150000 pps for 3 minutes -> top - 21:53:58 up 2:16, 5 users, load average: 20.90, 11.89, 5.36 Tasks: 20 total, 10 running, 10 sleeping, 0 stopped, 0 zombie %Cpu(s): 55.3 us, 3.4 sy, 0.8 ni, 37.5 id, 3.0 wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem: 32900200 total, 32703156 used, 197044 free, 892 buffers KiB Swap: 1953076 total, 626996 used, 1326080 free. 331096 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 18301 root 25 5 134092 46032 676 S 0.3 0.1 0:00.78 bro 1 18299 root 25 5 134132 46052 676 S 0.3 0.1 0:00.79 bro 11 18294 root 25 5 134100 2024 676 S 0.7 0.0 0:00.88 bro 12 18289 root 25 5 134124 46000 676 S 0.3 0.1 0:00.82 bro 15 18285 root 25 5 134136 2016 676 S 0.3 0.0 0:00.87 bro 10 18284 root 25 5 134072 1992 676 S 0.7 0.0 0:00.87 bro 14 18283 root 25 5 134160 46064 676 S 0.7 0.1 0:00.80 bro 13 17863 root 20 0 404636 24628 3540 R 0.0 0.1 4:36.63 bro 16 17860 root 20 0 404668 67472 3544 R 100.0 0.2 4:29.44 bro 15 17836 root 20 0 404612 24580 3540 R 100.0 0.1 4:35.53 bro 14 17835 root 20 0 404608 67412 3544 R 0.0 0.2 4:33.82 bro 13 17834 root 20 0 404608 67388 3544 R 100.0 0.2 4:35.69 bro 11 17833 root 20 0 404636 24648 3540 R 0.0 0.1 4:32.05 bro 12 17832 root 20 0 404632 24588 3540 R 100.0 0.1 4:36.37 bro 10 17738 root 20 0 404624 67368 3544 R 100.0 0.2 4:38.29 bro 1 17651 root 25 5 145956 7488 648 S 0.0 0.0 0:00.00 bro 17 17650 root 20 0 109988 3892 2916 S 1.7 0.0 0:09.39 bro 29 17613 root 25 5 871884 763968 668 R 99.9 2.3 3:39.04 bro 17 17604 root 20 0 22.869g 0.022t 2932 R 195.4 71.2 7:41.59 bro 18 17457 root 20 0 196772 2272 2272 S 0.0 0.0 0:00.02 solar_clusterd 0 As you can see, the manager parent process seems to be using 71% of the memory and it's CPU utilization is also 195%. - Sampada -----Original Message----- From: Azoff, Justin S [mailto:jazoff at illinois.edu] Sent: Tuesday, September 29, 2015 12:19 PM To: Sampada Kathare Cc: bro at bro.org Subject: Re: [Bro] Memory Issue with Bro > On Sep 29, 2015, at 12:29 PM, Sampada Kathare > wrote: > > Hi, > > These are the bro workers. I haven't shown the memory usage of the Bro manager and proxy processes. I believe the manager is the one that takes up most of the memory as when I stop the manager, the available free memory goes up by almost 10G! I will send out that log shortly. There's actually a manager parent and child process, knowing which one is using the memory can help figure this out. > Could the manager in anyway be accumulating per worker or per flow state and not freeing it? > > Thanks! There is sort of a known issue if the manager can't keep up logging the amount of data it is being sent. What sort of data are you sending bro? Is it something like random data that will cause a LOT of logging? -- - Justin Azoff The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150929/3258b635/attachment-0001.html From skathare at solarflare.com Tue Sep 29 14:08:08 2015 From: skathare at solarflare.com (Sampada Kathare) Date: Tue, 29 Sep 2015 21:08:08 +0000 Subject: [Bro] Memory Issue with Bro References: <7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu> <69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com> <454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu> Message-ID: <518d3bcc041046a3a0dc9cb2b652e385@ocex03.SolarFlarecom.com> Continuing the previous test case - After another 7-8 minutes, I don't see the BRO manager process running and memory seems to have been restored - top - 22:04:54 up 2:27, 5 users, load average: 16.04, 16.06, 11.24 Tasks: 18 total, 8 running, 10 sleeping, 0 stopped, 0 zombie %Cpu(s): 47.7 us, 0.1 sy, 0.0 ni, 52.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32900200 total, 8363464 used, 24536736 free, 4152 buffers KiB Swap: 1953076 total, 743412 used, 1209664 free. 608176 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 18301 root 25 5 132036 19816 540 S 0.0 0.1 0:00.88 bro 1 18299 root 25 5 132076 25040 468 S 0.0 0.1 0:00.83 bro 11 18294 root 25 5 132044 4208 480 S 0.0 0.0 0:01.04 bro 12 18289 root 25 5 132068 22424 548 S 0.0 0.1 0:00.84 bro 15 18285 root 25 5 132080 2276 468 S 0.0 0.0 0:00.89 bro 10 18284 root 25 5 132016 2276 444 S 0.0 0.0 0:00.92 bro 14 18283 root 25 5 132104 20792 520 S 0.0 0.1 0:00.85 bro 13 17863 root 20 0 404724 24268 1568 R 100.0 0.1 15:23.31 bro 16 17860 root 20 0 404592 44080 1568 R 91.2 0.1 14:33.31 bro 15 17836 root 20 0 405076 24604 1592 R 94.5 0.1 15:09.50 bro 14 17835 root 20 0 404632 42456 1568 R 0.0 0.1 15:06.73 bro 13 17834 root 20 0 404616 42740 1592 R 95.2 0.1 15:02.40 bro 11 17833 root 20 0 404612 24252 1592 R 94.9 0.1 15:00.49 bro 12 17832 root 20 0 404740 24052 1528 R 100.0 0.1 15:20.78 bro 10 17738 root 20 0 404676 41420 1528 R 0.0 0.1 15:27.26 bro 1 17651 root 25 5 143900 4820 360 S 0.0 0.0 0:00.01 bro 19 17650 root 20 0 109988 1916 736 S 1.3 0.0 0:19.00 bro 19 17457 root 20 0 196772 272 272 S 0.0 0.0 0:00.02 solar_clusterd 0 PID 17604 and 17613 missing above? Is this an expected behavior? During the initial few minutes, does the manager do some sort of stabilization to get everything in order, possibly due to the high traffic rate? It is during these few minutes that I saw packets being dropped. I don't see any drops right now. - Sampada From: Sampada Kathare Sent: Tuesday, September 29, 2015 2:04 PM To: 'Azoff, Justin S' Cc: bro at bro.org Subject: RE: [Bro] Memory Issue with Bro Hi, I re-ran the same test case just now to get the data for you - 16 bro workers, each pinned to a core, no cpu pinning for the manager and proxy processes, data sent at 155000 pps - Before running traffic - top - 21:48:24 up 2:11, 5 users, load average: 0.84, 1.14, 0.88 Tasks: 20 total, 0 running, 20 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.6 us, 2.1 sy, 0.0 ni, 96.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32900200 total, 8381556 used, 24518644 free, 55736 buffers KiB Swap: 1953076 total, 0 used, 1953076 free. 249556 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 18301 root 25 5 129980 46096 884 S 0.0 0.1 0:00.00 bro 1 18299 root 25 5 130020 46116 892 S 0.0 0.1 0:00.00 bro 11 18294 root 25 5 129988 46116 892 S 0.0 0.1 0:00.00 bro 12 18289 root 25 5 130012 46076 840 S 0.0 0.1 0:00.00 bro 15 18285 root 25 5 130024 46128 896 S 0.0 0.1 0:00.00 bro 10 18284 root 25 5 129960 46132 892 S 0.0 0.1 0:00.00 bro 14 18283 root 25 5 130048 46136 888 S 0.0 0.1 0:00.00 bro 13 17863 root 20 0 387408 51964 5548 S 9.7 0.2 0:31.44 bro 16 17860 root 20 0 387344 51968 5548 S 9.3 0.2 0:29.73 bro 15 17836 root 20 0 387416 51956 5548 S 8.7 0.2 0:29.62 bro 14 17835 root 20 0 387348 51960 5548 S 9.7 0.2 0:30.02 bro 13 17834 root 20 0 387348 51948 5552 S 9.0 0.2 0:29.31 bro 11 17833 root 20 0 387292 51952 5548 S 9.3 0.2 0:29.26 bro 12 17832 root 20 0 387340 51960 5548 S 9.3 0.2 0:29.85 bro 10 17738 root 20 0 387296 51920 5548 S 10.0 0.2 0:30.59 bro 1 17651 root 25 5 145956 75268 960 S 0.0 0.2 0:00.00 bro 2 17650 root 20 0 109988 43080 5096 S 1.3 0.1 0:04.50 bro 0 17613 root 25 5 146096 75396 944 S 0.0 0.2 0:00.01 bro 9 17604 root 20 0 405392 45428 5116 S 0.7 0.1 0:03.71 bro 26 17457 root 20 0 196772 9016 3448 S 0.0 0.0 0:00.02 solar_clusterd 0 The highlighted rows are Bro manager processes (17613 being the child and 17604 being the parent) After running traffic at 150000 pps for 3 minutes -> top - 21:53:58 up 2:16, 5 users, load average: 20.90, 11.89, 5.36 Tasks: 20 total, 10 running, 10 sleeping, 0 stopped, 0 zombie %Cpu(s): 55.3 us, 3.4 sy, 0.8 ni, 37.5 id, 3.0 wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem: 32900200 total, 32703156 used, 197044 free, 892 buffers KiB Swap: 1953076 total, 626996 used, 1326080 free. 331096 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 18301 root 25 5 134092 46032 676 S 0.3 0.1 0:00.78 bro 1 18299 root 25 5 134132 46052 676 S 0.3 0.1 0:00.79 bro 11 18294 root 25 5 134100 2024 676 S 0.7 0.0 0:00.88 bro 12 18289 root 25 5 134124 46000 676 S 0.3 0.1 0:00.82 bro 15 18285 root 25 5 134136 2016 676 S 0.3 0.0 0:00.87 bro 10 18284 root 25 5 134072 1992 676 S 0.7 0.0 0:00.87 bro 14 18283 root 25 5 134160 46064 676 S 0.7 0.1 0:00.80 bro 13 17863 root 20 0 404636 24628 3540 R 0.0 0.1 4:36.63 bro 16 17860 root 20 0 404668 67472 3544 R 100.0 0.2 4:29.44 bro 15 17836 root 20 0 404612 24580 3540 R 100.0 0.1 4:35.53 bro 14 17835 root 20 0 404608 67412 3544 R 0.0 0.2 4:33.82 bro 13 17834 root 20 0 404608 67388 3544 R 100.0 0.2 4:35.69 bro 11 17833 root 20 0 404636 24648 3540 R 0.0 0.1 4:32.05 bro 12 17832 root 20 0 404632 24588 3540 R 100.0 0.1 4:36.37 bro 10 17738 root 20 0 404624 67368 3544 R 100.0 0.2 4:38.29 bro 1 17651 root 25 5 145956 7488 648 S 0.0 0.0 0:00.00 bro 17 17650 root 20 0 109988 3892 2916 S 1.7 0.0 0:09.39 bro 29 17613 root 25 5 871884 763968 668 R 99.9 2.3 3:39.04 bro 17 17604 root 20 0 22.869g 0.022t 2932 R 195.4 71.2 7:41.59 bro 18 17457 root 20 0 196772 2272 2272 S 0.0 0.0 0:00.02 solar_clusterd 0 As you can see, the manager parent process seems to be using 71% of the memory and it's CPU utilization is also 195%. - Sampada -----Original Message----- From: Azoff, Justin S [mailto:jazoff at illinois.edu] Sent: Tuesday, September 29, 2015 12:19 PM To: Sampada Kathare > Cc: bro at bro.org Subject: Re: [Bro] Memory Issue with Bro > On Sep 29, 2015, at 12:29 PM, Sampada Kathare > wrote: > > Hi, > > These are the bro workers. I haven't shown the memory usage of the Bro manager and proxy processes. I believe the manager is the one that takes up most of the memory as when I stop the manager, the available free memory goes up by almost 10G! I will send out that log shortly. There's actually a manager parent and child process, knowing which one is using the memory can help figure this out. > Could the manager in anyway be accumulating per worker or per flow state and not freeing it? > > Thanks! There is sort of a known issue if the manager can't keep up logging the amount of data it is being sent. What sort of data are you sending bro? Is it something like random data that will cause a LOT of logging? -- - Justin Azoff The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150929/187874d9/attachment-0001.html From skathare at solarflare.com Tue Sep 29 16:15:18 2015 From: skathare at solarflare.com (Sampada Kathare) Date: Tue, 29 Sep 2015 23:15:18 +0000 Subject: [Bro] Memory Issue with Bro References: <7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu> <69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com> <454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu> Message-ID: <7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com> What is the meaning of this error? - SolarCapture session=4382/1 log=/var/tmp/solar_capture_root_4382/1 ERROR: errno=-114 from core/sc_ef_vi.c:1323 in sc_ef_vi_alloc_ts(): ERROR: sc_ef_vi_alloc_ts: ef_vi_alloc_from_pd failed (-114) sfsc_activate: sc_vi_alloc failed (ERROR: sc_ef_vi_alloc_ts: ef_vi_alloc_from_pd failed (-114) ) errno=Unknown error -114 fatal error: problem with interface enp65s0f0 (enp65s0f0: sfsc_activate: sc_vi_alloc failed (ERROR: sc_ef_vi_alloc_ts: ef_vi_alloc_from_pd failed (-114) ) errno=Unknown error -114) I see this in diag while creating 16 workers (sometimes even for lesser number). As a result, some of the workers are not started. - Sampada From: Sampada Kathare Sent: Tuesday, September 29, 2015 2:09 PM To: 'Azoff, Justin S' Cc: 'bro at bro.org' Subject: RE: [Bro] Memory Issue with Bro Continuing the previous test case - After another 7-8 minutes, I don't see the BRO manager process running and memory seems to have been restored - top - 22:04:54 up 2:27, 5 users, load average: 16.04, 16.06, 11.24 Tasks: 18 total, 8 running, 10 sleeping, 0 stopped, 0 zombie %Cpu(s): 47.7 us, 0.1 sy, 0.0 ni, 52.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32900200 total, 8363464 used, 24536736 free, 4152 buffers KiB Swap: 1953076 total, 743412 used, 1209664 free. 608176 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 18301 root 25 5 132036 19816 540 S 0.0 0.1 0:00.88 bro 1 18299 root 25 5 132076 25040 468 S 0.0 0.1 0:00.83 bro 11 18294 root 25 5 132044 4208 480 S 0.0 0.0 0:01.04 bro 12 18289 root 25 5 132068 22424 548 S 0.0 0.1 0:00.84 bro 15 18285 root 25 5 132080 2276 468 S 0.0 0.0 0:00.89 bro 10 18284 root 25 5 132016 2276 444 S 0.0 0.0 0:00.92 bro 14 18283 root 25 5 132104 20792 520 S 0.0 0.1 0:00.85 bro 13 17863 root 20 0 404724 24268 1568 R 100.0 0.1 15:23.31 bro 16 17860 root 20 0 404592 44080 1568 R 91.2 0.1 14:33.31 bro 15 17836 root 20 0 405076 24604 1592 R 94.5 0.1 15:09.50 bro 14 17835 root 20 0 404632 42456 1568 R 0.0 0.1 15:06.73 bro 13 17834 root 20 0 404616 42740 1592 R 95.2 0.1 15:02.40 bro 11 17833 root 20 0 404612 24252 1592 R 94.9 0.1 15:00.49 bro 12 17832 root 20 0 404740 24052 1528 R 100.0 0.1 15:20.78 bro 10 17738 root 20 0 404676 41420 1528 R 0.0 0.1 15:27.26 bro 1 17651 root 25 5 143900 4820 360 S 0.0 0.0 0:00.01 bro 19 17650 root 20 0 109988 1916 736 S 1.3 0.0 0:19.00 bro 19 17457 root 20 0 196772 272 272 S 0.0 0.0 0:00.02 solar_clusterd 0 PID 17604 and 17613 missing above? Is this an expected behavior? During the initial few minutes, does the manager do some sort of stabilization to get everything in order, possibly due to the high traffic rate? It is during these few minutes that I saw packets being dropped. I don't see any drops right now. - Sampada From: Sampada Kathare Sent: Tuesday, September 29, 2015 2:04 PM To: 'Azoff, Justin S' > Cc: bro at bro.org Subject: RE: [Bro] Memory Issue with Bro Hi, I re-ran the same test case just now to get the data for you - 16 bro workers, each pinned to a core, no cpu pinning for the manager and proxy processes, data sent at 155000 pps - Before running traffic - top - 21:48:24 up 2:11, 5 users, load average: 0.84, 1.14, 0.88 Tasks: 20 total, 0 running, 20 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.6 us, 2.1 sy, 0.0 ni, 96.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32900200 total, 8381556 used, 24518644 free, 55736 buffers KiB Swap: 1953076 total, 0 used, 1953076 free. 249556 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 18301 root 25 5 129980 46096 884 S 0.0 0.1 0:00.00 bro 1 18299 root 25 5 130020 46116 892 S 0.0 0.1 0:00.00 bro 11 18294 root 25 5 129988 46116 892 S 0.0 0.1 0:00.00 bro 12 18289 root 25 5 130012 46076 840 S 0.0 0.1 0:00.00 bro 15 18285 root 25 5 130024 46128 896 S 0.0 0.1 0:00.00 bro 10 18284 root 25 5 129960 46132 892 S 0.0 0.1 0:00.00 bro 14 18283 root 25 5 130048 46136 888 S 0.0 0.1 0:00.00 bro 13 17863 root 20 0 387408 51964 5548 S 9.7 0.2 0:31.44 bro 16 17860 root 20 0 387344 51968 5548 S 9.3 0.2 0:29.73 bro 15 17836 root 20 0 387416 51956 5548 S 8.7 0.2 0:29.62 bro 14 17835 root 20 0 387348 51960 5548 S 9.7 0.2 0:30.02 bro 13 17834 root 20 0 387348 51948 5552 S 9.0 0.2 0:29.31 bro 11 17833 root 20 0 387292 51952 5548 S 9.3 0.2 0:29.26 bro 12 17832 root 20 0 387340 51960 5548 S 9.3 0.2 0:29.85 bro 10 17738 root 20 0 387296 51920 5548 S 10.0 0.2 0:30.59 bro 1 17651 root 25 5 145956 75268 960 S 0.0 0.2 0:00.00 bro 2 17650 root 20 0 109988 43080 5096 S 1.3 0.1 0:04.50 bro 0 17613 root 25 5 146096 75396 944 S 0.0 0.2 0:00.01 bro 9 17604 root 20 0 405392 45428 5116 S 0.7 0.1 0:03.71 bro 26 17457 root 20 0 196772 9016 3448 S 0.0 0.0 0:00.02 solar_clusterd 0 The highlighted rows are Bro manager processes (17613 being the child and 17604 being the parent) After running traffic at 150000 pps for 3 minutes -> top - 21:53:58 up 2:16, 5 users, load average: 20.90, 11.89, 5.36 Tasks: 20 total, 10 running, 10 sleeping, 0 stopped, 0 zombie %Cpu(s): 55.3 us, 3.4 sy, 0.8 ni, 37.5 id, 3.0 wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem: 32900200 total, 32703156 used, 197044 free, 892 buffers KiB Swap: 1953076 total, 626996 used, 1326080 free. 331096 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND P 18301 root 25 5 134092 46032 676 S 0.3 0.1 0:00.78 bro 1 18299 root 25 5 134132 46052 676 S 0.3 0.1 0:00.79 bro 11 18294 root 25 5 134100 2024 676 S 0.7 0.0 0:00.88 bro 12 18289 root 25 5 134124 46000 676 S 0.3 0.1 0:00.82 bro 15 18285 root 25 5 134136 2016 676 S 0.3 0.0 0:00.87 bro 10 18284 root 25 5 134072 1992 676 S 0.7 0.0 0:00.87 bro 14 18283 root 25 5 134160 46064 676 S 0.7 0.1 0:00.80 bro 13 17863 root 20 0 404636 24628 3540 R 0.0 0.1 4:36.63 bro 16 17860 root 20 0 404668 67472 3544 R 100.0 0.2 4:29.44 bro 15 17836 root 20 0 404612 24580 3540 R 100.0 0.1 4:35.53 bro 14 17835 root 20 0 404608 67412 3544 R 0.0 0.2 4:33.82 bro 13 17834 root 20 0 404608 67388 3544 R 100.0 0.2 4:35.69 bro 11 17833 root 20 0 404636 24648 3540 R 0.0 0.1 4:32.05 bro 12 17832 root 20 0 404632 24588 3540 R 100.0 0.1 4:36.37 bro 10 17738 root 20 0 404624 67368 3544 R 100.0 0.2 4:38.29 bro 1 17651 root 25 5 145956 7488 648 S 0.0 0.0 0:00.00 bro 17 17650 root 20 0 109988 3892 2916 S 1.7 0.0 0:09.39 bro 29 17613 root 25 5 871884 763968 668 R 99.9 2.3 3:39.04 bro 17 17604 root 20 0 22.869g 0.022t 2932 R 195.4 71.2 7:41.59 bro 18 17457 root 20 0 196772 2272 2272 S 0.0 0.0 0:00.02 solar_clusterd 0 As you can see, the manager parent process seems to be using 71% of the memory and it's CPU utilization is also 195%. - Sampada -----Original Message----- From: Azoff, Justin S [mailto:jazoff at illinois.edu] Sent: Tuesday, September 29, 2015 12:19 PM To: Sampada Kathare > Cc: bro at bro.org Subject: Re: [Bro] Memory Issue with Bro > On Sep 29, 2015, at 12:29 PM, Sampada Kathare > wrote: > > Hi, > > These are the bro workers. I haven't shown the memory usage of the Bro manager and proxy processes. I believe the manager is the one that takes up most of the memory as when I stop the manager, the available free memory goes up by almost 10G! I will send out that log shortly. There's actually a manager parent and child process, knowing which one is using the memory can help figure this out. > Could the manager in anyway be accumulating per worker or per flow state and not freeing it? > > Thanks! There is sort of a known issue if the manager can't keep up logging the amount of data it is being sent. What sort of data are you sending bro? Is it something like random data that will cause a LOT of logging? -- - Justin Azoff The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error, please notify the sender immediately and delete the message. Unless you are an addressee (or authorized to receive for an addressee), you may not use, copy or disclose to anyone this message or any information contained in this message. The unauthorized use, disclosure, copying or alteration of this message is strictly prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150929/9441d855/attachment-0001.html From norton.perry at gmail.com Wed Sep 30 03:39:22 2015 From: norton.perry at gmail.com (nortonperry@gmail.com) Date: Wed, 30 Sep 2015 11:39:22 +0100 Subject: [Bro] General advice on malware hunting? Message-ID: Hi Gents, I have had Bro installed as my gateway for a home network for about nine months now, with a complete (mostly uninterrupted) run of logs. I've also supplemented this with the critical stack plugin since July, with intel feeds up - focused mostly on malware and candc domains. The network is reasonably busy, has probably about 25 discreet hosts of which at any given time between 3 and 10 are up. I have suspected there is malware / a rootkit perhaps on the network for a while as arp -a shows a lot of hosts every now and then from the terminal of most systems on the network. Also, Nmap scans often report IP addresses that simply are not there. Also, Bro reports traffic to local NAT IP addresses that don't exist. eg my network is divided into a 192.168.2.x (Internal, all the hosts) and 192.168.1.x(airgap between Bro router and domestic DSL router). The 192.168.1.x network only really ever has two hosts - the bro router and the dsl router, but connections show to other addresses which don't exist. I have tried to put a methodology together for malware hunting based on what I can find online, but nothing has really come to light. I use zcat, bro-cut and regular expressions to query the logs. Would anyone on this list mind assisting me in a bug hunt / provide a methodology for tracking down suspicious traffic? I have looked and looked but can't seem to find any workflow / tolling which can isolate malware effectively. Any advice on this would be very gratefully received! regs Perry -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150930/36ba891b/attachment.html From liburdi.joshua at gmail.com Wed Sep 30 06:41:01 2015 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Wed, 30 Sep 2015 09:41:01 -0400 Subject: [Bro] General advice on malware hunting? In-Reply-To: References: Message-ID: If you're going to do any serious hunting, then you should probably use a tool that makes viewing the data easier. Try ELK or Splunk. ELK is good if you just want to retrieve log data, Splunk can do that and it includes fairly robust statistical analysis (this is very useful for hunting). With the amount of logs you likely have, you'll exceed Splunk's trial license limit, but if you upload all of the logs at once, they won't suspend your account or your ability to search your data. For what you described, I recommend getting a trial Splunk license and putting all your data in that. On Wed, Sep 30, 2015 at 6:39 AM, nortonperry at gmail.com wrote: > Hi Gents, > > I have had Bro installed as my gateway for a home network for about nine > months now, with a complete (mostly uninterrupted) run of logs. I've also > supplemented this with the critical stack plugin since July, with intel > feeds up - focused mostly on malware and candc domains. > > The network is reasonably busy, has probably about 25 discreet hosts of > which at any given time between 3 and 10 are up. I have suspected there is > malware / a rootkit perhaps on the network for a while as arp -a shows a lot > of hosts every now and then from the terminal of most systems on > the network. Also, Nmap scans often report IP addresses that simply are not > there. > > Also, Bro reports traffic to local NAT IP addresses that don't exist. eg my > network is divided into a 192.168.2.x (Internal, all the hosts) and > 192.168.1.x(airgap between Bro router and domestic DSL router). The > 192.168.1.x network only really ever has two hosts - the bro router and the > dsl router, but connections show to other addresses which don't exist. > > I have tried to put a methodology together for malware hunting based on what > I can find online, but nothing has really come to light. I use zcat, bro-cut > and regular expressions to query the logs. > > Would anyone on this list mind assisting me in a bug hunt / provide a > methodology for tracking down suspicious traffic? > > I have looked and looked but can't seem to find any workflow / tolling which > can isolate malware effectively. Any advice on this would be very gratefully > received! > > regs > > Perry > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at icir.org Wed Sep 30 06:48:37 2015 From: seth at icir.org (Seth Hall) Date: Wed, 30 Sep 2015 09:48:37 -0400 Subject: [Bro] Memory Issue with Bro In-Reply-To: <7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com> References: <7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu> <69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com> <454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu> <7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com> Message-ID: <3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org> > On Sep 29, 2015, at 7:15 PM, Sampada Kathare wrote: > > What is the meaning of this error? ? > > SolarCapture session=4382/1 log=/var/tmp/solar_capture_root_4382/1 > ERROR: errno=-114 from core/sc_ef_vi.c:1323 in sc_ef_vi_alloc_ts(): > ERROR: sc_ef_vi_alloc_ts: ef_vi_alloc_from_pd failed (-114) That appears to be a problem with your own NIC (I?m assuming... solarflare error messages, solarflare.com email address). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From blackhole.em at gmail.com Wed Sep 30 09:55:22 2015 From: blackhole.em at gmail.com (Joe Blow) Date: Wed, 30 Sep 2015 12:55:22 -0400 Subject: [Bro] Memory Issue with Bro In-Reply-To: <3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org> References: <7E1F7811-8496-4F0F-A2BD-7D2FB419C6D2@illinois.edu> <69b57edc540149be8c81d28423ae6be8@ocex03.SolarFlarecom.com> <454B7312-B84C-41B0-BD1A-2B0445D99480@illinois.edu> <7689d3f6ddb6454d855e6fd3a3b8ea7b@ocex03.SolarFlarecom.com> <3D4C7E0B-479A-4186-A169-77941BC432DC@icir.org> Message-ID: I'm super interested in this thread, as I believe i'm experiencing the same memory leak, using the solarflare cards. i'm running a similar setup, with 20 workers and lots of traffic, but i'm having to bounce the entire NIC once Bro goes haywire. Bro doesn't take too long before it's wiped the whole box out of memory (all 192GB). Please let me know how to troubleshooting goes. I'm happy to provide logs. Cheers, JB On Wed, Sep 30, 2015 at 9:48 AM, Seth Hall wrote: > > > On Sep 29, 2015, at 7:15 PM, Sampada Kathare > wrote: > > > > What is the meaning of this error? ? > > > > SolarCapture session=4382/1 log=/var/tmp/solar_capture_root_4382/1 > > ERROR: errno=-114 from core/sc_ef_vi.c:1323 in sc_ef_vi_alloc_ts(): > > ERROR: sc_ef_vi_alloc_ts: ef_vi_alloc_from_pd failed (-114) > > That appears to be a problem with your own NIC (I?m assuming... solarflare > error messages, solarflare.com email address). > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150930/1373665e/attachment.html