From asharma at lbl.gov Sat Jul 1 08:43:27 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Sat, 1 Jul 2017 08:43:27 -0700 Subject: [Bro] Shared object "libcaf_core.so.0.14.1" not found, required by "libbroker.so.0" Message-ID: <20170701154325.GS12970@mac-822.local> So FYI in case someone is upgrading to bro-2.5.1 with broker: Install actor-framework: https://github.com/actor-framework/actor-framework/ need to : git checkout tags/0.14.1 and then build it Aashish From dnthayer at illinois.edu Sat Jul 1 17:11:10 2017 From: dnthayer at illinois.edu (Daniel Thayer) Date: Sat, 1 Jul 2017 19:11:10 -0500 Subject: [Bro] Shared object "libcaf_core.so.0.14.1" not found, required by "libbroker.so.0" In-Reply-To: <20170701154325.GS12970@mac-822.local> References: <20170701154325.GS12970@mac-822.local> Message-ID: On 7/1/17 10:43 AM, Aashish Sharma wrote: > So FYI in case someone is upgrading to bro-2.5.1 with broker: > > Install actor-framework: > > https://github.com/actor-framework/actor-framework/ > > need to : > > git checkout tags/0.14.1 > > and then build it > > Aashish Just wanted to add that any particular release of CAF can also be obtained on this page: https://github.com/actor-framework/actor-framework/releases From marcin.nawrocki at fu-berlin.de Mon Jul 3 03:51:39 2017 From: marcin.nawrocki at fu-berlin.de (Marcin Nawrocki) Date: Mon, 3 Jul 2017 12:51:39 +0200 Subject: [Bro] Real-time reporting from multiple sensors to multiple analysis points Message-ID: <230078a0-adf3-61d1-52f7-ddd64df0f54a@fu-berlin.de> Dear bro mailing list, I have a question regarding the configuration of bro and its real-time reporting features. Right now, I have two sensors (s1, s2), each running one bro node with log files rotating every hour. After the rotation, I send the files from each sensor to an analysis point (a1) via scp and perform my analysis steps. My requirements changed now: I want to know what happens on the sensors in almost real-time. How do I send reports from (s1,s2) with a max. delay of 10 seconds to another analysis point (a2)? The reports can still reach (a1) every hour to keep the load low. My intuition tells me, that a very low rotation interval and scp are not the best practice here. Regards, Marcin Nawrocki From vladg at illinois.edu Mon Jul 3 12:26:20 2017 From: vladg at illinois.edu (Vlad Grigorescu) Date: Mon, 03 Jul 2017 14:26:20 -0500 Subject: [Bro] Problems in writing an analyzer for custom TCP-based protocol In-Reply-To: <201706221540.v5MFeqxA080772@vladg.net> References: <201706221540.v5MFeqxA080772@vladg.net> Message-ID: Valerio, As you probably know, BinPAC supports reading data in via two "modes" -- flowunit or datagram: flowunit (https://www.bro.org/sphinx/components/binpac/README.html#id20): > When flowunit is used, the analyzer uses flow buffer to handle > incremental input and provide support for &oneline/&length. For > further detail on this, see Buffering. datagram (https://www.bro.org/sphinx/components/binpac/README.html#id21): > Opposite to flowunit, by declaring data unit as datagram, flow buffer > is opted out. This results in faster parsing but no incremental input > or buffering support. Additionally, there is a TCP reassembler that you can use to handle data across multiple TCP packets. From what you said below, it sounds like you would want flowunit parsing and the TCP reassembler. The MySQL analyzer might be a good example that you can reference: https://github.com/bro/bro/tree/master/src/analyzer/protocol/mysql Other analyzers that use the TCP reassembler are: DNP3, GSSAPI, IMAP, KRB_TCP, Modbus, NTLM, RDP, RFB, SIP_TCP, SOCKS, SSH, SSL, Syslog and XMPP. --Vlad Valerio writes: > Hi all, > > I am trying to write a simple analyzer with BinPAC for a custom binary > TCP protocol with the following structure > > +------------+---------------------------------------------------------+ > |FIXED | MESSAGE_1 # MESSAGE_2 # .......# MESSAGE_N | > > |HEX PREFIX | | > +------------+---------------------------------------------------------+ > > The problem is that the above structure doesn't have a fixed lenght and, > therefore, it can be spread across several TCP packets. > > At the moment, my prot_protocol.pac file has the following structure: > > type PROT_Message = record { > entry: uint8[] &until($input.length() == 0); > }; > > type PROT_PDU(is_orig: bool) = record { > entry : bytestring &restofdata; > } &byteorder=bigendian; > > While the analyzer processes every packet by calling the following > function (as defined in prot_analyzer.pac > > refine flow PROT_FLOW += { > function proc_prot_message(msg: PROT_PDU): bool > [...] > > The analyzer is activated with a dpd signature that matches the FIXED > HEX PREFIX. > Once activated, the function proc_prot_message is called for each packet > in the session exposing in msg its payload. > > What is the best way to feed the analyzer with the whole reassembled TCP > payload so that I can process it once without having to keep state while > processing each single packet? > > many thanks in advance, > Valerio > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170703/8cf294c2/attachment.bin From dave.a.florek at gmail.com Mon Jul 3 12:29:53 2017 From: dave.a.florek at gmail.com (David Florek) Date: Mon, 3 Jul 2017 15:29:53 -0400 Subject: [Bro] Bro Digest, Vol 135, Issue 3 In-Reply-To: References: Message-ID: Have you considered syslog? > On Jul 3, 2017, at 3:00 PM, bro-request at bro.org wrote: > > Send Bro mailing list submissions to > bro at bro.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > or, via email, send a message with subject or body 'help' to > bro-request at bro.org > > You can reach the person managing the list at > bro-owner at bro.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Bro digest..." > > > Today's Topics: > > 1. Real-time reporting from multiple sensors to multiple > analysis points (Marcin Nawrocki) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 3 Jul 2017 12:51:39 +0200 > From: Marcin Nawrocki > Subject: [Bro] Real-time reporting from multiple sensors to multiple > analysis points > To: bro at bro.org > Message-ID: <230078a0-adf3-61d1-52f7-ddd64df0f54a at fu-berlin.de> > Content-Type: text/plain; charset=utf-8; format=flowed > > Dear bro mailing list, > > > I have a question regarding the configuration of bro and its real-time > reporting features. > > Right now, I have two sensors (s1, s2), each running one bro node with > log files rotating every hour. After the rotation, I send the files from > each sensor to an analysis point (a1) via scp and perform my analysis steps. > > My requirements changed now: I want to know what happens on the sensors > in almost real-time. How do I send reports from (s1,s2) with a max. > delay of 10 seconds to another analysis point (a2)? The reports can > still reach (a1) every hour to keep the load low. My intuition tells me, > that a very low rotation interval and scp are not the best practice here. > > > Regards, > > Marcin Nawrocki > > > > > ------------------------------ > > _______________________________________________ > Bro mailing list > Bro at bro.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > End of Bro Digest, Vol 135, Issue 3 > *********************************** From jazoff at illinois.edu Mon Jul 3 12:32:16 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 3 Jul 2017 19:32:16 +0000 Subject: [Bro] Real-time reporting from multiple sensors to multiple analysis points In-Reply-To: <230078a0-adf3-61d1-52f7-ddd64df0f54a@fu-berlin.de> References: <230078a0-adf3-61d1-52f7-ddd64df0f54a@fu-berlin.de> Message-ID: > On Jul 3, 2017, at 6:51 AM, Marcin Nawrocki wrote: > > Dear bro mailing list, > > > I have a question regarding the configuration of bro and its real-time > reporting features. > > Right now, I have two sensors (s1, s2), each running one bro node with > log files rotating every hour. After the rotation, I send the files from > each sensor to an analysis point (a1) via scp and perform my analysis steps. > > My requirements changed now: I want to know what happens on the sensors > in almost real-time. How do I send reports from (s1,s2) with a max. > delay of 10 seconds to another analysis point (a2)? The reports can > still reach (a1) every hour to keep the load low. My intuition tells me, > that a very low rotation interval and scp are not the best practice here. Based on your requirements you probably want to use something like the bro kafka log writer, or a process running on each system like logstash that can forward logs. -- - Justin Azoff From vladg at illinois.edu Mon Jul 3 12:44:49 2017 From: vladg at illinois.edu (Vlad Grigorescu) Date: Mon, 03 Jul 2017 14:44:49 -0500 Subject: [Bro] Digging through Source Code In-Reply-To: <201706202044.v5KKi90L025920@vladg.net> References: <201706202044.v5KKi90L025920@vladg.net> Message-ID: "Weasel, Gary W Jr CIV DISA RE (US)" writes: > case 8: > if ( element->data()->etype()->data()->size() ) > rv->Assign(11, proc_cipher_list(element->data()->etype())); > > Following the breadcrumb trail in the if statement here... > > element is type KRB_REQ_Arg (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-protocol.pac) > -> data is type KRB_REQ_Arg_Data (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/krb/krb-protocol.pac) > -> etype is type Array (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/asn1/asn1.pac) > -> data is type ASN1Encoding (defined - https://github.com/bro/bro/blob/master/src/analyzer/protocol/asn1/asn1.pac) > -> size is type ? > > Following this line of thought, I'm a little confused by what "size()" > is supposed to mean here, since it's not an attribute. I can infer > that it's simply returning the size of the record, but I don't have > any information as to how or where that would be defined. I've also > tried looking through the source of BinPAC > (https://www.bro.org/sphinx/components/binpac/README.html) but have > come up empty so far. I'll add on to what Johanna said. Many attributes in Kerberos are technically optional (though the Microsoft and MIT implementations don't necessarily treat them that way). Other attributes aren't optional, but we check to make sure that they were actually seen before accessing the underlying data. The if statement is just checking to see if that attribute was in the request packet (even though etype isn't optional according to the RFC). > I have a sample of kerberos pcap that populates the msg$pa_data$encryption_type vector (from event krb_tgs_request), so I know that the aforementioned if statement is returning true - - but the other two vectors "host_addrs" and "additional"tickets" (that from documentation seem to imply they're parallel with the encryption_type vector) come up as . They're not really parallel. What the RFC has is: > etype [8] SEQUENCE OF Int32 -- EncryptionType > -- in preference order --, > addresses [9] HostAddresses OPTIONAL, > enc-authorization-data [10] EncryptedData OPTIONAL > -- AuthorizationData --, > additional-tickets [11] SEQUENCE OF Ticket OPTIONAL > -- NOTE: not empty addresses (called host_addrs by Bro) and additional-tickets are listed as optional, and likely won't be present in your PCAP (outside of the padata, at least). > This made me question that maybe there was something wrong with the > code that was causing it to miss the host_addr and ticket data, I > clearly find this data in my pcap sample under padata. This is my > current theory anyway, and wanted to see if I'm making a bad > assumption somewhere or if someone can shed light on what's going on > here. The padata is in a separate field (pa_data). The overall structure is that first you have the request itself, with the padata: > KDC-REQ ::= SEQUENCE { > -- NOTE: first tag is [1], not [0] > pvno [1] INTEGER (5) , > msg-type [2] INTEGER (10 -- AS -- | 12 -- TGS --), > padata [3] SEQUENCE OF PA-DATA OPTIONAL > -- NOTE: not empty --, > req-body [4] KDC-REQ-BODY > } req-body is then parsed as: > KDC-REQ-BODY ::= SEQUENCE { > kdc-options [0] KDCOptions, > cname [1] PrincipalName OPTIONAL > -- Used only in AS-REQ --, > realm [2] Realm > -- Server's realm > -- Also client's in AS-REQ --, > sname [3] PrincipalName OPTIONAL, > from [4] KerberosTime OPTIONAL, > till [5] KerberosTime, > rtime [6] KerberosTime OPTIONAL, > nonce [7] UInt32, > etype [8] SEQUENCE OF Int32 -- EncryptionType > -- in preference order --, > addresses [9] HostAddresses OPTIONAL, > enc-authorization-data [10] EncryptedData OPTIONAL > -- AuthorizationData --, > additional-tickets [11] SEQUENCE OF Ticket OPTIONAL > -- NOTE: not empty > } So it's possible that you're seeing these fields in padata, but not in the req-body. Hope that helps. I forgot how complex this all was. --Vlad -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 800 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170703/63b05ffa/attachment.bin From oelnaggar04 at gmail.com Mon Jul 3 22:29:52 2017 From: oelnaggar04 at gmail.com (Osama Elnaggar) Date: Tue, 4 Jul 2017 01:29:52 -0400 Subject: [Bro] Real-time reporting from multiple sensors to multiple analysis points In-Reply-To: References: <230078a0-adf3-61d1-52f7-ddd64df0f54a@fu-berlin.de> Message-ID: You can also have a look at lsyncd (https://github.com/axkibe/lsyncd) which is a synching daemon that uses rsync in the background. By default, lsyncd triggers copying when the file is closed but you can change this behavior by modifying the inotifyMode option - https://axkibe.github.io/lsyncd/manual/config/file/ -- Osama Elnaggar On July 4, 2017 at 5:35:34 AM, Azoff, Justin S (jazoff at illinois.edu) wrote: > On Jul 3, 2017, at 6:51 AM, Marcin Nawrocki wrote: > > Dear bro mailing list, > > > I have a question regarding the configuration of bro and its real-time > reporting features. > > Right now, I have two sensors (s1, s2), each running one bro node with > log files rotating every hour. After the rotation, I send the files from > each sensor to an analysis point (a1) via scp and perform my analysis steps. > > My requirements changed now: I want to know what happens on the sensors > in almost real-time. How do I send reports from (s1,s2) with a max. > delay of 10 seconds to another analysis point (a2)? The reports can > still reach (a1) every hour to keep the load low. My intuition tells me, > that a very low rotation interval and scp are not the best practice here. Based on your requirements you probably want to use something like the bro kafka log writer, or a process running on each system like logstash that can forward logs. -- - Justin Azoff _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170704/dc660522/attachment-0001.html From jlay at slave-tothe-box.net Fri Jul 7 13:11:39 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 07 Jul 2017 14:11:39 -0600 Subject: [Bro] Finer detail on mime types Message-ID: <0023e3a347a9896e4b6ac378c3510941@localhost> So in looking at xlsm/docm files I noticed this...where bro says: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet but the pcap says: application/vnd.ms-excel.sheet.macroenabled.12 Is there a way to fine tune this in bro? Identifying files flying around with macros would be wonderful...thank you. From darkheaven1983 at gmail.com Fri Jul 7 23:07:05 2017 From: darkheaven1983 at gmail.com (duhang) Date: Sat, 8 Jul 2017 14:07:05 +0800 Subject: [Bro] FTP with PF_RING Message-ID: Hello, I'm using PF_RING as LB for bro. 4-tuple works pretty well for me before I start to work with FTP protocol. I found that the command channel and data channel are balanced to different bro workers which cause ftp data cannot be handled properly. It can be resolved by using 2-tuple as pfring cluster type. However, using 2-tuple cause the load to different workers is not even especially for the case that one IP has a peak time to the same dest ip. Is there a better solution to cope with this? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170708/1f9e84cf/attachment.html From mydevmail at gmx.de Mon Jul 10 05:23:50 2017 From: mydevmail at gmx.de (Dominique Sellehr) Date: Mon, 10 Jul 2017 14:23:50 +0200 Subject: [Bro] Bro logging connections after specific daytime In-Reply-To: <530881CC-29DD-4001-B19D-1F60BE3DC43F@icir.org> References: <530881CC-29DD-4001-B19D-1F60BE3DC43F@icir.org> Message-ID: An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170710/40a5e978/attachment.html From mydevmail at gmx.de Mon Jul 10 05:25:35 2017 From: mydevmail at gmx.de (Dominique Sellehr) Date: Mon, 10 Jul 2017 14:25:35 +0200 Subject: [Bro] Connections and Crossed_Threshold-Event Message-ID: Hello Bro Community, ? i wrote a script which is able to determine the amount of data transfered by a given connection. If the datatransfer crosses a set limit, a crossed_threshold event is triggered. The connection which crossed the threshold is logged into a log file. Now, if i set a limit of 150.000 Bytes for example, the log file holds connections with 75.000 or 100.000 Bytes of transfered data. This looks a bit confusing because the connections crossed the 150.000 Byte threshold but the log file entry states a different, lower number. May someone please explain why this occurs and how to fix/change this behaviour/problem? From jdopheid at illinois.edu Mon Jul 10 08:26:01 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Mon, 10 Jul 2017 15:26:01 +0000 Subject: [Bro] =?utf-8?q?BroCon_=E2=80=9917=3A_Announcing_=28most_of=29_sp?= =?utf-8?q?eakers?= Message-ID: <8B233C36-0D8B-4FBF-87AA-1FFF8B49061B@illinois.edu> Hello Bro Community, We?re in the process of wrapping up our speaker list and I wanted to share the list of presentations we have confirmed so far. * Brian Wylie, Kitware, ?Data Analysis, Machine Learning, Bro, and You!? * Mark Krenz, CTSC and formerly SWAMP, ?Using awk to analyze Bro logs? * Adam Pumphrey, Bricata, ?A Bro Primer? * Jan Grashoefer, researcher at KIT, ?The Intelligence Framework Update? * Joseph Mosby, Deloitte & Touche, ?Spark and Bro: When Bro-Cut Won't Cut It? * Aashish Sharma, Lawrence Berkeley National Lab, ?Detecting Credential Spear-phishing Attacks at LBNL? * Fatema Bannat Wala, University of Delaware, ?Unconstrained Endpoint Security System (UEPtSS)? * Benjamin Klimkowski, former Asst. Prof. United States Military Academy, ?Using Bro to Hunt Persistent Threats? We?ll announce more speakers and topics as we receive confirmations. Don?t forget to register: https://brocon2017.eventbrite.com/ Early bird registration expires on July 30th. See you in September, The Bro Team ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From jdopheid at illinois.edu Mon Jul 10 09:12:39 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Mon, 10 Jul 2017 16:12:39 +0000 Subject: [Bro] =?utf-8?q?BroCon_=E2=80=9917=3A_Announcing_=28most_of=29_sp?= =?utf-8?q?eakers?= Message-ID: <52412479-70BB-4703-B70F-D0D77FFF7351@illinois.edu> And Alan Commike, Reservoir Labs, ?Running Bro in the Cloud at Scale? ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign On 7/10/17, 10:26 AM, "bro-bounces at bro.org on behalf of Dopheide, Jeannette M" wrote: Hello Bro Community, We?re in the process of wrapping up our speaker list and I wanted to share the list of presentations we have confirmed so far. * Brian Wylie, Kitware, ?Data Analysis, Machine Learning, Bro, and You!? * Mark Krenz, CTSC and formerly SWAMP, ?Using awk to analyze Bro logs? * Adam Pumphrey, Bricata, ?A Bro Primer? * Jan Grashoefer, researcher at KIT, ?The Intelligence Framework Update? * Joseph Mosby, Deloitte & Touche, ?Spark and Bro: When Bro-Cut Won't Cut It? * Aashish Sharma, Lawrence Berkeley National Lab, ?Detecting Credential Spear-phishing Attacks at LBNL? * Fatema Bannat Wala, University of Delaware, ?Unconstrained Endpoint Security System (UEPtSS)? * Benjamin Klimkowski, former Asst. Prof. United States Military Academy, ?Using Bro to Hunt Persistent Threats? We?ll announce more speakers and topics as we receive confirmations. Don?t forget to register: https://brocon2017.eventbrite.com/ Early bird registration expires on July 30th. See you in September, The Bro Team ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at corelight.com Mon Jul 10 13:09:07 2017 From: seth at corelight.com (Seth Hall) Date: Mon, 10 Jul 2017 16:09:07 -0400 Subject: [Bro] Finer detail on mime types In-Reply-To: <0023e3a347a9896e4b6ac378c3510941@localhost> References: <0023e3a347a9896e4b6ac378c3510941@localhost> Message-ID: That is a bit of an overloaded mime-type I'm afraid. We did build the files framework in Bro so that it could be extended to provide quite a bit of extra information when the file is "sniffed". The primary problem that we'd have with providing that information at the moment is lack of a way to analyze excel files. .Seth On Fri, Jul 7, 2017 at 4:11 PM, James Lay wrote: > So in looking at xlsm/docm files I noticed this...where bro says: > > application/vnd.openxmlformats-officedocument.spreadsheetml.sheet > > but the pcap says: > > application/vnd.ms-excel.sheet.macroenabled.12 > > Is there a way to fine tune this in bro? Identifying files flying > around with macros would be wonderful...thank you. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From fatema.bannatwala at gmail.com Mon Jul 10 13:22:14 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Mon, 10 Jul 2017 16:22:14 -0400 Subject: [Bro] Connections and Crossed_Threshold-Event Message-ID: Just out of curiosity, have you tried playing with the https://github.com/theflakes/bro-large_uploads/blob/master/large_uploads.bro script? Works well, and it's pretty good at producing different kinds of notices for large uploads to the internet, also it can easily be tweaked for detecting the large downloads as well, if that's included in your use-case. Also, you can change the limits of total byte transfer to whatever you would like. Thanks, Fatema. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170710/ea290807/attachment.html From jlay at slave-tothe-box.net Mon Jul 10 15:04:13 2017 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 10 Jul 2017 16:04:13 -0600 Subject: [Bro] Finer detail on mime types In-Reply-To: References: <0023e3a347a9896e4b6ac378c3510941@localhost> Message-ID: Understood and thanks Seth. At this point an analysis of the Macro enabled Excel/Word file is secondary to bro just being able to read and report the "macroeanbled" mime type. Lemme see if I can get protosigs to do something exciting....thank you! James On 2017-07-10 14:09, Seth Hall wrote: > That is a bit of an overloaded mime-type I'm afraid. We did build the > files framework in Bro so that it could be extended to provide quite a > bit of extra information when the file is "sniffed". The primary > problem that we'd have with providing that information at the moment > is lack of a way to analyze excel files. > > .Seth > > On Fri, Jul 7, 2017 at 4:11 PM, James Lay > wrote: >> So in looking at xlsm/docm files I noticed this...where bro says: >> >> application/vnd.openxmlformats-officedocument.spreadsheetml.sheet >> >> but the pcap says: >> >> application/vnd.ms-excel.sheet.macroenabled.12 >> >> Is there a way to fine tune this in bro? Identifying files flying >> around with macros would be wonderful...thank you. >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From christian at corelight.com Mon Jul 10 23:33:00 2017 From: christian at corelight.com (Christian Kreibich) Date: Mon, 10 Jul 2017 23:33:00 -0700 Subject: [Bro] Finer detail on mime types In-Reply-To: References: <0023e3a347a9896e4b6ac378c3510941@localhost> Message-ID: On 07/10/2017 01:09 PM, Seth Hall wrote: > That is a bit of an overloaded mime-type I'm afraid. We did build the > files framework in Bro so that it could be extended to provide quite a > bit of extra information when the file is "sniffed". The primary > problem that we'd have with providing that information at the moment > is lack of a way to analyze excel files. Once you know you're dealing with an OOXML archive, in my experience the following works well: take the presence of a vbaproject.bin file in the archive as a prerequisite for macro-enabledness, then leverage a .docm/.pptm/.xlsm filename suffix to distinguish application, and fall back to Word for others. I'd be interested to hear what approaches others have used. Thanks, -C. From christian at corelight.com Tue Jul 11 00:25:44 2017 From: christian at corelight.com (Christian Kreibich) Date: Tue, 11 Jul 2017 00:25:44 -0700 Subject: [Bro] Finer detail on mime types In-Reply-To: References: <0023e3a347a9896e4b6ac378c3510941@localhost> Message-ID: <8a0c6b33-87bb-89c7-051c-8a28dbad7974@corelight.com> On 07/10/2017 11:33 PM, Christian Kreibich wrote: > Once you know you're dealing with an OOXML archive, in my experience the > following works well: take the presence of a vbaproject.bin file in the > archive as a prerequisite for macro-enabledness, then leverage a > .docm/.pptm/.xlsm filename suffix to distinguish application, and fall > back to Word for others. I forgot: the directory layout in such archives is also telling -- look for word/, xl/, ppt/ ... It's been a while. :) Cheers, -C. From shirisha.kasarla96 at gmail.com Tue Jul 11 02:52:41 2017 From: shirisha.kasarla96 at gmail.com (Shirisha Kasarla) Date: Tue, 11 Jul 2017 15:22:41 +0530 Subject: [Bro] X509 Plugin does not support OpenSSL 1.1 Message-ID: can anyone help with this issue openssl should be downgraded to 1.0.0 for compiling bro openssl 1.1 is not compatible with bro compilation OS- ubuntu 16.04 The following error is occurred In file included from /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, from /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: forward declaration of ?EVP_PKEY {aka struct evp_pkey_st}? typedef struct evp_pkey_st EVP_PKEY; ^ /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:479:25: error: invalid use of incomplete type ?EVP_PKEY {aka struct evp_pkey_st}? return BN_num_bits(key->pkey.rsa->n); ^ In file included from /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, from /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: forward declaration of ?EVP_PKEY {aka struct evp_pkey_st}? typedef struct evp_pkey_st EVP_PKEY; ^ /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:482:25: error: invalid use of incomplete type ?EVP_PKEY {aka struct evp_pkey_st}? return BN_num_bits(key->pkey.dsa->p); ^ In file included from /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, from /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: forward declaration of ?EVP_PKEY {aka struct evp_pkey_st}? typedef struct evp_pkey_st EVP_PKEY; ^ /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:492:48: error: invalid use of incomplete type ?EVP_PKEY {aka struct evp_pkey_st}? const EC_GROUP *group = EC_KEY_get0_group(key->pkey.ec); ^ In file included from /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, from /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: forward declaration of ?EVP_PKEY {aka struct evp_pkey_st}? typedef struct evp_pkey_st EVP_PKEY; ^ src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/build.make:119: recipe for target 'src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/X509.cc.o' failed make[3]: *** [src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/X509.cc.o] Error 1 make[3]: Leaving directory '/home/shrisha/bro-2.5/build' CMakeFiles/Makefile2:9455: recipe for target 'src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/all' failed make[2]: *** [src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/all] Error 2 make[2]: Leaving directory '/home/shrisha/bro-2.5/build' Makefile:138: recipe for target 'all' failed make[1]: *** [all] Error 2 make[1]: Leaving directory '/home/shrisha/bro-2.5/build' Makefile:15: recipe for target 'all' failed make: *** [all] Error 2 shrisha at cdesktop:~/bro-2.5$ with Regards kasarla shirisha From mike at swedishmike.org Tue Jul 11 03:48:24 2017 From: mike at swedishmike.org (Mike Eriksson) Date: Tue, 11 Jul 2017 10:48:24 +0000 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs Message-ID: Hi all, We're currently working on deploying Bro sensors to various offices and I've come to realise that the Bro logs are quite 'expensive' when it comes to Splunk licenses. To say the least. We have discussed various solutions but most of them fall down on us losing the ability to correlate events unless we shift all the logs in to Splunk. At the moment we're running it pretty much 'out of the box' so we can save some GB's per day to turn of certain scripts, but it will probably not be enough. Someone mentioned that turning on JSON logging instead of the standard logging on Bro could save considerable amounts of space on your SIEM. Have any of you guys tested this and can you back that statement up? I was hoping that someone else had encountered this before and had come up with some solution(s) to this issue? Thanks in advance, Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170711/a7427f9b/attachment-0001.html From bro at pingtrip.com Tue Jul 11 07:57:06 2017 From: bro at pingtrip.com (Dave Crawford) Date: Tue, 11 Jul 2017 10:57:06 -0400 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: References: Message-ID: <7B38E6FD-D769-4A85-89D7-A4BA056E39A8@pingtrip.com> The JSON logs will always be larger than the default tab delimited. With JSON every log event includes the "column" names versus a set of headers in he delimited format. > On Jul 11, 2017, at 6:48 AM, Mike Eriksson wrote: > > Hi all, > > We're currently working on deploying Bro sensors to various offices and I've come to realise that the Bro logs are quite 'expensive' when it comes to Splunk licenses. To say the least. > > We have discussed various solutions but most of them fall down on us losing the ability to correlate events unless we shift all the logs in to Splunk. > > At the moment we're running it pretty much 'out of the box' so we can save some GB's per day to turn of certain scripts, but it will probably not be enough. > > Someone mentioned that turning on JSON logging instead of the standard logging on Bro could save considerable amounts of space on your SIEM. Have any of you guys tested this and can you back that statement up? > > I was hoping that someone else had encountered this before and had come up with some solution(s) to this issue? > > Thanks in advance, Mike > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From slagell at illinois.edu Tue Jul 11 08:03:26 2017 From: slagell at illinois.edu (Slagell, Adam J) Date: Tue, 11 Jul 2017 15:03:26 +0000 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: References: Message-ID: <1ACB1F91-60F2-487B-A64E-BA807143C508@illinois.edu> Well if you are collecting net flows and conn.logs, you could get rid of one. If you are recapturing syslogs with Bro and sending them to Splunk, you could trim duplication there. If you are finding a ton of certificate information in your bro logs, you might realize some cost savings there. But I don?t have much advice beyond don?t send the same info twice and look for large amounts of data that you don?t really use in Splunk, like maybe certificates. > On Jul 11, 2017, at 5:48 AM, Mike Eriksson wrote: > > Hi all, > > We're currently working on deploying Bro sensors to various offices and I've come to realise that the Bro logs are quite 'expensive' when it comes to Splunk licenses. To say the least. > > We have discussed various solutions but most of them fall down on us losing the ability to correlate events unless we shift all the logs in to Splunk. > > At the moment we're running it pretty much 'out of the box' so we can save some GB's per day to turn of certain scripts, but it will probably not be enough. > > Someone mentioned that turning on JSON logging instead of the standard logging on Bro could save considerable amounts of space on your SIEM. Have any of you guys tested this and can you back that statement up? > > I was hoping that someone else had encountered this before and had come up with some solution(s) to this issue? > > Thanks in advance, Mike > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From hosom at battelle.org Tue Jul 11 10:41:32 2017 From: hosom at battelle.org (Hosom, Stephen M) Date: Tue, 11 Jul 2017 17:41:32 +0000 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: <1ACB1F91-60F2-487B-A64E-BA807143C508@illinois.edu> References: <1ACB1F91-60F2-487B-A64E-BA807143C508@illinois.edu> Message-ID: Some people choose to implement Bro log filters that can result in a significant reduction in log volume. For example, if you filter all S0 connections originating from outside of your organization (and you also happen to listen outside of a firewall) this could reduce a substantial amount of log volume. -----Original Message----- From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Slagell, Adam J Sent: Tuesday, July 11, 2017 11:03 AM To: Mike Eriksson Cc: bro at bro.org Subject: Re: [Bro] Get the license usage down in Splunk when indexing Bro logs Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. Well if you are collecting net flows and conn.logs, you could get rid of one. If you are recapturing syslogs with Bro and sending them to Splunk, you could trim duplication there. If you are finding a ton of certificate information in your bro logs, you might realize some cost savings there. But I don?t have much advice beyond don?t send the same info twice and look for large amounts of data that you don?t really use in Splunk, like maybe certificates. > On Jul 11, 2017, at 5:48 AM, Mike Eriksson wrote: > > Hi all, > > We're currently working on deploying Bro sensors to various offices and I've come to realise that the Bro logs are quite 'expensive' when it comes to Splunk licenses. To say the least. > > We have discussed various solutions but most of them fall down on us losing the ability to correlate events unless we shift all the logs in to Splunk. > > At the moment we're running it pretty much 'out of the box' so we can save some GB's per day to turn of certain scripts, but it will probably not be enough. > > Someone mentioned that turning on JSON logging instead of the standard logging on Bro could save considerable amounts of space on your SIEM. Have any of you guys tested this and can you back that statement up? > > I was hoping that someone else had encountered this before and had come up with some solution(s) to this issue? > > Thanks in advance, Mike > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From zeolla at gmail.com Tue Jul 11 11:06:48 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Tue, 11 Jul 2017 18:06:48 +0000 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: References: <1ACB1F91-60F2-487B-A64E-BA807143C508@illinois.edu> Message-ID: In order to reduce logging load for some of our logging plugins we've applied filters that do things like drop S0 connections from conn, only send traffic to/from the Internet, or only send traffic to/from sensitive zones. I have an example of one of our configs here (the example also filters out IPv6 in a way that works for pre-2.5 - now there are is_v{4,6}_subnet() functions to handle this). Jon On Tue, Jul 11, 2017 at 1:43 PM Hosom, Stephen M wrote: > Some people choose to implement Bro log filters that can result in a > significant reduction in log volume. For example, if you filter all S0 > connections originating from outside of your organization (and you also > happen to listen outside of a firewall) this could reduce a substantial > amount of log volume. > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of > Slagell, Adam J > Sent: Tuesday, July 11, 2017 11:03 AM > To: Mike Eriksson > Cc: bro at bro.org > Subject: Re: [Bro] Get the license usage down in Splunk when indexing Bro > logs > > Message received from outside the Battelle network. Carefully examine it > before you open any links or attachments. > > Well if you are collecting net flows and conn.logs, you could get rid of > one. If you are recapturing syslogs with Bro and sending them to Splunk, > you could trim duplication there. If you are finding a ton of certificate > information in your bro logs, you might realize some cost savings there. > But I don?t have much advice beyond don?t send the same info twice and look > for large amounts of data that you don?t really use in Splunk, like maybe > certificates. > > > > On Jul 11, 2017, at 5:48 AM, Mike Eriksson wrote: > > > > Hi all, > > > > We're currently working on deploying Bro sensors to various offices and > I've come to realise that the Bro logs are quite 'expensive' when it comes > to Splunk licenses. To say the least. > > > > We have discussed various solutions but most of them fall down on us > losing the ability to correlate events unless we shift all the logs in to > Splunk. > > > > At the moment we're running it pretty much 'out of the box' so we can > save some GB's per day to turn of certain scripts, but it will probably not > be enough. > > > > Someone mentioned that turning on JSON logging instead of the standard > logging on Bro could save considerable amounts of space on your SIEM. Have > any of you guys tested this and can you back that statement up? > > > > I was hoping that someone else had encountered this before and had come > up with some solution(s) to this issue? > > > > Thanks in advance, Mike > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170711/10d443dc/attachment.html From promero at cenic.org Tue Jul 11 11:25:13 2017 From: promero at cenic.org (Philip Romero) Date: Tue, 11 Jul 2017 11:25:13 -0700 Subject: [Bro] Bug Report - Software Framework - Flash Player Version Parsing Message-ID: All, I was looking into updating my vulnerability alert configuration and noticed that the software framework may be incorrectly parsing the software version for Adobe Flash Player. Please see the below example. The full string details show the correct version (26.0.0.137), but the parsed versions that I believe the vulnerability scripts read for major and minor versions looks to be grabbing the "20" from that portion of the syntax in the full string. This email is information in case anyone actively updates the software framework. I'll see if I can try to work it a bit on my local development system as time permits. Thanks. Example Log: 1499796686.729596 137.164.83.xxx - HTTP::BROWSER Flash% 20 - - - Player/26 Flash%20Player/26.0.0.137 CFNetwork/811.5.4 Darwin/16.6.0 (x86_64) -- Philip Romero, CISSP, CISA Sr. Information Security Analyst CENIC promero at cenic.org Phone: (714) 220-3430 Mobile: (562) 237-9290 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170711/9a21d955/attachment-0001.html From buysse at umn.edu Tue Jul 11 11:55:13 2017 From: buysse at umn.edu (Joshua Buysse) Date: Tue, 11 Jul 2017 13:55:13 -0500 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: <7B38E6FD-D769-4A85-89D7-A4BA056E39A8@pingtrip.com> References: <7B38E6FD-D769-4A85-89D7-A4BA056E39A8@pingtrip.com> Message-ID: I think I know where Mike's misunderstanding comes from - the JSON logs are larger original size (for license volume), but will use less space on indexer disk than the default TSV because the extractions are search-time instead of index-time. On Tue, Jul 11, 2017 at 9:57 AM, Dave Crawford wrote: > > > The JSON logs will always be larger than the default tab delimited. With > JSON every log event includes the "column" names versus a set of headers in > he delimited format. > > > On Jul 11, 2017, at 6:48 AM, Mike Eriksson wrote: > > > > Hi all, > > > > We're currently working on deploying Bro sensors to various offices and > I've come to realise that the Bro logs are quite 'expensive' when it comes > to Splunk licenses. To say the least. > > > > We have discussed various solutions but most of them fall down on us > losing the ability to correlate events unless we shift all the logs in to > Splunk. > > > > At the moment we're running it pretty much 'out of the box' so we can > save some GB's per day to turn of certain scripts, but it will probably not > be enough. > > > > Someone mentioned that turning on JSON logging instead of the standard > logging on Bro could save considerable amounts of space on your SIEM. Have > any of you guys tested this and can you back that statement up? > > > > I was hoping that someone else had encountered this before and had come > up with some solution(s) to this issue? > > > > Thanks in advance, Mike > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Joshua Buysse University of Minnesota - University Information Security "On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question." - Charles Babbage -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170711/b813770f/attachment.html From pyrodie18 at gmail.com Tue Jul 11 12:49:44 2017 From: pyrodie18 at gmail.com (Troy Ward) Date: Tue, 11 Jul 2017 15:49:44 -0400 Subject: [Bro] Get the license usage down in Splunk when indexing Bro Message-ID: Mike, How much data are we talking about? Have you done the analysis to see what logs are actually causing you problems? I am currently ingesting somewhere in the neighborhood of 50GB of bro logs a day but at one point it was a lot more. After doing some digging we found out that our sensor was saturated and dropping a ton of packets which had the bro wierd log and conn log going through the roof because it some connections where appearing 3 or 4 times due to it thinking they were different connections. Troy > > Today's Topics: > > 1. Re: Get the license usage down in Splunk when indexing Bro > logs (Joshua Buysse) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 11 Jul 2017 13:55:13 -0500 > From: Joshua Buysse > Subject: Re: [Bro] Get the license usage down in Splunk when indexing > Bro logs > To: Dave Crawford > Cc: bro at bro.org > Message-ID: > gmail.com> > Content-Type: text/plain; charset="utf-8" > > I think I know where Mike's misunderstanding comes from - the JSON logs are > larger original size (for license volume), but will use less space on > indexer disk than the default TSV because the extractions are search-time > instead of index-time. > > On Tue, Jul 11, 2017 at 9:57 AM, Dave Crawford wrote: > > > > > > > The JSON logs will always be larger than the default tab delimited. With > > JSON every log event includes the "column" names versus a set of headers > in > > he delimited format. > > > > > On Jul 11, 2017, at 6:48 AM, Mike Eriksson > wrote: > > > > > > Hi all, > > > > > > We're currently working on deploying Bro sensors to various offices and > > I've come to realise that the Bro logs are quite 'expensive' when it > comes > > to Splunk licenses. To say the least. > > > > > > We have discussed various solutions but most of them fall down on us > > losing the ability to correlate events unless we shift all the logs in to > > Splunk. > > > > > > At the moment we're running it pretty much 'out of the box' so we can > > save some GB's per day to turn of certain scripts, but it will probably > not > > be enough. > > > > > > Someone mentioned that turning on JSON logging instead of the standard > > logging on Bro could save considerable amounts of space on your SIEM. > Have > > any of you guys tested this and can you back that statement up? > > > > > > I was hoping that someone else had encountered this before and had come > > up with some solution(s) to this issue? > > > > > > Thanks in advance, Mike > > > > > > > > > _______________________________________________ > > > Bro mailing list > > > bro at bro-ids.org > > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > > -- > Joshua Buysse > University of Minnesota - University Information Security > > "On two occasions I have been asked, 'Pray, Mr. Babbage, if you > put into the machine wrong figures, will the right answers come > out?' I am not able rightly to apprehend the kind of confusion of > ideas that could provoke such a question." > - Charles Babbage > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/ > 20170711/b813770f/attachment-0001.html > > ------------------------------ > > _______________________________________________ > Bro mailing list > Bro at bro.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > End of Bro Digest, Vol 135, Issue 9 > *********************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170711/d38193af/attachment.html From oelnaggar04 at gmail.com Tue Jul 11 14:55:50 2017 From: oelnaggar04 at gmail.com (Osama Elnaggar) Date: Tue, 11 Jul 2017 14:55:50 -0700 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: References: <7B38E6FD-D769-4A85-89D7-A4BA056E39A8@pingtrip.com> Message-ID: To build on some of the suggestions, you may also want to tier your logging. You can send high value logs which can kick off an investigation to Splunk while sending less valuable logs which will help in investigations but not start them to something like ELK or ELSA / ODE. This will add a little overhead to your operations and involves more moving parts but can significantly reduce your licensing costs. -- Osama Elnaggar On July 12, 2017 at 4:57:21 AM, Joshua Buysse (buysse at umn.edu) wrote: I think I know where Mike's misunderstanding comes from - the JSON logs are larger original size (for license volume), but will use less space on indexer disk than the default TSV because the extractions are search-time instead of index-time. On Tue, Jul 11, 2017 at 9:57 AM, Dave Crawford wrote: > > > The JSON logs will always be larger than the default tab delimited. With > JSON every log event includes the "column" names versus a set of headers in > he delimited format. > > > On Jul 11, 2017, at 6:48 AM, Mike Eriksson wrote: > > > > Hi all, > > > > We're currently working on deploying Bro sensors to various offices and > I've come to realise that the Bro logs are quite 'expensive' when it comes > to Splunk licenses. To say the least. > > > > We have discussed various solutions but most of them fall down on us > losing the ability to correlate events unless we shift all the logs in to > Splunk. > > > > At the moment we're running it pretty much 'out of the box' so we can > save some GB's per day to turn of certain scripts, but it will probably not > be enough. > > > > Someone mentioned that turning on JSON logging instead of the standard > logging on Bro could save considerable amounts of space on your SIEM. Have > any of you guys tested this and can you back that statement up? > > > > I was hoping that someone else had encountered this before and had come > up with some solution(s) to this issue? > > > > Thanks in advance, Mike > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Joshua Buysse University of Minnesota - University Information Security "On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question." - Charles Babbage _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170711/93b5eeef/attachment.html From mike at swedishmike.org Tue Jul 11 22:07:20 2017 From: mike at swedishmike.org (Mike Eriksson) Date: Wed, 12 Jul 2017 05:07:20 +0000 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: References: <1ACB1F91-60F2-487B-A64E-BA807143C508@illinois.edu> Message-ID: Jon, Many thanks for your reply - that looks really useful. Many thanks for your examples, I'll have a look at them. Cheers, Mike On Tue, Jul 11, 2017 at 7:07 PM Zeolla at GMail.com wrote: > In order to reduce logging load for some of our logging plugins we've > applied filters that do things like drop S0 connections from conn, only > send traffic to/from the Internet, or only send traffic to/from sensitive > zones. I have an example of one of our configs here > > (the example also filters out IPv6 in a way that works for pre-2.5 - now > there are is_v{4,6}_subnet() functions to handle this). > > Jon > > On Tue, Jul 11, 2017 at 1:43 PM Hosom, Stephen M > wrote: > >> Some people choose to implement Bro log filters that can result in a >> significant reduction in log volume. For example, if you filter all S0 >> connections originating from outside of your organization (and you also >> happen to listen outside of a firewall) this could reduce a substantial >> amount of log volume. >> >> -----Original Message----- >> From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of >> Slagell, Adam J >> Sent: Tuesday, July 11, 2017 11:03 AM >> To: Mike Eriksson >> Cc: bro at bro.org >> Subject: Re: [Bro] Get the license usage down in Splunk when indexing Bro >> logs >> >> Message received from outside the Battelle network. Carefully examine it >> before you open any links or attachments. >> >> Well if you are collecting net flows and conn.logs, you could get rid of >> one. If you are recapturing syslogs with Bro and sending them to Splunk, >> you could trim duplication there. If you are finding a ton of certificate >> information in your bro logs, you might realize some cost savings there. >> But I don?t have much advice beyond don?t send the same info twice and look >> for large amounts of data that you don?t really use in Splunk, like maybe >> certificates. >> >> >> > On Jul 11, 2017, at 5:48 AM, Mike Eriksson >> wrote: >> > >> > Hi all, >> > >> > We're currently working on deploying Bro sensors to various offices and >> I've come to realise that the Bro logs are quite 'expensive' when it comes >> to Splunk licenses. To say the least. >> > >> > We have discussed various solutions but most of them fall down on us >> losing the ability to correlate events unless we shift all the logs in to >> Splunk. >> > >> > At the moment we're running it pretty much 'out of the box' so we can >> save some GB's per day to turn of certain scripts, but it will probably not >> be enough. >> > >> > Someone mentioned that turning on JSON logging instead of the standard >> logging on Bro could save considerable amounts of space on your SIEM. Have >> any of you guys tested this and can you back that statement up? >> > >> > I was hoping that someone else had encountered this before and had come >> up with some solution(s) to this issue? >> > >> > Thanks in advance, Mike >> > >> > >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > > Jon > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/c5e0fb2d/attachment-0001.html From mike at swedishmike.org Tue Jul 11 22:08:20 2017 From: mike at swedishmike.org (Mike Eriksson) Date: Wed, 12 Jul 2017 05:08:20 +0000 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: References: <7B38E6FD-D769-4A85-89D7-A4BA056E39A8@pingtrip.com> Message-ID: Joshua, Many thanks for your reply. That's the one! ;) It will not make much use for our license usage - but will save space on the disk then. Cheers, Mike On Tue, Jul 11, 2017 at 7:55 PM Joshua Buysse wrote: > I think I know where Mike's misunderstanding comes from - the JSON logs > are larger original size (for license volume), but will use less space on > indexer disk than the default TSV because the extractions are search-time > instead of index-time. > > On Tue, Jul 11, 2017 at 9:57 AM, Dave Crawford wrote: > >> >> >> The JSON logs will always be larger than the default tab delimited. With >> JSON every log event includes the "column" names versus a set of headers in >> he delimited format. >> >> > On Jul 11, 2017, at 6:48 AM, Mike Eriksson >> wrote: >> > >> > Hi all, >> > >> > We're currently working on deploying Bro sensors to various offices and >> I've come to realise that the Bro logs are quite 'expensive' when it comes >> to Splunk licenses. To say the least. >> > >> > We have discussed various solutions but most of them fall down on us >> losing the ability to correlate events unless we shift all the logs in to >> Splunk. >> > >> > At the moment we're running it pretty much 'out of the box' so we can >> save some GB's per day to turn of certain scripts, but it will probably not >> be enough. >> > >> > Someone mentioned that turning on JSON logging instead of the standard >> logging on Bro could save considerable amounts of space on your SIEM. Have >> any of you guys tested this and can you back that statement up? >> > >> > I was hoping that someone else had encountered this before and had come >> up with some solution(s) to this issue? >> > >> > Thanks in advance, Mike >> > >> > >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > > > -- > Joshua Buysse > University of Minnesota - University Information Security > > "On two occasions I have been asked, 'Pray, Mr. Babbage, if you > put into the machine wrong figures, will the right answers come > out?' I am not able rightly to apprehend the kind of confusion of > ideas that could provoke such a question." > - Charles Babbage > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/5951c792/attachment.html From mike at swedishmike.org Tue Jul 11 22:09:16 2017 From: mike at swedishmike.org (Mike Eriksson) Date: Wed, 12 Jul 2017 05:09:16 +0000 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: References: <7B38E6FD-D769-4A85-89D7-A4BA056E39A8@pingtrip.com> Message-ID: Osama, I'm starting to lean towards this as being the next step to look into once I've looked at the suggestions from the list. More suggestions are of course very welcome ;-) Cheers, Mike On Tue, Jul 11, 2017 at 11:08 PM Osama Elnaggar wrote: > To build on some of the suggestions, you may also want to tier your > logging. You can send high value logs which can kick off an investigation > to Splunk while sending less valuable logs which will help in > investigations but not start them to something like ELK or ELSA / ODE. > This will add a little overhead to your operations and involves more moving > parts but can significantly reduce your licensing costs. > > -- > Osama Elnaggar > > On July 12, 2017 at 4:57:21 AM, Joshua Buysse (buysse at umn.edu) wrote: > > I think I know where Mike's misunderstanding comes from - the JSON logs > are larger original size (for license volume), but will use less space on > indexer disk than the default TSV because the extractions are search-time > instead of index-time. > > On Tue, Jul 11, 2017 at 9:57 AM, Dave Crawford wrote: > >> >> >> The JSON logs will always be larger than the default tab delimited. With >> JSON every log event includes the "column" names versus a set of headers in >> he delimited format. >> >> > On Jul 11, 2017, at 6:48 AM, Mike Eriksson >> wrote: >> > >> > Hi all, >> > >> > We're currently working on deploying Bro sensors to various offices and >> I've come to realise that the Bro logs are quite 'expensive' when it comes >> to Splunk licenses. To say the least. >> > >> > We have discussed various solutions but most of them fall down on us >> losing the ability to correlate events unless we shift all the logs in to >> Splunk. >> > >> > At the moment we're running it pretty much 'out of the box' so we can >> save some GB's per day to turn of certain scripts, but it will probably not >> be enough. >> > >> > Someone mentioned that turning on JSON logging instead of the standard >> logging on Bro could save considerable amounts of space on your SIEM. Have >> any of you guys tested this and can you back that statement up? >> > >> > I was hoping that someone else had encountered this before and had come >> up with some solution(s) to this issue? >> > >> > Thanks in advance, Mike >> > >> > >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > > > > -- > Joshua Buysse > University of Minnesota - University Information Security > > "On two occasions I have been asked, 'Pray, Mr. Babbage, if you > put into the machine wrong figures, will the right answers come > out?' I am not able rightly to apprehend the kind of confusion of > ideas that could provoke such a question." > - Charles Babbage > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/a5aa4c7d/attachment.html From mike at swedishmike.org Tue Jul 11 22:15:33 2017 From: mike at swedishmike.org (Mike Eriksson) Date: Wed, 12 Jul 2017 05:15:33 +0000 Subject: [Bro] Get the license usage down in Splunk when indexing Bro In-Reply-To: References: Message-ID: Troy, Many thanks for your reply. For this one host we're looking at around 170 - 190GB of data per day according to Splunk. Which feels like a lot, but could also be the 'true' value for a host with our traffic and all logs turned on if you see what I mean? So far we've not looked at duplicates in the logs - and I'll have to check the size of the weird.log when I get in to work - so that's definitely worth looking at. If you don't mind me asking, if we find that we have a similar issue like yours with dropped packets etc - what steps did you go through in order to work on this issue? Spread the load out on more cores? Lessen the amount of scripts run? All of the above, none of the above or something completely different? Cheers, Mike On Tue, Jul 11, 2017 at 8:52 PM Troy Ward wrote: > Mike, > > How much data are we talking about? Have you done the analysis to see > what logs are actually causing you problems? I am currently ingesting > somewhere in the neighborhood of 50GB of bro logs a day but at one point it > was a lot more. After doing some digging we found out that our sensor was > saturated and dropping a ton of packets which had the bro wierd log and > conn log going through the roof because it some connections where appearing > 3 or 4 times due to it thinking they were different connections. > > Troy > > >> >> Today's Topics: >> >> 1. Re: Get the license usage down in Splunk when indexing Bro >> logs (Joshua Buysse) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Tue, 11 Jul 2017 13:55:13 -0500 >> From: Joshua Buysse >> Subject: Re: [Bro] Get the license usage down in Splunk when indexing >> Bro logs >> To: Dave Crawford >> Cc: bro at bro.org >> Message-ID: >> < >> CAGp0rD1qGn8jyPR5eZn1kq0RbEetRuwBABFdPLkOO6zrTek4uw at mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> I think I know where Mike's misunderstanding comes from - the JSON logs >> are >> larger original size (for license volume), but will use less space on >> indexer disk than the default TSV because the extractions are search-time >> instead of index-time. >> >> On Tue, Jul 11, 2017 at 9:57 AM, Dave Crawford wrote: >> >> > >> > >> > The JSON logs will always be larger than the default tab delimited. With >> > JSON every log event includes the "column" names versus a set of >> headers in >> > he delimited format. >> > >> > > On Jul 11, 2017, at 6:48 AM, Mike Eriksson >> wrote: >> > > >> > > Hi all, >> > > >> > > We're currently working on deploying Bro sensors to various offices >> and >> > I've come to realise that the Bro logs are quite 'expensive' when it >> comes >> > to Splunk licenses. To say the least. >> > > >> > > We have discussed various solutions but most of them fall down on us >> > losing the ability to correlate events unless we shift all the logs in >> to >> > Splunk. >> > > >> > > At the moment we're running it pretty much 'out of the box' so we can >> > save some GB's per day to turn of certain scripts, but it will probably >> not >> > be enough. >> > > >> > > Someone mentioned that turning on JSON logging instead of the standard >> > logging on Bro could save considerable amounts of space on your SIEM. >> Have >> > any of you guys tested this and can you back that statement up? >> > > >> > > I was hoping that someone else had encountered this before and had >> come >> > up with some solution(s) to this issue? >> > > >> > > Thanks in advance, Mike >> > > >> > > >> > > _______________________________________________ >> > > Bro mailing list >> > > bro at bro-ids.org >> > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > >> > >> > _______________________________________________ >> > Bro mailing list >> > bro at bro-ids.org >> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > >> >> >> >> -- >> Joshua Buysse >> University of Minnesota - University Information Security >> >> "On two occasions I have been asked, 'Pray, Mr. Babbage, if you >> put into the machine wrong figures, will the right answers come >> out?' I am not able rightly to apprehend the kind of confusion of >> ideas that could provoke such a question." >> - Charles Babbage >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170711/b813770f/attachment-0001.html >> >> ------------------------------ >> >> _______________________________________________ >> Bro mailing list >> Bro at bro.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> >> >> End of Bro Digest, Vol 135, Issue 9 >> *********************************** >> > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/5f9fc882/attachment-0001.html From mike at swedishmike.org Wed Jul 12 02:07:59 2017 From: mike at swedishmike.org (Mike Eriksson) Date: Wed, 12 Jul 2017 09:07:59 +0000 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: References: <1ACB1F91-60F2-487B-A64E-BA807143C508@illinois.edu> Message-ID: Stephen, I've been looking at the various ways of dropping the S0 connections from the outside but haven't found a simple way of doing this. My Google-Fu seems to have abandoned me, hopefully just temporarily. Would you mind sharing some examples on how you and your organisation implemented this? Cheers, Mike On Tue, Jul 11, 2017 at 6:41 PM Hosom, Stephen M wrote: > Some people choose to implement Bro log filters that can result in a > significant reduction in log volume. For example, if you filter all S0 > connections originating from outside of your organization (and you also > happen to listen outside of a firewall) this could reduce a substantial > amount of log volume. > > -----Original Message----- > From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of > Slagell, Adam J > Sent: Tuesday, July 11, 2017 11:03 AM > To: Mike Eriksson > Cc: bro at bro.org > Subject: Re: [Bro] Get the license usage down in Splunk when indexing Bro > logs > > Message received from outside the Battelle network. Carefully examine it > before you open any links or attachments. > > Well if you are collecting net flows and conn.logs, you could get rid of > one. If you are recapturing syslogs with Bro and sending them to Splunk, > you could trim duplication there. If you are finding a ton of certificate > information in your bro logs, you might realize some cost savings there. > But I don?t have much advice beyond don?t send the same info twice and look > for large amounts of data that you don?t really use in Splunk, like maybe > certificates. > > > > On Jul 11, 2017, at 5:48 AM, Mike Eriksson wrote: > > > > Hi all, > > > > We're currently working on deploying Bro sensors to various offices and > I've come to realise that the Bro logs are quite 'expensive' when it comes > to Splunk licenses. To say the least. > > > > We have discussed various solutions but most of them fall down on us > losing the ability to correlate events unless we shift all the logs in to > Splunk. > > > > At the moment we're running it pretty much 'out of the box' so we can > save some GB's per day to turn of certain scripts, but it will probably not > be enough. > > > > Someone mentioned that turning on JSON logging instead of the standard > logging on Bro could save considerable amounts of space on your SIEM. Have > any of you guys tested this and can you back that statement up? > > > > I was hoping that someone else had encountered this before and had come > up with some solution(s) to this issue? > > > > Thanks in advance, Mike > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/bb126f45/attachment.html From jdopheid at illinois.edu Wed Jul 12 10:48:55 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Wed, 12 Jul 2017 17:48:55 +0000 Subject: [Bro] =?utf-8?q?BroCon_=E2=80=9917=3A_Announcing_=28most_of=29_sp?= =?utf-8?q?eakers?= In-Reply-To: <52412479-70BB-4703-B70F-D0D77FFF7351@illinois.edu> References: <52412479-70BB-4703-B70F-D0D77FFF7351@illinois.edu> Message-ID: A couple more speakers to announce: * Vijay Pasapuleti, Acumera, Travails of running Bro IDS on the Embedded Platform * Kevin McMahon, MITRE corp, Protocol Parser Development to Inspect Malware Command and Control Traffic ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign On 7/10/17, 11:12 AM, "bro-bounces at bro.org on behalf of Dopheide, Jeannette M" wrote: And Alan Commike, Reservoir Labs, ?Running Bro in the Cloud at Scale? ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign On 7/10/17, 10:26 AM, "bro-bounces at bro.org on behalf of Dopheide, Jeannette M" wrote: Hello Bro Community, We?re in the process of wrapping up our speaker list and I wanted to share the list of presentations we have confirmed so far. * Brian Wylie, Kitware, ?Data Analysis, Machine Learning, Bro, and You!? * Mark Krenz, CTSC and formerly SWAMP, ?Using awk to analyze Bro logs? * Adam Pumphrey, Bricata, ?A Bro Primer? * Jan Grashoefer, researcher at KIT, ?The Intelligence Framework Update? * Joseph Mosby, Deloitte & Touche, ?Spark and Bro: When Bro-Cut Won't Cut It? * Aashish Sharma, Lawrence Berkeley National Lab, ?Detecting Credential Spear-phishing Attacks at LBNL? * Fatema Bannat Wala, University of Delaware, ?Unconstrained Endpoint Security System (UEPtSS)? * Benjamin Klimkowski, former Asst. Prof. United States Military Academy, ?Using Bro to Hunt Persistent Threats? We?ll announce more speakers and topics as we receive confirmations. Don?t forget to register: https://brocon2017.eventbrite.com/ Early bird registration expires on July 30th. See you in September, The Bro Team ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From cherdt at umn.edu Wed Jul 12 11:38:32 2017 From: cherdt at umn.edu (Chris Herdt) Date: Wed, 12 Jul 2017 13:38:32 -0500 Subject: [Bro] Manager and logger threads crash immediately on deploy Message-ID: I've successfully run smaller Bro clusters, but now that I'm scaling out I'm seeing the manager and logger threads crash immediately when I deploy the configuration. What I'm trying to run: - 1 manager, 1 logger on 1 host - 8 proxies and 32 workers on 8 hosts I'm using Bro 2.5.1. Each worker host has 2 Myricom 10G NICs w/2 ports each, using the 3.0.10 Myricom SNF driver. I'm attempting to run 9 processes (lb_procs) per worker node, each pinned to its own CPU core. What I'm finding is that any time the number of worker processes exceeds ~160 (not a magic number--not consistent, but around that value based on observation), the manager and logger threads crash. If I keep the number of worker processes at or below ~160 (either by reducing processes per node, reducing nodes per host, or reducing hosts in the cluster) it runs successfully. Ideally, the cluster would have 288 worker processes. This does not seem to be related packet volume, as the manager and logger threads crash even if I am not sending any traffic to the worker nodes. Any troubleshooting or optimization suggestions are appreciated. -- Chris Herdt Systems Administrator University of Minnesota cherdt at umn.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/63946520/attachment.html From dopheide at gmail.com Wed Jul 12 11:56:24 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Wed, 12 Jul 2017 18:56:24 +0000 Subject: [Bro] Manager and logger threads crash immediately on deploy In-Reply-To: References: Message-ID: Is that (8 proxies and 32 workers) on ALL 8 hosts? For 64 total proxies? That seems like a lot to me. On Wed, Jul 12, 2017 at 1:44 PM Chris Herdt wrote: > I've successfully run smaller Bro clusters, but now that I'm scaling out > I'm seeing the manager and logger threads crash immediately when I deploy > the configuration. > > What I'm trying to run: > - 1 manager, 1 logger on 1 host > - 8 proxies and 32 workers on 8 hosts > > I'm using Bro 2.5.1. Each worker host has 2 Myricom 10G NICs w/2 ports > each, using the 3.0.10 Myricom SNF driver. I'm attempting to run 9 > processes (lb_procs) per worker node, each pinned to its own CPU core. > > What I'm finding is that any time the number of worker processes exceeds > ~160 (not a magic number--not consistent, but around that value based on > observation), the manager and logger threads crash. If I keep the number of > worker processes at or below ~160 (either by reducing processes per node, > reducing nodes per host, or reducing hosts in the cluster) it runs > successfully. Ideally, the cluster would have 288 worker processes. > > This does not seem to be related packet volume, as the manager and logger > threads crash even if I am not sending any traffic to the worker nodes. > > Any troubleshooting or optimization suggestions are appreciated. > > -- > Chris Herdt > Systems Administrator > University of Minnesota > cherdt at umn.edu > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/cbfcec3b/attachment-0001.html From cherdt at umn.edu Wed Jul 12 12:01:46 2017 From: cherdt at umn.edu (Chris Herdt) Date: Wed, 12 Jul 2017 14:01:46 -0500 Subject: [Bro] Manager and logger threads crash immediately on deploy In-Reply-To: References: Message-ID: On Wed, Jul 12, 2017 at 1:56 PM, Mike Dopheide wrote: > Is that (8 proxies and 32 workers) on ALL 8 hosts? For 64 total proxies? > > That seems like a lot to me. > > On Wed, Jul 12, 2017 at 1:44 PM Chris Herdt wrote: > >> I've successfully run smaller Bro clusters, but now that I'm scaling out >> I'm seeing the manager and logger threads crash immediately when I deploy >> the configuration. >> >> What I'm trying to run: >> - 1 manager, 1 logger on 1 host >> - 8 proxies and 32 workers on 8 hosts >> >> I'm using Bro 2.5.1. Each worker host has 2 Myricom 10G NICs w/2 ports >> each, using the 3.0.10 Myricom SNF driver. I'm attempting to run 9 >> processes (lb_procs) per worker node, each pinned to its own CPU core. >> >> What I'm finding is that any time the number of worker processes exceeds >> ~160 (not a magic number--not consistent, but around that value based on >> observation), the manager and logger threads crash. If I keep the number of >> worker processes at or below ~160 (either by reducing processes per node, >> reducing nodes per host, or reducing hosts in the cluster) it runs >> successfully. Ideally, the cluster would have 288 worker processes. >> >> This does not seem to be related packet volume, as the manager and logger >> threads crash even if I am not sending any traffic to the worker nodes. >> >> Any troubleshooting or optimization suggestions are appreciated. >> >> 8 total proxies, 32 total workers. (1 proxy node + 4 worker nodes) * 8 hosts -- Chris Herdt Systems Administrator University of Minnesota cherdt at umn.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/f0db8499/attachment.html From dopheide at gmail.com Wed Jul 12 12:32:19 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Wed, 12 Jul 2017 14:32:19 -0500 Subject: [Bro] Manager and logger threads crash immediately on deploy In-Reply-To: References: Message-ID: Okay, I think I'm following now, but I want to restate it so that other's with more large cluster experience can chime in. 1 physical host = manager and logger 8 x physical hosts = proxy + 4*(worker w/ 9lb_procs) I'm not sure if having multiple worker nodes per physical host is all that common. I assume you're doing that so each 'worker' node only monitors one of your four 10G links per physical host. Ignoring the actual traffic capture aspect, have you tried running only one worker node w/ 36 lb_procs per physical host? -Dop On Wed, Jul 12, 2017 at 2:01 PM, Chris Herdt wrote: > On Wed, Jul 12, 2017 at 1:56 PM, Mike Dopheide wrote: > >> Is that (8 proxies and 32 workers) on ALL 8 hosts? For 64 total proxies? >> >> That seems like a lot to me. >> >> On Wed, Jul 12, 2017 at 1:44 PM Chris Herdt wrote: >> >>> I've successfully run smaller Bro clusters, but now that I'm scaling out >>> I'm seeing the manager and logger threads crash immediately when I deploy >>> the configuration. >>> >>> What I'm trying to run: >>> - 1 manager, 1 logger on 1 host >>> - 8 proxies and 32 workers on 8 hosts >>> >>> I'm using Bro 2.5.1. Each worker host has 2 Myricom 10G NICs w/2 ports >>> each, using the 3.0.10 Myricom SNF driver. I'm attempting to run 9 >>> processes (lb_procs) per worker node, each pinned to its own CPU core. >>> >>> What I'm finding is that any time the number of worker processes exceeds >>> ~160 (not a magic number--not consistent, but around that value based on >>> observation), the manager and logger threads crash. If I keep the number of >>> worker processes at or below ~160 (either by reducing processes per node, >>> reducing nodes per host, or reducing hosts in the cluster) it runs >>> successfully. Ideally, the cluster would have 288 worker processes. >>> >>> This does not seem to be related packet volume, as the manager and >>> logger threads crash even if I am not sending any traffic to the worker >>> nodes. >>> >>> Any troubleshooting or optimization suggestions are appreciated. >>> >>> > > 8 total proxies, 32 total workers. > (1 proxy node + 4 worker nodes) * 8 hosts > > > -- > Chris Herdt > Systems Administrator > University of Minnesota > cherdt at umn.edu > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/28b57ab0/attachment.html From jazoff at illinois.edu Wed Jul 12 12:49:04 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 12 Jul 2017 19:49:04 +0000 Subject: [Bro] Manager and logger threads crash immediately on deploy In-Reply-To: References: Message-ID: > On Jul 12, 2017, at 1:38 PM, Chris Herdt wrote: > > What I'm finding is that any time the number of worker processes exceeds ~160 (not a magic number--not consistent, but around that value based on observation), the manager and logger threads crash. If I keep the number of worker processes at or below ~160 (either by reducing processes per node, reducing nodes per host, or reducing hosts in the cluster) it runs successfully. Ideally, the cluster would have 288 worker processes. Yes.. this is a problem. Bro currently uses select() internally for the IO loop and select can't handle more than 1024 file descriptors. Around 170 worker processes is where the manager will accumulate more than 1024 fds. There are a few options here: * Run less lb_procs per port to stay under the limit. * Run two separate bro manager installations so that each manager/logger only handles half the workers. You can currently run more than one logger, but that doesn't help for the manager. * Wait until the broker work is done and the old select code is removed. * Swap out all the uses of select in the communication code with poll - I had started doing this a while back, but it got put on hold. It's probably not that much work to update it for 2.5. From what I remember it seemed to work but I didn't have a chance to do much testing on it. https://github.com/bro/bro/commit/99dbd850b4caa0ed1af351cb7e2695b318ee54ad -- - Justin Azoff From fatema.bannatwala at gmail.com Wed Jul 12 13:14:38 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 12 Jul 2017 16:14:38 -0400 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: References: Message-ID: We only do filtering on conn logs, as they are the heaviest (in our environment at least), before indexing it in Splunk. Also, if you are ingesting files.log as well, then you can build some similar filters in props and transforms for the mime-type you can ignore (like plain/text etc), that will also reduce some of the volume indexed by your Splunk cluster. I do not know much about the cloud deployment, hence can't comment on that. Regards, Fatema. On Wed, Jul 12, 2017 at 3:51 PM, Mike Eriksson wrote: > Hi Fatema, > > Thats looks ace - I'll definitely have to have a try at implementing that. > Hopefully we'll be able to get that done even though we're on Cloud > instances. > > Many thanks for this - it's really apprecaited. > > Cheers, Mike > > On Wed, Jul 12, 2017 at 8:43 PM fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > >> Hi Mike, >> >> We also have something similar for brologs indexing in Splunk. >> What we do currently is to drop all the connections whose history had >> just a "Syn" and nothing else, >> i.e dropping all the tcp connections that were just connection attempts. >> >> And the way we implemented it in Splunk, is with following filter on the >> indexers: >> >> In props.conf: >> [bro_conn_sourcetype] >> TRANSFORMS-null= bro_conn_setnull >> >> In transforms.conf >> [bro_conn_setnull] >> REGEX = \b[S]{1}\b >> DEST_KEY = queue >> FORMAT = nullQueue >> >> Hope this helps. >> >> Thanks, >> Fatema. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/8c3f8cad/attachment-0001.html From mike at swedishmike.org Wed Jul 12 13:35:56 2017 From: mike at swedishmike.org (Mike Eriksson) Date: Wed, 12 Jul 2017 20:35:56 +0000 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: References: Message-ID: Fatema, Trying to filter out on types in the the files.log as well sounds like a great idea. We're a bit more limited as to what we can do ourselves when it comes to cloud Splunk but I'm sure they're more than happy to sell some PS time if need be. ;) Once again - many thanks for a very helpful suggestion. Cheers, Mike On Wed, Jul 12, 2017 at 9:14 PM fatema bannatwala < fatema.bannatwala at gmail.com> wrote: > We only do filtering on conn logs, as they are the heaviest (in our > environment at least), before indexing it in Splunk. > Also, if you are ingesting files.log as well, then you can build some > similar filters in props and transforms for the > mime-type you can ignore (like plain/text etc), that will also reduce some > of the volume indexed by your Splunk cluster. > I do not know much about the cloud deployment, hence can't comment on that. > > Regards, > Fatema. > > > On Wed, Jul 12, 2017 at 3:51 PM, Mike Eriksson > wrote: > >> Hi Fatema, >> >> Thats looks ace - I'll definitely have to have a try at implementing >> that. Hopefully we'll be able to get that done even though we're on Cloud >> instances. >> >> Many thanks for this - it's really apprecaited. >> >> Cheers, Mike >> >> On Wed, Jul 12, 2017 at 8:43 PM fatema bannatwala < >> fatema.bannatwala at gmail.com> wrote: >> >>> Hi Mike, >>> >>> We also have something similar for brologs indexing in Splunk. >>> What we do currently is to drop all the connections whose history had >>> just a "Syn" and nothing else, >>> i.e dropping all the tcp connections that were just connection attempts. >>> >>> And the way we implemented it in Splunk, is with following filter on the >>> indexers: >>> >>> In props.conf: >>> [bro_conn_sourcetype] >>> TRANSFORMS-null= bro_conn_setnull >>> >>> In transforms.conf >>> [bro_conn_setnull] >>> REGEX = \b[S]{1}\b >>> DEST_KEY = queue >>> FORMAT = nullQueue >>> >>> Hope this helps. >>> >>> Thanks, >>> Fatema. >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/40fdf668/attachment.html From fatema.bannatwala at gmail.com Wed Jul 12 16:48:44 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Wed, 12 Jul 2017 19:48:44 -0400 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: References: Message-ID: Hey Mike, So just out of curiosity, I ran a quick search on the files.log for the past hour, to see the top most logged mime-types, and here is the top 12 mime-types in the file: 2630139 application/pkix-cert 366285 text/plain 259828 - 258732 image/gif 175465 text/html 142375 image/jpeg 116151 application/xml 103263 text/json 70691 image/png 48208 application/ocsp-response 18720 application/ocsp-request 16267 application/javascript The Splunk filter can be easily built to ignore or filter the logs with these mime-types (of-course, only if you don't want them in Splunk): In profs.conf: [bro_files_sourcetype] TRANSFORMS-null= bro_files_setnull In transforms.conf: [brol_files_setnull] REGEX = (application\/pkix-cert|text\/plain|image\/gif|text\/html|image\/jpeg) DEST_KEY = queue FORMAT = nullQueue You can add more mime-types in the above REGEX, to filter and send to null queue in Splunk. I added top five, just to give you an idea of how it can be implemented. Hope this helps. :) Thanks, Fatema. On Wed, Jul 12, 2017 at 4:35 PM, Mike Eriksson wrote: > Fatema, > > Trying to filter out on types in the the files.log as well sounds like a > great idea. > > We're a bit more limited as to what we can do ourselves when it comes to > cloud Splunk but I'm sure they're more than happy to sell some PS time if > need be. ;) > > Once again - many thanks for a very helpful suggestion. > > Cheers, Mike > > On Wed, Jul 12, 2017 at 9:14 PM fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > >> We only do filtering on conn logs, as they are the heaviest (in our >> environment at least), before indexing it in Splunk. >> Also, if you are ingesting files.log as well, then you can build some >> similar filters in props and transforms for the >> mime-type you can ignore (like plain/text etc), that will also reduce >> some of the volume indexed by your Splunk cluster. >> I do not know much about the cloud deployment, hence can't comment on >> that. >> >> Regards, >> Fatema. >> >> >> On Wed, Jul 12, 2017 at 3:51 PM, Mike Eriksson >> wrote: >> >>> Hi Fatema, >>> >>> Thats looks ace - I'll definitely have to have a try at implementing >>> that. Hopefully we'll be able to get that done even though we're on Cloud >>> instances. >>> >>> Many thanks for this - it's really apprecaited. >>> >>> Cheers, Mike >>> >>> On Wed, Jul 12, 2017 at 8:43 PM fatema bannatwala < >>> fatema.bannatwala at gmail.com> wrote: >>> >>>> Hi Mike, >>>> >>>> We also have something similar for brologs indexing in Splunk. >>>> What we do currently is to drop all the connections whose history had >>>> just a "Syn" and nothing else, >>>> i.e dropping all the tcp connections that were just connection attempts. >>>> >>>> And the way we implemented it in Splunk, is with following filter on >>>> the indexers: >>>> >>>> In props.conf: >>>> [bro_conn_sourcetype] >>>> TRANSFORMS-null= bro_conn_setnull >>>> >>>> In transforms.conf >>>> [bro_conn_setnull] >>>> REGEX = \b[S]{1}\b >>>> DEST_KEY = queue >>>> FORMAT = nullQueue >>>> >>>> Hope this helps. >>>> >>>> Thanks, >>>> Fatema. >>>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/cc9518e9/attachment.html From seth at corelight.com Wed Jul 12 23:23:59 2017 From: seth at corelight.com (Seth Hall) Date: Thu, 13 Jul 2017 02:23:59 -0400 Subject: [Bro] Bug Report - Software Framework - Flash Player Version Parsing In-Reply-To: References: Message-ID: Oh, that's annoying. I fixed the issue in git master. Thanks for the report! https://github.com/bro/bro/commit/71c9945f266096e1e375461758ade515e9336692 .Seth On Tue, Jul 11, 2017 at 2:25 PM, Philip Romero wrote: > All, > > I was looking into updating my vulnerability alert configuration and noticed > that the software framework may be incorrectly parsing the software version > for Adobe Flash Player. Please see the below example. The full string > details show the correct version (26.0.0.137), but the parsed versions that > I believe the vulnerability scripts read for major and minor versions looks > to be grabbing the "20" from that portion of the syntax in the full string. > > This email is information in case anyone actively updates the software > framework. I'll see if I can try to work it a bit on my local development > system as time permits. Thanks. > > Example Log: > 1499796686.729596 137.164.83.xxx - HTTP::BROWSER Flash% 20 > - - - Player/26 Flash%20Player/26.0.0.137 CFNetwork/811.5.4 > Darwin/16.6.0 (x86_64) > > -- > Philip Romero, CISSP, CISA > Sr. Information Security Analyst > CENIC > promero at cenic.org > Phone: (714) 220-3430 > Mobile: (562) 237-9290 > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com From mike at swedishmike.org Wed Jul 12 23:53:04 2017 From: mike at swedishmike.org (Mike Eriksson) Date: Thu, 13 Jul 2017 06:53:04 +0000 Subject: [Bro] Get the license usage down in Splunk when indexing Bro logs In-Reply-To: References: Message-ID: Fatema, Many thanks yet again. Just for your comparison - here's the top 12 from our current files.log, which is about 50 minutes old and about to rotate out. 1271142 application/pkix-cert 182904 text/plain 54187 text/json 49992 application/xml 46694 text/html 30609 - 19430 image/png 9421 image/jpeg 4778 image/gif 3664 application/x-dosexec 3049 text/ini 1304 application/vnd.ms-cab-compressed Getting rid of just the 'application/pkix-cert' would probably save us lots of license here so your tip is really good. Cheers, Mike On Thu, Jul 13, 2017 at 12:48 AM fatema bannatwala < fatema.bannatwala at gmail.com> wrote: > Hey Mike, > > So just out of curiosity, I ran a quick search on the files.log for the > past hour, to see > the top most logged mime-types, and here is the top 12 mime-types in the > file: > > 2630139 application/pkix-cert > 366285 text/plain > 259828 - > 258732 image/gif > 175465 text/html > 142375 image/jpeg > 116151 application/xml > 103263 text/json > 70691 image/png > 48208 application/ocsp-response > 18720 application/ocsp-request > 16267 application/javascript > > The Splunk filter can be easily built to ignore or filter the logs with > these mime-types (of-course, only if you don't > want them in Splunk): > > In profs.conf: > [bro_files_sourcetype] > TRANSFORMS-null= bro_files_setnull > > In transforms.conf: > [brol_files_setnull] > REGEX = > (application\/pkix-cert|text\/plain|image\/gif|text\/html|image\/jpeg) > > DEST_KEY = queue > FORMAT = nullQueue > > You can add more mime-types in the above REGEX, to filter and send to null > queue in Splunk. > I added top five, just to give you an idea of how it can be implemented. > > Hope this helps. :) > > Thanks, > Fatema. > > On Wed, Jul 12, 2017 at 4:35 PM, Mike Eriksson > wrote: > >> Fatema, >> >> Trying to filter out on types in the the files.log as well sounds like a >> great idea. >> >> We're a bit more limited as to what we can do ourselves when it comes to >> cloud Splunk but I'm sure they're more than happy to sell some PS time if >> need be. ;) >> >> Once again - many thanks for a very helpful suggestion. >> >> Cheers, Mike >> >> On Wed, Jul 12, 2017 at 9:14 PM fatema bannatwala < >> fatema.bannatwala at gmail.com> wrote: >> >>> We only do filtering on conn logs, as they are the heaviest (in our >>> environment at least), before indexing it in Splunk. >>> Also, if you are ingesting files.log as well, then you can build some >>> similar filters in props and transforms for the >>> mime-type you can ignore (like plain/text etc), that will also reduce >>> some of the volume indexed by your Splunk cluster. >>> I do not know much about the cloud deployment, hence can't comment on >>> that. >>> >>> Regards, >>> Fatema. >>> >>> >>> On Wed, Jul 12, 2017 at 3:51 PM, Mike Eriksson >>> wrote: >>> >>>> Hi Fatema, >>>> >>>> Thats looks ace - I'll definitely have to have a try at implementing >>>> that. Hopefully we'll be able to get that done even though we're on Cloud >>>> instances. >>>> >>>> Many thanks for this - it's really apprecaited. >>>> >>>> Cheers, Mike >>>> >>>> On Wed, Jul 12, 2017 at 8:43 PM fatema bannatwala < >>>> fatema.bannatwala at gmail.com> wrote: >>>> >>>>> Hi Mike, >>>>> >>>>> We also have something similar for brologs indexing in Splunk. >>>>> What we do currently is to drop all the connections whose history had >>>>> just a "Syn" and nothing else, >>>>> i.e dropping all the tcp connections that were just connection >>>>> attempts. >>>>> >>>>> And the way we implemented it in Splunk, is with following filter on >>>>> the indexers: >>>>> >>>>> In props.conf: >>>>> [bro_conn_sourcetype] >>>>> TRANSFORMS-null= bro_conn_setnull >>>>> >>>>> In transforms.conf >>>>> [bro_conn_setnull] >>>>> REGEX = \b[S]{1}\b >>>>> DEST_KEY = queue >>>>> FORMAT = nullQueue >>>>> >>>>> Hope this helps. >>>>> >>>>> Thanks, >>>>> Fatema. >>>>> >>>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170713/b4454647/attachment-0001.html From philosnef at gmail.com Thu Jul 13 06:12:13 2017 From: philosnef at gmail.com (erik clark) Date: Thu, 13 Jul 2017 09:12:13 -0400 Subject: [Bro] rss queue conflict question Message-ID: What happens when you squash the rss queue down to one on a bro box with multiple worker nodes, while using the af_packet plugin? With pf_ring (this may be outdated), its recommended you have a queue for each worker. Is this necessary, when af_packet has fanout? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170713/9c3b90c4/attachment.html From promero at cenic.org Thu Jul 13 07:24:10 2017 From: promero at cenic.org (Philip Romero) Date: Thu, 13 Jul 2017 07:24:10 -0700 Subject: [Bro] Bro Digest, Vol 135, Issue 14 In-Reply-To: References: Message-ID: Seth, Thanks a ton for the quick update!!!. Looks great. 1499954837.093491 137.164.83.xxx - HTTP::BROWSER Adobe Flash Player Install Manager 26 0 0 137 CFNetwork/811 Adobe%20Flash%20Player%20Install%20Manager/26.0.0.137 CFNetwork/811.5.4 Darwin/16.6.0 (x86_64) 1499955553.719204 137.164.83.xxx - HTTP::BROWSER Flash 26 0 0 137 CFNetwork/811 Flash%20Player/26.0.0.137 CFNetwork/811.5.4 Darwin/16.6.0 (x86_64) Philip On 7/12/17 11:53 PM, bro-request at bro.org wrote: > Send Bro mailing list submissions to > bro at bro.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > or, via email, send a message with subject or body 'help' to > bro-request at bro.org > > You can reach the person managing the list at > bro-owner at bro.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Bro digest..." > > > Today's Topics: > > 1. Re: Get the license usage down in Splunk when indexing Bro > logs (Mike Eriksson) > 2. Re: Get the license usage down in Splunk when indexing Bro > logs (fatema bannatwala) > 3. Re: Bug Report - Software Framework - Flash Player Version > Parsing (Seth Hall) > 4. Re: Get the license usage down in Splunk when indexing Bro > logs (Mike Eriksson) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 12 Jul 2017 20:35:56 +0000 > From: Mike Eriksson > Subject: Re: [Bro] Get the license usage down in Splunk when indexing > Bro logs > To: fatema bannatwala > Cc: bro > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > Fatema, > > Trying to filter out on types in the the files.log as well sounds like a > great idea. > > We're a bit more limited as to what we can do ourselves when it comes to > cloud Splunk but I'm sure they're more than happy to sell some PS time if > need be. ;) > > Once again - many thanks for a very helpful suggestion. > > Cheers, Mike > > On Wed, Jul 12, 2017 at 9:14 PM fatema bannatwala < > fatema.bannatwala at gmail.com> wrote: > >> We only do filtering on conn logs, as they are the heaviest (in our >> environment at least), before indexing it in Splunk. >> Also, if you are ingesting files.log as well, then you can build some >> similar filters in props and transforms for the >> mime-type you can ignore (like plain/text etc), that will also reduce some >> of the volume indexed by your Splunk cluster. >> I do not know much about the cloud deployment, hence can't comment on that. >> >> Regards, >> Fatema. >> >> >> On Wed, Jul 12, 2017 at 3:51 PM, Mike Eriksson >> wrote: >> >>> Hi Fatema, >>> >>> Thats looks ace - I'll definitely have to have a try at implementing >>> that. Hopefully we'll be able to get that done even though we're on Cloud >>> instances. >>> >>> Many thanks for this - it's really apprecaited. >>> >>> Cheers, Mike >>> >>> On Wed, Jul 12, 2017 at 8:43 PM fatema bannatwala < >>> fatema.bannatwala at gmail.com> wrote: >>> >>>> Hi Mike, >>>> >>>> We also have something similar for brologs indexing in Splunk. >>>> What we do currently is to drop all the connections whose history had >>>> just a "Syn" and nothing else, >>>> i.e dropping all the tcp connections that were just connection attempts. >>>> >>>> And the way we implemented it in Splunk, is with following filter on the >>>> indexers: >>>> >>>> In props.conf: >>>> [bro_conn_sourcetype] >>>> TRANSFORMS-null= bro_conn_setnull >>>> >>>> In transforms.conf >>>> [bro_conn_setnull] >>>> REGEX = \b[S]{1}\b >>>> DEST_KEY = queue >>>> FORMAT = nullQueue >>>> >>>> Hope this helps. >>>> >>>> Thanks, >>>> Fatema. >>>> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/40fdf668/attachment-0001.html > > ------------------------------ > > Message: 2 > Date: Wed, 12 Jul 2017 19:48:44 -0400 > From: fatema bannatwala > Subject: Re: [Bro] Get the license usage down in Splunk when indexing > Bro logs > To: Mike Eriksson > Cc: bro > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > Hey Mike, > > So just out of curiosity, I ran a quick search on the files.log for the > past hour, to see > the top most logged mime-types, and here is the top 12 mime-types in the > file: > > 2630139 application/pkix-cert > 366285 text/plain > 259828 - > 258732 image/gif > 175465 text/html > 142375 image/jpeg > 116151 application/xml > 103263 text/json > 70691 image/png > 48208 application/ocsp-response > 18720 application/ocsp-request > 16267 application/javascript > > The Splunk filter can be easily built to ignore or filter the logs with > these mime-types (of-course, only if you don't > want them in Splunk): > > In profs.conf: > [bro_files_sourcetype] > TRANSFORMS-null= bro_files_setnull > > In transforms.conf: > [brol_files_setnull] > REGEX = > (application\/pkix-cert|text\/plain|image\/gif|text\/html|image\/jpeg) > DEST_KEY = queue > FORMAT = nullQueue > > You can add more mime-types in the above REGEX, to filter and send to null > queue in Splunk. > I added top five, just to give you an idea of how it can be implemented. > > Hope this helps. :) > > Thanks, > Fatema. > > On Wed, Jul 12, 2017 at 4:35 PM, Mike Eriksson wrote: > >> Fatema, >> >> Trying to filter out on types in the the files.log as well sounds like a >> great idea. >> >> We're a bit more limited as to what we can do ourselves when it comes to >> cloud Splunk but I'm sure they're more than happy to sell some PS time if >> need be. ;) >> >> Once again - many thanks for a very helpful suggestion. >> >> Cheers, Mike >> >> On Wed, Jul 12, 2017 at 9:14 PM fatema bannatwala < >> fatema.bannatwala at gmail.com> wrote: >> >>> We only do filtering on conn logs, as they are the heaviest (in our >>> environment at least), before indexing it in Splunk. >>> Also, if you are ingesting files.log as well, then you can build some >>> similar filters in props and transforms for the >>> mime-type you can ignore (like plain/text etc), that will also reduce >>> some of the volume indexed by your Splunk cluster. >>> I do not know much about the cloud deployment, hence can't comment on >>> that. >>> >>> Regards, >>> Fatema. >>> >>> >>> On Wed, Jul 12, 2017 at 3:51 PM, Mike Eriksson >>> wrote: >>> >>>> Hi Fatema, >>>> >>>> Thats looks ace - I'll definitely have to have a try at implementing >>>> that. Hopefully we'll be able to get that done even though we're on Cloud >>>> instances. >>>> >>>> Many thanks for this - it's really apprecaited. >>>> >>>> Cheers, Mike >>>> >>>> On Wed, Jul 12, 2017 at 8:43 PM fatema bannatwala < >>>> fatema.bannatwala at gmail.com> wrote: >>>> >>>>> Hi Mike, >>>>> >>>>> We also have something similar for brologs indexing in Splunk. >>>>> What we do currently is to drop all the connections whose history had >>>>> just a "Syn" and nothing else, >>>>> i.e dropping all the tcp connections that were just connection attempts. >>>>> >>>>> And the way we implemented it in Splunk, is with following filter on >>>>> the indexers: >>>>> >>>>> In props.conf: >>>>> [bro_conn_sourcetype] >>>>> TRANSFORMS-null= bro_conn_setnull >>>>> >>>>> In transforms.conf >>>>> [bro_conn_setnull] >>>>> REGEX = \b[S]{1}\b >>>>> DEST_KEY = queue >>>>> FORMAT = nullQueue >>>>> >>>>> Hope this helps. >>>>> >>>>> Thanks, >>>>> Fatema. >>>>> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170712/cc9518e9/attachment-0001.html > > ------------------------------ > > Message: 3 > Date: Thu, 13 Jul 2017 02:23:59 -0400 > From: Seth Hall > Subject: Re: [Bro] Bug Report - Software Framework - Flash Player > Version Parsing > To: Philip Romero > Cc: "bro at bro.org" > Message-ID: > > Content-Type: text/plain; charset="UTF-8" > > Oh, that's annoying. I fixed the issue in git master. Thanks for the report! > > https://github.com/bro/bro/commit/71c9945f266096e1e375461758ade515e9336692 > > .Seth > > On Tue, Jul 11, 2017 at 2:25 PM, Philip Romero wrote: >> All, >> >> I was looking into updating my vulnerability alert configuration and noticed >> that the software framework may be incorrectly parsing the software version >> for Adobe Flash Player. Please see the below example. The full string >> details show the correct version (26.0.0.137), but the parsed versions that >> I believe the vulnerability scripts read for major and minor versions looks >> to be grabbing the "20" from that portion of the syntax in the full string. >> >> This email is information in case anyone actively updates the software >> framework. I'll see if I can try to work it a bit on my local development >> system as time permits. Thanks. >> >> Example Log: >> 1499796686.729596 137.164.83.xxx - HTTP::BROWSER Flash% 20 >> - - - Player/26 Flash%20Player/26.0.0.137 CFNetwork/811.5.4 >> Darwin/16.6.0 (x86_64) >> >> -- >> Philip Romero, CISSP, CISA >> Sr. Information Security Analyst >> CENIC >> promero at cenic.org >> Phone: (714) 220-3430 >> Mobile: (562) 237-9290 >> >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- Philip Romero, CISSP, CISA Sr. Information Security Analyst CENIC promero at cenic.org Phone: (714) 220-3430 Mobile: (562) 237-9290 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170713/5eea1aef/attachment.html From nahum at us.ibm.com Thu Jul 13 07:44:41 2017 From: nahum at us.ibm.com (Erich M Nahum) Date: Thu, 13 Jul 2017 10:44:41 -0400 Subject: [Bro] Bro Elastic/ELK Experiences Message-ID: After some sweat, I finally have Bro integrated into the ELK stack (Elasticsearch, Logstash, Kibana). While there's a lot of stuff online about doing this, a bunch of it is incomplete and/or out of date. Here is a summary of my experience so as to hopefully save others time. I am using the latest releases of everything (Bro 2.5.0, ELK 5.4.4). I do plan to go further and use the Kafka plugin to pipe into Logstash, but haven't gotten that far yet. Logstash: a lot of the examples have complex regular expression matches to convert the default tab-separated bro logs. This is not necessary. All you need is to emit JSON in Bro: @load tuning/json-logs then in the logstash config file set your codec to JSON: codec => "json" Timestamps: This is important to get right so that the Kibana time-based functions work properly. Again, a lot of examples are out of date. What worked for me was to have Bro use ISO timestamps rather than the default Unix Epoch: redef LogAscii::json_timestamps = JSON::TS_ISO8601 ; then in the logstash filter section, add the following: date { match => [ "ts", "ISO8601" ] } GeoIP: This was a pain to get right, with many questions on the ELK forums, First, you need to add a geoip to the filter section in the logstash config file: geoip { source => "id.resp_h" target => "geoip" } If you use the default logstash output to elasticsearch with no changes, this works (i.e., Kibana recognizes the geoip.location filed as a "geo_point"). However, all your indexes are called "logstash*", which is a pain, and causes problems the moment you add a new bro log type. I wanted to have an index per Bro log type, so I did the following: In the input section of logstash, I put: file { start_position => "beginning" type => "bro_conn_logs" path => "/home/nahum/conn.log" codec => "json" } Then, in the output section of logstash, I put: if [type] == "bro_conn_logs" { elasticsearch { index => "bro-conn" } } This, however, breaks the GeoIP because it relies on the default index template for logstash, which defines how to map the geoip to a geo_point. The moment you change the index name, you lose that template mapping. So, I created a new bro template doing the following: 1: Get the logstash template curl -XGET localhost:9200/_template/logstash?pretty > bro.template 2: Edit bro.template to change logstash to bro 3: Delete all existing indexes 4: Install the new template into Elasticsearch curl -XPUT 'localhost:9200/_template/bro?pretty' \ -H 'Content-Type: application/json' -d ' ' where is the inline template JSON. 5: Reload data into ES indexes via logstash Note you must have no indexes before loading data, since templates are auto-generated at index creation time. I.e., if there's no pre-existing template for your new index, Elasticsearch will automatically create one, which doesn't handle the geoip properly, and thus it's too late. So create the template BEFORE creating any new indexes. Thanks to Aaron Gee-Clough for answering some questions. I'm also attaching my logstash config for reference. -Erich (See attached file: bro-to-elastic.conf) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170713/bc373bd0/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: bro-to-elastic.conf Type: application/octet-stream Size: 1203 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170713/bc373bd0/attachment-0001.obj From mus3 at lehigh.edu Thu Jul 13 08:20:28 2017 From: mus3 at lehigh.edu (Munroe Sollog) Date: Thu, 13 Jul 2017 11:20:28 -0400 Subject: [Bro] Bro Elastic/ELK Experiences In-Reply-To: References: Message-ID: We are using NSQ rather than kafka to get stuff into ELK, FYI works great. On Thu, Jul 13, 2017 at 10:44 AM, Erich M Nahum wrote: > After some sweat, I finally have Bro integrated into the ELK stack > (Elasticsearch, > Logstash, Kibana). While there's a lot of stuff online about doing this, a > bunch of it is incomplete and/or out of date. Here is a summary of my > experience > so as to hopefully save others time. I am using the latest releases of > everything (Bro 2.5.0, ELK 5.4.4). > > I do plan to go further and use the Kafka plugin to pipe into Logstash, > but haven't gotten that far yet. > > Logstash: a lot of the examples have complex regular expression matches to > convert the default tab-separated bro logs. This is not necessary. All you > need is to emit JSON in Bro: > > @load tuning/json-logs > > then in the logstash config file set your codec to JSON: > > codec => "json" > > Timestamps: This is important to get right so that the Kibana time-based > functions > work properly. Again, a lot of examples are out of date. What worked for me > was to have Bro use ISO timestamps rather than the default Unix Epoch: > > redef LogAscii::json_timestamps = JSON::TS_ISO8601 ; > > then in the logstash filter section, add the following: > > date { > match => [ "ts", "ISO8601" ] > } > > GeoIP: This was a pain to get right, with many questions on the ELK forums, > First, you need to add a geoip to the filter section in the logstash config > file: > > geoip { > source => "id.resp_h" > target => "geoip" > } > If you use the default logstash output to elasticsearch with no changes, > this works (i.e., Kibana recognizes the geoip.location filed as a > "geo_point"). > However, all your indexes are called "logstash*", which is a pain, and > causes > problems the moment you add a new bro log type. I wanted to have an index > per > Bro log type, so I did the following: > > In the input section of logstash, I put: > > file { > start_position => "beginning" > type => "bro_conn_logs" > path => "/home/nahum/conn.log" > codec => "json" > } > > Then, in the output section of logstash, I put: > > if [type] == "bro_conn_logs" { > elasticsearch { > index => "bro-conn" > } > } > > This, however, breaks the GeoIP because it relies on the default index > template > for logstash, which defines how to map the geoip to a geo_point. The > moment you > change the index name, you lose that template mapping. So, I created a new > bro template doing the following: > > 1: Get the logstash template > > curl -XGET localhost:9200/_template/logstash?pretty > bro.template > > 2: Edit bro.template to change logstash to bro > 3: Delete all existing indexes > 4: Install the new template into Elasticsearch > > curl -XPUT 'localhost:9200/_template/bro?pretty' \ > -H 'Content-Type: application/json' -d ' ' > > where is the inline template JSON. > > 5: Reload data into ES indexes via logstash > > Note you must have no indexes before loading data, since templates > are auto-generated at index creation time. I.e., if there's no pre-existing > template for your new index, Elasticsearch will automatically create one, > which > doesn't handle the geoip properly, and thus it's too late. > So create the template BEFORE creating any new indexes. > > Thanks to Aaron Gee-Clough for answering some questions. I'm also > attaching my logstash config for reference. > > -Erich > > *(See attached file: bro-to-elastic.conf)* > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Munroe Sollog Senior Network Engineer munroe at lehigh.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170713/b8a556b9/attachment.html From zhangxu1115 at gmail.com Thu Jul 13 09:42:56 2017 From: zhangxu1115 at gmail.com (Xu Zhang) Date: Thu, 13 Jul 2017 09:42:56 -0700 Subject: [Bro] SumStats framework Message-ID: Hi, I'm using SumStats framework to record features in the SSL handshake packets. There are lots of features (30+) I need to record and I created a reducer for each feature. In the SumStats::create(), I check if "feature_x" in result, and record result["feature_x"]$num. However, the SumStats::create function looks absurdly long. My question is: is it more efficient to break up the current SumStats::create function into multiple (each only have one reducer), or is it better to keep the code I currently have? Which one is faster? Thanks a lot! -- Sincerely, Xu Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170713/1d2300e8/attachment.html From anthony.kasza at gmail.com Thu Jul 13 14:40:34 2017 From: anthony.kasza at gmail.com (anthony kasza) Date: Thu, 13 Jul 2017 15:40:34 -0600 Subject: [Bro] SumStats framework In-Reply-To: References: Message-ID: Hi Xu, Can you share the script you've written? -AK On Jul 13, 2017 10:52 AM, "Xu Zhang" wrote: > Hi, > > I'm using SumStats framework to record features in the SSL handshake > packets. There are lots of features (30+) I need to record and I created a > reducer for each feature. In the SumStats::create(), I check if > "feature_x" in result, and record result["feature_x"]$num. However, the > SumStats::create function looks absurdly long. My question is: is it more > efficient to break up the current SumStats::create function into multiple > (each only have one reducer), or is it better to keep the code I currently > have? Which one is faster? > > Thanks a lot! > > -- > Sincerely, > Xu Zhang > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170713/77c8311a/attachment.html From zhangxu1115 at gmail.com Thu Jul 13 14:51:40 2017 From: zhangxu1115 at gmail.com (Xu Zhang) Date: Thu, 13 Jul 2017 14:51:40 -0700 Subject: [Bro] SumStats framework In-Reply-To: References: Message-ID: Hi Anthony, I have not finished the whole script yet. But Basically it is event bro_init() { local r: set[SumStats::Reducer]; local chellos = SumStats::Reducer($stream="client_hello_num", $apply=set(SumStats::SUM)); add r[chellos]; local shellos = SumStats::Reducer($stream="server_hello_num", $apply=set(SumStats::SUM)); add r[shellos]; ..... (a couple of other reducers ) SumStats::create([$name = "ssl stats", $epoch = 1hr, $reducers = r, $epoch_result(ts: time, key: SumStats::Key, result: SumStats::Result) = { if ("client_hello_num" in result) bla; if ("server_hello_num" in result) bla; ......(a couple of IFs) }]); } On Thu, Jul 13, 2017 at 2:40 PM, anthony kasza wrote: > Hi Xu, > > Can you share the script you've written? > > -AK > > On Jul 13, 2017 10:52 AM, "Xu Zhang" wrote: > >> Hi, >> >> I'm using SumStats framework to record features in the SSL handshake >> packets. There are lots of features (30+) I need to record and I created a >> reducer for each feature. In the SumStats::create(), I check if >> "feature_x" in result, and record result["feature_x"]$num. However, the >> SumStats::create function looks absurdly long. My question is: is it more >> efficient to break up the current SumStats::create function into multiple >> (each only have one reducer), or is it better to keep the code I currently >> have? Which one is faster? >> >> Thanks a lot! >> >> -- >> Sincerely, >> Xu Zhang >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > -- Sincerely, Xu Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170713/25cd29da/attachment.html From oelnaggar04 at gmail.com Thu Jul 13 16:58:50 2017 From: oelnaggar04 at gmail.com (Osama Elnaggar) Date: Thu, 13 Jul 2017 16:58:50 -0700 Subject: [Bro] Bro Elastic/ELK Experiences In-Reply-To: References: Message-ID: +1. Thanks for sharing. You might want to have NXLog or Filebeat as the shipper instead of Logstash on the Bro instance as they are a lot lighter resource wise. You could even forward from Filebeat to Redis and then have Logstash read from Redis for a quick improvement or go the Kafka way as you mentioned. -- Osama Elnaggar On July 14, 2017 at 1:00:22 AM, Erich M Nahum (nahum at us.ibm.com) wrote: After some sweat, I finally have Bro integrated into the ELK stack (Elasticsearch, Logstash, Kibana). While there's a lot of stuff online about doing this, a bunch of it is incomplete and/or out of date. Here is a summary of my experience so as to hopefully save others time. I am using the latest releases of everything (Bro 2.5.0, ELK 5.4.4). I do plan to go further and use the Kafka plugin to pipe into Logstash, but haven't gotten that far yet. Logstash: a lot of the examples have complex regular expression matches to convert the default tab-separated bro logs. This is not necessary. All you need is to emit JSON in Bro: @load tuning/json-logs then in the logstash config file set your codec to JSON: codec => "json" Timestamps: This is important to get right so that the Kibana time-based functions work properly. Again, a lot of examples are out of date. What worked for me was to have Bro use ISO timestamps rather than the default Unix Epoch: redef LogAscii::json_timestamps = JSON::TS_ISO8601 ; then in the logstash filter section, add the following: date { match => [ "ts", "ISO8601" ] } GeoIP: This was a pain to get right, with many questions on the ELK forums, First, you need to add a geoip to the filter section in the logstash config file: geoip { source => "id.resp_h" target => "geoip" } If you use the default logstash output to elasticsearch with no changes, this works (i.e., Kibana recognizes the geoip.location filed as a "geo_point"). However, all your indexes are called "logstash*", which is a pain, and causes problems the moment you add a new bro log type. I wanted to have an index per Bro log type, so I did the following: In the input section of logstash, I put: file { start_position => "beginning" type => "bro_conn_logs" path => "/home/nahum/conn.log" codec => "json" } Then, in the output section of logstash, I put: if [type] == "bro_conn_logs" { elasticsearch { index => "bro-conn" } } This, however, breaks the GeoIP because it relies on the default index template for logstash, which defines how to map the geoip to a geo_point. The moment you change the index name, you lose that template mapping. So, I created a new bro template doing the following: 1: Get the logstash template curl -XGET localhost:9200/_template/logstash?pretty > bro.template 2: Edit bro.template to change logstash to bro 3: Delete all existing indexes 4: Install the new template into Elasticsearch curl -XPUT 'localhost:9200/_template/bro?pretty' \ -H 'Content-Type: application/json' -d ' ' where is the inline template JSON. 5: Reload data into ES indexes via logstash Note you must have no indexes before loading data, since templates are auto-generated at index creation time. I.e., if there's no pre-existing template for your new index, Elasticsearch will automatically create one, which doesn't handle the geoip properly, and thus it's too late. So create the template BEFORE creating any new indexes. Thanks to Aaron Gee-Clough for answering some questions. I'm also attaching my logstash config for reference. -Erich *(See attached file: bro-to-elastic.conf)* _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170713/515cb6bd/attachment-0001.html From jazoff at illinois.edu Fri Jul 14 06:36:03 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 14 Jul 2017 13:36:03 +0000 Subject: [Bro] SumStats framework In-Reply-To: References: Message-ID: > On Jul 13, 2017, at 5:51 PM, Xu Zhang wrote: > > Hi Anthony, > I have not finished the whole script yet. > But Basically it is > event bro_init() > { > local r: set[SumStats::Reducer]; > local chellos = SumStats::Reducer($stream="client_hello_num", $apply=set(SumStats::SUM)); > add r[chellos]; > local shellos = SumStats::Reducer($stream="server_hello_num", $apply=set(SumStats::SUM)); > add r[shellos]; > ..... (a couple of other reducers ) > > SumStats::create([$name = "ssl stats", > $epoch = 1hr, > $reducers = r, > $epoch_result(ts: time, key: SumStats::Key, result: SumStats::Result) = > { > if ("client_hello_num" in result) > bla; > if ("server_hello_num" in result) > bla; > ......(a couple of IFs) > }]); > } > Yeah, it looks like you're making that a lot more complicated than it needs to be. You just need a single stream and a single reducer. When you do the observations just do SumStats::observe("ssl_events", [$str="client_hello"], [$num=1]); or SumStats::observe("ssl_events", [$str="server_hello"], [$num=1]); etc and then look at 'key' inside of the reducer, not result. -- - Justin Azoff From zhangxu1115 at gmail.com Fri Jul 14 10:02:43 2017 From: zhangxu1115 at gmail.com (Xu Zhang) Date: Fri, 14 Jul 2017 10:02:43 -0700 Subject: [Bro] SumStats framework In-Reply-To: References: Message-ID: Hi, Just make sure I understand correctly. So you are saying make a couple of SumStats::create(), each SumStat::create() has only one reducer. Could you give an example of "looking at 'key' inside of the reducer, not result"? Thanks a lot! On Fri, Jul 14, 2017 at 6:36 AM, Azoff, Justin S wrote: > > > On Jul 13, 2017, at 5:51 PM, Xu Zhang wrote: > > > > Hi Anthony, > > I have not finished the whole script yet. > > But Basically it is > > event bro_init() > > { > > local r: set[SumStats::Reducer]; > > local chellos = SumStats::Reducer($stream="client_hello_num", > $apply=set(SumStats::SUM)); > > add r[chellos]; > > local shellos = SumStats::Reducer($stream="server_hello_num", > $apply=set(SumStats::SUM)); > > add r[shellos]; > > ..... (a couple of other reducers ) > > > > SumStats::create([$name = "ssl stats", > > $epoch = 1hr, > > $reducers = r, > > $epoch_result(ts: time, key: > SumStats::Key, result: SumStats::Result) = > > { > > if ("client_hello_num" in result) > > bla; > > if ("server_hello_num" in result) > > bla; > > ......(a couple of IFs) > > }]); > > } > > > > Yeah, it looks like you're making that a lot more complicated than it > needs to be. You just need a single stream and a single reducer. > > When you do the observations just do > > SumStats::observe("ssl_events", [$str="client_hello"], [$num=1]); > > or > > SumStats::observe("ssl_events", [$str="server_hello"], [$num=1]); > > etc > > and then look at 'key' inside of the reducer, not result. > > > > -- > - Justin Azoff > > -- Sincerely, Xu Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170714/689e1fd9/attachment.html From jazoff at illinois.edu Fri Jul 14 10:35:05 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 14 Jul 2017 17:35:05 +0000 Subject: [Bro] SumStats framework In-Reply-To: References: Message-ID: > On Jul 14, 2017, at 1:02 PM, Xu Zhang wrote: > > Hi, > Just make sure I understand correctly. So you are saying make a couple of SumStats::create(), each SumStat::create() has only one reducer. > Could you give an example of "looking at 'key' inside of the reducer, not result"? > Thanks a lot! No.. I'm saying that you should have a single create. By looking at the key I mean use the 'key' variable that is present in the epoch_result function. Attached is a script I wrote a few years ago. It lets you track arbitrary statistics using sumstats - but it should only be used for a finite number of 'key' values... 1-500 keys would be ok.. using something like an id.orig_h as a key will break sumstats. To use it you can just do event ssl_server_hello(c: connection, version: count, possible_ts: time, server_random: string, session_id: string, cipher: count, comp_method: count) { StatMetrics::increment("server_hello", 1); } event ssl_client_hello(c: connection, version: count, possible_ts: time, client_random: string, session_id: string, ciphers: index_vec) { StatMetrics::increment("client_hello", 1); } -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170714/449aec53/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: stat_metrics.bro Type: application/octet-stream Size: 1558 bytes Desc: stat_metrics.bro Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170714/449aec53/attachment.obj From zhangxu1115 at gmail.com Fri Jul 14 11:26:35 2017 From: zhangxu1115 at gmail.com (Xu Zhang) Date: Fri, 14 Jul 2017 11:26:35 -0700 Subject: [Bro] SumStats framework In-Reply-To: References: Message-ID: Sorry I did not provide enough information for my problem. You approach would work for client hello and server hello. But for other features, i need to record the value: for example event ssl_server_hello(...) { SumStats::observe("server_hello_version",[$str=SSL::version_strings[version]],[$num=1]); } I'm using the key field to keep the actual value of that feature. So I cannot reuse the same reducer "ssl_events" because it will lose the actual value of that feature. SumStats::observe("ssl_events",[$str="server_hello_version"],[$num=1]); On Fri, Jul 14, 2017 at 10:35 AM, Azoff, Justin S wrote: > > > On Jul 14, 2017, at 1:02 PM, Xu Zhang wrote: > > > > Hi, > > Just make sure I understand correctly. So you are saying make a couple > of SumStats::create(), each SumStat::create() has only one reducer. > > Could you give an example of "looking at 'key' inside of the reducer, > not result"? > > Thanks a lot! > > No.. I'm saying that you should have a single create. > > By looking at the key I mean use the 'key' variable that is present in the > epoch_result function. > > Attached is a script I wrote a few years ago. It lets you track arbitrary > statistics using sumstats - but it should only be used for a finite number > of 'key' values... 1-500 keys would be ok.. using something like an > id.orig_h as a key will break sumstats. > > > To use it you can just do > > event ssl_server_hello(c: connection, version: count, possible_ts: time, > server_random: string, session_id: string, cipher: count, comp_method: > count) > { > StatMetrics::increment("server_hello", 1); > } > > event ssl_client_hello(c: connection, version: count, possible_ts: time, > client_random: string, session_id: string, ciphers: index_vec) > { > StatMetrics::increment("client_hello", 1); > > } > > > > -- > - Justin Azoff > > -- Sincerely, Xu Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170714/6903fc6b/attachment-0001.html From jazoff at illinois.edu Fri Jul 14 11:30:42 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 14 Jul 2017 18:30:42 +0000 Subject: [Bro] SumStats framework In-Reply-To: References: Message-ID: > On Jul 14, 2017, at 2:26 PM, Xu Zhang wrote: > > Sorry I did not provide enough information for my problem. You approach would work for client hello and server hello. But for other features, i need to record the value: for example > event ssl_server_hello(...) > { > SumStats::observe("server_hello_version",[$str=SSL::version_strings[version]],[$num=1]); > } > I'm using the key field to keep the actual value of that feature. So I cannot reuse the same reducer "ssl_events" because it will lose the actual value of that feature. > SumStats::observe("ssl_events",[$str="server_hello_version"],[$num=1]); For a small number of values like this you can just set the string to fmt("server_hello_version.%s", SSL::version_strings[version]) and then you'll get counts of server_hello_version.SSLv3 server_hello_version.TLSv13 etc. -- - Justin Azoff From zhangxu1115 at gmail.com Fri Jul 14 12:54:07 2017 From: zhangxu1115 at gmail.com (Xu Zhang) Date: Fri, 14 Jul 2017 12:54:07 -0700 Subject: [Bro] SumStats framework In-Reply-To: References: Message-ID: Hi Justin, I rewrote my code as you suggested. It works great and reduces more than 100 lines. Thanks a lot! Now I just need to pay attention to the number of possible values for each feature (str$key). I assume some values may be more than 500, for example, possible ssl_extensions or cipher suites. On Fri, Jul 14, 2017 at 11:30 AM, Azoff, Justin S wrote: > > > On Jul 14, 2017, at 2:26 PM, Xu Zhang wrote: > > > > Sorry I did not provide enough information for my problem. You approach > would work for client hello and server hello. But for other features, i > need to record the value: for example > > event ssl_server_hello(...) > > { > > SumStats::observe("server_hello_version",[$str=SSL:: > version_strings[version]],[$num=1]); > > } > > I'm using the key field to keep the actual value of that feature. So I > cannot reuse the same reducer "ssl_events" because it will lose the actual > value of that feature. > > SumStats::observe("ssl_events",[$str="server_hello_version"],[$num=1]); > > For a small number of values like this you can just set the string to > > fmt("server_hello_version.%s", SSL::version_strings[version]) > > and then you'll get counts of > > server_hello_version.SSLv3 > server_hello_version.TLSv13 > > etc. > > > > > -- > - Justin Azoff > > -- Sincerely, Xu Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170714/d7d68452/attachment.html From archeldeeb at gmail.com Sun Jul 16 00:15:13 2017 From: archeldeeb at gmail.com (Sherif Eldeeb) Date: Sun, 16 Jul 2017 10:15:13 +0300 Subject: [Bro] Send BRO logs over TCP/UDP Message-ID: I am aware that there are many ways to send bro logs to syslog/SIEM/etc., but as far as I understand they're all relying on reading/tailing the log files created by bro then send them over to their destination (e.g. logstash & NiFi). Is it possible to send logs over TCP/UDP without having them written to disk to be read by a third party app? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170716/05af24fb/attachment.html From mike at swedishmike.org Mon Jul 17 01:19:46 2017 From: mike at swedishmike.org (Mike Eriksson) Date: Mon, 17 Jul 2017 08:19:46 +0000 Subject: [Bro] Exclude S0 connections from conn.log? Message-ID: All, I've been looking at cutting down the size of my logs and after some great advice on this list one of the things that seems to help is to exclude S0 connections from conn.log I've been looking at doing this but sadly I'm still too much of a beginner to get this to work so I was hoping that someone out there can give me some guidance? Basically what I'd like to achieve is for the script to not log any events with a conn_state of S0 if the originating node is not in my local networks. If someone could give me some guidance on how to achieve this I'd be forever grateful. Thanks in advance, Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170717/aef057fc/attachment.html From jan.grashoefer at gmail.com Mon Jul 17 01:40:35 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Mon, 17 Jul 2017 10:40:35 +0200 Subject: [Bro] Exclude S0 connections from conn.log? In-Reply-To: References: Message-ID: Hi Mike, > Basically what I'd like to achieve is for the script to not log any events > with a conn_state of S0 if the originating node is not in my local > networks. > > If someone could give me some guidance on how to achieve this I'd be > forever grateful. you can use a filter (e.g., change the default one): https://www.bro.org/sphinx/frameworks/logging.html#filter-log-records There is also a blog post (http://blog.bro.org/2012/02/filtering-logs-with-bro.html) with a couple of examples as well as scripts available on github (e.g., https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_conn.bro). Jan From mike at swedishmike.org Mon Jul 17 05:36:48 2017 From: mike at swedishmike.org (Mike Eriksson) Date: Mon, 17 Jul 2017 12:36:48 +0000 Subject: [Bro] Exclude S0 connections from conn.log? In-Reply-To: References: Message-ID: All, With the pointers from Jan, and some earlier replies to an earlier question, I've managed to cobble together the following: <- Cut -> module LogFilter; event bro_init() { Log::remove_default_filter(Conn::LOG); Log::add_filter(Conn::LOG, [$name = "conn-filter-external-S0", $pred(rec: Conn::Info) = { local result = T; if ((/^S0$/ in rec$conn_state) && (!Site::is_local_addr(rec$id$orig_h))) result = F; return result; } ]); } <- Cut -> It _seems_ to be working as expected so I thought I'd post it here. For two reasons ;) 1. It could help someone else in the same predicament. 2. There could be a better way of doing it - or even a correct one ;) - and someone might spot my mistake(s). Cheers, Mike On Mon, Jul 17, 2017 at 9:42 AM Jan Grash?fer wrote: > Hi Mike, > > > Basically what I'd like to achieve is for the script to not log any > events > > with a conn_state of S0 if the originating node is not in my local > > networks. > > > > If someone could give me some guidance on how to achieve this I'd be > > forever grateful. > > you can use a filter (e.g., change the default one): > https://www.bro.org/sphinx/frameworks/logging.html#filter-log-records > > There is also a blog post > (http://blog.bro.org/2012/02/filtering-logs-with-bro.html) with a couple > of examples as well as scripts available on github (e.g., > > https://github.com/michalpurzynski/bro-gramming/blob/master/filter_noise_conn.bro > ). > > Jan > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170717/a4fdc02c/attachment.html From jeffrey.bencteux at ssi.gouv.fr Tue Jul 18 02:56:53 2017 From: jeffrey.bencteux at ssi.gouv.fr (Bencteux Jeffrey) Date: Tue, 18 Jul 2017 11:56:53 +0200 Subject: [Bro] Atomic operations on Broker store Message-ID: <596DDB65.80002@ssi.gouv.fr> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi all, I am playing around with Broker framework and I am having some trouble trying to share elements such as set or table between multiple instances of Bro. What I am doing follows : * A master store script creates a table and add it to the store * A frontend store script retreive the table, add elements to it and push it back to the store I got strange results doing that such as only one element I was trying to add was added (never the same) so I suspected the problem was concurrency and that my operations were not atomic. Indeed, what happens it that every call to my function in the slave script do not retreive an up-to-date table. Retreiving and adding a new element to a table is not an atomic operation and no function in the API is defined to do it in one call. If I do the same test with Broker::add_to_set() with a set instead of a table it works. And it seems that it is because it is implemented as an atomic operation in aux/broker/src/store/frontend.cc. Any idea how to that with a table with existing functions? Or does it needs further developments? If so, any hints on how to implement that? Here is the Bro scripts I used (debug prints removed) : master.bro: @load base/frameworks/broker @load broker_wrapper const broker_port: port = 6666/tcp &redef; redef exit_only_after_terminate = T; global h: opaque of Broker::Handle; event bro_init() { Broker::enable(); Broker::listen(broker_port, "127.0.0.1"); h = Broker::create_master("test_store"); WRAPPER::broker_table_create(h, "test"); } event Broker::incoming_connection_established(peer_name: string) { print "Conn established: ", peer_name; } frontend.bro: @load base/frameworks/broker @load broker_wrapper const broker_port: port = 6666/tcp &redef; redef exit_only_after_terminate = T; global h: opaque of Broker::Handle; event bro_init() { Broker::enable(); Broker::connect("127.0.0.1", broker_port, 1secs); } event Broker::outgoing_connection_established(peer_address: string, peer_port: port, peer_name: string) { print "Conn established: ", peer_address, peer_port, peer_name; h = Broker::create_clone("test_store"); WRAPPER::broker_table_insert(h, "test", Broker::data("one"), Broker::data("this")); WRAPPER::broker_table_insert(h, "test", Broker::data("two"), Broker::data("is")); WRAPPER::broker_table_insert(h, "test", Broker::data("three"), Broker::data("sparta")); } event Broker::outgoing_connection_broken(peer_name: string, peer_port: port) { print "Connection closed by remote peer"; terminate(); } broker_wrapper.bro: module WRAPPER; export { global broker_table_create: function(h: opaque of Broker::Handle, name: string); global broker_table_insert: function(h: opaque of Broker::Handle, name: string, key: Broker::Data, val: Broker::Data); } function broker_table_create(h: opaque of Broker::Handle, name: string) { local tab = Broker::table_create(); Broker::insert(h, Broker::data(name), tab); } function broker_table_insert(h: opaque of Broker::Handle, name: string, key: Broker::Data, val: Broker::Data) { # look for table with name 'name' when (local res = Broker::lookup(h, Broker::data(name))) { # insert element "key" = "value" into table local status = Broker::table_insert(res$result, key, val); print res$result; # insert table back into store Broker::insert(h, Broker::data(name), res$result); } timeout 10sec { print fmt("timeout broker_table_insert: %s key: %s val: %s", name, key, val); } } I also did a second test: If I try to add an element to a set, then test its existence in it with Broker::set_contains() and it appears not to be in in the set. Any idea why? Any help is welcome! - -- Jeffrey BENCTEUX ANSSI/COSSI/DTO/BSD -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJZbdtlAAoJEHTjT1vhqZtXV/sP/jmbNea4tCqnyX7wti2co5zc 6yL3aX7UyXHkuElzr3weKHA+oQZ3yguoQjodmigrwmBOMiFGp0bxzPvWUvYbnxWu p7yiZZxooJNEyxkoBmauk05p79dqdhmDGLQ3kIBOGFkD0lMHx9bGPRIifEI4oDcp hUwDi5LaEb/x//L6IPkByxxEoNrbI0HxSLS4Bl4TikUpkYbUICVsihdrlN8cPhXk ysaO9S1jv9iJvHSOrktrhv9WuDup0HM0LDE142d/V+Dao9nu+P+HdfKd4ieRVq2O AXFnoIp9flFksgn+PN/Y1FqF9pRWm1MUOQeUCLYTf9br7Z0z0jTR35NRogDvVFtH at/pKuotWOozZhTPf3QnbOdpdLgXIlk6kdIqQ3ENgmkj72L8Giyn/7cEqpl0dBgB 9iR9p6VQBuI8vsRigRUvffyC7TwonCECE+CTL9AzYCv0Zs25sTZTOw7JFMS99qav spGY5EK6H108Amc3COWQ3Gz5IUps3K4j6OwNIbOZKyaHztx8amaF6j2BPpU+V4Ox 2RuThAjW3kOECr2/fseN6HYR2AX7lNCn+HJqWHS+C+pbdOBUhGWucwYj+R+28u+h PVlWwYxcSb0ezZxckX3w32oi0I0YkX9ejLtxdLMd2L9XY4S1nRhxeFHnESqx51gi y6Uh6WaD+yfu9o4IXIUL =6UD8 -----END PGP SIGNATURE----- From vikrambasu059 at gmail.com Wed Jul 19 04:57:16 2017 From: vikrambasu059 at gmail.com (Vikram Basu) Date: Wed, 19 Jul 2017 17:27:16 +0530 Subject: [Bro] detect-external-names.bro Message-ID: <596f491d.9b43620a.4cb9b.06c2@mx.google.com> Hi, I am confused what the protocols/dns/detect-external-names.bro script is actually doing. The documentation reads ?This script detects names which are not within zones considered to be local but resolving to addresses considered local. The Site::local_zones variable must be set appropriately for this detection.? What does ?names which are not within zones considered to be local but resolving to addresses considered local? mean? And how is it determined ? Can you give an example which makes this clearer ? Regards Vikram Basu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170719/39645656/attachment.html From apumphrey at bricata.com Wed Jul 19 12:07:37 2017 From: apumphrey at bricata.com (Adam Pumphrey) Date: Wed, 19 Jul 2017 19:07:37 +0000 Subject: [Bro] detect-external-names.bro In-Reply-To: <596f491d.9b43620a.4cb9b.06c2@mx.google.com> References: <596f491d.9b43620a.4cb9b.06c2@mx.google.com> Message-ID: It checks responses to DNS A record queries for an IP address in the answer that is considered local (based on the Site::local_nets variable), but the resolved DNS name (the query) is not in a local DNS zone (based on the Site::local_zones variable). The IP and query locality tests in this script depend on both the Site::local_nets and Site::local_zones variables. If Site::local_zones is populated correctly, this script uses the Site::is_local_name function to see if the queried name belongs to a local DNS zone. Underneath its using a regular expression to match any subdomains of a zone specified in local_zones. If the query doesn?t appear to be in a local zone, but the IP in the answer was, the script generates a Notice. From: on behalf of Vikram Basu Date: Wednesday, July 19, 2017 at 7:57 AM To: "bro at bro.org" Subject: [Bro] detect-external-names.bro Hi, I am confused what the protocols/dns/detect-external-names.bro script is actually doing. The documentation reads ?This script detects names which are not within zones considered to be local but resolving to addresses considered local. The Site::local_zones variable must be set appropriately for this detection.? What does ?names which are not within zones considered to be local but resolving to addresses considered local? mean? And how is it determined ? Can you give an example which makes this clearer ? Regards Vikram Basu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170719/7f88918a/attachment.html From jazoff at illinois.edu Wed Jul 19 12:20:18 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 19 Jul 2017 19:20:18 +0000 Subject: [Bro] detect-external-names.bro In-Reply-To: References: <596f491d.9b43620a.4cb9b.06c2@mx.google.com> Message-ID: <8F737DA5-55D1-437B-AAF5-7BC0E50AD703@illinois.edu> > On Jul 19, 2017, at 3:07 PM, Adam Pumphrey wrote: > > It checks responses to DNS A record queries for an IP address in the answer that is considered local (based on the Site::local_nets variable), but the resolved DNS name (the query) is not in a local DNS zone (based on the Site::local_zones variable). > > The IP and query locality tests in this script depend on both the Site::local_nets and Site::local_zones variables. If Site::local_zones is populated correctly, this script uses the Site::is_local_name function to see if the queried name belongs to a local DNS zone. Underneath its using a regular expression to match any subdomains of a zone specified in local_zones. > > If the query doesn?t appear to be in a local zone, but the IP in the answer was, the script generates a Notice. > Exactly right. If you haven't already, I would say to just read the script. It's probably one of the most simple, but still useful scripts that ship with bro. Aside from some boilerplate that is common to any script that raises a notice and the event handling, the entire script is just a single if statement: if ( Site::is_local_addr(a) && # referring to a local host ! Site::is_local_name(ans$query) ) # name isn't in a local zone. # raise notice here -- - Justin Azoff From johanna at icir.org Thu Jul 20 22:06:20 2017 From: johanna at icir.org (Johanna Amann) Date: Thu, 20 Jul 2017 22:06:20 -0700 Subject: [Bro] X509 Plugin does not support OpenSSL 1.1 In-Reply-To: References: Message-ID: <20170721050620.v4pzjisv6kptahx5@Beezling.local> Hi, in case you have not solved it yourself so far - you are right, OpenSSL 1.1 is not yet supported (it is on the list). However I am a bit stumped by the fact that you encounter this on Ubuntu 16.04; 16.04 shups with OpenSSL 1.0.2 - so you should not have a problem there. Is there any chance that you compiled OpenSSL 1.1 yourself and have that laying around somewhere? Johanna On Tue, Jul 11, 2017 at 03:22:41PM +0530, Shirisha Kasarla wrote: > can anyone help with this issue > openssl should be downgraded to 1.0.0 > for compiling bro > openssl 1.1 is not compatible with bro compilation > OS- ubuntu 16.04 > The following error is occurred > > In file included from > /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, > from > /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: > /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: forward > declaration of ?EVP_PKEY {aka struct evp_pkey_st}? > typedef struct evp_pkey_st EVP_PKEY; > ^ > /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:479:25: > error: invalid use of incomplete type ?EVP_PKEY {aka struct > evp_pkey_st}? > return BN_num_bits(key->pkey.rsa->n); > ^ > In file included from > /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, > from > /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: > /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: forward > declaration of ?EVP_PKEY {aka struct evp_pkey_st}? > typedef struct evp_pkey_st EVP_PKEY; > ^ > /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:482:25: > error: invalid use of incomplete type ?EVP_PKEY {aka struct > evp_pkey_st}? > return BN_num_bits(key->pkey.dsa->p); > ^ > In file included from > /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, > from > /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: > /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: forward > declaration of ?EVP_PKEY {aka struct evp_pkey_st}? > typedef struct evp_pkey_st EVP_PKEY; > ^ > /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:492:48: > error: invalid use of incomplete type ?EVP_PKEY {aka struct > evp_pkey_st}? > const EC_GROUP *group = EC_KEY_get0_group(key->pkey.ec); > ^ > In file included from > /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, > from > /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: > /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: forward > declaration of ?EVP_PKEY {aka struct evp_pkey_st}? > typedef struct evp_pkey_st EVP_PKEY; > ^ > src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/build.make:119: > recipe for target > 'src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/X509.cc.o' > failed > make[3]: *** [src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/X509.cc.o] > Error 1 > make[3]: Leaving directory '/home/shrisha/bro-2.5/build' > CMakeFiles/Makefile2:9455: recipe for target > 'src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/all' > failed > make[2]: *** [src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/all] > Error 2 > make[2]: Leaving directory '/home/shrisha/bro-2.5/build' > Makefile:138: recipe for target 'all' failed > make[1]: *** [all] Error 2 > make[1]: Leaving directory '/home/shrisha/bro-2.5/build' > Makefile:15: recipe for target 'all' failed > make: *** [all] Error 2 > shrisha at cdesktop:~/bro-2.5$ > > > with Regards > > kasarla shirisha > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From shirisha.kasarla96 at gmail.com Fri Jul 21 00:30:43 2017 From: shirisha.kasarla96 at gmail.com (Shirisha Kasarla) Date: Fri, 21 Jul 2017 13:00:43 +0530 Subject: [Bro] X509 Plugin does not support OpenSSL 1.1 In-Reply-To: <20170721050620.v4pzjisv6kptahx5@Beezling.local> References: <20170721050620.v4pzjisv6kptahx5@Beezling.local> Message-ID: Hi, I have just checked the version of openssl On Fri, Jul 21, 2017 at 10:36 AM, Johanna Amann wrote: > Hi, > > in case you have not solved it yourself so far - you are right, OpenSSL > 1.1 is not yet supported (it is on the list). > > However I am a bit stumped by the fact that you encounter this on Ubuntu > 16.04; 16.04 shups with OpenSSL 1.0.2 - so you should not have a problem > there. Is there any chance that you compiled OpenSSL 1.1 yourself and have > that laying around somewhere? > > Johanna > > On Tue, Jul 11, 2017 at 03:22:41PM +0530, Shirisha Kasarla wrote: >> can anyone help with this issue >> openssl should be downgraded to 1.0.0 >> for compiling bro >> openssl 1.1 is not compatible with bro compilation >> OS- ubuntu 16.04 >> The following error is occurred >> >> In file included from >> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, >> from >> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: >> /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: forward >> declaration of ?EVP_PKEY {aka struct evp_pkey_st}? >> typedef struct evp_pkey_st EVP_PKEY; >> ^ >> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:479:25: >> error: invalid use of incomplete type ?EVP_PKEY {aka struct >> evp_pkey_st}? >> return BN_num_bits(key->pkey.rsa->n); >> ^ >> In file included from >> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, >> from >> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: >> /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: forward >> declaration of ?EVP_PKEY {aka struct evp_pkey_st}? >> typedef struct evp_pkey_st EVP_PKEY; >> ^ >> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:482:25: >> error: invalid use of incomplete type ?EVP_PKEY {aka struct >> evp_pkey_st}? >> return BN_num_bits(key->pkey.dsa->p); >> ^ >> In file included from >> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, >> from >> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: >> /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: forward >> declaration of ?EVP_PKEY {aka struct evp_pkey_st}? >> typedef struct evp_pkey_st EVP_PKEY; >> ^ >> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:492:48: >> error: invalid use of incomplete type ?EVP_PKEY {aka struct >> evp_pkey_st}? >> const EC_GROUP *group = EC_KEY_get0_group(key->pkey.ec); >> ^ >> In file included from >> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, >> from >> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: >> /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: forward >> declaration of ?EVP_PKEY {aka struct evp_pkey_st}? >> typedef struct evp_pkey_st EVP_PKEY; >> ^ >> src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/build.make:119: >> recipe for target >> 'src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/X509.cc.o' >> failed >> make[3]: *** [src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/X509.cc.o] >> Error 1 >> make[3]: Leaving directory '/home/shrisha/bro-2.5/build' >> CMakeFiles/Makefile2:9455: recipe for target >> 'src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/all' >> failed >> make[2]: *** [src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/all] >> Error 2 >> make[2]: Leaving directory '/home/shrisha/bro-2.5/build' >> Makefile:138: recipe for target 'all' failed >> make[1]: *** [all] Error 2 >> make[1]: Leaving directory '/home/shrisha/bro-2.5/build' >> Makefile:15: recipe for target 'all' failed >> make: *** [all] Error 2 >> shrisha at cdesktop:~/bro-2.5$ >> >> >> with Regards >> >> kasarla shirisha >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> -------------- next part -------------- A non-text attachment was scrubbed... Name: Selection_021.png Type: image/png Size: 9397 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170721/b1b35498/attachment.bin From giedrius.ramas at gmail.com Fri Jul 21 01:03:53 2017 From: giedrius.ramas at gmail.com (Giedrius Ramas) Date: Fri, 21 Jul 2017 11:03:53 +0300 Subject: [Bro] bro output log in different file names and format Message-ID: Hello, It is possible to have bro log output (the same log for example http.log) in different file names and format one ascii and another json ? For example http_ascii.log and http_json.log -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170721/0d23f274/attachment-0001.html From jan.grashoefer at gmail.com Fri Jul 21 01:22:33 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Fri, 21 Jul 2017 10:22:33 +0200 Subject: [Bro] bro output log in different file names and format In-Reply-To: References: Message-ID: <003ab889-40cf-5bef-5f7b-ac20b558d69e@gmail.com> On 21/07/17 10:03, Giedrius Ramas wrote: > Hello, > It is possible to have bro log output (the same log for example http.log) > in different file names and format one ascii and another json ? For example > http_ascii.log and http_json.log There is a package that provides more or less exactly this functionality: https://github.com/J-Gras/add-json If you have installed the Bro Package Manager: bro-pkg install add-json Jan From giedrius.ramas at gmail.com Fri Jul 21 03:05:43 2017 From: giedrius.ramas at gmail.com (Giedrius Ramas) Date: Fri, 21 Jul 2017 13:05:43 +0300 Subject: [Bro] bro output log in different file names and format In-Reply-To: <003ab889-40cf-5bef-5f7b-ac20b558d69e@gmail.com> References: <003ab889-40cf-5bef-5f7b-ac20b558d69e@gmail.com> Message-ID: Thanks, It is exactly what I need. On Fri, Jul 21, 2017 at 11:22 AM, Jan Grash?fer wrote: > On 21/07/17 10:03, Giedrius Ramas wrote: > > Hello, > > It is possible to have bro log output (the same log for example http.log) > > in different file names and format one ascii and another json ? For > example > > http_ascii.log and http_json.log > > There is a package that provides more or less exactly this > functionality: https://github.com/J-Gras/add-json > > If you have installed the Bro Package Manager: > bro-pkg install add-json > > Jan > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170721/169e6916/attachment.html From openjaf at gmail.com Fri Jul 21 09:56:07 2017 From: openjaf at gmail.com (James Feister) Date: Fri, 21 Jul 2017 12:56:07 -0400 Subject: [Bro] Bro Elastic/ELK Experiences In-Reply-To: References: Message-ID: When creating your elasticsearch template make sure you designate the version field as a string. Logstash will pick up the http.log and ssl.log json version fields as strings but ssh.log will be recognized as number. Doesn't look like you started ingesting ssh.log yet, so just a heads up. ---- JIm Feister On Thu, Jul 13, 2017 at 10:44 AM, Erich M Nahum wrote: > After some sweat, I finally have Bro integrated into the ELK stack > (Elasticsearch, > Logstash, Kibana). While there's a lot of stuff online about doing this, a > bunch of it is incomplete and/or out of date. Here is a summary of my > experience > so as to hopefully save others time. I am using the latest releases of > everything (Bro 2.5.0, ELK 5.4.4). > > I do plan to go further and use the Kafka plugin to pipe into Logstash, > but haven't gotten that far yet. > > Logstash: a lot of the examples have complex regular expression matches to > convert the default tab-separated bro logs. This is not necessary. All you > need is to emit JSON in Bro: > > @load tuning/json-logs > > then in the logstash config file set your codec to JSON: > > codec => "json" > > Timestamps: This is important to get right so that the Kibana time-based > functions > work properly. Again, a lot of examples are out of date. What worked for me > was to have Bro use ISO timestamps rather than the default Unix Epoch: > > redef LogAscii::json_timestamps = JSON::TS_ISO8601 ; > > then in the logstash filter section, add the following: > > date { > match => [ "ts", "ISO8601" ] > } > > GeoIP: This was a pain to get right, with many questions on the ELK forums, > First, you need to add a geoip to the filter section in the logstash config > file: > > geoip { > source => "id.resp_h" > target => "geoip" > } > If you use the default logstash output to elasticsearch with no changes, > this works (i.e., Kibana recognizes the geoip.location filed as a > "geo_point"). > However, all your indexes are called "logstash*", which is a pain, and > causes > problems the moment you add a new bro log type. I wanted to have an index > per > Bro log type, so I did the following: > > In the input section of logstash, I put: > > file { > start_position => "beginning" > type => "bro_conn_logs" > path => "/home/nahum/conn.log" > codec => "json" > } > > Then, in the output section of logstash, I put: > > if [type] == "bro_conn_logs" { > elasticsearch { > index => "bro-conn" > } > } > > This, however, breaks the GeoIP because it relies on the default index > template > for logstash, which defines how to map the geoip to a geo_point. The > moment you > change the index name, you lose that template mapping. So, I created a new > bro template doing the following: > > 1: Get the logstash template > > curl -XGET localhost:9200/_template/logstash?pretty > bro.template > > 2: Edit bro.template to change logstash to bro > 3: Delete all existing indexes > 4: Install the new template into Elasticsearch > > curl -XPUT 'localhost:9200/_template/bro?pretty' \ > -H 'Content-Type: application/json' -d ' ' > > where is the inline template JSON. > > 5: Reload data into ES indexes via logstash > > Note you must have no indexes before loading data, since templates > are auto-generated at index creation time. I.e., if there's no pre-existing > template for your new index, Elasticsearch will automatically create one, > which > doesn't handle the geoip properly, and thus it's too late. > So create the template BEFORE creating any new indexes. > > Thanks to Aaron Gee-Clough for answering some questions. I'm also > attaching my logstash config for reference. > > -Erich > > *(See attached file: bro-to-elastic.conf)* > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170721/04f2c55c/attachment.html From jdopheid at illinois.edu Fri Jul 21 11:57:32 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Fri, 21 Jul 2017 18:57:32 +0000 Subject: [Bro] REN-ISAC regional meeting the day before BroCon Message-ID: <09583F26-9047-4E26-B37F-F8B11D39EA57@illinois.edu> Bro community, Are you an active member in REN-ISAC and thinking about attending BroCon? REN-ISAC is hosting a regional meeting on Monday September 11th, the day before BroCon begins. More information about the event can be found here: https://www.ren-isac.net/events/midwest_regional_0917.html REN-ISAC is a members-only community, this event is not open to the public. ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From vikrambasu059 at gmail.com Fri Jul 21 23:26:39 2017 From: vikrambasu059 at gmail.com (Vikram Basu) Date: Sat, 22 Jul 2017 11:56:39 +0530 Subject: [Bro] detect-external-names.bro In-Reply-To: <8F737DA5-55D1-437B-AAF5-7BC0E50AD703@illinois.edu> References: <596f491d.9b43620a.4cb9b.06c2@mx.google.com> <8F737DA5-55D1-437B-AAF5-7BC0E50AD703@illinois.edu> Message-ID: <5972f01f.cdc0620a.d2359.c496@mx.google.com> Am I correct in saying if xyz.com points to a local IP address but a.xyz.com resolves to an external IP address then the notice is generated. So both site::local_nets and site::local_zones need to be defined externally in local.bro file using redef statements or does Bro automatically do by analysing traffic. In addition what does defining the private IP address in the networks.cfg in bro/etc folder do ? Regards Vikram Basu From: Azoff, Justin S Sent: 20 July 2017 12:50 AM To: Adam Pumphrey Cc: Vikram Basu; bro at bro.org Subject: Re: [Bro] detect-external-names.bro > On Jul 19, 2017, at 3:07 PM, Adam Pumphrey wrote: > > It checks responses to DNS A record queries for an IP address in the answer that is considered local (based on the Site::local_nets variable), but the resolved DNS name (the query) is not in a local DNS zone (based on the Site::local_zones variable). > > The IP and query locality tests in this script depend on both the Site::local_nets and Site::local_zones variables. If Site::local_zones is populated correctly, this script uses the Site::is_local_name function to see if the queried name belongs to a local DNS zone. Underneath its using a regular expression to match any subdomains of a zone specified in local_zones. > > If the query doesn?t appear to be in a local zone, but the IP in the answer was, the script generates a Notice. > Exactly right. If you haven't already, I would say to just read the script. It's probably one of the most simple, but still useful scripts that ship with bro. Aside from some boilerplate that is common to any script that raises a notice and the event handling, the entire script is just a single if statement: if ( Site::is_local_addr(a) && # referring to a local host ! Site::is_local_name(ans$query) ) # name isn't in a local zone. # raise notice here -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170722/e1429f9c/attachment.html From vikrambasu059 at gmail.com Sat Jul 22 05:56:51 2017 From: vikrambasu059 at gmail.com (Vikram Basu) Date: Sat, 22 Jul 2017 18:26:51 +0530 Subject: [Bro] Successful and failed login details Message-ID: <59734b94.9c47620a.3a00d.f3e9@mx.google.com> Is it possible to get successful and failed login details for HTTP/FTP/SSH connections using Bro IDS ? Also can it identify which user is trying to do the connections, in addition to the IP address of the machine ? Regards Vikram Basu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170722/2f6ab953/attachment.html From robin at icir.org Sat Jul 22 07:43:08 2017 From: robin at icir.org (Robin Sommer) Date: Sat, 22 Jul 2017 07:43:08 -0700 Subject: [Bro] Atomic operations on Broker store In-Reply-To: <596DDB65.80002@ssi.gouv.fr> References: <596DDB65.80002@ssi.gouv.fr> Message-ID: <20170722144308.GA99577@icir.org> Hi Bencteux, You are exactly right: the table operations aren't atomic currently (i.e., they don't happen server-side). Set manipulations are. The main difference is that for sets there are explicit store operations for inserting/removing elements, whereas for tables there are not. I don't think there's a particular reason that Broker doesn't offer table operations. We are working on a 2nd-gen Broker API currently which simplifies usage quite a bit overall. It doesn't have table operations either yet (nor set operations in fact) but I'm planing on adding them shortly. If you want to track progress there, it's in the topic/actor-framework branch. This work shouldn't take very long anymore to get ready. Robin On Tue, Jul 18, 2017 at 11:56 +0200, Bencteux Jeffrey wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Hi all, > > I am playing around with Broker framework and I am having some trouble > trying to share elements such as set or table between multiple instances > of Bro. > > What I am doing follows : > > * A master store script creates a table and add it to the store > * A frontend store script retreive the table, add elements to it and > push it back to the store > > I got strange results doing that such as only one element I was trying > to add was added (never the same) so I suspected the problem was > concurrency and that my operations were not atomic. Indeed, what happens > it that every call to my function in the slave script do not retreive an > up-to-date table. Retreiving and adding a new element to a table is not > an atomic operation and no function in the API is defined to do it in > one call. > > If I do the same test with Broker::add_to_set() with a set instead of a > table it works. And it seems that it is because it is implemented as an > atomic operation in aux/broker/src/store/frontend.cc. > > Any idea how to that with a table with existing functions? Or does it > needs further developments? If so, any hints on how to implement that? > > Here is the Bro scripts I used (debug prints removed) : > > master.bro: > > @load base/frameworks/broker > @load broker_wrapper > > const broker_port: port = 6666/tcp &redef; > redef exit_only_after_terminate = T; > > global h: opaque of Broker::Handle; > > event bro_init() > { > Broker::enable(); > Broker::listen(broker_port, "127.0.0.1"); > > h = Broker::create_master("test_store"); > > WRAPPER::broker_table_create(h, "test"); > } > > event Broker::incoming_connection_established(peer_name: string) > { > print "Conn established: ", peer_name; > } > > frontend.bro: > > @load base/frameworks/broker > @load broker_wrapper > > const broker_port: port = 6666/tcp &redef; > redef exit_only_after_terminate = T; > > global h: opaque of Broker::Handle; > > event bro_init() > { > Broker::enable(); > Broker::connect("127.0.0.1", broker_port, 1secs); > } > > event Broker::outgoing_connection_established(peer_address: string, > peer_port: port, > peer_name: string) > { > print "Conn established: ", peer_address, peer_port, peer_name; > > h = Broker::create_clone("test_store"); > > WRAPPER::broker_table_insert(h, "test", Broker::data("one"), > Broker::data("this")); > WRAPPER::broker_table_insert(h, "test", Broker::data("two"), > Broker::data("is")); > WRAPPER::broker_table_insert(h, "test", Broker::data("three"), > Broker::data("sparta")); > } > > event Broker::outgoing_connection_broken(peer_name: string, peer_port: port) > { > print "Connection closed by remote peer"; > terminate(); > } > > broker_wrapper.bro: > > module WRAPPER; > > export { > global broker_table_create: function(h: opaque of Broker::Handle, > name: string); > global broker_table_insert: function(h: opaque of Broker::Handle, > name: string, key: Broker::Data, val: Broker::Data); > } > > function broker_table_create(h: opaque of Broker::Handle, > name: string) > { > local tab = Broker::table_create(); > Broker::insert(h, Broker::data(name), tab); > } > > function broker_table_insert(h: opaque of Broker::Handle, > name: string, > key: Broker::Data, > val: Broker::Data) > { > # look for table with name 'name' > when (local res = Broker::lookup(h, Broker::data(name))) > { > # insert element "key" = "value" into table > local status = Broker::table_insert(res$result, > key, > val); > > print res$result; > > # insert table back into store > Broker::insert(h, Broker::data(name), res$result); > } > timeout 10sec > { > print fmt("timeout broker_table_insert: %s key: %s val: %s", > name, key, val); > } > } > > I also did a second test: > > If I try to add an element to a set, then test its existence in it with > Broker::set_contains() and it appears not to be in in the set. Any idea why? > > Any help is welcome! > > - -- > Jeffrey BENCTEUX > ANSSI/COSSI/DTO/BSD > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > > iQIcBAEBCAAGBQJZbdtlAAoJEHTjT1vhqZtXV/sP/jmbNea4tCqnyX7wti2co5zc > 6yL3aX7UyXHkuElzr3weKHA+oQZ3yguoQjodmigrwmBOMiFGp0bxzPvWUvYbnxWu > p7yiZZxooJNEyxkoBmauk05p79dqdhmDGLQ3kIBOGFkD0lMHx9bGPRIifEI4oDcp > hUwDi5LaEb/x//L6IPkByxxEoNrbI0HxSLS4Bl4TikUpkYbUICVsihdrlN8cPhXk > ysaO9S1jv9iJvHSOrktrhv9WuDup0HM0LDE142d/V+Dao9nu+P+HdfKd4ieRVq2O > AXFnoIp9flFksgn+PN/Y1FqF9pRWm1MUOQeUCLYTf9br7Z0z0jTR35NRogDvVFtH > at/pKuotWOozZhTPf3QnbOdpdLgXIlk6kdIqQ3ENgmkj72L8Giyn/7cEqpl0dBgB > 9iR9p6VQBuI8vsRigRUvffyC7TwonCECE+CTL9AzYCv0Zs25sTZTOw7JFMS99qav > spGY5EK6H108Amc3COWQ3Gz5IUps3K4j6OwNIbOZKyaHztx8amaF6j2BPpU+V4Ox > 2RuThAjW3kOECr2/fseN6HYR2AX7lNCn+HJqWHS+C+pbdOBUhGWucwYj+R+28u+h > PVlWwYxcSb0ezZxckX3w32oi0I0YkX9ejLtxdLMd2L9XY4S1nRhxeFHnESqx51gi > y6Uh6WaD+yfu9o4IXIUL > =6UD8 > -----END PGP SIGNATURE----- > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From bill.de.ping at gmail.com Sun Jul 23 06:19:44 2017 From: bill.de.ping at gmail.com (william de ping) Date: Sun, 23 Jul 2017 16:19:44 +0300 Subject: [Bro] cluster VS single instance - different results Message-ID: Hi all, I know issues related to this exists out there, but I havent manage to fix it yet.. The issue is that a single bro instance produces relevant logs upon listening to interface eth0. Yet broctl configured with a single worker that is listening to the same interface, running the same scripts, fail to produce relevant log files. I tried setting ignore_checksum = T and I have tried to run bro worker the way broctl runs it. Can anyone think of an explanation for this issue ? Both single instance and single worker have the same additional scripts.. Thanks alot B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170723/18d62b5c/attachment.html From seth at corelight.com Sun Jul 23 12:58:10 2017 From: seth at corelight.com (Seth Hall) Date: Sun, 23 Jul 2017 19:58:10 +0000 Subject: [Bro] cluster VS single instance - different results In-Reply-To: References: Message-ID: On Sun, Jul 23, 2017 at 9:31 AM william de ping wrote: > The issue is that a single bro instance produces relevant logs upon > listening to interface eth0. > Yet broctl configured with a single worker that is listening to the same > interface, running the same scripts, fail to produce relevant log files. > Hm, I'm not sure what could be causing that. Could you show the output of "ps auwwxf" while you have it running with broctl? .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170723/e3e929a4/attachment.html From seth at corelight.com Sun Jul 23 13:02:30 2017 From: seth at corelight.com (Seth Hall) Date: Sun, 23 Jul 2017 20:02:30 +0000 Subject: [Bro] Successful and failed login details In-Reply-To: <59734b94.9c47620a.3a00d.f3e9@mx.google.com> References: <59734b94.9c47620a.3a00d.f3e9@mx.google.com> Message-ID: On Sat, Jul 22, 2017 at 9:08 AM Vikram Basu wrote: > Is it possible to get successful and failed login details for HTTP/FTP/SSH > connections using Bro IDS ? Also can it identify which user is trying to do > the connections, in addition to the IP address of the machine ? > It is possible, but at the moment you will need to do it in a less-than-pleasant way. You would do it by finding the events for each of the relevant protocols where the data is available. I've been hoping to find some time to get a version of the long discussed "authentication framework" into 2.6. Once that's available you would be able to access authentication information directly through there as an abstraction. .Seth -- Seth Hall * Corelight, Inc * seth at corelight.com * www.corelight.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170723/b8f2aff5/attachment.html From johanna at icir.org Sun Jul 23 21:50:30 2017 From: johanna at icir.org (Johanna Amann) Date: Sun, 23 Jul 2017 21:50:30 -0700 Subject: [Bro] X509 Plugin does not support OpenSSL 1.1 In-Reply-To: References: <20170721050620.v4pzjisv6kptahx5@Beezling.local> Message-ID: Hi Shirisha, it should actually compile cleanly against that version of OpenSSL. Which version of openssl does the output of configure find, and what paths does it show? Johanna On 21 Jul 2017, at 0:30, Shirisha Kasarla wrote: > Hi, > > I have just checked the version of openssl > > > On Fri, Jul 21, 2017 at 10:36 AM, Johanna Amann > wrote: >> Hi, >> >> in case you have not solved it yourself so far - you are right, >> OpenSSL >> 1.1 is not yet supported (it is on the list). >> >> However I am a bit stumped by the fact that you encounter this on >> Ubuntu >> 16.04; 16.04 shups with OpenSSL 1.0.2 - so you should not have a >> problem >> there. Is there any chance that you compiled OpenSSL 1.1 yourself and >> have >> that laying around somewhere? >> >> Johanna >> >> On Tue, Jul 11, 2017 at 03:22:41PM +0530, Shirisha Kasarla wrote: >>> can anyone help with this issue >>> openssl should be downgraded to 1.0.0 >>> for compiling bro >>> openssl 1.1 is not compatible with bro compilation >>> OS- ubuntu 16.04 >>> The following error is occurred >>> >>> In file included from >>> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, >>> from >>> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: >>> /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: >>> forward >>> declaration of ?EVP_PKEY {aka struct evp_pkey_st}? >>> typedef struct evp_pkey_st EVP_PKEY; >>> ^ >>> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:479:25: >>> error: invalid use of incomplete type ?EVP_PKEY {aka struct >>> evp_pkey_st}? >>> return BN_num_bits(key->pkey.rsa->n); >>> ^ >>> In file included from >>> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, >>> from >>> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: >>> /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: >>> forward >>> declaration of ?EVP_PKEY {aka struct evp_pkey_st}? >>> typedef struct evp_pkey_st EVP_PKEY; >>> ^ >>> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:482:25: >>> error: invalid use of incomplete type ?EVP_PKEY {aka struct >>> evp_pkey_st}? >>> return BN_num_bits(key->pkey.dsa->p); >>> ^ >>> In file included from >>> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, >>> from >>> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: >>> /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: >>> forward >>> declaration of ?EVP_PKEY {aka struct evp_pkey_st}? >>> typedef struct evp_pkey_st EVP_PKEY; >>> ^ >>> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:492:48: >>> error: invalid use of incomplete type ?EVP_PKEY {aka struct >>> evp_pkey_st}? >>> const EC_GROUP *group = EC_KEY_get0_group(key->pkey.ec); >>> ^ >>> In file included from >>> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.h:9:0, >>> from >>> /home/shrisha/bro-2.5/src/file_analysis/analyzer/x509/X509.cc:5: >>> /home/shrisha/bro-2.5/src/file_analysis/../File.h:17:16: note: >>> forward >>> declaration of ?EVP_PKEY {aka struct evp_pkey_st}? >>> typedef struct evp_pkey_st EVP_PKEY; >>> ^ >>> src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/build.make:119: >>> recipe for target >>> 'src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/X509.cc.o' >>> failed >>> make[3]: *** >>> [src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/X509.cc.o] >>> Error 1 >>> make[3]: Leaving directory '/home/shrisha/bro-2.5/build' >>> CMakeFiles/Makefile2:9455: recipe for target >>> 'src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/all' >>> failed >>> make[2]: *** >>> [src/file_analysis/analyzer/x509/CMakeFiles/plugin-Bro-X509.dir/all] >>> Error 2 >>> make[2]: Leaving directory '/home/shrisha/bro-2.5/build' >>> Makefile:138: recipe for target 'all' failed >>> make[1]: *** [all] Error 2 >>> make[1]: Leaving directory '/home/shrisha/bro-2.5/build' >>> Makefile:15: recipe for target 'all' failed >>> make: *** [all] Error 2 >>> shrisha at cdesktop:~/bro-2.5$ >>> >>> >>> with Regards >>> >>> kasarla shirisha >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> From apumphrey at bricata.com Mon Jul 24 06:49:07 2017 From: apumphrey at bricata.com (Adam Pumphrey) Date: Mon, 24 Jul 2017 13:49:07 +0000 Subject: [Bro] detect-external-names.bro In-Reply-To: <5972f01f.cdc0620a.d2359.c496@mx.google.com> References: <596f491d.9b43620a.4cb9b.06c2@mx.google.com> <8F737DA5-55D1-437B-AAF5-7BC0E50AD703@illinois.edu> <5972f01f.cdc0620a.d2359.c496@mx.google.com> Message-ID: <7CDC318E-50F7-422B-8839-1024C762DC83@bricata.com> It?s actually kind of the other way around, a Notice would only be generated if a.xyz.com is not a local domain name (in other words ?xyz.com? is not in Site::local_zones) however Bro sees a DNS query/response where that name resolved to a local IP address. Think rogue DNS? if a Notice is generated by this script, you?re likely seeing an unauthorized DNS server, using a DNS zone you don?t own or manage, resolve A record queries to your local IP addresses. Site::local_zones and Site::local_nets must be set manually, Bro won?t do this by analyzing traffic. However, if you populate networks.cfg, Bro will set Site::local_nets for you when it starts up. From: Vikram Basu Date: Saturday, July 22, 2017 at 2:26 AM To: "Azoff, Justin S" , Adam Pumphrey Cc: "bro at bro.org" Subject: RE: [Bro] detect-external-names.bro Am I correct in saying if xyz.com points to a local IP address but a.xyz.com resolves to an external IP address then the notice is generated. So both site::local_nets and site::local_zones need to be defined externally in local.bro file using redef statements or does Bro automatically do by analysing traffic. In addition what does defining the private IP address in the networks.cfg in bro/etc folder do ? Regards Vikram Basu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170724/d94d7f92/attachment-0001.html From craig.edgmand at okstate.edu Mon Jul 24 08:00:16 2017 From: craig.edgmand at okstate.edu (Edgmand, Craig) Date: Mon, 24 Jul 2017 15:00:16 +0000 Subject: [Bro] Bro Elastic/ELK Experiences In-Reply-To: References: Message-ID: For those of you using ELK for Bro how many are using a message broker like RabbitMQ or Apache Kafka? Why did you choose the message broker you did? Thanks, Craig Edgmand IT Security Oklahoma State University From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of James Feister Sent: Friday, July 21, 2017 11:56 AM To: Erich M Nahum Cc: bro at bro.org Subject: Re: [Bro] Bro Elastic/ELK Experiences When creating your elasticsearch template make sure you designate the version field as a string. Logstash will pick up the http.log and ssl.log json version fields as strings but ssh.log will be recognized as number. Doesn't look like you started ingesting ssh.log yet, so just a heads up. ---- JIm Feister On Thu, Jul 13, 2017 at 10:44 AM, Erich M Nahum > wrote: After some sweat, I finally have Bro integrated into the ELK stack (Elasticsearch, Logstash, Kibana). While there's a lot of stuff online about doing this, a bunch of it is incomplete and/or out of date. Here is a summary of my experience so as to hopefully save others time. I am using the latest releases of everything (Bro 2.5.0, ELK 5.4.4). I do plan to go further and use the Kafka plugin to pipe into Logstash, but haven't gotten that far yet. Logstash: a lot of the examples have complex regular expression matches to convert the default tab-separated bro logs. This is not necessary. All you need is to emit JSON in Bro: @load tuning/json-logs then in the logstash config file set your codec to JSON: codec => "json" Timestamps: This is important to get right so that the Kibana time-based functions work properly. Again, a lot of examples are out of date. What worked for me was to have Bro use ISO timestamps rather than the default Unix Epoch: redef LogAscii::json_timestamps = JSON::TS_ISO8601 ; then in the logstash filter section, add the following: date { match => [ "ts", "ISO8601" ] } GeoIP: This was a pain to get right, with many questions on the ELK forums, First, you need to add a geoip to the filter section in the logstash config file: geoip { source => "id.resp_h" target => "geoip" } If you use the default logstash output to elasticsearch with no changes, this works (i.e., Kibana recognizes the geoip.location filed as a "geo_point"). However, all your indexes are called "logstash*", which is a pain, and causes problems the moment you add a new bro log type. I wanted to have an index per Bro log type, so I did the following: In the input section of logstash, I put: file { start_position => "beginning" type => "bro_conn_logs" path => "/home/nahum/conn.log" codec => "json" } Then, in the output section of logstash, I put: if [type] == "bro_conn_logs" { elasticsearch { index => "bro-conn" } } This, however, breaks the GeoIP because it relies on the default index template for logstash, which defines how to map the geoip to a geo_point. The moment you change the index name, you lose that template mapping. So, I created a new bro template doing the following: 1: Get the logstash template curl -XGET localhost:9200/_template/logstash?pretty > bro.template 2: Edit bro.template to change logstash to bro 3: Delete all existing indexes 4: Install the new template into Elasticsearch curl -XPUT 'localhost:9200/_template/bro?pretty' \ -H 'Content-Type: application/json' -d ' ' where is the inline template JSON. 5: Reload data into ES indexes via logstash Note you must have no indexes before loading data, since templates are auto-generated at index creation time. I.e., if there's no pre-existing template for your new index, Elasticsearch will automatically create one, which doesn't handle the geoip properly, and thus it's too late. So create the template BEFORE creating any new indexes. Thanks to Aaron Gee-Clough for answering some questions. I'm also attaching my logstash config for reference. -Erich (See attached file: bro-to-elastic.conf) _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170724/c484437d/attachment.html From landy-bible at utulsa.edu Mon Jul 24 09:25:12 2017 From: landy-bible at utulsa.edu (Landy Bible) Date: Mon, 24 Jul 2017 16:25:12 +0000 Subject: [Bro] Bro Elastic/ELK Experiences In-Reply-To: References: Message-ID: Craig, I'm currently Redis, but I'm migrating to Kafka. I initially chose Redis because it was super easy to set up. It's worked well for me so far, but it's limited by the amount of ram I have in my servers. I currently run 3 servers, but they aren't actually clustered. Logstash runs on my Bro server to forward the logs to Redis, and will fail over in the event that one of my Redis servers goes down. I had to write some scripts to block incoming writes if the node got too full while still allowing the Logstash agents on the other side to read from the queue. When I'm doing ELK maintenance I have to keep a close eye on my Redis boxes, I've only got a few hours worth of headroom on them, and if they fill up they'll crash and I lose data. Also, Redis operates as a single queue. I can't have multiple processes consuming the same data for different uses. Migrating to Kafka will allow me to do real clustering at the queuing layer. Since it writes to disk I have days or weeks worth of storage rather than hours, so my ELK maintenance won't be so stressful. And since it has the concept of consumer offsets I can have the ELK system consuming the data, as well as my other custom processes consume it for other uses without impacting ELK. The downside is that it's a bit more complex to get a Kafka cluster up and running than it is to get a single Redis node up. I'm only testing it right now but so far it seems like it's worth the effort. If you're ever in the Tulsa area, hit me up. I'd be happy to show you some details about how we're running things at TU. -Landy On Mon, Jul 24, 2017 at 10:02 AM Edgmand, Craig wrote: > For those of you using ELK for Bro how many are using a message broker > like RabbitMQ or Apache Kafka? Why did you choose the message broker you > did? > > Thanks, > > > > Craig Edgmand > > IT Security > > Oklahoma State University > > > > *From:* bro-bounces at bro.org [mailto:bro-bounces at bro.org] *On Behalf Of *James > Feister > *Sent:* Friday, July 21, 2017 11:56 AM > *To:* Erich M Nahum > *Cc:* bro at bro.org > *Subject:* Re: [Bro] Bro Elastic/ELK Experiences > > > > When creating your elasticsearch template make sure you designate the > version field as a string. Logstash will pick up the http.log and ssl.log > json version fields as strings but ssh.log will be recognized as number. > > > > Doesn't look like you started ingesting ssh.log yet, so just a heads up. > > > > ---- > > JIm Feister > > > > > > On Thu, Jul 13, 2017 at 10:44 AM, Erich M Nahum wrote: > > After some sweat, I finally have Bro integrated into the ELK stack > (Elasticsearch, > Logstash, Kibana). While there's a lot of stuff online about doing this, a > bunch of it is incomplete and/or out of date. Here is a summary of my > experience > so as to hopefully save others time. I am using the latest releases of > everything (Bro 2.5.0, ELK 5.4.4). > > I do plan to go further and use the Kafka plugin to pipe into Logstash, > but haven't gotten that far yet. > > Logstash: a lot of the examples have complex regular expression matches to > convert the default tab-separated bro logs. This is not necessary. All you > need is to emit JSON in Bro: > > @load tuning/json-logs > > then in the logstash config file set your codec to JSON: > > codec => "json" > > Timestamps: This is important to get right so that the Kibana time-based > functions > work properly. Again, a lot of examples are out of date. What worked for me > was to have Bro use ISO timestamps rather than the default Unix Epoch: > > redef LogAscii::json_timestamps = JSON::TS_ISO8601 ; > > then in the logstash filter section, add the following: > > date { > match => [ "ts", "ISO8601" ] > } > > GeoIP: This was a pain to get right, with many questions on the ELK forums, > First, you need to add a geoip to the filter section in the logstash config > file: > > geoip { > source => "id.resp_h" > target => "geoip" > } > If you use the default logstash output to elasticsearch with no changes, > this works (i.e., Kibana recognizes the geoip.location filed as a > "geo_point"). > However, all your indexes are called "logstash*", which is a pain, and > causes > problems the moment you add a new bro log type. I wanted to have an index > per > Bro log type, so I did the following: > > In the input section of logstash, I put: > > file { > start_position => "beginning" > type => "bro_conn_logs" > path => "/home/nahum/conn.log" > codec => "json" > } > > Then, in the output section of logstash, I put: > > if [type] == "bro_conn_logs" { > elasticsearch { > index => "bro-conn" > } > } > > This, however, breaks the GeoIP because it relies on the default index > template > for logstash, which defines how to map the geoip to a geo_point. The > moment you > change the index name, you lose that template mapping. So, I created a new > bro template doing the following: > > 1: Get the logstash template > > curl -XGET localhost:9200/_template/logstash?pretty > bro.template > > 2: Edit bro.template to change logstash to bro > 3: Delete all existing indexes > 4: Install the new template into Elasticsearch > > curl -XPUT 'localhost:9200/_template/bro?pretty' \ > -H 'Content-Type: application/json' -d ' ' > > where is the inline template JSON. > > 5: Reload data into ES indexes via logstash > > Note you must have no indexes before loading data, since templates > are auto-generated at index creation time. I.e., if there's no pre-existing > template for your new index, Elasticsearch will automatically create one, > which > doesn't handle the geoip properly, and thus it's too late. > So create the template BEFORE creating any new indexes. > > Thanks to Aaron Gee-Clough for answering some questions. I'm also > attaching my logstash config for reference. > > -Erich > > *(See attached file: bro-to-elastic.conf)* > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Landy Bible Information Security Analyst The University of Tulsa -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170724/4d8a20f5/attachment-0001.html From jazoff at illinois.edu Mon Jul 24 09:52:12 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 24 Jul 2017 16:52:12 +0000 Subject: [Bro] Bro Elastic/ELK Experiences In-Reply-To: References: Message-ID: > On Jul 24, 2017, at 12:25 PM, Landy Bible wrote: > > Craig, > > I'm currently Redis, but I'm migrating to Kafka. > > I initially chose Redis because it was super easy to set up. It's worked well for me so far, but it's limited by the amount of ram I have in my servers. I currently run 3 servers, but they aren't actually clustered. Logstash runs on my Bro server to forward the logs to Redis, and will fail over in the event that one of my Redis servers goes down. I had to write some scripts to block incoming writes if the node got too full while still allowing the Logstash agents on the other side to read from the queue. When I'm doing ELK maintenance I have to keep a close eye on my Redis boxes, I've only got a few hours worth of headroom on them, and if they fill up they'll crash and I lose data. Also, Redis operates as a single queue. I can't have multiple processes consuming the same data for different uses. > > Migrating to Kafka will allow me to do real clustering at the queuing layer. Since it writes to disk I have days or weeks worth of storage rather than hours, so my ELK maintenance won't be so stressful. And since it has the concept of consumer offsets I can have the ELK system consuming the data, as well as my other custom processes consume it for other uses without impacting ELK. The downside is that it's a bit more complex to get a Kafka cluster up and running than it is to get a single Redis node up. I'm only testing it right now but so far it seems like it's worth the effort. > > If you're ever in the Tulsa area, hit me up. I'd be happy to show you some details about how we're running things at TU. You should also consider http://nsq.io/ as a replacement for redis instead of kafka. Log stash or the bro elasticsearch plugin can write to it. It's a lot simpler to run compared to kafka. -- - Justin Azoff From mus3 at lehigh.edu Mon Jul 24 10:03:18 2017 From: mus3 at lehigh.edu (Munroe Sollog) Date: Mon, 24 Jul 2017 12:03:18 -0500 Subject: [Bro] Bro Elastic/ELK Experiences In-Reply-To: References: Message-ID: <17E2A4CB-5B0D-4F6C-B0E3-0BEACFE04C25@lehigh.edu> I agree with this. I am running NSQ on the bro box and using log stash to pull from it and push to ES. Works like a charm (10+Gbps of traffic) > On Jul 24, 2017, at 11:52 AM, Azoff, Justin S wrote: > >> On Jul 24, 2017, at 12:25 PM, Landy Bible wrote: >> >> Craig, >> >> I'm currently Redis, but I'm migrating to Kafka. >> >> I initially chose Redis because it was super easy to set up. It's worked well for me so far, but it's limited by the amount of ram I have in my servers. I currently run 3 servers, but they aren't actually clustered. Logstash runs on my Bro server to forward the logs to Redis, and will fail over in the event that one of my Redis servers goes down. I had to write some scripts to block incoming writes if the node got too full while still allowing the Logstash agents on the other side to read from the queue. When I'm doing ELK maintenance I have to keep a close eye on my Redis boxes, I've only got a few hours worth of headroom on them, and if they fill up they'll crash and I lose data. Also, Redis operates as a single queue. I can't have multiple processes consuming the same data for different uses. >> >> Migrating to Kafka will allow me to do real clustering at the queuing layer. Since it writes to disk I have days or weeks worth of storage rather than hours, so my ELK maintenance won't be so stressful. And since it has the concept of consumer offsets I can have the ELK system consuming the data, as well as my other custom processes consume it for other uses without impacting ELK. The downside is that it's a bit more complex to get a Kafka cluster up and running than it is to get a single Redis node up. I'm only testing it right now but so far it seems like it's worth the effort. >> >> If you're ever in the Tulsa area, hit me up. I'd be happy to show you some details about how we're running things at TU. > > You should also consider http://nsq.io/ as a replacement for redis instead of kafka. > > Log stash or the bro elasticsearch plugin can write to it. It's a lot simpler to run compared to kafka. > > -- > - Justin Azoff > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170724/acd4831e/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170724/acd4831e/attachment.bin From landy-bible at utulsa.edu Mon Jul 24 11:03:35 2017 From: landy-bible at utulsa.edu (Landy Bible) Date: Mon, 24 Jul 2017 18:03:35 +0000 Subject: [Bro] Bro Elastic/ELK Experiences In-Reply-To: References: Message-ID: The timing of your email was perfect. As it turns out last week I was preparing a presentation on using Bro with ELK. Your email helped me nail down parts of my demo configuration. I still need to integrate your geoip mapping. If anyone is interested I posted my own setup guide and presentation on GitHub. https://github.com/ljb2of3/techfest2017 -Landy On Thu, Jul 13, 2017 at 9:46 AM Erich M Nahum wrote: > After some sweat, I finally have Bro integrated into the ELK stack > (Elasticsearch, > Logstash, Kibana). While there's a lot of stuff online about doing this, a > bunch of it is incomplete and/or out of date. Here is a summary of my > experience > so as to hopefully save others time. I am using the latest releases of > everything (Bro 2.5.0, ELK 5.4.4). > > I do plan to go further and use the Kafka plugin to pipe into Logstash, > but haven't gotten that far yet. > > Logstash: a lot of the examples have complex regular expression matches to > convert the default tab-separated bro logs. This is not necessary. All you > need is to emit JSON in Bro: > > @load tuning/json-logs > > then in the logstash config file set your codec to JSON: > > codec => "json" > > Timestamps: This is important to get right so that the Kibana time-based > functions > work properly. Again, a lot of examples are out of date. What worked for me > was to have Bro use ISO timestamps rather than the default Unix Epoch: > > redef LogAscii::json_timestamps = JSON::TS_ISO8601 ; > > then in the logstash filter section, add the following: > > date { > match => [ "ts", "ISO8601" ] > } > > GeoIP: This was a pain to get right, with many questions on the ELK forums, > First, you need to add a geoip to the filter section in the logstash config > file: > > geoip { > source => "id.resp_h" > target => "geoip" > } > If you use the default logstash output to elasticsearch with no changes, > this works (i.e., Kibana recognizes the geoip.location filed as a > "geo_point"). > However, all your indexes are called "logstash*", which is a pain, and > causes > problems the moment you add a new bro log type. I wanted to have an index > per > Bro log type, so I did the following: > > In the input section of logstash, I put: > > file { > start_position => "beginning" > type => "bro_conn_logs" > path => "/home/nahum/conn.log" > codec => "json" > } > > Then, in the output section of logstash, I put: > > if [type] == "bro_conn_logs" { > elasticsearch { > index => "bro-conn" > } > } > > This, however, breaks the GeoIP because it relies on the default index > template > for logstash, which defines how to map the geoip to a geo_point. The > moment you > change the index name, you lose that template mapping. So, I created a new > bro template doing the following: > > 1: Get the logstash template > > curl -XGET localhost:9200/_template/logstash?pretty > bro.template > > 2: Edit bro.template to change logstash to bro > 3: Delete all existing indexes > 4: Install the new template into Elasticsearch > > curl -XPUT 'localhost:9200/_template/bro?pretty' \ > -H 'Content-Type: application/json' -d ' ' > > where is the inline template JSON. > > 5: Reload data into ES indexes via logstash > > Note you must have no indexes before loading data, since templates > are auto-generated at index creation time. I.e., if there's no pre-existing > template for your new index, Elasticsearch will automatically create one, > which > doesn't handle the geoip properly, and thus it's too late. > So create the template BEFORE creating any new indexes. > > Thanks to Aaron Gee-Clough for answering some questions. I'm also > attaching my logstash config for reference. > > -Erich > > *(See attached file: bro-to-elastic.conf)* > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Landy Bible Information Security Analyst The University of Tulsa -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170724/9b90274d/attachment-0001.html From fatema.bannatwala at gmail.com Tue Jul 25 06:12:50 2017 From: fatema.bannatwala at gmail.com (fatema bannatwala) Date: Tue, 25 Jul 2017 09:12:50 -0400 Subject: [Bro] Split-ed connection for some UDP traffic? Message-ID: Hey all, Just had a recent case where we started to see in the bro conn logs traffic originating from src port 389 for some of the systems, and I scratching my head thinking why would the ldap server initiate the UDP "connection" (I know that's not a correct term to use here), looking more into the logs, realized that, it is actually the response from the server, that Bro is logging in a complete new connection, for example: 1500927487.398576 CLr9ebnHeAYNOGzei 24.132.204.62 41600 128.175.235.216 389 udp - 93.677712 39999 0 S0 F T 0 D 597 56715 0 0 (empty) 1500927487.404591 CapBfs1lhI2XFt4gJb 128.175.235.216 389 24.132.204.62 41600 udp - 93.672242 1773687 0 S0 T F 0 D 597 1790403 0 0 (empty) Here, in the above case, shouldn't Bro be logging only a single connection with src: 24.132.204.62 and dest: 128.175.235.216, with History 'Dd' ? or I might be missing something important here :) Thanks, Fatema. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170725/aff40d27/attachment.html From zeolla at gmail.com Tue Jul 25 06:34:59 2017 From: zeolla at gmail.com (Zeolla@GMail.com) Date: Tue, 25 Jul 2017 13:34:59 +0000 Subject: [Bro] Bro Elastic/ELK Experiences In-Reply-To: References: Message-ID: I used to run nsq but we moved to Kafka. The biggest reason why is because of our heavy use of Apache Metron, which leverages Kafka and handles insertion into ES after normalization, enrichment, threat triage, etc. Jon On Mon, Jul 24, 2017, 11:08 AM Landy Bible wrote: > The timing of your email was perfect. As it turns out last week I was > preparing a presentation on using Bro with ELK. Your email helped me nail > down parts of my demo configuration. I still need to integrate your geoip > mapping. > > If anyone is interested I posted my own setup guide and presentation on > GitHub. > > https://github.com/ljb2of3/techfest2017 > > -Landy > > On Thu, Jul 13, 2017 at 9:46 AM Erich M Nahum wrote: > >> After some sweat, I finally have Bro integrated into the ELK stack >> (Elasticsearch, >> Logstash, Kibana). While there's a lot of stuff online about doing this, a >> bunch of it is incomplete and/or out of date. Here is a summary of my >> experience >> so as to hopefully save others time. I am using the latest releases of >> everything (Bro 2.5.0, ELK 5.4.4). >> >> I do plan to go further and use the Kafka plugin to pipe into Logstash, >> but haven't gotten that far yet. >> >> Logstash: a lot of the examples have complex regular expression matches to >> convert the default tab-separated bro logs. This is not necessary. All you >> need is to emit JSON in Bro: >> >> @load tuning/json-logs >> >> then in the logstash config file set your codec to JSON: >> >> codec => "json" >> >> Timestamps: This is important to get right so that the Kibana time-based >> functions >> work properly. Again, a lot of examples are out of date. What worked for >> me >> was to have Bro use ISO timestamps rather than the default Unix Epoch: >> >> redef LogAscii::json_timestamps = JSON::TS_ISO8601 ; >> >> then in the logstash filter section, add the following: >> >> date { >> match => [ "ts", "ISO8601" ] >> } >> >> GeoIP: This was a pain to get right, with many questions on the ELK >> forums, >> First, you need to add a geoip to the filter section in the logstash >> config >> file: >> >> geoip { >> source => "id.resp_h" >> target => "geoip" >> } >> If you use the default logstash output to elasticsearch with no changes, >> this works (i.e., Kibana recognizes the geoip.location filed as a >> "geo_point"). >> However, all your indexes are called "logstash*", which is a pain, and >> causes >> problems the moment you add a new bro log type. I wanted to have an index >> per >> Bro log type, so I did the following: >> >> In the input section of logstash, I put: >> >> file { >> start_position => "beginning" >> type => "bro_conn_logs" >> path => "/home/nahum/conn.log" >> codec => "json" >> } >> >> Then, in the output section of logstash, I put: >> >> if [type] == "bro_conn_logs" { >> elasticsearch { >> index => "bro-conn" >> } >> } >> >> This, however, breaks the GeoIP because it relies on the default index >> template >> for logstash, which defines how to map the geoip to a geo_point. The >> moment you >> change the index name, you lose that template mapping. So, I created a new >> bro template doing the following: >> >> 1: Get the logstash template >> >> curl -XGET localhost:9200/_template/logstash?pretty > bro.template >> >> 2: Edit bro.template to change logstash to bro >> 3: Delete all existing indexes >> 4: Install the new template into Elasticsearch >> >> curl -XPUT 'localhost:9200/_template/bro?pretty' \ >> -H 'Content-Type: application/json' -d ' ' >> >> where is the inline template JSON. >> >> 5: Reload data into ES indexes via logstash >> >> Note you must have no indexes before loading data, since templates >> are auto-generated at index creation time. I.e., if there's no >> pre-existing >> template for your new index, Elasticsearch will automatically create one, >> which >> doesn't handle the geoip properly, and thus it's too late. >> So create the template BEFORE creating any new indexes. >> >> Thanks to Aaron Gee-Clough for answering some questions. I'm also >> attaching my logstash config for reference. >> >> -Erich >> >> *(See attached file: bro-to-elastic.conf)* >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -- > Landy Bible > Information Security Analyst > The University of Tulsa > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170725/22420757/attachment.html From jdopheid at illinois.edu Tue Jul 25 11:54:04 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Tue, 25 Jul 2017 18:54:04 +0000 Subject: [Bro] =?utf-8?q?BroCon_=E2=80=9917=3A_Agenda_is_posted!?= Message-ID: <131877B7-6BCE-4D08-8CF6-AFFEA7BEFC7D@illinois.edu> The BroCon agenda is now available: https://www.bro.org/community/brocon2017.html#agenda (Note: the agenda is subject to change.) And, we?ve added a lightning talk session to the agenda this year, see our abstract below: Have you contributed a package to the new Bro Package Manager? Or do you have something interesting to share related to Bro that doesn?t fit into a traditional presentation? Great! We?d love to hear from you. We have scheduled a 45-minute session for lightning talks. Be prepared to quickly identify your point, demonstrate it, and provide a link or contact info for later follow-up. We?d like to accommodate as many talks as possible so please limit your talk to less than 5 minutes. No product presentations. Early bird registration ends Sunday night (Central time), register now: https://brocon2017.eventbrite.com/ ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign From robin at icir.org Tue Jul 25 17:30:22 2017 From: robin at icir.org (Robin Sommer) Date: Tue, 25 Jul 2017 17:30:22 -0700 Subject: [Bro] Atomic operations on Broker store In-Reply-To: <20170722144308.GA99577@icir.org> References: <596DDB65.80002@ssi.gouv.fr> <20170722144308.GA99577@icir.org> Message-ID: <20170726003022.GA31936@icir.org> On Sat, Jul 22, 2017 at 07:43 -0700, I wrote: > We are working on a 2nd-gen Broker API currently which simplifies > usage quite a bit overall. It doesn't have table operations either yet I stand corrected: it actually did have them already, the API just made it easy to miss. I've now added explicit methods for set/table/vector manipulations to the store API in that branch branch (Note that they aren't accessible from the Bro side yet, that'll come shortly). Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From Izik.Birka at hot.net.il Wed Jul 26 00:00:06 2017 From: Izik.Birka at hot.net.il (Izik Birka) Date: Wed, 26 Jul 2017 07:00:06 +0000 Subject: [Bro] no intel.log | critical-stack intel Message-ID: <592228F4D0C8504187F2F76658040CB6E0120C6D@HOT-MAILBOX-02.HOT.NET.IL> Hi I installed the critical-stack agent I pulled the feeds and the master file was created successfully But when I trying to test it , and connect to forbidden address , I'm not getting the log in the intel.log like I should Actually there is no file "intel.log" What am I missing ? I was following this article https://intel.criticalstack.com/client/0-4-x/usage [Enjoy] ????? ????? ??? ???? ????? ???? ?????? ???? ????? ?????? ???? 077-7077790 | 053-6064571 P ???? ?? ?????? ???? ?????? ???? ?? [Enjoy] This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain materials protected by copyright or information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or agreement. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication by error, notify the sender immediately and delete this message immediately. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170726/eca7edf1/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 18831 bytes Desc: image001.jpg Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170726/eca7edf1/attachment-0002.jpg -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 49161 bytes Desc: image002.jpg Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170726/eca7edf1/attachment-0003.jpg From kmidwinter at exoendo.com Wed Jul 26 06:38:02 2017 From: kmidwinter at exoendo.com (Keith Midwinter) Date: Wed, 26 Jul 2017 13:38:02 +0000 Subject: [Bro] Adding dns entry to bro logs Message-ID: Hi, I am using bro internally on a network that uses dhcp to assign ip addresses so if I want to investigate something that happened yesterday then doing a nslookup today wont tell me what host it was assigned to at the time the log was created. So is there a way to do an nslookup at the time of log creation and add it to the logs? I did some googling and found a reference to extending the log format and running scripts but it wasn't enough for me to figure it out. Thanks, Keith -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170726/c6ed4e4a/attachment.html From reswob10 at gmail.com Wed Jul 26 08:26:46 2017 From: reswob10 at gmail.com (craig bowser) Date: Wed, 26 Jul 2017 11:26:46 -0400 Subject: [Bro] Bro IDS on CentOS 6.x Message-ID: I'm trying to install Bro on CentOS 6.9. When I'm trying to configure I get an error complaining that gcc needs to be version 4.8 or above. CentOS 6.9 comes with 4.7.7 and the main repo does not have 4.8. And you are not supposed to upgrade the 6.x working environment to 4.8. So I followed this forum post (of many similar) to create an environment where 4.8 is installed: https://gist.github.com/stephenturner/e3bc5cfacc2dc67eca8b # sudo yum install centos-release-scl # sudo yum install devtoolset-3-toolchain # scl enable devtoolset-3 bash # gcc --version gcc (GCC) 4.9.2 20150212 (Red Hat 4.9.2-6) Copyright (C) 2014 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. But when I run ./configure, it quits with this error: CMake Error at cmake/RequireCXX11.cmake:40 (message): GCC version must be at least 4.8 for C++11 support, detected: 4.4.7 Call Stack (most recent call first): CMakeLists.txt:168 (include) -- Configuring incomplete, errors occurred! See also "/opt/bro-2.5.1/build/CMakeFiles/CMakeOutput.log". See also "/opt/bro-2.5.1/build/CMakeFiles/CMakeError.log". which seems like it running in the original environment not in the scl devtoolset-3 environment. Besides upgrading to CentOS 7.x (because enterprise standards), any suggestions? Craig L Bowser ____________________________ This email is measured by size. Bits and bytes may have settled during transport. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170726/97c2fc00/attachment.html From jazoff at illinois.edu Wed Jul 26 08:38:25 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 26 Jul 2017 15:38:25 +0000 Subject: [Bro] Bro IDS on CentOS 6.x In-Reply-To: References: Message-ID: <9C771D5E-8F1A-47C7-811D-CCE967440723@illinois.edu> > On Jul 26, 2017, at 11:26 AM, craig bowser wrote: > > > I'm trying to install Bro on CentOS 6.9. > > When I'm trying to configure I get an error complaining that gcc needs to be version 4.8 or above. > > CentOS 6.9 comes with 4.7.7 and the main repo does not have 4.8. > > And you are not supposed to upgrade the 6.x working environment to 4.8. > > So I followed this forum post (of many similar) to create an environment where 4.8 is installed: > > https://gist.github.com/stephenturner/e3bc5cfacc2dc67eca8b > > # sudo yum install centos-release-scl > # sudo yum install devtoolset-3-toolchain > # scl enable devtoolset-3 bash > > # gcc --version > gcc (GCC) 4.9.2 20150212 (Red Hat 4.9.2-6) > Copyright (C) 2014 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > But when I run ./configure, it quits with this error: > > CMake Error at cmake/RequireCXX11.cmake:40 (message): > GCC version must be at least 4.8 for C++11 support, detected: 4.4.7 > Call Stack (most recent call first): > CMakeLists.txt:168 (include) > > > -- Configuring incomplete, errors occurred! > See also "/opt/bro-2.5.1/build/CMakeFiles/CMakeOutput.log". > See also "/opt/bro-2.5.1/build/CMakeFiles/CMakeError.log". > > which seems like it running in the original environment not in the scl devtoolset-3 environment. rm -rf build and then ./configure again. That should fix it. -- - Justin Azoff From jdopheid at illinois.edu Wed Jul 26 08:48:19 2017 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Wed, 26 Jul 2017 15:48:19 +0000 Subject: [Bro] =?utf-8?q?BroCon_=E2=80=9917=3A_Agenda_is_posted!?= Message-ID: A couple more updates on the agenda and other news. We are happy to announce this year?s keynote speaker: Michael Bailey, Associate Professor of electrical and computer engineering at the University of Illinois at Urbana-Champaign. Mike?s experience and research lies in networking and security, with a specific focus on solving real-world problems. We thank Mike for sharing his time and knowledge with our community. Also, the Bro Team?s talks have been finalized: * Johanna Amann, SSL Research with Bro * Justin Azoff?s LaBROtory, and update on what Justin has been up to this year * Seth Hall, the Bro Package Manager and You! * Robin Sommer, Following the Packets: A Walk Through Bro?s Internal Processing Pipeline * And our annual Broadmap and panel talks We are still accepting sponsorships, contact us at info at bro.org for more information. Early bird registration expires this Sunday! https://brocon2017.eventbrite.com/ ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign On 7/25/17, 1:54 PM, "bro-bounces at bro.org on behalf of Dopheide, Jeannette M" wrote: The BroCon agenda is now available: https://www.bro.org/community/brocon2017.html#agenda (Note: the agenda is subject to change.) And, we?ve added a lightning talk session to the agenda this year, see our abstract below: Have you contributed a package to the new Bro Package Manager? Or do you have something interesting to share related to Bro that doesn?t fit into a traditional presentation? Great! We?d love to hear from you. We have scheduled a 45-minute session for lightning talks. Be prepared to quickly identify your point, demonstrate it, and provide a link or contact info for later follow-up. We?d like to accommodate as many talks as possible so please limit your talk to less than 5 minutes. No product presentations. Early bird registration ends Sunday night (Central time), register now: https://brocon2017.eventbrite.com/ ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dopheide at gmail.com Wed Jul 26 09:37:32 2017 From: dopheide at gmail.com (Mike Dopheide) Date: Wed, 26 Jul 2017 11:37:32 -0500 Subject: [Bro] Adding dns entry to bro logs In-Reply-To: References: Message-ID: The only way I know to do a lookup (someone please chime in), is using when(), something like this, which I stole from urls.bro originally: when(local m_addrs = lookup_hostname(m_no_scheme)){ for(m_addr in m_addrs){ # ie, the lookup succeeded if(m_addr != 0.0.0.0){ ### DO YOUR THING } } } # timeout 5sec{ # print fmt("timeout"); # } However, I'd heavily caution you against doing that every time a log writes. For one, it's extremely expensive. Second, the when() call spawns a separate process, so if it works at all, you'd need to somehow delay your log writes while that field populates. Just thinking out loud, if your DHCP pool isn't too huge, you could do the lookups on some interval and just populate a table that you reference later. Not perfect, but close. I'd probably just recommend having the DHCP logs pushed to the same SIEM as my Bro data. -Dop On Wed, Jul 26, 2017 at 8:38 AM, Keith Midwinter wrote: > Hi, > I am using bro internally on a network that uses dhcp to assign ip > addresses so if I want to investigate something that happened yesterday > then doing a nslookup today wont tell me what host it was assigned to at > the time the log was created. So is there a way to do an nslookup at the > time of log creation and add it to the logs? > > I did some googling and found a reference to extending the log format and > running scripts but it wasn't enough for me to figure it out. > Thanks, > Keith > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170726/e284d259/attachment-0001.html From jazoff at illinois.edu Wed Jul 26 09:48:28 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 26 Jul 2017 16:48:28 +0000 Subject: [Bro] Adding dns entry to bro logs In-Reply-To: References: Message-ID: > On Jul 26, 2017, at 12:37 PM, Mike Dopheide wrote: > > However, I'd heavily caution you against doing that every time a log writes. For one, it's extremely expensive. Second, the when() call spawns a separate process, so if it works at all, you'd need to somehow delay your log writes while that field populates. Yeah this would not work that well in practice. > Just thinking out loud, if your DHCP pool isn't too huge, you could do the lookups on some interval and just populate a table that you reference later. Not perfect, but close. I was thinking exactly this. You just need some tool written in any language to output a file like #fields ip name 10.0.0.1 boxone 10.0.0.2 otherbox 10.0.0.3 thirdbox (with tabs and not spaces) and then bro can load that into a table[addr] of string; and you can reference it as often as you need. -- - Justin Azoff From jan.grashoefer at gmail.com Wed Jul 26 10:21:25 2017 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Wed, 26 Jul 2017 19:21:25 +0200 Subject: [Bro] Adding dns entry to bro logs In-Reply-To: References: Message-ID: <85f5c916-c4bf-32ad-51e7-770ea7fac131@gmail.com> >> Just thinking out loud, if your DHCP pool isn't too huge, you could do the lookups on some interval and just populate a table that you reference later. Not perfect, but close. > > I was thinking exactly this. You just need some tool written in any language to output a file like > > #fields ip name > 10.0.0.1 boxone > 10.0.0.2 otherbox > 10.0.0.3 thirdbox > > (with tabs and not spaces) and then bro can load that into a table[addr] of string; and you can reference it as often as you need. Another idea: If you monitor the DHCP traffic with Bro as well, wouldn't it be possible to react on new leases, do the lookup using "when" and store that info in the table? Jan From jazoff at illinois.edu Wed Jul 26 10:35:06 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 26 Jul 2017 17:35:06 +0000 Subject: [Bro] Adding dns entry to bro logs In-Reply-To: <85f5c916-c4bf-32ad-51e7-770ea7fac131@gmail.com> References: <85f5c916-c4bf-32ad-51e7-770ea7fac131@gmail.com> Message-ID: <8781234A-CA34-41D1-B8AB-887C703D4F0E@illinois.edu> > On Jul 26, 2017, at 1:21 PM, Jan Grash?fer wrote: > > > Another idea: If you monitor the DHCP traffic with Bro as well, wouldn't > it be possible to react on new leases, do the lookup using "when" and > store that info in the table? Yes.. if bro saw the DHCP traffic it could do this directly. There are some other challenges with that approach, like what happens when bro first starts up and it hasn't seen any dhcp traffic yet. If the lease times are long it could be a while before it has any data. -- - Justin Azoff From briford.wylie at gmail.com Wed Jul 26 10:37:04 2017 From: briford.wylie at gmail.com (Brian Wylie) Date: Wed, 26 Jul 2017 10:37:04 -0700 Subject: [Bro] Adding dns entry to bro logs In-Reply-To: References: Message-ID: Just FWIW, if you like/use Python. The BroThon package will automatically tail any Bro IDS log and then you can use the nice set of python modules to do nslookup and whatever else (dump file, etc). https://github.com/Kitware/BroThon This approach 'offloads' work from Bro IDS (which should stay focused on high performance network packet processing). Code Example: In [1]: from brothon import bro_log_reader In [2]: bro_log = bro_log_reader.BroLogReader('/path/to/bro/dhcp.log', tail= True) In [3]: for row in bro_log.readrows(): ...: # Do whatever you want with dhcp fields here (socket module/nslookup module/etc) ...: print(row['assigned_ip'], row['mac']) Output: Successfully monitoring /Users/briford/work/BroThon/data/dhcp.log... 192.168.33.10 00:20:18:eb:ca:21 192.168.33.11 46:22:28:b6:a3:24 On Wed, Jul 26, 2017 at 9:48 AM, Azoff, Justin S wrote: > > > On Jul 26, 2017, at 12:37 PM, Mike Dopheide wrote: > > > > However, I'd heavily caution you against doing that every time a log > writes. For one, it's extremely expensive. Second, the when() call spawns > a separate process, so if it works at all, you'd need to somehow delay your > log writes while that field populates. > > Yeah this would not work that well in practice. > > > Just thinking out loud, if your DHCP pool isn't too huge, you could do > the lookups on some interval and just populate a table that you reference > later. Not perfect, but close. > > I was thinking exactly this. You just need some tool written in any > language to output a file like > > #fields ip name > 10.0.0.1 boxone > 10.0.0.2 otherbox > 10.0.0.3 thirdbox > > (with tabs and not spaces) and then bro can load that into a table[addr] > of string; and you can reference it as often as you need. > > > -- > - Justin Azoff > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170726/8e94350c/attachment.html From kmidwinter at exoendo.com Wed Jul 26 10:39:21 2017 From: kmidwinter at exoendo.com (Keith Midwinter) Date: Wed, 26 Jul 2017 17:39:21 +0000 Subject: [Bro] Adding dns entry to bro logs In-Reply-To: References: , Message-ID: Hi Mike, Thanks for the info. I would like to do a load test as the box is pretty beefy and doesn't see a lot of traffic (2000 logs per minute). What file would I be editing to add the below code segment? Thanks! Keith Ps currently dhcp and bro logs do go to the Splunk. ________________________________ From: Mike Dopheide Sent: Wednesday, July 26, 2017 12:37:32 PM To: Keith Midwinter Cc: bro at bro.org Subject: Re: [Bro] Adding dns entry to bro logs The only way I know to do a lookup (someone please chime in), is using when(), something like this, which I stole from urls.bro originally: when(local m_addrs = lookup_hostname(m_no_scheme)){ for(m_addr in m_addrs){ # ie, the lookup succeeded if(m_addr != 0.0.0.0){ ### DO YOUR THING } } } # timeout 5sec{ # print fmt("timeout"); # } However, I'd heavily caution you against doing that every time a log writes. For one, it's extremely expensive. Second, the when() call spawns a separate process, so if it works at all, you'd need to somehow delay your log writes while that field populates. Just thinking out loud, if your DHCP pool isn't too huge, you could do the lookups on some interval and just populate a table that you reference later. Not perfect, but close. I'd probably just recommend having the DHCP logs pushed to the same SIEM as my Bro data. -Dop On Wed, Jul 26, 2017 at 8:38 AM, Keith Midwinter > wrote: Hi, I am using bro internally on a network that uses dhcp to assign ip addresses so if I want to investigate something that happened yesterday then doing a nslookup today wont tell me what host it was assigned to at the time the log was created. So is there a way to do an nslookup at the time of log creation and add it to the logs? I did some googling and found a reference to extending the log format and running scripts but it wasn't enough for me to figure it out. Thanks, Keith _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170726/4b8bb051/attachment-0001.html From valerio.click at gmx.com Thu Jul 27 23:47:08 2017 From: valerio.click at gmx.com (Valerio) Date: Fri, 28 Jul 2017 08:47:08 +0200 Subject: [Bro] Protocl Analyzer: when to Unref a variable Message-ID: <92bf769e-5a0e-ae7e-4639-7a566a6351d8@gmx.com> By looking at some analyzers (e.g., sip, TCP) I noticed that some here and there (usually after the content of a variable is assigned to another structure with Append) a call to Unref() is made. Are there any general principles or guidelines regarding when to explicitly Unref a variable when writing a Bro Analyzer with BinPAC? thanks, Valerio From hacecky at jlab.org Fri Jul 28 07:46:07 2017 From: hacecky at jlab.org (Eric Hacecky) Date: Fri, 28 Jul 2017 10:46:07 -0400 (EDT) Subject: [Bro] Strange behavior with interesting-hostnames.bro Message-ID: <252429094.2798246.1501253167135.JavaMail.zimbra@jlab.org> I have people connecting to an ftp server on my network via port 22, which is triggering the SSH::interesting_hostnames notice. I'm guessing this is for sftp, but either way it's not something I'm concerned with. Easy question: In my local.bro I'm trying to redef the interesting_hostnames variable to remove the ftp regex. reference bro/share/bro/policy/protocols/ssh/interesting-hostnames.bro // redef SSH::interesting_hostnames = { /^d?ns[0-9]*\./ | /^smtp[0-9]*\./ | /^mail[0-9]*\./ | /^pop[0-9]*\./ | /^imap[0-9]*\./ | /^www[0-9]*\./}; // I've also tried it without the brackets, {}. Both are accepted by broctl, but neither actually seem to do anything as the script continues to match port 22 traffic to my ftp server. Can't find much on google syntax related aside from having the -= operator available as a different way to specify it. What's wrong here? ========== Second...and this one is strange When the notice is generated, instead of having the note as SSH:Interesting_Hostname_Login, it's attaching a note of HTTP::SQL_Injection_Attacker. Here's a log line // 1501207574.941860 C3EEas45G6RoObEML9 7.7.7.7 46160 10.10.10.10 22 - - - tcp HTTP::SQL_Injection_Attacker Possible SSH login involving a local server with an interesting hostname. myftpserver.mydomain.com Notice::ACTION_EMAIL,IPBLOCK::ACTION_IPBLOCK,Notice::ACTION_LOG 3600.000000 // How is Bro attaching SQL_Injection_Attacker from a completely different module as the note? Thanks, Eric From jazoff at illinois.edu Fri Jul 28 08:02:07 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 28 Jul 2017 15:02:07 +0000 Subject: [Bro] Strange behavior with interesting-hostnames.bro In-Reply-To: <252429094.2798246.1501253167135.JavaMail.zimbra@jlab.org> References: <252429094.2798246.1501253167135.JavaMail.zimbra@jlab.org> Message-ID: <1814DDD4-F75E-4AE8-8728-E6E5838B6ADE@illinois.edu> > On Jul 28, 2017, at 10:46 AM, Eric Hacecky wrote: > > I have people connecting to an ftp server on my network via port 22, which is triggering the SSH::interesting_hostnames notice. I'm guessing this is for sftp, but either way it's not something I'm concerned with. > > Easy question: > > In my local.bro I'm trying to redef the interesting_hostnames variable to remove the ftp regex. reference bro/share/bro/policy/protocols/ssh/interesting-hostnames.bro > > // > redef SSH::interesting_hostnames = { > /^d?ns[0-9]*\./ | > /^smtp[0-9]*\./ | > /^mail[0-9]*\./ | > /^pop[0-9]*\./ | > /^imap[0-9]*\./ | > /^www[0-9]*\./}; > // > > I've also tried it without the brackets, {}. > > Both are accepted by broctl, but neither actually seem to do anything as the script continues to match port 22 traffic to my ftp server. > > Can't find much on google syntax related aside from having the -= operator available as a different way to specify it. What's wrong here? Without the braces is correct. You can use broctl print SSH::interesting_hostnames to see what version your cluster is using. You can also just use a notice policy hook to ignore the notices for that specific hostname, instead of ignoring ftp logins entirely. Your problem is probably related to the second issue you have... > Second...and this one is strange > > When the notice is generated, instead of having the note as SSH:Interesting_Hostname_Login, it's attaching a note of HTTP::SQL_Injection_Attacker. > > Here's a log line > > // > 1501207574.941860 C3EEas45G6RoObEML9 7.7.7.7 46160 10.10.10.10 22 - - - tcp HTTP::SQL_Injection_Attacker Possible SSH login involving a local server with an interesting hostname. myftpserver.mydomain.com Notice::ACTION_EMAIL,IPBLOCK::ACTION_IPBLOCK,Notice::ACTION_LOG 3600.000000 > // > > How is Bro attaching SQL_Injection_Attacker from a completely different module as the note? Did you make changes to your local.bro without restarting the entire cluster? Or did you make changes to one of the local-logger.bro local-manager.bro local-proxy.bro local-worker.bro files? I believe enums being wrong is a direct result of running different policies on different boxes causing the metadata for enums to get out of sync. Running different policies on different boxes would also explain why your changes to interesting_hostnames has not worked. -- - Justin Azoff From hacecky at jlab.org Fri Jul 28 08:22:52 2017 From: hacecky at jlab.org (Eric Hacecky) Date: Fri, 28 Jul 2017 11:22:52 -0400 (EDT) Subject: [Bro] Strange behavior with interesting-hostnames.bro In-Reply-To: <703194136.2806220.1501255354720.JavaMail.zimbra@jlab.org> References: <252429094.2798246.1501253167135.JavaMail.zimbra@jlab.org> <1814DDD4-F75E-4AE8-8728-E6E5838B6ADE@illinois.edu> Message-ID: <901449892.2806378.1501255372359.JavaMail.zimbra@jlab.org> Justin, Yeah seems like you're right on the money. // [BroControl] > print SSH::interesting_hostnames manager SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ proxy-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ worker-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ worker-2 SSH::interesting_hostnames = /((((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?))|(^?(^ftp[0-9]*\.)$?)/ // I haven't made any changes to the other local files. They're all empty on the manager/worker nodes. > Did you make changes to your local.bro without restarting the entire cluster? I've been using broctl deploy or restart mostly. I did get a crash message from worker-2 once, where I issued broctl restart worker-2 instead of the entire cluster. // 1501254343.937122 fatal error in /usr/local/bro/spool/installed-scripts-do-not-touch/site/local.bro, line 157: Val::CONST_ACCESSOR (types/string) (/(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/) // Any ideas? Eric ----- Original Message ----- From: "Justin S Azoff" To: "Eric Hacecky" Cc: bro at bro.org Sent: Friday, July 28, 2017 11:02:07 AM Subject: Re: [Bro] Strange behavior with interesting-hostnames.bro > On Jul 28, 2017, at 10:46 AM, Eric Hacecky wrote: > > I have people connecting to an ftp server on my network via port 22, which is triggering the SSH::interesting_hostnames notice. I'm guessing this is for sftp, but either way it's not something I'm concerned with. > > Easy question: > > In my local.bro I'm trying to redef the interesting_hostnames variable to remove the ftp regex. reference bro/share/bro/policy/protocols/ssh/interesting-hostnames.bro > > // > redef SSH::interesting_hostnames = { > /^d?ns[0-9]*\./ | > /^smtp[0-9]*\./ | > /^mail[0-9]*\./ | > /^pop[0-9]*\./ | > /^imap[0-9]*\./ | > /^www[0-9]*\./}; > // > > I've also tried it without the brackets, {}. > > Both are accepted by broctl, but neither actually seem to do anything as the script continues to match port 22 traffic to my ftp server. > > Can't find much on google syntax related aside from having the -= operator available as a different way to specify it. What's wrong here? Without the braces is correct. You can use broctl print SSH::interesting_hostnames to see what version your cluster is using. You can also just use a notice policy hook to ignore the notices for that specific hostname, instead of ignoring ftp logins entirely. Your problem is probably related to the second issue you have... > Second...and this one is strange > > When the notice is generated, instead of having the note as SSH:Interesting_Hostname_Login, it's attaching a note of HTTP::SQL_Injection_Attacker. > > Here's a log line > > // > 1501207574.941860 C3EEas45G6RoObEML9 7.7.7.7 46160 10.10.10.10 22 - - - tcp HTTP::SQL_Injection_Attacker Possible SSH login involving a local server with an interesting hostname. myftpserver.mydomain.com Notice::ACTION_EMAIL,IPBLOCK::ACTION_IPBLOCK,Notice::ACTION_LOG 3600.000000 > // > > How is Bro attaching SQL_Injection_Attacker from a completely different module as the note? Did you make changes to your local.bro without restarting the entire cluster? Or did you make changes to one of the local-logger.bro local-manager.bro local-proxy.bro local-worker.bro files? I believe enums being wrong is a direct result of running different policies on different boxes causing the metadata for enums to get out of sync. Running different policies on different boxes would also explain why your changes to interesting_hostnames has not worked. -- - Justin Azoff From jazoff at illinois.edu Fri Jul 28 08:35:18 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 28 Jul 2017 15:35:18 +0000 Subject: [Bro] Strange behavior with interesting-hostnames.bro In-Reply-To: <901449892.2806378.1501255372359.JavaMail.zimbra@jlab.org> References: <252429094.2798246.1501253167135.JavaMail.zimbra@jlab.org> <1814DDD4-F75E-4AE8-8728-E6E5838B6ADE@illinois.edu> <901449892.2806378.1501255372359.JavaMail.zimbra@jlab.org> Message-ID: > On Jul 28, 2017, at 11:22 AM, Eric Hacecky wrote: > > Justin, > > Yeah seems like you're right on the money. > > // > [BroControl] > print SSH::interesting_hostnames > manager SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > proxy-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > worker-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > worker-2 SSH::interesting_hostnames = /((((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?))|(^?(^ftp[0-9]*\.)$?)/ > // > > I haven't made any changes to the other local files. They're all empty on the manager/worker nodes. > >> Did you make changes to your local.bro without restarting the entire cluster? > > I've been using broctl deploy or restart mostly. I did get a crash message from worker-2 once, where I issued broctl restart worker-2 instead of the entire cluster. > > // > 1501254343.937122 fatal error in /usr/local/bro/spool/installed-scripts-do-not-touch/site/local.bro, line 157: Val::CONST_ACCESSOR (types/string) (/(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/) > // > > Any ideas? > > Eric > What does a broctl deploy output at this point? Does your local.bro currently have the redef with the {}'s? you want it without them. -- - Justin Azoff From hacecky at jlab.org Fri Jul 28 10:42:49 2017 From: hacecky at jlab.org (Eric Hacecky) Date: Fri, 28 Jul 2017 13:42:49 -0400 (EDT) Subject: [Bro] Strange behavior with interesting-hostnames.bro In-Reply-To: <848079435.2829391.1501263738171.JavaMail.zimbra@jlab.org> References: <252429094.2798246.1501253167135.JavaMail.zimbra@jlab.org> <1814DDD4-F75E-4AE8-8728-E6E5838B6ADE@illinois.edu> <901449892.2806378.1501255372359.JavaMail.zimbra@jlab.org> Message-ID: <1894308815.2829429.1501263769905.JavaMail.zimbra@jlab.org> [BroControl] > deploy checking configurations ... installing ... removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/site ... removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/auto ... creating policy directories ... installing site policies ... generating cluster-layout.bro ... generating local-networks.bro ... generating broctl-config.bro ... generating broctl-config.sh ... updating nodes ... stopping ... stopping worker-2 ... stopping worker-1 ... stopping proxy-1 ... stopping manager ... starting ... starting manager ... starting proxy-1 ... starting worker-1 ... starting worker-2 ... [BroControl] > print SSH::interesting_hostnames manager SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ proxy-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ worker-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ worker-2 SSH::interesting_hostnames = /((((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?))|(^?(^ftp[0-9]*\.)$?)/ The line for worker-2 is even formatted strangely. There's an extra set of parentheses surrounding the string of ((dns|smtp|mail|pop|imap|www) | ftp ) >From my local.bro # Remove match for hostname with "ftp" for the SSH login success to interesting hostname from /share/bro/policy/protocols/ssh/interesting-hostnames.bro redef SSH::interesting_hostnames = /^d?ns[0-9]*\./ | /^smtp[0-9]*\./ | /^mail[0-9]*\./ | /^pop[0-9]*\./ | /^imap[0-9]*\./ | /^www[0-9]*\./; Eric ----- Original Message ----- From: "Justin S Azoff" To: "Eric Hacecky" Cc: bro at bro.org Sent: Friday, July 28, 2017 11:35:18 AM Subject: Re: [Bro] Strange behavior with interesting-hostnames.bro > On Jul 28, 2017, at 11:22 AM, Eric Hacecky wrote: > > Justin, > > Yeah seems like you're right on the money. > > // > [BroControl] > print SSH::interesting_hostnames > manager SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > proxy-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > worker-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > worker-2 SSH::interesting_hostnames = /((((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?))|(^?(^ftp[0-9]*\.)$?)/ > // > > I haven't made any changes to the other local files. They're all empty on the manager/worker nodes. > >> Did you make changes to your local.bro without restarting the entire cluster? > > I've been using broctl deploy or restart mostly. I did get a crash message from worker-2 once, where I issued broctl restart worker-2 instead of the entire cluster. > > // > 1501254343.937122 fatal error in /usr/local/bro/spool/installed-scripts-do-not-touch/site/local.bro, line 157: Val::CONST_ACCESSOR (types/string) (/(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/) > // > > Any ideas? > > Eric > What does a broctl deploy output at this point? Does your local.bro currently have the redef with the {}'s? you want it without them. -- - Justin Azoff From jazoff at illinois.edu Fri Jul 28 11:06:50 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 28 Jul 2017 18:06:50 +0000 Subject: [Bro] Strange behavior with interesting-hostnames.bro In-Reply-To: <1894308815.2829429.1501263769905.JavaMail.zimbra@jlab.org> References: <252429094.2798246.1501253167135.JavaMail.zimbra@jlab.org> <1814DDD4-F75E-4AE8-8728-E6E5838B6ADE@illinois.edu> <901449892.2806378.1501255372359.JavaMail.zimbra@jlab.org> <1894308815.2829429.1501263769905.JavaMail.zimbra@jlab.org> Message-ID: <5A29B39D-C549-4A85-A0D6-C70DC00BC57F@illinois.edu> > On Jul 28, 2017, at 1:42 PM, Eric Hacecky wrote: > > [BroControl] > deploy > checking configurations ... > installing ... > removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/site ... > removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/auto ... > creating policy directories ... > installing site policies ... > generating cluster-layout.bro ... > generating local-networks.bro ... > generating broctl-config.bro ... > generating broctl-config.sh ... > updating nodes ... > stopping ... > stopping worker-2 ... > stopping worker-1 ... > stopping proxy-1 ... > stopping manager ... > starting ... > starting manager ... > starting proxy-1 ... > starting worker-1 ... > starting worker-2 ... > [BroControl] > print SSH::interesting_hostnames > manager SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > proxy-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > worker-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > worker-2 SSH::interesting_hostnames = /((((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?))|(^?(^ftp[0-9]*\.)$?)/ > > The line for worker-2 is even formatted strangely. There's an extra set of parentheses surrounding the string of ((dns|smtp|mail|pop|imap|www) | ftp ) > Ah.. that's normal It's actually more like.. so one extra item will have one extra parens added. ((((dns) |smtp) |mail) |pop) |imap) > From my local.bro > > > # Remove match for hostname with "ftp" for the SSH login success to interesting hostname from /share/bro/policy/protocols/ssh/interesting-hostnames.bro > redef SSH::interesting_hostnames = /^d?ns[0-9]*\./ | /^smtp[0-9]*\./ | /^mail[0-9]*\./ | /^pop[0-9]*\./ | /^imap[0-9]*\./ | /^www[0-9]*\./; > > Eric > I bet you have a half broken process for worker-2 lying around from when it had an issue. It may not have completely crashed. .. if you run broctl stop and then btoctl ps.bro Are any bro processes returned related to worker-2? Ensure that every bro process is stopped and then do a new deploy, that should clear things up. -- - Justin Azoff From hacecky at jlab.org Fri Jul 28 11:13:13 2017 From: hacecky at jlab.org (Eric Hacecky) Date: Fri, 28 Jul 2017 14:13:13 -0400 (EDT) Subject: [Bro] Strange behavior with interesting-hostnames.bro In-Reply-To: <1451786706.2833020.1501265570415.JavaMail.zimbra@jlab.org> References: <252429094.2798246.1501253167135.JavaMail.zimbra@jlab.org> <1814DDD4-F75E-4AE8-8728-E6E5838B6ADE@illinois.edu> <901449892.2806378.1501255372359.JavaMail.zimbra@jlab.org> <1894308815.2829429.1501263769905.JavaMail.zimbra@jlab.org> <5A29B39D-C549-4A85-A0D6-C70DC00BC57F@illinois.edu> Message-ID: <362935337.2833070.1501265593467.JavaMail.zimbra@jlab.org> Yep that was it. // [BroControl] > stop stopping worker-2 ... stopping worker-1 ... stopping proxy-1 ... stopping manager ... [BroControl] > ps.bro USER PID PPID %CPU %MEM VSZ RSS TT S STARTED TIME COMMAND >>> worker-2 (-) root 29973 29967 36.8 1.8 2580256 2458068 ? S Jul 25 1-02:41:59 bro (-) root 29974 29973 0.2 0.2 387928 328412 ? S Jul 25 00:11:20 bro // Cleaned it up and now the output for print matches across them. Well done, thanks again. Eric ----- Original Message ----- From: "Justin S Azoff" To: "Eric Hacecky" Cc: bro at bro.org Sent: Friday, July 28, 2017 2:06:50 PM Subject: Re: [Bro] Strange behavior with interesting-hostnames.bro > On Jul 28, 2017, at 1:42 PM, Eric Hacecky wrote: > > [BroControl] > deploy > checking configurations ... > installing ... > removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/site ... > removing old policies in /usr/local/bro/spool/installed-scripts-do-not-touch/auto ... > creating policy directories ... > installing site policies ... > generating cluster-layout.bro ... > generating local-networks.bro ... > generating broctl-config.bro ... > generating broctl-config.sh ... > updating nodes ... > stopping ... > stopping worker-2 ... > stopping worker-1 ... > stopping proxy-1 ... > stopping manager ... > starting ... > starting manager ... > starting proxy-1 ... > starting worker-1 ... > starting worker-2 ... > [BroControl] > print SSH::interesting_hostnames > manager SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > proxy-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > worker-1 SSH::interesting_hostnames = /(((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?)/ > worker-2 SSH::interesting_hostnames = /((((((^?(^d?ns[0-9]*\.)$?)|(^?(^smtp[0-9]*\.)$?))|(^?(^mail[0-9]*\.)$?))|(^?(^pop[0-9]*\.)$?))|(^?(^imap[0-9]*\.)$?))|(^?(^www[0-9]*\.)$?))|(^?(^ftp[0-9]*\.)$?)/ > > The line for worker-2 is even formatted strangely. There's an extra set of parentheses surrounding the string of ((dns|smtp|mail|pop|imap|www) | ftp ) > Ah.. that's normal It's actually more like.. so one extra item will have one extra parens added. ((((dns) |smtp) |mail) |pop) |imap) > From my local.bro > > > # Remove match for hostname with "ftp" for the SSH login success to interesting hostname from /share/bro/policy/protocols/ssh/interesting-hostnames.bro > redef SSH::interesting_hostnames = /^d?ns[0-9]*\./ | /^smtp[0-9]*\./ | /^mail[0-9]*\./ | /^pop[0-9]*\./ | /^imap[0-9]*\./ | /^www[0-9]*\./; > > Eric > I bet you have a half broken process for worker-2 lying around from when it had an issue. It may not have completely crashed. .. if you run broctl stop and then btoctl ps.bro Are any bro processes returned related to worker-2? Ensure that every bro process is stopped and then do a new deploy, that should clear things up. -- - Justin Azoff From jazoff at illinois.edu Fri Jul 28 11:17:16 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 28 Jul 2017 18:17:16 +0000 Subject: [Bro] Strange behavior with interesting-hostnames.bro In-Reply-To: <362935337.2833070.1501265593467.JavaMail.zimbra@jlab.org> References: <252429094.2798246.1501253167135.JavaMail.zimbra@jlab.org> <1814DDD4-F75E-4AE8-8728-E6E5838B6ADE@illinois.edu> <901449892.2806378.1501255372359.JavaMail.zimbra@jlab.org> <1894308815.2829429.1501263769905.JavaMail.zimbra@jlab.org> <5A29B39D-C549-4A85-A0D6-C70DC00BC57F@illinois.edu> <362935337.2833070.1501265593467.JavaMail.zimbra@jlab.org> Message-ID: > On Jul 28, 2017, at 2:13 PM, Eric Hacecky wrote: > > Yep that was it. > > // > [BroControl] > stop > stopping worker-2 ... > stopping worker-1 ... > stopping proxy-1 ... > stopping manager ... > [BroControl] > ps.bro > USER PID PPID %CPU %MEM VSZ RSS TT S STARTED TIME COMMAND >>>> worker-2 > (-) root 29973 29967 36.8 1.8 2580256 2458068 ? S Jul 25 1-02:41:59 bro > (-) root 29974 29973 0.2 0.2 387928 328412 ? S Jul 25 00:11:20 bro > // > > Cleaned it up and now the output for print matches across them. > > Well done, thanks again. > > Eric Awesome! One of the things in the works for broctl is a better mechanism for process supervision which will help prevent this exact problem. -- - Justin Azoff From logan_miller at byu.edu Mon Jul 31 09:40:52 2017 From: logan_miller at byu.edu (Logan Miller) Date: Mon, 31 Jul 2017 16:40:52 +0000 Subject: [Bro] Arista Traffic Shunting Message-ID: <2a290211b378405ea1099d71b7461159@MB4.byu.local> Hello everyone, We have a bro cluster setup and running but we are getting a lot of packet loss from elephant flows. We've seen that a lot of people use an Arista switch to block these flows but we haven't seen how to interface with Arista from bro. How do people do traffic shunting using Arista? These are the sources where we've seen people shunting traffic with Arista: http://www.cspi.com/wp-content/uploads/2016/09/Berkeley-100GIntrusionDetection.pdf http://mailman.icsi.berkeley.edu/pipermail/bro/2015-January/008038.html http://www.ucop.edu/information-technology-services/initiatives/sautter-award-program/sautter-2015/berkeley_lab-sautterawardnomination2015.pdf Thanks, Logan Miller Network Security Engineer Brigham Young University Office of IT -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170731/1cda2fd6/attachment.html From jdickenson at gmail.com Mon Jul 31 10:13:27 2017 From: jdickenson at gmail.com (James Dickenson) Date: Mon, 31 Jul 2017 10:13:27 -0700 Subject: [Bro] Arista Traffic Shunting In-Reply-To: <2a290211b378405ea1099d71b7461159@MB4.byu.local> References: <2a290211b378405ea1099d71b7461159@MB4.byu.local> Message-ID: I would also be interested to see what an implementation of this looks like. From my very very limited understanding is they are using the Arista's RESTful API. In the Berkley document the have the arista switch configs and it includes ' management api http?commands'. Unsure how on the bro end they are determining which flows to terminate and what logic controls that. https://eos.arista.com/arista-eapi-101/ -James On Mon, Jul 31, 2017 at 9:40 AM, Logan Miller wrote: > Hello everyone, > > > > We have a bro cluster setup and running but we are getting a lot of packet > loss from elephant flows. We?ve seen that a lot of people use an Arista > switch to block these flows but we haven?t seen how to interface with Arista > from bro. How do people do traffic shunting using Arista? > > > > These are the sources where we?ve seen people shunting traffic with Arista: > > http://www.cspi.com/wp-content/uploads/2016/09/Berkeley-100GIntrusionDetection.pdf > > http://mailman.icsi.berkeley.edu/pipermail/bro/2015-January/008038.html > > http://www.ucop.edu/information-technology-services/initiatives/sautter-award-program/sautter-2015/berkeley_lab-sautterawardnomination2015.pdf > > > > Thanks, > > > > Logan Miller > > Network Security Engineer > > Brigham Young University > > Office of IT > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From asharma at lbl.gov Mon Jul 31 10:20:40 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Mon, 31 Jul 2017 10:20:40 -0700 Subject: [Bro] Arista Traffic Shunting In-Reply-To: <2a290211b378405ea1099d71b7461159@MB4.byu.local> References: <2a290211b378405ea1099d71b7461159@MB4.byu.local> Message-ID: <20170731172039.GB10859@Aashishs-MacBook-Pro.local> (While Justin and others chime in) We are relying on : https://github.com/esnet/dumbno (this one has IPv6 support) Originally we started with Justin's branch here: https://github.com/ncsa/dumbno On bro side there is a conn-bulk.bro and react framework Here: https://github.com/JustinAzoff/bro-react This ties bro with dumbno.py which talks with arista to apply and remove ACLs. If you need specific arista configurations, I can send you our arista configs too. Aashish On Mon, Jul 31, 2017 at 04:40:52PM +0000, Logan Miller wrote: > Hello everyone, > > We have a bro cluster setup and running but we are getting a lot of packet loss from elephant flows. We've seen that a lot of people use an Arista switch to block these flows but we haven't seen how to interface with Arista from bro. How do people do traffic shunting using Arista? > > These are the sources where we've seen people shunting traffic with Arista: > http://www.cspi.com/wp-content/uploads/2016/09/Berkeley-100GIntrusionDetection.pdf > http://mailman.icsi.berkeley.edu/pipermail/bro/2015-January/008038.html > http://www.ucop.edu/information-technology-services/initiatives/sautter-award-program/sautter-2015/berkeley_lab-sautterawardnomination2015.pdf > > Thanks, > > Logan Miller > Network Security Engineer > Brigham Young University > Office of IT > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From eyrich at illinois.edu Mon Jul 31 10:21:47 2017 From: eyrich at illinois.edu (James Eyrich) Date: Mon, 31 Jul 2017 12:21:47 -0500 Subject: [Bro] Arista Traffic Shunting In-Reply-To: <2a290211b378405ea1099d71b7461159@MB4.byu.local> References: <2a290211b378405ea1099d71b7461159@MB4.byu.local> Message-ID: https://github.com/ncsa/dumbno On 7/31/2017 11:40 AM, Logan Miller wrote: > > Hello everyone, > > > > We have a bro cluster setup and running but we are getting a lot of > packet loss from elephant flows. We?ve seen that a lot of people use > an Arista switch to block these flows but we haven?t seen how to > interface with Arista from bro. How do people do traffic shunting > using Arista? > > > > These are the sources where we?ve seen people shunting traffic with > Arista: > > http://www.cspi.com/wp-content/uploads/2016/09/Berkeley-100GIntrusionDetection.pdf > > > http://mailman.icsi.berkeley.edu/pipermail/bro/2015-January/008038.html > > > http://www.ucop.edu/information-technology-services/initiatives/sautter-award-program/sautter-2015/berkeley_lab-sautterawardnomination2015.pdf > > > > > Thanks, > > > > Logan Miller > > Network Security Engineer > > Brigham Young University > > Office of IT > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- ---------------------------------------- James Eyrich Manager - Incident Response and Security National Center for Supercomputer Applications University of Illinois at Urbana-Champaign eyrich at illinois.edu 217-265-6867 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170731/fc03e80a/attachment.html From jazoff at illinois.edu Mon Jul 31 10:35:03 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 31 Jul 2017 17:35:03 +0000 Subject: [Bro] Arista Traffic Shunting In-Reply-To: <20170731172039.GB10859@Aashishs-MacBook-Pro.local> References: <2a290211b378405ea1099d71b7461159@MB4.byu.local> <20170731172039.GB10859@Aashishs-MacBook-Pro.local> Message-ID: <9D2A3BC2-AF19-49E9-B6B7-CD756D9D1A87@illinois.edu> > On Jul 31, 2017, at 1:20 PM, Aashish Sharma wrote: > > (While Justin and others chime in) > > We are relying on : > > https://github.com/esnet/dumbno (this one has IPv6 support) Originally > we started with Justin's branch here: https://github.com/ncsa/dumbno Oh, I merged in the ipv6 support over a year ago, so you should probably be running the upstream version. -- - Justin Azoff From logan_miller at byu.edu Mon Jul 31 10:54:10 2017 From: logan_miller at byu.edu (Logan Miller) Date: Mon, 31 Jul 2017 17:54:10 +0000 Subject: [Bro] Arista Traffic Shunting In-Reply-To: <9D2A3BC2-AF19-49E9-B6B7-CD756D9D1A87@illinois.edu> References: <2a290211b378405ea1099d71b7461159@MB4.byu.local> <20170731172039.GB10859@Aashishs-MacBook-Pro.local> <9D2A3BC2-AF19-49E9-B6B7-CD756D9D1A87@illinois.edu> Message-ID: Aashish, I saw those github scripts and I wasn't sure where to put them in the bro directory and what bro configs had to be changed to make it them work. Where did you put the scripts? Also, as long as api management is enabled on the Arista switch, is there anything else on the switch that needs to be configured? - Logan Miller From asharma at lbl.gov Mon Jul 31 12:40:22 2017 From: asharma at lbl.gov (Aashish Sharma) Date: Mon, 31 Jul 2017 12:40:22 -0700 Subject: [Bro] Arista Traffic Shunting In-Reply-To: References: <2a290211b378405ea1099d71b7461159@MB4.byu.local> <20170731172039.GB10859@Aashishs-MacBook-Pro.local> <9D2A3BC2-AF19-49E9-B6B7-CD756D9D1A87@illinois.edu> Message-ID: <20170731194021.GD10859@Aashishs-MacBook-Pro.local> Here is how I have stuff setup: 1) in your site.bro: @load conn-bulk.bro @load react redef GridFTP::size_threshold = 1048576; # 2 MB redef Bulk::size_threshold = 134217728 ; # 128 MB 2) make sure react/dumbno.cfg has right values 3) then: a) ./dumbno.py dumbno.cfg setup b) ./dumbno.py dumbno.cfg >& /var/log/dumbno.log & c) ./dumbno.py dumbno.cfg stats >& /var/log/dumbno.stats & 4) @load conn-bulk.bro and "@load react" should get bro to start flagging and acting on fat flows. tail on /var/log/dumbno.log should give you reasonable visibility into on-going operations Hope this helps, Aashish On Mon, Jul 31, 2017 at 05:54:10PM +0000, Logan Miller wrote: > Aashish, > > I saw those github scripts and I wasn't sure where to put them in the bro directory and what bro configs had to be changed to make it them work. Where did you put the scripts? Also, as long as api management is enabled on the Arista switch, is there anything else on the switch that needs to be configured? > > - Logan Miller From mabossert at gmail.com Mon Jul 31 13:50:07 2017 From: mabossert at gmail.com (M. Aaron Bossert) Date: Mon, 31 Jul 2017 16:50:07 -0400 Subject: [Bro] strategies for ingesting large number of PCAP files Message-ID: All, I am working with a storm topology to process a large number of PCAP files which can be of variable sizes, but tend to be in the range of 100MB to 200MB, give or take. My current batch to work on contains about 42K files...I am aiming to process with as much parallelism as possible while avoiding the issue of sessions that span more than one file (so you know why I am doing this) my main constraints/focus: 1. take advantage of large number of cores (56) and RAM (~750GB) on my node(s) 2. Avoid disk as much as possible (I have relatively slow spinning disks, though quite a few of them that can be addressed individually, which could mitigate the disk IO bottleneck to some degree) 3. Prioritize completeness above all else...get as many sessions reconstructed as possible by stitching the packets back together in one of the ways below...or another if you folks have a better idea... my thinking...and hope for suggestions on the best approach...or a completely different one if you have a better solution: 1. run mergecap and setup bro to run as a cluster and hope for the best 1. *upside*: relatively simple and lowest level of effort 2. *downside*: not sure it will scale the way I want. I'd prefer to isolate Bro to running on no more than two nodes in my cluster...each node has 56 cores and ~750GB RAM. Also, it will be one more hack to have to work into my Storm topology 2. use Storm topology (or something else) to re-write packets to individual files based on SIP/DIP/SPORT/DPORT or similar 1. *upside*: this will ensure a certain level of parallelism and keep the logic inside my topology where I can control it to the greatest extent 2. *downside*: This seems like it is horribly inefficient because I will have to read the PCAP files twice: once to split and once again when Bro get them, and again to read the Bro logs (if I don't get the Kafka plugins to do what I want). Also, this will require some sort of load balancing to ensure that IP's that represent a disproportionate percentage of traffic don't gum up the works, nor do IP's that have relatively little traffic take up too many resources. My thought here is to simply keep track of approximate file sizes and send IP's in rough balance (though still always sending any given IP/port pair to the same file). Also, this makes me interact with the HDD's at least three times (once to read PCAP, next to write PCAP, again to read Bro logs, which is undesirable) 3. Use Storm topology or TCP replay (or similar) to read in PCAP files, then write to virtual interfaces (a pool setup manually) so that Bro can simply listen on each interface and process as appropriate. 1. *upside*: Seems like this could be the most efficient option as it probably avoids disk the most, seems like it could scale very well, and would support clustering by simply creating pools of interfaces on multiple nodes, session-ization takes care of itself and I just need to tell Bro to wait longer for packets to show up so it doesn't think the interface went dead if there are lulls is traffic 2. *downside*: Most complex of the bunch and I am uncertain of my ability to preserve timestamps when sending the packets over the interface to Bro 4. Extend Bro to not only write directly to Kafka topics, but also to read from them such that I could use one of the methods above to split traffic and load balance and then have Bro simply spit out logs to another topic of my choosing 1. *upside*: This could be the most elegant solution because it will allow me to handle failures and hiccups using Kafka offsets 2. *downside*: This is easily the most difficult to implement for me as I have not messed with extending Bro at all. Any suggestions or feedback would be greatly appreciated! Thanks in advance... Aaron P.S. sorry for the verbose message...but was hoping to give as complete a problem/solution statement as I can -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170731/e8ddea26/attachment.html From jazoff at illinois.edu Mon Jul 31 14:06:50 2017 From: jazoff at illinois.edu (Azoff, Justin S) Date: Mon, 31 Jul 2017 21:06:50 +0000 Subject: [Bro] strategies for ingesting large number of PCAP files In-Reply-To: References: Message-ID: <16290E7A-078E-4277-B90C-438E3A16F214@illinois.edu> Is this for that Metron thing? I had chatted with someone on irc about this a ways back. I think the simplest way to integrate bro would be to write a 'pcapdir' packet source plugin that works similar to the 'live' pcap mode, but instead reads packets from sequentially numbered pcap files in a directory. That way fetching the next packet would boil down to, in pythonish pseudocode loop: while not current file: current file = find next pcap file sleep for 100ms packet = get next packet from current pcap file if packet: return packet close current pcap file delete current pcap file current file = None packet source plugins are pretty easy to write, you could probably get this working in a few hours. It would be a lot easier to implement than interfacing with kafka directly. Then you just need to atomically move pcap files into the directory that bro is watching. Since a single instance of bro is running you don't have to worry about sessions that span more than one file... that should just work normally. To use more than one bro process you would just need to write a tool that can read pcaps, hash by 2 or 4 tuple, and output sliced pcaps to different places. You should be able to do everything but the last part without pcaps ever touching the disk. You probably want to avoid using something like tcpreplay since you'd lose a lot of the performance benefits of bro reading from pcap files. -- - Justin Azoff > On Jul 31, 2017, at 4:50 PM, M. Aaron Bossert wrote: > > All, > > I am working with a storm topology to process a large number of PCAP files which can be of variable sizes, but tend to be in the range of 100MB to 200MB, give or take. My current batch to work on contains about 42K files...I am aiming to process with as much parallelism as possible while avoiding the issue of sessions that span more than one file (so you know why I am doing this) > > my main constraints/focus: > ? take advantage of large number of cores (56) and RAM (~750GB) on my node(s) > ? Avoid disk as much as possible (I have relatively slow spinning disks, though quite a few of them that can be addressed individually, which could mitigate the disk IO bottleneck to some degree) > ? Prioritize completeness above all else...get as many sessions reconstructed as possible by stitching the packets back together in one of the ways below...or another if you folks have a better idea... > > my thinking...and hope for suggestions on the best approach...or a completely different one if you have a better solution: > > ? run mergecap and setup bro to run as a cluster and hope for the best > ? upside: relatively simple and lowest level of effort > ? downside: not sure it will scale the way I want. I'd prefer to isolate Bro to running on no more than two nodes in my cluster...each node has 56 cores and ~750GB RAM. Also, it will be one more hack to have to work into my Storm topology > ? use Storm topology (or something else) to re-write packets to individual files based on SIP/DIP/SPORT/DPORT or similar > ? upside: this will ensure a certain level of parallelism and keep the logic inside my topology where I can control it to the greatest extent > ? downside: This seems like it is horribly inefficient because I will have to read the PCAP files twice: once to split and once again when Bro get them, and again to read the Bro logs (if I don't get the Kafka plugins to do what I want). Also, this will require some sort of load balancing to ensure that IP's that represent a disproportionate percentage of traffic don't gum up the works, nor do IP's that have relatively little traffic take up too many resources. My thought here is to simply keep track of approximate file sizes and send IP's in rough balance (though still always sending any given IP/port pair to the same file). Also, this makes me interact with the HDD's at least three times (once to read PCAP, next to write PCAP, again to read Bro logs, which is undesirable) > ? Use Storm topology or TCP replay (or similar) to read in PCAP files, then write to virtual interfaces (a pool setup manually) so that Bro can simply listen on each interface and process as appropriate. > ? upside: Seems like this could be the most efficient option as it probably avoids disk the most, seems like it could scale very well, and would support clustering by simply creating pools of interfaces on multiple nodes, session-ization takes care of itself and I just need to tell Bro to wait longer for packets to show up so it doesn't think the interface went dead if there are lulls is traffic > ? downside: Most complex of the bunch and I am uncertain of my ability to preserve timestamps when sending the packets over the interface to Bro > ? Extend Bro to not only write directly to Kafka topics, but also to read from them such that I could use one of the methods above to split traffic and load balance and then have Bro simply spit out logs to another topic of my choosing > ? upside: This could be the most elegant solution because it will allow me to handle failures and hiccups using Kafka offsets > ? downside: This is easily the most difficult to implement for me as I have not messed with extending Bro at all. > Any suggestions or feedback would be greatly appreciated! Thanks in advance... > > Aaron > > P.S. sorry for the verbose message...but was hoping to give as complete a problem/solution statement as I can From mabossert at gmail.com Mon Jul 31 14:14:41 2017 From: mabossert at gmail.com (M. Aaron Bossert) Date: Mon, 31 Jul 2017 17:14:41 -0400 Subject: [Bro] strategies for ingesting large number of PCAP files In-Reply-To: <16290E7A-078E-4277-B90C-438E3A16F214@illinois.edu> References: <16290E7A-078E-4277-B90C-438E3A16F214@illinois.edu> Message-ID: This is not for Metron...I am doing something different...research...I could see the read from a directory approach, but unfortunately, I cannot control the file naming scheme nor can I be certain the "last-modified" timestamp will be entirely reliable. That is why I was trying to deal with the packets and their respective timestamps directly... I write code in Scala/Java, and Perl, but never have written anything in Python...which is also a hurdle I will have to deal with...(but am willing to deal with if needed) On Mon, Jul 31, 2017 at 5:06 PM, Azoff, Justin S wrote: > Is this for that Metron thing? I had chatted with someone on irc about > this a ways back. I think the simplest way to integrate bro would be to > write a 'pcapdir' packet source plugin that works similar to the 'live' > pcap mode, but instead reads packets from sequentially numbered pcap files > in a directory. That way fetching the next packet would boil down to, in > pythonish pseudocode > > > loop: > while not current file: > current file = find next pcap file > sleep for 100ms > packet = get next packet from current pcap file > if packet: > return packet > close current pcap file > delete current pcap file > current file = None > > packet source plugins are pretty easy to write, you could probably get > this working in a few hours. It would be a lot easier to implement than > interfacing with kafka directly. > > Then you just need to atomically move pcap files into the directory that > bro is watching. Since a single instance of bro is running you don't have > to worry about sessions that span more than one file... that should just > work normally. > > To use more than one bro process you would just need to write a tool that > can read pcaps, hash by 2 or 4 tuple, and output sliced pcaps to different > places. > > You should be able to do everything but the last part without pcaps ever > touching the disk. > > You probably want to avoid using something like tcpreplay since you'd lose > a lot of the performance benefits of bro reading from pcap files. > > -- > - Justin Azoff > > > On Jul 31, 2017, at 4:50 PM, M. Aaron Bossert > wrote: > > > > All, > > > > I am working with a storm topology to process a large number of PCAP > files which can be of variable sizes, but tend to be in the range of 100MB > to 200MB, give or take. My current batch to work on contains about 42K > files...I am aiming to process with as much parallelism as possible while > avoiding the issue of sessions that span more than one file (so you know > why I am doing this) > > > > my main constraints/focus: > > ? take advantage of large number of cores (56) and RAM (~750GB) on > my node(s) > > ? Avoid disk as much as possible (I have relatively slow spinning > disks, though quite a few of them that can be addressed individually, which > could mitigate the disk IO bottleneck to some degree) > > ? Prioritize completeness above all else...get as many sessions > reconstructed as possible by stitching the packets back together in one of > the ways below...or another if you folks have a better idea... > > > > my thinking...and hope for suggestions on the best approach...or a > completely different one if you have a better solution: > > > > ? run mergecap and setup bro to run as a cluster and hope for the > best > > ? upside: relatively simple and lowest level of effort > > ? downside: not sure it will scale the way I want. I'd > prefer to isolate Bro to running on no more than two nodes in my > cluster...each node has 56 cores and ~750GB RAM. Also, it will be one more > hack to have to work into my Storm topology > > ? use Storm topology (or something else) to re-write packets to > individual files based on SIP/DIP/SPORT/DPORT or similar > > ? upside: this will ensure a certain level of parallelism > and keep the logic inside my topology where I can control it to the > greatest extent > > ? downside: This seems like it is horribly inefficient > because I will have to read the PCAP files twice: once to split and once > again when Bro get them, and again to read the Bro logs (if I don't get the > Kafka plugins to do what I want). Also, this will require some sort of > load balancing to ensure that IP's that represent a disproportionate > percentage of traffic don't gum up the works, nor do IP's that have > relatively little traffic take up too many resources. My thought here is > to simply keep track of approximate file sizes and send IP's in rough > balance (though still always sending any given IP/port pair to the same > file). Also, this makes me interact with the HDD's at least three times > (once to read PCAP, next to write PCAP, again to read Bro logs, which is > undesirable) > > ? Use Storm topology or TCP replay (or similar) to read in PCAP > files, then write to virtual interfaces (a pool setup manually) so that Bro > can simply listen on each interface and process as appropriate. > > ? upside: Seems like this could be the most efficient > option as it probably avoids disk the most, seems like it could scale very > well, and would support clustering by simply creating pools of interfaces > on multiple nodes, session-ization takes care of itself and I just need to > tell Bro to wait longer for packets to show up so it doesn't think the > interface went dead if there are lulls is traffic > > ? downside: Most complex of the bunch and I am uncertain > of my ability to preserve timestamps when sending the packets over the > interface to Bro > > ? Extend Bro to not only write directly to Kafka topics, but also > to read from them such that I could use one of the methods above to split > traffic and load balance and then have Bro simply spit out logs to another > topic of my choosing > > ? upside: This could be the most elegant solution because > it will allow me to handle failures and hiccups using Kafka offsets > > ? downside: This is easily the most difficult to implement > for me as I have not messed with extending Bro at all. > > Any suggestions or feedback would be greatly appreciated! Thanks in > advance... > > > > Aaron > > > > P.S. sorry for the verbose message...but was hoping to give as complete > a problem/solution statement as I can > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170731/c9d5db97/attachment-0001.html From mabossert at gmail.com Mon Jul 31 14:21:38 2017 From: mabossert at gmail.com (M. Aaron Bossert) Date: Mon, 31 Jul 2017 17:21:38 -0400 Subject: [Bro] strategies for ingesting large number of PCAP files In-Reply-To: References: <16290E7A-078E-4277-B90C-438E3A16F214@illinois.edu> Message-ID: Sorry, missed the last part...I already have code written (Storm topology) that reads the PCAP files and sends them to the follow--on bolt to write to disk...or whatever...it already hashes on the 2/4/5 tuple (configurable). I am using pcap4j for that...SO, writing files to disk is trivial and mostly done...but I kept getting heartburn when I was looking at the disk IO implications (read,write,read,write,read,send to Kafka...which is another write once it persists). This approach also means that I am stuck with having to probably read PCAP twice...once to process for Bro and once to process the PCAP itself, part of which involves an in-memory JOIN of sorts that associates each Bro log (conn.log) to the individual packets that made up the session. On Mon, Jul 31, 2017 at 5:14 PM, M. Aaron Bossert wrote: > This is not for Metron...I am doing something different...research...I > could see the read from a directory approach, but unfortunately, I cannot > control the file naming scheme nor can I be certain the "last-modified" > timestamp will be entirely reliable. That is why I was trying to deal with > the packets and their respective timestamps directly... > > I write code in Scala/Java, and Perl, but never have written anything in > Python...which is also a hurdle I will have to deal with...(but am willing > to deal with if needed) > > On Mon, Jul 31, 2017 at 5:06 PM, Azoff, Justin S > wrote: > >> Is this for that Metron thing? I had chatted with someone on irc about >> this a ways back. I think the simplest way to integrate bro would be to >> write a 'pcapdir' packet source plugin that works similar to the 'live' >> pcap mode, but instead reads packets from sequentially numbered pcap files >> in a directory. That way fetching the next packet would boil down to, in >> pythonish pseudocode >> >> >> loop: >> while not current file: >> current file = find next pcap file >> sleep for 100ms >> packet = get next packet from current pcap file >> if packet: >> return packet >> close current pcap file >> delete current pcap file >> current file = None >> >> packet source plugins are pretty easy to write, you could probably get >> this working in a few hours. It would be a lot easier to implement than >> interfacing with kafka directly. >> >> Then you just need to atomically move pcap files into the directory that >> bro is watching. Since a single instance of bro is running you don't have >> to worry about sessions that span more than one file... that should just >> work normally. >> >> To use more than one bro process you would just need to write a tool that >> can read pcaps, hash by 2 or 4 tuple, and output sliced pcaps to different >> places. >> >> You should be able to do everything but the last part without pcaps ever >> touching the disk. >> >> You probably want to avoid using something like tcpreplay since you'd >> lose a lot of the performance benefits of bro reading from pcap files. >> >> -- >> - Justin Azoff >> >> > On Jul 31, 2017, at 4:50 PM, M. Aaron Bossert >> wrote: >> > >> > All, >> > >> > I am working with a storm topology to process a large number of PCAP >> files which can be of variable sizes, but tend to be in the range of 100MB >> to 200MB, give or take. My current batch to work on contains about 42K >> files...I am aiming to process with as much parallelism as possible while >> avoiding the issue of sessions that span more than one file (so you know >> why I am doing this) >> > >> > my main constraints/focus: >> > ? take advantage of large number of cores (56) and RAM (~750GB) >> on my node(s) >> > ? Avoid disk as much as possible (I have relatively slow spinning >> disks, though quite a few of them that can be addressed individually, which >> could mitigate the disk IO bottleneck to some degree) >> > ? Prioritize completeness above all else...get as many sessions >> reconstructed as possible by stitching the packets back together in one of >> the ways below...or another if you folks have a better idea... >> > >> > my thinking...and hope for suggestions on the best approach...or a >> completely different one if you have a better solution: >> > >> > ? run mergecap and setup bro to run as a cluster and hope for the >> best >> > ? upside: relatively simple and lowest level of effort >> > ? downside: not sure it will scale the way I want. I'd >> prefer to isolate Bro to running on no more than two nodes in my >> cluster...each node has 56 cores and ~750GB RAM. Also, it will be one more >> hack to have to work into my Storm topology >> > ? use Storm topology (or something else) to re-write packets to >> individual files based on SIP/DIP/SPORT/DPORT or similar >> > ? upside: this will ensure a certain level of parallelism >> and keep the logic inside my topology where I can control it to the >> greatest extent >> > ? downside: This seems like it is horribly inefficient >> because I will have to read the PCAP files twice: once to split and once >> again when Bro get them, and again to read the Bro logs (if I don't get the >> Kafka plugins to do what I want). Also, this will require some sort of >> load balancing to ensure that IP's that represent a disproportionate >> percentage of traffic don't gum up the works, nor do IP's that have >> relatively little traffic take up too many resources. My thought here is >> to simply keep track of approximate file sizes and send IP's in rough >> balance (though still always sending any given IP/port pair to the same >> file). Also, this makes me interact with the HDD's at least three times >> (once to read PCAP, next to write PCAP, again to read Bro logs, which is >> undesirable) >> > ? Use Storm topology or TCP replay (or similar) to read in PCAP >> files, then write to virtual interfaces (a pool setup manually) so that Bro >> can simply listen on each interface and process as appropriate. >> > ? upside: Seems like this could be the most efficient >> option as it probably avoids disk the most, seems like it could scale very >> well, and would support clustering by simply creating pools of interfaces >> on multiple nodes, session-ization takes care of itself and I just need to >> tell Bro to wait longer for packets to show up so it doesn't think the >> interface went dead if there are lulls is traffic >> > ? downside: Most complex of the bunch and I am uncertain >> of my ability to preserve timestamps when sending the packets over the >> interface to Bro >> > ? Extend Bro to not only write directly to Kafka topics, but also >> to read from them such that I could use one of the methods above to split >> traffic and load balance and then have Bro simply spit out logs to another >> topic of my choosing >> > ? upside: This could be the most elegant solution because >> it will allow me to handle failures and hiccups using Kafka offsets >> > ? downside: This is easily the most difficult to >> implement for me as I have not messed with extending Bro at all. >> > Any suggestions or feedback would be greatly appreciated! Thanks in >> advance... >> > >> > Aaron >> > >> > P.S. sorry for the verbose message...but was hoping to give as complete >> a problem/solution statement as I can >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170731/24bc5dca/attachment.html