From paul.halliday at gmail.com Wed Oct 1 06:32:43 2014 From: paul.halliday at gmail.com (Paul Halliday) Date: Wed, 1 Oct 2014 10:32:43 -0300 Subject: [Bro] File log Message-ID: Is it normal for the 'filename' field to always be empty? The mime_type is almost always identified but the filename field is always '-' application/vnd.ms-cab-compressed - application/x-dosexec - text/plain - application/x-dosexec - text/plain - application/vnd.ms-fontobject - application/vnd.ms-fontobject - application/vnd.ms-fontobject - application/octet-stream - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/x-dosexec - application/vnd.ms-cab-compressed - image/jpeg - image/jpeg - image/jpeg - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/x-dosexec - application/vnd.ms-cab-compressed - text/plain - text/html - text/html - application/x-dosexec - application/vnd.ms-cab-compressed - application/x-dosexec - application/vnd.ms-cab-compressed - application/x-dosexec - image/jpeg - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/x-dosexec - text/plain - image/jpeg - application/vnd.ms-cab-compressed - application/octet-stream - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - image/jpeg - image/jpeg - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - image/jpeg - application/x-dosexec - application/x-dosexec - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - text/html - text/html - Thanks. -- Paul Halliday http://www.pintumbler.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141001/b3e2d6de/attachment.html From hosom at battelle.org Wed Oct 1 06:44:31 2014 From: hosom at battelle.org (Hosom, Stephen M) Date: Wed, 1 Oct 2014 13:44:31 +0000 Subject: [Bro] File log In-Reply-To: References: Message-ID: This is normal. Filename is used for protocols that identify the file name when it is in transit on the network (like HTTP). Generally though? you don?t actually want the filename, so this doesn?t have much impact on Bro?s ability to do cool stuff with files (how would you deal with a trillion copies of index.html, for example?). From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Paul Halliday Sent: Wednesday, October 01, 2014 9:33 AM To: bro at bro.org Subject: [Bro] File log Is it normal for the 'filename' field to always be empty? The mime_type is almost always identified but the filename field is always '-' application/vnd.ms-cab-compressed - application/x-dosexec - text/plain - application/x-dosexec - text/plain - application/vnd.ms-fontobject - application/vnd.ms-fontobject - application/vnd.ms-fontobject - application/octet-stream - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/x-dosexec - application/vnd.ms-cab-compressed - image/jpeg - image/jpeg - image/jpeg - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/x-dosexec - application/vnd.ms-cab-compressed - text/plain - text/html - text/html - application/x-dosexec - application/vnd.ms-cab-compressed - application/x-dosexec - application/vnd.ms-cab-compressed - application/x-dosexec - image/jpeg - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/x-dosexec - text/plain - image/jpeg - application/vnd.ms-cab-compressed - application/octet-stream - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - image/jpeg - image/jpeg - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - image/jpeg - application/x-dosexec - application/x-dosexec - application/vnd.ms-cab-compressed - application/vnd.ms-cab-compressed - text/html - text/html - Thanks. -- Paul Halliday http://www.pintumbler.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141001/4b59c515/attachment.html From paul.halliday at gmail.com Wed Oct 1 07:27:26 2014 From: paul.halliday at gmail.com (Paul Halliday) Date: Wed, 1 Oct 2014 11:27:26 -0300 Subject: [Bro] File log In-Reply-To: References: Message-ID: Good to know. Out of curiosity though, if the field is of little value then why even have it? (I have to deal with a trillion copies of '-') ;) On Wed, Oct 1, 2014 at 10:44 AM, Hosom, Stephen M wrote: > This is normal. Filename is used for protocols that identify the file > name when it is in transit on the network (like HTTP). Generally though? > you don?t actually want the filename, so this doesn?t have much impact on > Bro?s ability to do cool stuff with files (how would you deal with a > trillion copies of index.html, for example?). > > > > *From:* bro-bounces at bro.org [mailto:bro-bounces at bro.org] *On Behalf Of *Paul > Halliday > *Sent:* Wednesday, October 01, 2014 9:33 AM > *To:* bro at bro.org > *Subject:* [Bro] File log > > > > Is it normal for the 'filename' field to always be empty? The mime_type is > almost always identified but the filename field is always '-' > > > > application/vnd.ms-cab-compressed - > > application/x-dosexec - > > text/plain - > > application/x-dosexec - > > text/plain - > > application/vnd.ms-fontobject - > > application/vnd.ms-fontobject - > > application/vnd.ms-fontobject - > > application/octet-stream - > > application/vnd.ms-cab-compressed - > > application/vnd.ms-cab-compressed - > > application/x-dosexec - > > application/vnd.ms-cab-compressed - > > image/jpeg - > > image/jpeg - > > image/jpeg - > > application/vnd.ms-cab-compressed - > > application/vnd.ms-cab-compressed - > > application/vnd.ms-cab-compressed - > > application/x-dosexec - > > application/vnd.ms-cab-compressed - > > text/plain - > > text/html - > > text/html - > > application/x-dosexec - > > application/vnd.ms-cab-compressed - > > application/x-dosexec - > > application/vnd.ms-cab-compressed - > > application/x-dosexec - > > image/jpeg - > > application/vnd.ms-cab-compressed - > > application/vnd.ms-cab-compressed - > > application/x-dosexec - > > text/plain - > > image/jpeg - > > application/vnd.ms-cab-compressed - > > application/octet-stream - > > application/vnd.ms-cab-compressed - > > application/vnd.ms-cab-compressed - > > application/vnd.ms-cab-compressed - > > application/vnd.ms-cab-compressed - > > application/vnd.ms-cab-compressed - > > application/vnd.ms-cab-compressed - > > image/jpeg - > > image/jpeg - > > application/vnd.ms-cab-compressed - > > application/vnd.ms-cab-compressed - > > image/jpeg - > > application/x-dosexec - > > application/x-dosexec - > > application/vnd.ms-cab-compressed - > > application/vnd.ms-cab-compressed - > > text/html - > > text/html - > > > > Thanks. > > > > -- > Paul Halliday > http://www.pintumbler.org/ > -- Paul Halliday http://www.pintumbler.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141001/522c3a7b/attachment.html From seth at icir.org Wed Oct 1 08:07:05 2014 From: seth at icir.org (Seth Hall) Date: Wed, 1 Oct 2014 11:07:05 -0400 Subject: [Bro] File log In-Reply-To: References: Message-ID: <9B98889F-6CD1-4FEA-8150-413E5340A5BA@icir.org> On Oct 1, 2014, at 10:27 AM, Paul Halliday wrote: > Good to know. Out of curiosity though, if the field is of little value then why even have it? (I have to deal with a trillion copies of '-') For a little more explanation, I'll point to a mailing list post I did a while ago: http://marc.info/?l=bro&m=139882790812212&w=2 I'm not sure that I'd say that the field is of little value though. It's actually pretty valuable, the only problem is that for the most frequently seen protocol in your files log (HTTP), filename are rarely made available. If you look at SMTP traffic, you will much more frequently see that attachments have filenames. Also, for the upcoming SMB analyzer, filenames are always (or should always) be available. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From donaldson8 at llnl.gov Wed Oct 1 11:56:08 2014 From: donaldson8 at llnl.gov (Donaldson, John) Date: Wed, 1 Oct 2014 18:56:08 +0000 Subject: [Bro] High drop rates on recent builds Message-ID: On recent builds from the master branch, I'm seeing anomalously high drop rates. >From notice.log 458745 packets dropped after filtering, 747736 received, 288991 on link 262140 packets dropped after filtering, 581026 received, 318886 on link 524280 packets dropped after filtering, 826789 received, 302509 on link If I flip some of these values around to make sense (so that dropped+received=onlink), I'm still left with really bad capture rates. I'm running on DAG cards, and I can confirm that they, too, think that I'm madly dropping packets on the floor, even though CPU utilization for the Bro processes is hovering around 3%. By reverting back to the 2.3.1 tag, these problems go away. Any thoughts? Is anyone else seeing this? v/r John Donaldson From seth at icir.org Wed Oct 1 12:26:25 2014 From: seth at icir.org (Seth Hall) Date: Wed, 1 Oct 2014 15:26:25 -0400 Subject: [Bro] High drop rates on recent builds In-Reply-To: References: Message-ID: <40FE5F41-657C-4A83-93B7-1728BF5A9070@icir.org> On Oct 1, 2014, at 2:56 PM, Donaldson, John wrote: > Any thoughts? Is anyone else seeing this?  Hm, I've actually heard a similar story from one or two other people. I'm not sure where to start debugging this one though. Fortunately the diffs between 2.3 and 2.3.1 are pretty minimal so it might take someone digging through those by hand to see if there is something that changed in there that could impact some people. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From wren3 at illinois.edu Wed Oct 1 12:37:34 2014 From: wren3 at illinois.edu (Ren, Wenyu) Date: Wed, 1 Oct 2014 19:37:34 +0000 Subject: [Bro] question: "intel.log" not generated Message-ID: Hi everyone, I am new to Bro and I am learning to use the Intelligence framework. I followed the Bro Intel Framework Exercises which can be found in the following link. https://www.bro.org/current/exercises/intel/index.html For the first exercise, everything went well except that no "intel.log" file was generated. Does anyone know the reason and how to get it work? Thanks a lot. Wenyu From robin at icir.org Wed Oct 1 12:58:21 2014 From: robin at icir.org (Robin Sommer) Date: Wed, 1 Oct 2014 12:58:21 -0700 Subject: [Bro] High drop rates on recent builds In-Reply-To: References: Message-ID: <20141001195821.GP76556@icir.org> Any idea at what time this started? Two candidates I can think of: commit e9692958f05fb17bc946a04a78313cc5dd0922de Date: Thu Sep 25 12:46:39 2014 -0700 Merge remote-tracking branch 'origin/topic/jsiwek/improve_comm_loop' Merge: 3caecad 265438b Date: Tue Sep 9 12:35:38 2014 -0500 Merge remote-tracking branch 'origin/topic/robin/pktsrc' If you're up for some experiments, you could switch to versions just before these merges and see if that helps. Robin On Wed, Oct 01, 2014 at 18:56 +0000, Donaldson, John wrote: > On recent builds from the master branch, I'm seeing anomalously high drop rates. > > >From notice.log > 458745 packets dropped after filtering, 747736 received, 288991 on link > 262140 packets dropped after filtering, 581026 received, 318886 on link > 524280 packets dropped after filtering, 826789 received, 302509 on link > > If I flip some of these values around to make sense (so that > dropped+received=onlink), I'm still left with really bad capture > rates. I'm running on DAG cards, and I can confirm that they, too, > think that I'm madly dropping packets on the floor, even though CPU > utilization for the Bro processes is hovering around 3%. > > By reverting back to the 2.3.1 tag, these problems go away. > > Any thoughts? Is anyone else seeing this? > > v/r > > John Donaldson > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From seth at icir.org Wed Oct 1 14:03:19 2014 From: seth at icir.org (Seth Hall) Date: Wed, 1 Oct 2014 17:03:19 -0400 Subject: [Bro] question: "intel.log" not generated In-Reply-To: References: Message-ID: <8C796C5C-1B7E-44BE-8557-CFC6423532F3@icir.org> On Oct 1, 2014, at 3:37 PM, Ren, Wenyu wrote: > For the first exercise, everything went well except that no "intel.log" file was generated. It sounds like it everything actually didn't go very well. :) Could you give us some more information about what you did? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From donaldson8 at llnl.gov Wed Oct 1 15:44:39 2014 From: donaldson8 at llnl.gov (Donaldson, John) Date: Wed, 1 Oct 2014 22:44:39 +0000 Subject: [Bro] High drop rates on recent builds In-Reply-To: <20141001195821.GP76556@icir.org> References: <20141001195821.GP76556@icir.org> Message-ID: Robin, It looks like this problem existed prior to or on 22 September, which was when I most recently rebuilt, and, looking back, started seeing the odd rates. I know that I rebuilt on the 9th, as well, because of the (since-resolved) issues with 3caecad265438b and specifying DAG streams. I'm stepping through commits now. John Donaldson > -----Original Message----- > From: Robin Sommer [mailto:robin at icir.org] > Sent: Wednesday, October 01, 2014 12:58 PM > To: Donaldson, John > Cc: bro at bro.org > Subject: Re: [Bro] High drop rates on recent builds > > Any idea at what time this started? > > Two candidates I can think of: > > commit e9692958f05fb17bc946a04a78313cc5dd0922de > Date: Thu Sep 25 12:46:39 2014 -0700 > > Merge remote-tracking branch 'origin/topic/jsiwek/improve_comm_loop' > > Merge: 3caecad 265438b > Date: Tue Sep 9 12:35:38 2014 -0500 > > Merge remote-tracking branch 'origin/topic/robin/pktsrc' > > If you're up for some experiments, you could switch to versions just before > these merges and see if that helps. > > Robin > > On Wed, Oct 01, 2014 at 18:56 +0000, Donaldson, John wrote: > > > On recent builds from the master branch, I'm seeing anomalously high drop > rates. > > > > >From notice.log > > 458745 packets dropped after filtering, 747736 received, 288991 on > > link > > 262140 packets dropped after filtering, 581026 received, 318886 on > > link > > 524280 packets dropped after filtering, 826789 received, 302509 on > > link > > > > If I flip some of these values around to make sense (so that > > dropped+received=onlink), I'm still left with really bad capture > > rates. I'm running on DAG cards, and I can confirm that they, too, > > think that I'm madly dropping packets on the floor, even though CPU > > utilization for the Bro processes is hovering around 3%. > > > > By reverting back to the 2.3.1 tag, these problems go away. > > > > Any thoughts? Is anyone else seeing this? > > > > v/r > > > > John Donaldson > > > > > > > > _______________________________________________ > > Bro mailing list > > bro at bro-ids.org > > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > > > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From dnthayer at illinois.edu Thu Oct 2 09:36:30 2014 From: dnthayer at illinois.edu (Daniel Thayer) Date: Thu, 2 Oct 2014 11:36:30 -0500 Subject: [Bro] question: "intel.log" not generated In-Reply-To: References: Message-ID: <542D7F0E.6080204@illinois.edu> On 10/01/2014 02:37 PM, Ren, Wenyu wrote: > Hi everyone, > > I am new to Bro and I am learning to use the Intelligence framework. I followed the Bro Intel Framework Exercises which can be found in the following link. > https://www.bro.org/current/exercises/intel/index.html > > For the first exercise, everything went well except that no "intel.log" file was generated. Does anyone know the reason and how to get it work? > > Thanks a lot. > > Wenyu > I tried it and encountered the same problem. It seems that the pcap file got truncated somehow. I still have a copy of the pcap file that was originally distributed for these exercises and the original file is much larger. From jsiwek at illinois.edu Thu Oct 2 09:52:15 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 2 Oct 2014 16:52:15 +0000 Subject: [Bro] High drop rates on recent builds In-Reply-To: References: Message-ID: <82807749-FD15-4156-A28A-1B18750889DD@illinois.edu> On Oct 1, 2014, at 1:56 PM, Donaldson, John wrote: > On recent builds from the master branch, I'm seeing anomalously high drop rates. If you?re able to test and provide feedback on latest master again, 9cd85be308 fixes a problem that caused the main loop to spin more frequently than it used to. - Jon From donaldson8 at llnl.gov Thu Oct 2 10:26:49 2014 From: donaldson8 at llnl.gov (Donaldson, John) Date: Thu, 2 Oct 2014 17:26:49 +0000 Subject: [Bro] High drop rates on recent builds In-Reply-To: <82807749-FD15-4156-A28A-1B18750889DD@illinois.edu> References: <82807749-FD15-4156-A28A-1B18750889DD@illinois.edu> Message-ID: No success. It jumps right back up to around 50% loss. John Donaldson > -----Original Message----- > From: Siwek, Jon [mailto:jsiwek at illinois.edu] > Sent: Thursday, October 02, 2014 9:52 AM > To: Donaldson, John > Cc: bro at bro.org > Subject: Re: [Bro] High drop rates on recent builds > > > On Oct 1, 2014, at 1:56 PM, Donaldson, John wrote: > > > On recent builds from the master branch, I'm seeing anomalously high drop > rates. > > If you're able to test and provide feedback on latest master again, > 9cd85be308 fixes a problem that caused the main loop to spin more > frequently than it used to. > > - Jon From dnthayer at illinois.edu Fri Oct 3 09:58:58 2014 From: dnthayer at illinois.edu (Daniel Thayer) Date: Fri, 3 Oct 2014 11:58:58 -0500 Subject: [Bro] question: "intel.log" not generated In-Reply-To: <542D7F0E.6080204@illinois.edu> References: <542D7F0E.6080204@illinois.edu> Message-ID: <542ED5D2.1010700@illinois.edu> On 10/02/2014 11:36 AM, Daniel Thayer wrote: > On 10/01/2014 02:37 PM, Ren, Wenyu wrote: >> Hi everyone, >> >> I am new to Bro and I am learning to use the Intelligence framework. I followed the Bro Intel Framework Exercises which can be found in the following link. >> https://www.bro.org/current/exercises/intel/index.html >> >> For the first exercise, everything went well except that no "intel.log" file was generated. Does anyone know the reason and how to get it work? >> >> Thanks a lot. >> >> Wenyu >> > > I tried it and encountered the same problem. > > It seems that the pcap file got truncated somehow. > I still have a copy of the pcap file that was > originally distributed for these exercises and the > original file is much larger. I've replaced the "exercise-traffic.pcap" file on the web site with the original file, so the exercise should work now. From zryzregister at 163.com Sat Oct 4 19:00:54 2014 From: zryzregister at 163.com (=?GBK?B?1dTcx9Sq?=) Date: Sun, 5 Oct 2014 10:00:54 +0800 (CST) Subject: [Bro] =?gbk?q?Bro_Cannot_Get_=A1=AEResp=5Fmime=5Ftypes=27_properl?= =?gbk?q?y_in_http=2Elog?= In-Reply-To: <542D7F0E.6080204@illinois.edu> References: <542D7F0E.6080204@illinois.edu> Message-ID: <7544a229.c32.148de09c150.Coremail.zryzregister@163.com> Hi Brolist, I run some interesting pcaps using Bro-2.3, but there are some HTTP sessions that Bro-2.3 cannot tackle properly. For example, this pcap file from the malware-traffic-analysis.net. http://malware-traffic-analysis.net/2014/10/03/2014-10-03-Sweet-Orange-EK-traffic.pcap This is a exploit traffic and Bro cannot get 'Resp_mime_types' in the request to 'b.epavers.com/alterra/lLWZm'. As shown above Bro-2.3 parses the 'Resp_mime_types' as '-'. But in fact, when I use wireshark to parse this stream, the type is ''application/x-shockwave-flash'. In fact I have encountered this problem for quite several times, so I wonder why this happened and how to solve it ! Thanks a lot if anyone can answer my question! Yours, Rui-Yuan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141005/35809765/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: QQ??20141005095216.png Type: image/png Size: 19022 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141005/35809765/attachment.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: QQ??20141005095541.png Type: image/png Size: 16072 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141005/35809765/attachment-0001.bin From blackhole.em at gmail.com Sun Oct 5 13:18:49 2014 From: blackhole.em at gmail.com (Joe Blow) Date: Sun, 5 Oct 2014 16:18:49 -0400 Subject: [Bro] Add Geo data to Conn log? Message-ID: Hey folks, Was wondering: what the easiest way to add the geo data to conn log? Cheers, JB -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141005/556ddd7f/attachment.html From doug.burks at gmail.com Sun Oct 5 14:49:03 2014 From: doug.burks at gmail.com (Doug Burks) Date: Sun, 5 Oct 2014 17:49:03 -0400 Subject: [Bro] Add Geo data to Conn log? In-Reply-To: References: Message-ID: Hi JB, Please see: https://github.com/sethhall/bro-securityonion/blob/master/conn-add-country.bro On Sun, Oct 5, 2014 at 4:18 PM, Joe Blow wrote: > Hey folks, > > Was wondering: what the easiest way to add the geo data to conn log? > > Cheers, > > JB > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Doug Burks Need Security Onion Training or Commercial Support? http://securityonionsolutions.com From seth at icir.org Mon Oct 6 08:05:29 2014 From: seth at icir.org (Seth Hall) Date: Mon, 6 Oct 2014 11:05:29 -0400 Subject: [Bro] =?utf-8?q?Bro_Cannot_Get_=E2=80=98Resp=5Fmime=5Ftypes=27_pr?= =?utf-8?q?operly_in_http=2Elog?= In-Reply-To: <7544a229.c32.148de09c150.Coremail.zryzregister@163.com> References: <542D7F0E.6080204@illinois.edu> <7544a229.c32.148de09c150.Coremail.zryzregister@163.com> Message-ID: <5D9739EA-27D0-4B16-B178-6514C6855E73@icir.org> On Oct 4, 2014, at 10:00 PM, ??? wrote: > As shown above Bro-2.3 parses the 'Resp_mime_types' as '-'. But in fact, when I use wireshark to parse this stream, the type is ''application/x-shockwave-flash'. What you're seeing there is what the server declared the content to be. Bro ignores that value and sniffs the content to try and identify it. You have found a weakness in our shockwave detection fingerprint though. I'm going to be doing a commit into master soon that improves on our Flash detection (our signatures don't detect LZMA compressed flash files). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From damian.gerow at shopify.com Mon Oct 6 10:58:32 2014 From: damian.gerow at shopify.com (Damian Gerow) Date: Mon, 6 Oct 2014 13:58:32 -0400 Subject: [Bro] Cluster state synchronization Message-ID: I'm having some troubles wrapping my head around synchronization of set values in a cluster. We use a relatively simple bro script that correlates sets of whitelisted/blacklisted DNS names with new connections. To accomplish this, we have sets that are just the IP addresses returned by DNS lookups, which we then use to check against new connections. i.e. Host "foo.internal" looks up "blacklist.example.com", and receives response "10.0.0.1". Bro then adds IP address "10.0.0.1" to the set named "blacklisted_ips". "foo.internal" then proceeds to contact "10.0.0.1" on TCP/443. Bro looks up "10.0.0.1" in "blacklisted_ips" and, as there is a match, raises a notice. After migrating from a standalone to a single-node cluster configuration (manager, proxy, worker), it now appears as though the sets containing IP addresses are updated after the TCP connection is initialized. As a result, our notice log is now growing with entries that should never have been raised in the first place, and is missing entries that should have been raised. Does this theory make sense? Is there a way to speed up set additions/removals, or otherwise force synchronization whenever a modification is made, before processing any further traffic? Alternatively, does the Bro scripting language have any concept of a 'sleep'? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141006/ce192263/attachment.html From anthony.kasza at gmail.com Mon Oct 6 12:54:32 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Mon, 6 Oct 2014 12:54:32 -0700 Subject: [Bro] Cluster state synchronization In-Reply-To: References: Message-ID: I'm not sure about forcing synchronization. In reply to your question about sleep, you may want to look at scriptland's suspend_processing() and continue_processing(). -AK On Oct 6, 2014 11:06 AM, "Damian Gerow" wrote: > I'm having some troubles wrapping my head around synchronization of set > values in a cluster. > > We use a relatively simple bro script that correlates sets of > whitelisted/blacklisted DNS names with new connections. To accomplish > this, we have sets that are just the IP addresses returned by DNS lookups, > which we then use to check against new connections. > > i.e. Host "foo.internal" looks up "blacklist.example.com", and receives > response "10.0.0.1". Bro then adds IP address "10.0.0.1" to the set named > "blacklisted_ips". "foo.internal" then proceeds to contact "10.0.0.1" on > TCP/443. Bro looks up "10.0.0.1" in "blacklisted_ips" and, as there is a > match, raises a notice. > > After migrating from a standalone to a single-node cluster configuration > (manager, proxy, worker), it now appears as though the sets containing IP > addresses are updated after the TCP connection is initialized. As a > result, our notice log is now growing with entries that should never have > been raised in the first place, and is missing entries that should have > been raised. > > Does this theory make sense? Is there a way to speed up set > additions/removals, or otherwise force synchronization whenever a > modification is made, before processing any further traffic? > Alternatively, does the Bro scripting language have any concept of a > 'sleep'? > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141006/e5ca0cc5/attachment.html From npratley at redhat.com Mon Oct 6 19:07:35 2014 From: npratley at redhat.com (Nick Pratley) Date: Tue, 07 Oct 2014 12:07:35 +1000 Subject: [Bro] BitTorrent protocol analyzer help Message-ID: <54334AE7.9040403@redhat.com> Hi, I need some help with the BitTorrent protocol analyzer. My aim is to log info_hash values for files downloaded over bittorrent. I can see bittorrent-related events in base/bif/plugins/Bro_BitTorrent.events.bif.bro but these events don't seem to be getting raised. I'm testing with a .pcap generated on my laptop while opening Transmission and starting a Fedora torrent download. I'm running Bro 2.3.1 on RHEL 6, installed via the RPM. I'm new to Bro and have been reading a lot of the documentation but I'm still not sure exactly how I'm supposed to go about achieving this, so if someone could give me a pointer to get started that would be greatly appreciated. Thanks, Nick. From seth at icir.org Mon Oct 6 20:42:15 2014 From: seth at icir.org (Seth Hall) Date: Mon, 6 Oct 2014 23:42:15 -0400 Subject: [Bro] BitTorrent protocol analyzer help In-Reply-To: <54334AE7.9040403@redhat.com> References: <54334AE7.9040403@redhat.com> Message-ID: <567A6A45-4442-447B-97B1-E1F8CAD01C4A@icir.org> On Oct 6, 2014, at 10:07 PM, Nick Pratley wrote: > Hi, I need some help with the BitTorrent protocol analyzer. My aim is to log info_hash values for > files downloaded over bittorrent. The bittorrent analyzer has undergone some bitrot and doesn't currently have scripts that enable it. > I can see bittorrent-related events in base/bif/plugins/Bro_BitTorrent.events.bif.bro but these > events don't seem to be getting raised. If you look at the base scripts for other protocols, you will see where the analyzer is attached to connections by a port heuristic or by a signature heuristic in the accompanying .sig file (in scripts/base/protocols/xxx/).  Generally, unless you're prepared to do some heavier core and scriptland work, bittorrent isn't going to be something you can just use right now. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From damian.gerow at shopify.com Tue Oct 7 08:56:31 2014 From: damian.gerow at shopify.com (Damian Gerow) Date: Tue, 7 Oct 2014 11:56:31 -0400 Subject: [Bro] Cluster state synchronization In-Reply-To: References: Message-ID: I think I may have mis-spoken when I used the term 'synchronization': our cluster is a single-node cluster, with one each of manager, proxy, and worker. So the sets are not tagged with '&synchronized'. What we're seeing is: 1. DNS lookup is performed. Query matches whitelist. Bro is told to place the query response into the 'whitelisted_ips' set. 2. Connection is established to the IP address. Connection is verified against the 'whitelisted_ips' set. No match is found, so a notice is raised. 3. I confirm that the destination IP exists in the 'whitelisted_ips' set. I've just done a quick test, and I have confirmed that one DNS name that was present in a whitelist took upwards of 10 seconds to show up in the whitelisted_ips set. While this was happening, multiple DNS lookups were performed and recorded by Bro in the DNS log. Why does it take so long for a set to be updated, if that set is not set for synchronization? On Mon, Oct 6, 2014 at 3:54 PM, anthony kasza wrote: > I'm not sure about forcing synchronization. In reply to your question > about sleep, you may want to look at scriptland's suspend_processing() and > continue_processing(). > > -AK > On Oct 6, 2014 11:06 AM, "Damian Gerow" wrote: > >> I'm having some troubles wrapping my head around synchronization of set >> values in a cluster. >> >> We use a relatively simple bro script that correlates sets of >> whitelisted/blacklisted DNS names with new connections. To accomplish >> this, we have sets that are just the IP addresses returned by DNS lookups, >> which we then use to check against new connections. >> >> i.e. Host "foo.internal" looks up "blacklist.example.com", and receives >> response "10.0.0.1". Bro then adds IP address "10.0.0.1" to the set named >> "blacklisted_ips". "foo.internal" then proceeds to contact "10.0.0.1" on >> TCP/443. Bro looks up "10.0.0.1" in "blacklisted_ips" and, as there is a >> match, raises a notice. >> >> After migrating from a standalone to a single-node cluster configuration >> (manager, proxy, worker), it now appears as though the sets containing IP >> addresses are updated after the TCP connection is initialized. As a >> result, our notice log is now growing with entries that should never have >> been raised in the first place, and is missing entries that should have >> been raised. >> >> Does this theory make sense? Is there a way to speed up set >> additions/removals, or otherwise force synchronization whenever a >> modification is made, before processing any further traffic? >> Alternatively, does the Bro scripting language have any concept of a >> 'sleep'? >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141007/dede9436/attachment.html From anthony.kasza at gmail.com Tue Oct 7 09:23:53 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Tue, 7 Oct 2014 09:23:53 -0700 Subject: [Bro] Cluster state synchronization In-Reply-To: References: Message-ID: I think I understand the question now. What does the &synchronize attribute do for values in a single node cluster? Someone else on the list will have to jump in on this one... I'm curious if changing the priority of the event where you add to the whitelist set would make any difference. -AK On Oct 7, 2014 8:56 AM, "Damian Gerow" wrote: > I think I may have mis-spoken when I used the term 'synchronization': our > cluster is a single-node cluster, with one each of manager, proxy, and > worker. So the sets are not tagged with '&synchronized'. > > What we're seeing is: > > 1. DNS lookup is performed. Query matches whitelist. Bro is told to > place the query response into the 'whitelisted_ips' set. > 2. Connection is established to the IP address. Connection is verified > against the 'whitelisted_ips' set. No match is found, so a notice is > raised. > 3. I confirm that the destination IP exists in the 'whitelisted_ips' set. > > I've just done a quick test, and I have confirmed that one DNS name that > was present in a whitelist took upwards of 10 seconds to show up in the > whitelisted_ips set. While this was happening, multiple DNS lookups were > performed and recorded by Bro in the DNS log. > > Why does it take so long for a set to be updated, if that set is not set > for synchronization? > > On Mon, Oct 6, 2014 at 3:54 PM, anthony kasza > wrote: > >> I'm not sure about forcing synchronization. In reply to your question >> about sleep, you may want to look at scriptland's suspend_processing() and >> continue_processing(). >> >> -AK >> On Oct 6, 2014 11:06 AM, "Damian Gerow" wrote: >> >>> I'm having some troubles wrapping my head around synchronization of set >>> values in a cluster. >>> >>> We use a relatively simple bro script that correlates sets of >>> whitelisted/blacklisted DNS names with new connections. To accomplish >>> this, we have sets that are just the IP addresses returned by DNS lookups, >>> which we then use to check against new connections. >>> >>> i.e. Host "foo.internal" looks up "blacklist.example.com", and receives >>> response "10.0.0.1". Bro then adds IP address "10.0.0.1" to the set named >>> "blacklisted_ips". "foo.internal" then proceeds to contact "10.0.0.1" on >>> TCP/443. Bro looks up "10.0.0.1" in "blacklisted_ips" and, as there is a >>> match, raises a notice. >>> >>> After migrating from a standalone to a single-node cluster configuration >>> (manager, proxy, worker), it now appears as though the sets containing IP >>> addresses are updated after the TCP connection is initialized. As a >>> result, our notice log is now growing with entries that should never have >>> been raised in the first place, and is missing entries that should have >>> been raised. >>> >>> Does this theory make sense? Is there a way to speed up set >>> additions/removals, or otherwise force synchronization whenever a >>> modification is made, before processing any further traffic? >>> Alternatively, does the Bro scripting language have any concept of a >>> 'sleep'? >>> >>> _______________________________________________ >>> Bro mailing list >>> bro at bro-ids.org >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141007/783785ab/attachment.html From seth at icir.org Tue Oct 7 09:51:40 2014 From: seth at icir.org (Seth Hall) Date: Tue, 7 Oct 2014 12:51:40 -0400 Subject: [Bro] Cluster state synchronization In-Reply-To: References: Message-ID: <2BE14E49-64A6-4ACD-9EDC-6B817A346999@icir.org> On Oct 6, 2014, at 1:58 PM, Damian Gerow wrote: > I'm having some troubles wrapping my head around synchronization of set values in a cluster. > > We use a relatively simple bro script that correlates sets of whitelisted/blacklisted DNS names with new connections. To accomplish this, we have sets that are just the IP addresses returned by DNS lookups, which we then use to check against new connections.  Is this a script that you wrote locally or are you using the Broala script? https://github.com/broala/bro-snippets/blob/master/intel-dns.bro (this script works like it sounds like your does, but it uses data you have fed into the intel framework) If you're curious about your script though, post is somewhere and someone can take a look. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From npratley at redhat.com Tue Oct 7 15:24:59 2014 From: npratley at redhat.com (Nick Pratley) Date: Wed, 08 Oct 2014 08:24:59 +1000 Subject: [Bro] BitTorrent protocol analyzer help In-Reply-To: <567A6A45-4442-447B-97B1-E1F8CAD01C4A@icir.org> References: <54334AE7.9040403@redhat.com> <567A6A45-4442-447B-97B1-E1F8CAD01C4A@icir.org> Message-ID: <5434683B.5060809@redhat.com> Hi Seth, thanks for the response. On 10/07/2014 01:42 PM, Seth Hall wrote: > The bittorrent analyzer has undergone some bitrot and doesn't currently have scripts that enable it. Curious to know what you mean by bitrot exactly? Was it not complete in the first place, not maintained to keep up with changes in Bro itself..? > If you look at the base scripts for other protocols, you will see where the analyzer is attached to connections by a port heuristic or by a signature heuristic in the accompanying .sig file (in scripts/base/protocols/xxx/). >  > Generally, unless you're prepared to do some heavier core and scriptland work, bittorrent isn't going to be something you can just use right now. BitTorrent analysis would be quite useful to me so I'll have a look around. Even if I don't get it working I should at least learn a bit about Bro :) - Nick From seth at icir.org Tue Oct 7 18:29:45 2014 From: seth at icir.org (Seth Hall) Date: Tue, 7 Oct 2014 21:29:45 -0400 Subject: [Bro] BitTorrent protocol analyzer help In-Reply-To: <5434683B.5060809@redhat.com> References: <54334AE7.9040403@redhat.com> <567A6A45-4442-447B-97B1-E1F8CAD01C4A@icir.org> <5434683B.5060809@redhat.com> Message-ID: <76ADEE52-CCBF-4DCB-B07A-387F43699E13@icir.org> On Oct 7, 2014, at 6:24 PM, Nick Pratley wrote: > Curious to know what you mean by bitrot exactly? Was it not complete in the first place, not > maintained to keep up with changes in Bro itself..? It had some small issues and hasn't been updated to use the file analysis api internally. There were also never any 2.x style scripts written for it. > BitTorrent analysis would be quite useful to me so I'll have a look around. Even if I don't get it > working I should at least learn a bit about Bro :) Please ask if you have any questions! You'll learn a lot about Bro if you do it. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From damian.gerow at shopify.com Wed Oct 8 06:25:07 2014 From: damian.gerow at shopify.com (Damian Gerow) Date: Wed, 8 Oct 2014 09:25:07 -0400 Subject: [Bro] Cluster state synchronization In-Reply-To: <2BE14E49-64A6-4ACD-9EDC-6B817A346999@icir.org> References: <2BE14E49-64A6-4ACD-9EDC-6B817A346999@icir.org> Message-ID: On Tue, Oct 7, 2014 at 12:51 PM, Seth Hall wrote: > Is this a script that you wrote locally or are you using the Broala > script? > https://github.com/broala/bro-snippets/blob/master/intel-dns.bro > (this script works like it sounds like your does, but it uses data > you have fed into the intel framework) > It's a script that I inherited, originally written locally (I believe). It is quite similar to the Broala script, but we're not using the intel framework. If you're curious about your script though, post is somewhere and someone > can take a look. :) > A shortened version of the script I'm using for testing is at https://gist.github.com/mutemule/a36f49b16db51eccd159. If I move the 'add' commands into their own functions, and then prioritize the 'add_' over the 'is_' functions, would that be a reasonable way to ensure my sets are updated before being used for lookups? I'm already planning to migrate some of our stuff over to Intel, but I'm not quite there yet. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141008/64bb7f45/attachment.html From seth at icir.org Wed Oct 8 07:19:52 2014 From: seth at icir.org (Seth Hall) Date: Wed, 8 Oct 2014 10:19:52 -0400 Subject: [Bro] Cluster state synchronization In-Reply-To: References: <2BE14E49-64A6-4ACD-9EDC-6B817A346999@icir.org> Message-ID: <897E1997-31E2-410F-A5A6-EF34B62D4FCB@icir.org> On Oct 8, 2014, at 9:25 AM, Damian Gerow wrote: > On Tue, Oct 7, 2014 at 12:51 PM, Seth Hall wrote: > Is this a script that you wrote locally or are you using the Broala script? > > https://github.com/broala/bro-snippets/blob/master/intel-dns.bro > (this script works like it sounds like your does, but it uses data you have fed into the intel framework) > > It's a script that I inherited, originally written locally (I believe). It is quite similar to the Broala script, but we're not using the intel framework. > > If you're curious about your script though, post is somewhere and someone can take a look. :) > > A shortened version of the script I'm using for testing is at https://gist.github.com/mutemule/a36f49b16db51eccd159. If I move the 'add' commands into their own functions, and then prioritize the 'add_' over the 'is_' functions, would that be a reasonable way to ensure my sets are updated before being used for lookups? I'm already planning to migrate some of our stuff over to Intel, but I'm not quite there yet. Oh, nice. I like the idea behind that script. I think I understand the rationale behind it too. I made some updates to your script (also attached to the email)... http://try.bro.org/#/trybro/saved/d1269a5c-4099-4f55-aed9-82f59fc9e3dd I don't see any reason why this script wouldn't work (on single workers, it won't work well on a cluster). You'll need to add your own list of authorized fqdns (probably in local.bro after you load this script), like this... @load connection_validation redef ConnectionValidation::authorized_fqdns += { "a.example.com", "b.example.com", }; If you try it, let me know how it works for you! -------------- next part -------------- A non-text attachment was scrubbed... Name: connection_validation.bro Type: application/octet-stream Size: 2294 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141008/32538449/attachment.obj -------------- next part -------------- .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From damian.gerow at shopify.com Wed Oct 8 08:15:27 2014 From: damian.gerow at shopify.com (Damian Gerow) Date: Wed, 8 Oct 2014 11:15:27 -0400 Subject: [Bro] Cluster state synchronization In-Reply-To: <897E1997-31E2-410F-A5A6-EF34B62D4FCB@icir.org> References: <2BE14E49-64A6-4ACD-9EDC-6B817A346999@icir.org> <897E1997-31E2-410F-A5A6-EF34B62D4FCB@icir.org> Message-ID: On Wed, Oct 8, 2014 at 10:19 AM, Seth Hall wrote: > > A shortened version of the script I'm using for testing is at > https://gist.github.com/mutemule/a36f49b16db51eccd159. If I move the > 'add' commands into their own functions, and then prioritize the 'add_' > over the 'is_' functions, would that be a reasonable way to ensure my sets > are updated before being used for lookups? I'm already planning to migrate > some of our stuff over to Intel, but I'm not quite there yet. > > Oh, nice. I like the idea behind that script. I think I understand the > rationale behind it too. > > I made some updates to your script (also attached to the email)... > > http://try.bro.org/#/trybro/saved/d1269a5c-4099-4f55-aed9-82f59fc9e3dd Thanks! That's the first step in my cleanup done. ;) But I don't see much of a difference between these scripts, as it relates to my problem with the timeliness of set updates. Is there anything in particular you've done that might result in sets being updated before being referenced for lookups? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141008/e2f6efb3/attachment.html From seth at icir.org Wed Oct 8 09:21:35 2014 From: seth at icir.org (Seth Hall) Date: Wed, 8 Oct 2014 12:21:35 -0400 Subject: [Bro] Cluster state synchronization In-Reply-To: References: <2BE14E49-64A6-4ACD-9EDC-6B817A346999@icir.org> <897E1997-31E2-410F-A5A6-EF34B62D4FCB@icir.org> Message-ID: <9D582A7E-3399-43C4-AF70-F0B6F5F6FA96@icir.org> On Oct 8, 2014, at 11:15 AM, Damian Gerow wrote: > But I don't see much of a difference between these scripts, as it relates to my problem with the timeliness of set updates. Is there anything in particular you've done that might result in sets being updated before being referenced for lookups? Not really. You had some code in there that was a little difficult to follow which was why I started doing the reorganization and cleanup. I'm wondering if you might have had a bug in there before. Could you try running my updated script and seeing if that works (I wish I had a concrete answer, but I don't yet). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From damian.gerow at shopify.com Wed Oct 8 09:38:27 2014 From: damian.gerow at shopify.com (Damian Gerow) Date: Wed, 8 Oct 2014 12:38:27 -0400 Subject: [Bro] Cluster state synchronization In-Reply-To: <9D582A7E-3399-43C4-AF70-F0B6F5F6FA96@icir.org> References: <2BE14E49-64A6-4ACD-9EDC-6B817A346999@icir.org> <897E1997-31E2-410F-A5A6-EF34B62D4FCB@icir.org> <9D582A7E-3399-43C4-AF70-F0B6F5F6FA96@icir.org> Message-ID: On Wed, Oct 8, 2014 at 12:21 PM, Seth Hall wrote: > > But I don't see much of a difference between these scripts, as it > relates to my problem with the timeliness of set updates. Is there > anything in particular you've done that might result in sets being updated > before being referenced for lookups? > > Not really. You had some code in there that was a little difficult to > follow which was why I started doing the reorganization and cleanup. I'm > wondering if you might have had a bug in there before. Could you try > running my updated script and seeing if that works (I wish I had a concrete > answer, but I don't yet). > Done, and the end result is the same: sets are being updated slowly. For one CNAME in question, it took multiple queries and a few minutes for it to show up in the CNAMEs set. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141008/535d72dc/attachment.html From seth at icir.org Wed Oct 8 09:47:08 2014 From: seth at icir.org (Seth Hall) Date: Wed, 8 Oct 2014 12:47:08 -0400 Subject: [Bro] Cluster state synchronization In-Reply-To: References: <2BE14E49-64A6-4ACD-9EDC-6B817A346999@icir.org> <897E1997-31E2-410F-A5A6-EF34B62D4FCB@icir.org> <9D582A7E-3399-43C4-AF70-F0B6F5F6FA96@icir.org> Message-ID: <8EB75B6E-0F7A-49E1-8F63-71B931E356B2@icir.org> On Oct 8, 2014, at 12:38 PM, Damian Gerow wrote: > Done, and the end result is the same: sets are being updated slowly. For one CNAME in question, it took multiple queries and a few minutes for it to show up in the CNAMEs set.  Weird. Could you try removing the &persistent attribute from those variables with it set? That attribute hasn't been used much ever, especially in the past couple of years. It's probably going to be removed before too many more releases too (replaced by something else).  It would help if you had some traffic that showed this effect too. It possible that there is something weird going on with your traffic that is giving this appearance. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From damian.gerow at shopify.com Wed Oct 8 11:17:00 2014 From: damian.gerow at shopify.com (Damian Gerow) Date: Wed, 8 Oct 2014 14:17:00 -0400 Subject: [Bro] Cluster state synchronization In-Reply-To: <8EB75B6E-0F7A-49E1-8F63-71B931E356B2@icir.org> References: <2BE14E49-64A6-4ACD-9EDC-6B817A346999@icir.org> <897E1997-31E2-410F-A5A6-EF34B62D4FCB@icir.org> <9D582A7E-3399-43C4-AF70-F0B6F5F6FA96@icir.org> <8EB75B6E-0F7A-49E1-8F63-71B931E356B2@icir.org> Message-ID: On Wed, Oct 8, 2014 at 12:47 PM, Seth Hall wrote: > > Done, and the end result is the same: sets are being updated slowly. > For one CNAME in question, it took multiple queries and a few minutes for > it to show up in the CNAMEs set. > > Weird. Could you try removing the &persistent attribute from those > variables with it set? That attribute hasn't been used much ever, > especially in the past couple of years. It's probably going to be removed > before too many more releases too (replaced by something else). > I can try, but we kind of depend on persistence: without it, Bro goes a bit nuts on startup until it sees all the DNS queries again. I've been toying with the idea of resolving each member of the list and adding to the appropriate address set during bro_init(), but that would still leave some gaps. > It would help if you had some traffic that showed this effect too. It > possible that there is something weird going on with your traffic that is > giving this appearance. > The CNAME I mentioned above was a bad example: looks like the host was caching some aspects of it (namely, that it was a CNAME), so Bro never had a chance. I'll have another look at the network: we just had another false positive arrive, but the DNS log shows the query happening after the connection. Thanks! Looks like the issue may be in the network itself. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141008/5134e4c6/attachment.html From liburdi.joshua at gmail.com Thu Oct 9 07:50:23 2014 From: liburdi.joshua at gmail.com (Josh Liburdi) Date: Thu, 9 Oct 2014 07:50:23 -0700 Subject: [Bro] Cluster state synchronization In-Reply-To: References: <2BE14E49-64A6-4ACD-9EDC-6B817A346999@icir.org> <897E1997-31E2-410F-A5A6-EF34B62D4FCB@icir.org> <9D582A7E-3399-43C4-AF70-F0B6F5F6FA96@icir.org> <8EB75B6E-0F7A-49E1-8F63-71B931E356B2@icir.org> Message-ID: As far as persistence goes, I had a similar need and ignored the &persistent attribute; instead I manually append the data to a file and then read the file into a table on bro_init. This seemed to work well on some large scale (10 GB pcap) tests, but I haven't finished the script to run in prod yet. On Wed, Oct 8, 2014 at 11:17 AM, Damian Gerow wrote: > On Wed, Oct 8, 2014 at 12:47 PM, Seth Hall wrote: >> >> > Done, and the end result is the same: sets are being updated slowly. >> > For one CNAME in question, it took multiple queries and a few minutes for it >> > to show up in the CNAMEs set. >> >> Weird. Could you try removing the &persistent attribute from those >> variables with it set? That attribute hasn't been used much ever, >> especially in the past couple of years. It's probably going to be removed >> before too many more releases too (replaced by something else). > > > I can try, but we kind of depend on persistence: without it, Bro goes a bit > nuts on startup until it sees all the DNS queries again. I've been toying > with the idea of resolving each member of the list and adding to the > appropriate address set during bro_init(), but that would still leave some > gaps. > >> >> It would help if you had some traffic that showed this effect too. It >> possible that there is something weird going on with your traffic that is >> giving this appearance. > > > The CNAME I mentioned above was a bad example: looks like the host was > caching some aspects of it (namely, that it was a CNAME), so Bro never had a > chance. I'll have another look at the network: we just had another false > positive arrive, but the DNS log shows the query happening after the > connection. > > Thanks! Looks like the issue may be in the network itself. > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at icir.org Thu Oct 9 09:36:12 2014 From: seth at icir.org (Seth Hall) Date: Thu, 9 Oct 2014 12:36:12 -0400 Subject: [Bro] Cluster state synchronization In-Reply-To: References: <2BE14E49-64A6-4ACD-9EDC-6B817A346999@icir.org> <897E1997-31E2-410F-A5A6-EF34B62D4FCB@icir.org> <9D582A7E-3399-43C4-AF70-F0B6F5F6FA96@icir.org> <8EB75B6E-0F7A-49E1-8F63-71B931E356B2@icir.org> Message-ID: <860719C8-198B-43AC-998B-51AEC5DEF8DC@icir.org> On Oct 8, 2014, at 2:17 PM, Damian Gerow wrote: > Thanks! Looks like the issue may be in the network itself. Ok, that makes me feel better about the persistence at least. I'd be curious about what's causing things on your network to show up in a weird order though. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jlay at slave-tothe-box.net Thu Oct 9 12:55:10 2014 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 09 Oct 2014 13:55:10 -0600 Subject: [Bro] Question on "already defined (Notice::policy)" error Message-ID: Hi All, I'm dabbling with getting Bro to email, so I've added this to my local.bro: redef Notice::mail_dest = "myemail at address.com"; redef Notice::policy += { [$result = Notice::ACTION_EMAIL, $pred(n: Notice::Info) = { return n$note == PacketFilter::Dropped_Packets; } ] }; but I'm getting: error in /usr/local/bro/share/bro/base/frameworks/notice/./main.bro, line 183 and /usr/local/bro/share/bro/site/local.bro, line 101: already defined (Notice::policy) Is there something I'm missing? Thank you. James From gfaulkner.nsm at gmail.com Thu Oct 9 13:39:14 2014 From: gfaulkner.nsm at gmail.com (Gary Faulkner) Date: Thu, 09 Oct 2014 15:39:14 -0500 Subject: [Bro] Question on "already defined (Notice::policy)" error In-Reply-To: References: Message-ID: <5436F272.2030703@gmail.com> Have you tried something like this for defining notices you want emails on: redef Notice::emailed_types += { PacketFilter::Dropped_Packets, }; On 10/9/2014 2:55 PM, James Lay wrote: > Hi All, > > I'm dabbling with getting Bro to email, so I've added this to my > local.bro: > > redef Notice::mail_dest = "myemail at address.com"; > > redef Notice::policy += { > [$result = Notice::ACTION_EMAIL, > $pred(n: Notice::Info) = > { return n$note == PacketFilter::Dropped_Packets; } > ] > }; > > but I'm getting: > > error in /usr/local/bro/share/bro/base/frameworks/notice/./main.bro, > line 183 and /usr/local/bro/share/bro/site/local.bro, line 101: already > defined (Notice::policy) > > Is there something I'm missing? Thank you. > > James > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jlay at slave-tothe-box.net Thu Oct 9 13:44:12 2014 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 09 Oct 2014 14:44:12 -0600 Subject: [Bro] Question on "already defined (Notice::policy)" error In-Reply-To: <5436F272.2030703@gmail.com> References: <5436F272.2030703@gmail.com> Message-ID: <46aae2f6245b4e6d89cb6cfb235dcf94@localhost> On 2014-10-09 14:39, Gary Faulkner wrote: > Have you tried something like this for defining notices you want > emails on: > > redef Notice::emailed_types += { > PacketFilter::Dropped_Packets, > }; > > > On 10/9/2014 2:55 PM, James Lay wrote: >> Hi All, >> >> I'm dabbling with getting Bro to email, so I've added this to my >> local.bro: >> >> redef Notice::mail_dest = "myemail at address.com"; >> >> redef Notice::policy += { >> [$result = Notice::ACTION_EMAIL, >> $pred(n: Notice::Info) = >> { return n$note == PacketFilter::Dropped_Packets; } >> ] >> }; >> >> but I'm getting: >> >> error in /usr/local/bro/share/bro/base/frameworks/notice/./main.bro, >> line 183 and /usr/local/bro/share/bro/site/local.bro, line 101: >> already >> defined (Notice::policy) >> >> Is there something I'm missing? Thank you. >> >> James >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro Giving that a go now...thanks Gary. James From jlay at slave-tothe-box.net Thu Oct 9 13:53:46 2014 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 09 Oct 2014 14:53:46 -0600 Subject: [Bro] Question on "already defined (Notice::policy)" error In-Reply-To: <5436F272.2030703@gmail.com> References: <5436F272.2030703@gmail.com> Message-ID: <53dc763f686492b1ccccb5eff09889fc@localhost> On 2014-10-09 14:39, Gary Faulkner wrote: > Have you tried something like this for defining notices you want > emails on: > > redef Notice::emailed_types += { > PacketFilter::Dropped_Packets, > }; > > > On 10/9/2014 2:55 PM, James Lay wrote: >> Hi All, >> >> I'm dabbling with getting Bro to email, so I've added this to my >> local.bro: >> >> redef Notice::mail_dest = "myemail at address.com"; >> >> redef Notice::policy += { >> [$result = Notice::ACTION_EMAIL, >> $pred(n: Notice::Info) = >> { return n$note == PacketFilter::Dropped_Packets; } >> ] >> }; >> >> but I'm getting: >> >> error in /usr/local/bro/share/bro/base/frameworks/notice/./main.bro, >> line 183 and /usr/local/bro/share/bro/site/local.bro, line 101: >> already >> defined (Notice::policy) >> >> Is there something I'm missing? Thank you. >> >> James Thanks again Gary...I remember now looking at emailing direct from bro when I first started out...and this is why I think I stopped: sh: 1: /usr/sbin/sendmail: not found I do not have sendmail installed....is there a way to redefine which email application that gets used? I use an app called sendEmail: http://caspian.dotconf.net/menu/Software/SendEmail/ Thanks again for all the assistance...getting closer. James From jlay at slave-tothe-box.net Thu Oct 9 14:48:26 2014 From: jlay at slave-tothe-box.net (James Lay) Date: Thu, 09 Oct 2014 15:48:26 -0600 Subject: [Bro] Mal-dnssearch issue Message-ID: Hey again all, Got almost all the intel feeds that I'm looking to get save one...malips. From: http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html I'm running: mal-dnssearch -M malips -p | mal-dns2bro -T ip -s malips > malips.intel However the results looks muffed: head malips.intel #fields indicator indicator_type meta.source meta.url meta.do_notice meta.if_in 100.42.5Intel::ADDR malips - F - 103.14.1Intel::ADDR malips - F - 103.19.8Intel::ADDR malips - F - The others all look fine. Again, am I missing a flag or something? Thank you. James From dnthayer at illinois.edu Thu Oct 9 15:39:11 2014 From: dnthayer at illinois.edu (Daniel Thayer) Date: Thu, 9 Oct 2014 17:39:11 -0500 Subject: [Bro] Question on "already defined (Notice::policy)" error In-Reply-To: <53dc763f686492b1ccccb5eff09889fc@localhost> References: <5436F272.2030703@gmail.com> <53dc763f686492b1ccccb5eff09889fc@localhost> Message-ID: <54370E8F.8000704@illinois.edu> On 10/09/2014 03:53 PM, James Lay wrote: > On 2014-10-09 14:39, Gary Faulkner wrote: >> Have you tried something like this for defining notices you want >> emails on: >> >> redef Notice::emailed_types += { >> PacketFilter::Dropped_Packets, >> }; >> >> >> On 10/9/2014 2:55 PM, James Lay wrote: >>> Hi All, >>> >>> I'm dabbling with getting Bro to email, so I've added this to my >>> local.bro: >>> >>> redef Notice::mail_dest = "myemail at address.com"; >>> >>> redef Notice::policy += { >>> [$result = Notice::ACTION_EMAIL, >>> $pred(n: Notice::Info) = >>> { return n$note == PacketFilter::Dropped_Packets; } >>> ] >>> }; >>> >>> but I'm getting: >>> >>> error in /usr/local/bro/share/bro/base/frameworks/notice/./main.bro, >>> line 183 and /usr/local/bro/share/bro/site/local.bro, line 101: >>> already >>> defined (Notice::policy) >>> >>> Is there something I'm missing? Thank you. >>> >>> James > > Thanks again Gary...I remember now looking at emailing direct from bro > when I first started out...and this is why I think I stopped: > > sh: 1: /usr/sbin/sendmail: not found > > I do not have sendmail installed....is there a way to redefine which > email application that gets used? I use an app called sendEmail: > > http://caspian.dotconf.net/menu/Software/SendEmail/ > > Thanks again for all the assistance...getting closer. > > James You could try this: https://www.bro.org/sphinx-git/scripts/base/frameworks/notice/main.bro.html#id-Notice::sendmail From jlay at slave-tothe-box.net Fri Oct 10 07:44:17 2014 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 10 Oct 2014 08:44:17 -0600 Subject: [Bro] Mal-dnssearch issue In-Reply-To: References: Message-ID: <331364aba4ecf19b60b6023302ea6c7a@localhost> On 2014-10-09 15:48, James Lay wrote: > Hey again all, > > Got almost all the intel feeds that I'm looking to get save > one...malips. From: > > http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html > > I'm running: > > mal-dnssearch -M malips -p | mal-dns2bro -T ip -s malips > > malips.intel > > However the results looks muffed: > > head malips.intel > #fields indicator indicator_type meta.source meta.url > meta.do_notice meta.if_in > 100.42.5Intel::ADDR malips - F - > 103.14.1Intel::ADDR malips - F - > 103.19.8Intel::ADDR malips - F - > > The others all look fine. Again, am I missing a flag or something? > Thank you. > > James Some additional info shows that there's a carriage return after the IP...doing a :set list in vim shows: 100.42.50.110^M^IIntel::ADDR^Imalips^I-^IF^I-$ None of the other .intel files show the ^M. Thanks all. James From jonschipp at gmail.com Fri Oct 10 10:22:41 2014 From: jonschipp at gmail.com (Jon Schipp) Date: Fri, 10 Oct 2014 12:22:41 -0500 Subject: [Bro] Mal-dnssearch issue In-Reply-To: <331364aba4ecf19b60b6023302ea6c7a@localhost> References: <331364aba4ecf19b60b6023302ea6c7a@localhost> Message-ID: Hello James, Sorry, I've been really busy. Thanks for reporting, I'll look into it. For any specific issue with the script you can create an issue on Github and I'll take care of it :) On Fri, Oct 10, 2014 at 9:44 AM, James Lay wrote: > On 2014-10-09 15:48, James Lay wrote: >> Hey again all, >> >> Got almost all the intel feeds that I'm looking to get save >> one...malips. From: >> >> http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html >> >> I'm running: >> >> mal-dnssearch -M malips -p | mal-dns2bro -T ip -s malips > >> malips.intel >> >> However the results looks muffed: >> >> head malips.intel >> #fields indicator indicator_type meta.source meta.url >> meta.do_notice meta.if_in >> 100.42.5Intel::ADDR malips - F - >> 103.14.1Intel::ADDR malips - F - >> 103.19.8Intel::ADDR malips - F - >> >> The others all look fine. Again, am I missing a flag or something? >> Thank you. >> >> James > > Some additional info shows that there's a carriage return after the > IP...doing a :set list in vim shows: > > 100.42.50.110^M^IIntel::ADDR^Imalips^I-^IF^I-$ > > None of the other .intel files show the ^M. Thanks all. > > James > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Jon Schipp, jonschipp.com, sickbits.net From jlay at slave-tothe-box.net Fri Oct 10 10:49:42 2014 From: jlay at slave-tothe-box.net (James Lay) Date: Fri, 10 Oct 2014 11:49:42 -0600 Subject: [Bro] Mal-dnssearch issue In-Reply-To: References: <331364aba4ecf19b60b6023302ea6c7a@localhost> Message-ID: On 2014-10-10 11:22, Jon Schipp wrote: > Hello James, > > Sorry, I've been really busy. Thanks for reporting, I'll look into > it. > For any specific issue with the script you can create an issue on > Github and I'll take care of it :) > > On Fri, Oct 10, 2014 at 9:44 AM, James Lay > wrote: >> On 2014-10-09 15:48, James Lay wrote: >>> Hey again all, >>> >>> Got almost all the intel feeds that I'm looking to get save >>> one...malips. From: >>> >>> http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html >>> >>> I'm running: >>> >>> mal-dnssearch -M malips -p | mal-dns2bro -T ip -s malips > >>> malips.intel >>> >>> However the results looks muffed: >>> >>> head malips.intel >>> #fields indicator indicator_type meta.source meta.url >>> meta.do_notice meta.if_in >>> 100.42.5Intel::ADDR malips - F - >>> 103.14.1Intel::ADDR malips - F - >>> 103.19.8Intel::ADDR malips - F - >>> >>> The others all look fine. Again, am I missing a flag or something? >>> Thank you. >>> >>> James >> >> Some additional info shows that there's a carriage return after the >> IP...doing a :set list in vim shows: >> >> 100.42.50.110^M^IIntel::ADDR^Imalips^I-^IF^I-$ >> >> None of the other .intel files show the ^M. Thanks all. >> >> James Did so thanks Jon...I'll get work with this off list. James From jonschipp at gmail.com Fri Oct 10 11:13:20 2014 From: jonschipp at gmail.com (Jon Schipp) Date: Fri, 10 Oct 2014 13:13:20 -0500 Subject: [Bro] Mal-dnssearch issue In-Reply-To: References: <331364aba4ecf19b60b6023302ea6c7a@localhost> Message-ID: Oh you did, awesome. I didn't quite make it to that e-mail :) It's fixed: https://github.com/jonschipp/mal-dnssearch/commit/2b9e5bb6797e1dcfcbf5e6f5368704d18765e2b1 On Fri, Oct 10, 2014 at 12:49 PM, James Lay wrote: > On 2014-10-10 11:22, Jon Schipp wrote: >> >> Hello James, >> >> Sorry, I've been really busy. Thanks for reporting, I'll look into it. >> For any specific issue with the script you can create an issue on >> Github and I'll take care of it :) >> >> On Fri, Oct 10, 2014 at 9:44 AM, James Lay >> wrote: >>> >>> On 2014-10-09 15:48, James Lay wrote: >>>> >>>> Hey again all, >>>> >>>> Got almost all the intel feeds that I'm looking to get save >>>> one...malips. From: >>>> >>>> http://blog.bro.org/2014/01/intelligence-data-and-bro_4980.html >>>> >>>> I'm running: >>>> >>>> mal-dnssearch -M malips -p | mal-dns2bro -T ip -s malips > >>>> malips.intel >>>> >>>> However the results looks muffed: >>>> >>>> head malips.intel >>>> #fields indicator indicator_type meta.source meta.url >>>> meta.do_notice meta.if_in >>>> 100.42.5Intel::ADDR malips - F - >>>> 103.14.1Intel::ADDR malips - F - >>>> 103.19.8Intel::ADDR malips - F - >>>> >>>> The others all look fine. Again, am I missing a flag or something? >>>> Thank you. >>>> >>>> James >>> >>> >>> Some additional info shows that there's a carriage return after the >>> IP...doing a :set list in vim shows: >>> >>> 100.42.50.110^M^IIntel::ADDR^Imalips^I-^IF^I-$ >>> >>> None of the other .intel files show the ^M. Thanks all. >>> >>> James > > > Did so thanks Jon...I'll get work with this off list. > > James > -- Jon Schipp, jonschipp.com, sickbits.net From doris at bro.org Fri Oct 10 14:20:21 2014 From: doris at bro.org (Doris Schioberg) Date: Fri, 10 Oct 2014 14:20:21 -0700 Subject: [Bro] Bro Monthly #2 Message-ID: <54384D95.7040105@bro.org> This month's Bro newsletter covers Shellshock, BinPAC++, new features for the Intel Framework, and a lot of other topics. Find out what we did last month: http://blog.bro.org/2014/10/bro-monthly-2.html As always, feedback is very welcome! -The Bro Team -- Doris Schioberg Bro Outreach, Training, and Education Coordinator International Computer Science Institute (ICSI Berkeley) Phone: +1 (510) 289-8406 * doris at bro.org From jlay at slave-tothe-box.net Mon Oct 13 10:36:06 2014 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 13 Oct 2014 11:36:06 -0600 Subject: [Bro] Redefining the email application Message-ID: <806c97b26f28d8e9b4ed6f94e778159a@localhost> Hey All, I've been trying to figured out how to get a different application as the sendmail app. I've not been able to find anything for this, so I'm coming here for an assist. If not being able to redefine sendmail to something different, is there a way I can instead fire off a script that will run my email application instead? I've looked at: https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html For a while now, but I'm not able to figure it out. Thanks for any assistance. James From gc355804 at ohio.edu Mon Oct 13 11:08:19 2014 From: gc355804 at ohio.edu (Clark, Gilbert) Date: Mon, 13 Oct 2014 18:08:19 +0000 Subject: [Bro] Redefining the email application In-Reply-To: <806c97b26f28d8e9b4ed6f94e778159a@localhost> References: <806c97b26f28d8e9b4ed6f94e778159a@localhost> Message-ID: <1413223700763.95073@ohio.edu> Hi James: Disclaimer: I've never done this before, so the following may be completely wrong. with that said ... In base/frameworks/notice/main.bro, there is a function called 'email_notice_to'. In this function, there is a call that looks like this: piped_exec(fmt("%s -t -oi", sendmail), email_text); piped_exec is, I think, what actually makes the call to sendmail, where sendmail is defined to be (in the same file): ## Local system sendmail program. ## ## Note that this is overridden by the BroControl SendMail option. const sendmail = "/usr/sbin/sendmail" &redef; So, based on the above, I can see a few options: * Use an application with sendmail compatibility and redefine Notice::sendmail to point to that * Maybe check broctl's SendMail option to make sure it's not overwriting the value you'd like (since it mentions that broctl takes precedence in the comment above)? * If neither of the above work, symlink /usr/bin/sendmail to the actual application you're trying to forward the mail to and see if that works? * If the above isn't an option, write a shim that accepts the -t -oi options and forwards the body of the mail to the desired application * If none of the above work, maybe modify email_notice_to to make a different piped_exec() call? Note that this might be bad because changes would be lost at upgrade ... Hope something in there is useful. -Gilbert ________________________________________ From: bro-bounces at bro.org on behalf of James Lay Sent: Monday, October 13, 2014 1:36 PM To: bro at bro-ids.org Subject: [Bro] Redefining the email application Hey All, I've been trying to figured out how to get a different application as the sendmail app. I've not been able to find anything for this, so I'm coming here for an assist. If not being able to redefine sendmail to something different, is there a way I can instead fire off a script that will run my email application instead? I've looked at: https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html For a while now, but I'm not able to figure it out. Thanks for any assistance. James _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jlay at slave-tothe-box.net Mon Oct 13 13:19:29 2014 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 13 Oct 2014 14:19:29 -0600 Subject: [Bro] Interesting intel.log issue Message-ID: <3b7b270e63afca08b0ea75d5289776ff@localhost> Topic says it...for the most part most of my intel.log looks like: 1413230008.288997 CV0p4G1epvXb4Cagma x.x.x.x 41918 50.63.40.1 80 - - - 50.63.40.1 Intel::ADDR Conn::IN_RESP alienvault 1413230008.357789 CR6AUc3tAVZKpxue2c x.x.x.x 38068 50.63.40.1 80 - - - 50.63.40.1 Intel::ADDR Conn::IN_RESP alienvault 1413230267.919296 C4AHVH2Y7UUpTBbl2 x.x.x.x 49880 208.109.181.58 80 - - - 208.109.181.58 Intel::ADDR Conn::IN_RESP alienvault 1413230268.588344 CkxyU02h5MNCvSl4jc x.x.x.x 59045 208.109.181.58 80 - - - 208.109.181.58 Intel::ADDR Conn::IN_RESP alienvault But sometimes I see: 1413230008.288997 CV0p4G1epvXb4Cagma x.x.x.x 41918 50.63.40.1 80 - - - 50.63.40.1 Intel::ADDR Conn::IN_RESP alienvault 1413230008.357789 CR6AUc3tAVZKpxue2c x.x.x.x 38068 50.63.40.1 80 - - - 50.63.40.1 Intel::ADDR Conn::IN_RESP alienvaul1413230267.919296 C4AHVH2Y7UUpTBbl2 x.x.x.x 49880 208.109.181.58 80 - - - 208.109.181.58 Intel::ADDR Conn::IN_RESP alienvault 1411413230268.588344 CkxyU02h5MNCvSl4jc x.x.x.x 59045 208.109.181.58 80 - - - 208.109.181.58 Intel::ADDR Conn::IN_RESP alienvault where the timestamp has been tagged on to the end of the previous line..in this case it nukes the 't' in alienvault in the second line. Weird....currently I'm tailing this file and while it's a mild issue, if I was going to generate a report with grep/sed/awk as I do with some of the others this would be a problem. Thank you. James From jlay at slave-tothe-box.net Mon Oct 13 13:30:22 2014 From: jlay at slave-tothe-box.net (James Lay) Date: Mon, 13 Oct 2014 14:30:22 -0600 Subject: [Bro] Redefining the email application In-Reply-To: <1413223700763.95073@ohio.edu> References: <806c97b26f28d8e9b4ed6f94e778159a@localhost> <1413223700763.95073@ohio.edu> Message-ID: <1ca9cda861cfe80b9d2183194066f9a7@localhost> On 2014-10-13 12:08, Clark, Gilbert wrote: > Hi James: > > Disclaimer: I've never done this before, so the following may be > completely wrong. with that said ... > > In base/frameworks/notice/main.bro, there is a function called > 'email_notice_to'. In this function, there is a call that looks like > this: > > piped_exec(fmt("%s -t -oi", sendmail), email_text); > > piped_exec is, I think, what actually makes the call to sendmail, > where sendmail is defined to be (in the same file): > > ## Local system sendmail program. > ## > ## Note that this is overridden by the BroControl SendMail > option. > const sendmail = "/usr/sbin/sendmail" &redef; > > So, based on the above, I can see a few options: > > * Use an application with sendmail compatibility and redefine > Notice::sendmail to point to that > * Maybe check broctl's SendMail option to make sure it's not > overwriting the value you'd like (since it mentions that broctl takes > precedence in the comment above)? > * If neither of the above work, symlink /usr/bin/sendmail to the > actual application you're trying to forward the mail to and see if > that works? > * If the above isn't an option, write a shim that accepts the -t -oi > options and forwards the body of the mail to the desired application > * If none of the above work, maybe modify email_notice_to to make a > different piped_exec() call? Note that this might be bad because > changes would be lost at upgrade ... > > Hope something in there is useful. > > -Gilbert > ________________________________________ > From: bro-bounces at bro.org on behalf of James > Lay > Sent: Monday, October 13, 2014 1:36 PM > To: bro at bro-ids.org > Subject: [Bro] Redefining the email application > > Hey All, > > I've been trying to figured out how to get a different application as > the sendmail app. I've not been able to find anything for this, so > I'm > coming here for an assist. If not being able to redefine sendmail to > something different, is there a way I can instead fire off a script > that > will run my email application instead? I've looked at: > > > https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html > > For a while now, but I'm not able to figure it out. Thanks for any > assistance. > > James Thanks for looking at this Gilbert...that helps. James From silusilusilu at gmail.com Thu Oct 16 05:18:34 2014 From: silusilusilu at gmail.com (fasf safas) Date: Thu, 16 Oct 2014 14:18:34 +0200 Subject: [Bro] Newbie question on Bro and NetBIOS protocol Message-ID: Hi, i'm a newbie and i'm studying BRO: i'm using BRO in standard configuration without any plugin. I have some problems with NetBIOS protocol: i've executed the test described below 1. nbtstat "a specific IP": i've obtained the NetBIOS name. Wireshark and Bro were in background 2. From Wireshark i'm able to see two packets: the first is the NBSTAT name query, the second one its response. 3. In Bro (under dns.log) i'm able to see only the query, but not its response...so i'm not able to see the netbios name. What's wrong? Thanks in advance, Alvin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141016/07a6cac2/attachment.html From phyltr at flavourcountry.org Thu Oct 16 06:31:04 2014 From: phyltr at flavourcountry.org (Karl Hill) Date: Thu, 16 Oct 2014 07:31:04 -0600 Subject: [Bro] Fault tolerance Message-ID: Hello, I'm in a situation where I need to have some fault-tolerance with Bro. I need Bro data in two geographically disparate sites. Since I can't setup two managers I'm not sure how to go about accomplishing this. From davidvasil at gmail.com Fri Oct 17 05:49:20 2014 From: davidvasil at gmail.com (David Vasil) Date: Fri, 17 Oct 2014 07:49:20 -0500 Subject: [Bro] SSH:ignore_guessers Message-ID: I would like to redef my SSH::ignore_guessers to exclude hosts that I know will be consistently triggering the SSH::Password_Guessing alert due to legitimate business processes. I've tried the following (10.0.0.2 is the host performing the scanning in this example): redef SSH::ignore_guessers += { [10.0.0.2/32] = 211.11.11.211/32, [10.0.0.2/32] = 10.2.2.2/32, }; in my local.bro, did a broctl check/broctl install/broctl restart, but I still receive alerts. I am assuming that the key-value format I am trying to use is incorrect, but the code only states: ## The index represents client subnets and the yield value represents server subnets. How does one set SSH:ignore_guessers like I am trying to do? Thanks! -Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141017/02f0e62b/attachment.html From jdonnelly at dyn.com Fri Oct 17 12:27:53 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Fri, 17 Oct 2014 14:27:53 -0500 Subject: [Bro] Limiting the number of scripts and log files bro uses ? Message-ID: Hello , I am interested in learning how to limit the number of logs and scripts bro uses. With the default installation installed via source method (make install) bro uses a huge number of scripts loaded as reported by broctl cli's command "scripts" . Is there a way to reduce those numbers down ? Maybe a better question would be : " I just want dns logger generated " - is there a way to do that ? Thank you . JD. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141017/4cc97b61/attachment.html From seth at icir.org Fri Oct 17 13:15:08 2014 From: seth at icir.org (Seth Hall) Date: Fri, 17 Oct 2014 16:15:08 -0400 Subject: [Bro] Limiting the number of scripts and log files bro uses ? In-Reply-To: References: Message-ID: <2C24D46A-7678-440A-B0B1-D7C191043ADF@icir.org> On Oct 17, 2014, at 3:27 PM, John Donnelly wrote: > Maybe a better question would be : " I just want dns logger generated " - is there a way to do that ?  You can run in "bare mode" with the -b flag. That will cause Bro to only load stuff that is necessary to load due to a core dependency. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdonnelly at dyn.com Sun Oct 19 07:17:19 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Sun, 19 Oct 2014 09:17:19 -0500 Subject: [Bro] Limiting the number of scripts and log files bro uses ? In-Reply-To: <2C24D46A-7678-440A-B0B1-D7C191043ADF@icir.org> References: <2C24D46A-7678-440A-B0B1-D7C191043ADF@icir.org> Message-ID: Thank you. Can I start bro with using broctl shell? On Fri, Oct 17, 2014 at 3:15 PM, Seth Hall wrote: > > On Oct 17, 2014, at 3:27 PM, John Donnelly wrote: > > > Maybe a better question would be : " I just want dns logger generated " > - is there a way to do that ? > > You can run in "bare mode" with the -b flag. That will cause Bro to only > load stuff that is necessary to load due to a core dependency. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141019/cd77a560/attachment.html From vladimira at vfemail.net Sun Oct 19 19:19:08 2014 From: vladimira at vfemail.net (Vladimir Arseniev) Date: Sun, 19 Oct 2014 22:19:08 -0400 Subject: [Bro] Problem reading pcap file Message-ID: <5444711C.3040107@vfemail.net> While this is fundamentally an old question, the old answers aren't working for me. Using "bro -r", I get the classic "invalid UDP checksums" error. Using "bro -rC" (or "bro -r -C"), I get numerous errors about unrecognized characters (even with the "-r" flag). Using "bro -Cr" (or "bro -C -r"), I get no shell errors. However, I see just 13 packets in "conn.log" vs 24311 packets expected. Perhaps this is the new piece of my question (plus why "-rC" <> "-Cr"). How do I fix this? Some details might be useful. I compiled from bro-2.3.1.tar.gz in Debian 7.6 x64. I'm working with a 32MB capture from a Centos 6.5 VPS. I used dumpcap with a ring buffer: dumpcap -b filesize:102400 -b files:10 -i eth0 -w /home/user/eth0 I used Wireshark to restrict eth0_00001_20141014111022 to IPv4, yielding eth0_00001_20141014111022_IPv4. Then I used "editcap -F libpcap" to convert to eth0_00001_20141014111022_IPv4.pcap (hereinafter "eth0.pcap"). I get no joy reading eth0.pcap with bro: bro -r eth0.pcap 1413340801.822519 warning in /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54: Your trace file likely has invalid UDP checksums, most likely from NIC checksum offloading. bro -rC eth0.pcap error in ./eth0.pcap, line 1: unrecognized character - ... error in ./eth0.pcap, line 1: unknown identifier t, at or near "t" bro -Cr eth0.pcap [completes without errors, but conn.log is just 2.4KB] cat conn.log [see expected headers, but just 13 data lines] #close 2014-10-19-20-26-47 From anthony.kasza at gmail.com Sun Oct 19 20:43:52 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Sun, 19 Oct 2014 20:43:52 -0700 Subject: [Bro] Problem reading pcap file In-Reply-To: <5444711C.3040107@vfemail.net> References: <5444711C.3040107@vfemail.net> Message-ID: Are you willing to share a sample pcap that causes these errors? -AK On Oct 19, 2014 7:27 PM, "Vladimir Arseniev" wrote: > While this is fundamentally an old question, the old answers aren't > working for me. Using "bro -r", I get the classic "invalid UDP > checksums" error. Using "bro -rC" (or "bro -r -C"), I get numerous > errors about unrecognized characters (even with the "-r" flag). > > Using "bro -Cr" (or "bro -C -r"), I get no shell errors. However, I see > just 13 packets in "conn.log" vs 24311 packets expected. Perhaps this is > the new piece of my question (plus why "-rC" <> "-Cr"). > > How do I fix this? > > Some details might be useful. I compiled from bro-2.3.1.tar.gz in Debian > 7.6 x64. I'm working with a 32MB capture from a Centos 6.5 VPS. I used > dumpcap with a ring buffer: > > dumpcap -b filesize:102400 -b files:10 -i eth0 -w /home/user/eth0 > > I used Wireshark to restrict eth0_00001_20141014111022 to IPv4, yielding > eth0_00001_20141014111022_IPv4. Then I used "editcap -F libpcap" to > convert to eth0_00001_20141014111022_IPv4.pcap (hereinafter "eth0.pcap"). > > I get no joy reading eth0.pcap with bro: > > bro -r eth0.pcap > 1413340801.822519 warning in > /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line > 54: Your trace file likely has invalid UDP checksums, most likely from > NIC checksum offloading. > > bro -rC eth0.pcap > error in ./eth0.pcap, line 1: unrecognized character - > ... > error in ./eth0.pcap, line 1: unknown identifier t, at or near "t" > > bro -Cr eth0.pcap > [completes without errors, but conn.log is just 2.4KB] > > cat conn.log > [see expected headers, but just 13 data lines] > #close 2014-10-19-20-26-47 > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141019/3ab2e48f/attachment.html From vladimira at vfemail.net Sun Oct 19 21:53:57 2014 From: vladimira at vfemail.net (Vladimir Arseniev) Date: Mon, 20 Oct 2014 00:53:57 -0400 Subject: [Bro] Problem reading pcap file In-Reply-To: References: <5444711C.3040107@vfemail.net> Message-ID: <54449565.2060800@vfemail.net> On 10/19/2014 11:43 PM, anthony kasza wrote: > Are you willing to share a sample pcap that causes these errors? > > -AK Thank you. That was the perfect answer, because it led me to excerpt one conversation with a nonsensitive host. And reading that yielded a "conn.log" with just one line. [zap!] It's classic PEBKAC ;) Conversations <> packets. There are _only 13_ conversations in that capture. It's a bandwidth test, and ten of them are http with resp_bytes=1520560. Excuse: I've been reading about both Bro and Splunk, and got confused. > On Oct 19, 2014 7:27 PM, "Vladimir Arseniev" wrote: > >> While this is fundamentally an old question, the old answers aren't >> working for me. Using "bro -r", I get the classic "invalid UDP >> checksums" error. Using "bro -rC" (or "bro -r -C"), I get numerous >> errors about unrecognized characters (even with the "-r" flag). >> >> Using "bro -Cr" (or "bro -C -r"), I get no shell errors. However, I see >> just 13 packets in "conn.log" vs 24311 packets expected. Perhaps this is >> the new piece of my question (plus why "-rC" <> "-Cr"). >> >> How do I fix this? >> >> Some details might be useful. I compiled from bro-2.3.1.tar.gz in Debian >> 7.6 x64. I'm working with a 32MB capture from a Centos 6.5 VPS. I used >> dumpcap with a ring buffer: >> >> dumpcap -b filesize:102400 -b files:10 -i eth0 -w /home/user/eth0 >> >> I used Wireshark to restrict eth0_00001_20141014111022 to IPv4, yielding >> eth0_00001_20141014111022_IPv4. Then I used "editcap -F libpcap" to >> convert to eth0_00001_20141014111022_IPv4.pcap (hereinafter "eth0.pcap"). >> >> I get no joy reading eth0.pcap with bro: >> >> bro -r eth0.pcap >> 1413340801.822519 warning in >> /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line >> 54: Your trace file likely has invalid UDP checksums, most likely from >> NIC checksum offloading. >> >> bro -rC eth0.pcap >> error in ./eth0.pcap, line 1: unrecognized character - >> ... >> error in ./eth0.pcap, line 1: unknown identifier t, at or near "t" >> >> bro -Cr eth0.pcap >> [completes without errors, but conn.log is just 2.4KB] >> >> cat conn.log >> [see expected headers, but just 13 data lines] >> #close 2014-10-19-20-26-47 >> >> _______________________________________________ >> Bro mailing list >> bro at bro-ids.org >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro >> > From juan.caballero at imdea.org Mon Oct 20 03:31:15 2014 From: juan.caballero at imdea.org (Juan Caballero) Date: Mon, 20 Oct 2014 12:31:15 +0200 Subject: [Bro] arista & cpacket experience Message-ID: <007501cfec50$fa1e6af0$ee5b40d0$@imdea.org> Hi everyone, We would like to deploy a Bro Cluster at a 10 Gbps at about 35% peak usage. We already have a splitter in place and are discussing options for a front-end that can merge both traffic directions and load balance sessions to Bro workers based on session hash and MAC rewriting. Ideally we would like some equipment that supports multi-port mirroring so that we can add other monitoring tools in addition to the Bro Cluster (e.g., Snort, TimeMachine or other Storage). Robin mentioned to me that people are using Arista and CPacket switches for this kind of setup. After looking at their webpages the Arista 7150 seems like a possibility for us (I see on the web page the San Diego SDSC and Cornell use the larger 7500 series) and CPacket's cVu240NG may be another (although there is less information about CPacket products online). Does anyone have experience with these products? Do those models make sense for the description above? Any recommendations or things to consider for people without prior experience in such setups? Thanks! Juan Caballero Assistant Research Professor IMDEA Software Institute Madrid, Spain From vitologrillo at gmail.com Mon Oct 20 03:50:07 2014 From: vitologrillo at gmail.com (Vito Logrillo) Date: Mon, 20 Oct 2014 12:50:07 +0200 Subject: [Bro] Bro and NetBIOS Message-ID: Hi, i'm trying to use BRO to analyze data based on NetBIOS protocol: i'm using BRO 2.3.1. I've made a small script with these lines: ....snippet... const NetBIOSports = { 138/udp, 139/tcp, 445/tcp}; event bro_init() &priority=5 { Analyzer::register_for_ports(Analyzer::ANALYZER_NETBIOSSSN,NetBIOSports); } ................ Below you can see my stderr.log: Internal error: unknown analyzer name NETBIOS; mismatch with tag analyzer::Component? What does it mean? Another question: SMB2.0 protocol is supported by BRO or not? Thanks, Vito -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141020/57e59fad/attachment.html From mike.patterson at uwaterloo.ca Mon Oct 20 05:22:59 2014 From: mike.patterson at uwaterloo.ca (Mike Patterson) Date: Mon, 20 Oct 2014 12:22:59 +0000 Subject: [Bro] arista & cpacket experience In-Reply-To: <007501cfec50$fa1e6af0$ee5b40d0$@imdea.org> References: <007501cfec50$fa1e6af0$ee5b40d0$@imdea.org> Message-ID: <9B38ABCB-FF64-474C-B38A-1AE582EDE32D@uwaterloo.ca> > On Oct 20, 2014, at 6:31 AM, Juan Caballero wrote: > (about a 10 gig cluster expecting 3-4 gig peaks, wanting a way to more easily manage the distribution of traffic to monitoring systems, asking specifically about Arista and CPacket) > Does anyone have experience with these products? Do those models make sense > for the description above? We?ve been using an Arista 7150S for exactly this purpose, and our requirements are pretty similar (10 gig links, expected peaks are 4-4.5 gig, although we?ve seen 6-7). I can?t speak to CPacket. But yes, the Arista will fit exactly into what you want and pricing on them is pretty good. We don?t do load balancing on the switch though, we?re doing it on the NICs. I have a machine with an Endace DAG (older box, pre-Arista) and several with Intel x520 NICs using PF_RING. There should be several others on the list feeding Aristas to Intels, although I?ll let them speak about their own experiences. I?m finding I get better performance out of the DAG box, although not 10,000 dollars better performance (which is about the price gap). > Any recommendations or things to consider for people without prior > experience in such setups? Whatever you pick, figure out a way to integrate monitoring it into your environment, or accept that you?ll get to monitor it yourself. And unless your NOC is heavily invested in these things too, chances are it will be the latter, so you?ll get to play sysadmin too. :) Mike From seth at icir.org Mon Oct 20 06:07:38 2014 From: seth at icir.org (Seth Hall) Date: Mon, 20 Oct 2014 09:07:38 -0400 Subject: [Bro] Limiting the number of scripts and log files bro uses ? In-Reply-To: References: <2C24D46A-7678-440A-B0B1-D7C191043ADF@icir.org> Message-ID: On Oct 19, 2014, at 10:17 AM, John Donnelly wrote: > Can I start bro with using broctl shell? Not at this time. We don't provide a mechanism to load a limited subset of scripts with BroControl. You *could* try adding "BroArgs=-b" into broctl.cfg, but without looking, I have no clue if that would have any repercussions. If you did that, you'd also want to make sure you have a pretty empty local.bro since broctl loads local.bro too. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From dev25 at cornell.edu Mon Oct 20 07:00:13 2014 From: dev25 at cornell.edu (Dan Villanti) Date: Mon, 20 Oct 2014 14:00:13 +0000 Subject: [Bro] Limiting the number of scripts and log files bro uses ? In-Reply-To: References: <2C24D46A-7678-440A-B0B1-D7C191043ADF@icir.org> Message-ID: <1a7053c4d72f4dd28d65cf2b3df98a41@BLUPR04MB817.namprd04.prod.outlook.com> > You *could* try adding "BroArgs=-b" into broctl.cfg... We have been using the "BroArgs=-b" configuration parameter for a while. If you use this option in a cluster setup, be sure to load core frameworks such as cluster, notice, control, etc. in local.bro or you may see some strange things. You may just want to load all of the frameworks defined in /share/bro/base/init-default.bro and then selectively add desired functionality on top of that. It took us a little playing around to get all of the dependencies lined up for what we wanted to analyze, but the filtered results and performance increase were worth it. Dan From seth at icir.org Mon Oct 20 07:16:43 2014 From: seth at icir.org (Seth Hall) Date: Mon, 20 Oct 2014 10:16:43 -0400 Subject: [Bro] Limiting the number of scripts and log files bro uses ? In-Reply-To: <1a7053c4d72f4dd28d65cf2b3df98a41@BLUPR04MB817.namprd04.prod.outlook.com> References: <2C24D46A-7678-440A-B0B1-D7C191043ADF@icir.org> <1a7053c4d72f4dd28d65cf2b3df98a41@BLUPR04MB817.namprd04.prod.outlook.com> Message-ID: On Oct 20, 2014, at 10:00 AM, Dan Villanti wrote: >> If you use this option in a cluster setup, be sure to load core frameworks such as cluster, notice, control, etc. in local.bro or you may see some strange things. Hm, I'm surprised those things aren't automatically loaded. BroControl should be loading those even if you are running in bare mode. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Mon Oct 20 07:29:08 2014 From: seth at icir.org (Seth Hall) Date: Mon, 20 Oct 2014 10:29:08 -0400 Subject: [Bro] Fault tolerance In-Reply-To: References: Message-ID: <7B29334D-CC8D-46A6-B076-00653340E8F7@icir.org> On Oct 16, 2014, at 9:31 AM, Karl Hill wrote: > Hello, I'm in a situation where I need to have some fault-tolerance > with Bro. I need Bro data in two geographically disparate sites. Since > I can't setup two managers I'm not sure how to go about accomplishing > this.  It doesn't work to just copy the data to your secondary data center hourly? Maybe you could explain more about what sort of fault tolerance your looking for? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Mon Oct 20 07:30:10 2014 From: seth at icir.org (Seth Hall) Date: Mon, 20 Oct 2014 10:30:10 -0400 Subject: [Bro] Newbie question on Bro and NetBIOS protocol In-Reply-To: References: Message-ID: On Oct 16, 2014, at 8:18 AM, fasf safas wrote: > 3. In Bro (under dns.log) i'm able to see only the query, but not its response...so i'm not able to see the netbios name.  That should be working. The best way to help debug this would be if you provided us with a trace containing some packets that exhibit this problem. Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Mon Oct 20 07:33:27 2014 From: seth at icir.org (Seth Hall) Date: Mon, 20 Oct 2014 10:33:27 -0400 Subject: [Bro] Bro and NetBIOS In-Reply-To: References: Message-ID: <592A8733-F255-4CEA-BBD5-94F4F58863F4@icir.org> On Oct 20, 2014, at 6:50 AM, Vito Logrillo wrote: > i'm trying to use BRO to analyze data based on NetBIOS protocol: i'm using BRO 2.3.1. SMB support in 2.3.1 (and earlier) is broken and wildly incomplete. > Another question: SMB2.0 protocol is supported by BRO or not? It should be moderately supported in 2.4. Broala has contributed an SMB (1+2) analyzer and it should be in 2.4. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From vitologrillo at gmail.com Mon Oct 20 07:44:16 2014 From: vitologrillo at gmail.com (Vito Logrillo) Date: Mon, 20 Oct 2014 16:44:16 +0200 Subject: [Bro] Bro and NetBIOS In-Reply-To: <592A8733-F255-4CEA-BBD5-94F4F58863F4@icir.org> References: <592A8733-F255-4CEA-BBD5-94F4F58863F4@icir.org> Message-ID: Hi Seth, thanks for your support: about NetBIOS, do you have any suggestion? What's wrong? Vito 2014-10-20 16:33 GMT+02:00 Seth Hall : > > On Oct 20, 2014, at 6:50 AM, Vito Logrillo wrote: > > > i'm trying to use BRO to analyze data based on NetBIOS protocol: i'm > using BRO 2.3.1. > > SMB support in 2.3.1 (and earlier) is broken and wildly incomplete. > > > Another question: SMB2.0 protocol is supported by BRO or not? > > It should be moderately supported in 2.4. Broala has contributed an SMB > (1+2) analyzer and it should be in 2.4. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141020/dd62b353/attachment.html From seth at icir.org Mon Oct 20 08:10:45 2014 From: seth at icir.org (Seth Hall) Date: Mon, 20 Oct 2014 11:10:45 -0400 Subject: [Bro] Bro and NetBIOS In-Reply-To: References: <592A8733-F255-4CEA-BBD5-94F4F58863F4@icir.org> Message-ID: <5A77C5BF-5BAD-4672-9406-3FBA8EFDA686@icir.org> On Oct 20, 2014, at 10:44 AM, Vito Logrillo wrote: > thanks for your support: about NetBIOS, do you have any suggestion? What's wrong?  I don't know about the problem you're encountering, but I believe that even if you got the analyzer attached it wouldn't do much for you. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From vitologrillo at gmail.com Mon Oct 20 08:25:10 2014 From: vitologrillo at gmail.com (Vito Logrillo) Date: Mon, 20 Oct 2014 17:25:10 +0200 Subject: [Bro] Bro and NetBIOS In-Reply-To: <5A77C5BF-5BAD-4672-9406-3FBA8EFDA686@icir.org> References: <592A8733-F255-4CEA-BBD5-94F4F58863F4@icir.org> <5A77C5BF-5BAD-4672-9406-3FBA8EFDA686@icir.org> Message-ID: Sorry Seth, but i don't understand your answer: what you mean with "... I believe that even if you got the analyzer attached it wouldn't do much for you"? Because i want to analyze and decode all NetBIOS traffic, with the help of google and your useful mailing list i've wrote a test script like this: ....snippet... const NetBIOSports = { 138/udp, 139/tcp, 445/tcp}; event bro_init() &priority=5 { Analyzer::register_for_ports(Analyzer::ANALYZER_NETBIOSSSN,NetBIOSports); } event netbios_session_message(c: connection, is_orig: bool, msg_type:count, data_len: count) &priority=5 { print "netbios_session_message"; } ................ But Bro gives me this error: Internal error: unknown analyzer name NETBIOS; mismatch with tag analyzer::Component? I'm using Bro in the wrong way? 2014-10-20 17:10 GMT+02:00 Seth Hall : > > On Oct 20, 2014, at 10:44 AM, Vito Logrillo > wrote: > > > thanks for your support: about NetBIOS, do you have any suggestion? > What's wrong? > > I don't know about the problem you're encountering, but I believe that > even if you got the analyzer attached it wouldn't do much for you. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141020/278110fc/attachment.html From dnthayer at illinois.edu Mon Oct 20 09:01:50 2014 From: dnthayer at illinois.edu (Daniel Thayer) Date: Mon, 20 Oct 2014 11:01:50 -0500 Subject: [Bro] Limiting the number of scripts and log files bro uses ? In-Reply-To: References: <2C24D46A-7678-440A-B0B1-D7C191043ADF@icir.org> <1a7053c4d72f4dd28d65cf2b3df98a41@BLUPR04MB817.namprd04.prod.outlook.com> Message-ID: <544531EE.2060706@illinois.edu> On 10/20/2014 09:16 AM, Seth Hall wrote: > > On Oct 20, 2014, at 10:00 AM, Dan Villanti wrote: > >>> If you use this option in a cluster setup, be sure to load core frameworks such as cluster, notice, control, etc. in local.bro or you may see some strange things. > > Hm, I'm surprised those things aren't automatically loaded. BroControl should be loading those even if you are running in bare mode. > > .Seth > BroControl loads those things automatically (even when BroArgs=-b). This can be confirmed by running "broctl scripts" (the output shows all scripts loaded by Bro). From inetjunkmail at gmail.com Mon Oct 20 09:58:26 2014 From: inetjunkmail at gmail.com (inetjunkmail) Date: Mon, 20 Oct 2014 12:58:26 -0400 Subject: [Bro] arista & cpacket experience In-Reply-To: <007501cfec50$fa1e6af0$ee5b40d0$@imdea.org> References: <007501cfec50$fa1e6af0$ee5b40d0$@imdea.org> Message-ID: We are using Arista 7150S's but are in the process up upgrading to 7280SE's. The 7280SE's are their next-gen platform and a much needed feature request that we have (MPLS label popping) is roadmap for the 7280SE They will be evaluating it for there 7150S but no commitment yet). cPacket already has MPLS label popping but is considerably more expensive. We're happy enough with the Arista's to stay that route. On Mon, Oct 20, 2014 at 6:31 AM, Juan Caballero wrote: > Hi everyone, > We would like to deploy a Bro Cluster at a 10 Gbps at about 35% peak usage. > We already have a splitter in place and are discussing options for a > front-end that can merge both traffic directions and load balance sessions > to Bro workers based on session hash and MAC rewriting. Ideally we would > like some equipment that supports multi-port mirroring so that we can add > other monitoring tools in addition to the Bro Cluster (e.g., Snort, > TimeMachine or other Storage). > Robin mentioned to me that people are using Arista and CPacket switches for > this kind of setup. After looking at their webpages the Arista 7150 seems > like a possibility for us (I see on the web page the San Diego SDSC and > Cornell use the larger 7500 series) and CPacket's cVu240NG may be another > (although there is less information about CPacket products online). > > Does anyone have experience with these products? Do those models make sense > for the description above? > Any recommendations or things to consider for people without prior > experience in such setups? > > Thanks! > > Juan Caballero > Assistant Research Professor > IMDEA Software Institute > Madrid, Spain > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141020/c893766c/attachment.html From phyltr at flavourcountry.org Mon Oct 20 11:22:08 2014 From: phyltr at flavourcountry.org (Phyltr) Date: Mon, 20 Oct 2014 12:22:08 -0600 Subject: [Bro] Fault tolerance In-Reply-To: <7B29334D-CC8D-46A6-B076-00653340E8F7@icir.org> References: <7B29334D-CC8D-46A6-B076-00653340E8F7@icir.org> Message-ID: That's ok, but what happens if the first data center becomes a crater in the ground? The workers will still be trying to send to the non-existent manager. Even if you have shared SSH keys from the secondary server to push out a new config that makes it the manager, its still a manual process and if something happens to make a data center a crater, manually switching Bro would likely be low on the list of priorities. Ideally I'd like to see each worker send to two servers in tandem. On Mon, Oct 20, 2014 at 8:29 AM, Seth Hall wrote: > > On Oct 16, 2014, at 9:31 AM, Karl Hill wrote: > >> Hello, I'm in a situation where I need to have some fault-tolerance >> with Bro. I need Bro data in two geographically disparate sites. Since >> I can't setup two managers I'm not sure how to go about accomplishing >> this. > > It doesn't work to just copy the data to your secondary data center hourly? Maybe you could explain more about what sort of fault tolerance your looking for? > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > From Xxx.Xxx at Removed.com Mon Oct 20 11:44:39 2014 From: Xxx.Xxx at Removed.com (Xxx, Xxx) Date: Mon, 20 Oct 2014 18:44:39 +0000 Subject: [Bro] Error: value used but not set (bloomfilter_basic_init) Message-ID: All, This is my first time using the mailing list so please let me know if I need to follow a certain format when asking questions. We are trying to set up a Bro 2.3 development server but whenever we run 'broctl check' our SMTP script throws the following error: Value used but not set (bloomfilter_basic_init) This is the offending line: global mail_links = bloomfilter_basic_init(0.00000001, 10000000); We got the script from GitHub. Looks nearly identical to this: https://github.com/initconf/smtp-analysis/blob/master/smtp-embedded-url-bloom.bro It's worth noting that the script runs with no issues on a previous version of Bro. Any help would be appreciated!! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141020/9d1c47bc/attachment.html From raj at bivio.net Mon Oct 20 12:19:44 2014 From: raj at bivio.net (Raj Srinivasan) Date: Mon, 20 Oct 2014 19:19:44 +0000 Subject: [Bro] Bro threads usage question Message-ID: <485e5a0e40194ca6990dbba86e6131a9@CO1PR01MB270.prod.exchangelabs.com> This is a question regarding bro's use of threads in Linux, for packet processing using the PCAP interface. Looking at the source code, it looks like a worker will receive a packet using the pcap_next() call, process the packet in the same thread, and invoke pcap_next() again to receive the next packet after the thread is done processing the current packet. I just want to confirm that this is indeed the case. It is important in our environment for a packet to be processed fully by a thread before it retrieves the next packet from PCAP. I would also like to know under what circumstances a bro worker will create additional threads to process packets or for other functions, if indeed this happens... from my reading of the code, this does not seem to happen, but not being familiar with the code, I am sure I missed something! I have searched the archives and did not find an answer. Any information, or pointers to information will be highly appreciated. Thanks! Raj -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141020/0db2899e/attachment.html From seth at icir.org Mon Oct 20 12:36:29 2014 From: seth at icir.org (Seth Hall) Date: Mon, 20 Oct 2014 15:36:29 -0400 Subject: [Bro] Bro threads usage question In-Reply-To: <485e5a0e40194ca6990dbba86e6131a9@CO1PR01MB270.prod.exchangelabs.com> References: <485e5a0e40194ca6990dbba86e6131a9@CO1PR01MB270.prod.exchangelabs.com> Message-ID: On Oct 20, 2014, at 3:19 PM, Raj Srinivasan wrote: > I would also like to know under what circumstances a bro worker will create additional threads to process packets or for other functions, if indeed this happens? from my reading of the code, this does not seem to happen, but not being familiar with the code, I am sure I missed something! You are correct, Bro is single threaded in terms of packet processing. Threads are currently only used for support tasks right now (logging, input). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From asharma at lbl.gov Mon Oct 20 12:44:13 2014 From: asharma at lbl.gov (Aashish Sharma) Date: Mon, 20 Oct 2014 12:44:13 -0700 Subject: [Bro] Error: value used but not set (bloomfilter_basic_init) In-Reply-To: References: Message-ID: <20141020194411.GA29977@yaksha.lbl.gov> I just tested this script with latest bro-2.3.1 and script seems to be running fine. I think there is some other issue which is causing bro to incorrectly point to mail_links as error location. I might be wrong but "value used but not set" shouldn't cause bro to fail. Could you do the following: broctl install; broctl start ; (and if bro fails ) paste me the error or output of: broctl diag Aashish On Mon, Oct 20, 2014 at 06:44:39PM +0000, Xxx, Xxx wrote: > > All, > > > This is my first time using the mailing list so please let me know if I need > to follow a certain format when asking questions. > > > We are trying to set up a Bro 2.3 development server but whenever we run > ?broctl check? our SMTP script throws the following error: > > > Value used but not set (bloomfilter_basic_init) > > > This is the offending line: > > > global mail_links = bloomfilter_basic_init(0.00000001, 10000000); > > > We got the script from GitHub. Looks nearly identical to this: > > > https://github.com/initconf/smtp-analysis/blob/master/smtp-embedded-url-bloo > m.bro > > > It?s worth noting that the script runs with no issues on a previous version > of Bro. > > > > Any help would be appreciated!! > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Aashish Sharma (asharma at lbl.gov) Cyber Security, Lawrence Berkeley National Laboratory http://go.lbl.gov/pgp-aashish Office: (510)-495-2680 Cell: (510)-612-7971 From seth at icir.org Mon Oct 20 12:53:49 2014 From: seth at icir.org (Seth Hall) Date: Mon, 20 Oct 2014 15:53:49 -0400 Subject: [Bro] Error: value used but not set (bloomfilter_basic_init) In-Reply-To: References: Message-ID: <67BC5B28-8552-4535-A789-96796F0B2C08@icir.org> On Oct 20, 2014, at 2:44 PM, Xxx, Xxx wrote: > This is my first time using the mailing list so please let me know if I need to follow a certain format when asking questions. Hi. This is perfectly fine. :) > global mail_links = bloomfilter_basic_init(0.00000001, 10000000); > > We got the script from GitHub. Looks nearly identical to this: > https://github.com/initconf/smtp-analysis/blob/master/smtp-embedded-url-bloom.bro Yeah, I noticed that issue with that script as well. There is a thread from a while ago where I promised to get that script heavily updated and haven't finished it yet. If you only declare the type and don't initialize it there then do the call to bloomfilter_basic_init in a bro_init handler it should work. So, while I was in the middle of writing this email, I went ahead and finished the smtp-url.bro script. This is very similar to Aashish's smtp-embedded-url-bloom.bro script but has been very lightly tested. I'd appreciate feedback from anyone that decides to run it. :) I put the script into the "official" external Bro scripts repository: https://github.com/bro/bro-scripts/blob/master/smtp-url.bro .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Mon Oct 20 13:02:59 2014 From: seth at icir.org (Seth Hall) Date: Mon, 20 Oct 2014 16:02:59 -0400 Subject: [Bro] Fault tolerance In-Reply-To: References: <7B29334D-CC8D-46A6-B076-00653340E8F7@icir.org> Message-ID: <6BC3E6AD-8F89-42CB-A36F-D51C7C39BCAE@icir.org> On Oct 20, 2014, at 2:22 PM, Phyltr wrote: > That's ok, but what happens if the first data center becomes a crater > in the ground? The workers will still be trying to send to the > non-existent manager. Ahh, the first data center being where your manager is? > Ideally I'd like to see each worker send to two servers in tandem.  Sorry, that's not something that is possible right now. There is some ongoing work to make cluster deployments a bit more flexible, but it will be some time still until that's in place. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From vstoffer at lbl.gov Mon Oct 20 17:54:51 2014 From: vstoffer at lbl.gov (Vincent Stoffer) Date: Mon, 20 Oct 2014 17:54:51 -0700 Subject: [Bro] arista & cpacket experience In-Reply-To: <007501cfec50$fa1e6af0$ee5b40d0$@imdea.org> References: <007501cfec50$fa1e6af0$ee5b40d0$@imdea.org> Message-ID: Hi Juan, We use both the cPacket (cVue 240) and Arista (7150s) and both are quite capable of handling the traffic you suggest. In our older setups we use a custom cPacket device to do MAC re-writing from 10G input to 1G Bro worker nodes. As Mike mentioned, load-balancing traffic to workers on a multi-core box with specialized NIC driver is a more common and often more cost effective configuration these days. We're currently ramping up our 100G Bro cluster with a combination of Arista hardware and collection of Myricom 10G workers on FreeBSD. I would suggest that you use the device you choose to aggregate, filter and distribute your traffic to the different tools and then experiment with running a Bro cluster on a single box. I think with the traffic volumes you mention you should be able to monitor everything with a single 10G card and multiple worker threads. One thing not to forget is that you'll need 1 port for each direction of "input" traffic on these devices to monitor full duplex taps, so make sure you take that into account when counting ports. The cVue is a very nice piece of hardware with great flexibility, however, the cost is not comparable with the Arista. The Arista feature set is quite good and they have been receptive to our feature requests. We're also very excited to be using Arista's API which lets us do dynamic shunting based on feedback from Bro. If you have specific questions, let me know and I'd be happy to answer them. Thank you, Vince On Mon, Oct 20, 2014 at 3:31 AM, Juan Caballero wrote: > Hi everyone, > We would like to deploy a Bro Cluster at a 10 Gbps at about 35% peak usage. > We already have a splitter in place and are discussing options for a > front-end that can merge both traffic directions and load balance sessions > to Bro workers based on session hash and MAC rewriting. Ideally we would > like some equipment that supports multi-port mirroring so that we can add > other monitoring tools in addition to the Bro Cluster (e.g., Snort, > TimeMachine or other Storage). > Robin mentioned to me that people are using Arista and CPacket switches for > this kind of setup. After looking at their webpages the Arista 7150 seems > like a possibility for us (I see on the web page the San Diego SDSC and > Cornell use the larger 7500 series) and CPacket's cVu240NG may be another > (although there is less information about CPacket products online). > > Does anyone have experience with these products? Do those models make sense > for the description above? > Any recommendations or things to consider for people without prior > experience in such setups? > > Thanks! > > Juan Caballero > Assistant Research Professor > IMDEA Software Institute > Madrid, Spain > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -- Vincent Stoffer, Cyber Security Engineer Cyber Security, Information Technology Division Lawrence Berkeley National Laboratory (510) 486-4531 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141020/6bd68a8b/attachment.html From Xxx.Xxx at Removed.com Tue Oct 21 06:12:38 2014 From: Xxx.Xxx at Removed.com (Xxx, Xxx) Date: Tue, 21 Oct 2014 13:12:38 +0000 Subject: [Bro] Error: value used but not set (bloomfilter_basic_init) Message-ID: Thank you all for the prompt responses. We implemented Seth's recommendation (initializing in the bro_init handler) and the issue has been resolved. Seth, I will apply your smtp-url script on the dev server for testing. Thanks again, -----Original Message----- From: Seth Hall [mailto:seth at icir.org] Sent: Monday, October 20, 2014 3:54 PM To: Xxx, Xxx Cc: bro at bro.org Subject: EXT :Re: [Bro] Error: value used but not set (bloomfilter_basic_init) On Oct 20, 2014, at 2:44 PM, Xxx, Xxx wrote: > This is my first time using the mailing list so please let me know if I need to follow a certain format when asking questions. Hi. This is perfectly fine. :) > global mail_links = bloomfilter_basic_init(0.00000001, 10000000); > > We got the script from GitHub. Looks nearly identical to this: > https://github.com/initconf/smtp-analysis/blob/master/smtp-embedded-url-bloom.bro Yeah, I noticed that issue with that script as well. There is a thread from a while ago where I promised to get that script heavily updated and haven't finished it yet. If you only declare the type and don't initialize it there then do the call to bloomfilter_basic_init in a bro_init handler it should work. So, while I was in the middle of writing this email, I went ahead and finished the smtp-url.bro script. This is very similar to Aashish's smtp-embedded-url-bloom.bro script but has been very lightly tested. I'd appreciate feedback from anyone that decides to run it. :) I put the script into the "official" external Bro scripts repository: https://github.com/bro/bro-scripts/blob/master/smtp-url.bro .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdonnelly at dyn.com Tue Oct 21 06:49:18 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Tue, 21 Oct 2014 08:49:18 -0500 Subject: [Bro] Startup error : error in /usr/local/bro/share/bro/base/utils/files.bro, line 32 Message-ID: Hi, I am getting this startup error : - Any idea how to debug this ? [BroControl] > start starting bro ... warning: cannot send mail bro terminated immediately after starting; check output with "diag" [BroControl] > diag [bro] Bro 2.2-135 Linux 3.13.0-24-generic ==== No reporter.log ==== stderr.log error in /usr/local/bro/share/bro/base/utils/files.bro, line 32: unknown identifier unescape_URI, at or near "unescape_URI" ==== stdout.log unlimited unlimited unlimited ==== .cmdline -i eth0 -U .status -p broctl -p broctl-live -p standalone -p local -p bro local.bro broctl broctl/standalone broctl/auto -b -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141021/f549b172/attachment.html From juan.caballero at imdea.org Tue Oct 21 06:59:40 2014 From: juan.caballero at imdea.org (Juan Caballero) Date: Tue, 21 Oct 2014 15:59:40 +0200 Subject: [Bro] arista & cpacket experience In-Reply-To: References: <007501cfec50$fa1e6af0$ee5b40d0$@imdea.org> Message-ID: <006e01cfed37$425ab9e0$c7102da0$@imdea.org> Hi Vincent, Thanks a lot for your feedback. Indeed, we plan using a multi-core machine with one Bro worker per core ( plus 1-2 cores for other stuff) and distribute traffic to them either using an Endance card (already available), Myricom cards, or PF-RING. I wasn?t sure if our current machine would be enough that is why I was thinking to support multiple machines, but starting with a single machine sounds like a great idea. From the answers to my questions the Arista may be a cost-effective option for an initial deployment. Juan From: Vincent Stoffer [mailto:vstoffer at lbl.gov] Sent: Tuesday, October 21, 2014 2:55 AM To: Juan Caballero Cc: bro at bro-ids.org Subject: Re: [Bro] arista & cpacket experience Hi Juan, We use both the cPacket (cVue 240) and Arista (7150s) and both are quite capable of handling the traffic you suggest. In our older setups we use a custom cPacket device to do MAC re-writing from 10G input to 1G Bro worker nodes. As Mike mentioned, load-balancing traffic to workers on a multi-core box with specialized NIC driver is a more common and often more cost effective configuration these days. We're currently ramping up our 100G Bro cluster with a combination of Arista hardware and collection of Myricom 10G workers on FreeBSD. I would suggest that you use the device you choose to aggregate, filter and distribute your traffic to the different tools and then experiment with running a Bro cluster on a single box. I think with the traffic volumes you mention you should be able to monitor everything with a single 10G card and multiple worker threads. One thing not to forget is that you'll need 1 port for each direction of "input" traffic on these devices to monitor full duplex taps, so make sure you take that into account when counting ports. The cVue is a very nice piece of hardware with great flexibility, however, the cost is not comparable with the Arista. The Arista feature set is quite good and they have been receptive to our feature requests. We're also very excited to be using Arista's API which lets us do dynamic shunting based on feedback from Bro. If you have specific questions, let me know and I'd be happy to answer them. Thank you, Vince On Mon, Oct 20, 2014 at 3:31 AM, Juan Caballero wrote: Hi everyone, We would like to deploy a Bro Cluster at a 10 Gbps at about 35% peak usage. We already have a splitter in place and are discussing options for a front-end that can merge both traffic directions and load balance sessions to Bro workers based on session hash and MAC rewriting. Ideally we would like some equipment that supports multi-port mirroring so that we can add other monitoring tools in addition to the Bro Cluster (e.g., Snort, TimeMachine or other Storage). Robin mentioned to me that people are using Arista and CPacket switches for this kind of setup. After looking at their webpages the Arista 7150 seems like a possibility for us (I see on the web page the San Diego SDSC and Cornell use the larger 7500 series) and CPacket's cVu240NG may be another (although there is less information about CPacket products online). Does anyone have experience with these products? Do those models make sense for the description above? Any recommendations or things to consider for people without prior experience in such setups? Thanks! Juan Caballero Assistant Research Professor IMDEA Software Institute Madrid, Spain _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -- Vincent Stoffer, Cyber Security Engineer Cyber Security, Information Technology Division Lawrence Berkeley National Laboratory (510) 486-4531 From juan.caballero at imdea.org Tue Oct 21 07:11:30 2014 From: juan.caballero at imdea.org (Juan Caballero) Date: Tue, 21 Oct 2014 16:11:30 +0200 Subject: [Bro] arista & cpacket experience In-Reply-To: <9B38ABCB-FF64-474C-B38A-1AE582EDE32D@uwaterloo.ca> References: <007501cfec50$fa1e6af0$ee5b40d0$@imdea.org> <9B38ABCB-FF64-474C-B38A-1AE582EDE32D@uwaterloo.ca> Message-ID: <007701cfed38$e9529a50$bbf7cef0$@imdea.org> > We don?t do load balancing on the switch though, we?re doing it on the NICs. It seems that load-balancing on the NICs is the preferred approach by different groups, so we will definitely give that a try. Thanks! We may try an Endace DAG that we have around, but would like to experiment with Intel x520+PF_RING as well > figure out a way to integrate monitoring it into your environment, or accept that you?ll get to monitor it yourself. Right, thanks for the advice. I am counting we will have to do some sys admin work ourselves at least at the start :) Thanks! Juan From jeff at jeffhammett.com Tue Oct 21 09:02:07 2014 From: jeff at jeffhammett.com (Jeff Hammett) Date: Tue, 21 Oct 2014 09:02:07 -0700 Subject: [Bro] Running Bro on MacBook with USB/Thunderbolt network interfaces References: <21870A69-342F-482A-AD59-C6CD8C53B779@jeffhammett.com> Message-ID: I am running Bro on my MacBook. I usually use wifi, but sometimes use either a USB or Thunderbolt network adapter and I?d like Bro to monitor whatever interface I am using, but am running into some problems. My wifi is en0 USB network adapter is en3 Thunderbolt Ethernet is en5 In node.cfg I have interface=en0 In broctl.cfg I have broargs = -i en3 -i en5 This seems to be working as long as the USB network adapter and the Thunderbolt network adapter are physically connected. But when one of them is not connected the interface does not exist and Bro will not start ? error: fatal error: /opt/bro/bin/bro: problem with interface en5 - pcap_open_live: en5: No such device exists (BIOCSETIF failed: Device not configured) when the Thunderbolt adapter is not connected. Is there a way to tell Bro to monitor an interface if it exists, but ignore it otherwise? Or is there a way to force my Mac to keep the en3/en5 devices available even when the USB/Thunderbolt network adapters are not attached? Or is there a better way to accomplish what I am trying to do? Jeff From seth at icir.org Tue Oct 21 10:15:03 2014 From: seth at icir.org (Seth Hall) Date: Tue, 21 Oct 2014 13:15:03 -0400 Subject: [Bro] Startup error : error in /usr/local/bro/share/bro/base/utils/files.bro, line 32 In-Reply-To: References: Message-ID: <976F64AA-9ED1-4675-A760-EC80EF56D2C4@icir.org> On Oct 21, 2014, at 9:49 AM, John Donnelly wrote: > I am getting this startup error : - Any idea how to debug this ? > error in /usr/local/bro/share/bro/base/utils/files.bro, line 32: unknown identifier unescape_URI, at or near "unescape_URI" Did you make any changes to the scripts that Bro scripts with? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdonnelly at dyn.com Wed Oct 22 05:58:24 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Wed, 22 Oct 2014 07:58:24 -0500 Subject: [Bro] Where are the log files when DNS monitoring ran by cli ? Message-ID: Hi, When I run : bro -i eth0 -i eth1 /usr/local/bro/share/bro/base/protocols/dns/main.bro listening on eth0, capture length 8192 bytes listening on eth1, capture length 8192 bytes 1413981834.692222 warning in /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54: Your interface is likely receiving invalid TCP and UDP checksums, most likely from NIC checksum offloading. ^C 1413982439.080452 received termination signal 1413982439.080452 5 packets received on interface eth0, 0 dropped 1413982439.080452 1977 packets received on interface eth1, 0 dropped I don't see any log file created under: /usr/local/bro/logs/current Suggestions ? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141022/bb093505/attachment.html From seth at icir.org Wed Oct 22 06:21:08 2014 From: seth at icir.org (Seth Hall) Date: Wed, 22 Oct 2014 09:21:08 -0400 Subject: [Bro] Where are the log files when DNS monitoring ran by cli ? In-Reply-To: References: Message-ID: <66609325-D57E-4FD2-BD9A-AE13113878E3@icir.org> On Oct 22, 2014, at 8:58 AM, John Donnelly wrote: > 1413981834.692222 warning in /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54: Your interface is likely receiving invalid TCP and UDP checksums, most likely from NIC checksum offloading. It looks like you have bad checksums probably from NIC checksum offloading. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdonnelly at dyn.com Wed Oct 22 06:25:32 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Wed, 22 Oct 2014 08:25:32 -0500 Subject: [Bro] Where are the log files when DNS monitoring ran by cli ? In-Reply-To: <66609325-D57E-4FD2-BD9A-AE13113878E3@icir.org> References: <66609325-D57E-4FD2-BD9A-AE13113878E3@icir.org> Message-ID: I have bro on Ubuntu 14.04 VM's running under VirtualBox on Linux . Should the dns.logs appear /usr/local/bro/logs if those errors were corrected ? The logs appear when I start bro using broctl . On Wed, Oct 22, 2014 at 8:21 AM, Seth Hall wrote: > > On Oct 22, 2014, at 8:58 AM, John Donnelly wrote: > > > 1413981834.692222 warning in > /usr/local/bro/share/bro/base/misc/find-checksum-offloading.bro, line 54: > Your interface is likely receiving invalid TCP and UDP checksums, most > likely from NIC checksum offloading. > > It looks like you have bad checksums probably from NIC checksum > offloading. :) > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141022/4e90062a/attachment.html From seth at icir.org Wed Oct 22 06:34:37 2014 From: seth at icir.org (Seth Hall) Date: Wed, 22 Oct 2014 09:34:37 -0400 Subject: [Bro] Where are the log files when DNS monitoring ran by cli ? In-Reply-To: References: <66609325-D57E-4FD2-BD9A-AE13113878E3@icir.org> Message-ID: <3F8DB155-0A99-4DD8-A109-CA5B95E336D0@icir.org> On Oct 22, 2014, at 9:25 AM, John Donnelly wrote: > The logs appear when I start bro using broctl . I get the sense you've made changes to broctl's configuration then. Did you add broars=-C to broctl.cfg perhaps? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdonnelly at dyn.com Wed Oct 22 06:42:31 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Wed, 22 Oct 2014 08:42:31 -0500 Subject: [Bro] Where are the log files when DNS monitoring ran by cli ? In-Reply-To: <3F8DB155-0A99-4DD8-A109-CA5B95E336D0@icir.org> References: <66609325-D57E-4FD2-BD9A-AE13113878E3@icir.org> <3F8DB155-0A99-4DD8-A109-CA5B95E336D0@icir.org> Message-ID: No changes made to broctl.cfg ! I am running bro outside of broctl .. are those setting read by bro during startup ? On Wed, Oct 22, 2014 at 8:34 AM, Seth Hall wrote: > > On Oct 22, 2014, at 9:25 AM, John Donnelly wrote: > > > The logs appear when I start bro using broctl . > > I get the sense you've made changes to broctl's configuration then. Did > you add broars=-C to broctl.cfg perhaps? > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141022/f90fbddf/attachment.html From seth at icir.org Wed Oct 22 07:26:06 2014 From: seth at icir.org (Seth Hall) Date: Wed, 22 Oct 2014 10:26:06 -0400 Subject: [Bro] Where are the log files when DNS monitoring ran by cli ? In-Reply-To: References: <66609325-D57E-4FD2-BD9A-AE13113878E3@icir.org> <3F8DB155-0A99-4DD8-A109-CA5B95E336D0@icir.org> Message-ID: On Oct 22, 2014, at 9:42 AM, John Donnelly wrote: > No changes made to broctl.cfg ! > > I am running bro outside of broctl .. are those setting read by bro during startup ? I'm confused. You first said that you weren't getting logs when you ran Bro outside of BroControl but then you said you were getting logs when you ran Bro with BroControl. If you run bro directly at the command line, it won't load any of the broctl scripts or implement any of the broctl configuration. You are almost certainly seeing invalid checksums on one of the interfaces you're sniffing. If you want to see if that's it, you could temporarily disable checksum checking with the -C flag on the command line. I don't recommend running with that configuration for normal use though.  It seemed like you were also confused about where logs would be written when running bro directly. They should be written to your current working directory by default. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdonnelly at dyn.com Wed Oct 22 07:28:54 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Wed, 22 Oct 2014 09:28:54 -0500 Subject: [Bro] Where are the log files when DNS monitoring ran by cli ? In-Reply-To: References: <66609325-D57E-4FD2-BD9A-AE13113878E3@icir.org> <3F8DB155-0A99-4DD8-A109-CA5B95E336D0@icir.org> Message-ID: Bingo - the dns.log is in the cwd ! Q answered ! On Wed, Oct 22, 2014 at 9:26 AM, Seth Hall wrote: > > On Oct 22, 2014, at 9:42 AM, John Donnelly wrote: > > > No changes made to broctl.cfg ! > > > > I am running bro outside of broctl .. are those setting read by bro > during startup ? > > I'm confused. You first said that you weren't getting logs when you ran > Bro outside of BroControl but then you said you were getting logs when you > ran Bro with BroControl. > > If you run bro directly at the command line, it won't load any of the > broctl scripts or implement any of the broctl configuration. You are > almost certainly seeing invalid checksums on one of the interfaces you're > sniffing. If you want to see if that's it, you could temporarily disable > checksum checking with the -C flag on the command line. I don't recommend > running with that configuration for normal use though. > > It seemed like you were also confused about where logs would be written > when running bro directly. They should be written to your current working > directory by default. :) > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141022/790cb365/attachment.html From anthony.kasza at gmail.com Wed Oct 22 07:30:01 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Wed, 22 Oct 2014 07:30:01 -0700 Subject: [Bro] Where are the log files when DNS monitoring ran by cli ? In-Reply-To: References: <66609325-D57E-4FD2-BD9A-AE13113878E3@icir.org> <3F8DB155-0A99-4DD8-A109-CA5B95E336D0@icir.org> Message-ID: Try running bro -C -i eth0 -i eth1 By default, all scripts in base/ are loaded when calling Bro. Log files are dropped in your working directory when running bro without broctl. Just a note, don't ever change scripts in base/. -AK On Oct 22, 2014 6:58 AM, "John Donnelly" wrote: > No changes made to broctl.cfg ! > > I am running bro outside of broctl .. are those setting read by bro during > startup ? > > On Wed, Oct 22, 2014 at 8:34 AM, Seth Hall wrote: > >> >> On Oct 22, 2014, at 9:25 AM, John Donnelly wrote: >> >> > The logs appear when I start bro using broctl . >> >> I get the sense you've made changes to broctl's configuration then. Did >> you add broars=-C to broctl.cfg perhaps? >> >> .Seth >> >> -- >> Seth Hall >> International Computer Science Institute >> (Bro) because everyone has a network >> http://www.bro.org/ >> >> > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141022/51b7e442/attachment.html From vitologrillo at gmail.com Thu Oct 23 05:16:06 2014 From: vitologrillo at gmail.com (Vito Logrillo) Date: Thu, 23 Oct 2014 14:16:06 +0200 Subject: [Bro] How filter machine name registration? Message-ID: Hi, i want to filter machine name registration on 137/udp port: as you know, all traffic is based on NetBios protocol. I've saw bro source code and seems that the traffic on this port is managed by dns analyzer and not directly by Netbios analyzer. How can i filter netbios name service registration? Regards, Vito -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141023/afb805e4/attachment.html From jdonnelly at dyn.com Thu Oct 23 06:11:23 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Thu, 23 Oct 2014 08:11:23 -0500 Subject: [Bro] How to turn off -O2 build flags ? Message-ID: Hi, How can I turn off the optimization settings when I build bro ? I would like to run gdb against a running proc but it gets confused due to optimizations . -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141023/a9bd244a/attachment.html From jdonnelly at dyn.com Thu Oct 23 06:22:24 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Thu, 23 Oct 2014 08:22:24 -0500 Subject: [Bro] How to turn off -O2 build flags ? In-Reply-To: References: Message-ID: looks like ./configure --enable-debug will do it ;-) On Thu, Oct 23, 2014 at 8:11 AM, John Donnelly wrote: > Hi, > How can I turn off the optimization settings when I build bro ? > > I would like to run gdb against a running proc but it gets confused due > to optimizations . > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141023/df81ce84/attachment.html From seth at icir.org Thu Oct 23 06:50:55 2014 From: seth at icir.org (Seth Hall) Date: Thu, 23 Oct 2014 09:50:55 -0400 Subject: [Bro] How to turn off -O2 build flags ? In-Reply-To: References: Message-ID: On Oct 23, 2014, at 9:11 AM, John Donnelly wrote: > \ How can I turn off the optimization settings when I build bro ? > > I would like to run gdb against a running proc but it gets confused due to optimizations . You could build with --enable-debug, but that turns on a lot of extra debugging and make Bro run quite a bit slower. It seems that if you give -O0 as a CXXFLAG when you run configure we're currently appending those values in the wrong order when the compiler runs so you can't override it that way at the moment. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Thu Oct 23 06:52:18 2014 From: seth at icir.org (Seth Hall) Date: Thu, 23 Oct 2014 09:52:18 -0400 Subject: [Bro] How filter machine name registration? In-Reply-To: References: Message-ID: <193EC0EC-FD65-4AA2-B170-AAEE573923A2@icir.org> On Oct 23, 2014, at 8:16 AM, Vito Logrillo wrote: > How can i filter netbios name service registration? It all shows up in dns.log and you are given access to it through the various DNS events. Could you describe what you are trying to accomplish? Providing a packet capture and describing what you want to get out of it would be the most useful. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jsiwek at illinois.edu Thu Oct 23 07:46:05 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 23 Oct 2014 14:46:05 +0000 Subject: [Bro] How to turn off -O2 build flags ? In-Reply-To: References: Message-ID: > On Oct 23, 2014, at 8:50 AM, Seth Hall wrote: > > You could build with --enable-debug, but that turns on a lot of extra debugging and make Bro run quite a bit slower. Just wondering if you?ve done any recent measurements? I can?t remember if [1] completely eliminated the overhead of ?enable-debug without any `bro -B` flags versus just -O0, but it did help a lot. - Jon [1] https://github.com/bro/bro/commit/302c063874f03c94eb0f96c974a2d0e2f137e51f From seth at icir.org Thu Oct 23 07:55:05 2014 From: seth at icir.org (Seth Hall) Date: Thu, 23 Oct 2014 10:55:05 -0400 Subject: [Bro] How to turn off -O2 build flags ? In-Reply-To: References: Message-ID: <16B45696-A638-4179-8A49-BE55480E2234@icir.org> On Oct 23, 2014, at 10:46 AM, Siwek, Jon wrote: > Just wondering if you?ve done any recent measurements? I can?t remember if [1] completely eliminated the overhead of ?enable-debug without any `bro -B` flags versus just -O0, but it did help a lot. Oh, I haven't. That would be great if the debug overhead was gone or nearly gone. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jamie.gausemel at gmail.com Fri Oct 24 08:46:16 2014 From: jamie.gausemel at gmail.com (Jamie Gausemel) Date: Fri, 24 Oct 2014 11:46:16 -0400 Subject: [Bro] Parsing HTTP Traffic Message-ID: Could someone point me in the right direction... I simply need to parse out usernames from HTTP packets that look like: HTTP/1.1 200 OK Server: nginx/1.4.2 Date: Wed, 22 Oct 2014 14:58:11 GMT Content-Type: application/json; charset=UTF-8 Content-Length: 104 Connection: keep-alive Set-Cookie: si=xxxxx; Max-Age=7199; Path=/; expires=Wed, 22-Oct-2014 16:58:11 GMT; HttpOnly {"username": "first.last", "name": "first last", "groups": ["group name goes here"]} -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141024/b993ac50/attachment.html From anthony.kasza at gmail.com Fri Oct 24 10:39:58 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Fri, 24 Oct 2014 10:39:58 -0700 Subject: [Bro] Parsing HTTP Traffic In-Reply-To: References: Message-ID: You'll have to reconstruct HTTP bodies and parse the json. There are a few scripts that do the body reconstruction floating around github. -AK On Oct 24, 2014 9:08 AM, "Jamie Gausemel" wrote: > Could someone point me in the right direction... I simply need to parse > out usernames from HTTP packets that look like: > > HTTP/1.1 200 OK > Server: nginx/1.4.2 > Date: Wed, 22 Oct 2014 14:58:11 GMT > Content-Type: application/json; charset=UTF-8 > Content-Length: 104 > Connection: keep-alive > Set-Cookie: si=xxxxx; Max-Age=7199; Path=/; expires=Wed, 22-Oct-2014 > 16:58:11 GMT; HttpOnly > > {"username": "first.last", "name": "first last", "groups": ["group name > goes here"]} > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141024/77493f35/attachment.html From nweaver at ICSI.Berkeley.EDU Fri Oct 24 10:46:30 2014 From: nweaver at ICSI.Berkeley.EDU (Nicholas Weaver) Date: Fri, 24 Oct 2014 10:46:30 -0700 Subject: [Bro] Parsing HTTP Traffic In-Reply-To: References: Message-ID: <28BEDD83-2A28-4A18-B51F-39B0B2F3BC36@icsi.berkeley.edu> > On Oct 24, 2014, at 10:39 AM, anthony kasza wrote: > > You'll have to reconstruct HTTP bodies and parse the json. There are a few scripts that do the body reconstruction floating around github. > > -AK The other option if things are always the same is to just use a couple of regular expressions to indicate where the data should be. -- Nicholas Weaver it is a tale, told by an idiot, nweaver at icsi.berkeley.edu full of sound and fury, 510-666-2903 .signifying nothing PGP: http://www1.icsi.berkeley.edu/~nweaver/data/nweaver_pub.asc From robin at icir.org Fri Oct 24 14:23:18 2014 From: robin at icir.org (Robin Sommer) Date: Fri, 24 Oct 2014 14:23:18 -0700 Subject: [Bro] bro.org outage on Tuesday Message-ID: <20141024212318.GS96754@icir.org> Services on bro.org will be inaccesible next Tuesday between 10am to noon PST for server room work. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From vitologrillo at gmail.com Mon Oct 27 01:55:43 2014 From: vitologrillo at gmail.com (Vito Logrillo) Date: Mon, 27 Oct 2014 09:55:43 +0100 Subject: [Bro] How filter machine name registration? In-Reply-To: <193EC0EC-FD65-4AA2-B170-AAEE573923A2@icir.org> References: <193EC0EC-FD65-4AA2-B170-AAEE573923A2@icir.org> Message-ID: Thanks for your reply, i'll try to explain my problem better. I'm trying to log all netbios service name registration: as you have suggested, i've filtered dns traffic on 137/udp port and used a filter for a specific opcode (Netbios_registration == 5). In this way, i'm able to log all netbios registrations, but i'm not able to discern a group name registration from an unique name registration. Using wireshark, i find this information in an additional record that i can't see in bro. For example, using this event event dns_request (c:connection, msg: dns_msg, query: string, qtype: count, qclass: count) { print (msg$num_addl); } I can see the presence of an additional record in the packet (msg$num_addl =1), but i can't see its value. How can i do in Bro? Thanks Vito 2014-10-23 15:52 GMT+02:00 Seth Hall : > > On Oct 23, 2014, at 8:16 AM, Vito Logrillo wrote: > > > How can i filter netbios name service registration? > > It all shows up in dns.log and you are given access to it through the > various DNS events. Could you describe what you are trying to accomplish? > Providing a packet capture and describing what you want to get out of it > would be the most useful. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141027/8b8248da/attachment.html From seth at icir.org Mon Oct 27 07:56:48 2014 From: seth at icir.org (Seth Hall) Date: Mon, 27 Oct 2014 10:56:48 -0400 Subject: [Bro] How filter machine name registration? In-Reply-To: References: <193EC0EC-FD65-4AA2-B170-AAEE573923A2@icir.org> Message-ID: On Oct 27, 2014, at 4:55 AM, Vito Logrillo wrote: > I can see the presence of an additional record in the packet (msg$num_addl =1), but i can't see its value. > How can i do in Bro? redef dns_skip_all_addl=F; Long ago there was a decision in the DNS analyzer to not process auth and addl records due to load issues. If you make the setting change that I recommended, you can get the extra DNS records. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdonnelly at dyn.com Mon Oct 27 08:22:50 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Mon, 27 Oct 2014 10:22:50 -0500 Subject: [Bro] Link error: magic_open Message-ID: Hi, Do I need another package other than libmagic-dev ? Linking CXX executable bro CMakeFiles/bro.dir/util.cc.o: In function `bro_init_magic(magic_set**, int)': /work/jpd/dyn/src/dynect2/edge_processing/bro/src/util.cc:1705: undefined reference to `magic_open' /work/jpd/dyn/src/dynect2/edge_processing/bro/src/util.cc:1712: undefined reference to `magic_error' /work/jpd/dyn/src/dynect2/edge_processing/bro/src/util.cc:1717: undefined reference to `magic_load' /work/jpd/dyn/src/dynect2/edge_processing/bro/src/util.cc:1719: undefined reference to `magic_error' /work/jpd/dyn/src/dynect2/edge_processing/bro/src/util.cc:1722: undefined reference to `magic_close' CMakeFiles/bro.dir/util.cc.o: In function `bro_magic_buffer(magic_set*, void const*, unsigned long)': /work/jpd/dyn/src/dynect2/edge_processing/bro/src/util.cc:1729: undefined reference to `magic_buffer' /work/jpd/dyn/src/dynect2/edge_processing/bro/src/util.cc:1732: undefined reference to `magic_error' -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141027/1aa86129/attachment.html From seth at icir.org Mon Oct 27 09:15:35 2014 From: seth at icir.org (Seth Hall) Date: Mon, 27 Oct 2014 12:15:35 -0400 Subject: [Bro] Link error: magic_open In-Reply-To: References: Message-ID: On Oct 27, 2014, at 11:22 AM, John Donnelly wrote: > Do I need another package other than libmagic-dev ? What version of Bro is this you're building? As of 2.3 we don't use libmagic anymore. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jdonnelly at dyn.com Mon Oct 27 11:03:57 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Mon, 27 Oct 2014 13:03:57 -0500 Subject: [Bro] Link error: magic_open In-Reply-To: References: Message-ID: I see where the libmagic version 5.16 was manually added to the build: https://github.com/bro/bro/commit/c48c5316299898d05533ee9f5b0273b575391a1f Then at a later point this was removed. Ubuntu 14.04 currently uses: libmagic-dev:amd64 1:5.14-2ubuntu3.2 Any insight appreciated . On Mon, Oct 27, 2014 at 11:15 AM, Seth Hall wrote: > > On Oct 27, 2014, at 11:22 AM, John Donnelly wrote: > > > Do I need another package other than libmagic-dev ? > > What version of Bro is this you're building? As of 2.3 we don't use > libmagic anymore. > > .Seth > > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141027/8bf99249/attachment.html From jdonnelly at dyn.com Mon Oct 27 11:56:37 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Mon, 27 Oct 2014 13:56:37 -0500 Subject: [Bro] Link error: magic_open In-Reply-To: References: Message-ID: I see libmagic was pulled out and new plug module was added: https://github.com/bro/bro/commit/bbd409d274fa36fe66f0ac3a6d43bc56d2e0d67f On Mon, Oct 27, 2014 at 1:03 PM, John Donnelly wrote: > I see where the libmagic version 5.16 was manually added to the build: > > https://github.com/bro/bro/commit/c48c5316299898d05533ee9f5b0273b575391a1f > > Then at a later point this was removed. > > Ubuntu 14.04 currently uses: libmagic-dev:amd64 > 1:5.14-2ubuntu3.2 > > Any insight appreciated . > > > > On Mon, Oct 27, 2014 at 11:15 AM, Seth Hall wrote: > >> >> On Oct 27, 2014, at 11:22 AM, John Donnelly wrote: >> >> > Do I need another package other than libmagic-dev ? >> >> What version of Bro is this you're building? As of 2.3 we don't use >> libmagic anymore. >> >> .Seth >> >> >> -- >> Seth Hall >> International Computer Science Institute >> (Bro) because everyone has a network >> http://www.bro.org/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141027/036b85e3/attachment.html From jsiwek at illinois.edu Mon Oct 27 14:38:43 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 27 Oct 2014 21:38:43 +0000 Subject: [Bro] Link error: magic_open In-Reply-To: References: Message-ID: <73AEB394-1D42-4A14-BE7E-E72229D0B018@illinois.edu> > On Oct 27, 2014, at 1:03 PM, John Donnelly wrote: > > I see where the libmagic version 5.16 was manually added to the build: > > https://github.com/bro/bro/commit/c48c5316299898d05533ee9f5b0273b575391a1f > > Then at a later point this was removed. Yes, IIRC, this method of integrating libmagic never made it in to a release version of Bro, so it?s probably not something you need to worry about. > Ubuntu 14.04 currently uses: libmagic-dev:amd64 1:5.14-2ubuntu3.2 If you are using a version of Bro older than 2.3, that libmagic-dev should work fine ? at least I don?t have a problem with it when compiling Bro 2.2. As mentioned, Bro 2.3 and later won?t require libmagic, so if you can use that instead, it would sidestep whatever your issue is. Else, we?ll need more details: best if you can give step-by-step instructions to reproduce your problem, including which Bro version and any non-default configuration flags used. - Jon From hckim at narusec.com Mon Oct 27 17:11:20 2014 From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=) Date: Tue, 28 Oct 2014 09:11:20 +0900 Subject: [Bro] (no subject) Message-ID: Hi every one I like to know is there any way Bro could decrypt SSL session If I have a private open keys from each side, Can I use this information for decrypt SSL session and log it? If bro can do it could you point me in the right direction. I try to search bro.org and bro mailing list but cat't find any answer. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141028/d01c2bfb/attachment.html From johanna at icir.org Mon Oct 27 18:24:07 2014 From: johanna at icir.org (Johanna Amann) Date: Mon, 27 Oct 2014 18:24:07 -0700 Subject: [Bro] (no subject) In-Reply-To: References: Message-ID: <20141028012407.GA20693@LadyMacbeth.local> Hello, Sorry, Bro currently does not support decrypting SSL/TLS sessions, even if you are in posession of the private or session keys. Johanna On Tue, Oct 28, 2014 at 09:11:20AM +0900, ??? wrote: > Hi every one > I like to know is there any way Bro could decrypt SSL session > If I have a private open keys from each side, Can I use this information > for decrypt SSL session and log it? > If bro can do it could you point me in the right direction. > > > I try to search bro.org and bro mailing list but cat't find any answer. > > Thank you > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From seth at icir.org Mon Oct 27 18:30:45 2014 From: seth at icir.org (Seth Hall) Date: Mon, 27 Oct 2014 21:30:45 -0400 Subject: [Bro] (no subject) In-Reply-To: References: Message-ID: <8DCCBD1D-5CF8-445A-8EDA-6206DEE6F455@icir.org> On Oct 27, 2014, at 8:11 PM, ??? wrote: > I like to know is there any way Bro could decrypt SSL session > If I have a private open keys from each side, Can I use this information for decrypt SSL session and log it? > If bro can do it could you point me in the right direction. Nope, Bro has no mechanism to decrypt SSL/TLS right now. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From justin at justinthomas.name Mon Oct 27 18:43:45 2014 From: justin at justinthomas.name (Justin Thomas) Date: Mon, 27 Oct 2014 15:43:45 -1000 Subject: [Bro] (no subject) In-Reply-To: References: Message-ID: You might try https://github.com/plashchynski/viewssld . On Oct 27, 2014 5:16 PM, "???" wrote: > Hi every one > I like to know is there any way Bro could decrypt SSL session > If I have a private open keys from each side, Can I use this information > for decrypt SSL session and log it? > If bro can do it could you point me in the right direction. > > > I try to search bro.org and bro mailing list but cat't find any answer. > > Thank you > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141027/fbe2fd07/attachment.html From Robert_Yang at trendmicro.com.cn Mon Oct 27 23:55:53 2014 From: Robert_Yang at trendmicro.com.cn (Robert_Yang at trendmicro.com.cn) Date: Tue, 28 Oct 2014 06:55:53 +0000 Subject: [Bro] Is it is a code issue about bro-2.3? Message-ID: <6FCE7872AA66C246990EC5623F91A014C6CEAD9C@CDCEXMBX03.tw.trendnet.org> Dear, BroFile::Write() in src/File.cc, len = fwrite(data, 1, len, f); if ( len <=0 ) return false; Maybe, the method to check the return value of fwrite is incorrect. We should call ferror to check file operation's result. The following is fixing. len = fwrite(data, 1, len, f); if ( ferror(f) ) { clearerr(f); return false; } Would you please verify this question? Thanks! 
TREND MICRO EMAIL NOTICE
The information contained in this email and any attachments is confidential 
and may be subject to copyright or other intellectual property protection. 
If you are not the intended recipient, you are not authorized to use or 
disclose this information, and we request that you notify us by reply mail or
telephone and delete the original message from your mail system.
-------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141028/3300cbc7/attachment.html From vitologrillo at gmail.com Tue Oct 28 00:59:47 2014 From: vitologrillo at gmail.com (Vito Logrillo) Date: Tue, 28 Oct 2014 08:59:47 +0100 Subject: [Bro] How filter machine name registration? In-Reply-To: References: <193EC0EC-FD65-4AA2-B170-AAEE573923A2@icir.org> Message-ID: I've tried your solution without any result. Below you can see the bro script that i've used -----Script.bro---- module Scriptlog; redef dns_skip_all_addl=F; export { redef enum Log::ID += { LOG }; type Info: record { ts: time &log; orig_h: addr &log; orig_p: port &log; resp_h: addr &log; resp_p: port &log; addl: set [string] &log; }; global Scriptlog_Log: event(rec: Info); } event bro_init() &priority=5 { Log::create_stream(Scriptlog::LOG, [$columns = Info, $ev = Scriptlog_Log]); } event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count) &priority=5 { if(msg$num_addl != 0) { local myinfo: Info; myinfo$ts = network_time(); myinfo$orig_h = c$id$orig_h; myinfo$orig_p = c$id$orig_p; myinfo$resp_h = c$id$resp_h; myinfo$resp_p = c$id$resp_p; myinfo$addl = c$dns$addl; Log::write(Scriptlog::LOG, myinfo); } } --------------- In reporter.log i have this error Reporter::ERROR field value missing [Scriptlog::c$dns$addl] What can i do? Thanks, Vito 2014-10-27 15:56 GMT+01:00 Seth Hall : > > On Oct 27, 2014, at 4:55 AM, Vito Logrillo wrote: > > > I can see the presence of an additional record in the packet > (msg$num_addl =1), but i can't see its value. > > How can i do in Bro? > > redef dns_skip_all_addl=F; > > Long ago there was a decision in the DNS analyzer to not process auth and > addl records due to load issues. If you make the setting change that I > recommended, you can get the extra DNS records. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141028/501e10dc/attachment.html From seth at icir.org Tue Oct 28 05:13:26 2014 From: seth at icir.org (Seth Hall) Date: Tue, 28 Oct 2014 08:13:26 -0400 Subject: [Bro] How filter machine name registration? In-Reply-To: References: <193EC0EC-FD65-4AA2-B170-AAEE573923A2@icir.org> Message-ID: <51092425-775A-45AA-88A0-7011B860EA1D@icir.org> On Oct 28, 2014, at 3:59 AM, Vito Logrillo wrote: > I've tried your solution without any result. > Below you can see the bro script that i've used Oh, sorry. That was only part of the solution. Those records attached to the connection record are filled out by scripts, but we don't have scripts that deal with additional RRs. You will have to handle the appropriate events and write your own script to do something with the data. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From vitologrillo at gmail.com Tue Oct 28 05:21:43 2014 From: vitologrillo at gmail.com (Vito Logrillo) Date: Tue, 28 Oct 2014 13:21:43 +0100 Subject: [Bro] How filter machine name registration? In-Reply-To: <51092425-775A-45AA-88A0-7011B860EA1D@icir.org> References: <193EC0EC-FD65-4AA2-B170-AAEE573923A2@icir.org> <51092425-775A-45AA-88A0-7011B860EA1D@icir.org> Message-ID: Ok, thanks for your reply. Without any change on source code, what event you suggest to use to handle these data? Also an event able to give me additional RRs as row data could be fine. Vito 2014-10-28 13:13 GMT+01:00 Seth Hall : > > On Oct 28, 2014, at 3:59 AM, Vito Logrillo wrote: > > > I've tried your solution without any result. > > Below you can see the bro script that i've used > > Oh, sorry. That was only part of the solution. Those records attached to > the connection record are filled out by scripts, but we don't have scripts > that deal with additional RRs. You will have to handle the appropriate > events and write your own script to do something with the data. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141028/08542488/attachment.html From seth at icir.org Tue Oct 28 05:52:40 2014 From: seth at icir.org (Seth Hall) Date: Tue, 28 Oct 2014 08:52:40 -0400 Subject: [Bro] How filter machine name registration? In-Reply-To: References: <193EC0EC-FD65-4AA2-B170-AAEE573923A2@icir.org> <51092425-775A-45AA-88A0-7011B860EA1D@icir.org> Message-ID: <6A814F54-114D-4EC3-935B-2165DC53DBFF@icir.org> On Oct 28, 2014, at 8:21 AM, Vito Logrillo wrote: > Without any change on source code, what event you suggest to use to handle these data? > Also an event able to give me additional RRs as row data could be fine. It depends on the RR type. You can look at the different events for the different RRs here: https://www.bro.org/sphinx/scripts/base/bif/plugins/Bro_DNS.events.bif.bro.html I also forgot that there is a script that may add what you are looking for. @load policy/protocols/dns/auth-addl You may want to take a look at how that script works to see if it's doing what you want. (also, the DNS::do_reply hook is defined in the DNS scripts and not in the core analyzer) -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From johanna at icir.org Tue Oct 28 07:07:02 2014 From: johanna at icir.org (Johanna Amann) Date: Tue, 28 Oct 2014 07:07:02 -0700 Subject: [Bro] Is it is a code issue about bro-2.3? In-Reply-To: <6FCE7872AA66C246990EC5623F91A014C6CEAD9C@CDCEXMBX03.tw.trendnet.org> References: <6FCE7872AA66C246990EC5623F91A014C6CEAD9C@CDCEXMBX03.tw.trendnet.org> Message-ID: <20141028140702.GA23116@LadyMacbeth.local> Hello Robert, according to the fwrite manual pages: "If an error occurs, or the end-of-file is reached, the return value is a short object count (or zero)." According to this, checking ferror is unnecessary for fwrites. However you are right that just checking that the len is 0 is not enough, instead we should check that the number of bytes written is equal to the number of bytes we wanted to write, i.e. if ( fwrite(data, 1, len, f) < len ) return false; There are a few more fwrite calls in the file before that, I will give it a quick pass and submit a patch later today. Johanna On Tue, Oct 28, 2014 at 06:55:53AM +0000, Robert_Yang at trendmicro.com.cn wrote: > Dear, > > BroFile::Write() in src/File.cc, > > len = fwrite(data, 1, len, f); > if ( len <=0 ) > return false; > > Maybe, the method to check the return value of fwrite is incorrect. We should call ferror to check file operation's result. The following is fixing. > > len = fwrite(data, 1, len, f); > if ( ferror(f) ) { > clearerr(f); > return false; > } > > Would you please verify this question? > > Thanks! > > > 
> TREND MICRO EMAIL NOTICE
> The information contained in this email and any attachments is confidential
> and may be subject to copyright or other intellectual property protection.
> If you are not the intended recipient, you are not authorized to use or
> disclose this information, and we request that you notify us by reply mail or
> telephone and delete the original message from your mail system.
>
> _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From jdonnelly at dyn.com Tue Oct 28 12:53:08 2014 From: jdonnelly at dyn.com (John Donnelly) Date: Tue, 28 Oct 2014 14:53:08 -0500 Subject: [Bro] Adding addition args to CFLAGS and CXXFLAGS Message-ID: Hi, This there a way I can append additional -D args to the CFLAGS and CXXFLAGS when I first ./configure bro to build ? There is a comment in ./configure --help I don't understand: Influential Environment Variables (only on first invocation per build directory): CC C compiler command CFLAGS C compiler flags CXX C++ compiler command CXXFLAGS C++ compiler flags -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141028/a6800317/attachment.html From jsiwek at illinois.edu Tue Oct 28 13:08:03 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 28 Oct 2014 20:08:03 +0000 Subject: [Bro] Adding addition args to CFLAGS and CXXFLAGS In-Reply-To: References: Message-ID: > On Oct 28, 2014, at 2:53 PM, John Donnelly wrote: > > This there a way I can append additional -D args to the > CFLAGS and CXXFLAGS when I first ./configure bro to build ? As an example, this should prepend to the default set of flags: CXXFLAGS="-DMYDEFINITION" CFLAGS="-DMYDEFINITION" ./configure - Jon From anthony.kasza at gmail.com Tue Oct 28 18:10:46 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Tue, 28 Oct 2014 18:10:46 -0700 Subject: [Bro] Attributes and Ports Questions Message-ID: Hi All, Is anyone using the following attributes? How are you using them? I believe some of these have been deprecated. &rotate_interval &rotate_size &mergeable &synchronize (I think there was a post earlier last month about this one) &persistent &group &add_func &delete_func &encrypt (applying this to a file causes Bro to "elegantly terminate" for me) bro -Ci eth0 -e 'global f1: file = open("f.out") &encrypt' What is the purpose of the unknown port type? And why do they only range from 0-255? Compare the results of the following commands. bro -e 'print 0/unknown; print 255/unknown;' bro -e 'print 0/unknown; print 255/unknown; print 256/unknown;' How are ports flattened? See the results of the following command. bro -e 'print 0/udp; print |0/udp|; print |32/tcp|; print |11/tcp|; print |132/unknown|; print 132/unknown;' Any insights are appreciated. -AK From christian.kollee at fkie.fraunhofer.de Wed Oct 29 07:48:09 2014 From: christian.kollee at fkie.fraunhofer.de (Christian Kollee) Date: Wed, 29 Oct 2014 15:48:09 +0100 Subject: [Bro] Using the sqlite logger in cluster mode Message-ID: <5450FE29.2010505@fkie.fraunhofer.de> Hi, we try to use the sqlite logger with a simple cluster configuration (SecurityOnion with manager, proxy and one worker on the same machine). We added a module to $PREFIX/share/bro containing just the example script from bro.org [1]. After restarting bro (using broctl restart --clean) the manager will crash on the next connection. However if we start bro using standalone mode the script works as intended. The database file is created and the connections are added. Switching back to cluster mode everything works now. Removing the database file and create an empty one using the schema extracted previously will also crash the manager in cluster mode. We are a little bit puzzeled what went wrong here and how to get the sqlite logger working in cluster mode. Did we miss something or is this a bug (or a feature)? Best regards Christian [1] https://www.bro.org/sphinx-git/frameworks/logging-input-sqlite.html From jsiwek at illinois.edu Wed Oct 29 12:35:23 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 29 Oct 2014 19:35:23 +0000 Subject: [Bro] Attributes and Ports Questions In-Reply-To: References: Message-ID: > On Oct 28, 2014, at 8:10 PM, anthony kasza wrote: > > What is the purpose of the unknown port type? It should mostly be used internally to signify an uninitialized/invalid transport protocol. I don?t think it?s common for that to actually be exposed to the scripting-layer for practical usage. > And why do they only > range from 0-255? Compare the results of the following commands. > bro -e 'print 0/unknown; print 255/unknown;' > bro -e 'print 0/unknown; print 255/unknown; print 256/unknown;? Likely arbitrary and just due to copy-paste of the code that parses ICMP port literals (for ICMP, Bro uses 0-255 to correspond to the Type/Code values). > How are ports flattened? See the results of the following command. > bro -e 'print 0/udp; print |0/udp|; print |32/tcp|; print |11/tcp|; > print |132/unknown|; print 132/unknown;' Internally, a port is a single uint64 with some of the high-bits set to indicate which port-space it belongs to. You?re seeing that value here. E.g. $ bro -e 'print |32/tcp| == 0x10000 + 32' T $ bro -e 'print |37/udp| == 0x20000 + 37? T - Jon From robin at icir.org Thu Oct 30 07:53:58 2014 From: robin at icir.org (Robin Sommer) Date: Thu, 30 Oct 2014 07:53:58 -0700 Subject: [Bro] Attributes and Ports Questions In-Reply-To: References: Message-ID: <20141030145358.GA22582@icir.org> Hi Anthony, have you seen this page? https://www.bro.org/sphinx-git/script-reference/attributes.html It's pretty new (though maybe it's actually where your questiosns are coming from :) To add a bit to that: On Tue, Oct 28, 2014 at 18:10 -0700, anthony kasza wrote: > &rotate_interval > &rotate_size This used to be primary log rotation mechanism before we switched to the new logging system/format. I've been wondering if we should just remove these attributes. > &mergeable > &synchronize (I think there was a post earlier last month about this one) > &persistent These are going to go away, but we aren't there yet. We may start deprecating them with the next release, which is scheduled to ship with a first version of their replacement, the new Broker library. > &group A bit of an obscure feature, originally added to toggle selected sets of analysis dynamically from BroControl. Don't think that's used anywhere and I'm inclined to remove it. > &add_func > &delete_func These aren't used very often, but can be useful in individual cases. > &encrypt (applying this to a file causes Bro to "elegantly terminate" for me) > bro -Ci eth0 -e 'global f1: file = open("f.out") &encrypt' Another relict from old-style logging, although the new framework doesn't have any equivalent functionality yet. Mind filing a ticket for the crash? We should either fix it or remove the attribute. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From anthony.kasza at gmail.com Thu Oct 30 08:40:02 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Thu, 30 Oct 2014 08:40:02 -0700 Subject: [Bro] Attributes and Ports Questions In-Reply-To: References: Message-ID: Thanks for the explanations, Jon. -AK On Oct 29, 2014 12:35 PM, "Siwek, Jon" wrote: > > > On Oct 28, 2014, at 8:10 PM, anthony kasza > wrote: > > > > What is the purpose of the unknown port type? > > It should mostly be used internally to signify an uninitialized/invalid > transport protocol. I don?t think it?s common for that to actually be > exposed to the scripting-layer for practical usage. > > > And why do they only > > range from 0-255? Compare the results of the following commands. > > bro -e 'print 0/unknown; print 255/unknown;' > > bro -e 'print 0/unknown; print 255/unknown; print 256/unknown;? > > Likely arbitrary and just due to copy-paste of the code that parses ICMP > port literals (for ICMP, Bro uses 0-255 to correspond to the Type/Code > values). > > > How are ports flattened? See the results of the following command. > > bro -e 'print 0/udp; print |0/udp|; print |32/tcp|; print |11/tcp|; > > print |132/unknown|; print 132/unknown;' > > Internally, a port is a single uint64 with some of the high-bits set to > indicate which port-space it belongs to. You?re seeing that value here. > E.g. > > $ bro -e 'print |32/tcp| == 0x10000 + 32' > T > $ bro -e 'print |37/udp| == 0x20000 + 37? > T > > - Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141030/1c58ec35/attachment.html From anthony.kasza at gmail.com Thu Oct 30 08:44:00 2014 From: anthony.kasza at gmail.com (anthony kasza) Date: Thu, 30 Oct 2014 08:44:00 -0700 Subject: [Bro] Attributes and Ports Questions In-Reply-To: <20141030145358.GA22582@icir.org> References: <20141030145358.GA22582@icir.org> Message-ID: That page is exactly where my questions are coming from. I tried using each of the attributes in a few toy scripts and was wondering if people are using them in production as I could not find some of them used in base or policy. Thanks for the insight, Robin. -AK On Oct 30, 2014 7:54 AM, "Robin Sommer" wrote: > Hi Anthony, > > have you seen this page? > > https://www.bro.org/sphinx-git/script-reference/attributes.html > > It's pretty new (though maybe it's actually where your questiosns are > coming from :) > > To add a bit to that: > > On Tue, Oct 28, 2014 at 18:10 -0700, anthony kasza wrote: > > > &rotate_interval > > &rotate_size > > This used to be primary log rotation mechanism before we switched to > the new logging system/format. I've been wondering if we should just > remove these attributes. > > > &mergeable > > &synchronize (I think there was a post earlier last month about this one) > > &persistent > > These are going to go away, but we aren't there yet. We may start > deprecating them with the next release, which is scheduled to ship > with a first version of their replacement, the new Broker library. > > > &group > > A bit of an obscure feature, originally added to toggle selected sets > of analysis dynamically from BroControl. Don't think that's used > anywhere and I'm inclined to remove it. > > > &add_func > > &delete_func > > These aren't used very often, but can be useful in individual cases. > > > &encrypt (applying this to a file causes Bro to "elegantly terminate" > for me) > > bro -Ci eth0 -e 'global f1: file = open("f.out") &encrypt' > > Another relict from old-style logging, although the new framework > doesn't have any equivalent functionality yet. > > Mind filing a ticket for the crash? We should either fix it or remove > the attribute. > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141030/e20c362d/attachment.html From hckim at narusec.com Thu Oct 30 17:24:17 2014 From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=) Date: Fri, 31 Oct 2014 09:24:17 +0900 Subject: [Bro] ssl decode Message-ID: Thank you for fast replay Bro it self decoding ssl could be hard. and I will check https://github.com/plashchynski/viewssld Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141031/c23c1bc4/attachment.html From vitologrillo at gmail.com Fri Oct 31 01:48:52 2014 From: vitologrillo at gmail.com (Vito Logrillo) Date: Fri, 31 Oct 2014 09:48:52 +0100 Subject: [Bro] How "priority" keyword works? Message-ID: Hi, i have some questions about priority keyword: i'll try to explain with an example. If i make a script able to write a new log file ------snippet code.bro------ event dns_message(c: connection, is_orig: bool, msg: dns_msg, len: count) &priority=5 { if(c$id$orig_p == 138/udp) { ...do something and write in my custom log file... } } ------------------------------------- My custom event dns_message overrides the standard event? The standard event is executed or not in this case? Should i find the same packet logged in dns.log and in my custom log or not? It depends on priority keyword? And what happens if i set priority = -5? Thanks Vito -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141031/d3a18526/attachment.html From jsiwek at illinois.edu Fri Oct 31 06:41:08 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 31 Oct 2014 13:41:08 +0000 Subject: [Bro] How "priority" keyword works? In-Reply-To: References: Message-ID: > On Oct 31, 2014, at 3:48 AM, Vito Logrillo wrote: > > ------snippet code.bro------ > event dns_message(c: connection, is_orig: bool, msg: dns_msg, len: count) &priority=5 > { > if(c$id$orig_p == 138/udp) > { > ...do something and write in my custom log file... > } > } > ------------------------------------- > > My custom event dns_message overrides the standard event? The body of it just gets executed before any other dns_message event handlers with a lower priority. > The standard event is executed or not in this case? It still executes. > Should i find the same packet logged in dns.log and in my custom log or not? In both (technically not in your custom log if the condition you show isn?t true). > It depends on priority keyword? No, &priority just changes the order that the event handlers execute (highest goes first). > And what happens if i set priority = -5? The logic in your event handler runs after other event handlers that have priority greater than -5 (if no &priority is given, it defaults to 0). If the priority of two event handlers is the same, the order is not well-defined. - Jon From vitologrillo at gmail.com Fri Oct 31 07:42:03 2014 From: vitologrillo at gmail.com (Vito Logrillo) Date: Fri, 31 Oct 2014 15:42:03 +0100 Subject: [Bro] How "priority" keyword works? In-Reply-To: References: Message-ID: Hi Jon, thanks for your reply. Only a question: How can avoid to execute the standard event and permit only the execution of my custom event? In the example below, how can avoid the log of the same packet (in dns.log and in my custom log) if the condition "if(c$id$orig_p == 138/udp)" is true? Thanks Vito 2014-10-31 14:41 GMT+01:00 Siwek, Jon : > > > On Oct 31, 2014, at 3:48 AM, Vito Logrillo > wrote: > > > > ------snippet code.bro------ > > event dns_message(c: connection, is_orig: bool, msg: dns_msg, len: > count) &priority=5 > > { > > if(c$id$orig_p == 138/udp) > > { > > ...do something and write in my custom log file... > > } > > } > > ------------------------------------- > > > > My custom event dns_message overrides the standard event? > > The body of it just gets executed before any other dns_message event > handlers with a lower priority. > > > The standard event is executed or not in this case? > > It still executes. > > > Should i find the same packet logged in dns.log and in my custom log or > not? > > In both (technically not in your custom log if the condition you show > isn?t true). > > > It depends on priority keyword? > > No, &priority just changes the order that the event handlers execute > (highest goes first). > > > And what happens if i set priority = -5? > > The logic in your event handler runs after other event handlers that have > priority greater than -5 (if no &priority is given, it defaults to 0). If > the priority of two event handlers is the same, the order is not > well-defined. > > - Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141031/e6e5fc1c/attachment.html From jsiwek at illinois.edu Fri Oct 31 08:21:24 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 31 Oct 2014 15:21:24 +0000 Subject: [Bro] How "priority" keyword works? In-Reply-To: References: Message-ID: > On Oct 31, 2014, at 9:42 AM, Vito Logrillo wrote: > > thanks for your reply. Only a question: How can avoid to execute the standard event and permit only the execution of my custom event? The main option is to simply not load the script that contains unwanted event handlers. You may have to run Bro with the ?-b? flag to do that. But it may also not load a lot of other default functionality that you want and you?ll either have to replicate some portions of the default scripts in your own, or pick and choose which scripts are ok to @load individually. > In the example below, how can avoid the log of the same packet (in dns.log and in my custom log) if the condition "if(c$id$orig_p == 138/udp)" is true? If you just care about modifying the logging aspects of the standard event rather than preventing it from running entirely, you may be able to customize that via logging filters. In this case, it seems you could supply the ?pred? field [1] for the default DNS logging filter. More reading at [2] that may help explain options for customized logging. - Jon [1] https://www.bro.org/sphinx/scripts/base/frameworks/logging/main.bro.html?highlight=filter#type-Log::Filter [2] https://www.bro.org/sphinx/scripting/index.html#custom-logging From brianallen at wustl.edu Fri Oct 31 12:45:58 2014 From: brianallen at wustl.edu (Allen, Brian) Date: Fri, 31 Oct 2014 19:45:58 +0000 Subject: [Bro] bro question with SIEM In-Reply-To: References: Message-ID: Hi - Our Medschool uses the IBM Qradar SIEM tool, and we have a project to expand it to cover the rest of the University. Since we have a SEIM now, I figured I might as well put the best logs I have in it - which include BRO logs: http, dns, conn, etc. IBM is asking me the following question: Is BRO able to forward raw flow data that has not been normalized or altered? I'm pretty sure the answer is no because I have worked with raw flow data with flow-tools a lot, but I wanted to post it here to make sure, plus see if anyone is using BRO with a SIEM and what those setups might look like. Thanks, -Brian Brian Allen, CISSP Information Security Manager Washington University brianallen at wustl.edu 314-935-5380 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141031/b4d9f954/attachment.html From branthale at gmail.com Fri Oct 31 13:08:51 2014 From: branthale at gmail.com (Brant Hale) Date: Fri, 31 Oct 2014 16:08:51 -0400 Subject: [Bro] bro question with SIEM In-Reply-To: References: Message-ID: Brian, I also have Qradar and am looking to supplement it with BRO - mainly the Security Onion platform. The systems have some overlap, I suspect that they are just going to want raw network data as they have their own tools to pull info out. I am planning on sending my syslog data to Qradar and pulling the BRO data from a network tap. So both systems will run in parallel not one reporting to the other. Do let us know what you end up with. Brant On Fri, Oct 31, 2014 at 3:45 PM, Allen, Brian wrote: > Hi - > > Our Medschool uses the IBM Qradar SIEM tool, and we have a project to > expand it to cover the rest of the University. Since we have a SEIM now, I > figured I might as well put the best logs I have in it - which include BRO > logs: http, dns, conn, etc. > > IBM is asking me the following question: Is BRO able to forward raw > flow data that has not been normalized or altered? > > I'm pretty sure the answer is no because I have worked with raw flow > data with flow-tools a lot, but I wanted to post it here to make sure, plus > see if anyone is using BRO with a SIEM and what those setups might look > like. > > Thanks, > -Brian > > Brian Allen, CISSP > Information Security Manager > Washington University > brianallen at wustl.edu > 314-935-5380 > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141031/c3ccbbd5/attachment.html From slagell at illinois.edu Fri Oct 31 13:15:02 2014 From: slagell at illinois.edu (Slagell, Adam J) Date: Fri, 31 Oct 2014 20:15:02 +0000 Subject: [Bro] bro question with SIEM In-Reply-To: References: Message-ID: <6D841E35-5A1F-4443-B9D3-CCED039E39DC@illinois.edu> You could send the logs or even the raw bro events. I'm not sure what they mean by raw flow data, but am guessing they mean like v9 netflows. That it won't do. On Oct 31, 2014, at 3:08 PM, Brant Hale > wrote: Brian, I also have Qradar and am looking to supplement it with BRO - mainly the Security Onion platform. The systems have some overlap, I suspect that they are just going to want raw network data as they have their own tools to pull info out. I am planning on sending my syslog data to Qradar and pulling the BRO data from a network tap. So both systems will run in parallel not one reporting to the other. Do let us know what you end up with. Brant On Fri, Oct 31, 2014 at 3:45 PM, Allen, Brian > wrote: Hi - Our Medschool uses the IBM Qradar SIEM tool, and we have a project to expand it to cover the rest of the University. Since we have a SEIM now, I figured I might as well put the best logs I have in it - which include BRO logs: http, dns, conn, etc. IBM is asking me the following question: Is BRO able to forward raw flow data that has not been normalized or altered? I'm pretty sure the answer is no because I have worked with raw flow data with flow-tools a lot, but I wanted to post it here to make sure, plus see if anyone is using BRO with a SIEM and what those setups might look like. Thanks, -Brian Brian Allen, CISSP Information Security Manager Washington University brianallen at wustl.edu 314-935-5380 _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro ------ Adam J. Slagell Chief Information Security Officer Assistant Director, Cybersecurity Directorate National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141031/8a1f59f4/attachment.html From rjenkins at rmjconsulting.net Fri Oct 31 13:44:39 2014 From: rjenkins at rmjconsulting.net (Ron Jenkins) Date: Fri, 31 Oct 2014 20:44:39 +0000 Subject: [Bro] bro question with SIEM In-Reply-To: References: Message-ID: We setup the Linux?s Syslog-NG v3x to monitor and forward the logs to Log Siphon?s Syslog daemon. Below is a example syslog.conf setup. Thanks! Below is the sample settings for syslog-ng.conf concerning sending the Bro2 logs to Log Siphon; items in dark red ARE required. * #BRO IDS source s1_broHttp { file("/opt/bro2/spool/bro/http.log" follow_freq(1)); }; source s1_broConn { file("/opt/bro2/spool/bro/conn.log" follow_freq(1)); }; source s1_broDNS { file("/opt/bro2/spool/bro/dns.log" follow_freq(1)); }; source s1_broFILES { file("/opt/bro2/spool/bro/files.log" follow_freq(1)); }; source s1_broSMTP { file("/opt/bro2/spool/bro/smtp.log" follow_freq(1)); }; source s1_broSMTPentities { file("/opt/bro2/spool/bro/smtp_entities.log" follow_freq(1)); }; source s1_broSoftware { file("/opt/bro2/spool/bro/software.log" follow_freq(1)); }; source s1_broSNMP { file("/opt/bro2/spool/bro/snmp.log" follow_freq(1)); }; source s1_broSSL { file("/opt/bro2/spool/bro/ssl.log" follow_freq(1)); }; source s1_broDPD { file("/opt/bro2/spool/bro/dpd.log" follow_freq(1)); }; source s1_broNotice { file("/opt/bro2/spool/bro/notice.log" follow_freq(1)); }; source s1_broSSH { file("/opt/bro2/spool/bro/ssh.log" follow_freq(1)); }; source s1_broSYSLOG { file("/opt/bro2/spool/bro/syslog.log" follow_freq(1)); }; source s1_broFTP { file("/opt/bro2/spool/bro/ftp.log" follow_freq(1)); }; source s1_broTUNNEL { file("/opt/bro2/spool/bro/tunnel.log" follow_freq(1)); }; source s1_broX509 { file("/opt/bro2/spool/bro/x509.log" follow_freq(1)); }; source s1_broDHCP { file("/opt/bro2/spool/bro/dhcp.log" follow_freq(1)); }; destination logsiphon1 { udp("X.X.X.X" port(514) template("BRO2 HTTP -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon2 { udp("X.X.X.X" port(514) template("BRO2 CONN -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon3 { udp("X.X.X.X" port(514) template("BRO2 DNS -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon4 { udp("X.X.X.X" port(514) template("BRO2 SMTP -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon5 { udp("X.X.X.X" port(514) template("BRO2 SMTP_ENTITIES -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon6 { udp("X.X.X.X" port(514) template("BRO2 SOFTWARE -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon7 { udp("X.X.X.X" port(514) template("BRO2 SSL -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon8 { udp("X.X.X.X" port(514) template("BRO2 DPD -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon9 { udp("X.X.X.X" port(514) template("BRO2 NOTICE -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon10 { udp("X.X.X.X" port(514) template("BRO2 FTP -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon11 { udp("X.X.X.X" port(514) template("BRO2 SSH -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon12 { udp("X.X.X.X" port(514) template("BRO2 SYSLOG -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon13 { udp("x.x.x.x" port(514) template("BRO2 TUNNEL -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon14 { udp("x.x.x.x" port(514) template("BRO2 FILES -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon15 { udp("x.x.x.x" port(514) template("BRO2 SNMP -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon16 { udp("x.x.x.x" port(514) template("BRO2 X509 -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; destination logsiphon17 { udp("x.x.x.x" port(514) template("BRO2 DHCP -> $FULLDATE $PROGRAM $MSGONLY \n") template_escape(no)); }; log { source(s1_broHttp); destination(logsiphon1); }; log { source(s1_broConn); destination(logsiphon2); }; log { source(s1_broDNS); destination(logsiphon3); }; log { source(s1_broSMTP); destination(logsiphon4); }; log { source(s1_broSMTPentities); destination(logsiphon5); }; log { source(s1_broSoftware); destination(logsiphon6); }; log { source(s1_broSSL); destination(logsiphon7); }; log { source(s1_broDPD); destination(logsiphon8); }; log { source(s1_broNotice); destination(logsiphon9); }; log { source(s1_broFTP); destination(logsiphon10); }; log { source(s1_broSSH); destination(logsiphon11); }; log { source(s1_broSYSLOG); destination(logsiphon12); }; log { source(s1_broTUNNEL); destination(logsiphon13); }; log { source(s1_broFILES); destination(logsiphon14); }; log { source(s1_broSNMP); destination(logsiphon15); }; log { source(s1_broX509); destination(logsiphon16); }; log { source(s1_broDHCP); destination(logsiphon17); }; Ron Jenkins (Owner / Senior Architect) RMJ Consulting, LLC. "Bringing Companies and Solutions Together" 11715 Bricksome Ave STE B-7 Baton Rouge, LA 70816 Toll: 855-448-5214 Direct. 225-448-5214 Ext #101 Fax. 225-448-5324 Cell. 225-931-1632 Email. rjenkins at rmjconsulting.net Web. http://www.rmjconsulting.net Log Siphon. http://www.logsiphon.com Linkedin. www.linkedin.com/in/ronmjenkins/ Twitter: www.twitter.com/RMJConsulting RMJ Consulting?s Technology Corner. https://www.rmjconsulting.net/main/paper.php From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Brant Hale Sent: Friday, October 31, 2014 3:09 PM To: Allen, Brian Cc: bro at bro.org Subject: Re: [Bro] bro question with SIEM Brian, I also have Qradar and am looking to supplement it with BRO - mainly the Security Onion platform. The systems have some overlap, I suspect that they are just going to want raw network data as they have their own tools to pull info out. I am planning on sending my syslog data to Qradar and pulling the BRO data from a network tap. So both systems will run in parallel not one reporting to the other. Do let us know what you end up with. Brant On Fri, Oct 31, 2014 at 3:45 PM, Allen, Brian > wrote: Hi - Our Medschool uses the IBM Qradar SIEM tool, and we have a project to expand it to cover the rest of the University. Since we have a SEIM now, I figured I might as well put the best logs I have in it - which include BRO logs: http, dns, conn, etc. IBM is asking me the following question: Is BRO able to forward raw flow data that has not been normalized or altered? I'm pretty sure the answer is no because I have worked with raw flow data with flow-tools a lot, but I wanted to post it here to make sure, plus see if anyone is using BRO with a SIEM and what those setups might look like. Thanks, -Brian Brian Allen, CISSP Information Security Manager Washington University brianallen at wustl.edu 314-935-5380 _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141031/3ece15b2/attachment.html From mike.patterson at uwaterloo.ca Fri Oct 31 15:51:20 2014 From: mike.patterson at uwaterloo.ca (Mike Patterson) Date: Fri, 31 Oct 2014 22:51:20 +0000 Subject: [Bro] bro question with SIEM In-Reply-To: References: , Message-ID: <7D0D94AA-28C9-4D44-B2D9-C855A7840D84@uwaterloo.ca> They definitely mean netflowv5 or 9. Bro can't do this, but you probably could generate flows from the same device you're running Bro on. I'm pretty sure there are some open source options here. -- Mike Patterson Manager, Information Security Operations Information Security Services, University of Waterloo +1 519-888-4567, x47178 / mike.patterson at uwaterloo.ca On Oct 31, 2014, at 16:00, Allen, Brian > wrote: Hi - Our Medschool uses the IBM Qradar SIEM tool, and we have a project to expand it to cover the rest of the University. Since we have a SEIM now, I figured I might as well put the best logs I have in it - which include BRO logs: http, dns, conn, etc. IBM is asking me the following question: Is BRO able to forward raw flow data that has not been normalized or altered? I'm pretty sure the answer is no because I have worked with raw flow data with flow-tools a lot, but I wanted to post it here to make sure, plus see if anyone is using BRO with a SIEM and what those setups might look like. Thanks, -Brian Brian Allen, CISSP Information Security Manager Washington University brianallen at wustl.edu 314-935-5380 _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141031/caa2130c/attachment.html From slagell at illinois.edu Fri Oct 31 16:01:13 2014 From: slagell at illinois.edu (Slagell, Adam J) Date: Fri, 31 Oct 2014 23:01:13 +0000 Subject: [Bro] bro question with SIEM In-Reply-To: <7D0D94AA-28C9-4D44-B2D9-C855A7840D84@uwaterloo.ca> References: , , <7D0D94AA-28C9-4D44-B2D9-C855A7840D84@uwaterloo.ca> Message-ID: <28256CFC-A32E-46D1-9DF3-80CE0A7FFE51@illinois.edu> We were going to do that once with Bro & Argus on the same boxes. But then we got better flow data from the routers with a network upgrade and moved to just setting up a single collector box. On Oct 31, 2014, at 5:52 PM, Mike Patterson > wrote: They definitely mean netflowv5 or 9. Bro can't do this, but you probably could generate flows from the same device you're running Bro on. I'm pretty sure there are some open source options here. -- Mike Patterson Manager, Information Security Operations Information Security Services, University of Waterloo +1 519-888-4567, x47178 / mike.patterson at uwaterloo.ca On Oct 31, 2014, at 16:00, Allen, Brian > wrote: Hi - Our Medschool uses the IBM Qradar SIEM tool, and we have a project to expand it to cover the rest of the University. Since we have a SEIM now, I figured I might as well put the best logs I have in it - which include BRO logs: http, dns, conn, etc. IBM is asking me the following question: Is BRO able to forward raw flow data that has not been normalized or altered? I'm pretty sure the answer is no because I have worked with raw flow data with flow-tools a lot, but I wanted to post it here to make sure, plus see if anyone is using BRO with a SIEM and what those setups might look like. Thanks, -Brian Brian Allen, CISSP Information Security Manager Washington University brianallen at wustl.edu 314-935-5380 _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20141031/99160d6f/attachment.html