From noreply at bro.org Tue Jul 1 00:00:17 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 1 Jul 2014 00:00:17 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407010700.s6170HUl026155@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ --------------- --------------- ---------- ---------- ------------- ---------- -------------------------------------------------------- BIT-1213 [1] broccoli-python Nicholas Weaver - 2014-06-30 - Normal broccoli/bindings/broccoli-python not building correctly [1] BIT-1213 https://bro-tracker.atlassian.net/browse/BIT-1213 From robin at icir.org Tue Jul 1 09:12:30 2014 From: robin at icir.org (Robin Sommer) Date: Tue, 1 Jul 2014 09:12:30 -0700 Subject: [Bro-Dev] Documenting Weirds In-Reply-To: References: <20140627215410.GL9039@icir.org> Message-ID: <20140701161230.GR38176@icir.org> On Sat, Jun 28, 2014 at 11:46 -0400, you wrote: > Maybe more generally, we should to make a Weird closer to a Notice. For > example, if a file analyzer generates a weird, there are no fields in the > weird.log to map it back to the offending file. Yeah, that would make a lot sense. > I realize that that's trickier, since Weirds can be generated from > either the core or script-land. One thing to keep in mind is that it shouldn't become burdensome to write the code for generating a Weird, in particular in core land. The situations that they report are (supposedly) rare, so it's not worth spending much time on. Right now, one appeal of the Weirds is that it's to say "alright, shouldn't happen, weird() if it does"; but if one would need to start building Vals etc. to pass the right parameters with it, one might become more inclined to skip the check (because it shouldn't happen anyways :) This is not a huge issue, and could probably be solved with a few wrappers covering common cases (and we have some of that already, like the WeirdConn() etc.; could extend that a bit more). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From noreply at bro.org Wed Jul 2 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 2 Jul 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407020700.s6270GER015577@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ --------------- --------------- ---------- ---------- ------------- ---------- -------------------------------------------------------- BIT-1213 [1] broccoli-python Nicholas Weaver - 2014-06-30 - Normal broccoli/bindings/broccoli-python not building correctly [1] BIT-1213 https://bro-tracker.atlassian.net/browse/BIT-1213 From noreply at bro.org Thu Jul 3 00:00:21 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 3 Jul 2014 00:00:21 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407030700.s6370LSV027530@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ --------------- --------------- ---------- ---------- ------------- ---------- -------------------------------------------------------- BIT-1213 [1] broccoli-python Nicholas Weaver - 2014-06-30 - Normal broccoli/bindings/broccoli-python not building correctly [1] BIT-1213 https://bro-tracker.atlassian.net/browse/BIT-1213 From noreply at bro.org Fri Jul 4 00:00:34 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 4 Jul 2014 00:00:34 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407040700.s6470YOk027546@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ --------------- --------------- ---------- ---------- ------------- ---------- -------------------------------------------------------- BIT-1213 [1] broccoli-python Nicholas Weaver - 2014-06-30 - Normal broccoli/bindings/broccoli-python not building correctly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ------------------------------------------------------ 1031cfb [2] bro-aux Daniel Thayer 2014-07-03 Add more tests of bro-cut 4ab6b08 [3] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when duplicate fields are specified 3be30b1 [4] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when log file has missing field 05982e3 [5] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut output of "#types" header line ecd71c1 [6] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when separator is not hexadecimal [1] BIT-1213 https://bro-tracker.atlassian.net/browse/BIT-1213 [2] 1031cfb https://github.com/bro/bro-aux/commit/1031cfb8c4871ebc33b62548644253fbdeb89bc4 [3] 4ab6b08 https://github.com/bro/bro-aux/commit/4ab6b087a3f7cf2b7875491798f7df9d1e2eece9 [4] 3be30b1 https://github.com/bro/bro-aux/commit/3be30b12ae1e1ca9dad7c68806b7649576b834d9 [5] 05982e3 https://github.com/bro/bro-aux/commit/05982e396c424333ff007b59f17467dfedaef5bc [6] ecd71c1 https://github.com/bro/bro-aux/commit/ecd71c1a6886e7982be8f7836decafc23a7fc175 From noreply at bro.org Sat Jul 5 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 5 Jul 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407050700.s6570Gp8005076@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ --------------- --------------- ---------- ---------- ------------- ---------- -------------------------------------------------------- BIT-1213 [1] broccoli-python Nicholas Weaver - 2014-06-30 - Normal broccoli/bindings/broccoli-python not building correctly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ------------------------------------------------------ 1031cfb [2] bro-aux Daniel Thayer 2014-07-03 Add more tests of bro-cut 4ab6b08 [3] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when duplicate fields are specified 3be30b1 [4] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when log file has missing field 05982e3 [5] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut output of "#types" header line ecd71c1 [6] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when separator is not hexadecimal [1] BIT-1213 https://bro-tracker.atlassian.net/browse/BIT-1213 [2] 1031cfb https://github.com/bro/bro-aux/commit/1031cfb8c4871ebc33b62548644253fbdeb89bc4 [3] 4ab6b08 https://github.com/bro/bro-aux/commit/4ab6b087a3f7cf2b7875491798f7df9d1e2eece9 [4] 3be30b1 https://github.com/bro/bro-aux/commit/3be30b12ae1e1ca9dad7c68806b7649576b834d9 [5] 05982e3 https://github.com/bro/bro-aux/commit/05982e396c424333ff007b59f17467dfedaef5bc [6] ecd71c1 https://github.com/bro/bro-aux/commit/ecd71c1a6886e7982be8f7836decafc23a7fc175 From noreply at bro.org Sun Jul 6 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 6 Jul 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407060700.s6670Ffw011035@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ --------------- --------------- ---------- ---------- ------------- ---------- -------------------------------------------------------- BIT-1213 [1] broccoli-python Nicholas Weaver - 2014-06-30 - Normal broccoli/bindings/broccoli-python not building correctly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ------------------------------------------------------ 1031cfb [2] bro-aux Daniel Thayer 2014-07-03 Add more tests of bro-cut 4ab6b08 [3] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when duplicate fields are specified 3be30b1 [4] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when log file has missing field 05982e3 [5] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut output of "#types" header line ecd71c1 [6] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when separator is not hexadecimal [1] BIT-1213 https://bro-tracker.atlassian.net/browse/BIT-1213 [2] 1031cfb https://github.com/bro/bro-aux/commit/1031cfb8c4871ebc33b62548644253fbdeb89bc4 [3] 4ab6b08 https://github.com/bro/bro-aux/commit/4ab6b087a3f7cf2b7875491798f7df9d1e2eece9 [4] 3be30b1 https://github.com/bro/bro-aux/commit/3be30b12ae1e1ca9dad7c68806b7649576b834d9 [5] 05982e3 https://github.com/bro/bro-aux/commit/05982e396c424333ff007b59f17467dfedaef5bc [6] ecd71c1 https://github.com/bro/bro-aux/commit/ecd71c1a6886e7982be8f7836decafc23a7fc175 From noreply at bro.org Mon Jul 7 00:00:28 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 7 Jul 2014 00:00:28 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407070700.s6770SdT015928@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ --------------- --------------- ---------- ---------- ------------- ---------- -------------------------------------------------------- BIT-1213 [1] broccoli-python Nicholas Weaver - 2014-06-30 - Normal broccoli/bindings/broccoli-python not building correctly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ------------------------------------------------------ 1031cfb [2] bro-aux Daniel Thayer 2014-07-03 Add more tests of bro-cut 4ab6b08 [3] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when duplicate fields are specified 3be30b1 [4] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when log file has missing field 05982e3 [5] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut output of "#types" header line ecd71c1 [6] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when separator is not hexadecimal [1] BIT-1213 https://bro-tracker.atlassian.net/browse/BIT-1213 [2] 1031cfb https://github.com/bro/bro-aux/commit/1031cfb8c4871ebc33b62548644253fbdeb89bc4 [3] 4ab6b08 https://github.com/bro/bro-aux/commit/4ab6b087a3f7cf2b7875491798f7df9d1e2eece9 [4] 3be30b1 https://github.com/bro/bro-aux/commit/3be30b12ae1e1ca9dad7c68806b7649576b834d9 [5] 05982e3 https://github.com/bro/bro-aux/commit/05982e396c424333ff007b59f17467dfedaef5bc [6] ecd71c1 https://github.com/bro/bro-aux/commit/ecd71c1a6886e7982be8f7836decafc23a7fc175 From jira at bro-tracker.atlassian.net Mon Jul 7 06:56:10 2014 From: jira at bro-tracker.atlassian.net (grigorescu (JIRA)) Date: Mon, 7 Jul 2014 08:56:10 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1166) installation does not take place in given prefix entirely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17100#comment-17100 ] grigorescu commented on BIT-1166: --------------------------------- After some off-list e-mails, the conclusion reached was: * broctl's CMakeList will install files in /var/opt/bro if the prefix is not /usr and the OS is not OS X. * This is in line with [FHS|http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard#Directory_structure]. * Modifying this behavior would be a breaking change for many places, but we could introduce a configure option to override the location of the directory. Personally, I build packages to install Bro, and these are the default paths in packages. I've had to work around core dumps running /var out of disk space, since the partitioning scheme didn't leave much room in /var. Having such an option would be useful for me, at least, but it's not immediately clear to me how trivial of a change this would be. > installation does not take place in given prefix entirely > --------------------------------------------------------- > > Key: BIT-1166 > URL: https://bro-tracker.atlassian.net/browse/BIT-1166 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, BroControl > Affects Versions: git/master > Reporter: Matthias Vallentin > Labels: build > > When configuring Bro to remain in a given prefix, say {{/opt/bro}}, the installation of BroControl still attempts to create a spool directory outside of the prefix: > {code} > ./configure --prefix=/opt/bro > make > make install > [...] > CMake Error at aux/broctl/cmake_install.cmake:200 (FILE): > file cannot create directory: /var/opt/bro/spool. Maybe need > administrative privileges. > {code} -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Mon Jul 7 07:37:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 7 Jul 2014 09:37:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1166) installation does not take place in given prefix entirely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1166: --------------------------- Status: Reopened (was: Closed) > installation does not take place in given prefix entirely > --------------------------------------------------------- > > Key: BIT-1166 > URL: https://bro-tracker.atlassian.net/browse/BIT-1166 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, BroControl > Affects Versions: git/master > Reporter: Matthias Vallentin > Labels: build > > When configuring Bro to remain in a given prefix, say {{/opt/bro}}, the installation of BroControl still attempts to create a spool directory outside of the prefix: > {code} > ./configure --prefix=/opt/bro > make > make install > [...] > CMake Error at aux/broctl/cmake_install.cmake:200 (FILE): > file cannot create directory: /var/opt/bro/spool. Maybe need > administrative privileges. > {code} -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Mon Jul 7 07:39:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 7 Jul 2014 09:39:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1166) installation does not take place in given prefix entirely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1166: --------------------------- Fix Version/s: 2.4 > installation does not take place in given prefix entirely > --------------------------------------------------------- > > Key: BIT-1166 > URL: https://bro-tracker.atlassian.net/browse/BIT-1166 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, BroControl > Affects Versions: git/master > Reporter: Matthias Vallentin > Labels: build > Fix For: 2.4 > > > When configuring Bro to remain in a given prefix, say {{/opt/bro}}, the installation of BroControl still attempts to create a spool directory outside of the prefix: > {code} > ./configure --prefix=/opt/bro > make > make install > [...] > CMake Error at aux/broctl/cmake_install.cmake:200 (FILE): > file cannot create directory: /var/opt/bro/spool. Maybe need > administrative privileges. > {code} -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Mon Jul 7 07:46:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 7 Jul 2014 09:46:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1166) installation does not take place in given prefix entirely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17101#comment-17101 ] Jon Siwek commented on BIT-1166: -------------------------------- Adding a configure option, {{--localstatedir}}, to set where the{{var}} dir goes should not be that difficult. > installation does not take place in given prefix entirely > --------------------------------------------------------- > > Key: BIT-1166 > URL: https://bro-tracker.atlassian.net/browse/BIT-1166 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, BroControl > Affects Versions: git/master > Reporter: Matthias Vallentin > Labels: build > Fix For: 2.4 > > > When configuring Bro to remain in a given prefix, say {{/opt/bro}}, the installation of BroControl still attempts to create a spool directory outside of the prefix: > {code} > ./configure --prefix=/opt/bro > make > make install > [...] > CMake Error at aux/broctl/cmake_install.cmake:200 (FILE): > file cannot create directory: /var/opt/bro/spool. Maybe need > administrative privileges. > {code} -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Mon Jul 7 07:46:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 7 Jul 2014 09:46:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1166) installation does not take place in given prefix entirely In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17101#comment-17101 ] Jon Siwek edited comment on BIT-1166 at 7/7/14 9:46 AM: -------------------------------------------------------- Adding a configure option, {{--localstatedir}}, to set where the {{var}} dir goes should not be that difficult. was (Author: jsiwek): Adding a configure option, {{--localstatedir}}, to set where the{{var}} dir goes should not be that difficult. > installation does not take place in given prefix entirely > --------------------------------------------------------- > > Key: BIT-1166 > URL: https://bro-tracker.atlassian.net/browse/BIT-1166 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, BroControl > Affects Versions: git/master > Reporter: Matthias Vallentin > Labels: build > Fix For: 2.4 > > > When configuring Bro to remain in a given prefix, say {{/opt/bro}}, the installation of BroControl still attempts to create a spool directory outside of the prefix: > {code} > ./configure --prefix=/opt/bro > make > make install > [...] > CMake Error at aux/broctl/cmake_install.cmake:200 (FILE): > file cannot create directory: /var/opt/bro/spool. Maybe need > administrative privileges. > {code} -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From noreply at bro.org Tue Jul 8 00:00:18 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 8 Jul 2014 00:00:18 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407080700.s6870Ixq004711@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ --------------- --------------- ---------- ---------- ------------- ---------- -------------------------------------------------------- BIT-1213 [1] broccoli-python Nicholas Weaver - 2014-06-30 - Normal broccoli/bindings/broccoli-python not building correctly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ------------------------------------------------------ 1031cfb [2] bro-aux Daniel Thayer 2014-07-03 Add more tests of bro-cut 4ab6b08 [3] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when duplicate fields are specified 3be30b1 [4] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when log file has missing field 05982e3 [5] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut output of "#types" header line ecd71c1 [6] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when separator is not hexadecimal [1] BIT-1213 https://bro-tracker.atlassian.net/browse/BIT-1213 [2] 1031cfb https://github.com/bro/bro-aux/commit/1031cfb8c4871ebc33b62548644253fbdeb89bc4 [3] 4ab6b08 https://github.com/bro/bro-aux/commit/4ab6b087a3f7cf2b7875491798f7df9d1e2eece9 [4] 3be30b1 https://github.com/bro/bro-aux/commit/3be30b12ae1e1ca9dad7c68806b7649576b834d9 [5] 05982e3 https://github.com/bro/bro-aux/commit/05982e396c424333ff007b59f17467dfedaef5bc [6] ecd71c1 https://github.com/bro/bro-aux/commit/ecd71c1a6886e7982be8f7836decafc23a7fc175 From jira at bro-tracker.atlassian.net Tue Jul 8 09:51:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 8 Jul 2014 11:51:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1213) broccoli/bindings/broccoli-python not building correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1213: --------------------------------- Assignee: Robin Sommer > broccoli/bindings/broccoli-python not building correctly > -------------------------------------------------------- > > Key: BIT-1213 > URL: https://bro-tracker.atlassian.net/browse/BIT-1213 > Project: Bro Issue Tracker > Issue Type: Problem > Components: broccoli-python > Affects Versions: 2.3 > Environment: OS-X 10.9.3 > Reporter: Nicholas Weaver > Assignee: Robin Sommer > > The setup.py routine fails due to path changes in 2.3, namely that the broccoli.h file is now in ../../build/src, as is the resulting library. > This patch appears to work: > diff --git a/setup.py b/setup.py > index 8a017f1..9cd19ae 100755 > --- a/setup.py > +++ b/setup.py > @@ -12,8 +12,8 @@ setup(name="broccoli-python", > py_modules=['broccoli'], > ext_modules = [ > Extension("_broccoli_intern", ["broccoli_intern_wrap.c"], > - include_dirs=["../../src"], > - library_dirs=["../../src/.libs"], > + include_dirs=["../../build/src"], > + library_dirs=["../../build/src"], > libraries=["broccoli"])] > ) -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Tue Jul 8 21:59:07 2014 From: jira at bro-tracker.atlassian.net (Robert W (JIRA)) Date: Tue, 8 Jul 2014 23:59:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1214) Updating Root CAs used for ssl.log In-Reply-To: References: Message-ID: Robert W created BIT-1214: ----------------------------- Summary: Updating Root CAs used for ssl.log Key: BIT-1214 URL: https://bro-tracker.atlassian.net/browse/BIT-1214 Project: Bro Issue Tracker Issue Type: Task Components: Bro Environment: Running on RHEL 6.5 Reporter: Robert W Need assistance confirming how to update the root CAs that Bro uses for the ssl.log. When list of websites are visited from the logs that have used a self-signed cert but within that list a number of sites are actually trusted. I found some documentation that states you need to take a DER formatted version of your root public key and convert it to Bro's hex string, etc. http://comments.gmane.org/gmane.comp.security.detection.bro/4117 Could you confirm the steps to take to resolve this specific issue? I am trying to ensure there isn't a specific location in a local config that will allow me to set the path. Please advise if you need any additional information. -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From noreply at bro.org Wed Jul 9 00:00:21 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 9 Jul 2014 00:00:21 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407090700.s6970L1E028200@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ --------------- --------------- ------------ ---------- ------------- ---------- -------------------------------------------------------- BIT-1213 [1] broccoli-python Nicholas Weaver Robin Sommer 2014-07-08 - Normal broccoli/bindings/broccoli-python not building correctly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ------------------------------------------------------ 1031cfb [2] bro-aux Daniel Thayer 2014-07-03 Add more tests of bro-cut 4ab6b08 [3] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when duplicate fields are specified 3be30b1 [4] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when log file has missing field 05982e3 [5] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut output of "#types" header line ecd71c1 [6] bro-aux Daniel Thayer 2014-07-03 Fix bug in bro-cut when separator is not hexadecimal [1] BIT-1213 https://bro-tracker.atlassian.net/browse/BIT-1213 [2] 1031cfb https://github.com/bro/bro-aux/commit/1031cfb8c4871ebc33b62548644253fbdeb89bc4 [3] 4ab6b08 https://github.com/bro/bro-aux/commit/4ab6b087a3f7cf2b7875491798f7df9d1e2eece9 [4] 3be30b1 https://github.com/bro/bro-aux/commit/3be30b12ae1e1ca9dad7c68806b7649576b834d9 [5] 05982e3 https://github.com/bro/bro-aux/commit/05982e396c424333ff007b59f17467dfedaef5bc [6] ecd71c1 https://github.com/bro/bro-aux/commit/ecd71c1a6886e7982be8f7836decafc23a7fc175 From jira at bro-tracker.atlassian.net Wed Jul 9 06:10:07 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Wed, 9 Jul 2014 08:10:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1214) Updating Root CAs used for ssl.log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann reassigned BIT-1214: ---------------------------------- Assignee: Johanna Amann > Updating Root CAs used for ssl.log > ---------------------------------- > > Key: BIT-1214 > URL: https://bro-tracker.atlassian.net/browse/BIT-1214 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Environment: Running on RHEL 6.5 > Reporter: Robert W > Assignee: Johanna Amann > Labels: logging > > Need assistance confirming how to update the root CAs that Bro uses for the ssl.log. When list of websites are visited from the logs that have used a self-signed cert but within that list a number of sites are actually trusted. I found some documentation that states you need to take a DER formatted version of your root public key and convert it to Bro's hex string, etc. > http://comments.gmane.org/gmane.comp.security.detection.bro/4117 > Could you confirm the steps to take to resolve this specific issue? I am trying to ensure there isn't a specific location in a local config that will allow me to set the path. Please advise if you need any additional information. -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Wed Jul 9 07:42:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 9 Jul 2014 09:42:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1213) broccoli/bindings/broccoli-python not building correctly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1213: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > broccoli/bindings/broccoli-python not building correctly > -------------------------------------------------------- > > Key: BIT-1213 > URL: https://bro-tracker.atlassian.net/browse/BIT-1213 > Project: Bro Issue Tracker > Issue Type: Problem > Components: broccoli-python > Affects Versions: 2.3 > Environment: OS-X 10.9.3 > Reporter: Nicholas Weaver > Assignee: Robin Sommer > > The setup.py routine fails due to path changes in 2.3, namely that the broccoli.h file is now in ../../build/src, as is the resulting library. > This patch appears to work: > diff --git a/setup.py b/setup.py > index 8a017f1..9cd19ae 100755 > --- a/setup.py > +++ b/setup.py > @@ -12,8 +12,8 @@ setup(name="broccoli-python", > py_modules=['broccoli'], > ext_modules = [ > Extension("_broccoli_intern", ["broccoli_intern_wrap.c"], > - include_dirs=["../../src"], > - library_dirs=["../../src/.libs"], > + include_dirs=["../../build/src"], > + library_dirs=["../../build/src"], > libraries=["broccoli"])] > ) -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Wed Jul 9 09:55:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Wed, 9 Jul 2014 11:55:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: Daniel Thayer created BIT-1215: ---------------------------------- Summary: bro-cut should be rewritten in C for speed and to not depend on gawk Key: BIT-1215 URL: https://bro-tracker.atlassian.net/browse/BIT-1215 Project: Bro Issue Tracker Issue Type: Improvement Components: bro-aux Reporter: Daniel Thayer Fix For: 2.4 The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Wed Jul 9 16:04:07 2014 From: jira at bro-tracker.atlassian.net (grigorescu (JIRA)) Date: Wed, 9 Jul 2014 18:04:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1216) Add Modbus record documentation In-Reply-To: References: Message-ID: grigorescu created BIT-1216: ------------------------------- Summary: Add Modbus record documentation Key: BIT-1216 URL: https://bro-tracker.atlassian.net/browse/BIT-1216 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: grigorescu Priority: Low Modbus records aren't documented: http://www.bro.org/sphinx-git/scripts/base/init-bare.bro.html#type-ModbusCoils While uid, tid, pid might mean something in Modbus terminology, it'd be nice to at least mention what they are (even something like: Modbus transaction ID). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Wed Jul 9 22:00:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 10 Jul 2014 00:00:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1215: ------------------------------- Component/s: Bro > bro-cut should be rewritten in C for speed and to not depend on gawk > -------------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Wed Jul 9 22:15:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 10 Jul 2014 00:15:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102#comment-17102 ] Daniel Thayer commented on BIT-1215: ------------------------------------ Branch topic/dnthayer/ticket1215 in bro and bro-aux repos contains the new bro-cut, and a couple of doc changes (remove gawk from list of optional Bro dependencies, and update btest sphinx PATH so that the documentation examples that use bro-cut can find the new bro-cut). > bro-cut should be rewritten in C for speed and to not depend on gawk > -------------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Wed Jul 9 22:17:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Thu, 10 Jul 2014 00:17:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1215: ------------------------------- Status: Merge Request (was: Open) > bro-cut should be rewritten in C for speed and to not depend on gawk > -------------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Wed Jul 9 22:32:07 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 10 Jul 2014 00:32:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1214) Updating Root CAs used for ssl.log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17103#comment-17103 ] Johanna Amann commented on BIT-1214: ------------------------------------ So - this question has two different answers, depending on what exactly is happening in your case. You mention that Bro does not validate certificates of sites that are actually trusted. In case the root-certificates that those sites chain back to are listed on http://www.mozilla.org/en-US/about/governance/policies/security-group/certs/included, this almost certainly means that the server is not sending one of the necessary intermediate certificates needed to verify the chain. Many browsers either cache those intermediate certificates or download them on-the-fly, so this kind of server configuration can go unnoticed for quite a while. You can use sites like https://www.ssllabs.com/ssltest/ to check your servers for this. In case the server is using a root certificate that is not included in the Mozilla root store (and hence not shipped with Bro), you have to add the extra root certificate to the list of root certificates known to Bro. The steps in the email thread should still be applicable - you can add your extra certificate to SSL::root_certs by adding it to local.bro like suggested in that thread. > Updating Root CAs used for ssl.log > ---------------------------------- > > Key: BIT-1214 > URL: https://bro-tracker.atlassian.net/browse/BIT-1214 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Environment: Running on RHEL 6.5 > Reporter: Robert W > Assignee: Johanna Amann > Labels: logging > > Need assistance confirming how to update the root CAs that Bro uses for the ssl.log. When list of websites are visited from the logs that have used a self-signed cert but within that list a number of sites are actually trusted. I found some documentation that states you need to take a DER formatted version of your root public key and convert it to Bro's hex string, etc. > http://comments.gmane.org/gmane.comp.security.detection.bro/4117 > Could you confirm the steps to take to resolve this specific issue? I am trying to ensure there isn't a specific location in a local config that will allow me to set the path. Please advise if you need any additional information. -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From noreply at bro.org Thu Jul 10 00:00:23 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 10 Jul 2014 00:00:23 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407100700.s6A70Ngm017580@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- -------------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer - 2014-07-10 2.4 Normal bro-cut should be rewritten in C for speed and to not depend on gawk [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From jira at bro-tracker.atlassian.net Thu Jul 10 14:14:07 2014 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Thu, 10 Jul 2014 16:14:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17104#comment-17104 ] Justin Azoff commented on BIT-1215: ----------------------------------- so, the MAX_LINE_LEN needs a closer look. I did a quick check against some of our http logs. Across 351536081 lines, there were 21 lines longer than about a megabyte. All of these were requests for webcams that use mjpeg / multi-part http responses. 350,000 responses in a single connection causes a very large log line of almost 10 megabytes. I think we should look into reallocing the line. The following check needs to exist either way, so resizing the array and re-reading instead of exiting shouldn't be too much more work, or affect performance. {code} linelen = strlen(line); if (linelen == MAX_LINE_LEN - 1) { {code} > bro-cut should be rewritten in C for speed and to not depend on gawk > -------------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From robin at icir.org Thu Jul 10 14:29:55 2014 From: robin at icir.org (Robin Sommer) Date: Thu, 10 Jul 2014 14:29:55 -0700 Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: <20140710212955.GJ42066@icir.org> I haven't looked at the code yet but if there's hard line length limit in there, that's a problem. bro-cut shouldn't care how long lines are. From jira at bro-tracker.atlassian.net Thu Jul 10 14:30:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 10 Jul 2014 16:30:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105#comment-17105 ] Robin Sommer commented on BIT-1215: ----------------------------------- I haven't looked at the code yet but if there's hard line length limit in there, that's a problem. bro-cut shouldn't care how long lines are. > bro-cut should be rewritten in C for speed and to not depend on gawk > -------------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From slagell at illinois.edu Thu Jul 10 15:15:22 2014 From: slagell at illinois.edu (Slagell, Adam J) Date: Thu, 10 Jul 2014 22:15:22 +0000 Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: , Message-ID: We are going to make it configurable and default to like a 1000KB line. Otherwise, you add a check to see if you need to reallocate memory for every line processed, which seems inefficient for edge cases. Letting the user override the default is a good compromise though. > On Jul 10, 2014, at 4:30 PM, "Robin Sommer (JIRA)" wrote: > > I haven't looked at the code yet but if there's hard line length > limit in there, that's a problem. bro-cut shouldn't care how long > lines are. From jira at bro-tracker.atlassian.net Thu Jul 10 15:16:07 2014 From: jira at bro-tracker.atlassian.net (Adam Slagell (JIRA)) Date: Thu, 10 Jul 2014 17:16:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106#comment-17106 ] Adam Slagell commented on BIT-1215: ----------------------------------- We are going to make it configurable and default to like a 1000KB line. Otherwise, you add a check to see if you need to reallocate memory for every line processed, which seems inefficient for edge cases. Letting the user override the default is a good compromise though. > bro-cut should be rewritten in C for speed and to not depend on gawk > -------------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Thu Jul 10 15:27:07 2014 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Thu, 10 Jul 2014 17:27:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107#comment-17107 ] Justin Azoff commented on BIT-1215: ----------------------------------- I think start with 1M and realloc 2x as needed is the way to go after all. We need (and already have) the check to see if fgets truncated the line. I think the only thing to do would be to add an absolute max line length of 64M or something to handle the case where someone accidentally runs bro-cut against a binary file (like a compressed bro log) that just doesn't contain any newlines. > bro-cut should be rewritten in C for speed and to not depend on gawk > -------------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Thu Jul 10 15:36:07 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 10 Jul 2014 17:36:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1217) Documentation: include type for vectors In-Reply-To: References: Message-ID: Johanna Amann created BIT-1217: ---------------------------------- Summary: Documentation: include type for vectors Key: BIT-1217 URL: https://bro-tracker.atlassian.net/browse/BIT-1217 Project: Bro Issue Tracker Issue Type: Problem Components: Bro, Website Affects Versions: git/master Reporter: Johanna Amann Fix For: 2.4 While browsing our documentation, I noticed that at the moment the script reference does not contain the type that is stored inside of a vector. This would be highly convenient sometimes. At the moment, it is e.g. impossible to find out what kind of Data a vector in an Info record contains. See http://www.bro.org/sphinx-git/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info for an example/ -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From robin at icir.org Thu Jul 10 15:41:33 2014 From: robin at icir.org (Robin Sommer) Date: Thu, 10 Jul 2014 15:41:33 -0700 Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: <20140710224133.GM42066@icir.org> On Thu, Jul 10, 2014 at 17:27 -0500, you wrote: > I think start with 1M and realloc 2x as needed is the way to go after > all. Yes. Maybe a bit less than 2x, exponential grows quickly. :) > I think the only thing to do would be to add an absolute max line > length of 64M or something to handle the case where someone > accidentally runs bro-cut against a binary file (like a compressed bro > log) that just doesn't contain any newlines. Would be nicer to recognize that differently, like by not finding a log header; that way we can give a good error message. If such a check is in place, I wouldn't actually bother with another double-check on line length; in the unlikely case that the file has a correct header but totally broken content, I'm sure there are plenty other cases where bro-cut would fail, and it seems there's not more here that can happen in addition than running out of memory (which the OS will catch). From jira at bro-tracker.atlassian.net Thu Jul 10 15:42:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 10 Jul 2014 17:42:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17108#comment-17108 ] Robin Sommer commented on BIT-1215: ----------------------------------- Yes. Maybe a bit less than 2x, exponential grows quickly. :) Would be nicer to recognize that differently, like by not finding a log header; that way we can give a good error message. If such a check is in place, I wouldn't actually bother with another double-check on line length; in the unlikely case that the file has a correct header but totally broken content, I'm sure there are plenty other cases where bro-cut would fail, and it seems there's not more here that can happen in addition than running out of memory (which the OS will catch). > bro-cut should be rewritten in C for speed and to not depend on gawk > -------------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Thu Jul 10 17:26:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 10 Jul 2014 19:26:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1217) Documentation: include type for vectors In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1217: --------------------------- Resolution: Fixed Status: Closed (was: Open) > Documentation: include type for vectors > --------------------------------------- > > Key: BIT-1217 > URL: https://bro-tracker.atlassian.net/browse/BIT-1217 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro, Website > Affects Versions: git/master > Reporter: Johanna Amann > Fix For: 2.4 > > > While browsing our documentation, I noticed that at the moment the script reference does not contain the type that is stored inside of a vector. > This would be highly convenient sometimes. At the moment, it is e.g. impossible to find out what kind of Data a vector in an Info record contains. See http://www.bro.org/sphinx-git/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info for an example/ -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From noreply at bro.org Fri Jul 11 00:00:17 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 11 Jul 2014 00:00:17 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407110700.s6B70Hq2005605@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- -------------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer - 2014-07-10 2.4 Normal bro-cut should be rewritten in C for speed and to not depend on gawk [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From jira at bro-tracker.atlassian.net Fri Jul 11 00:36:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Fri, 11 Jul 2014 02:36:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109#comment-17109 ] Daniel Thayer commented on BIT-1215: ------------------------------------ I've removed the hard-coded line length limit (now we're limited only by available memory). > bro-cut should be rewritten in C for speed and to not depend on gawk > -------------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Fri Jul 11 05:04:07 2014 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Fri, 11 Jul 2014 07:04:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110#comment-17110 ] Justin Azoff commented on BIT-1215: ----------------------------------- Ah, that was simple. I thought getline was a GNU thing, but OS X/bsd apparently has it too. > bro-cut should be rewritten in C for speed and to not depend on gawk > -------------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Fri Jul 11 07:00:07 2014 From: jira at bro-tracker.atlassian.net (gclark (JIRA)) Date: Fri, 11 Jul 2014 09:00:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten in C for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111#comment-17111 ] gclark commented on BIT-1215: ----------------------------- Why a static array with local code to resize instead of using something like std::vector? Is it a requirement that bro-cut be C and not C++? > bro-cut should be rewritten in C for speed and to not depend on gawk > -------------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Fri Jul 11 07:29:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Fri, 11 Jul 2014 09:29:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1215: ------------------------------- Summary: bro-cut should be rewritten for speed and to not depend on gawk (was: bro-cut should be rewritten in C for speed and to not depend on gawk) The current implementation can be compiled with a C++ compiler (and it works), so I guess it's already C++. > bro-cut should be rewritten for speed and to not depend on gawk > --------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Fri Jul 11 18:06:07 2014 From: jira at bro-tracker.atlassian.net (grigorescu (JIRA)) Date: Fri, 11 Jul 2014 20:06:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1218) misc/dump-events only dumps events handled by other scripts In-Reply-To: References: Message-ID: grigorescu created BIT-1218: ------------------------------- Summary: misc/dump-events only dumps events handled by other scripts Key: BIT-1218 URL: https://bro-tracker.atlassian.net/browse/BIT-1218 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: grigorescu Priority: Low misc/dump-events is a very handy script, and I often use it as a script writing tool. If I have a PCAP, I run it with misc/dump-events to get a quick sense of which events fire on it, and how many times each event fires. This helps me pick out the best event to handle. The issue is that events that aren't handled elsewhere don't get reported, as unhandled events aren't generated. Would it be possible to have dump-events (or perhaps dump-all-events) pretend like all events are handled, to get a more complete event listing? -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From noreply at bro.org Sat Jul 12 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 12 Jul 2014 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407120700.s6C70EtY011355@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- --------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer - 2014-07-11 2.4 Normal bro-cut should be rewritten for speed and to not depend on gawk [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From noreply at bro.org Sun Jul 13 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 13 Jul 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407130700.s6D70F2M031960@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- --------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer - 2014-07-11 2.4 Normal bro-cut should be rewritten for speed and to not depend on gawk [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From noreply at bro.org Mon Jul 14 00:00:22 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 14 Jul 2014 00:00:22 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407140700.s6E70MRv009965@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- --------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer - 2014-07-11 2.4 Normal bro-cut should be rewritten for speed and to not depend on gawk [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From jira at bro-tracker.atlassian.net Mon Jul 14 00:05:09 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Mon, 14 Jul 2014 02:05:09 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1214) Updating Root CAs used for ssl.log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1214: ------------------------------- Resolution: Solved Status: Closed (was: Open) > Updating Root CAs used for ssl.log > ---------------------------------- > > Key: BIT-1214 > URL: https://bro-tracker.atlassian.net/browse/BIT-1214 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Environment: Running on RHEL 6.5 > Reporter: Robert W > Assignee: Johanna Amann > Labels: logging > > Need assistance confirming how to update the root CAs that Bro uses for the ssl.log. When list of websites are visited from the logs that have used a self-signed cert but within that list a number of sites are actually trusted. I found some documentation that states you need to take a DER formatted version of your root public key and convert it to Bro's hex string, etc. > http://comments.gmane.org/gmane.comp.security.detection.bro/4117 > Could you confirm the steps to take to resolve this specific issue? I am trying to ensure there isn't a specific location in a local config that will allow me to set the path. Please advise if you need any additional information. -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Mon Jul 14 12:00:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Mon, 14 Jul 2014 14:00:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1219) broctl should have options to turn off cron emails In-Reply-To: References: Message-ID: Daniel Thayer created BIT-1219: ---------------------------------- Summary: broctl should have options to turn off cron emails Key: BIT-1219 URL: https://bro-tracker.atlassian.net/browse/BIT-1219 Project: Bro Issue Tracker Issue Type: Problem Components: BroControl Reporter: Daniel Thayer Fix For: 2.4 Several users have requested an easy way to turn off some emails that broctl cron sends (such as host up/down, "...not seeing any packets on interface...", etc.). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From noreply at bro.org Tue Jul 15 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 15 Jul 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407150700.s6F70FPo018709@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- --------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer - 2014-07-11 2.4 Normal bro-cut should be rewritten for speed and to not depend on gawk [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From noreply at bro.org Wed Jul 16 00:00:18 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 16 Jul 2014 00:00:18 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407160700.s6G70INa004341@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- --------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer - 2014-07-11 2.4 Normal bro-cut should be rewritten for speed and to not depend on gawk [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From noreply at bro.org Thu Jul 17 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 17 Jul 2014 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407170700.s6H70EMT015027@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- --------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer - 2014-07-11 2.4 Normal bro-cut should be rewritten for speed and to not depend on gawk [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From noreply at bro.org Fri Jul 18 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 18 Jul 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407180700.s6I70GiR023855@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- --------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer - 2014-07-11 2.4 Normal bro-cut should be rewritten for speed and to not depend on gawk [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From noreply at bro.org Sat Jul 19 00:00:18 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 19 Jul 2014 00:00:18 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407190700.s6J70IHv010721@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- --------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer - 2014-07-11 2.4 Normal bro-cut should be rewritten for speed and to not depend on gawk [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From noreply at bro.org Sun Jul 20 00:00:17 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 20 Jul 2014 00:00:17 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407200700.s6K70HBZ016379@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ---------- ---------- ------------- ---------- --------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer - 2014-07-11 2.4 Normal bro-cut should be rewritten for speed and to not depend on gawk [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From jira at bro-tracker.atlassian.net Sun Jul 20 00:05:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Sun, 20 Jul 2014 02:05:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1215: --------------------------------- Assignee: Robin Sommer > bro-cut should be rewritten for speed and to not depend on gawk > --------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Assignee: Robin Sommer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From noreply at bro.org Mon Jul 21 00:00:21 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 21 Jul 2014 00:00:21 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407210700.s6L70L21028765@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ------------ ---------- ------------- ---------- --------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer Robin Sommer 2014-07-20 2.4 Normal bro-cut should be rewritten for speed and to not depend on gawk [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From noreply at bro.org Tue Jul 22 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 22 Jul 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407220700.s6M70GEe009397@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ------------ ---------- ------------- ---------- --------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer Robin Sommer 2014-07-20 2.4 Normal bro-cut should be rewritten for speed and to not depend on gawk Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- --------- ---------- ------------------------------------------------- 4f1a504 [2] btest Jon Siwek 2014-07-21 Update MANIFEST.in and setup.py to fix packaging. [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 [2] 4f1a504 https://github.com/bro/btest/commit/4f1a5041525f08f982b502afcaafdce9e1e72682 From jira at bro-tracker.atlassian.net Tue Jul 22 17:41:08 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 22 Jul 2014 19:41:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1215: ------------------------------ Status: Open (was: Merge Request) > bro-cut should be rewritten for speed and to not depend on gawk > --------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Assignee: Robin Sommer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Tue Jul 22 17:41:08 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 22 Jul 2014 19:41:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1215: --------------------------------- Assignee: Daniel Thayer (was: Robin Sommer) > bro-cut should be rewritten for speed and to not depend on gawk > --------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Assignee: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Tue Jul 22 17:41:08 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 22 Jul 2014 19:41:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200#comment-17200 ] Robin Sommer commented on BIT-1215: ----------------------------------- I noticed a regression compared to the awk-version: the C bro-cut cannot handle more than one time column when converting to readable output. The branch {{topic/robin/ticket1215-merge}} has a test case in {{bro-cut/multiple-times.test}}. Might be a bit painful to fix, but I think we should ... > bro-cut should be rewritten for speed and to not depend on gawk > --------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Assignee: Robin Sommer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Wed Jul 23 17:20:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 23 Jul 2014 19:20:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1220) topic/robin/dynamic-plugins-2.3 In-Reply-To: References: Message-ID: Robin Sommer created BIT-1220: --------------------------------- Summary: topic/robin/dynamic-plugins-2.3 Key: BIT-1220 URL: https://bro-tracker.atlassian.net/browse/BIT-1220 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Robin Sommer This implements dynamic plugins for Bro, in the form of shared libraries loaded at startup. Tested on Linux, MacOS, and FreeBSD. Branches topic/robin/dynamic-plugins-2.3 are in bro, cmake, and bro-aux. An overview of the main functionality is in doc/devel/plugins.rst. This is a large change, and not everything is cast in stone yet. However, I think it would be good to get merged at this point to then fine-tune further later. I also have a few further branches based on this one that move more functionality over to the plugin structure (readers, writers, pktsrcs). I'll prepare them for merging later once this is in. Further notes about the code changes: - This removes the old Plugin macro magic, and hence touches all the existing analyzers to move them to the new API. Sorry. :) - The plugin API changed to generally use std::strings instead of const char*. - There are a number of invocations of PLUGIN_HOOK_{VOID,WITH_RESULT} across the code base, which allow plugins to hook into the processing at those locations. These are macros to make sure the overhead remains as low as possible when no plugin actually defines a hook (i.e., the normal case). See src/plugin/Manager.h for the macros' definition. - There's one hook which could be potentially expensive: plugins can be notified if a BroObj they are interested in gets destroyed. But I didn't see a performance impact in my tests (with no such hook defined), and the memory usage doesn't change due to field alignment. - The branch also adds a few new accessor methods to various classes to allow plugins to get to that information. - network_time cannot be just assigned to anymore, there's now function net_update_time() for that. - The branch redos how builtin variables are initialized, so that it works for plugins as well. No more init_net_var(), but instead bifcl-generated code that registers them. - same_type() gets an optional extra argument allowing record type comparision to ignore if field names don't match. - There are various changes for adjusting to the now dynamic generation of analyzer instances. - The file analysis API gets unified further with the protocol analyzer API (assigning IDs to analyzers; adding Init()/Done() methods; adding subtypes). - Adding a new command line option -Q that prints some basic execution time stats. Seems generally useful, and I'm planing to provide a plugin hook for measuring custom stuff. - I'm not yet happy with the current conventions for the C++ namespaces that plugins are in. I'm planing to clean that up later though, as I have some more branches relying on the current scheme and it will be easier to clean things up once everything is in. - My cmake style is probably not fully consistent with the rest of the build system. Feel free to adapt (or also to leave as it is). - There's a new piece of functionality for the file analysis framework: activate analyzers by MIME type. Pieces going in there: - File::register_for_mime_type(tag: Analyzer::Tag, mt: string): Associates a file analyzer with a MIME type. - File::add_analyzers_for_mime_type(f: fa_file, mtype: string): Activates all analyzers registered for a MIME type for the file. - The default file_new() handler calls File::add_analyzers_for_mime_type() with the file's MIME type. This isn't actually used yet by any existing file analyzer (because we don't have any yet that would target a specific file format), but there's a test making sure it works. -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From jira at bro-tracker.atlassian.net Wed Jul 23 17:20:08 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 23 Jul 2014 19:20:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1220) topic/robin/dynamic-plugins-2.3 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1220: ------------------------------ Status: Merge Request (was: Open) > topic/robin/dynamic-plugins-2.3 > ------------------------------- > > Key: BIT-1220 > URL: https://bro-tracker.atlassian.net/browse/BIT-1220 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > > This implements dynamic plugins for Bro, in the form of shared > libraries loaded at startup. Tested on Linux, MacOS, and FreeBSD. > Branches topic/robin/dynamic-plugins-2.3 are in bro, cmake, and bro-aux. > An overview of the main functionality is in doc/devel/plugins.rst. > This is a large change, and not everything is cast in stone yet. > However, I think it would be good to get merged at this point to then fine-tune further later. > I also have a few further branches based on this one that move more > functionality over to the plugin structure (readers, writers, > pktsrcs). I'll prepare them for merging later once this is in. > Further notes about the code changes: > - This removes the old Plugin macro magic, and hence touches all the > existing analyzers to move them to the new API. Sorry. :) > - The plugin API changed to generally use std::strings instead of > const char*. > - There are a number of invocations of PLUGIN_HOOK_{VOID,WITH_RESULT} > across the code base, which allow plugins to hook into the > processing at those locations. These are macros to make sure the > overhead remains as low as possible when no plugin actually defines > a hook (i.e., the normal case). See src/plugin/Manager.h for the > macros' definition. > - There's one hook which could be potentially expensive: plugins can > be notified if a BroObj they are interested in gets destroyed. But I > didn't see a performance impact in my tests (with no such hook > defined), and the memory usage doesn't change due to field > alignment. > - The branch also adds a few new accessor methods to various classes > to allow plugins to get to that information. > - network_time cannot be just assigned to anymore, there's now > function net_update_time() for that. > - The branch redos how builtin variables are initialized, so that it > works for plugins as well. No more init_net_var(), but instead > bifcl-generated code that registers them. > - same_type() gets an optional extra argument allowing record type > comparision to ignore if field names don't match. > - There are various changes for adjusting to the now dynamic > generation of analyzer instances. > - The file analysis API gets unified further with the protocol > analyzer API (assigning IDs to analyzers; adding Init()/Done() > methods; adding subtypes). > - Adding a new command line option -Q that prints some basic execution > time stats. Seems generally useful, and I'm planing to provide a > plugin hook for measuring custom stuff. > - I'm not yet happy with the current conventions for the C++ > namespaces that plugins are in. I'm planing to clean that up later > though, as I have some more branches relying on the current scheme > and it will be easier to clean things up once everything is in. > - My cmake style is probably not fully consistent with the rest of the > build system. Feel free to adapt (or also to leave as it is). > - There's a new piece of functionality for the file analysis > framework: activate analyzers by MIME type. Pieces going in there: > - File::register_for_mime_type(tag: Analyzer::Tag, mt: string): > Associates a file analyzer with a MIME type. > - File::add_analyzers_for_mime_type(f: fa_file, mtype: string): > Activates all analyzers registered for a MIME type for the file. > - The default file_new() handler calls > File::add_analyzers_for_mime_type() with the file's MIME type. > This isn't actually used yet by any existing file analyzer (because > we don't have any yet that would target a specific file format), > but there's a test making sure it works. -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From noreply at bro.org Thu Jul 24 00:00:29 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 24 Jul 2014 00:00:29 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407240700.s6O70TKb029749@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------- BIT-1220 [1] Bro Robin Sommer - 2014-07-23 - Normal topic/robin/dynamic-plugins-2.3 [2] [1] BIT-1220 https://bro-tracker.atlassian.net/browse/BIT-1220 [2] dynamic-plugins-2.3 https://github.com/bro/bro/tree/topic/robin/dynamic-plugins-2.3 From jira at bro-tracker.atlassian.net Thu Jul 24 12:54:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 24 Jul 2014 14:54:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1221) DPD website docs out of date In-Reply-To: References: Message-ID: Jon Siwek created BIT-1221: ------------------------------ Summary: DPD website docs out of date Key: BIT-1221 URL: https://bro-tracker.atlassian.net/browse/BIT-1221 Project: Bro Issue Tracker Issue Type: Problem Components: Website Reporter: Jon Siwek Fix For: 2.4 http://www.bro.org/development/howtos/dpd.html Some parts of that document reference old code. At a glance, {{dpd_config}}, {{DPM}}, and the use of {{int}} as the type for sequence numbers are things that pop out at me. -- This message was sent by Atlassian JIRA (v6.3-OD-08-005-WN#6328) From noreply at bro.org Fri Jul 25 00:00:20 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 25 Jul 2014 00:00:20 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407250700.s6P70KqB005033@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------- BIT-1220 [1] Bro Robin Sommer - 2014-07-23 - Normal topic/robin/dynamic-plugins-2.3 [2] [1] BIT-1220 https://bro-tracker.atlassian.net/browse/BIT-1220 [2] dynamic-plugins-2.3 https://github.com/bro/bro/tree/topic/robin/dynamic-plugins-2.3 From noreply at bro.org Sat Jul 26 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 26 Jul 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407260700.s6Q70FIP023094@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------- BIT-1220 [1] Bro Robin Sommer - 2014-07-23 - Normal topic/robin/dynamic-plugins-2.3 [2] [1] BIT-1220 https://bro-tracker.atlassian.net/browse/BIT-1220 [2] dynamic-plugins-2.3 https://github.com/bro/bro/tree/topic/robin/dynamic-plugins-2.3 From noreply at bro.org Sun Jul 27 00:00:19 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 27 Jul 2014 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407270700.s6R70JMq029062@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------- BIT-1220 [1] Bro Robin Sommer - 2014-07-23 - Normal topic/robin/dynamic-plugins-2.3 [2] [1] BIT-1220 https://bro-tracker.atlassian.net/browse/BIT-1220 [2] dynamic-plugins-2.3 https://github.com/bro/bro/tree/topic/robin/dynamic-plugins-2.3 From noreply at bro.org Mon Jul 28 00:00:17 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 28 Jul 2014 00:00:17 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407280700.s6S70HcV008177@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------- BIT-1220 [1] Bro Robin Sommer - 2014-07-23 - Normal topic/robin/dynamic-plugins-2.3 [2] [1] BIT-1220 https://bro-tracker.atlassian.net/browse/BIT-1220 [2] dynamic-plugins-2.3 https://github.com/bro/bro/tree/topic/robin/dynamic-plugins-2.3 From jira at bro-tracker.atlassian.net Mon Jul 28 07:52:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 28 Jul 2014 09:52:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1220) topic/robin/dynamic-plugins-2.3 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1220: --------------------------- Assignee: Jon Siwek > topic/robin/dynamic-plugins-2.3 > ------------------------------- > > Key: BIT-1220 > URL: https://bro-tracker.atlassian.net/browse/BIT-1220 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > > This implements dynamic plugins for Bro, in the form of shared > libraries loaded at startup. Tested on Linux, MacOS, and FreeBSD. > Branches topic/robin/dynamic-plugins-2.3 are in bro, cmake, and bro-aux. > An overview of the main functionality is in doc/devel/plugins.rst. > This is a large change, and not everything is cast in stone yet. > However, I think it would be good to get merged at this point to then fine-tune further later. > I also have a few further branches based on this one that move more > functionality over to the plugin structure (readers, writers, > pktsrcs). I'll prepare them for merging later once this is in. > Further notes about the code changes: > - This removes the old Plugin macro magic, and hence touches all the > existing analyzers to move them to the new API. Sorry. :) > - The plugin API changed to generally use std::strings instead of > const char*. > - There are a number of invocations of PLUGIN_HOOK_{VOID,WITH_RESULT} > across the code base, which allow plugins to hook into the > processing at those locations. These are macros to make sure the > overhead remains as low as possible when no plugin actually defines > a hook (i.e., the normal case). See src/plugin/Manager.h for the > macros' definition. > - There's one hook which could be potentially expensive: plugins can > be notified if a BroObj they are interested in gets destroyed. But I > didn't see a performance impact in my tests (with no such hook > defined), and the memory usage doesn't change due to field > alignment. > - The branch also adds a few new accessor methods to various classes > to allow plugins to get to that information. > - network_time cannot be just assigned to anymore, there's now > function net_update_time() for that. > - The branch redos how builtin variables are initialized, so that it > works for plugins as well. No more init_net_var(), but instead > bifcl-generated code that registers them. > - same_type() gets an optional extra argument allowing record type > comparision to ignore if field names don't match. > - There are various changes for adjusting to the now dynamic > generation of analyzer instances. > - The file analysis API gets unified further with the protocol > analyzer API (assigning IDs to analyzers; adding Init()/Done() > methods; adding subtypes). > - Adding a new command line option -Q that prints some basic execution > time stats. Seems generally useful, and I'm planing to provide a > plugin hook for measuring custom stuff. > - I'm not yet happy with the current conventions for the C++ > namespaces that plugins are in. I'm planing to clean that up later > though, as I have some more branches relying on the current scheme > and it will be easier to clean things up once everything is in. > - My cmake style is probably not fully consistent with the rest of the > build system. Feel free to adapt (or also to leave as it is). > - There's a new piece of functionality for the file analysis > framework: activate analyzers by MIME type. Pieces going in there: > - File::register_for_mime_type(tag: Analyzer::Tag, mt: string): > Associates a file analyzer with a MIME type. > - File::add_analyzers_for_mime_type(f: fa_file, mtype: string): > Activates all analyzers registered for a MIME type for the file. > - The default file_new() handler calls > File::add_analyzers_for_mime_type() with the file's MIME type. > This isn't actually used yet by any existing file analyzer (because > we don't have any yet that would target a specific file format), > but there's a test making sure it works. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From noreply at bro.org Tue Jul 29 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 29 Jul 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407290700.s6T70Fnd018339@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------- BIT-1220 [1] Bro Robin Sommer Jon Siwek 2014-07-28 - Normal topic/robin/dynamic-plugins-2.3 [2] [1] BIT-1220 https://bro-tracker.atlassian.net/browse/BIT-1220 [2] dynamic-plugins-2.3 https://github.com/bro/bro/tree/topic/robin/dynamic-plugins-2.3 From noreply at bro.org Wed Jul 30 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 30 Jul 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407300700.s6U70FaB004407@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ----------------------------------- BIT-1220 [1] Bro Robin Sommer Jon Siwek 2014-07-28 - Normal topic/robin/dynamic-plugins-2.3 [2] [1] BIT-1220 https://bro-tracker.atlassian.net/browse/BIT-1220 [2] dynamic-plugins-2.3 https://github.com/bro/bro/tree/topic/robin/dynamic-plugins-2.3 From jira at bro-tracker.atlassian.net Wed Jul 30 09:58:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Wed, 30 Jul 2014 11:58:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300#comment-17300 ] Daniel Thayer commented on BIT-1215: ------------------------------------ In branch topic/dnthayer/ticket1215, I've made the following changes: 1) bro-cut now handles time conversion for multiple time columns in a log file (and there is a new test case), 2) bro-cut no longer has a hard-coded limit on the number of columns that it can handle, 3) all tests now pass on OS X (previously, some were failing due to strftime("%z") behavior on OS X) > bro-cut should be rewritten for speed and to not depend on gawk > --------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Assignee: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Wed Jul 30 09:58:07 2014 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Wed, 30 Jul 2014 11:58:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1215: ------------------------------- Status: Merge Request (was: Open) > bro-cut should be rewritten for speed and to not depend on gawk > --------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Assignee: Daniel Thayer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Wed Jul 30 12:53:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 30 Jul 2014 14:53:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1220) topic/robin/dynamic-plugins-2.3 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17301#comment-17301 ] Jon Siwek commented on BIT-1220: -------------------------------- Nice code, I like the idea of the plugin-hooks. Questions: * NEWS worthy? * It seemed like Plugin::Configure should be a private method and Plugin::InitPreScript, Plugin::InitPostScript, and Plugin::Done should be protected? (I've changed it already, but can revert if you think otherwise). * Was there any code at this point that uses the new same_type() parameter for ignoring record field names? I wasn't finding any usages and I was just curious what it's for (or will be for) ? * I think I saw some failures of testing/btest/plugins/bifs-and-scripts* and it's due to the initialization order of global plugin::__RegisterBif objects being undefined. Maybe just makes sense to sort the baselines for those tests, or do you think something more is needed? * Any thoughts/plans for making it possible to hot-swap dynamic plugins? * What's the general story going to look like for ABI compatibility? ** When does BRO_PLUGIN_API_VERSION get incremented? Is it when the plugin API changes, or when anything in Bro changes that breaks ABI? ** Is there a missing check of Plugin::APIVersion(); was expecting plugin manager to do it sometime after dlopen, but didn't find it? ** Currently, if a dynamic plugin is found to be incompatible, that would mean it's already run its Configure() implementation and global plugin::__RegisterBif objects were initialized? Does it need to be possible to check compatibility at an earlier stage where it will have not yet been allowed to modify anything? Otherwise, I'll complete the merge later today or early tomorrow. > topic/robin/dynamic-plugins-2.3 > ------------------------------- > > Key: BIT-1220 > URL: https://bro-tracker.atlassian.net/browse/BIT-1220 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > > This implements dynamic plugins for Bro, in the form of shared > libraries loaded at startup. Tested on Linux, MacOS, and FreeBSD. > Branches topic/robin/dynamic-plugins-2.3 are in bro, cmake, and bro-aux. > An overview of the main functionality is in doc/devel/plugins.rst. > This is a large change, and not everything is cast in stone yet. > However, I think it would be good to get merged at this point to then fine-tune further later. > I also have a few further branches based on this one that move more > functionality over to the plugin structure (readers, writers, > pktsrcs). I'll prepare them for merging later once this is in. > Further notes about the code changes: > - This removes the old Plugin macro magic, and hence touches all the > existing analyzers to move them to the new API. Sorry. :) > - The plugin API changed to generally use std::strings instead of > const char*. > - There are a number of invocations of PLUGIN_HOOK_{VOID,WITH_RESULT} > across the code base, which allow plugins to hook into the > processing at those locations. These are macros to make sure the > overhead remains as low as possible when no plugin actually defines > a hook (i.e., the normal case). See src/plugin/Manager.h for the > macros' definition. > - There's one hook which could be potentially expensive: plugins can > be notified if a BroObj they are interested in gets destroyed. But I > didn't see a performance impact in my tests (with no such hook > defined), and the memory usage doesn't change due to field > alignment. > - The branch also adds a few new accessor methods to various classes > to allow plugins to get to that information. > - network_time cannot be just assigned to anymore, there's now > function net_update_time() for that. > - The branch redos how builtin variables are initialized, so that it > works for plugins as well. No more init_net_var(), but instead > bifcl-generated code that registers them. > - same_type() gets an optional extra argument allowing record type > comparision to ignore if field names don't match. > - There are various changes for adjusting to the now dynamic > generation of analyzer instances. > - The file analysis API gets unified further with the protocol > analyzer API (assigning IDs to analyzers; adding Init()/Done() > methods; adding subtypes). > - Adding a new command line option -Q that prints some basic execution > time stats. Seems generally useful, and I'm planing to provide a > plugin hook for measuring custom stuff. > - I'm not yet happy with the current conventions for the C++ > namespaces that plugins are in. I'm planing to clean that up later > though, as I have some more branches relying on the current scheme > and it will be easier to clean things up once everything is in. > - My cmake style is probably not fully consistent with the rest of the > build system. Feel free to adapt (or also to leave as it is). > - There's a new piece of functionality for the file analysis > framework: activate analyzers by MIME type. Pieces going in there: > - File::register_for_mime_type(tag: Analyzer::Tag, mt: string): > Associates a file analyzer with a MIME type. > - File::add_analyzers_for_mime_type(f: fa_file, mtype: string): > Activates all analyzers registered for a MIME type for the file. > - The default file_new() handler calls > File::add_analyzers_for_mime_type() with the file's MIME type. > This isn't actually used yet by any existing file analyzer (because > we don't have any yet that would target a specific file format), > but there's a test making sure it works. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Wed Jul 30 13:34:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 30 Jul 2014 15:34:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1220) topic/robin/dynamic-plugins-2.3 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17302#comment-17302 ] Robin Sommer commented on BIT-1220: ----------------------------------- Yeah, at a high level ("we now have a plugin interface that can do X, Y, Z"). I can add that later if you want. I'm also planing a blog posting once the dust has settled. Sounds good. It's actually used only by the BinPAC++ plugin right now, which auto-generates Bro records that need to match Bro events, but it can only tell field types and not names. (Independent of that, I have wondered before if we really want the names-must-match semantics in Bro, but that's a different topic.) More generally, as I haven't explicitly said it (though I'm sure you have realized :): Some of the features in this branch are driven by what the BinPAC++/HILTI work needs. However I've tried to limit it to things that I can see more generally useful. Yeah, sorting should solve it. Now you are getting ambitious. :-) Haven't though about that, but it's probably hard because once in particular the script-level stuff is in use, it's tricky to change it later. I would like it to be the latter, but I'm afraid it will be the former. The problem is that we haven't defined (yet?) what part of Bro's ABI plugins are allowed to use. They have access to everything, but in practice they should really limit themselves to a subset; just what that subset that is exactly, remains unclear. For now I see the API version mostly as way to catch some easy mistakes, but nothing foolproof. Oh, yes, that was there once, but seems it got lost. If you see the right place, just add it, otherwise I can do that later. I think we should just generally abort when a plugin is incompatible, and then it doesn't really matter. Not sure it's worth being more clever. Cool, thanks for reviewing! > topic/robin/dynamic-plugins-2.3 > ------------------------------- > > Key: BIT-1220 > URL: https://bro-tracker.atlassian.net/browse/BIT-1220 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > > This implements dynamic plugins for Bro, in the form of shared > libraries loaded at startup. Tested on Linux, MacOS, and FreeBSD. > Branches topic/robin/dynamic-plugins-2.3 are in bro, cmake, and bro-aux. > An overview of the main functionality is in doc/devel/plugins.rst. > This is a large change, and not everything is cast in stone yet. > However, I think it would be good to get merged at this point to then fine-tune further later. > I also have a few further branches based on this one that move more > functionality over to the plugin structure (readers, writers, > pktsrcs). I'll prepare them for merging later once this is in. > Further notes about the code changes: > - This removes the old Plugin macro magic, and hence touches all the > existing analyzers to move them to the new API. Sorry. :) > - The plugin API changed to generally use std::strings instead of > const char*. > - There are a number of invocations of PLUGIN_HOOK_{VOID,WITH_RESULT} > across the code base, which allow plugins to hook into the > processing at those locations. These are macros to make sure the > overhead remains as low as possible when no plugin actually defines > a hook (i.e., the normal case). See src/plugin/Manager.h for the > macros' definition. > - There's one hook which could be potentially expensive: plugins can > be notified if a BroObj they are interested in gets destroyed. But I > didn't see a performance impact in my tests (with no such hook > defined), and the memory usage doesn't change due to field > alignment. > - The branch also adds a few new accessor methods to various classes > to allow plugins to get to that information. > - network_time cannot be just assigned to anymore, there's now > function net_update_time() for that. > - The branch redos how builtin variables are initialized, so that it > works for plugins as well. No more init_net_var(), but instead > bifcl-generated code that registers them. > - same_type() gets an optional extra argument allowing record type > comparision to ignore if field names don't match. > - There are various changes for adjusting to the now dynamic > generation of analyzer instances. > - The file analysis API gets unified further with the protocol > analyzer API (assigning IDs to analyzers; adding Init()/Done() > methods; adding subtypes). > - Adding a new command line option -Q that prints some basic execution > time stats. Seems generally useful, and I'm planing to provide a > plugin hook for measuring custom stuff. > - I'm not yet happy with the current conventions for the C++ > namespaces that plugins are in. I'm planing to clean that up later > though, as I have some more branches relying on the current scheme > and it will be easier to clean things up once everything is in. > - My cmake style is probably not fully consistent with the rest of the > build system. Feel free to adapt (or also to leave as it is). > - There's a new piece of functionality for the file analysis > framework: activate analyzers by MIME type. Pieces going in there: > - File::register_for_mime_type(tag: Analyzer::Tag, mt: string): > Associates a file analyzer with a MIME type. > - File::add_analyzers_for_mime_type(f: fa_file, mtype: string): > Activates all analyzers registered for a MIME type for the file. > - The default file_new() handler calls > File::add_analyzers_for_mime_type() with the file's MIME type. > This isn't actually used yet by any existing file analyzer (because > we don't have any yet that would target a specific file format), > but there's a test making sure it works. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Wed Jul 30 15:16:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 30 Jul 2014 17:16:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1220) topic/robin/dynamic-plugins-2.3 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303#comment-17303 ] Jon Siwek commented on BIT-1220: -------------------------------- It's merged now, leaving NEWS and the missing APIVersion() check for you. > topic/robin/dynamic-plugins-2.3 > ------------------------------- > > Key: BIT-1220 > URL: https://bro-tracker.atlassian.net/browse/BIT-1220 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.4 > > > This implements dynamic plugins for Bro, in the form of shared > libraries loaded at startup. Tested on Linux, MacOS, and FreeBSD. > Branches topic/robin/dynamic-plugins-2.3 are in bro, cmake, and bro-aux. > An overview of the main functionality is in doc/devel/plugins.rst. > This is a large change, and not everything is cast in stone yet. > However, I think it would be good to get merged at this point to then fine-tune further later. > I also have a few further branches based on this one that move more > functionality over to the plugin structure (readers, writers, > pktsrcs). I'll prepare them for merging later once this is in. > Further notes about the code changes: > - This removes the old Plugin macro magic, and hence touches all the > existing analyzers to move them to the new API. Sorry. :) > - The plugin API changed to generally use std::strings instead of > const char*. > - There are a number of invocations of PLUGIN_HOOK_{VOID,WITH_RESULT} > across the code base, which allow plugins to hook into the > processing at those locations. These are macros to make sure the > overhead remains as low as possible when no plugin actually defines > a hook (i.e., the normal case). See src/plugin/Manager.h for the > macros' definition. > - There's one hook which could be potentially expensive: plugins can > be notified if a BroObj they are interested in gets destroyed. But I > didn't see a performance impact in my tests (with no such hook > defined), and the memory usage doesn't change due to field > alignment. > - The branch also adds a few new accessor methods to various classes > to allow plugins to get to that information. > - network_time cannot be just assigned to anymore, there's now > function net_update_time() for that. > - The branch redos how builtin variables are initialized, so that it > works for plugins as well. No more init_net_var(), but instead > bifcl-generated code that registers them. > - same_type() gets an optional extra argument allowing record type > comparision to ignore if field names don't match. > - There are various changes for adjusting to the now dynamic > generation of analyzer instances. > - The file analysis API gets unified further with the protocol > analyzer API (assigning IDs to analyzers; adding Init()/Done() > methods; adding subtypes). > - Adding a new command line option -Q that prints some basic execution > time stats. Seems generally useful, and I'm planing to provide a > plugin hook for measuring custom stuff. > - I'm not yet happy with the current conventions for the C++ > namespaces that plugins are in. I'm planing to clean that up later > though, as I have some more branches relying on the current scheme > and it will be easier to clean things up once everything is in. > - My cmake style is probably not fully consistent with the rest of the > build system. Feel free to adapt (or also to leave as it is). > - There's a new piece of functionality for the file analysis > framework: activate analyzers by MIME type. Pieces going in there: > - File::register_for_mime_type(tag: Analyzer::Tag, mt: string): > Associates a file analyzer with a MIME type. > - File::add_analyzers_for_mime_type(f: fa_file, mtype: string): > Activates all analyzers registered for a MIME type for the file. > - The default file_new() handler calls > File::add_analyzers_for_mime_type() with the file's MIME type. > This isn't actually used yet by any existing file analyzer (because > we don't have any yet that would target a specific file format), > but there's a test making sure it works. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Wed Jul 30 15:16:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 30 Jul 2014 17:16:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1220) topic/robin/dynamic-plugins-2.3 In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1220: --------------------------- Fix Version/s: 2.4 Status: Closed (was: Merge Request) > topic/robin/dynamic-plugins-2.3 > ------------------------------- > > Key: BIT-1220 > URL: https://bro-tracker.atlassian.net/browse/BIT-1220 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.4 > > > This implements dynamic plugins for Bro, in the form of shared > libraries loaded at startup. Tested on Linux, MacOS, and FreeBSD. > Branches topic/robin/dynamic-plugins-2.3 are in bro, cmake, and bro-aux. > An overview of the main functionality is in doc/devel/plugins.rst. > This is a large change, and not everything is cast in stone yet. > However, I think it would be good to get merged at this point to then fine-tune further later. > I also have a few further branches based on this one that move more > functionality over to the plugin structure (readers, writers, > pktsrcs). I'll prepare them for merging later once this is in. > Further notes about the code changes: > - This removes the old Plugin macro magic, and hence touches all the > existing analyzers to move them to the new API. Sorry. :) > - The plugin API changed to generally use std::strings instead of > const char*. > - There are a number of invocations of PLUGIN_HOOK_{VOID,WITH_RESULT} > across the code base, which allow plugins to hook into the > processing at those locations. These are macros to make sure the > overhead remains as low as possible when no plugin actually defines > a hook (i.e., the normal case). See src/plugin/Manager.h for the > macros' definition. > - There's one hook which could be potentially expensive: plugins can > be notified if a BroObj they are interested in gets destroyed. But I > didn't see a performance impact in my tests (with no such hook > defined), and the memory usage doesn't change due to field > alignment. > - The branch also adds a few new accessor methods to various classes > to allow plugins to get to that information. > - network_time cannot be just assigned to anymore, there's now > function net_update_time() for that. > - The branch redos how builtin variables are initialized, so that it > works for plugins as well. No more init_net_var(), but instead > bifcl-generated code that registers them. > - same_type() gets an optional extra argument allowing record type > comparision to ignore if field names don't match. > - There are various changes for adjusting to the now dynamic > generation of analyzer instances. > - The file analysis API gets unified further with the protocol > analyzer API (assigning IDs to analyzers; adding Init()/Done() > methods; adding subtypes). > - Adding a new command line option -Q that prints some basic execution > time stats. Seems generally useful, and I'm planing to provide a > plugin hook for measuring custom stuff. > - I'm not yet happy with the current conventions for the C++ > namespaces that plugins are in. I'm planing to clean that up later > though, as I have some more branches relying on the current scheme > and it will be easier to clean things up once everything is in. > - My cmake style is probably not fully consistent with the rest of the > build system. Feel free to adapt (or also to leave as it is). > - There's a new piece of functionality for the file analysis > framework: activate analyzers by MIME type. Pieces going in there: > - File::register_for_mime_type(tag: Analyzer::Tag, mt: string): > Associates a file analyzer with a MIME type. > - File::add_analyzers_for_mime_type(f: fa_file, mtype: string): > Activates all analyzers registered for a MIME type for the file. > - The default file_new() handler calls > File::add_analyzers_for_mime_type() with the file's MIME type. > This isn't actually used yet by any existing file analyzer (because > we don't have any yet that would target a specific file format), > but there's a test making sure it works. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From noreply at bro.org Thu Jul 31 00:00:17 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 31 Jul 2014 00:00:17 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201407310700.s6V70HLA018607@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ------------- ---------- ------------- ---------- --------------------------------------------------------------- BIT-1215 [1] Bro,bro-aux Daniel Thayer Daniel Thayer 2014-07-30 2.4 Normal bro-cut should be rewritten for speed and to not depend on gawk [1] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From jira at bro-tracker.atlassian.net Thu Jul 31 15:21:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 31 Jul 2014 17:21:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: Robin Sommer created BIT-1222: --------------------------------- Summary: topic/robin/reader-writer-plugins Key: BIT-1222 URL: https://bro-tracker.atlassian.net/browse/BIT-1222 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Robin Sommer This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Jul 31 15:21:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 31 Jul 2014 17:21:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1222: ------------------------------ Status: Merge Request (was: Open) > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000)