From noreply at bro.org Fri Aug 1 00:00:47 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 1 Aug 2014 00:00:47 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408010700.s7170l31022834@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------- ------------- ---------- ------------- ---------- --------------------------------------------------------------- BIT-1222 [1] Bro Robin Sommer - 2014-07-31 - Normal topic/robin/reader-writer-plugins [2] BIT-1215 [3] Bro,bro-aux Daniel Thayer Daniel Thayer 2014-07-30 2.4 Normal bro-cut should be rewritten for speed and to not depend on gawk [1] BIT-1222 https://bro-tracker.atlassian.net/browse/BIT-1222 [2] reader-writer-plugins https://github.com/bro/bro/tree/topic/robin/reader-writer-plugins [3] BIT-1215 https://bro-tracker.atlassian.net/browse/BIT-1215 From jira at bro-tracker.atlassian.net Fri Aug 1 11:11:07 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 1 Aug 2014 13:11:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1223) Merge topic/johanna/dhcp-log In-Reply-To: References: Message-ID: Johanna Amann created BIT-1223: ---------------------------------- Summary: Merge topic/johanna/dhcp-log Key: BIT-1223 URL: https://bro-tracker.atlassian.net/browse/BIT-1223 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Johanna Amann Priority: Trivial Fix For: 2.4 Please merge topic/johanna/dhcp-log. It contains a minimal change that splits dhcp record creation from log-writing. This allows users to customize dhcp.log by changing the record in their own dhcp_ack event. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Fri Aug 1 11:20:07 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 1 Aug 2014 13:20:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1223) Merge topic/johanna/dhcp-log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304#comment-17304 ] Johanna Amann commented on BIT-1223: ------------------------------------ branch also contains a small documentation fix. > Merge topic/johanna/dhcp-log > ---------------------------- > > Key: BIT-1223 > URL: https://bro-tracker.atlassian.net/browse/BIT-1223 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Priority: Trivial > Fix For: 2.4 > > > Please merge topic/johanna/dhcp-log. It contains a minimal change that splits dhcp record creation from log-writing. This allows users to customize dhcp.log by changing the record in their own dhcp_ack event. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Fri Aug 1 11:21:07 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 1 Aug 2014 13:21:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1223) Merge topic/johanna/dhcp-log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1223: ------------------------------- Status: Merge Request (was: Open) > Merge topic/johanna/dhcp-log > ---------------------------- > > Key: BIT-1223 > URL: https://bro-tracker.atlassian.net/browse/BIT-1223 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Priority: Trivial > Fix For: 2.4 > > > Please merge topic/johanna/dhcp-log. It contains a minimal change that splits dhcp record creation from log-writing. This allows users to customize dhcp.log by changing the record in their own dhcp_ack event. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Fri Aug 1 13:58:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 1 Aug 2014 15:58:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1215: --------------------------------- Assignee: Robin Sommer (was: Daniel Thayer) > bro-cut should be rewritten for speed and to not depend on gawk > --------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Assignee: Robin Sommer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Fri Aug 1 14:07:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 1 Aug 2014 16:07:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1223) Merge topic/johanna/dhcp-log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1223: --------------------------------- Assignee: Robin Sommer > Merge topic/johanna/dhcp-log > ---------------------------- > > Key: BIT-1223 > URL: https://bro-tracker.atlassian.net/browse/BIT-1223 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Priority: Trivial > Fix For: 2.4 > > > Please merge topic/johanna/dhcp-log. It contains a minimal change that splits dhcp record creation from log-writing. This allows users to customize dhcp.log by changing the record in their own dhcp_ack event. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Fri Aug 1 15:19:07 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 1 Aug 2014 17:19:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1224) Defining a record missing a colon after entry name results in a segfault In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1224: ------------------------------- Issue Type: Problem (was: Improvement) > Defining a record missing a colon after entry name results in a segfault > ------------------------------------------------------------------------ > > Key: BIT-1224 > URL: https://bro-tracker.atlassian.net/browse/BIT-1224 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Priority: Low > Fix For: 2.4 > > > The following script results in a segmentation fault when running Bro. > {code} > type test: record { > a bool; > }; > {code} > Result > {code} > error in ./test.bro, line 2: syntax error, at or near "bool" > zsh: segmentation fault bro test.bro > {code} -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Fri Aug 1 15:19:07 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Fri, 1 Aug 2014 17:19:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1224) Defining a record missing a colon after entry name results in a segfault In-Reply-To: References: Message-ID: Johanna Amann created BIT-1224: ---------------------------------- Summary: Defining a record missing a colon after entry name results in a segfault Key: BIT-1224 URL: https://bro-tracker.atlassian.net/browse/BIT-1224 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Johanna Amann Priority: Low Fix For: 2.4 The following script results in a segmentation fault when running Bro. {code} type test: record { a bool; }; {code} Result {code} error in ./test.bro, line 2: syntax error, at or near "bool" zsh: segmentation fault bro test.bro {code} -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Fri Aug 1 15:32:08 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 1 Aug 2014 17:32:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1223) Merge topic/johanna/dhcp-log In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1223: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Merge topic/johanna/dhcp-log > ---------------------------- > > Key: BIT-1223 > URL: https://bro-tracker.atlassian.net/browse/BIT-1223 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Assignee: Robin Sommer > Priority: Trivial > Fix For: 2.4 > > > Please merge topic/johanna/dhcp-log. It contains a minimal change that splits dhcp record creation from log-writing. This allows users to customize dhcp.log by changing the record in their own dhcp_ack event. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Fri Aug 1 15:32:08 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 1 Aug 2014 17:32:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1215: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > bro-cut should be rewritten for speed and to not depend on gawk > --------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Assignee: Robin Sommer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From seth at icir.org Fri Aug 1 20:58:00 2014 From: seth at icir.org (Seth Hall) Date: Fri, 1 Aug 2014 23:58:00 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Merge remote-tracking branch 'origin/topic/dnthayer/ticket1215' (25b8efe) In-Reply-To: <201408012242.s71MgtKm009075@bro-ids.icir.org> References: <201408012242.s71MgtKm009075@bro-ids.icir.org> Message-ID: On Aug 1, 2014, at 6:42 PM, Robin Sommer wrote: > Update PATH so that documentation btests can find bro-cut > Remove gawk from list of optional packages in documentation Woo! Faster bro-cut! For everyone on bro-dev that didn't know what was going on, Justin and Daniel just rewrote bro-cut as a C tool and *greatly* improved it's speed. Justin/Daniel, could one of you comment on here with a rough estimate of how much faster bro-cut is now? Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140801/f531cf89/attachment.bin From noreply at bro.org Sat Aug 2 00:00:18 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 2 Aug 2014 00:00:18 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408020700.s7270Ic1018545@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ------------------------------------- BIT-1222 [1] Bro Robin Sommer - 2014-07-31 - Normal topic/robin/reader-writer-plugins [2] [1] BIT-1222 https://bro-tracker.atlassian.net/browse/BIT-1222 [2] reader-writer-plugins https://github.com/bro/bro/tree/topic/robin/reader-writer-plugins From noreply at bro.org Sun Aug 3 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 3 Aug 2014 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408030700.s7370ELr026054@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ------------------------------------- BIT-1222 [1] Bro Robin Sommer - 2014-07-31 - Normal topic/robin/reader-writer-plugins [2] [1] BIT-1222 https://bro-tracker.atlassian.net/browse/BIT-1222 [2] reader-writer-plugins https://github.com/bro/bro/tree/topic/robin/reader-writer-plugins From dnthayer at illinois.edu Sun Aug 3 08:10:37 2014 From: dnthayer at illinois.edu (Daniel Thayer) Date: Sun, 3 Aug 2014 10:10:37 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Merge remote-tracking branch 'origin/topic/dnthayer/ticket1215' (25b8efe) In-Reply-To: References: <201408012242.s71MgtKm009075@bro-ids.icir.org> Message-ID: <53DE50ED.1040801@illinois.edu> On 08/01/2014 10:58 PM, Seth Hall wrote: > > On Aug 1, 2014, at 6:42 PM, Robin Sommer wrote: > >> Update PATH so that documentation btests can find bro-cut >> Remove gawk from list of optional packages in documentation > > Woo! Faster bro-cut! For everyone on bro-dev that didn't know what was going on, Justin and Daniel just rewrote bro-cut as a C tool and *greatly* improved it's speed. Justin/Daniel, could one of you comment on here with a rough estimate of how much faster bro-cut is now? > > Thanks! > .Seth > For me, the new bro-cut runs roughly 10x faster than the old one (the exact amount of speedup depends on which hardware and bro-cut options you're using). Another benefit of the new bro-cut is that Bro no longer depends on gawk for anything. -Daniel From jira at bro-tracker.atlassian.net Sun Aug 3 09:02:07 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Sun, 3 Aug 2014 11:02:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1224) Defining a record missing a colon after entry name results in a segfault In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1224: ------------------------------- Resolution: Cannot Reproduce Status: Closed (was: Open) ...and I was stupid and managed to use a completely broken Bro version without noticing it. > Defining a record missing a colon after entry name results in a segfault > ------------------------------------------------------------------------ > > Key: BIT-1224 > URL: https://bro-tracker.atlassian.net/browse/BIT-1224 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Johanna Amann > Priority: Low > Fix For: 2.4 > > > The following script results in a segmentation fault when running Bro. > {code} > type test: record { > a bool; > }; > {code} > Result > {code} > error in ./test.bro, line 2: syntax error, at or near "bool" > zsh: segmentation fault bro test.bro > {code} -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From noreply at bro.org Mon Aug 4 00:00:21 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 4 Aug 2014 00:00:21 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408040700.s7470LdC030800@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ------------------------------------- BIT-1222 [1] Bro Robin Sommer - 2014-07-31 - Normal topic/robin/reader-writer-plugins [2] [1] BIT-1222 https://bro-tracker.atlassian.net/browse/BIT-1222 [2] reader-writer-plugins https://github.com/bro/bro/tree/topic/robin/reader-writer-plugins From jira at bro-tracker.atlassian.net Mon Aug 4 06:49:07 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 4 Aug 2014 08:49:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1225) ReadFile API: topic/seth/readfile In-Reply-To: References: Message-ID: Seth Hall created BIT-1225: ------------------------------ Summary: ReadFile API: topic/seth/readfile Key: BIT-1225 URL: https://bro-tracker.atlassian.net/browse/BIT-1225 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Seth Hall The ReadFile module provides a simplified API for reading files off of disk. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Mon Aug 4 06:49:08 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 4 Aug 2014 08:49:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1225) ReadFile API: topic/seth/readfile In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1225: --------------------------- Status: Merge Request (was: Open) > ReadFile API: topic/seth/readfile > --------------------------------- > > Key: BIT-1225 > URL: https://bro-tracker.atlassian.net/browse/BIT-1225 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > > The ReadFile module provides a simplified API for reading files off of disk. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Mon Aug 4 08:50:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 4 Aug 2014 10:50:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek reassigned BIT-1222: ------------------------------ Assignee: Jon Siwek > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Mon Aug 4 08:51:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 4 Aug 2014 10:51:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1222: --------------------------- Fix Version/s: 2.4 > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.4 > > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Mon Aug 4 10:25:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 4 Aug 2014 12:25:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17400#comment-17400 ] Jon Siwek commented on BIT-1222: -------------------------------- For optional components like the DataSeries writer, if the plugin is not built (e.g. missing dependencies), the enum value associated w/ it's tag won't be created, but it may still be referenced from scripts that are loaded. e.g. when I compile without DataSeries, tests fail and complain about {{Log::WRITER_DATASERIES}} in {{scripts/base/frameworks/logging/writers/dataseries.bro}} being an unknown ID. > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.4 > > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Mon Aug 4 10:26:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 4 Aug 2014 12:26:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1222: --------------------------- Assignee: Robin Sommer (was: Jon Siwek) > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.4 > > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Mon Aug 4 10:40:07 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 4 Aug 2014 12:40:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17401#comment-17401 ] Seth Hall commented on BIT-1222: -------------------------------- I think that script and any tests (assuming the plugin test infrastructure is in place?) need to move into the plugin. Once functionality is removed from the Bro core, all bits and pieces of that functionality need to move as well. I'm just worried that if we don't make this a pretty hard rule that we'll slip a little bit and allow bits of plugins to remain in the Bro core. > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.4 > > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From robin at icir.org Mon Aug 4 13:48:38 2014 From: robin at icir.org (Robin Sommer) Date: Mon, 4 Aug 2014 13:48:38 -0700 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: References: Message-ID: <20140804204838.GA77517@icir.org> (Taking this to the mailing list for discussion.) On Mon, Aug 04, 2014 at 12:40 -0500, you wrote: > I think that script and any tests (assuming the plugin test > infrastructure is in place?) need to move into the plugin. Agreed in general. But there are two more general questions going in here I think: - Part of the problem is that right now, Bro's standard tree of scripts is still unchanged: while the core analyzers/readers/writers are now plugins, their corresponding scripts remains where they always were (and hence get pulled in unconditionally). Question is: do we want to change that? I'm reluctant to do that right now, as it would be major structural change, and we don't have much experience yet with the plugins' organization. I would prefer to leave the standard scripts as they are for now. - What's our strategy for moving non-standard stuff out of the main distribution? Generally, I think we should start a separate bro-plugins repository where we keep non-standard plugins (both from us, and from external folks as long as there's a clear maintainer). We could then take the stance that everything dependending on optional functionality would go there, rather than into Bro itself. Right now, I think that would mean support for DataSeries and ElasticSearch. So, in short: what would you guys think about solving the problem by moving DataSeries and ElasticSearch (including their scripts and tests) out into a new bro-plugin repository, but otherwise leaving things as they are right now? Robin From jira at bro-tracker.atlassian.net Mon Aug 4 13:57:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 4 Aug 2014 15:57:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1215) bro-cut should be rewritten for speed and to not depend on gawk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17402#comment-17402 ] Jon Siwek commented on BIT-1215: -------------------------------- Just an FYI: I've added a job to Jenkins to run the bro-aux test suite, so bro-cut is now being regression tested automatically. > bro-cut should be rewritten for speed and to not depend on gawk > --------------------------------------------------------------- > > Key: BIT-1215 > URL: https://bro-tracker.atlassian.net/browse/BIT-1215 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro, bro-aux > Reporter: Daniel Thayer > Assignee: Robin Sommer > Fix For: 2.4 > > > The current implementation of bro-cut is too slow when processing large log files (takes more than a minute to process a single log file a few hundred MB in size). Justin Azoff rewrote bro-cut in C and found that it runs an order of magnitude faster. Another benefit of a C version of bro-cut is that we will no longer depend on gawk for anything (and some of Bro's supported platforms do not include gawk by default). -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jsiwek at illinois.edu Mon Aug 4 15:03:57 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 4 Aug 2014 22:03:57 +0000 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <20140804204838.GA77517@icir.org> References: <20140804204838.GA77517@icir.org> Message-ID: <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> On Aug 4, 2014, at 3:48 PM, Robin Sommer wrote: > So, in short: what would you guys think about solving the problem by > moving DataSeries and ElasticSearch (including their scripts and > tests) out into a new bro-plugin repository, but otherwise leaving > things as they are right now? Yeah, seems like a reasonable first-step. I?m wondering if it makes sense to break them up even further in to separate repos like "dataseries-bro-plugin" and "elasticsearch-bro-plugin? ? Done like that, it may make the maintainer of the plugin more obvious and so also distinguishes ones that are externally contributed as the owner will have their own independent repo. If the aim was to have a central place for people to discover plugins, simply keeping a running list of repo URLs somewhere could also provide that. - Jon From seth at icir.org Mon Aug 4 18:15:34 2014 From: seth at icir.org (Seth Hall) Date: Mon, 4 Aug 2014 21:15:34 -0400 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <20140804204838.GA77517@icir.org> References: <20140804204838.GA77517@icir.org> Message-ID: On Aug 4, 2014, at 4:48 PM, Robin Sommer wrote: > Question is: do we want to change that? I'm reluctant to do > that right now, as it would be major structural change, and we > don't have much experience yet with the plugins' organization. > I would prefer to leave the standard scripts as they are for > now. Mmmppphhh... Not sure if I should say "let's do it!" or not. I'm *really* tempted to say that we should make the break. We're early in the 2.4 dev cycle and now's the perfect time to get that plugin organizational experience. At the very least, this would force some better practices on us and the community. Only one way to get the experience. :)  I'm actually starting to wonder now what you mean by "standard scripts"? You're obviously saying that the ES and dataseries scripts should be broken out, but could you give an example where you think we should leave it alone for now? I may be losing track of this conversation. > Generally, I think we should start a separate bro-plugins > repository where we keep non-standard plugins (both from us, > and from external folks as long as there's a clear maintainer). Agreed. Those should be easy to break out even further into separate repositories once we have an easy system for managing dependencies (i.e. a package manager). > We could then take the stance that everything dependending on > optional functionality would go there, rather than into Bro > itself. Right now, I think that would mean support for > DataSeries and ElasticSearch. And libgeoip! > So, in short: what would you guys think about solving the problem by > moving DataSeries and ElasticSearch (including their scripts and > tests) out into a new bro-plugin repository, but otherwise leaving > things as they are right now? In case it's not obvious, I'm voting for making the larger change, whatever that is. It just feels wrong to leave this code split up half way done. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140804/3eb95e1d/attachment.bin From jira at bro-tracker.atlassian.net Tue Aug 5 08:10:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 5 Aug 2014 10:10:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1226) bad example in quickstart guide In-Reply-To: References: Message-ID: Jon Siwek created BIT-1226: ------------------------------ Summary: bad example in quickstart guide Key: BIT-1226 URL: https://bro-tracker.atlassian.net/browse/BIT-1226 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.3, git/master Reporter: Jon Siwek Fix For: 2.4 The quickstart has a "deployment customization" involving watching for an SSH login to a specific set of hosts. The first problem is the code is wrong; an updated example is at https://gist.github.com/jsiwek/2a7692aa9f24e197ca9c. But there's other reasons why this example is not straightforward for new users. I think it should be replaced with a different example. Should add a unit test for it as well to make sure it doesn't become outdated. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Tue Aug 5 08:14:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 5 Aug 2014 10:14:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1222: --------------------------- Status: Open (was: Merge Request) > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.4 > > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Tue Aug 5 08:25:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 5 Aug 2014 10:25:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1225) ReadFile API: topic/seth/readfile In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek reassigned BIT-1225: ------------------------------ Assignee: Jon Siwek > ReadFile API: topic/seth/readfile > --------------------------------- > > Key: BIT-1225 > URL: https://bro-tracker.atlassian.net/browse/BIT-1225 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > Assignee: Jon Siwek > > The ReadFile module provides a simplified API for reading files off of disk. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Tue Aug 5 09:11:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 5 Aug 2014 11:11:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1225) ReadFile API: topic/seth/readfile In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1225: --------------------------- Status: Open (was: Merge Request) > ReadFile API: topic/seth/readfile > --------------------------------- > > Key: BIT-1225 > URL: https://bro-tracker.atlassian.net/browse/BIT-1225 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > Assignee: Jon Siwek > > The ReadFile module provides a simplified API for reading files off of disk. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Tue Aug 5 09:12:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 5 Aug 2014 11:12:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1225) ReadFile API: topic/seth/readfile In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1225: --------------------------- Fix Version/s: 2.4 > ReadFile API: topic/seth/readfile > --------------------------------- > > Key: BIT-1225 > URL: https://bro-tracker.atlassian.net/browse/BIT-1225 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > Assignee: Seth Hall > Fix For: 2.4 > > > The ReadFile module provides a simplified API for reading files off of disk. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Tue Aug 5 09:15:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 5 Aug 2014 11:15:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1225) ReadFile API: topic/seth/readfile In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403#comment-17403 ] Jon Siwek commented on BIT-1225: -------------------------------- In order to make it possible to detect errors reading the file, Seth said he'd reimplement this to use the Exec module internally -- return the stdout from `cat $filename` and also look for abnormal exit status to detect read errors (e.g. permissions, nonexisting). > ReadFile API: topic/seth/readfile > --------------------------------- > > Key: BIT-1225 > URL: https://bro-tracker.atlassian.net/browse/BIT-1225 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > Assignee: Seth Hall > Fix For: 2.4 > > > The ReadFile module provides a simplified API for reading files off of disk. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Tue Aug 5 09:12:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 5 Aug 2014 11:12:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1225) ReadFile API: topic/seth/readfile In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1225: --------------------------- Assignee: Seth Hall (was: Jon Siwek) > ReadFile API: topic/seth/readfile > --------------------------------- > > Key: BIT-1225 > URL: https://bro-tracker.atlassian.net/browse/BIT-1225 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > Assignee: Seth Hall > Fix For: 2.4 > > > The ReadFile module provides a simplified API for reading files off of disk. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From robin at icir.org Tue Aug 5 14:33:52 2014 From: robin at icir.org (Robin Sommer) Date: Tue, 5 Aug 2014 14:33:52 -0700 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: References: <20140804204838.GA77517@icir.org> Message-ID: <20140805213352.GL79884@icir.org> On Mon, Aug 04, 2014 at 21:15 -0400, you wrote: > Mmmppphhh... Not sure if I should say "let's do it!" or not. I'm > *really* tempted to say that we should make the break. We're early in > the 2.4 dev cycle and now's the perfect time to get that plugin > organizational experience. Well, my preference would be to first get experience with maintaining some external plugins before we take the step to reorganize all the existing stuff. That's quite a bit of work, and if we get it wrong we'll have to do it again later ... > the community. Only one way to get the experience. :)  I'm > actually starting to wonder now what you mean by "standard scripts"? > You're obviously saying that the ES and dataseries scripts should be > broken out, but could you give an example where you think we should > leave it alone for now? Take the HTTP analyzer for example. The event engine code lives in src/analyzer/protocol/http/. If this were an external plugin, it would have its corresponding Bro scripts in src/analyzer/protocol/http/. So I guess that means we would now move scripts/{base,policy}/protocols/http in there. But I'm not sure it's always clear-cut where existign scripts would move; and what about those that don't have a corresponding src/* part? I think the answer is that those would move into script-only plugins, which in principle should already be supported as well; but where do they live? Maybe we want to move all the plugins out of src/ anyways? And how does this all play along with the envisioned CBAN (or whatever we call it these days)? So, I'm with you that should figure this all out, but I would prefer to do that as a separate step that for now leaves the existing structure in place until we know these answers. > And libgeoip! Good point, although this would hurt a bit more: geoip is optional but still pretty standard functionality that we would now require people to install separately ... So while I agree that this should move into its own plugin as well, maybe that's also somethign for later (generally, most of the bifs should move to plugins as well; we should reorg them broadly by functionality). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From robin at icir.org Tue Aug 5 14:41:53 2014 From: robin at icir.org (Robin Sommer) Date: Tue, 5 Aug 2014 14:41:53 -0700 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> Message-ID: <20140805214153.GM79884@icir.org> On Mon, Aug 04, 2014 at 22:03 +0000, you wrote: > Yeah, seems like a reasonable first-step. I?m wondering if it makes > sense to break them up even further in to separate repos like > "dataseries-bro-plugin" and "elasticsearch-bro-plugin? ? Yeah, I'm torn on this. It does make sense for the reasons you give, but one repository also has its appeal: - it's easy to just get them all by cloning, or packaging, the one thing. - administratively, we just need to manage/mirror one repo. - we can add some infrastructure to the repo to easily build and test them all at once, including as part of Jenkins. - we can market that one repo as a vetted source for plugins, including plugins maintained externally that follow certain standards, like having a maintainer who fixes problems and makes sure it works with the current release (we'd ping that person when something breaks and remove the plugin if there's no fix). [1] - independent of what we do, people can of course still have their own repos elsewhere anyways. Opinions? Robin [1] That said, maybe even that is already more effort than we really want to invest into external code? -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From jsiwek at illinois.edu Tue Aug 5 17:04:20 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 6 Aug 2014 00:04:20 +0000 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <20140805214153.GM79884@icir.org> References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> <20140805214153.GM79884@icir.org> Message-ID: <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> On Aug 5, 2014, at 4:41 PM, Robin Sommer wrote: > On Mon, Aug 04, 2014 at 22:03 +0000, you wrote: > >> Yeah, seems like a reasonable first-step. I?m wondering if it makes >> sense to break them up even further in to separate repos like >> "dataseries-bro-plugin" and "elasticsearch-bro-plugin? ? > > Yeah, I'm torn on this. It does make sense for the reasons you give, > but one repository also has its appeal: > > - it's easy to just get them all by cloning, or packaging, the one > thing. > > - administratively, we just need to manage/mirror one repo. > > - we can add some infrastructure to the repo to easily build and > test them all at once, including as part of Jenkins. > > - we can market that one repo as a vetted source for plugins, > including plugins maintained externally that follow certain > standards, like having a maintainer who fixes problems and makes > sure it works with the current release (we'd ping that person > when something breaks and remove the plugin if there's no fix). > [1] > > - independent of what we do, people can of course still have their > own repos elsewhere anyways. > > Opinions? Maybe still have one repo that relies on git submodules, one per plugin? - Easy to clone everything w/ ?recursive. - Could hold common packaging and testing infrastructure. - Still has administrative overhead of having to create/mirror many repos. Was there more concern regarding admin overhead other than the initial cost of setting-up/mirroring? Is there a limit to how many repos can be mirrored? Could also do a hybrid approach where only external plugins are submodules, but internally maintained ones just get committed directly to main "bro-plugin? repo. That would cut down on the admin overhead. Another worry: should the way plugins are organized make it easy to be selective about which plugins to build/install ? Say that I just want the dataseries plugin, but also happen satisfy dependencies of the elasticsearch plugin, would it be more ?awkward" for me if all plugins live in same repo and share build infrastructure? ?Awkward? meaning it?s going to download/build/install things I don?t want. - Jon From robin at icir.org Wed Aug 6 08:00:38 2014 From: robin at icir.org (Robin Sommer) Date: Wed, 6 Aug 2014 08:00:38 -0700 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> <20140805214153.GM79884@icir.org> <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> Message-ID: <20140806150038.GV79884@icir.org> On Wed, Aug 06, 2014 at 00:04 +0000, you wrote: > Was there more concern regarding admin overhead other than the initial > cost of setting-up/mirroring? Is there a limit to how many repos can > be mirrored? No, it's indeed mostly the setup, plus the potential messiness of having 1000s of individial repositories show up when somebody goes to github (to be a bit overly optimistic :-) > Could also do a hybrid approach where only external plugins are > submodules, but internally maintained ones just get committed directly > to main "bro-plugin? repo. That would cut down on the admin overhead. I like this model. Keeping our own stuff in a single repo feels more convinient to me, but I can see pulling in external ones separately. And we'd still have "vetting power" becaue we have to move the submodule forward. > Another worry: should the way plugins are organized make it easy to be > selective about which plugins to build/install ? Yes, actually I think so. Building/testing all together is convenient, but I'm not sure it should be the default (*installing* all together should pretty certainly not be the default). So I'm thinking to do it selectively, with the option to do them all. Maybe that's just a top-level Makefile with individual targets {build,install,test}-dataseries, {build,install}-elasticsearch, and then an overall {build,install,test}-all. In addition one can also always go to the subdirectories individually and build there, they are all supposed to build standalone as well. So do we want to go ahead with this model? Then I'd set that up and move DS and ES over. One more question: would we then want to pull bro-plugin as a submodule into bro/aux? Then people would automatically get the stuff there as well (but it wouldn't be build automatically), and it would allow for easier testing as it's clear where things are located. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From jsiwek at illinois.edu Wed Aug 6 08:47:17 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 6 Aug 2014 15:47:17 +0000 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <20140806150038.GV79884@icir.org> References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> <20140805214153.GM79884@icir.org> <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> <20140806150038.GV79884@icir.org> Message-ID: <84B38D4E-C33E-4749-85F4-04B2B9D1BB9C@illinois.edu> On Aug 6, 2014, at 10:00 AM, Robin Sommer wrote: > So do we want to go ahead with this model? Sure. I?m thinking if a problem is found, it?s not hard to convert to one of the other models since they should share the same directory structure and only differ in which dirs are chosen to be git submodules. > One more question: would we then want to pull bro-plugin as a > submodule into bro/aux? I think that would be ok unless you?re worried about keeping the size/overhead of cloning bro down. - Jon From robin at icir.org Wed Aug 6 08:51:51 2014 From: robin at icir.org (Robin Sommer) Date: Wed, 6 Aug 2014 08:51:51 -0700 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <84B38D4E-C33E-4749-85F4-04B2B9D1BB9C@illinois.edu> References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> <20140805214153.GM79884@icir.org> <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> <20140806150038.GV79884@icir.org> <84B38D4E-C33E-4749-85F4-04B2B9D1BB9C@illinois.edu> Message-ID: <20140806155151.GZ79884@icir.org> On Wed, Aug 06, 2014 at 15:47 +0000, you wrote: > I think that would be ok unless you?re worried about keeping the size/overhead of cloning bro down. No, that's fine I think. I don't have particular worries, most of my questions are just fishing for opinions on how we structure this best. :-) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From seth at icir.org Wed Aug 6 09:26:06 2014 From: seth at icir.org (Seth Hall) Date: Wed, 6 Aug 2014 12:26:06 -0400 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <20140805213352.GL79884@icir.org> References: <20140804204838.GA77517@icir.org> <20140805213352.GL79884@icir.org> Message-ID: On Aug 5, 2014, at 5:33 PM, Robin Sommer wrote: > So, I'm with you that should figure this all out, but I would prefer > to do that as a separate step that for now leaves the existing > structure in place until we know these answers. I think you gave a good justification. Personally I am still a bit mixed on if we should even include the scripts that we put in base/protocols/ in with the analyzers. I can actually see real justification for not including them. >> And libgeoip! > > Good point, although this would hurt a bit more: geoip is optional but > still pretty standard functionality that we would now require people > to install separately ... So while I agree that this should move into > its own plugin as well, maybe that's also somethign for later Fair enough. Eventually when this reorganization happens in earnest I think we'll have to use the rule of thumb we discussed yesterday. Always seek to shorten init-bare.bro. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140806/225ad0b0/attachment-0001.bin From seth at icir.org Wed Aug 6 09:33:14 2014 From: seth at icir.org (Seth Hall) Date: Wed, 6 Aug 2014 12:33:14 -0400 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <20140806150038.GV79884@icir.org> References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> <20140805214153.GM79884@icir.org> <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> <20140806150038.GV79884@icir.org> Message-ID: On Aug 6, 2014, at 11:00 AM, Robin Sommer wrote: > One more question: would we then want to pull bro-plugin as a > submodule into bro/aux? Then people would automatically get the stuff > there as well (but it wouldn't be build automatically), and it would > allow for easier testing as it's clear where things are located.  I don't like that stuff might not automatically build. Would it be possible to have the plugins add stuff to bro's configure output? So that plugins that are available and able to find their dependencies automatically build? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140806/731211cd/attachment.bin From robin at icir.org Wed Aug 6 12:14:56 2014 From: robin at icir.org (Robin Sommer) Date: Wed, 6 Aug 2014 12:14:56 -0700 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: References: <20140804204838.GA77517@icir.org> <20140805213352.GL79884@icir.org> Message-ID: <20140806191456.GB79884@icir.org> On Wed, Aug 06, 2014 at 12:26 -0400, you wrote: > I think you gave a good justification. Personally I am still a bit > mixed on if we should even include the scripts that we put in > base/protocols/ in with the analyzers. Yeah, that's yet another decision to make ... Undecided. :) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From robin at icir.org Wed Aug 6 12:17:50 2014 From: robin at icir.org (Robin Sommer) Date: Wed, 6 Aug 2014 12:17:50 -0700 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> <20140805214153.GM79884@icir.org> <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> <20140806150038.GV79884@icir.org> Message-ID: <20140806191750.GC79884@icir.org> On Wed, Aug 06, 2014 at 12:33 -0400, you wrote: > I don't like that stuff might not automatically build. Would it be > possible to have the plugins add stuff to bro's configure output? So > that plugins that are available and able to find their dependencies > automatically build? Hmmm ... Not sure I like that. To me these separately maintained plugins are optional things that shouldn't be pulled in automatically. Would you say they should also install automatically? What about those that don't even have external dependencies? Would they always be installed/loaded? Also, if we integrated them into the central configure, we'd probably also need to provide their options, like --with-dataseries=/path/to/ds ... Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From seth at icir.org Wed Aug 6 12:23:03 2014 From: seth at icir.org (Seth Hall) Date: Wed, 6 Aug 2014 15:23:03 -0400 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <20140806191750.GC79884@icir.org> References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> <20140805214153.GM79884@icir.org> <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> <20140806150038.GV79884@icir.org> <20140806191750.GC79884@icir.org> Message-ID: On Aug 6, 2014, at 3:17 PM, Robin Sommer wrote: > Also, if we integrated them into the central configure, we'd probably > also need to provide their options, like --with-dataseries=/path/to/ds ... Yep. I feel more and more like we're going to need the package management system soon after (or in conjunction?) with this. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140806/c650c140/attachment.bin From robin at icir.org Wed Aug 6 12:37:20 2014 From: robin at icir.org (Robin Sommer) Date: Wed, 6 Aug 2014 12:37:20 -0700 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> <20140805214153.GM79884@icir.org> <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> <20140806150038.GV79884@icir.org> <20140806191750.GC79884@icir.org> Message-ID: <20140806193720.GF79884@icir.org> On Wed, Aug 06, 2014 at 15:23 -0400, you wrote: > Yep. I feel more and more like we're going to need the package > management system soon after (or in conjunction?) with this. We'll need it eventually, but I wouldn't like to wait for it. Do you think we'd break much by moving DS and ES into bro-plugins/, without building them automatically as well? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From seth at icir.org Wed Aug 6 12:39:49 2014 From: seth at icir.org (Seth Hall) Date: Wed, 6 Aug 2014 15:39:49 -0400 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <20140806193720.GF79884@icir.org> References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> <20140805214153.GM79884@icir.org> <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> <20140806150038.GV79884@icir.org> <20140806191750.GC79884@icir.org> <20140806193720.GF79884@icir.org> Message-ID: <0BDB1487-D555-4D87-8F4A-6682D3C7B4D5@icir.org> On Aug 6, 2014, at 3:37 PM, Robin Sommer wrote: > We'll need it eventually, but I wouldn't like to wait for it. Do you > think we'd break much by moving DS and ES into bro-plugins/, without > building them automatically as well? I'm not very fond of the ES support being even harder to get working. I'm still very hopeful that we'll get mechanisms and documentation in place that make the ES stuff much easier. Ultimately it's probably a small concern. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140806/e3134a8b/attachment.bin From robin at icir.org Wed Aug 6 12:50:22 2014 From: robin at icir.org (Robin Sommer) Date: Wed, 6 Aug 2014 12:50:22 -0700 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <0BDB1487-D555-4D87-8F4A-6682D3C7B4D5@icir.org> References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> <20140805214153.GM79884@icir.org> <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> <20140806150038.GV79884@icir.org> <20140806191750.GC79884@icir.org> <20140806193720.GF79884@icir.org> <0BDB1487-D555-4D87-8F4A-6682D3C7B4D5@icir.org> Message-ID: <20140806195022.GH79884@icir.org> On Wed, Aug 06, 2014 at 15:39 -0400, you wrote: > I'm not very fond of the ES support being even harder to get working. Any idea how widely that's used currently? It wouldn't be harder, but a bit. I think the alternatives right now are: (1) holding off completely on moving readers/writers to plugins, or (2) getting some hack in place so that we avoid the problem that Jon noticed with readers/writers being used in script-land that aren't available (could probably done with some ugly ifdefs). I could see (2) if we had to. I would prefer to avoid (1). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From seth at icir.org Wed Aug 6 13:03:56 2014 From: seth at icir.org (Seth Hall) Date: Wed, 6 Aug 2014 16:03:56 -0400 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <20140806195022.GH79884@icir.org> References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> <20140805214153.GM79884@icir.org> <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> <20140806150038.GV79884@icir.org> <20140806191750.GC79884@icir.org> <20140806193720.GF79884@icir.org> <0BDB1487-D555-4D87-8F4A-6682D3C7B4D5@icir.org> <20140806195022.GH79884@icir.org> Message-ID: On Aug 6, 2014, at 3:50 PM, Robin Sommer wrote: > Any idea how widely that's used currently? It wouldn't be harder, but > a bit. Not very widely. It still has issues we're looking to correct too. I think we can skip 1 and 2. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140806/237ab8e0/attachment.bin From gc355804 at ohio.edu Wed Aug 6 14:38:59 2014 From: gc355804 at ohio.edu (Clark, Gilbert) Date: Wed, 6 Aug 2014 21:38:59 +0000 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <20140806191750.GC79884@icir.org> References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> <20140805214153.GM79884@icir.org> <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> <20140806150038.GV79884@icir.org> , <20140806191750.GC79884@icir.org> Message-ID: <1407361138164.6307@ohio.edu> Are there any plans for packaging plugins and pushing those into various distributions' repositories (e.g. CentOS, Debian, FreeBSD)? 'sudo yum install bro-2.4-elasticsearch-writer' seems like it would be pretty convenient for users, assuming there are plans to support it. On a related note, it seems like individual maintainers could acquire blessed status pretty quickly without getting the bro team involved by pushing their individual plugin upstream somewhere: anything that 'yum search bro-plugin' (or equivalent) yields would probably be assumed to be somewhat stable (or at least stable enough to install without thinking about it too hard first :) Are there plans to package a bro-2.4-plugin-devel or equivalent to make it possible for the folks who have installed bro via e.g. apt or yum to build plugins without having to also pull down and compile a complete version of bro? I think this could make plugin development quite a bit more accessible for new folks, assuming the overhead of maintaining such a package wasn't unreasonable. -Gilbert ________________________________________ From: bro-dev-bounces at bro.org on behalf of Robin Sommer Sent: Wednesday, August 06, 2014 3:17 PM To: Seth Hall Cc: bro-dev at bro.org Subject: Re: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) On Wed, Aug 06, 2014 at 12:33 -0400, you wrote: > I don't like that stuff might not automatically build. Would it be > possible to have the plugins add stuff to bro's configure output? So > that plugins that are available and able to find their dependencies > automatically build? Hmmm ... Not sure I like that. To me these separately maintained plugins are optional things that shouldn't be pulled in automatically. Would you say they should also install automatically? What about those that don't even have external dependencies? Would they always be installed/loaded? Also, if we integrated them into the central configure, we'd probably also need to provide their options, like --with-dataseries=/path/to/ds ... Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin _______________________________________________ bro-dev mailing list bro-dev at bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From robin at icir.org Thu Aug 7 09:14:34 2014 From: robin at icir.org (Robin Sommer) Date: Thu, 7 Aug 2014 09:14:34 -0700 Subject: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) In-Reply-To: <1407361138164.6307@ohio.edu> References: <20140804204838.GA77517@icir.org> <7676EB4C-1555-4594-9340-B58C00B0C1AC@illinois.edu> <20140805214153.GM79884@icir.org> <2015FBD8-BD25-42A8-9460-AF6578AA5935@illinois.edu> <20140806150038.GV79884@icir.org> <20140806191750.GC79884@icir.org> <1407361138164.6307@ohio.edu> Message-ID: <20140807161434.GZ79884@icir.org> All good questions. We still have a lot figure out with all of this, would certainly be nice to be able to do "yum install bro-my-plugin". Right now the only built-in way to distribute plugins as binaries is "make bdist" with provided skeleton Makefile: that builds tarball with everything that the plugin needs at runtime. I also realized another problem (maybe) with the current skeleton for writing plugins: it doesn't keep things nicely inside a single build directory, as we usually do with all cmake-built stuff (there's a reason for why not, but a few sylinks could probably solve that). Anyways, I would still like to go ahead with things just as they are now (assuming we manage to not break anything/much). Nothing's cast in stone, but I think it will work better to work this out once things are in master so that people start using it. Robin On Wed, Aug 06, 2014 at 21:38 +0000, you wrote: > Are there any plans for packaging plugins and pushing those into > various distributions' repositories (e.g. CentOS, Debian, FreeBSD)? > 'sudo yum install bro-2.4-elasticsearch-writer' seems like it would be > pretty convenient for users, assuming there are plans to support it. > On a related note, it seems like individual maintainers could acquire > blessed status pretty quickly without getting the bro team involved by > pushing their individual plugin upstream somewhere: anything that 'yum > search bro-plugin' (or equivalent) yields would probably be assumed to > be somewhat stable (or at least stable enough to install without > thinking about it too hard first :) > > Are there plans to package a bro-2.4-plugin-devel or equivalent to > make it possible for the folks who have installed bro via e.g. apt or > yum to build plugins without having to also pull down and compile a > complete version of bro? I think this could make plugin development > quite a bit more accessible for new folks, assuming the overhead of > maintaining such a package wasn't unreasonable. > > -Gilbert > ________________________________________ > From: bro-dev-bounces at bro.org on behalf of Robin Sommer > Sent: Wednesday, August 06, 2014 3:17 PM > To: Seth Hall > Cc: bro-dev at bro.org > Subject: Re: [Bro-Dev] Organizing plugins (Re: [JIRA] (BIT-1222) topic/robin/reader-writer-plugins) > > On Wed, Aug 06, 2014 at 12:33 -0400, you wrote: > > > I don't like that stuff might not automatically build. Would it be > > possible to have the plugins add stuff to bro's configure output? So > > that plugins that are available and able to find their dependencies > > automatically build? > > Hmmm ... Not sure I like that. To me these separately maintained > plugins are optional things that shouldn't be pulled in automatically. > Would you say they should also install automatically? What about those > that don't even have external dependencies? Would they always be > installed/loaded? > > Also, if we integrated them into the central configure, we'd probably > also need to provide their options, like --with-dataseries=/path/to/ds ... > > Robin > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From jira at bro-tracker.atlassian.net Thu Aug 7 09:51:07 2014 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Thu, 7 Aug 2014 11:51:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1227) netstats should compute statistics In-Reply-To: References: Message-ID: Justin Azoff created BIT-1227: --------------------------------- Summary: netstats should compute statistics Key: BIT-1227 URL: https://bro-tracker.atlassian.net/browse/BIT-1227 Project: Bro Issue Tracker Issue Type: New Feature Components: BroControl Reporter: Justin Azoff Assignee: Justin Azoff Priority: Trivial Someone on irc had shared a really hackish script that parsed the output of broctl netstats to add drop percentages and totals. This is trivial to do inside of broctl. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 7 09:57:07 2014 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Thu, 7 Aug 2014 11:57:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1228) broctl needs to keep track of desired state In-Reply-To: References: Message-ID: Justin Azoff created BIT-1228: --------------------------------- Summary: broctl needs to keep track of desired state Key: BIT-1228 URL: https://bro-tracker.atlassian.net/browse/BIT-1228 Project: Bro Issue Tracker Issue Type: Problem Components: BroControl Reporter: Justin Azoff Assignee: Justin Azoff Priority: Low On a multi-node cluster the following sequence of events can happen: * A cluster node (node-2) has a power problem and is shut down * broctl stop is ran on the manager * broctl then fails to stop bro on node-2 * node-2 reboots * broctl cron restarts bro on node-2 because the last known state is up The problem can happen in reverse as well, where broctl will not restart bro on a node that was down. The problem arises because broctl stores the actual state of the nodes, but not the desired state. commands like stop and start need to set the desired start first, and then attempt to sync reality with that state information. broctl cron then just needs to attempt the similar sync. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 7 10:19:07 2014 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Thu, 7 Aug 2014 12:19:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1229) loading a non-existant enum from an input file terminates bro In-Reply-To: References: Message-ID: Justin Azoff created BIT-1229: --------------------------------- Summary: loading a non-existant enum from an input file terminates bro Key: BIT-1229 URL: https://bro-tracker.atlassian.net/browse/BIT-1229 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Reporter: Justin Azoff Attachments: ignored_notices.csv, ignore-notices.bro If you have an input file with an enum in it and it does not exist, bro terminates: internal error: Value not found in enum mappimg. Module: NoSuch, var: Notice, var size: 6 -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 7 10:21:07 2014 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Thu, 7 Aug 2014 12:21:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1230) Input::REREAD should retry after errors In-Reply-To: References: Message-ID: Justin Azoff created BIT-1230: --------------------------------- Summary: Input::REREAD should retry after errors Key: BIT-1230 URL: https://bro-tracker.atlassian.net/browse/BIT-1230 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Reporter: Justin Azoff Priority: Low Right now if you are trying to input a file and there is an error, the input thread exits: {code} 0.000000 Reporter::ERROR ignored_notices.csv/Input::READER_ASCII: could not read first line (empty) 0.000000 Reporter::ERROR ignored_notices.csv/Input::READER_ASCII: Init: cannot open ignored_notices.csv; headers are incorrect (empty) 0.000000 Reporter::ERROR ignored_notices.csv/Input::READER_ASCII: Init failed (empty) 0.000000 Reporter::ERROR ignored_notices.csv/Input::READER_ASCII: terminating thread (empty) {code} This is counter intuitive and causes some operational issues. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 7 10:29:07 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 7 Aug 2014 12:29:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1227) netstats should compute statistics In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1227: --------------------------- Attachment: signature.asc Would it make more sense to just write a Bro script that does it directly? I can't remember the exact mechanism that broctl netstats uses to collect that information. We could just write it out as a log if the data is ultimately coming from Bro. > netstats should compute statistics > ---------------------------------- > > Key: BIT-1227 > URL: https://bro-tracker.atlassian.net/browse/BIT-1227 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BroControl > Reporter: Justin Azoff > Assignee: Justin Azoff > Priority: Trivial > Attachments: signature.asc > > > Someone on irc had shared a really hackish script that parsed the output of broctl netstats to add drop percentages and totals. This is trivial to do inside of broctl. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 7 10:33:07 2014 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Thu, 7 Aug 2014 12:33:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1227) netstats should compute statistics In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17405#comment-17405 ] Justin Azoff commented on BIT-1227: ----------------------------------- That does sound like it makes more sense. I'm not sure what people are doing with the netstats output. > netstats should compute statistics > ---------------------------------- > > Key: BIT-1227 > URL: https://bro-tracker.atlassian.net/browse/BIT-1227 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: BroControl > Reporter: Justin Azoff > Assignee: Justin Azoff > Priority: Trivial > Attachments: signature.asc > > > Someone on irc had shared a really hackish script that parsed the output of broctl netstats to add drop percentages and totals. This is trivial to do inside of broctl. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 7 10:40:07 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 7 Aug 2014 12:40:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1230) Input::REREAD should retry after errors In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1230: ------------------------------- Resolution: Duplicate Status: Closed (was: Open) This is (basically) a duplicate of BIT-1180. I know, I should try to fix that soon... > Input::REREAD should retry after errors > --------------------------------------- > > Key: BIT-1230 > URL: https://bro-tracker.atlassian.net/browse/BIT-1230 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Justin Azoff > Priority: Low > > Right now if you are trying to input a file and there is an error, the input thread exits: > {code} > 0.000000 Reporter::ERROR ignored_notices.csv/Input::READER_ASCII: could not read first line (empty) > 0.000000 Reporter::ERROR ignored_notices.csv/Input::READER_ASCII: Init: cannot open ignored_notices.csv; headers are incorrect (empty) > 0.000000 Reporter::ERROR ignored_notices.csv/Input::READER_ASCII: Init failed (empty) > 0.000000 Reporter::ERROR ignored_notices.csv/Input::READER_ASCII: terminating thread (empty) > {code} > This is counter intuitive and causes some operational issues. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 7 10:42:07 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 7 Aug 2014 12:42:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1229) loading a non-existant enum from an input file terminates bro In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann reassigned BIT-1229: ---------------------------------- Assignee: Johanna Amann > loading a non-existant enum from an input file terminates bro > ------------------------------------------------------------- > > Key: BIT-1229 > URL: https://bro-tracker.atlassian.net/browse/BIT-1229 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Justin Azoff > Assignee: Johanna Amann > Fix For: 2.4 > > Attachments: ignored_notices.csv, ignore-notices.bro > > > If you have an input file with an enum in it and it does not exist, bro terminates: > internal error: Value not found in enum mappimg. Module: NoSuch, var: Notice, var size: 6 -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 7 10:42:08 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 7 Aug 2014 12:42:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1230) Input::REREAD should retry after errors In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1230: --------------------------- Attachment: signature.asc Johanna, can you link this ticket to your other ticket where you filed basically this same error reporting issue with the input framework and close this ticket as a dup? Thanks! > Input::REREAD should retry after errors > --------------------------------------- > > Key: BIT-1230 > URL: https://bro-tracker.atlassian.net/browse/BIT-1230 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Justin Azoff > Priority: Low > Attachments: signature.asc > > > Right now if you are trying to input a file and there is an error, the input thread exits: > {code} > 0.000000 Reporter::ERROR ignored_notices.csv/Input::READER_ASCII: could not read first line (empty) > 0.000000 Reporter::ERROR ignored_notices.csv/Input::READER_ASCII: Init: cannot open ignored_notices.csv; headers are incorrect (empty) > 0.000000 Reporter::ERROR ignored_notices.csv/Input::READER_ASCII: Init failed (empty) > 0.000000 Reporter::ERROR ignored_notices.csv/Input::READER_ASCII: terminating thread (empty) > {code} > This is counter intuitive and causes some operational issues. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 7 10:42:07 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 7 Aug 2014 12:42:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1229) loading a non-existant enum from an input file terminates bro In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1229: ------------------------------- Fix Version/s: 2.4 > loading a non-existant enum from an input file terminates bro > ------------------------------------------------------------- > > Key: BIT-1229 > URL: https://bro-tracker.atlassian.net/browse/BIT-1229 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Justin Azoff > Assignee: Johanna Amann > Fix For: 2.4 > > Attachments: ignored_notices.csv, ignore-notices.bro > > > If you have an input file with an enum in it and it does not exist, bro terminates: > internal error: Value not found in enum mappimg. Module: NoSuch, var: Notice, var size: 6 -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 7 10:42:08 2014 From: jira at bro-tracker.atlassian.net (Johanna Amann (JIRA)) Date: Thu, 7 Aug 2014 12:42:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1229) loading a non-existant enum from an input file terminates bro In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-1229: ------------------------------- Affects Version/s: git/master > loading a non-existant enum from an input file terminates bro > ------------------------------------------------------------- > > Key: BIT-1229 > URL: https://bro-tracker.atlassian.net/browse/BIT-1229 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Justin Azoff > Assignee: Johanna Amann > Fix For: 2.4 > > Attachments: ignored_notices.csv, ignore-notices.bro > > > If you have an input file with an enum in it and it does not exist, bro terminates: > internal error: Value not found in enum mappimg. Module: NoSuch, var: Notice, var size: 6 -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 7 15:32:07 2014 From: jira at bro-tracker.atlassian.net (Justin Azoff (JIRA)) Date: Thu, 7 Aug 2014 17:32:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1180) Input framework subsiquient REREAD fails after file update In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17408#comment-17408 ] Justin Azoff commented on BIT-1180: ----------------------------------- One potential cause of this issue is not atomically changing the input file. Basically, if you are doing something like this: {code} fetch_data > file.csv {code} You should instead do: {code} fetch_data > file.csv.new && mv file.csv.new file.csv {code} > Input framework subsiquient REREAD fails after file update > ----------------------------------------------------------- > > Key: BIT-1180 > URL: https://bro-tracker.atlassian.net/browse/BIT-1180 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Aashish Sharma > Assignee: Johanna Amann > Priority: High > Labels: input-framework > Fix For: 2.4 > > > I have a file that gets updated every hour and I am using it as a feed into bro using input framework. Every hour I write a list of IP addresses into this file. For many updates everything works fine but Occasionally, I see the following error: > Apr 6 05:00:09 Reporter::ERROR /feeds/Blacklist/CURRENT.24hrs_BRO/Input::READER_ASCII: could not read first line (empty) > After this failure/message, any subsequent updates on the file are ignored by the input framework. > From visual inspection the file looks just fine and header/data (1 column of IP addresses) is there as expected but somehow input framework doesn't like it. It seems that every hour when update the file using a cron script, on a rare occasion the file is empty for a minuscule duration after which this error starts. > for further REREADS data won't get updated into the tables anymore once the above Reporter::ERROR kicks in. > Please let me know if you need ways to reproduce this error condition or have more questions for me. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 7 19:48:07 2014 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 7 Aug 2014 21:48:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1181) Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409#comment-17409 ] Seth Hall commented on BIT-1181: -------------------------------- I actually don't think we should use another global event here or do this through the reporter. I think we should either be able to register an event ($ev?) or a callback ($fn?) if there was a failure. > Input-framework errors should be fatal (or Notice_Alarm) instead of silent reporter::error failures > --------------------------------------------------------------------------------------------------- > > Key: BIT-1181 > URL: https://bro-tracker.atlassian.net/browse/BIT-1181 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.2 > Reporter: Aashish Sharma > Assignee: Johanna Amann > Labels: input-framework > > I noticed many times that if there is a problem in a feed file (syntax, or some other issue) and input-framework is unable to read the file, it generates a Reporter::Error. This is a silent failure condition ie bro continues to operate as normal and the error is logged into reporter log. > Ideally above is the right thing to do. However, This failure results in no data in the tables getting updated any more while I continue to operate under-impression that Bro is working fine (unless I have explicitly been looking at reporter log for this issue , which now I do). > If input-framework is unable to read/digest data from a feed, I believe that should be a (configurable) fatal error or something which at least triggers an alarm/alert/email. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From noreply at bro.org Fri Aug 8 00:00:20 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 8 Aug 2014 00:00:20 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408080700.s7870KY0000902@bro-ids.icir.org> Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- --------- ---------- ---------------------------- #10 [1] bro hosom [2] 2014-08-07 Log urls sent over SMTP. [3] [1] Pull Request #10 https://github.com/bro/bro/pull/10 [2] hosom https://github.com/hosom [3] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From noreply at bro.org Sat Aug 9 00:00:18 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 9 Aug 2014 00:00:18 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408090700.s7970IVl007769@bro-ids.icir.org> Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- --------- ---------- ---------------------------- #10 [1] bro hosom [2] 2014-08-07 Log urls sent over SMTP. [3] [1] Pull Request #10 https://github.com/bro/bro/pull/10 [2] hosom https://github.com/hosom [3] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From noreply at bro.org Sun Aug 10 00:00:20 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 10 Aug 2014 00:00:20 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408100700.s7A70Kon011274@bro-ids.icir.org> Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- --------- ---------- ---------------------------- #10 [1] bro hosom [2] 2014-08-07 Log urls sent over SMTP. [3] [1] Pull Request #10 https://github.com/bro/bro/pull/10 [2] hosom https://github.com/hosom [3] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From noreply at bro.org Mon Aug 11 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 11 Aug 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408110700.s7B70GcE016040@bro-ids.icir.org> Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- --------- ---------- ---------------------------- #10 [1] bro hosom [2] 2014-08-07 Log urls sent over SMTP. [3] [1] Pull Request #10 https://github.com/bro/bro/pull/10 [2] hosom https://github.com/hosom [3] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From noreply at bro.org Tue Aug 12 00:00:18 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 12 Aug 2014 00:00:18 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408120700.s7C70IOK000326@bro-ids.icir.org> Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- --------- ---------- ---------------------------- #10 [1] bro hosom [2] 2014-08-07 Log urls sent over SMTP. [3] [1] Pull Request #10 https://github.com/bro/bro/pull/10 [2] hosom https://github.com/hosom [3] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From noreply at bro.org Wed Aug 13 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 13 Aug 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408130700.s7D70G95008807@bro-ids.icir.org> Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- --------- ---------- ---------------------------- #10 [1] bro hosom [2] 2014-08-07 Log urls sent over SMTP. [3] [1] Pull Request #10 https://github.com/bro/bro/pull/10 [2] hosom https://github.com/hosom [3] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From noreply at bro.org Thu Aug 14 00:00:27 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 14 Aug 2014 00:00:27 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408140700.s7E70RAH009975@bro-ids.icir.org> Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- --------- ---------- ---------------------------- #10 [1] bro hosom [2] 2014-08-07 Log urls sent over SMTP. [3] [1] Pull Request #10 https://github.com/bro/bro/pull/10 [2] hosom https://github.com/hosom [3] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From jira at bro-tracker.atlassian.net Thu Aug 14 13:47:07 2014 From: jira at bro-tracker.atlassian.net (hui (JIRA)) Date: Thu, 14 Aug 2014 15:47:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: hui created BIT-1231: ------------------------ Summary: DNP3 Analyzer Supports for DNP3-over-UDP Key: BIT-1231 URL: https://bro-tracker.atlassian.net/browse/BIT-1231 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: 2.3 Reporter: hui Two major changes are made for the DNP3 analyzer 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. The changes are made in DNP3.cc, DNP3.h and dpd.sig 2. Fix a bug in the binpac codes of the DNP3 analyzer The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 14 15:53:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 14 Aug 2014 17:53:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17500#comment-17500 ] Robin Sommer commented on BIT-1222: ----------------------------------- Next attempt. Branch topic/robin/reader-writer-plugins in bro, bro-aux, and cmake; plus a new repository bro-plugins (master branch there). Per discussion on the mailing list, I've moved DS and ES into their own repository. I've also committed some further tweaks to plugin skeleton setup, including a simple configure template that makes it easier to pass further options (like for DS, where the DS libraries are installed). > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.4 > > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Thu Aug 14 15:53:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 14 Aug 2014 17:53:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1222: ------------------------------ Status: Merge Request (was: Open) > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Robin Sommer > Fix For: 2.4 > > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From robin at icir.org Thu Aug 14 20:01:24 2014 From: robin at icir.org (Robin Sommer) Date: Thu, 14 Aug 2014 20:01:24 -0700 Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: <20140815030124.GC50325@icir.org> Is this ready already, and if so, what's the branch to merge? On Thu, Aug 14, 2014 at 15:47 -0500, you wrote: > hui created BIT-1231: > ------------------------ > > Summary: DNP3 Analyzer Supports for DNP3-over-UDP > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > > > > > > -- > This message was sent by Atlassian JIRA > (v6.4-OD-02-003#64000) > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From jira at bro-tracker.atlassian.net Thu Aug 14 20:02:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 14 Aug 2014 22:02:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17501#comment-17501 ] Robin Sommer commented on BIT-1231: ----------------------------------- Is this ready already, and if so, what's the branch to merge? > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From noreply at bro.org Fri Aug 15 00:00:22 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 15 Aug 2014 00:00:22 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408150700.s7F70MXB021709@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ------------ ---------- ------------- ---------- ------------------------------------- BIT-1222 [1] Bro Robin Sommer Robin Sommer 2014-08-14 2.4 Normal topic/robin/reader-writer-plugins [2] Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- --------- ---------- ---------------------------- #10 [3] bro hosom [4] 2014-08-07 Log urls sent over SMTP. [5] [1] BIT-1222 https://bro-tracker.atlassian.net/browse/BIT-1222 [2] reader-writer-plugins https://github.com/bro/bro/tree/topic/robin/reader-writer-plugins [3] Pull Request #10 https://github.com/bro/bro/pull/10 [4] hosom https://github.com/hosom [5] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From jira at bro-tracker.atlassian.net Fri Aug 15 08:02:07 2014 From: jira at bro-tracker.atlassian.net (hui (JIRA)) Date: Fri, 15 Aug 2014 10:02:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502#comment-17502 ] hui commented on BIT-1231: -------------------------- I committed all the changes in the branch "topic/hui/dnp3-udp". It is ready to merge. > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Fri Aug 15 09:12:07 2014 From: jira at bro-tracker.atlassian.net (hui (JIRA)) Date: Fri, 15 Aug 2014 11:12:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503#comment-17503 ] hui commented on BIT-1231: -------------------------- Not yet to merge. Let me first remove some debug printf function in DNP3.cc and DNP3.h. > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From jira at bro-tracker.atlassian.net Fri Aug 15 12:30:07 2014 From: jira at bro-tracker.atlassian.net (hui (JIRA)) Date: Fri, 15 Aug 2014 14:30:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504#comment-17504 ] hui commented on BIT-1231: -------------------------- Just removed the debug printf in DNP3.cc. Now it is ready to merge. > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-02-003#64000) From noreply at bro.org Sat Aug 16 00:00:20 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 16 Aug 2014 00:00:20 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408160700.s7G70K09024382@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ------------ ---------- ------------- ---------- ------------------------------------- BIT-1222 [1] Bro Robin Sommer Robin Sommer 2014-08-14 2.4 Normal topic/robin/reader-writer-plugins [2] Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- --------------------------------------------------------------------------- #11 [3] bro balintm [4] 2014-08-15 Flip roles if [Syn,Ack] is seen as a first packet in TCP communication. [5] #10 [6] bro hosom [7] 2014-08-07 Log urls sent over SMTP. [8] [1] BIT-1222 https://bro-tracker.atlassian.net/browse/BIT-1222 [2] reader-writer-plugins https://github.com/bro/bro/tree/topic/robin/reader-writer-plugins [3] Pull Request #11 https://github.com/bro/bro/pull/11 [4] balintm https://github.com/balintm [5] Merge Pull Request #11 with git pull -no-ff --no-commit https://github.com/balintm/bro.git master [6] Pull Request #10 https://github.com/bro/bro/pull/10 [7] hosom https://github.com/hosom [8] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From noreply at bro.org Sun Aug 17 00:00:20 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 17 Aug 2014 00:00:20 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408170700.s7H70Kmo014111@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ------------ ---------- ------------- ---------- ------------------------------------- BIT-1222 [1] Bro Robin Sommer Robin Sommer 2014-08-14 2.4 Normal topic/robin/reader-writer-plugins [2] Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- --------------------------------------------------------------------------- #11 [3] bro balintm [4] 2014-08-15 Flip roles if [Syn,Ack] is seen as a first packet in TCP communication. [5] #10 [6] bro hosom [7] 2014-08-07 Log urls sent over SMTP. [8] [1] BIT-1222 https://bro-tracker.atlassian.net/browse/BIT-1222 [2] reader-writer-plugins https://github.com/bro/bro/tree/topic/robin/reader-writer-plugins [3] Pull Request #11 https://github.com/bro/bro/pull/11 [4] balintm https://github.com/balintm [5] Merge Pull Request #11 with git pull -no-ff --no-commit https://github.com/balintm/bro.git master [6] Pull Request #10 https://github.com/bro/bro/pull/10 [7] hosom https://github.com/hosom [8] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From noreply at bro.org Mon Aug 18 00:00:28 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 18 Aug 2014 00:00:28 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408180700.s7I70SC3018676@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ------------ ---------- ------------- ---------- ------------------------------------- BIT-1222 [1] Bro Robin Sommer Robin Sommer 2014-08-14 2.4 Normal topic/robin/reader-writer-plugins [2] Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- --------------------------------------------------------------------------- #11 [3] bro balintm [4] 2014-08-15 Flip roles if [Syn,Ack] is seen as a first packet in TCP communication. [5] #10 [6] bro hosom [7] 2014-08-07 Log urls sent over SMTP. [8] [1] BIT-1222 https://bro-tracker.atlassian.net/browse/BIT-1222 [2] reader-writer-plugins https://github.com/bro/bro/tree/topic/robin/reader-writer-plugins [3] Pull Request #11 https://github.com/bro/bro/pull/11 [4] balintm https://github.com/balintm [5] Merge Pull Request #11 with git pull -no-ff --no-commit https://github.com/balintm/bro.git master [6] Pull Request #10 https://github.com/bro/bro/pull/10 [7] hosom https://github.com/hosom [8] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From jira at bro-tracker.atlassian.net Mon Aug 18 13:41:07 2014 From: jira at bro-tracker.atlassian.net (hui (JIRA)) Date: Mon, 18 Aug 2014 15:41:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hui updated BIT-1231: --------------------- Status: Merge Request (was: Open) > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From noreply at bro.org Tue Aug 19 00:00:25 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 19 Aug 2014 00:00:25 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408190700.s7J70PZP013650@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ------------ ---------- ------------- ---------- ---------------------------------------- BIT-1231 [1] Bro hui - 2014-08-18 - Normal DNP3 Analyzer Supports for DNP3-over-UDP BIT-1222 [2] Bro Robin Sommer Robin Sommer 2014-08-14 2.4 Normal topic/robin/reader-writer-plugins [3] Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- --------------------------------------------------------------------------- #11 [4] bro balintm [5] 2014-08-15 Flip roles if [Syn,Ack] is seen as a first packet in TCP communication. [6] #10 [7] bro hosom [8] 2014-08-07 Log urls sent over SMTP. [9] [1] BIT-1231 https://bro-tracker.atlassian.net/browse/BIT-1231 [2] BIT-1222 https://bro-tracker.atlassian.net/browse/BIT-1222 [3] reader-writer-plugins https://github.com/bro/bro/tree/topic/robin/reader-writer-plugins [4] Pull Request #11 https://github.com/bro/bro/pull/11 [5] balintm https://github.com/balintm [6] Merge Pull Request #11 with git pull -no-ff --no-commit https://github.com/balintm/bro.git master [7] Pull Request #10 https://github.com/bro/bro/pull/10 [8] hosom https://github.com/hosom [9] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From jira at bro-tracker.atlassian.net Tue Aug 19 07:31:07 2014 From: jira at bro-tracker.atlassian.net (Tim Yardley (JIRA)) Date: Tue, 19 Aug 2014 09:31:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1232) Command line script execution has limitations In-Reply-To: References: Message-ID: Tim Yardley created BIT-1232: -------------------------------- Summary: Command line script execution has limitations Key: BIT-1232 URL: https://bro-tracker.atlassian.net/browse/BIT-1232 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.3 Reporter: Tim Yardley root at lubuntu:/exercises/BroCon14/analyzers# bro -e "GridFTP::size_threshold = 1000;" error in , line 1: const is not a modifiable lvalue (GridFTP::size_threshold) and then to do it right... root at lubuntu:/exercises/BroCon14/analyzers# bro -e "redef GridFTP::size_threshold = 1000;" internal warning in , line 1: Can't document redef of GridFTP::size_threshold, lookup of failed Neither work. However... root at lubuntu:/exercises/BroCon14/analyzers# bro "GridFTP::size_threshold=1000" root at lubuntu:/exercises/BroCon14/analyzers# cat blah.bro redef GridFTP::size_threshold = 1000; root at lubuntu:/exercises/BroCon14/analyzers# bro blah.bro root at lubuntu:/exercises/BroCon14/analyzers# All fine. The redef from the command line is the interesting part. -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Tue Aug 19 07:56:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 19 Aug 2014 09:56:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1232) Command line script execution has limitations In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600#comment-17600 ] Jon Siwek commented on BIT-1232: -------------------------------- {quote} root at lubuntu:/exercises/BroCon14/analyzers# bro -e "GridFTP::size_threshold = 1000;" error in , line 1: const is not a modifiable lvalue (GridFTP::size_threshold) {quote} Yes, that's correct behavior because the variable is const w/ &redef and so can only be re-assigned via redef. {quote} and then to do it right... root at lubuntu:/exercises/BroCon14/analyzers# bro -e "redef GridFTP::size_threshold = 1000;" internal warning in , line 1: Can't document redef of GridFTP::size_threshold, lookup of failed Neither work. {quote} This way does work -- it emits a warning not an error. The warning isn't anything to worry about, so I'll work on silencing that since there's nothing wrong about what you're doing here. {quote} However... root at lubuntu:/exercises/BroCon14/analyzers# bro "GridFTP::size_threshold=1000" root at lubuntu:/exercises/BroCon14/analyzers# cat blah.bro redef GridFTP::size_threshold = 1000; root at lubuntu:/exercises/BroCon14/analyzers# bro blah.bro root at lubuntu:/exercises/BroCon14/analyzers# All fine. The redef from the command line is the interesting part. {quote} Yeah, that's an alternative/convenience way to implicitly supply redefs via the command line (arbitrary code would need to use -e, but "identifier=expr" arguments on the command line are treated as if you mean to do a redef). > Command line script execution has limitations > --------------------------------------------- > > Key: BIT-1232 > URL: https://bro-tracker.atlassian.net/browse/BIT-1232 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Reporter: Tim Yardley > > root at lubuntu:/exercises/BroCon14/analyzers# bro -e "GridFTP::size_threshold = 1000;" > error in , line 1: const is not a modifiable lvalue (GridFTP::size_threshold) > and then to do it right... > root at lubuntu:/exercises/BroCon14/analyzers# bro -e "redef GridFTP::size_threshold = 1000;" > internal warning in , line 1: Can't document redef of GridFTP::size_threshold, lookup of failed > Neither work. > However... > root at lubuntu:/exercises/BroCon14/analyzers# bro "GridFTP::size_threshold=1000" > root at lubuntu:/exercises/BroCon14/analyzers# cat blah.bro > redef GridFTP::size_threshold = 1000; > root at lubuntu:/exercises/BroCon14/analyzers# bro blah.bro > root at lubuntu:/exercises/BroCon14/analyzers# > All fine. The redef from the command line is the interesting part. -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Tue Aug 19 07:57:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 19 Aug 2014 09:57:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1232) Command line script execution has limitations In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek reassigned BIT-1232: ------------------------------ Assignee: Jon Siwek > Command line script execution has limitations > --------------------------------------------- > > Key: BIT-1232 > URL: https://bro-tracker.atlassian.net/browse/BIT-1232 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Reporter: Tim Yardley > Assignee: Jon Siwek > > root at lubuntu:/exercises/BroCon14/analyzers# bro -e "GridFTP::size_threshold = 1000;" > error in , line 1: const is not a modifiable lvalue (GridFTP::size_threshold) > and then to do it right... > root at lubuntu:/exercises/BroCon14/analyzers# bro -e "redef GridFTP::size_threshold = 1000;" > internal warning in , line 1: Can't document redef of GridFTP::size_threshold, lookup of failed > Neither work. > However... > root at lubuntu:/exercises/BroCon14/analyzers# bro "GridFTP::size_threshold=1000" > root at lubuntu:/exercises/BroCon14/analyzers# cat blah.bro > redef GridFTP::size_threshold = 1000; > root at lubuntu:/exercises/BroCon14/analyzers# bro blah.bro > root at lubuntu:/exercises/BroCon14/analyzers# > All fine. The redef from the command line is the interesting part. -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Tue Aug 19 08:11:07 2014 From: jira at bro-tracker.atlassian.net (Tim Yardley (JIRA)) Date: Tue, 19 Aug 2014 10:11:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1232) Command line script execution has limitations In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601#comment-17601 ] Tim Yardley commented on BIT-1232: ---------------------------------- Thanks. I suppose I should have printed the output. Didn't notice the error vs warning difference there. root at lubuntu:/exercises/BroCon14/analyzers# bro -e "redef GridFTP::size_threshold = 1000; print GridFTP::size_threshold;" internal warning in , line 1: Can't document redef of GridFTP::size_threshold, lookup of failed 1000 Thanks! > Command line script execution has limitations > --------------------------------------------- > > Key: BIT-1232 > URL: https://bro-tracker.atlassian.net/browse/BIT-1232 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Reporter: Tim Yardley > Assignee: Jon Siwek > > root at lubuntu:/exercises/BroCon14/analyzers# bro -e "GridFTP::size_threshold = 1000;" > error in , line 1: const is not a modifiable lvalue (GridFTP::size_threshold) > and then to do it right... > root at lubuntu:/exercises/BroCon14/analyzers# bro -e "redef GridFTP::size_threshold = 1000;" > internal warning in , line 1: Can't document redef of GridFTP::size_threshold, lookup of failed > Neither work. > However... > root at lubuntu:/exercises/BroCon14/analyzers# bro "GridFTP::size_threshold=1000" > root at lubuntu:/exercises/BroCon14/analyzers# cat blah.bro > redef GridFTP::size_threshold = 1000; > root at lubuntu:/exercises/BroCon14/analyzers# bro blah.bro > root at lubuntu:/exercises/BroCon14/analyzers# > All fine. The redef from the command line is the interesting part. -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Tue Aug 19 09:08:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 19 Aug 2014 11:08:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1232) Command line script execution has limitations In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1232: --------------------------- Resolution: Fixed Status: Closed (was: Open) > Command line script execution has limitations > --------------------------------------------- > > Key: BIT-1232 > URL: https://bro-tracker.atlassian.net/browse/BIT-1232 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Reporter: Tim Yardley > Assignee: Jon Siwek > > root at lubuntu:/exercises/BroCon14/analyzers# bro -e "GridFTP::size_threshold = 1000;" > error in , line 1: const is not a modifiable lvalue (GridFTP::size_threshold) > and then to do it right... > root at lubuntu:/exercises/BroCon14/analyzers# bro -e "redef GridFTP::size_threshold = 1000;" > internal warning in , line 1: Can't document redef of GridFTP::size_threshold, lookup of failed > Neither work. > However... > root at lubuntu:/exercises/BroCon14/analyzers# bro "GridFTP::size_threshold=1000" > root at lubuntu:/exercises/BroCon14/analyzers# cat blah.bro > redef GridFTP::size_threshold = 1000; > root at lubuntu:/exercises/BroCon14/analyzers# bro blah.bro > root at lubuntu:/exercises/BroCon14/analyzers# > All fine. The redef from the command line is the interesting part. -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From noreply at bro.org Wed Aug 20 00:00:19 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 20 Aug 2014 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408200700.s7K70Jlh024735@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ------------ ---------- ------------- ---------- ---------------------------------------- BIT-1231 [1] Bro hui - 2014-08-18 - Normal DNP3 Analyzer Supports for DNP3-over-UDP BIT-1222 [2] Bro Robin Sommer Robin Sommer 2014-08-14 2.4 Normal topic/robin/reader-writer-plugins [3] Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- --------------------------------------------------------------------------- #11 [4] bro balintm [5] 2014-08-15 Flip roles if [Syn,Ack] is seen as a first packet in TCP communication. [6] #10 [7] bro hosom [8] 2014-08-07 Log urls sent over SMTP. [9] [1] BIT-1231 https://bro-tracker.atlassian.net/browse/BIT-1231 [2] BIT-1222 https://bro-tracker.atlassian.net/browse/BIT-1222 [3] reader-writer-plugins https://github.com/bro/bro/tree/topic/robin/reader-writer-plugins [4] Pull Request #11 https://github.com/bro/bro/pull/11 [5] balintm https://github.com/balintm [6] Merge Pull Request #11 with git pull -no-ff --no-commit https://github.com/balintm/bro.git master [7] Pull Request #10 https://github.com/bro/bro/pull/10 [8] hosom https://github.com/hosom [9] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From jira at bro-tracker.atlassian.net Wed Aug 20 13:44:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 20 Aug 2014 15:44:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek reassigned BIT-1222: ------------------------------ Assignee: Jon Siwek (was: Robin Sommer) > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.4 > > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From noreply at bro.org Thu Aug 21 00:00:14 2014 From: noreply at bro.org (Merge Tracker) Date: Thu, 21 Aug 2014 00:00:14 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408210700.s7L70EmP030702@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ------------ ---------- ---------- ------------- ---------- ---------------------------------------- BIT-1231 [1] Bro hui - 2014-08-18 - Normal DNP3 Analyzer Supports for DNP3-over-UDP BIT-1222 [2] Bro Robin Sommer Jon Siwek 2014-08-20 2.4 Normal topic/robin/reader-writer-plugins [3] Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- ----------- ---------- --------------------------------------------------------------------------- #12 [4] bro zeebr [5] 2014-08-20 SMB & NTLM analyzers. [6] #11 [7] bro balintm [8] 2014-08-15 Flip roles if [Syn,Ack] is seen as a first packet in TCP communication. [9] #10 [10] bro hosom [11] 2014-08-07 Log urls sent over SMTP. [12] [1] BIT-1231 https://bro-tracker.atlassian.net/browse/BIT-1231 [2] BIT-1222 https://bro-tracker.atlassian.net/browse/BIT-1222 [3] reader-writer-plugins https://github.com/bro/bro/tree/topic/robin/reader-writer-plugins [4] Pull Request #12 https://github.com/bro/bro/pull/12 [5] zeebr https://github.com/zeebr [6] Merge Pull Request #12 with git pull -no-ff --no-commit https://github.com/bro/bro.git topic/vladg/smb [7] Pull Request #11 https://github.com/bro/bro/pull/11 [8] balintm https://github.com/balintm [9] Merge Pull Request #11 with git pull -no-ff --no-commit https://github.com/balintm/bro.git master [10] Pull Request #10 https://github.com/bro/bro/pull/10 [11] hosom https://github.com/hosom [12] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From jira at bro-tracker.atlassian.net Thu Aug 21 11:13:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 21 Aug 2014 13:13:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602#comment-17602 ] Jon Siwek commented on BIT-1222: -------------------------------- Quick questions about the bro-plugins repo: There's directories called {{elasticsearch/scripts/Bro/ElasticSearch}} and {{dataseries/scripts/bro/dataseries}}. Should the capitalization be more consistent? If so, which casing scheme should I go with? {{elasticsearch/scripts/Bro/ElasticSearch/main.bro}} is an empty file. Is it meant to contain something, or can I just remove? > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.4 > > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Thu Aug 21 11:52:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 21 Aug 2014 13:52:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603#comment-17603 ] Robin Sommer commented on BIT-1222: ----------------------------------- Yeah, should be consistent, but looks like I couldn't make up my mind which version I prefer. :-) The skeleton puts the capitalized version in place, and I think that's indeed my preference, as it matches the plugin's name directly. On the other hand, that doesn't align with our current naming scheme for scripts, so it would put a new standard in place for plugins (which might be fine and indeed help to differentiate their scripts). I guess in the end I'm fine either way; do you have a preference? Just remove. > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.4 > > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Thu Aug 21 12:49:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 21 Aug 2014 14:49:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17604#comment-17604 ] Jon Siwek commented on BIT-1222: -------------------------------- Also no preference, will choose the capitalized version maybe just because that's the one that requires less changes. > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.4 > > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Thu Aug 21 14:07:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Thu, 21 Aug 2014 16:07:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1222) topic/robin/reader-writer-plugins In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1222: --------------------------- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > topic/robin/reader-writer-plugins > --------------------------------- > > Key: BIT-1222 > URL: https://bro-tracker.atlassian.net/browse/BIT-1222 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Assignee: Jon Siwek > Fix For: 2.4 > > > This moves log writers and input readers to the new plugin API. No functional differences, except that one can now implement them via external plugins as well. Test cases for that included. > Most of the change is just moving stuff around, plus adapting to the new API. There are a few changes to defining/handling of the corresponding builtin types, as they now have to be dynamic. -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Thu Aug 21 15:11:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 21 Aug 2014 17:11:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1233) Parameters inside dynamically instantiated functions bind incorrectly In-Reply-To: References: Message-ID: Robin Sommer created BIT-1233: --------------------------------- Summary: Parameters inside dynamically instantiated functions bind incorrectly Key: BIT-1233 URL: https://bro-tracker.atlassian.net/browse/BIT-1233 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Robin Sommer The example below should print "2 1", but prints "2 2". I'm not sure if we can actually fix that easily, but if not it should at least print an error message that binding to outer parameters isn't supported. {noformat} type Foo: record { x: function(a: string) : string; }; function bar(b: string) { local f: Foo; f = [$x=function(a: string) : string { return cat(a, " ", b); } ]; print f$x("2"); } event bro_init() { bar("1"); } {noformat} -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From noreply at bro.org Fri Aug 22 00:00:20 2014 From: noreply at bro.org (Merge Tracker) Date: Fri, 22 Aug 2014 00:00:20 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408220700.s7M70KTm022647@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ---------- ---------- ---------- ------------- ---------- ---------------------------------------- BIT-1231 [1] Bro hui - 2014-08-18 - Normal DNP3 Analyzer Supports for DNP3-over-UDP Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- --------------------------------------------------------------------------- #12 [2] bro zeebr [3] 2014-08-20 SMB & NTLM analyzers. [4] #11 [5] bro balintm [6] 2014-08-15 Flip roles if [Syn,Ack] is seen as a first packet in TCP communication. [7] #10 [8] bro hosom [9] 2014-08-07 Log urls sent over SMTP. [10] [1] BIT-1231 https://bro-tracker.atlassian.net/browse/BIT-1231 [2] Pull Request #12 https://github.com/bro/bro/pull/12 [3] zeebr https://github.com/zeebr [4] Merge Pull Request #12 with git pull -no-ff --no-commit https://github.com/bro/bro.git topic/vladg/smb [5] Pull Request #11 https://github.com/bro/bro/pull/11 [6] balintm https://github.com/balintm [7] Merge Pull Request #11 with git pull -no-ff --no-commit https://github.com/balintm/bro.git master [8] Pull Request #10 https://github.com/bro/bro/pull/10 [9] hosom https://github.com/hosom [10] Merge Pull Request #10 with git pull -no-ff --no-commit https://github.com/hosom/bro.git patch-1 From jira at bro-tracker.atlassian.net Fri Aug 22 14:16:07 2014 From: jira at bro-tracker.atlassian.net (mel (JIRA)) Date: Fri, 22 Aug 2014 16:16:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1234) Yosemite attempt In-Reply-To: References: Message-ID: mel created BIT-1234: ------------------------ Summary: Yosemite attempt Key: BIT-1234 URL: https://bro-tracker.atlassian.net/browse/BIT-1234 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.3 Environment: OS X 10.10 Yosemite, MacBook Pro Reporter: mel I'm setting up opengraphiti (http://www.opengraphiti.com/) and wanted to use the .pcap file to create a visual web. From what I gathered I need to run the .pcap through bro -r to create a .log file which will be usable. On to the issue. The ./configure goes off without a hitch with the final output being -- Generating done -- Build files have been written to: /Users/infosec/bro/build. Then when I try to run "make" I get this fun error message [ 19%] Building CXX object src/analyzer/protocol/conn-size/CMakeFiles/plugin-Bro-ConnSize.dir/ConnSize.cc.o In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: In file included from /Users/infosec/bro/src/Var.h:7: In file included from /Users/infosec/bro/src/Expr.h:11: In file included from /Users/infosec/bro/src/Val.h:11: /Users/infosec/bro/src/net_util.h:210:15: error: expected ')' inline uint64 ntohll(uint64 i) ^ /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' #define ntohll(x) __DARWIN_OSSwapInt64(x) ^ /usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64' (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) ^ /Users/infosec/bro/src/net_util.h:210:15: note: to match this '(' /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' #define ntohll(x) __DARWIN_OSSwapInt64(x) ^ /usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64' (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) ^ In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: In file included from /Users/infosec/bro/src/Var.h:7: In file included from /Users/infosec/bro/src/Expr.h:11: In file included from /Users/infosec/bro/src/Val.h:11: /Users/infosec/bro/src/net_util.h:210:15: error: functions that differ only in their return type cannot be overloaded inline uint64 ntohll(uint64 i) ^ /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' #define ntohll(x) __DARWIN_OSSwapInt64(x) ^ /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) ^ /Users/infosec/bro/src/net_util.h:210:15: note: previous implicit declaration is here /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' #define ntohll(x) __DARWIN_OSSwapInt64(x) ^ /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) ^ In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: In file included from /Users/infosec/bro/src/Var.h:7: In file included from /Users/infosec/bro/src/Expr.h:11: In file included from /Users/infosec/bro/src/Val.h:11: /Users/infosec/bro/src/net_util.h:226:15: error: expected ')' inline uint64 htonll(uint64 i) { return ntohll(i); } ^ /usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll' #define htonll(x) __DARWIN_OSSwapInt64(x) ^ /usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64' (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) ^ /Users/infosec/bro/src/net_util.h:226:15: note: to match this '(' /usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll' #define htonll(x) __DARWIN_OSSwapInt64(x) ^ /usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64' (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) ^ In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: In file included from /Users/infosec/bro/src/Var.h:7: In file included from /Users/infosec/bro/src/Expr.h:11: In file included from /Users/infosec/bro/src/Val.h:11: /Users/infosec/bro/src/net_util.h:226:15: error: functions that differ only in their return type cannot be overloaded inline uint64 htonll(uint64 i) { return ntohll(i); } ^ /usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll' #define htonll(x) __DARWIN_OSSwapInt64(x) ^ /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) ^ /Users/infosec/bro/src/net_util.h:210:15: note: previous implicit declaration is here inline uint64 ntohll(uint64 i) ^ /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' #define ntohll(x) __DARWIN_OSSwapInt64(x) ^ /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) ^ 4 errors generated. make[3]: *** [src/analyzer/protocol/conn-size/CMakeFiles/plugin-Bro-ConnSize.dir/ConnSize.cc.o] Error 1 make[2]: *** [src/analyzer/protocol/conn-size/CMakeFiles/plugin-Bro-ConnSize.dir/all] Error 2 make[1]: *** [all] Error 2 make: *** [all] Error 2 -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Fri Aug 22 14:33:08 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 22 Aug 2014 16:33:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1231: --------------------------------- Assignee: Robin Sommer > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Assignee: Robin Sommer > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Fri Aug 22 14:37:08 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 22 Aug 2014 16:37:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605#comment-17605 ] Robin Sommer commented on BIT-1231: ----------------------------------- {quote} For DNP3 over UDP analyzer. most of the codes are copied and pasted. Better way to reuse the code? {quote} Yeah, don't do that. :) You need to refactor this out so that the code exists only once, probably by creating a joint base class or something (also applies to crc_table_initialized and crc_table). > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Assignee: Robin Sommer > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Fri Aug 22 14:38:08 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 22 Aug 2014 16:38:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1231: ------------------------------ Status: Open (was: Merge Request) > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Assignee: Robin Sommer > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Fri Aug 22 14:38:08 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 22 Aug 2014 16:38:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1231: --------------------------------- Assignee: (was: Robin Sommer) > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Fri Aug 22 14:41:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 22 Aug 2014 16:41:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-1231: --------------------------------- Assignee: Robin Sommer > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Assignee: Robin Sommer > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Fri Aug 22 14:42:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 22 Aug 2014 16:42:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1231: ------------------------------ Assignee: hui (was: Robin Sommer) > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Assignee: hui > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Fri Aug 22 15:02:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 22 Aug 2014 17:02:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1233) Parameters inside dynamically instantiated functions bind incorrectly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606#comment-17606 ] Jon Siwek commented on BIT-1233: -------------------------------- Wasn't immediately seeing an easy way to make it work. Maybe take a look at topic/jsiwek/outer_param_binding and see if that's a decent way to detect it and raise an error for now. > Parameters inside dynamically instantiated functions bind incorrectly > --------------------------------------------------------------------- > > Key: BIT-1233 > URL: https://bro-tracker.atlassian.net/browse/BIT-1233 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Labels: language > > The example below should print "2 1", but prints "2 2". I'm not sure if we can actually fix that easily, but if not it should at least print an error message that binding to outer parameters isn't supported. > {noformat} > type Foo: record { > x: function(a: string) : string; > }; > function bar(b: string) > { > local f: Foo; > f = [$x=function(a: string) : string > { > return cat(a, " ", b); > } > ]; > print f$x("2"); > } > event bro_init() > { > bar("1"); > } > {noformat} -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Fri Aug 22 15:03:08 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 22 Aug 2014 17:03:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1233) Parameters inside dynamically instantiated functions bind incorrectly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1233: --------------------------- Status: Merge Request (was: Open) > Parameters inside dynamically instantiated functions bind incorrectly > --------------------------------------------------------------------- > > Key: BIT-1233 > URL: https://bro-tracker.atlassian.net/browse/BIT-1233 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Labels: language > > The example below should print "2 1", but prints "2 2". I'm not sure if we can actually fix that easily, but if not it should at least print an error message that binding to outer parameters isn't supported. > {noformat} > type Foo: record { > x: function(a: string) : string; > }; > function bar(b: string) > { > local f: Foo; > f = [$x=function(a: string) : string > { > return cat(a, " ", b); > } > ]; > print f$x("2"); > } > event bro_init() > { > bar("1"); > } > {noformat} -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Fri Aug 22 15:32:07 2014 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Fri, 22 Aug 2014 17:32:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1233) Parameters inside dynamically instantiated functions bind incorrectly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1233: ------------------------------ Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Parameters inside dynamically instantiated functions bind incorrectly > --------------------------------------------------------------------- > > Key: BIT-1233 > URL: https://bro-tracker.atlassian.net/browse/BIT-1233 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Labels: language > > The example below should print "2 1", but prints "2 2". I'm not sure if we can actually fix that easily, but if not it should at least print an error message that binding to outer parameters isn't supported. > {noformat} > type Foo: record { > x: function(a: string) : string; > }; > function bar(b: string) > { > local f: Foo; > f = [$x=function(a: string) : string > { > return cat(a, " ", b); > } > ]; > print f$x("2"); > } > event bro_init() > { > bar("1"); > } > {noformat} -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Fri Aug 22 17:59:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 22 Aug 2014 19:59:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1234) Yosemite attempt In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1234: --------------------------- Resolution: Fixed Status: Closed (was: Open) > Yosemite attempt > ---------------- > > Key: BIT-1234 > URL: https://bro-tracker.atlassian.net/browse/BIT-1234 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Environment: OS X 10.10 Yosemite, MacBook Pro > Reporter: mel > Labels: OSX10.10, bro, yosemite > > I'm setting up opengraphiti (http://www.opengraphiti.com/) and wanted to use the .pcap file to create a visual web. From what I gathered I need to run the .pcap through bro -r to create a .log file which will be usable. > On to the issue. > The ./configure goes off without a hitch with the final output being -- Generating done -- Build files have been written to: /Users/infosec/bro/build. > Then when I try to run "make" I get this fun error message > [ 19%] Building CXX object src/analyzer/protocol/conn-size/CMakeFiles/plugin-Bro-ConnSize.dir/ConnSize.cc.o > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: > In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: > In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: > In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: > In file included from /Users/infosec/bro/src/Var.h:7: > In file included from /Users/infosec/bro/src/Expr.h:11: > In file included from /Users/infosec/bro/src/Val.h:11: > /Users/infosec/bro/src/net_util.h:210:15: error: expected ')' > inline uint64 ntohll(uint64 i) > ^ > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > /Users/infosec/bro/src/net_util.h:210:15: note: to match this '(' > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: > In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: > In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: > In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: > In file included from /Users/infosec/bro/src/Var.h:7: > In file included from /Users/infosec/bro/src/Expr.h:11: > In file included from /Users/infosec/bro/src/Val.h:11: > /Users/infosec/bro/src/net_util.h:210:15: error: functions that differ only in their return type cannot be overloaded > inline uint64 ntohll(uint64 i) > ^ > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > /Users/infosec/bro/src/net_util.h:210:15: note: previous implicit declaration is here > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: > In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: > In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: > In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: > In file included from /Users/infosec/bro/src/Var.h:7: > In file included from /Users/infosec/bro/src/Expr.h:11: > In file included from /Users/infosec/bro/src/Val.h:11: > /Users/infosec/bro/src/net_util.h:226:15: error: expected ')' > inline uint64 htonll(uint64 i) { return ntohll(i); } > ^ > /usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll' > #define htonll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > /Users/infosec/bro/src/net_util.h:226:15: note: to match this '(' > /usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll' > #define htonll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: > In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: > In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: > In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: > In file included from /Users/infosec/bro/src/Var.h:7: > In file included from /Users/infosec/bro/src/Expr.h:11: > In file included from /Users/infosec/bro/src/Val.h:11: > /Users/infosec/bro/src/net_util.h:226:15: error: functions that differ only in their return type cannot be overloaded > inline uint64 htonll(uint64 i) { return ntohll(i); } > ^ > /usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll' > #define htonll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > /Users/infosec/bro/src/net_util.h:210:15: note: previous implicit declaration is here > inline uint64 ntohll(uint64 i) > ^ > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > 4 errors generated. > make[3]: *** [src/analyzer/protocol/conn-size/CMakeFiles/plugin-Bro-ConnSize.dir/ConnSize.cc.o] Error 1 > make[2]: *** [src/analyzer/protocol/conn-size/CMakeFiles/plugin-Bro-ConnSize.dir/all] Error 2 > make[1]: *** [all] Error 2 > make: *** [all] Error 2 -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Fri Aug 22 18:00:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 22 Aug 2014 20:00:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1234) Yosemite attempt In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1234: --------------------------- Fix Version/s: 2.4 > Yosemite attempt > ---------------- > > Key: BIT-1234 > URL: https://bro-tracker.atlassian.net/browse/BIT-1234 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Environment: OS X 10.10 Yosemite, MacBook Pro > Reporter: mel > Labels: OSX10.10, bro, yosemite > Fix For: 2.4 > > > I'm setting up opengraphiti (http://www.opengraphiti.com/) and wanted to use the .pcap file to create a visual web. From what I gathered I need to run the .pcap through bro -r to create a .log file which will be usable. > On to the issue. > The ./configure goes off without a hitch with the final output being -- Generating done -- Build files have been written to: /Users/infosec/bro/build. > Then when I try to run "make" I get this fun error message > [ 19%] Building CXX object src/analyzer/protocol/conn-size/CMakeFiles/plugin-Bro-ConnSize.dir/ConnSize.cc.o > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: > In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: > In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: > In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: > In file included from /Users/infosec/bro/src/Var.h:7: > In file included from /Users/infosec/bro/src/Expr.h:11: > In file included from /Users/infosec/bro/src/Val.h:11: > /Users/infosec/bro/src/net_util.h:210:15: error: expected ')' > inline uint64 ntohll(uint64 i) > ^ > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > /Users/infosec/bro/src/net_util.h:210:15: note: to match this '(' > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: > In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: > In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: > In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: > In file included from /Users/infosec/bro/src/Var.h:7: > In file included from /Users/infosec/bro/src/Expr.h:11: > In file included from /Users/infosec/bro/src/Val.h:11: > /Users/infosec/bro/src/net_util.h:210:15: error: functions that differ only in their return type cannot be overloaded > inline uint64 ntohll(uint64 i) > ^ > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > /Users/infosec/bro/src/net_util.h:210:15: note: previous implicit declaration is here > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: > In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: > In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: > In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: > In file included from /Users/infosec/bro/src/Var.h:7: > In file included from /Users/infosec/bro/src/Expr.h:11: > In file included from /Users/infosec/bro/src/Val.h:11: > /Users/infosec/bro/src/net_util.h:226:15: error: expected ')' > inline uint64 htonll(uint64 i) { return ntohll(i); } > ^ > /usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll' > #define htonll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > /Users/infosec/bro/src/net_util.h:226:15: note: to match this '(' > /usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll' > #define htonll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: > In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: > In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: > In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: > In file included from /Users/infosec/bro/src/Var.h:7: > In file included from /Users/infosec/bro/src/Expr.h:11: > In file included from /Users/infosec/bro/src/Val.h:11: > /Users/infosec/bro/src/net_util.h:226:15: error: functions that differ only in their return type cannot be overloaded > inline uint64 htonll(uint64 i) { return ntohll(i); } > ^ > /usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll' > #define htonll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > /Users/infosec/bro/src/net_util.h:210:15: note: previous implicit declaration is here > inline uint64 ntohll(uint64 i) > ^ > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > 4 errors generated. > make[3]: *** [src/analyzer/protocol/conn-size/CMakeFiles/plugin-Bro-ConnSize.dir/ConnSize.cc.o] Error 1 > make[2]: *** [src/analyzer/protocol/conn-size/CMakeFiles/plugin-Bro-ConnSize.dir/all] Error 2 > make[1]: *** [all] Error 2 > make: *** [all] Error 2 -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From jira at bro-tracker.atlassian.net Fri Aug 22 18:04:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Fri, 22 Aug 2014 20:04:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1234) Yosemite attempt In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17607#comment-17607 ] Jon Siwek commented on BIT-1234: -------------------------------- It should be fixed in the git repository now, try doing a "git pull" and then continue where you left off with "make". Let me know if you still get errors. > Yosemite attempt > ---------------- > > Key: BIT-1234 > URL: https://bro-tracker.atlassian.net/browse/BIT-1234 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Environment: OS X 10.10 Yosemite, MacBook Pro > Reporter: mel > Labels: OSX10.10, bro, yosemite > Fix For: 2.4 > > > I'm setting up opengraphiti (http://www.opengraphiti.com/) and wanted to use the .pcap file to create a visual web. From what I gathered I need to run the .pcap through bro -r to create a .log file which will be usable. > On to the issue. > The ./configure goes off without a hitch with the final output being -- Generating done -- Build files have been written to: /Users/infosec/bro/build. > Then when I try to run "make" I get this fun error message > [ 19%] Building CXX object src/analyzer/protocol/conn-size/CMakeFiles/plugin-Bro-ConnSize.dir/ConnSize.cc.o > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: > In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: > In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: > In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: > In file included from /Users/infosec/bro/src/Var.h:7: > In file included from /Users/infosec/bro/src/Expr.h:11: > In file included from /Users/infosec/bro/src/Val.h:11: > /Users/infosec/bro/src/net_util.h:210:15: error: expected ')' > inline uint64 ntohll(uint64 i) > ^ > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > /Users/infosec/bro/src/net_util.h:210:15: note: to match this '(' > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: > In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: > In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: > In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: > In file included from /Users/infosec/bro/src/Var.h:7: > In file included from /Users/infosec/bro/src/Expr.h:11: > In file included from /Users/infosec/bro/src/Val.h:11: > /Users/infosec/bro/src/net_util.h:210:15: error: functions that differ only in their return type cannot be overloaded > inline uint64 ntohll(uint64 i) > ^ > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > /Users/infosec/bro/src/net_util.h:210:15: note: previous implicit declaration is here > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: > In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: > In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: > In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: > In file included from /Users/infosec/bro/src/Var.h:7: > In file included from /Users/infosec/bro/src/Expr.h:11: > In file included from /Users/infosec/bro/src/Val.h:11: > /Users/infosec/bro/src/net_util.h:226:15: error: expected ')' > inline uint64 htonll(uint64 i) { return ntohll(i); } > ^ > /usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll' > #define htonll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:30: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > /Users/infosec/bro/src/net_util.h:226:15: note: to match this '(' > /usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll' > #define htonll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:5: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.cc:6: > In file included from /Users/infosec/bro/src/analyzer/protocol/conn-size/ConnSize.h:7: > In file included from /Users/infosec/bro/src/analyzer/Analyzer.h:8: > In file included from /Users/infosec/bro/src/analyzer/Tag.h:10: > In file included from /Users/infosec/bro/src/plugin/ComponentManager.h:10: > In file included from /Users/infosec/bro/src/Var.h:7: > In file included from /Users/infosec/bro/src/Expr.h:11: > In file included from /Users/infosec/bro/src/Val.h:11: > /Users/infosec/bro/src/net_util.h:226:15: error: functions that differ only in their return type cannot be overloaded > inline uint64 htonll(uint64 i) { return ntohll(i); } > ^ > /usr/include/sys/_endian.h:141:25: note: expanded from macro 'htonll' > #define htonll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > /Users/infosec/bro/src/net_util.h:210:15: note: previous implicit declaration is here > inline uint64 ntohll(uint64 i) > ^ > /usr/include/sys/_endian.h:140:25: note: expanded from macro 'ntohll' > #define ntohll(x) __DARWIN_OSSwapInt64(x) > ^ > /usr/include/libkern/_OSByteOrder.h:78:6: note: expanded from macro '__DARWIN_OSSwapInt64' > (__builtin_constant_p(x) ? __DARWIN_OSSwapConstInt64(x) : _OSSwapInt64(x)) > ^ > 4 errors generated. > make[3]: *** [src/analyzer/protocol/conn-size/CMakeFiles/plugin-Bro-ConnSize.dir/ConnSize.cc.o] Error 1 > make[2]: *** [src/analyzer/protocol/conn-size/CMakeFiles/plugin-Bro-ConnSize.dir/all] Error 2 > make[1]: *** [all] Error 2 > make: *** [all] Error 2 -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From noreply at bro.org Sat Aug 23 00:00:17 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 23 Aug 2014 00:00:17 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408230700.s7N70HmT018528@bro-ids.icir.org> Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- ----------- ---------- --------------------------------------------------------------------------- #11 [1] bro balintm [2] 2014-08-15 Flip roles if [Syn,Ack] is seen as a first packet in TCP communication. [3] [1] Pull Request #11 https://github.com/bro/bro/pull/11 [2] balintm https://github.com/balintm [3] Merge Pull Request #11 with git pull --no-ff --no-commit https://github.com/balintm/bro.git master From noreply at bro.org Sun Aug 24 00:00:17 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 24 Aug 2014 00:00:17 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408240700.s7O70HeX005164@bro-ids.icir.org> Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- --------------- ---------- --------------------------------------------------------------------------- #13 [1] bro jimmyjones2 [2] 2014-08-23 Better documentation for sub_bytes [3] #11 [4] bro balintm [5] 2014-08-15 Flip roles if [Syn,Ack] is seen as a first packet in TCP communication. [6] [1] Pull Request #13 https://github.com/bro/bro/pull/13 [2] jimmyjones2 https://github.com/jimmyjones2 [3] Merge Pull Request #13 with git pull --no-ff --no-commit https://github.com/jimmyjones2/bro.git topic/jimmyjones2/string-doc [4] Pull Request #11 https://github.com/bro/bro/pull/11 [5] balintm https://github.com/balintm [6] Merge Pull Request #11 with git pull --no-ff --no-commit https://github.com/balintm/bro.git master From jira at bro-tracker.atlassian.net Sun Aug 24 12:01:07 2014 From: jira at bro-tracker.atlassian.net (Brian O'Berry (JIRA)) Date: Sun, 24 Aug 2014 14:01:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1235) HTTP multipart POST request alters file contents In-Reply-To: References: Message-ID: Brian O'Berry created BIT-1235: ---------------------------------- Summary: HTTP multipart POST request alters file contents Key: BIT-1235 URL: https://bro-tracker.atlassian.net/browse/BIT-1235 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.3 Environment: CentOS 6.5, file extract analyzer Reporter: Brian O'Berry Attachments: bro-2.3-HTTP.patch, gdb.log, upload-api-http.pcap HTTP POST multipart processing converts bare CR or LF chars to CRLF pairs, corrupting most files when extracted with Files::ANALYZER_EXTRACT. This is clear in the attached gdb.log, which has a backtrace that shows a buffer with the start of a PDF file entering MIME/HTTP entity processing at frame 25, and emerging with LF chars converted to CRLF at frame 6. Also attached are the pcap file associated with the backtrace, and an initial patch that we've barely begun to test. A point of concern with the patch is that it changes a weird.log entry from "line_terminated_with_single_CR" to "http_no_crlf_in_header_list". It does enable Files::ANALYZER_EXTRACT to correctly extract the PDF file from the attached pcap. Please let me know if we can provide anything else to help with this. -- This message was sent by Atlassian JIRA (v6.4-OD-03-010#64001) From noreply at bro.org Mon Aug 25 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Mon, 25 Aug 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408250700.s7P70FIi016659@bro-ids.icir.org> Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- --------------- ---------- --------------------------------------------------------------------------- #13 [1] bro jimmyjones2 [2] 2014-08-23 Better documentation for sub_bytes [3] #11 [4] bro balintm [5] 2014-08-15 Flip roles if [Syn,Ack] is seen as a first packet in TCP communication. [6] [1] Pull Request #13 https://github.com/bro/bro/pull/13 [2] jimmyjones2 https://github.com/jimmyjones2 [3] Merge Pull Request #13 with git pull --no-ff --no-commit https://github.com/jimmyjones2/bro.git topic/jimmyjones2/string-doc [4] Pull Request #11 https://github.com/bro/bro/pull/11 [5] balintm https://github.com/balintm [6] Merge Pull Request #11 with git pull --no-ff --no-commit https://github.com/balintm/bro.git master From jira at bro-tracker.atlassian.net Mon Aug 25 06:25:07 2014 From: jira at bro-tracker.atlassian.net (Brian O'Berry (JIRA)) Date: Mon, 25 Aug 2014 08:25:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1235) HTTP multipart POST request alters file contents In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700#comment-17700 ] Brian O'Berry commented on BIT-1235: ------------------------------------ Okay, the patch is incorrect because it assumes only a single file in the multipart form, which I think means the solution has to be addressed in tcp and/or mime parsing code. Could it be as simple as changing the if statements in analyzer::tcp::Content_Analyzer::DoDeliverOnce() to be equality tests rather than bitwise ANDs (ContentLine.cc lines 242 and 256)? There's also this code to think about in DoDeliver() starting at line 141: {code} if ( (CR_LF_as_EOL & CR_as_EOL) && last_char == '\r' && *data == '\n' ) { // CR is already considered as EOL. // Compress CRLF to just one line termination. // // Note, we test this prior to checking for // "plain delivery" because (1) we might have // made the decision to switch to plain delivery // based on a line terminated with '\r' for // which a '\n' then arrived, and (2) we are // careful when executing plain delivery to // clear last_char once we do so. last_char = *data; --len; ++data; ++seq; ++seq_delivered_in_lines; } {code} > HTTP multipart POST request alters file contents > ------------------------------------------------ > > Key: BIT-1235 > URL: https://bro-tracker.atlassian.net/browse/BIT-1235 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Environment: CentOS 6.5, file extract analyzer > Reporter: Brian O'Berry > Attachments: bro-2.3-HTTP.patch, gdb.log, upload-api-http.pcap > > > HTTP POST multipart processing converts bare CR or LF chars to CRLF pairs, corrupting most files when extracted with Files::ANALYZER_EXTRACT. This is clear in the attached gdb.log, which has a backtrace that shows a buffer with the start of a PDF file entering MIME/HTTP entity processing at frame 25, and emerging with LF chars converted to CRLF at frame 6. > Also attached are the pcap file associated with the backtrace, and an initial patch that we've barely begun to test. A point of concern with the patch is that it changes a weird.log entry from "line_terminated_with_single_CR" to "http_no_crlf_in_header_list". It does enable Files::ANALYZER_EXTRACT to correctly extract the PDF file from the attached pcap. > Please let me know if we can provide anything else to help with this. -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Mon Aug 25 06:54:07 2014 From: jira at bro-tracker.atlassian.net (Brian O'Berry (JIRA)) Date: Mon, 25 Aug 2014 08:54:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1235) HTTP multipart POST request alters file contents In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700#comment-17700 ] Brian O'Berry edited comment on BIT-1235 at 8/25/14 8:53 AM: ------------------------------------------------------------- Okay, the patch is incorrect because it assumes only a single file in the multipart form, which I think means the solution has to be addressed in tcp and/or mime parsing code. Could it be as simple as changing the if statements in analyzer::tcp::Content_Analyzer::DoDeliverOnce() to be equality tests rather than bitwise ANDs (ContentLine.cc lines 242 and 256)? There's also this code to think about in DoDeliver() starting at line 141: {code} if ( (CR_LF_as_EOL & CR_as_EOL) && last_char == '\r' && *data == '\n' ) { // CR is already considered as EOL. // Compress CRLF to just one line termination. // // Note, we test this prior to checking for // "plain delivery" because (1) we might have // made the decision to switch to plain delivery // based on a line terminated with '\r' for // which a '\n' then arrived, and (2) we are // careful when executing plain delivery to // clear last_char once we do so. last_char = *data; --len; ++data; ++seq; ++seq_delivered_in_lines; } {code} UPDATE: Removing the previous patch and changing if stmts in Content_Analyzer::DoDeliverOnce() as described, to use '==' rather than '&', *almost* produced a correct file. Bare CR and LF chars were left alone, but the trailing CRLF (before the multipart boundary string) was included in the file. Also, I see 2 messages in weird.log, reporting line_terminated_with_single_LF and line_terminated_with_single_CR, in that order. The PDF ends with {{%%EOF}}, and given the order of the weird messages, I have to wonder if the parsing code is getting confused by the final LFCRLF sequence preceding the closing multipart boundary. was (Author: boberry): Okay, the patch is incorrect because it assumes only a single file in the multipart form, which I think means the solution has to be addressed in tcp and/or mime parsing code. Could it be as simple as changing the if statements in analyzer::tcp::Content_Analyzer::DoDeliverOnce() to be equality tests rather than bitwise ANDs (ContentLine.cc lines 242 and 256)? There's also this code to think about in DoDeliver() starting at line 141: {code} if ( (CR_LF_as_EOL & CR_as_EOL) && last_char == '\r' && *data == '\n' ) { // CR is already considered as EOL. // Compress CRLF to just one line termination. // // Note, we test this prior to checking for // "plain delivery" because (1) we might have // made the decision to switch to plain delivery // based on a line terminated with '\r' for // which a '\n' then arrived, and (2) we are // careful when executing plain delivery to // clear last_char once we do so. last_char = *data; --len; ++data; ++seq; ++seq_delivered_in_lines; } {code} > HTTP multipart POST request alters file contents > ------------------------------------------------ > > Key: BIT-1235 > URL: https://bro-tracker.atlassian.net/browse/BIT-1235 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Environment: CentOS 6.5, file extract analyzer > Reporter: Brian O'Berry > Attachments: bro-2.3-HTTP.patch, gdb.log, upload-api-http.pcap > > > HTTP POST multipart processing converts bare CR or LF chars to CRLF pairs, corrupting most files when extracted with Files::ANALYZER_EXTRACT. This is clear in the attached gdb.log, which has a backtrace that shows a buffer with the start of a PDF file entering MIME/HTTP entity processing at frame 25, and emerging with LF chars converted to CRLF at frame 6. > Also attached are the pcap file associated with the backtrace, and an initial patch that we've barely begun to test. A point of concern with the patch is that it changes a weird.log entry from "line_terminated_with_single_CR" to "http_no_crlf_in_header_list". It does enable Files::ANALYZER_EXTRACT to correctly extract the PDF file from the attached pcap. > Please let me know if we can provide anything else to help with this. -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Mon Aug 25 07:16:08 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 25 Aug 2014 09:16:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1233) Parameters inside dynamically instantiated functions bind incorrectly In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1233: --------------------------- Fix Version/s: 2.4 > Parameters inside dynamically instantiated functions bind incorrectly > --------------------------------------------------------------------- > > Key: BIT-1233 > URL: https://bro-tracker.atlassian.net/browse/BIT-1233 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: git/master > Reporter: Robin Sommer > Labels: language > Fix For: 2.4 > > > The example below should print "2 1", but prints "2 2". I'm not sure if we can actually fix that easily, but if not it should at least print an error message that binding to outer parameters isn't supported. > {noformat} > type Foo: record { > x: function(a: string) : string; > }; > function bar(b: string) > { > local f: Foo; > f = [$x=function(a: string) : string > { > return cat(a, " ", b); > } > ]; > print f$x("2"); > } > event bro_init() > { > bar("1"); > } > {noformat} -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Mon Aug 25 09:42:08 2014 From: jira at bro-tracker.atlassian.net (hui (JIRA)) Date: Mon, 25 Aug 2014 11:42:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701#comment-17701 ] hui commented on BIT-1231: -------------------------- Hi Robin, What I did to avoid the same code being copied and pasted is to define a independent class, i.e., DNP3_Analyzer which includes most of the repeated codes. Then I let DNP3_TCP_Analyzer inherits from both this DNP3_Analyzer and TCP_ApplicationAnalyzer. I can pass the test by doing this. The only issue is that because the DNP3_Analyzer is defined independently, it can not use the function such as Weird, ProtocolViolation and ProtocolConfirmation. Do you know how I can call them inside this class? If this is not possible, I am thinking about let the function in DNP3_Analyzer return different error values to DNP3_TCP_Analyzer and call the Weird function in the DNP3_TCP_Analyzer. Is this OK to you? Best, Hui Lin On Fri, Aug 22, 2014 at 4:37 PM, Robin Sommer (JIRA) < -- Hui Lin PhD Candidate, Research Assistant Electrical and Computer Engineering Department University of Illinois at Urbana-Champaign > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Assignee: hui > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Mon Aug 25 14:32:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 25 Aug 2014 16:32:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1236) topic/jsiwek/flip-on-syn-ack In-Reply-To: References: Message-ID: Jon Siwek created BIT-1236: ------------------------------ Summary: topic/jsiwek/flip-on-syn-ack Key: BIT-1236 URL: https://bro-tracker.atlassian.net/browse/BIT-1236 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Jon Siwek Assignee: Robin Sommer Fix For: 2.4 This branch is in bro and bro-testing-private. The goal is the same as https://github.com/bro/bro/pull/11, but I have it flip roles at an even earlier point in the code path or else I notice some inconsistencies in things like connection history strings or the connsize analyzer counters (which were probably also issues w/ the old flipping method). -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Mon Aug 25 14:32:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 25 Aug 2014 16:32:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1236) topic/jsiwek/flip-on-syn-ack In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1236: --------------------------- Status: Merge Request (was: Open) > topic/jsiwek/flip-on-syn-ack > ---------------------------- > > Key: BIT-1236 > URL: https://bro-tracker.atlassian.net/browse/BIT-1236 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Robin Sommer > Fix For: 2.4 > > > This branch is in bro and bro-testing-private. > The goal is the same as https://github.com/bro/bro/pull/11, but I have it flip roles at an even earlier point in the code path or else I notice some inconsistencies in things like connection history strings or the connsize analyzer counters (which were probably also issues w/ the old flipping method). -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From vlad at grigorescu.org Mon Aug 25 14:40:11 2014 From: vlad at grigorescu.org (Vlad Grigorescu) Date: Mon, 25 Aug 2014 16:40:11 -0500 Subject: [Bro-Dev] [JIRA] (BIT-1236) topic/jsiwek/flip-on-syn-ack In-Reply-To: References: Message-ID: This ties into something I had noticed recently. Certain scanning tools like to use the same source port per destination IP (I imagine to cache portions of the TCP header). During these scans, multiple TCP connections occur. Bro saw traffic that had: - A connection that was setup and torn down as expected (conn_state == "SF") - A few minutes pass - A second connection that was setup and torn down as expected, *except* that the first SYN was missed - either by Bro or upstream loss. Bro considered these the same connection. Does it makes sense that following a connection teardown, if a SYN-ACK is seen, a new connection begins, instead of using the existing connection? I can probably grab a PCAP if necessary. --Vlad On Mon, Aug 25, 2014 at 4:32 PM, Jon Siwek (JIRA) < jira at bro-tracker.atlassian.net> wrote: > > [ > https://bro-tracker.atlassian.net/browse/BIT-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel > ] > > Jon Siwek updated BIT-1236: > --------------------------- > Status: Merge Request (was: Open) > > > topic/jsiwek/flip-on-syn-ack > > ---------------------------- > > > > Key: BIT-1236 > > URL: https://bro-tracker.atlassian.net/browse/BIT-1236 > > Project: Bro Issue Tracker > > Issue Type: Improvement > > Components: Bro > > Affects Versions: git/master > > Reporter: Jon Siwek > > Assignee: Robin Sommer > > Fix For: 2.4 > > > > > > This branch is in bro and bro-testing-private. > > The goal is the same as https://github.com/bro/bro/pull/11, but I have > it flip roles at an even earlier point in the code path or else I notice > some inconsistencies in things like connection history strings or the > connsize analyzer counters (which were probably also issues w/ the old > flipping method). > > > > -- > This message was sent by Atlassian JIRA > (v6.4-OD-04-006#64001) > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140825/6bca1a1a/attachment.html From jsiwek at illinois.edu Mon Aug 25 15:04:00 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 25 Aug 2014 22:04:00 +0000 Subject: [Bro-Dev] [JIRA] (BIT-1236) topic/jsiwek/flip-on-syn-ack In-Reply-To: References: Message-ID: <91CCA31E-6AF5-4DD9-950D-6C01205FEFD7@illinois.edu> On Aug 25, 2014, at 4:40 PM, Vlad Grigorescu wrote: > Does it makes sense that following a connection teardown, if a SYN-ACK is seen, a new connection begins, instead of using the existing connection? I can probably grab a PCAP if necessary. Actually, I?m thinking it may already work like you expect in many ?normal? situations. One special case I can remember (there may be others) is that Bro may defer closing out a connection even if it sees the teardown control packets when it thinks it may be possible to fill in a content gap (i.e. it thinks there?s packets coming in out of order, but maybe in your case it?s just never seen at all). If that doesn?t fit with what you saw and you?ve got a pcap you can send me, I can try to make sense of it. - Jon From jmellander at lbl.gov Mon Aug 25 16:45:28 2014 From: jmellander at lbl.gov (Jim Mellander) Date: Mon, 25 Aug 2014 16:45:28 -0700 Subject: [Bro-Dev] [JIRA] (BIT-1236) topic/jsiwek/flip-on-syn-ack In-Reply-To: References: Message-ID: It would be nice to have an optional hook to the script-level, which could signal Bro as to which side is the originator, if the 3-way handshake was missed. There are a number of cases where we could use local site knowledge to definitively identify originator & responder. On Mon, Aug 25, 2014 at 2:40 PM, Vlad Grigorescu wrote: > This ties into something I had noticed recently. Certain scanning tools > like to use the same source port per destination IP (I imagine to cache > portions of the TCP header). During these scans, multiple TCP connections > occur. Bro saw traffic that had: > > - A connection that was setup and torn down as expected (conn_state == > "SF") > - A few minutes pass > - A second connection that was setup and torn down as expected, *except* > that the first SYN was missed - either by Bro or upstream loss. > > Bro considered these the same connection. > > Does it makes sense that following a connection teardown, if a SYN-ACK is > seen, a new connection begins, instead of using the existing connection? I > can probably grab a PCAP if necessary. > > --Vlad > > > On Mon, Aug 25, 2014 at 4:32 PM, Jon Siwek (JIRA) < > jira at bro-tracker.atlassian.net> wrote: > >> >> [ >> https://bro-tracker.atlassian.net/browse/BIT-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel >> ] >> >> Jon Siwek updated BIT-1236: >> --------------------------- >> Status: Merge Request (was: Open) >> >> > topic/jsiwek/flip-on-syn-ack >> > ---------------------------- >> > >> > Key: BIT-1236 >> > URL: https://bro-tracker.atlassian.net/browse/BIT-1236 >> > Project: Bro Issue Tracker >> > Issue Type: Improvement >> > Components: Bro >> > Affects Versions: git/master >> > Reporter: Jon Siwek >> > Assignee: Robin Sommer >> > Fix For: 2.4 >> > >> > >> > This branch is in bro and bro-testing-private. >> > The goal is the same as https://github.com/bro/bro/pull/11, but I have >> it flip roles at an even earlier point in the code path or else I notice >> some inconsistencies in things like connection history strings or the >> connsize analyzer counters (which were probably also issues w/ the old >> flipping method). >> >> >> >> -- >> This message was sent by Atlassian JIRA >> (v6.4-OD-04-006#64001) >> _______________________________________________ >> bro-dev mailing list >> bro-dev at bro.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev >> > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140825/d9abe282/attachment-0001.html From noreply at bro.org Tue Aug 26 00:00:15 2014 From: noreply at bro.org (Merge Tracker) Date: Tue, 26 Aug 2014 00:00:15 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408260700.s7Q70F9F011305@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ---------- ------------ ---------- ------------- ---------- -------------------------------- BIT-1236 [1] Bro Jon Siwek Robin Sommer 2014-08-25 2.4 Normal topic/jsiwek/flip-on-syn-ack [2] Open GitHub Pull Requests ========================= Issue Component User Updated Title ------- ----------- --------------- ---------- -------------------------------------- #13 [3] bro jimmyjones2 [4] 2014-08-23 Better documentation for sub_bytes [5] [1] BIT-1236 https://bro-tracker.atlassian.net/browse/BIT-1236 [2] flip-on-syn-ack https://github.com/bro/bro/tree/topic/jsiwek/flip-on-syn-ack [3] Pull Request #13 https://github.com/bro/bro/pull/13 [4] jimmyjones2 https://github.com/jimmyjones2 [5] Merge Pull Request #13 with git pull --no-ff --no-commit https://github.com/jimmyjones2/bro.git topic/jimmyjones2/string-doc From vlad at grigorescu.org Tue Aug 26 15:02:26 2014 From: vlad at grigorescu.org (Vlad Grigorescu) Date: Tue, 26 Aug 2014 17:02:26 -0500 Subject: [Bro-Dev] [JIRA] (BIT-1236) topic/jsiwek/flip-on-syn-ack In-Reply-To: <91CCA31E-6AF5-4DD9-950D-6C01205FEFD7@illinois.edu> References: <91CCA31E-6AF5-4DD9-950D-6C01205FEFD7@illinois.edu> Message-ID: Thanks, Jon. Here's a PCAP with an example. I've anonymized the IPs, so it can be shared publicly/used as a test if desired. It does look like the first connection wasn't torn down in a completely normal way - if I run just that connection through Bro, conn_state is S3, and there are some missed bytes. Unfortunately, this is a pretty common occurrence when we're being scanned - traffic spikes, causing Bro to miss more bytes, leading to more of these incorrect connections. The specific issue is that the jump in seq numbers between the first and second connection cause Bro to think that a lot of traffic was simply missed. This leads to false positives with the SSH heuristic, since now the byte total is over the threshold. Digging into this, I realize it wasn't as closely related to this ticket as I thought, so let me know if I should file a new ticket for this. --Vlad On Mon, Aug 25, 2014 at 5:04 PM, Siwek, Jon wrote: > > On Aug 25, 2014, at 4:40 PM, Vlad Grigorescu wrote: > > > Does it makes sense that following a connection teardown, if a SYN-ACK > is seen, a new connection begins, instead of using the existing connection? > I can probably grab a PCAP if necessary. > > Actually, I?m thinking it may already work like you expect in many > ?normal? situations. One special case I can remember (there may be others) > is that Bro may defer closing out a connection even if it sees the teardown > control packets when it thinks it may be possible to fill in a content gap > (i.e. it thinks there?s packets coming in out of order, but maybe in your > case it?s just never seen at all). If that doesn?t fit with what you saw > and you?ve got a pcap you can send me, I can try to make sense of it. > > - Jon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140826/7ab7a99c/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: bug_ssh_8_1_14_anon.pcap Type: application/octet-stream Size: 7332 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140826/7ab7a99c/attachment.obj From jsiwek at illinois.edu Tue Aug 26 15:49:40 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 26 Aug 2014 22:49:40 +0000 Subject: [Bro-Dev] [JIRA] (BIT-1236) topic/jsiwek/flip-on-syn-ack In-Reply-To: References: <91CCA31E-6AF5-4DD9-950D-6C01205FEFD7@illinois.edu> Message-ID: > On Aug 26, 2014, at 5:02 PM, Vlad Grigorescu wrote: > > The specific issue is that the jump in seq numbers between the first and second connection cause Bro to think that a lot of traffic was simply missed. This leads to false positives with the SSH heuristic, since now the byte total is over the threshold. As a workaround you may be able to filter out such cases by checking whether connection records report missing data and a history string with more than one handshake? > Digging into this, I realize it wasn't as closely related to this ticket as I thought, so let me know if I should file a new ticket for this. Yeah, make a ticket. - Jon From noreply at bro.org Wed Aug 27 00:00:17 2014 From: noreply at bro.org (Merge Tracker) Date: Wed, 27 Aug 2014 00:00:17 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408270700.s7R70H5u014408@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ---------- ------------ ---------- ------------- ---------- -------------------------------- BIT-1236 [1] Bro Jon Siwek Robin Sommer 2014-08-25 2.4 Normal topic/jsiwek/flip-on-syn-ack [2] [1] BIT-1236 https://bro-tracker.atlassian.net/browse/BIT-1236 [2] flip-on-syn-ack https://github.com/bro/bro/tree/topic/jsiwek/flip-on-syn-ack From jira at bro-tracker.atlassian.net Wed Aug 27 08:24:08 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 27 Aug 2014 10:24:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1235) HTTP multipart POST request alters file contents In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702#comment-17702 ] Jon Siwek commented on BIT-1235: -------------------------------- The fix I came up with is in the topic/jsiwek/bit-1235 branch, are you able to verify your own test cases work better with the change? My own testing seems to show things are only improved and nothing got worse, but as you've seen, some code/comments behind how this works are worrying (thanks for taking a crack at figuring it out). > HTTP multipart POST request alters file contents > ------------------------------------------------ > > Key: BIT-1235 > URL: https://bro-tracker.atlassian.net/browse/BIT-1235 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Environment: CentOS 6.5, file extract analyzer > Reporter: Brian O'Berry > Attachments: bro-2.3-HTTP.patch, gdb.log, upload-api-http.pcap > > > HTTP POST multipart processing converts bare CR or LF chars to CRLF pairs, corrupting most files when extracted with Files::ANALYZER_EXTRACT. This is clear in the attached gdb.log, which has a backtrace that shows a buffer with the start of a PDF file entering MIME/HTTP entity processing at frame 25, and emerging with LF chars converted to CRLF at frame 6. > Also attached are the pcap file associated with the backtrace, and an initial patch that we've barely begun to test. A point of concern with the patch is that it changes a weird.log entry from "line_terminated_with_single_CR" to "http_no_crlf_in_header_list". It does enable Files::ANALYZER_EXTRACT to correctly extract the PDF file from the attached pcap. > Please let me know if we can provide anything else to help with this. -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Wed Aug 27 08:47:09 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 27 Aug 2014 10:47:09 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1235) HTTP multipart POST request alters file contents In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1235: --------------------------- Fix Version/s: 2.4 > HTTP multipart POST request alters file contents > ------------------------------------------------ > > Key: BIT-1235 > URL: https://bro-tracker.atlassian.net/browse/BIT-1235 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Environment: CentOS 6.5, file extract analyzer > Reporter: Brian O'Berry > Fix For: 2.4 > > Attachments: bro-2.3-HTTP.patch, gdb.log, upload-api-http.pcap > > > HTTP POST multipart processing converts bare CR or LF chars to CRLF pairs, corrupting most files when extracted with Files::ANALYZER_EXTRACT. This is clear in the attached gdb.log, which has a backtrace that shows a buffer with the start of a PDF file entering MIME/HTTP entity processing at frame 25, and emerging with LF chars converted to CRLF at frame 6. > Also attached are the pcap file associated with the backtrace, and an initial patch that we've barely begun to test. A point of concern with the patch is that it changes a weird.log entry from "line_terminated_with_single_CR" to "http_no_crlf_in_header_list". It does enable Files::ANALYZER_EXTRACT to correctly extract the PDF file from the attached pcap. > Please let me know if we can provide anything else to help with this. -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Wed Aug 27 08:57:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 27 Aug 2014 10:57:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1236) topic/jsiwek/flip-on-syn-ack In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17800#comment-17800 ] Jon Siwek commented on BIT-1236: -------------------------------- It was pointed out in a github comment that flipping roles on just one packet may be problematic, so maybe don't consider this change for merging. Or do. I don't know. > topic/jsiwek/flip-on-syn-ack > ---------------------------- > > Key: BIT-1236 > URL: https://bro-tracker.atlassian.net/browse/BIT-1236 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Robin Sommer > Fix For: 2.4 > > > This branch is in bro and bro-testing-private. > The goal is the same as https://github.com/bro/bro/pull/11, but I have it flip roles at an even earlier point in the code path or else I notice some inconsistencies in things like connection history strings or the connsize analyzer counters (which were probably also issues w/ the old flipping method). -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Wed Aug 27 08:57:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 27 Aug 2014 10:57:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1236) topic/jsiwek/flip-on-syn-ack In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1236: --------------------------- Status: Open (was: Merge Request) > topic/jsiwek/flip-on-syn-ack > ---------------------------- > > Key: BIT-1236 > URL: https://bro-tracker.atlassian.net/browse/BIT-1236 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Robin Sommer > Fix For: 2.4 > > > This branch is in bro and bro-testing-private. > The goal is the same as https://github.com/bro/bro/pull/11, but I have it flip roles at an even earlier point in the code path or else I notice some inconsistencies in things like connection history strings or the connsize analyzer counters (which were probably also issues w/ the old flipping method). -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Wed Aug 27 08:58:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 27 Aug 2014 10:58:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1236) topic/jsiwek/flip-on-syn-ack In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1236: --------------------------- Fix Version/s: (was: 2.4) > topic/jsiwek/flip-on-syn-ack > ---------------------------- > > Key: BIT-1236 > URL: https://bro-tracker.atlassian.net/browse/BIT-1236 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > Assignee: Robin Sommer > > This branch is in bro and bro-testing-private. > The goal is the same as https://github.com/bro/bro/pull/11, but I have it flip roles at an even earlier point in the code path or else I notice some inconsistencies in things like connection history strings or the connsize analyzer counters (which were probably also issues w/ the old flipping method). -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Wed Aug 27 09:02:07 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 27 Aug 2014 11:02:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1236) topic/jsiwek/flip-on-syn-ack In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1236: --------------------------- Assignee: (was: Robin Sommer) > topic/jsiwek/flip-on-syn-ack > ---------------------------- > > Key: BIT-1236 > URL: https://bro-tracker.atlassian.net/browse/BIT-1236 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Jon Siwek > > This branch is in bro and bro-testing-private. > The goal is the same as https://github.com/bro/bro/pull/11, but I have it flip roles at an even earlier point in the code path or else I notice some inconsistencies in things like connection history strings or the connsize analyzer counters (which were probably also issues w/ the old flipping method). -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Wed Aug 27 12:54:07 2014 From: jira at bro-tracker.atlassian.net (Peter Kaloroumakis (JIRA)) Date: Wed, 27 Aug 2014 14:54:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1237) Bro script declaration ordering In-Reply-To: References: Message-ID: Peter Kaloroumakis created BIT-1237: --------------------------------------- Summary: Bro script declaration ordering Key: BIT-1237 URL: https://bro-tracker.atlassian.net/browse/BIT-1237 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: 2.3 Environment: Bro con training VM Reporter: Peter Kaloroumakis Priority: Trivial During one of the scripting exercises I noticed odd behavior with items declared in the global scope: ############################# error.bro not working: ------------------------------------------------ local test_var = "test_var"; function test_1() { print "test_1"; } print test_var; test_1(); >>> Output: error in ./test.bro, line 3: syntax error, at or near "test_1" ############################# working.bro working: ------------------------------------------------ function test_1() { print "test_1"; } local test_var = "test_var"; print test_var; test_1(); >>> Output: test_var test_1 ############################# To declare a function, bro 2.3 forced me to do it at the top of the file. On the exercise with the redef of the grid ftp size variable I noticed the same issue with redef, it required me to put the redef at the very top of the file. Robin asked me to open a ticket and mentioned this was low priority. -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Wed Aug 27 13:35:07 2014 From: jira at bro-tracker.atlassian.net (Brian O'Berry (JIRA)) Date: Wed, 27 Aug 2014 15:35:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1235) HTTP multipart POST request alters file contents In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17801#comment-17801 ] Brian O'Berry commented on BIT-1235: ------------------------------------ I ran your topic branch and got perfect results! We have more testing to do and will update this if we see any issues over the next couple weeks. I went for the quick fix and didn't think of SetCRLFAsEOL(0), but it's clear I need to learn the analyzer parent/child processing flow to help in this area. Thanks very much for the fast response, and I'll keep you posted! > HTTP multipart POST request alters file contents > ------------------------------------------------ > > Key: BIT-1235 > URL: https://bro-tracker.atlassian.net/browse/BIT-1235 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Environment: CentOS 6.5, file extract analyzer > Reporter: Brian O'Berry > Fix For: 2.4 > > Attachments: bro-2.3-HTTP.patch, gdb.log, upload-api-http.pcap > > > HTTP POST multipart processing converts bare CR or LF chars to CRLF pairs, corrupting most files when extracted with Files::ANALYZER_EXTRACT. This is clear in the attached gdb.log, which has a backtrace that shows a buffer with the start of a PDF file entering MIME/HTTP entity processing at frame 25, and emerging with LF chars converted to CRLF at frame 6. > Also attached are the pcap file associated with the backtrace, and an initial patch that we've barely begun to test. A point of concern with the patch is that it changes a weird.log entry from "line_terminated_with_single_CR" to "http_no_crlf_in_header_list". It does enable Files::ANALYZER_EXTRACT to correctly extract the PDF file from the attached pcap. > Please let me know if we can provide anything else to help with this. -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Wed Aug 27 14:58:08 2014 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 27 Aug 2014 16:58:08 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1237) Bro script declaration ordering In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17802#comment-17802 ] Jon Siwek commented on BIT-1237: -------------------------------- Thanks. It is a bit of a quirk of the parser -- it expects no statement to precede a declaration (at the global scope). In this case, the function definition and redef are types of declarations while locals are considered statements. For now, maybe a habit to pick up in order to avoid the situation is only using locals inside functions/events. > Bro script declaration ordering > ------------------------------- > > Key: BIT-1237 > URL: https://bro-tracker.atlassian.net/browse/BIT-1237 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Affects Versions: 2.3 > Environment: Bro con training VM > Reporter: Peter Kaloroumakis > Priority: Trivial > Labels: BroScript > > During one of the scripting exercises I noticed odd behavior with items declared in the global scope: > ############################# error.bro > not working: > ------------------------------------------------ > local test_var = "test_var"; > function test_1() > { > print "test_1"; > } > print test_var; > test_1(); > >>> Output: > error in ./test.bro, line 3: syntax error, at or near "test_1" > ############################# working.bro > working: > ------------------------------------------------ > function test_1() > { > print "test_1"; > } > local test_var = "test_var"; > print test_var; > test_1(); > >>> Output: > test_var > test_1 > ############################# > To declare a function, bro 2.3 forced me to do it at the top of the file. On the exercise with the redef of the grid ftp size variable I noticed the same issue with redef, it required me to put the redef at the very top of the file. > Robin asked me to open a ticket and mentioned this was low priority. -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jsiwek at illinois.edu Thu Aug 28 11:45:30 2014 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 28 Aug 2014 18:45:30 +0000 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/improve_comm_loop: Remove timeouts from remote communication loop. (675fba3) In-Reply-To: <201408281836.s7SIaNOn010555@bro-ids.icir.org> References: <201408281836.s7SIaNOn010555@bro-ids.icir.org> Message-ID: Feedback and/or help testing is welcome. - Jon On Aug 28, 2014, at 1:36 PM, Jonathan Siwek wrote: > Repository : ssh://git at bro-ids.icir.org/bro > > On branch : topic/jsiwek/improve_comm_loop > Link : https://github.com/bro/bro/commit/675fba3fdee0a391cfb6fc52d07b08caaca96c76 > >> --------------------------------------------------------------- > > commit 675fba3fdee0a391cfb6fc52d07b08caaca96c76 > Author: Jon Siwek > Date: Thu Aug 28 13:13:30 2014 -0500 > > Remove timeouts from remote communication loop. > > The select() now blocks until there's work to do instead of relying on a > small timeout value which can cause unproductive use of cpu cycles. > > >> --------------------------------------------------------------- > > 675fba3fdee0a391cfb6fc52d07b08caaca96c76 > src/CMakeLists.txt | 2 ++ > src/ChunkedIO.cc | 46 +++++++++++++++++++++++++++- > src/ChunkedIO.h | 16 +++++++++- > src/DNS_Mgr.cc | 5 +-- > src/DNS_Mgr.h | 3 +- > src/Flare.cc | 29 ++++++++++++++++++ > src/Flare.h | 45 +++++++++++++++++++++++++++ > src/FlowSrc.cc | 5 +-- > src/FlowSrc.h | 3 +- > src/IOSource.cc | 47 +++++++++++++++++++++------- > src/IOSource.h | 13 +++++--- > src/Pipe.cc | 79 ++++++++++++++++++++++++++++++++++++++++++++++++ > src/Pipe.h | 57 ++++++++++++++++++++++++++++++++++ > src/PktSrc.cc | 5 +-- > src/PktSrc.h | 3 +- > src/RemoteSerializer.cc | 47 +++++++++++++--------------- > src/RemoteSerializer.h | 3 +- > src/Serializer.cc | 5 +-- > src/Serializer.h | 3 +- > src/threading/Manager.cc | 3 +- > src/threading/Manager.h | 3 +- > 21 files changed, 364 insertions(+), 58 deletions(-) > > diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt > index 04867b7..3764533 100644 > --- a/src/CMakeLists.txt > +++ b/src/CMakeLists.txt > @@ -279,6 +279,7 @@ set(bro_SRCS > EventRegistry.cc > Expr.cc > File.cc > + Flare.cc > FlowSrc.cc > Frag.cc > Frame.cc > @@ -299,6 +300,7 @@ set(bro_SRCS > OSFinger.cc > PacketFilter.cc > PersistenceSerializer.cc > + Pipe.cc > PktSrc.cc > PolicyFile.cc > PrefixTable.cc > diff --git a/src/ChunkedIO.cc b/src/ChunkedIO.cc > index 54e2e59..a94eb98 100644 > --- a/src/ChunkedIO.cc > +++ b/src/ChunkedIO.cc > @@ -210,6 +210,7 @@ bool ChunkedIOFd::WriteChunk(Chunk* chunk, bool partial) > else > pending_head = pending_tail = q; > > + write_flare.Fire(); > return Flush(); > } > > @@ -232,6 +233,7 @@ bool ChunkedIOFd::PutIntoWriteBuffer(Chunk* chunk) > write_len += len; > > delete chunk; > + write_flare.Fire(); > > if ( network_time - last_flush > 0.005 ) > FlushWriteBuffer(); > @@ -269,6 +271,10 @@ bool ChunkedIOFd::FlushWriteBuffer() > if ( unsigned(written) == len ) > { > write_pos = write_len = 0; > + > + if ( ! pending_head ) > + write_flare.Extinguish(); > + > return true; > } > > @@ -318,7 +324,12 @@ bool ChunkedIOFd::Flush() > } > } > > - return FlushWriteBuffer(); > + bool rval = FlushWriteBuffer(); > + > + if ( ! pending_head && write_len == 0 ) > + write_flare.Extinguish(); > + > + return rval; > } > > uint32 ChunkedIOFd::ChunkAvailable() > @@ -394,6 +405,9 @@ bool ChunkedIOFd::Read(Chunk** chunk, bool may_block) > #ifdef DEBUG_COMMUNICATION > AddToBuffer("", true); > #endif > + if ( ! ChunkAvailable() ) > + read_flare.Extinguish(); > + > return false; > } > > @@ -402,9 +416,15 @@ bool ChunkedIOFd::Read(Chunk** chunk, bool may_block) > #ifdef DEBUG_COMMUNICATION > AddToBuffer("", true); > #endif > + read_flare.Extinguish(); > return true; > } > > + if ( ChunkAvailable() ) > + read_flare.Fire(); > + else > + read_flare.Extinguish(); > + > #ifdef DEBUG > if ( *chunk ) > DBG_LOG(DBG_CHUNKEDIO, "read of size %d %s[%s]", > @@ -481,6 +501,9 @@ bool ChunkedIOFd::ReadChunk(Chunk** chunk, bool may_block) > read_pos = 0; > read_len = bytes_left; > > + if ( ! ChunkAvailable() ) > + read_flare.Extinguish(); > + > // If allowed, wait a bit for something to read. > if ( may_block ) > { > @@ -607,6 +630,14 @@ bool ChunkedIOFd::IsFillingUp() > return stats.pending > MAX_BUFFERED_CHUNKS_SOFT; > } > > +std::vector ChunkedIOFd::FdSupplements() const > + { > + std::vector rval; > + rval.push_back(write_flare.FD()); > + rval.push_back(read_flare.FD()); > + return rval; > + } > + > void ChunkedIOFd::Clear() > { > while ( pending_head ) > @@ -618,6 +649,9 @@ void ChunkedIOFd::Clear() > } > > pending_head = pending_tail = 0; > + > + if ( write_len == 0 ) > + write_flare.Extinguish(); > } > > const char* ChunkedIOFd::Error() > @@ -830,6 +864,7 @@ bool ChunkedIOSSL::Write(Chunk* chunk) > else > write_head = write_tail = q; > > + write_flare.Fire(); > Flush(); > return true; > } > @@ -935,6 +970,7 @@ bool ChunkedIOSSL::Flush() > write_state = LEN; > } > > + write_flare.Extinguish(); > return true; > } > > @@ -1104,6 +1140,13 @@ bool ChunkedIOSSL::IsFillingUp() > return false; > } > > +std::vector ChunkedIOSSL::FdSupplements() const > + { > + std::vector rval; > + rval.push_back(write_flare.FD()); > + return rval; > + } > + > void ChunkedIOSSL::Clear() > { > while ( write_head ) > @@ -1114,6 +1157,7 @@ void ChunkedIOSSL::Clear() > write_head = next; > } > write_head = write_tail = 0; > + write_flare.Extinguish(); > } > > const char* ChunkedIOSSL::Error() > diff --git a/src/ChunkedIO.h b/src/ChunkedIO.h > index a9865e4..c640e52 100644 > --- a/src/ChunkedIO.h > +++ b/src/ChunkedIO.h > @@ -6,8 +6,9 @@ > #include "config.h" > #include "List.h" > #include "util.h" > - > +#include "Flare.h" > #include > +#include > > #ifdef NEED_KRB5_H > # include > @@ -95,6 +96,11 @@ public: > // Returns underlying fd if available, -1 otherwise. > virtual int Fd() { return -1; } > > + // Returns supplementary file descriptors that become read-ready in order > + // to signal that there is some work that can be performed. > + virtual std::vector FdSupplements() const > + { return std::vector(); } > + > // Makes sure that no additional protocol data is written into > // the output stream. If this is activated, the output cannot > // be read again by any of these classes! > @@ -177,6 +183,7 @@ public: > virtual void Clear(); > virtual bool Eof() { return eof; } > virtual int Fd() { return fd; } > + virtual std::vector FdSupplements() const; > virtual void Stats(char* buffer, int length); > > private: > @@ -240,6 +247,8 @@ private: > ChunkQueue* pending_tail; > > pid_t pid; > + bro::Flare write_flare; > + bro::Flare read_flare; > }; > > // Chunked I/O using an SSL connection. > @@ -262,6 +271,7 @@ public: > virtual void Clear(); > virtual bool Eof() { return eof; } > virtual int Fd() { return socket; } > + virtual std::vector FdSupplements() const; > virtual void Stats(char* buffer, int length); > > private: > @@ -303,6 +313,8 @@ private: > > // One SSL for all connections. > static SSL_CTX* ctx; > + > + bro::Flare write_flare; > }; > > #include > @@ -328,6 +340,8 @@ public: > > virtual bool Eof() { return io->Eof(); } > virtual int Fd() { return io->Fd(); } > + virtual std::vector FdSupplements() const > + { return io->FdSupplements(); } > virtual void Stats(char* buffer, int length); > > void EnableCompression(int level) > diff --git a/src/DNS_Mgr.cc b/src/DNS_Mgr.cc > index 9188d61..9fb5c8b 100644 > --- a/src/DNS_Mgr.cc > +++ b/src/DNS_Mgr.cc > @@ -1217,9 +1217,10 @@ void DNS_Mgr::IssueAsyncRequests() > } > } > > -void DNS_Mgr::GetFds(int* read, int* write, int* except) > +void DNS_Mgr::GetFds(std::vector* read, std::vector* write, > + std::vector* except) > { > - *read = nb_dns_fd(nb_dns); > + read->push_back(nb_dns_fd(nb_dns)); > } > > double DNS_Mgr::NextTimestamp(double* network_time) > diff --git a/src/DNS_Mgr.h b/src/DNS_Mgr.h > index 7864505..fa19914 100644 > --- a/src/DNS_Mgr.h > +++ b/src/DNS_Mgr.h > @@ -132,7 +132,8 @@ protected: > void DoProcess(bool flush); > > // IOSource interface. > - virtual void GetFds(int* read, int* write, int* except); > + virtual void GetFds(std::vector* read, std::vector* write, > + std::vector* except); > virtual double NextTimestamp(double* network_time); > virtual void Process(); > virtual const char* Tag() { return "DNS_Mgr"; } > diff --git a/src/Flare.cc b/src/Flare.cc > new file mode 100644 > index 0000000..8a0418f > --- /dev/null > +++ b/src/Flare.cc > @@ -0,0 +1,29 @@ > +// See the file "COPYING" in the main distribution directory for copyright. > + > +#include "Flare.h" > +#include "util.h" > +#include > +#include > +#include > + > +using namespace bro; > + > +Flare::Flare() > + : pipe(FD_CLOEXEC, FD_CLOEXEC, O_NONBLOCK, O_NONBLOCK) > + { > + } > + > +void Flare::Fire() > + { > + char tmp; > + safe_write(pipe.WriteFD(), &tmp, 1); > + } > + > +void Flare::Extinguish() > + { > + char tmp[256]; > + > + for ( ; ; ) > + if ( read(pipe.ReadFD(), &tmp, sizeof(tmp)) == -1 && errno == EAGAIN ) > + break; > + } > diff --git a/src/Flare.h b/src/Flare.h > new file mode 100644 > index 0000000..4e63788 > --- /dev/null > +++ b/src/Flare.h > @@ -0,0 +1,45 @@ > +// See the file "COPYING" in the main distribution directory for copyright. > + > +#ifndef BRO_FLARE_H > +#define BRO_FLARE_H > + > +#include "Pipe.h" > + > +namespace bro { > + > +class Flare { > +public: > + > + /** > + * Create a flare object that can be used to signal a "ready" status via > + * a file descriptor that may be integrated with select(), poll(), etc. > + * Not thread-safe, but that should only require Fire()/Extinguish() calls > + * to be made mutually exclusive (across all copies of a Flare). > + */ > + Flare(); > + > + /** > + * @return a file descriptor that will become ready if the flare has been > + * Fire()'d and not yet Extinguished()'d. > + */ > + int FD() const > + { return pipe.ReadFD(); } > + > + /** > + * Put the object in the "ready" state. > + */ > + void Fire(); > + > + /** > + * Take the object out of the "ready" state. > + */ > + void Extinguish(); > + > +private: > + > + Pipe pipe; > +}; > + > +} // namespace bro > + > +#endif // BRO_FLARE_H > diff --git a/src/FlowSrc.cc b/src/FlowSrc.cc > index 8eed94f..4999d9c 100644 > --- a/src/FlowSrc.cc > +++ b/src/FlowSrc.cc > @@ -28,10 +28,11 @@ FlowSrc::~FlowSrc() > delete netflow_analyzer; > } > > -void FlowSrc::GetFds(int* read, int* write, int* except) > +void FlowSrc::GetFds(std::vector* read, std::vector* write, > + std::vector* except) > { > if ( selectable_fd >= 0 ) > - *read = selectable_fd; > + read->push_back(selectable_fd); > } > > double FlowSrc::NextTimestamp(double* network_time) > diff --git a/src/FlowSrc.h b/src/FlowSrc.h > index 03dda27..ee92760 100644 > --- a/src/FlowSrc.h > +++ b/src/FlowSrc.h > @@ -34,7 +34,8 @@ public: > > // IOSource interface: > bool IsReady(); > - void GetFds(int* read, int* write, int* except); > + void GetFds(std::vector* read, std::vector* write, > + std::vector* except); > double NextTimestamp(double* network_time); > void Process(); > > diff --git a/src/IOSource.cc b/src/IOSource.cc > index d47007c..540b797 100644 > --- a/src/IOSource.cc > +++ b/src/IOSource.cc > @@ -24,6 +24,15 @@ void IOSourceRegistry::RemoveAll() > dont_counts = sources.size(); > } > > +static void fd_vector_set(const std::vector& fds, fd_set* set, int* max) > + { > + for ( size_t i = 0; i < fds.size(); ++i ) > + { > + FD_SET(fds[i], set); > + *max = ::max(fds[i], *max); > + } > + } > + > IOSource* IOSourceRegistry::FindSoonest(double* ts) > { > // Remove sources which have gone dry. For simplicity, we only > @@ -94,16 +103,14 @@ IOSource* IOSourceRegistry::FindSoonest(double* ts) > // be ready. > continue; > > - src->fd_read = src->fd_write = src->fd_except = 0; > + src->fd_read.clear(); > + src->fd_write.clear(); > + src->fd_except.clear(); > src->src->GetFds(&src->fd_read, &src->fd_write, &src->fd_except); > > - FD_SET(src->fd_read, &fd_read); > - FD_SET(src->fd_write, &fd_write); > - FD_SET(src->fd_except, &fd_except); > - > - maxx = max(src->fd_read, maxx); > - maxx = max(src->fd_write, maxx); > - maxx = max(src->fd_except, maxx); > + fd_vector_set(src->fd_read, &fd_read, &maxx); > + fd_vector_set(src->fd_write, &fd_write, &maxx); > + fd_vector_set(src->fd_except, &fd_except, &maxx); > } > > // We can't block indefinitely even when all sources are dry: > @@ -143,9 +150,7 @@ IOSource* IOSourceRegistry::FindSoonest(double* ts) > if ( ! src->src->IsIdle() ) > continue; > > - if ( FD_ISSET(src->fd_read, &fd_read) || > - FD_ISSET(src->fd_write, &fd_write) || > - FD_ISSET(src->fd_except, &fd_except) ) > + if ( src->Ready(&fd_read, &fd_write, &fd_except) ) > { > double local_network_time = 0; > double ts = src->src->NextTimestamp(&local_network_time); > @@ -174,3 +179,23 @@ void IOSourceRegistry::Register(IOSource* src, bool dont_count) > ++dont_counts; > return sources.push_back(s); > } > + > +static bool fd_vector_ready(const std::vector& fds, fd_set* set) > + { > + for ( size_t i = 0; i < fds.size(); ++i ) > + if ( FD_ISSET(fds[i], set) ) > + return true; > + > + return false; > + } > + > +bool IOSourceRegistry::Source::Ready(fd_set* read, fd_set* write, > + fd_set* except) const > + { > + if ( fd_vector_ready(fd_read, read) || > + fd_vector_ready(fd_write, write) || > + fd_vector_ready(fd_except, except) ) > + return true; > + > + return false; > + } > diff --git a/src/IOSource.h b/src/IOSource.h > index db50bbd..3da70af 100644 > --- a/src/IOSource.h > +++ b/src/IOSource.h > @@ -4,6 +4,8 @@ > #define iosource_h > > #include > +#include > +#include > #include "Timer.h" > > using namespace std; > @@ -22,7 +24,8 @@ public: > > // Returns select'able fds (leaves args untouched if we don't have > // selectable fds). > - virtual void GetFds(int* read, int* write, int* except) = 0; > + virtual void GetFds(std::vector* read, std::vector* write, > + std::vector* except) = 0; > > // The following two methods are only called when either IsIdle() > // returns false or select() on one of the fds indicates that there's > @@ -89,9 +92,11 @@ protected: > > struct Source { > IOSource* src; > - int fd_read; > - int fd_write; > - int fd_except; > + std::vector fd_read; > + std::vector fd_write; > + std::vector fd_except; > + > + bool Ready(fd_set* read, fd_set* write, fd_set* except) const; > }; > > typedef list SourceList; > diff --git a/src/Pipe.cc b/src/Pipe.cc > new file mode 100644 > index 0000000..51298d0 > --- /dev/null > +++ b/src/Pipe.cc > @@ -0,0 +1,79 @@ > +// See the file "COPYING" in the main distribution directory for copyright. > + > +#include "Pipe.h" > +#include "Reporter.h" > +#include > +#include > +#include > +#include > + > +using namespace bro; > + > +static void pipe_fail(int eno) > + { > + char tmp[256]; > + strerror_r(eno, tmp, sizeof(tmp)); > + reporter->FatalError("Pipe failure: %s", tmp); > + } > + > +static void set_flags(int fd, int flags) > + { > + if ( flags ) > + fcntl(fd, F_SETFD, fcntl(fd, F_GETFD) | flags); > + } > + > +static void set_status_flags(int fd, int flags) > + { > + if ( flags ) > + fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | flags); > + } > + > +static int dup_or_fail(int fd, int flags) > + { > + int rval = dup(fd); > + > + if ( rval < 0 ) > + pipe_fail(errno); > + > + set_flags(fd, flags); > + return rval; > + } > + > +Pipe::Pipe(int flags0, int flags1, int status_flags0, int status_flags1) > + { > + // pipe2 can set flags atomically, but not yet available everywhere. > + if ( ::pipe(fds) ) > + pipe_fail(errno); > + > + flags[0] = flags0; > + flags[1] = flags1; > + > + set_flags(fds[0], flags[0]); > + set_flags(fds[1], flags[1]); > + set_status_flags(fds[0], status_flags0); > + set_status_flags(fds[1], status_flags1); > + } > + > +Pipe::~Pipe() > + { > + close(fds[0]); > + close(fds[1]); > + } > + > +Pipe::Pipe(const Pipe& other) > + { > + fds[0] = dup_or_fail(other.fds[0], other.flags[0]); > + fds[1] = dup_or_fail(other.fds[1], other.flags[1]); > + } > + > +Pipe& Pipe::operator=(const Pipe& other) > + { > + if ( this == &other ) > + return *this; > + > + close(fds[0]); > + close(fds[1]); > + fds[0] = dup_or_fail(other.fds[0], other.flags[0]); > + fds[1] = dup_or_fail(other.fds[1], other.flags[1]); > + return *this; > + } > diff --git a/src/Pipe.h b/src/Pipe.h > new file mode 100644 > index 0000000..493169e > --- /dev/null > +++ b/src/Pipe.h > @@ -0,0 +1,57 @@ > +// See the file "COPYING" in the main distribution directory for copyright. > + > +#ifndef BRO_PIPE_H > +#define BRO_PIPE_H > + > +namespace bro { > + > +class Pipe { > +public: > + > + /** > + * Create a pair of file descriptors via pipe(), or aborts if it cannot. > + * @param flags0 file descriptor flags to set on read end of pipe. > + * @param flags1 file descriptor flags to set on write end of pipe. > + * @param status_flags0 descriptor status flags to set on read end of pipe. > + * @param status_flags1 descriptor status flags to set on write end of pipe. > + */ > + Pipe(int flags0 = 0, int flags1 = 0, int status_flags0 = 0, > + int status_flags1 = 0); > + > + /** > + * Close the pair of file descriptors owned by the object. > + */ > + ~Pipe(); > + > + /** > + * Make a copy of another Pipe object (file descriptors are dup'd). > + */ > + Pipe(const Pipe& other); > + > + /** > + * Assign a Pipe object by closing file descriptors and duping those of > + * the other. > + */ > + Pipe& operator=(const Pipe& other); > + > + /** > + * @return the file descriptor associated with the read-end of the pipe. > + */ > + int ReadFD() const > + { return fds[0]; } > + > + /** > + * @return the file descriptor associated with the write-end of the pipe. > + */ > + int WriteFD() const > + { return fds[1]; } > + > +private: > + > + int fds[2]; > + int flags[2]; > +}; > + > +} // namespace bro > + > +#endif // BRO_PIPE_H > diff --git a/src/PktSrc.cc b/src/PktSrc.cc > index b5ac3a5..04b7b7d 100644 > --- a/src/PktSrc.cc > +++ b/src/PktSrc.cc > @@ -51,7 +51,8 @@ PktSrc::~PktSrc() > delete [] readfile; > } > > -void PktSrc::GetFds(int* read, int* write, int* except) > +void PktSrc::GetFds(std::vector* read, std::vector* write, > + std::vector* except) > { > if ( pseudo_realtime ) > { > @@ -62,7 +63,7 @@ void PktSrc::GetFds(int* read, int* write, int* except) > } > > if ( selectable_fd >= 0 ) > - *read = selectable_fd; > + read->push_back(selectable_fd); > } > > int PktSrc::ExtractNextPacket() > diff --git a/src/PktSrc.h b/src/PktSrc.h > index 70eef4d..0d4be12 100644 > --- a/src/PktSrc.h > +++ b/src/PktSrc.h > @@ -98,7 +98,8 @@ public: > > // IOSource interface > bool IsReady(); > - void GetFds(int* read, int* write, int* except); > + void GetFds(std::vector* read, std::vector* write, > + std::vector* except); > double NextTimestamp(double* local_network_time); > void Process(); > const char* Tag() { return "PktSrc"; } > diff --git a/src/RemoteSerializer.cc b/src/RemoteSerializer.cc > index 3e46c5a..34c5f1a 100644 > --- a/src/RemoteSerializer.cc > +++ b/src/RemoteSerializer.cc > @@ -1368,12 +1368,17 @@ void RemoteSerializer::Unregister(ID* id) > } > } > > -void RemoteSerializer::GetFds(int* read, int* write, int* except) > +void RemoteSerializer::GetFds(std::vector* read, std::vector* write, > + std::vector* except) > { > - *read = io->Fd(); > + read->push_back(io->Fd()); > + std::vector supp = io->FdSupplements(); > + > + for ( size_t i = 0; i < supp.size(); ++i ) > + read->push_back(supp[i]); > > if ( io->CanWrite() ) > - *write = io->Fd(); > + write->push_back(io->Fd()); > } > > double RemoteSerializer::NextTimestamp(double* local_network_time) > @@ -3356,6 +3361,15 @@ SocketComm::~SocketComm() > > static unsigned int first_rtime = 0; > > +static void fd_vector_set(const std::vector& fds, fd_set* set, int* max) > + { > + for ( size_t i = 0; i < fds.size(); ++i ) > + { > + FD_SET(fds[i], set); > + *max = ::max(fds[i], *max); > + } > + } > + > void SocketComm::Run() > { > first_rtime = (unsigned int) current_time(true); > @@ -3381,6 +3395,7 @@ void SocketComm::Run() > > FD_SET(io->Fd(), &fd_read); > max_fd = io->Fd(); > + fd_vector_set(io->FdSupplements(), &fd_read, &max_fd); > > loop_over_list(peers, i) > { > @@ -3389,6 +3404,7 @@ void SocketComm::Run() > FD_SET(peers[i]->io->Fd(), &fd_read); > if ( peers[i]->io->Fd() > max_fd ) > max_fd = peers[i]->io->Fd(); > + fd_vector_set(peers[i]->io->FdSupplements(), &fd_read, &max_fd); > } > else > { > @@ -3439,38 +3455,17 @@ void SocketComm::Run() > if ( ! io->IsFillingUp() && shutting_conns_down ) > shutting_conns_down = false; > > - // We cannot rely solely on select() as the there may > - // be some data left in our input/output queues. So, we use > - // a small timeout for select and check for data > - // manually afterwards. > - > static long selects = 0; > static long canwrites = 0; > - static long timeouts = 0; > > ++selects; > if ( io->CanWrite() ) > ++canwrites; > > - // FIXME: Fine-tune this (timeouts, flush, etc.) > - struct timeval small_timeout; > - small_timeout.tv_sec = 0; > - small_timeout.tv_usec = > - io->CanWrite() || io->CanRead() ? 1 : 10; > - > -#if 0 > - if ( ! io->CanWrite() ) > - usleep(10); > -#endif > - > - int a = select(max_fd + 1, &fd_read, &fd_write, &fd_except, > - &small_timeout); > - > - if ( a == 0 ) > - ++timeouts; > + int a = select(max_fd + 1, &fd_read, &fd_write, &fd_except, 0); > > if ( selects % 100000 == 0 ) > - Log(fmt("selects=%ld canwrites=%ld timeouts=%ld", selects, canwrites, timeouts)); > + Log(fmt("selects=%ld canwrites=%ld", selects, canwrites)); > > if ( a < 0 ) > // Ignore errors for now. > diff --git a/src/RemoteSerializer.h b/src/RemoteSerializer.h > index 9dbfbd9..3aa4f91 100644 > --- a/src/RemoteSerializer.h > +++ b/src/RemoteSerializer.h > @@ -140,7 +140,8 @@ public: > void Finish(); > > // Overidden from IOSource: > - virtual void GetFds(int* read, int* write, int* except); > + virtual void GetFds(std::vector* read, std::vector* write, > + std::vector* except); > virtual double NextTimestamp(double* local_network_time); > virtual void Process(); > virtual TimerMgr::Tag* GetCurrentTag(); > diff --git a/src/Serializer.cc b/src/Serializer.cc > index 36b1c74..0ea79cf 100644 > --- a/src/Serializer.cc > +++ b/src/Serializer.cc > @@ -1067,9 +1067,10 @@ void EventPlayer::GotFunctionCall(const char* name, double time, > // We don't replay function calls. > } > > -void EventPlayer::GetFds(int* read, int* write, int* except) > +void EventPlayer::GetFds(std::vector* read, std::vector* write, > + std::vector* except) > { > - *read = fd; > + read->push_back(fd); > } > > double EventPlayer::NextTimestamp(double* local_network_time) > diff --git a/src/Serializer.h b/src/Serializer.h > index 543797a..0524906 100644 > --- a/src/Serializer.h > +++ b/src/Serializer.h > @@ -355,7 +355,8 @@ public: > EventPlayer(const char* file); > virtual ~EventPlayer(); > > - virtual void GetFds(int* read, int* write, int* except); > + virtual void GetFds(std::vector* read, std::vector* write, > + std::vector* except); > virtual double NextTimestamp(double* local_network_time); > virtual void Process(); > virtual const char* Tag() { return "EventPlayer"; } > diff --git a/src/threading/Manager.cc b/src/threading/Manager.cc > index 4491cd4..c16b9f4 100644 > --- a/src/threading/Manager.cc > +++ b/src/threading/Manager.cc > @@ -65,7 +65,8 @@ void Manager::AddMsgThread(MsgThread* thread) > msg_threads.push_back(thread); > } > > -void Manager::GetFds(int* read, int* write, int* except) > +void Manager::GetFds(std::vector* read, std::vector* write, > + std::vector* except) > { > } > > diff --git a/src/threading/Manager.h b/src/threading/Manager.h > index e839749..4f0e539 100644 > --- a/src/threading/Manager.h > +++ b/src/threading/Manager.h > @@ -103,7 +103,8 @@ protected: > /** > * Part of the IOSource interface. > */ > - virtual void GetFds(int* read, int* write, int* except); > + virtual void GetFds(std::vector* read, std::vector* write, > + std::vector* except); > > /** > * Part of the IOSource interface. > > _______________________________________________ > bro-commits mailing list > bro-commits at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits > From vlad at grigorescu.org Thu Aug 28 16:16:44 2014 From: vlad at grigorescu.org (Vlad Grigorescu) Date: Thu, 28 Aug 2014 18:16:44 -0500 Subject: [Bro-Dev] Adding a LOCAL option to the Direction type? Message-ID: The Direction type (defined in base/utils/directions-and-hosts.bro) currently has directions for: - remote orig, local resp - local orig, remote resp - bidirectional ("Only one endpoint is within the locally-monitored network, meaning the connection is either outbound or inbound.") - no_direction ("This value doesn't match any connection.") Does it make sense to add LOCAL == local orig, local resp? Similarly, do we want to add EXTERNAL == remote orig, remote resp? I'm looking at this for the SSH log in particular. --Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20140828/0fa29fc8/attachment.html From jira at bro-tracker.atlassian.net Fri Aug 29 09:10:07 2014 From: jira at bro-tracker.atlassian.net (hui (JIRA)) Date: Fri, 29 Aug 2014 11:10:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hui updated BIT-1231: --------------------- Status: Merge Request (was: Open) > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Assignee: hui > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From jira at bro-tracker.atlassian.net Fri Aug 29 09:10:07 2014 From: jira at bro-tracker.atlassian.net (hui (JIRA)) Date: Fri, 29 Aug 2014 11:10:07 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1231) DNP3 Analyzer Supports for DNP3-over-UDP In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803#comment-17803 ] hui commented on BIT-1231: -------------------------- I define a base class for both tcp and udp analyzer and include all the repeated codes there. In order to call the functions defined in analyzer::Analyzer (such as Weird), I just pass the this pointer back to this base class. At the same time, I fix a bug found in the DNP3.cc. All these changes do not introduce different test baseline results. > DNP3 Analyzer Supports for DNP3-over-UDP > ---------------------------------------- > > Key: BIT-1231 > URL: https://bro-tracker.atlassian.net/browse/BIT-1231 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: hui > Assignee: hui > Labels: DNP3, analyzer > > Two major changes are made for the DNP3 analyzer > 1. Make the analyzer support both the DNP3-over-UDP and the DNP3-over-TCP. > The changes are made in DNP3.cc, DNP3.h and dpd.sig > 2. Fix a bug in the binpac codes of the DNP3 analyzer > The changes are made in dnp3-protocol.pac. The changes results in different baseline results of testing/btest/Baseline/scripts.base.protocols.dnp3.dnp3_link_only > -- This message was sent by Atlassian JIRA (v6.4-OD-04-006#64001) From noreply at bro.org Sat Aug 30 00:00:16 2014 From: noreply at bro.org (Merge Tracker) Date: Sat, 30 Aug 2014 00:00:16 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408300700.s7U70GoG002823@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ---------- ---------- ---------- ------------- ---------- ---------------------------------------- BIT-1231 [1] Bro hui hui 2014-08-29 - Normal DNP3 Analyzer Supports for DNP3-over-UDP [1] BIT-1231 https://bro-tracker.atlassian.net/browse/BIT-1231 From noreply at bro.org Sun Aug 31 00:00:19 2014 From: noreply at bro.org (Merge Tracker) Date: Sun, 31 Aug 2014 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201408310700.s7V70Jv2007360@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- ---------- ---------- ---------- ------------- ---------- ---------------------------------------- BIT-1231 [1] Bro hui hui 2014-08-29 - Normal DNP3 Analyzer Supports for DNP3-over-UDP [1] BIT-1231 https://bro-tracker.atlassian.net/browse/BIT-1231