From robin at icir.org Mon Apr 1 09:46:58 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 1 Apr 2013 09:46:58 -0700 Subject: [Bro-Dev] topic/jsiwek/file-analysis In-Reply-To: References: Message-ID: <20130401164658.GP65900@icir.org> On Fri, Mar 29, 2013 at 18:52 +0000, you wrote: > I'm claiming that this branch now has parity with all the old file > analysis functionality. Great to hear! I'll take a closer look. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From seth at icir.org Mon Apr 1 13:22:30 2013 From: seth at icir.org (Seth Hall) Date: Mon, 1 Apr 2013 16:22:30 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/plugins: Lots of cleanup and API documentation for the analyzer/* classes. (e0c4bd1) In-Reply-To: <201304012017.r31KHQ6L030565@bro-ids.icir.org> References: <201304012017.r31KHQ6L030565@bro-ids.icir.org> Message-ID: <94639495-23D5-4ACC-9AC8-7F27422DFB2E@icir.org> On Apr 1, 2013, at 4:17 PM, Robin Sommer wrote: > - event groups are broken (do we want to keep them?) I don't think we need them. The original reason for them was for your old multicore work, right? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From robin at icir.org Mon Apr 1 13:27:59 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 1 Apr 2013 13:27:59 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/plugins: Lots of cleanup and API documentation for the analyzer/* classes. (e0c4bd1) In-Reply-To: <94639495-23D5-4ACC-9AC8-7F27422DFB2E@icir.org> References: <201304012017.r31KHQ6L030565@bro-ids.icir.org> <94639495-23D5-4ACC-9AC8-7F27422DFB2E@icir.org> Message-ID: <20130401202759.GC2114@icir.org> On Mon, Apr 01, 2013 at 16:22 -0400, you wrote: > I don't think we need them. The original reason for them was for your old multicore work, right? No, it was just an alternative way of disabling chunks of analyses (for load-levels I believe, though I don't actually remember). Btw, I'd appreciate any feedback on this branch if anybody is game to take a look. The main action is in src/{analyzer,plugin,protocols}/* and scripts/base/frameworks/analyzer/* Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From seth at icir.org Mon Apr 1 13:36:03 2013 From: seth at icir.org (Seth Hall) Date: Mon, 1 Apr 2013 16:36:03 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/plugins: Lots of cleanup and API documentation for the analyzer/* classes. (e0c4bd1) In-Reply-To: <20130401202759.GC2114@icir.org> References: <201304012017.r31KHQ6L030565@bro-ids.icir.org> <94639495-23D5-4ACC-9AC8-7F27422DFB2E@icir.org> <20130401202759.GC2114@icir.org> Message-ID: <61B097D9-6042-45D7-B279-BFF509C1B7D8@icir.org> On Apr 1, 2013, at 4:27 PM, Robin Sommer wrote: > No, it was just an alternative way of disabling chunks of analyses > (for load-levels I believe, though I don't actually remember). Ahh. I think the new approach of just disabling entire analyzers will be more straightforward although I see the desire for the attribute because it could make things even more flexible but I don't think it's necessary. > Btw, I'd appreciate any feedback on this branch if anybody is game to > take a look. The main action is in src/{analyzer,plugin,protocols}/* > and scripts/base/frameworks/analyzer/* I'll start playing with it soon. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From robin at icir.org Mon Apr 1 13:36:23 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 1 Apr 2013 13:36:23 -0700 Subject: [Bro-Dev] Removing --use-binpac Message-ID: <20130401203623.GD2114@icir.org> I propose to remove this switch. We shouldn't have more than one protocol analyzer per protocol; where we currently do, we should decide if we want the C++ or BinPAC version. The --use-binpac switch is currently used here (condensed version of "grep FLAGS_use_binpac"): HTTP.h: http_event || http_stats) && !FLAGS_use_binpac; } HTTP-binpac.h: { return (http_request || http_reply) && FLAGS_use_binpac; } DNS.h: ! FLAGS_use_binpac; DNS-binpac.h: { return (dns_request || dns_full_request) && FLAGS_use_binpac; } DNS-binpac.h: { return (dns_request || dns_full_request) && FLAGS_use_binpac; } DHCP-binpac.h: { return dhcp_request && FLAGS_use_binpac; } I suggest we do the following: - Remove the binpac version of HTTP (it's by default off and hence hardly used/tested by anybody). - Remove the binpac version of DNS (same reason). - Enable DHCP even without that flag (I'm actually surprised it isn't already, there's no alternative). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From robin at icir.org Mon Apr 1 13:37:36 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 1 Apr 2013 13:37:36 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/plugins: Lots of cleanup and API documentation for the analyzer/* classes. (e0c4bd1) In-Reply-To: <61B097D9-6042-45D7-B279-BFF509C1B7D8@icir.org> References: <201304012017.r31KHQ6L030565@bro-ids.icir.org> <94639495-23D5-4ACC-9AC8-7F27422DFB2E@icir.org> <20130401202759.GC2114@icir.org> <61B097D9-6042-45D7-B279-BFF509C1B7D8@icir.org> Message-ID: <20130401203736.GE2114@icir.org> On Mon, Apr 01, 2013 at 16:36 -0400, you wrote: > Ahh. I think the new approach of just disabling entire analyzers will > be more straightforward although I see the desire for the attribute > because it could make things even more flexible but I don't think it's > necessary. Yeah, for example one could turn off http_reply but not http_request. I'm fine either way, it's not a big deal to keep ot if we think it's still useful. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From seth at icir.org Mon Apr 1 13:54:29 2013 From: seth at icir.org (Seth Hall) Date: Mon, 1 Apr 2013 16:54:29 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/plugins: Lots of cleanup and API documentation for the analyzer/* classes. (e0c4bd1) In-Reply-To: <20130401203736.GE2114@icir.org> References: <201304012017.r31KHQ6L030565@bro-ids.icir.org> <94639495-23D5-4ACC-9AC8-7F27422DFB2E@icir.org> <20130401202759.GC2114@icir.org> <61B097D9-6042-45D7-B279-BFF509C1B7D8@icir.org> <20130401203736.GE2114@icir.org> Message-ID: On Apr 1, 2013, at 4:37 PM, Robin Sommer wrote: > Yeah, for example one could turn off http_reply but not http_request. > I'm fine either way, it's not a big deal to keep ot if we think it's > still useful. Yep, it's probably better overall if we stay away from the notion of disabling analyzers in parts. Having the foundation that analyzers are either fully on or fully off is a nice thing to be able to assume. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Mon Apr 1 13:55:26 2013 From: seth at icir.org (Seth Hall) Date: Mon, 1 Apr 2013 16:55:26 -0400 Subject: [Bro-Dev] Removing --use-binpac In-Reply-To: <20130401203623.GD2114@icir.org> References: <20130401203623.GD2114@icir.org> Message-ID: On Apr 1, 2013, at 4:36 PM, Robin Sommer wrote: > I propose to remove this switch. We shouldn't have more than one > protocol analyzer per protocol; where we currently do, we should > decide if we want the C++ or BinPAC version. Agreed! > - Remove the binpac version of HTTP (it's by default off and hence > hardly used/tested by anybody). > > - Remove the binpac version of DNS (same reason). Agreed on those two. > - Enable DHCP even without that flag (I'm actually surprised it > isn't already, there's no alternative). Vlad's DHCP updates do this. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From robin at icir.org Mon Apr 1 14:21:07 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 1 Apr 2013 14:21:07 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/plugins: Lots of cleanup and API documentation for the analyzer/* classes. (e0c4bd1) In-Reply-To: <201304012017.r31KHQ6L030565@bro-ids.icir.org> References: <201304012017.r31KHQ6L030565@bro-ids.icir.org> Message-ID: <20130401212107.GA12725@icir.org> On Mon, Apr 01, 2013 at 13:17 -0700, I wrote: > TODOs: One more, and a larger one actually: - Convert all the current analyzers over to the one-directory-per-analyzer structure. I might need some help with that eventually. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From robin at icir.org Tue Apr 2 12:40:28 2013 From: robin at icir.org (Robin Sommer) Date: Tue, 2 Apr 2013 12:40:28 -0700 Subject: [Bro-Dev] 2.2 items Message-ID: <20130402194028.GC93990@icir.org> To collect where we are, here's what I believe we want to have in 2.2: - Measurement framework (mostly done; Seth) - With Bloom filters and Hyperloglog? (state?; Soumya/Bernhard/Seth) - File analysis framework (mostly done; Jon) - With Windows executable analyzer (state?; Seth) - Sqlite/Postgresql readers/writers (mostly done?; Bernhard) - BroControl test-suite (state?; Daniel) - Fix for BroControl's log archival, #970 (state?; Daniel) - Internal analyzer reorg (partially done; Robin) - Thread state cleanup (done, needs testing with input framework; Robin/Bernhard) - DNP3 (mostly done; Hui) - DHCP (state?; Vlad) - Input framework extensions for exec module (mostly done; Bernhard/Seth) - Documentation (can wait for beta period) - Scripting guide (whatever is ready; Scott) - Remaining parts from old Wiki (Daniel) - Overall cleanup How does that look? Anything else in terms of larger tasks (there are plenty smaller items in tickets of course, though we'll probably end up bumping quite a few again). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From bro at tracker.bro.org Tue Apr 2 14:10:43 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 02 Apr 2013 21:10:43 -0000 Subject: [Bro-Dev] #976: regex change in syslog-analyzer.pac Message-ID: <045.4ffa1e8cb47d98f8a92d0eac529f48ff@tracker.bro.org> #976: regex change in syslog-analyzer.pac ------------------------+-------------------- Reporter: aashish | Type: Patch Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+-------------------- Changing {{{[[:digit:]] to [[:alnum:]]}}} allows to capture windows logs via syslog analyzer specifically when logs are forwarded via snare. I can provide a trace, if desired. I tested this change locally and seems to be working just fine. {{{ diff --git a/src/syslog-protocol.pac b/src/syslog-protocol.pac index a2bf8a3..bc6d931 100644 --- a/src/syslog-protocol.pac +++ b/src/syslog-protocol.pac @@ -5,7 +5,7 @@ type Syslog_Message = record { type Syslog_Priority = record { lt : uint8 &check(lt == "<"); - val : RE/[[:digit:]]+/; + val : RE/[[:alnum:]]+/; gt : uint8 &check(gt == ">"); } &let { val_length: int = sizeof(val) - 1; }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Apr 2 14:15:32 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 02 Apr 2013 21:15:32 -0000 Subject: [Bro-Dev] #976: regex change in syslog-analyzer.pac In-Reply-To: <045.4ffa1e8cb47d98f8a92d0eac529f48ff@tracker.bro.org> References: <045.4ffa1e8cb47d98f8a92d0eac529f48ff@tracker.bro.org> Message-ID: <060.87f2f889702040e3abdd59801db8756a@tracker.bro.org> #976: regex change in syslog-analyzer.pac ----------------------+------------------------ Reporter: aashish | Owner: Type: Patch | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): > - val : RE/[[:digit:]]+/; > + val : RE/[[:alnum:]]+/;= Please do provide a trace since I don't think that follows the spec (not that that will stop us from doing it, we just need to have a valid and tested reason for doing it). -- Ticket URL: Bro Tracker Bro Issue Tracker From asharma at lbl.gov Tue Apr 2 14:22:10 2013 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 2 Apr 2013 14:22:10 -0700 Subject: [Bro-Dev] #976: regex change in syslog-analyzer.pac In-Reply-To: <060.87f2f889702040e3abdd59801db8756a@tracker.bro.org> References: <045.4ffa1e8cb47d98f8a92d0eac529f48ff@tracker.bro.org> <060.87f2f889702040e3abdd59801db8756a@tracker.bro.org> Message-ID: <20130402212208.GA2911@yaksha.lbl.gov> Seth: Trace sent to you in a separate email. Aashish On Tue, Apr 02, 2013 at 09:15:32PM -0000, Bro Tracker wrote: > #976: regex change in syslog-analyzer.pac > ----------------------+------------------------ > Reporter: aashish | Owner: > Type: Patch | Status: new > Priority: Medium | Milestone: Bro2.2 > Component: Bro | Version: git/master > Resolution: | Keywords: > ----------------------+------------------------ > > Comment (by seth): > > > - val : RE/[[:digit:]]+/; > > + val : RE/[[:alnum:]]+/;= > > Please do provide a trace since I don't think that follows the spec (not > that that will stop us from doing it, we just need to have a valid and > tested reason for doing it). > > -- > Ticket URL: > Bro Tracker > Bro Issue Tracker -- Aashish Sharma (asharma at lbl.gov) Cyber Security, Information Technology Division Lawrence Berkeley National Laboratory http://www.lbl.gov/cyber/pgp-aashish.txt Office: (510)-495-2680 Cell: (510)-457-1525 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130402/2d7ef5c5/attachment.bin From bro at tracker.bro.org Tue Apr 2 14:24:22 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 02 Apr 2013 21:24:22 -0000 Subject: [Bro-Dev] #976: regex change in syslog-analyzer.pac In-Reply-To: <045.4ffa1e8cb47d98f8a92d0eac529f48ff@tracker.bro.org> References: <045.4ffa1e8cb47d98f8a92d0eac529f48ff@tracker.bro.org> Message-ID: <060.0c80e06827e3e11bc4c7820d1cccbd40@tracker.bro.org> #976: regex change in syslog-analyzer.pac ----------------------+------------------------ Reporter: aashish | Owner: Type: Patch | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by aashish): None Filename None could not be saved, problem: [Errno 13] Permission denied: '/da/trac/bro/attachments/ticket/976'\Seth: Trace sent to you in a separate email. Aashish On Tue, Apr 02, 2013 at 09:15:32PM -0000, Bro Tracker wrote: > #976: regex change in syslog-analyzer.pac > ----------------------+------------------------ > Reporter: aashish | Owner: > Type: Patch | Status: new > Priority: Medium | Milestone: Bro2.2 > Component: Bro | Version: git/master > Resolution: | Keywords: > ----------------------+------------------------ > > Comment (by seth): > > > - val : RE/[[:digit:]]+/; > > + val : RE/[[:alnum:]]+/;= > > Please do provide a trace since I don't think that follows the spec (not > that that will stop us from doing it, we just need to have a valid and > tested reason for doing it). > > -- > Ticket URL: > Bro Tracker > Bro Issue Tracker [attachment:"None"] -- Ticket URL: Bro Tracker Bro Issue Tracker From asharma at lbl.gov Tue Apr 2 15:53:51 2013 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 2 Apr 2013 15:53:51 -0700 Subject: [Bro-Dev] 2.2 items In-Reply-To: <20130402194028.GC93990@icir.org> References: <20130402194028.GC93990@icir.org> Message-ID: <20130402225349.GJ2911@yaksha.lbl.gov> Robin: Any plans to get the new scan detection policies in 2.2 ? Aashish On Tue, Apr 02, 2013 at 12:40:28PM -0700, Robin Sommer wrote: > To collect where we are, here's what I believe we want to have in 2.2: > > - Measurement framework (mostly done; Seth) > - With Bloom filters and Hyperloglog? (state?; Soumya/Bernhard/Seth) > - File analysis framework (mostly done; Jon) > - With Windows executable analyzer (state?; Seth) > - Sqlite/Postgresql readers/writers (mostly done?; Bernhard) > - BroControl test-suite (state?; Daniel) > - Fix for BroControl's log archival, #970 (state?; Daniel) > - Internal analyzer reorg (partially done; Robin) > - Thread state cleanup (done, needs testing with input framework; Robin/Bernhard) > - DNP3 (mostly done; Hui) > - DHCP (state?; Vlad) > - Input framework extensions for exec module (mostly done; Bernhard/Seth) > > - Documentation (can wait for beta period) > - Scripting guide (whatever is ready; Scott) > - Remaining parts from old Wiki (Daniel) > - Overall cleanup > > How does that look? Anything else in terms of larger tasks (there are > plenty smaller items in tickets of course, though we'll probably end > up bumping quite a few again). > > Robin > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Aashish Sharma (asharma at lbl.gov) Cyber Security, Information Technology Division Lawrence Berkeley National Laboratory http://www.lbl.gov/cyber/pgp-aashish.txt Office: (510)-495-2680 Cell: (510)-457-1525 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130402/a9ea21ab/attachment.bin From seth at icir.org Tue Apr 2 18:55:15 2013 From: seth at icir.org (Seth Hall) Date: Tue, 2 Apr 2013 21:55:15 -0400 Subject: [Bro-Dev] 2.2 items In-Reply-To: <20130402225349.GJ2911@yaksha.lbl.gov> References: <20130402194028.GC93990@icir.org> <20130402225349.GJ2911@yaksha.lbl.gov> Message-ID: On Apr 2, 2013, at 6:53 PM, Aashish Sharma wrote: > Any plans to get the new scan detection policies in 2.2 ? Yep, that's included in the measurement framework. Get in touch with me tomorrow and we can discuss it if you want. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130402/c2294c69/attachment.bin From seth at icir.org Wed Apr 3 11:17:17 2013 From: seth at icir.org (Seth Hall) Date: Wed, 3 Apr 2013 14:17:17 -0400 Subject: [Bro-Dev] merge status emails? Message-ID: <81D887D4-9A52-439B-9E9E-FC44A1F606F4@icir.org> Are those broken? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130403/5bebc093/attachment-0001.bin From robin at icir.org Wed Apr 3 11:54:56 2013 From: robin at icir.org (Robin Sommer) Date: Wed, 3 Apr 2013 11:54:56 -0700 Subject: [Bro-Dev] merge status emails? In-Reply-To: <81D887D4-9A52-439B-9E9E-FC44A1F606F4@icir.org> References: <81D887D4-9A52-439B-9E9E-FC44A1F606F4@icir.org> Message-ID: <20130403185456.GB7887@icir.org> On Wed, Apr 03, 2013 at 14:17 -0400, you wrote: > Are those broken? It doesn't look like there's anything in the queue ... The mails report the same as the link on tracker.bro.org, plus fastpath. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From seth at icir.org Wed Apr 3 13:22:38 2013 From: seth at icir.org (Seth Hall) Date: Wed, 3 Apr 2013 16:22:38 -0400 Subject: [Bro-Dev] merge status emails? In-Reply-To: <20130403185456.GB7887@icir.org> References: <81D887D4-9A52-439B-9E9E-FC44A1F606F4@icir.org> <20130403185456.GB7887@icir.org> Message-ID: On Apr 3, 2013, at 2:54 PM, Robin Sommer wrote: > It doesn't look like there's anything in the queue ... The mails > report the same as the link on tracker.bro.org, plus fast path. Ah, I wasn't even thinking of that. It's been a while since nothing was in the merge queue. ;) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From bro at tracker.bro.org Wed Apr 3 18:23:34 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 04 Apr 2013 01:23:34 -0000 Subject: [Bro-Dev] #976: regex change in syslog-analyzer.pac In-Reply-To: <045.4ffa1e8cb47d98f8a92d0eac529f48ff@tracker.bro.org> References: <045.4ffa1e8cb47d98f8a92d0eac529f48ff@tracker.bro.org> Message-ID: <060.ca252b0d5d2d8e5c020d93ddd95793f5@tracker.bro.org> #976: regex change in syslog-analyzer.pac ----------------------+------------------------ Reporter: aashish | Owner: Type: Patch | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): This issue will take more investigation, the patch isn't right and it only partly works on accident due to the &check attribute in binpac not doing anything. It looks like we may need a snare log analyzer or make a modification to the syslog analyzer to make it cope with Snare logs (which are definitely not using the syslog protocol). For reference, here's some documentation about the snare packet format: http://wiki.rsyslog.com/index.php/Snare_and_rsyslog -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Apr 3 18:27:45 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 04 Apr 2013 01:27:45 -0000 Subject: [Bro-Dev] #977: retransmit in connection history Message-ID: <042.a50d6237cbff321b1136db48f0842853@tracker.bro.org> #977: retransmit in connection history -----------------------------+------------------------ Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Keywords: | -----------------------------+------------------------ In the connection record's $history field, it would be useful to include another character to indicate retransmits sent during the connection. I think that T and t for first orig and resp retransmit respectively would be appropriate. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Apr 11 13:02:32 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 11 Apr 2013 20:02:32 -0000 Subject: [Bro-Dev] #978: delete seems to invalidate set iteration Message-ID: <046.15a8fb964a692433b3803e2217a3a83e@tracker.bro.org> #978: delete seems to invalidate set iteration ------------------------+--------------------- Reporter: dmandelb | Type: Problem Status: new | Priority: Low Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- Deleting an element of a set inside of a for loop iterating over the set seems to cause unpredictable behavior. I think this should either be documented as a known non-feature or fixed. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Sun Apr 14 04:44:37 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Sun, 14 Apr 2013 11:44:37 -0000 Subject: [Bro-Dev] #968: Add bytestring_to_uint16, uint32, uint64 functions In-Reply-To: <041.91bbf7ca939f2dab16b849fee5136a33@tracker.bro.org> References: <041.91bbf7ca939f2dab16b849fee5136a33@tracker.bro.org> Message-ID: <056.90dc98b919d67ce84dd71ec0dc0426c7@tracker.bro.org> #968: Add bytestring_to_uint16, uint32, uint64 functions --------------------+------------------------ Reporter: yun | Owner: Type: Patch | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: bytestring --------------------+------------------------ Comment (by yun): I haven't seen any commits yet for this patch. I would love to have this function in the next release :) So if the patch needs some re-work please let me know. I do think the current patch would be much cleaner if we had access to these functions: {{{ uint16_t htobe16(uint16_t host_16bits); uint16_t htole16(uint16_t host_16bits); uint16_t be16toh(uint16_t big_endian_16bits); uint16_t le16toh(uint16_t little_endian_16bits); uint32_t htobe32(uint32_t host_32bits); uint32_t htole32(uint32_t host_32bits); uint32_t be32toh(uint32_t big_endian_32bits); uint32_t le32toh(uint32_t little_endian_32bits); uint64_t htobe64(uint64_t host_64bits); uint64_t htole64(uint64_t host_64bits); uint64_t be64toh(uint64_t big_endian_64bits); uint64_t le64toh(uint64_t little_endian_64bits); }}} So proposed idea: check if these functions are available on the host OS, and if not use our own implementation of these functions? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Apr 16 07:50:04 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 16 Apr 2013 14:50:04 -0000 Subject: [Bro-Dev] #979: segmentation fault in Expr::Serialize Message-ID: <046.6eaf716bfb137734790f713f70e740f7@tracker.bro.org> #979: segmentation fault in Expr::Serialize ------------------------+--------------------- Reporter: dmandelb | Type: Problem Status: new | Priority: Low Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- {{{ $ gdb --args ~/opt/bro/bin/bro -C -r reps/reps/traffic-generator/test- traffic.pcap reps/bro-scripts/bbn-host-characterization.bro GNU gdb (GDB) 7.0.1-debian Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /home/dmandelb/opt/bro/bin/bro...done. (gdb) r Starting program: /home/dmandelb/opt/bro/bin/bro -C -r reps/reps/traffic- generator/test-traffic.pcap reps/bro-scripts/bbn-host-characterization.bro [Thread debugging using libthread_db enabled] [New Thread 0xb7b27b70 (LWP 3035)] [New Thread 0xb71ffb70 (LWP 3036)] [New Thread 0xb69feb70 (LWP 3037)] [New Thread 0xb5fffb70 (LWP 3038)] Program received signal SIGSEGV, Segmentation fault. 0x00000000 in ?? () (gdb) bt #0 0x00000000 in ?? () #1 0x08301d14 in Expr::Serialize (this=0x8f69ce0, info=0xb60058d4) at /home/dmandelb/bro/src/Expr.cc:184 #2 0x08294b92 in Attributes::DoSerialize (this=0x8f69998, info=0xb60058d4) at /home/dmandelb/bro/src/Attr.cc:512 #3 0x083d4a63 in SerialObj::Serialize (this=0x8f69998, info=0xb60058d4) at /home/dmandelb/bro/src/SerialObj.cc:121 #4 0x08294950 in Attributes::Serialize (this=0x8f69998, info=0xb60058d4) at /home/dmandelb/bro/src/Attr.cc:487 #5 0x0835f111 in ID::DoSerialize (this=0x8f69790, info=0xb60058d4) at /home/dmandelb/bro/src/ID.cc:495 #6 0x083d4a63 in SerialObj::Serialize (this=0x8f69790, info=0xb60058d4) at /home/dmandelb/bro/src/SerialObj.cc:121 #7 0x0835e96e in ID::Serialize (this=0x8f69790, info=0xb60058d4) at /home/dmandelb/bro/src/ID.cc:311 #8 0x083d7af9 in Serializer::Serialize (this=0x85ce3f0, info=0xb60058d4, id=...) at /home/dmandelb/bro/src/Serializer.cc:111 #9 0x0839a133 in PersistenceSerializer::DoIDSerialization (this=0x85ce3f0, status=0xb60058d0, id=0x8f69790) at /home/dmandelb/bro/src/PersistenceSerializer.cc:498 #10 0x08399d06 in PersistenceSerializer::RunSerialization (this=0x85ce3f0, status=0xb60058d0) at /home/dmandelb/bro/src/PersistenceSerializer.cc:405 #11 0x083996ed in PersistenceSerializer::WriteState (this=0x85ce3f0, may_suspend=false) at /home/dmandelb/bro/src/PersistenceSerializer.cc:262 #12 0x08280a2e in done_with_network () at /home/dmandelb/bro/src/main.cc:258 #13 0x08282a2a in main (argc=5, argv=0xbffff2b4) at /home/dmandelb/bro/src/main.cc:1078 (gdb) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Apr 16 08:29:21 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 16 Apr 2013 15:29:21 -0000 Subject: [Bro-Dev] #979: segmentation fault in Expr::Serialize In-Reply-To: <046.6eaf716bfb137734790f713f70e740f7@tracker.bro.org> References: <046.6eaf716bfb137734790f713f70e740f7@tracker.bro.org> Message-ID: <061.6aa0dd2928a16beef9a51df52025d83e@tracker.bro.org> #979: segmentation fault in Expr::Serialize -----------------------+------------------------ Reporter: dmandelb | Owner: Type: Problem | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by dmandelb): Here's a simple (but inconvenient) workaround in case anybody else runs into this problem: {{{ diff --git a/bbn-host-characterization.bro b/bbn-host-characterization.bro index a11a9eb..4fbf04b 100644 --- a/bbn-host-characterization.bro +++ b/bbn-host-characterization.bro @@ -266,7 +266,9 @@ type HostData: record { ## TODO: mergeable? -global data: table[addr] of HostData = {} &default=[] &synchronized &persistent; +## TODO: This should be &persistent once http://tracker.bro.org/bro/ticket/979 +## is fixed. +global data: table[addr] of HostData = {} &default=[] &synchronized; redef record endpoint += { }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Apr 17 15:58:51 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 17 Apr 2013 22:58:51 -0000 Subject: [Bro-Dev] #980: elements not added to set as expected when using &default Message-ID: <046.1f46f88afe0c8267d678905d2253bbd7@tracker.bro.org> #980: elements not added to set as expected when using &default ------------------------+--------------------- Reporter: dmandelb | Type: Problem Status: new | Priority: Low Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- This code {{{ type Foo: record { x: count &default=0; }; global foo: table[count] of Foo = {} &default=[]; print(foo); print(foo[0]$x); foo[0]$x += 42; print(foo); print(foo[0]$x); }}} prints {{{ { } 0 { } 42 }}} but I think it should instead print {{{ { } 0 { [0] = [x=42] } 42 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Apr 18 08:55:45 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 18 Apr 2013 15:55:45 -0000 Subject: [Bro-Dev] #980: elements not added to set as expected when using &default In-Reply-To: <046.1f46f88afe0c8267d678905d2253bbd7@tracker.bro.org> References: <046.1f46f88afe0c8267d678905d2253bbd7@tracker.bro.org> Message-ID: <061.2a5f79018d269ebb9fdfdeafb3b5ad5a@tracker.bro.org> #980: elements not added to set as expected when using &default -----------------------+------------------------ Reporter: dmandelb | Owner: Type: Problem | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by dmandelb): Here's a workaround: {{{ foo[0] = foo[0]; }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Apr 18 16:47:58 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 18 Apr 2013 23:47:58 -0000 Subject: [Bro-Dev] #981: &default record only creates one record Message-ID: <046.19e3bb096ee03fbe6a622f573705cb5f@tracker.bro.org> #981: &default record only creates one record ------------------------+--------------------- Reporter: dmandelb | Type: Problem Status: new | Priority: Low Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- It seems like when a table's &default is set to a record (either as returned by copy() or a literal []-expression), only one copy is created instead of one copy per table entry. This code {{{ type Foo: record { x: count &default=0; }; global foo: table[count] of Foo = {} &default=[]; print(foo[0]$x); print(foo[1]$x); foo[0]$x = 0; foo[1]$x = 1; print(foo[0]$x); print(foo[1]$x); }}} prints {{{ 0 0 1 1 }}} but I think it should print {{{ 0 0 0 1 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From david at mandelberg.org Fri Apr 19 10:10:45 2013 From: david at mandelberg.org (David Mandelberg) Date: Fri, 19 Apr 2013 13:10:45 -0400 Subject: [Bro-Dev] would a patch for #981 be accepted? Message-ID: <1366391445.3473.20.camel@solaria> Hi, Is http://tracker.bro.org/bro/ticket/981 a bug, or did I misunderstand how the language is supposed to work? If I write a patch, would it be accepted (assuming it's of good quality)? I'm not sure if I'll be able to write the patch yet, but I'm trying to get a sense of how to move forward. -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ Fri Apr 19 13:06:18 EDT 2013 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130419/3d119fad/attachment.bin From seth at icir.org Fri Apr 19 10:19:59 2013 From: seth at icir.org (Seth Hall) Date: Fri, 19 Apr 2013 13:19:59 -0400 Subject: [Bro-Dev] would a patch for #981 be accepted? In-Reply-To: <1366391445.3473.20.camel@solaria> References: <1366391445.3473.20.camel@solaria> Message-ID: <625D0C09-4A8D-46EB-A9F1-C85AD5165DC7@icir.org> On Apr 19, 2013, at 1:10 PM, David Mandelberg wrote: > Is http://tracker.bro.org/bro/ticket/981 a bug, or did I misunderstand > how the language is supposed to work?  I think that's a legit bug. Robin? Jon? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130419/ac104747/attachment.bin From jsiwek at illinois.edu Fri Apr 19 11:15:27 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Fri, 19 Apr 2013 18:15:27 +0000 Subject: [Bro-Dev] would a patch for #981 be accepted? In-Reply-To: <1366391445.3473.20.camel@solaria> References: <1366391445.3473.20.camel@solaria> Message-ID: On Apr 19, 2013, at 12:10 PM, David Mandelberg wrote: > Is http://tracker.bro.org/bro/ticket/981 a bug, or did I misunderstand > how the language is supposed to work? Provided that &default for tables is not supposed to modify table membership when accessing indices that don't exist, the example you gave looked correct to me (i.e. not a bug) Since foo[0] and foo[1] are never actually entries in the table, you're always working directly with the value associated with the &default attribute whenever you access the table at those indices. - Jon From robin at icir.org Fri Apr 19 12:02:40 2013 From: robin at icir.org (Robin Sommer) Date: Fri, 19 Apr 2013 12:02:40 -0700 Subject: [Bro-Dev] would a patch for #981 be accepted? In-Reply-To: References: <1366391445.3473.20.camel@solaria> Message-ID: <20130419190240.GB57001@icir.org> On Fri, Apr 19, 2013 at 18:15 +0000, you wrote: > Provided that &default for tables is not supposed to modify table > membership when accessing indices that don't exist, I'm wondering if it should modify the table here. I can see how the current behaviour is misleading, it violates the "principle of least surprise". :) Can we change tables so that if &default is a non-constant, the first time one accesses a non-existing index, that slot gets assigned a deep-copy of the &default value? The downside would be that if somebody is relying on the current behaviour, he might access lots of non-existing entries with the assumption that the table won't change (i.e., he won't run into memory trouble). An alternative would be to have the lookup return a copy of the default, but not modify the table. With that "foo[0]$x = 0" would still not work as expected, but at least it wouldn't have any surprising side effects. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From david at mandelberg.org Fri Apr 19 12:16:06 2013 From: david at mandelberg.org (David Mandelberg) Date: Fri, 19 Apr 2013 15:16:06 -0400 Subject: [Bro-Dev] would a patch for #981 be accepted? In-Reply-To: <20130419190240.GB57001@icir.org> References: <1366391445.3473.20.camel@solaria> <20130419190240.GB57001@icir.org> Message-ID: <1366398966.3473.25.camel@solaria> On Fri, 2013-04-19 at 12:02 -0700, Robin Sommer wrote: > Can we change tables so that if &default is a non-constant, the first > time one accesses a non-existing index, that slot gets assigned a > deep-copy of the &default value? The downside would be that if > somebody is relying on the current behaviour, he might access lots of > non-existing entries with the assumption that the table won't change > (i.e., he won't run into memory trouble). What about modifying the table on writes to empty slots but not reads of empty slots? Is that feasible, or would it require too much code change? -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ Fri Apr 19 15:11:51 EDT 2013 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130419/844e5e4e/attachment.bin From robin at icir.org Fri Apr 19 12:23:38 2013 From: robin at icir.org (Robin Sommer) Date: Fri, 19 Apr 2013 12:23:38 -0700 Subject: [Bro-Dev] would a patch for #981 be accepted? In-Reply-To: <1366398966.3473.25.camel@solaria> References: <1366391445.3473.20.camel@solaria> <20130419190240.GB57001@icir.org> <1366398966.3473.25.camel@solaria> Message-ID: <20130419192338.GH57001@icir.org> On Fri, Apr 19, 2013 at 15:16 -0400, you wrote: > What about modifying the table on writes to empty slots but not reads of > empty slots? Is that feasible, or would it require too much code change? Don't think that's going to work because the write operates on the returned record; at that point it's not clear anymore that the value came out of the table. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From jsiwek at illinois.edu Fri Apr 19 12:30:26 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Fri, 19 Apr 2013 19:30:26 +0000 Subject: [Bro-Dev] would a patch for #981 be accepted? In-Reply-To: <20130419190240.GB57001@icir.org> References: <1366391445.3473.20.camel@solaria> <20130419190240.GB57001@icir.org> Message-ID: On Apr 19, 2013, at 2:02 PM, Robin Sommer wrote: >> Provided that &default for tables is not supposed to modify table >> membership when accessing indices that don't exist, > > I'm wondering if it should modify the table here. I can see how the > current behaviour is misleading, it violates the "principle of least > surprise". :) Does seem more intuitive to me. > Can we change tables so that if &default is a non-constant, the first > time one accesses a non-existing index, that slot gets assigned a > deep-copy of the &default value? Probably. > The downside would be that if > somebody is relying on the current behaviour, he might access lots of > non-existing entries with the assumption that the table won't change > (i.e., he won't run into memory trouble). Maybe we can provide a script-layer flag that, when on, generates warnings for script locations that end up assigning &default values to non-existing indices? That at least would help someone pinpoint locations they need to change to work w/ new behavior. - Jon From seth at icir.org Fri Apr 19 12:31:19 2013 From: seth at icir.org (Seth Hall) Date: Fri, 19 Apr 2013 15:31:19 -0400 Subject: [Bro-Dev] would a patch for #981 be accepted? In-Reply-To: <20130419190240.GB57001@icir.org> References: <1366391445.3473.20.camel@solaria> <20130419190240.GB57001@icir.org> Message-ID: On Apr 19, 2013, at 3:02 PM, Robin Sommer wrote: > An alternative would be to have the lookup return a copy of the > default, but not modify the table. I think this is reasonable and the way I'd expect it to work at least. I've never seen anyone doing things with &default that would violate it either. On the upside I've never seen anyone give a non-constant &default value. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From vallentin at icir.org Sat Apr 20 09:16:28 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Sat, 20 Apr 2013 09:16:28 -0700 Subject: [Bro-Dev] Support for HTTP body extraction of originator Message-ID: The current HTTP body extraction mechanism only allows for recording responses: event http_entity_data(c: connection, is_orig: bool,... { # Client body extraction is not currently supported in this script. if ( is_orig ) return; Does anyone recall the reason for this? Later in the script, we have: local suffix = fmt("%s_%d.dat", is_orig ? "orig" : "resp", \ c$http_state$current_response); So simply removing the is_orig check readily enables extraction of HTTP request bodies, and also correctly tags the extraction file with "orig" or "resp". The current workaround at this point is to copy the entire event handler for http_entity_data and simply invert the above check, which is redundant and inefficient. Here's my suggestion: we'd introduce an enum that specifies the direction, e.g., ORIG, RESP, BOTH. Users can then decide what they'd like to have recorded. Matthias From vern at icir.org Sun Apr 21 02:06:19 2013 From: vern at icir.org (Vern Paxson) Date: Sun, 21 Apr 2013 02:06:19 -0700 Subject: [Bro-Dev] would a patch for #981 be accepted? In-Reply-To: <20130419190240.GB57001@icir.org> (Fri, 19 Apr 2013 12:02:40 PDT). Message-ID: <20130421090619.3D6322C4003@rock.ICSI.Berkeley.EDU> > Can we change tables so that if &default is a non-constant, the first > time one accesses a non-existing index, that slot gets assigned a > deep-copy of the &default value? I'm not a fan of this, as "&default is a non-constant" is a more nuanced semantic notion (and could get messy for complicated constructors) than the sort that I think language rules should draw upon. Rather, my thinking is that &default should apply to rvalues but not lvalues. I believe (but have not confirmed) that there's enough context in the parser to tell whether the access is in an lvalue context or an rvalue context. If so, then the former shoud be a run-time error; if the user wants to update a value in the table that's not already there, they need to first put the value in the table. > An alternative would be to have the lookup return a copy of the > default, but not modify the table. That makes sense to me in any case. In fact, I'm surprised this isn't what already happens. Vern From seth at icir.org Sun Apr 21 06:05:32 2013 From: seth at icir.org (Seth Hall) Date: Sun, 21 Apr 2013 09:05:32 -0400 Subject: [Bro-Dev] Support for HTTP body extraction of originator In-Reply-To: References: Message-ID: On Apr 20, 2013, at 12:16 PM, Matthias Vallentin wrote: > # Client body extraction is not currently supported in this script. > if ( is_orig ) > return; > > Does anyone recall the reason for this? Oversight on my part. :) > Here's my suggestion: we'd introduce an enum that specifies the > direction, e.g., ORIG, RESP, BOTH. Users can then decide what they'd > like to have recorded. This is all being done through the file analysis framework now and is being abstracted there now. The script you are having trouble with is being removed. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jsiwek at illinois.edu Mon Apr 22 08:05:48 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Mon, 22 Apr 2013 15:05:48 +0000 Subject: [Bro-Dev] Support for HTTP body extraction of originator In-Reply-To: References: Message-ID: >> Here's my suggestion: we'd introduce an enum that specifies the >> direction, e.g., ORIG, RESP, BOTH. Users can then decide what they'd >> like to have recorded. > > > This is all being done through the file analysis framework now and is being abstracted there now. The script you are having trouble with is being removed. The script isn't being removed, just changed to use the generic file analysis events instead of http_entity_data. And the generic file events don't currently specify any direction information, so HTTP extraction will do both request and response bodies, but they can't be controlled independently. Do I need to add an 'is_orig' flag to at least the 'file_new' event? - Jon From vallentin at icir.org Mon Apr 22 08:19:44 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 22 Apr 2013 08:19:44 -0700 Subject: [Bro-Dev] Support for HTTP body extraction of originator In-Reply-To: References: Message-ID: > Do I need to add an 'is_orig' flag to at least the 'file_new' event? I don't know the internals of the FA framework, I just recall a record fa_file which appears to be what the Info record is to the logging framework. Could it make sense to put the directionality in there for more flexibility? Then users can access this information in any event. Matthias From vallentin at icir.org Mon Apr 22 08:22:25 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 22 Apr 2013 08:22:25 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/topk: implement topk. (c21c18e) In-Reply-To: <201304220812.r3M8CRUt000786@bro-ids.icir.org> References: <201304220812.r3M8CRUt000786@bro-ids.icir.org> Message-ID: > This is _completely_ untested. It compiles. It will probably do > nothing else (well, besides crashing Bro). What algorithm do you use? Based on my own research, I found that we should use the Space-Saving algorithm that typically comes in the form of the Stream-Summary data structure. Matthias From bernhard at ICSI.Berkeley.EDU Mon Apr 22 08:24:49 2013 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Mon, 22 Apr 2013 08:24:49 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/bernhard/topk: implement topk. (c21c18e) In-Reply-To: References: <201304220812.r3M8CRUt000786@bro-ids.icir.org> Message-ID: <7A0507BF-C66E-4C0E-8134-309623B49C21@icsi.berkeley.edu> On Apr 22, 2013, at 8:22 AM, Matthias Vallentin wrote: >> This is _completely_ untested. It compiles. It will probably do >> nothing else (well, besides crashing Bro). > > What algorithm do you use? Based on my own research, I found that we > should use the Space-Saving algorithm that typically comes in the form > of the Stream-Summary data structure. Exactly that one. Both of them to be exact. From jsiwek at illinois.edu Mon Apr 22 08:48:17 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Mon, 22 Apr 2013 15:48:17 +0000 Subject: [Bro-Dev] Support for HTTP body extraction of originator In-Reply-To: References: Message-ID: On Apr 22, 2013, at 10:19 AM, Matthias Vallentin wrote: >> Do I need to add an 'is_orig' flag to at least the 'file_new' event? > > I don't know the internals of the FA framework, I just recall a record > fa_file which appears to be what the Info record is to the logging > framework. fa_file is more analogous to the connection record now. > Could it make sense to put the directionality in there for > more flexibility? Then users can access this information in any event. Yeah, that might be fine. Do you have an opinion, Seth (I thought you did when we talked about the loss of directionality before) ? - Jon From bro at tracker.bro.org Mon Apr 22 09:56:21 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 22 Apr 2013 16:56:21 -0000 Subject: [Bro-Dev] #968: Add bytestring_to_uint16, uint32, uint64 functions In-Reply-To: <041.91bbf7ca939f2dab16b849fee5136a33@tracker.bro.org> References: <041.91bbf7ca939f2dab16b849fee5136a33@tracker.bro.org> Message-ID: <056.0bd6f2c7b45fdac762acea9262218973@tracker.bro.org> #968: Add bytestring_to_uint16, uint32, uint64 functions --------------------+------------------------ Reporter: yun | Owner: Type: Patch | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: bytestring --------------------+------------------------ Comment (by jsiwek): Replying to [comment:31 yun]: > Replying to [comment:30 robin]: > > Thinking about it, would it be better to switch the boolean to > > indicate host- vs. network-byte order, rather than big-endian vs > > low-endian? > > I think that would not work as that would make the bro script platform dependent. So for example using the bro script on a little endian platform would read the bytestream differently on a big endian platform if you specified host order. If a bytestring is parsed from some network traffic and delivered to the script-layer, it's either going to be in host order or network order and usually no one has to care the endianness that's behind those if they've got access to `hton*/ntoh*` functions that will hide any of the byte swapping that may be needed. So I think it makes more sense for a `bytestring_to_count` function to have a boolean flag for network vs. host order (instead of big vs. little endian). Or has endianness explicitly been exposed to the script-layer for some reason in your case? -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Mon Apr 22 10:11:42 2013 From: seth at icir.org (Seth Hall) Date: Mon, 22 Apr 2013 13:11:42 -0400 Subject: [Bro-Dev] Support for HTTP body extraction of originator In-Reply-To: References: Message-ID: On Apr 22, 2013, at 11:48 AM, "Siwek, Jonathan Luke" wrote: > Yeah, that might be fine. Do you have an opinion, Seth (I thought you did when we talked about the loss of directionality before) ? I think we had discussed creating an enums values to represent each location for files. For example: HTTP::FILE_CLIENT HTTP::FILE_SERVER SMTP::FILE_ENTITY FTP::FILE_ENTITY SSL::FILE_CLIENT_CERT SSL::FILE_SERVER_CERT This would give the directionality while leaving the possibility for protocols to have multiple transport mechanisms. PROTO::FILE_CLIENT_WRITE_METHOD1 PROTO::FILE_CLIENT_WRITE_METHOD2 PROTO::FILE_CLIENT_READ_METHOD2 Do you think we need to go that far or do you think that directionality alone is enough? I'm also not completely sure how this should be conveyed since I don't think it should be an argument to file_new since file_new is used for files read off disk or extracted from other files (child files). Perhaps it should just be a field in the fa_file record? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jsiwek at illinois.edu Mon Apr 22 10:43:11 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Mon, 22 Apr 2013 17:43:11 +0000 Subject: [Bro-Dev] Support for HTTP body extraction of originator In-Reply-To: References: Message-ID: > This would give the directionality while leaving the possibility for protocols to have multiple transport mechanisms. > > PROTO::FILE_CLIENT_WRITE_METHOD1 > PROTO::FILE_CLIENT_WRITE_METHOD2 > PROTO::FILE_CLIENT_READ_METHOD2 > > Do you think we need to go that far or do you think that directionality alone is enough? That case seems maybe like overkill because the mechanism and other context is typically available in c$proto which people can inspect in the FAF events, but the part that's missing is a consistent and protocol independent way of determining the direction that the file is going. As long as they have that, any other context that's available at the time of the FAF event becomes usable. > Perhaps it should just be a field in the fa_file record? Seems fine for now. Will add unless there's other thoughts. - Jon From seth at icir.org Mon Apr 22 10:55:44 2013 From: seth at icir.org (Seth Hall) Date: Mon, 22 Apr 2013 13:55:44 -0400 Subject: [Bro-Dev] Support for HTTP body extraction of originator In-Reply-To: References: Message-ID: On Apr 22, 2013, at 1:43 PM, "Siwek, Jonathan Luke" wrote: > but the part that's missing is a consistent and protocol independent way of determining the direction that the file is going. Sure, that seems reasonable. I think I would use it too. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From bro at tracker.bro.org Mon Apr 22 11:03:23 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 22 Apr 2013 18:03:23 -0000 Subject: [Bro-Dev] #974: segmentation fault with schedule statement In-Reply-To: <046.d01e8d84d5ec2c07b9d0bdadc4b2feb5@tracker.bro.org> References: <046.d01e8d84d5ec2c07b9d0bdadc4b2feb5@tracker.bro.org> Message-ID: <061.36927fdc090afd4ae17d7a3ed91709b1@tracker.bro.org> #974: segmentation fault with schedule statement -----------------------+------------------------ Reporter: dmandelb | Owner: Type: Problem | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by jsiwek): In [changeset:fa30d4a31365c448cd044d1dc4a5dc08a26ce7f2/bro]: {{{ #!CommitTicketReference repository="bro" revision="fa30d4a31365c448cd044d1dc4a5dc08a26ce7f2" Fix schedule statements used outside event handlers (addresses #974). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Apr 22 11:04:25 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 22 Apr 2013 18:04:25 -0000 Subject: [Bro-Dev] #974: segmentation fault with schedule statement In-Reply-To: <046.d01e8d84d5ec2c07b9d0bdadc4b2feb5@tracker.bro.org> References: <046.d01e8d84d5ec2c07b9d0bdadc4b2feb5@tracker.bro.org> Message-ID: <061.91ef7690d86890440b02f10906729501@tracker.bro.org> #974: segmentation fault with schedule statement ----------------------------+------------------------ Reporter: dmandelb | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: In `topic/jsiwek/974` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Apr 22 12:32:04 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 22 Apr 2013 19:32:04 -0000 Subject: [Bro-Dev] #927: topic/seth/metrics-merge: Metrics framework updates In-Reply-To: <042.a8a353a2e7de5a01b629d7caf36d7acc@tracker.bro.org> References: <042.a8a353a2e7de5a01b629d7caf36d7acc@tracker.bro.org> Message-ID: <057.60968952d2db3169f4f437aa97c243dd@tracker.bro.org> #927: topic/seth/metrics-merge: Metrics framework updates ----------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by seth): * owner: seth => robin * type: Task => Merge Request Comment: Ready to merge. There is some problem with the coverage.bare-mode-errors test though and I need to stop messing with it, I just can't figure out the problem. Here's the output I get (seems to be some dependency issue): {{{ +error in /Users/seth/bro/bro.work8/scripts/base/frameworks/sumstats/./plugins/././variance.bro, line 37: no such field in record (SumStats::rv$average) +error in /Users/seth/bro/bro.work8/scripts/base/frameworks/sumstats/./plugins/././variance.bro, line 40: no such field in record (SumStats::rv$average) +error in /Users/seth/bro/bro.work8/scripts/base/frameworks/sumstats/./plugins/././variance.bro, line 47: no such field in record (SumStats::rv1?$average) +error in /Users/seth/bro/bro.work8/scripts/base/frameworks/sumstats/./plugins/././variance.bro, line 48: no such field in record (SumStats::rv2?$average) +error in /Users/seth/bro/bro.work8/scripts/base/frameworks/sumstats/./plugins/././variance.bro, line 50: no such field in record (SumStats::result$average) +error in /Users/seth/bro/bro.work8/scripts/base/frameworks/sumstats/./plugins/././variance.bro, line 50: no such field in record (SumStats::rv1$average) +error in /Users/seth/bro/bro.work8/scripts/base/frameworks/sumstats/./plugins/././variance.bro, line 52: no such field in record (SumStats::result$average) +error in /Users/seth/bro/bro.work8/scripts/base/frameworks/sumstats/./plugins/././variance.bro, line 52: no such field in record (SumStats::rv2$average) +error in /Users/seth/bro/bro.work8/scripts/base/frameworks/sumstats/./plugins /./std-dev.bro, line 20: no such field in record (SumStats::rv?$variance) +error in /Users/seth/bro/bro.work8/scripts/base/frameworks/sumstats/./plugins /./std-dev.bro, line 21 and double: type mismatch (SumStats::rv$ and double) +error in /Users/seth/bro/bro.work8/scripts/base/frameworks/sumstats/./plugins /./std-dev.bro, line 21: argument type mismatch in function call (sqrt(SumStats::rv$)) +error in /Users/seth/bro/bro.work8/scripts/base/frameworks/sumstats/./plugins /./std-dev.bro, line 21: no such field in record (SumStats::rv$variance) +error in double and /Users/seth/bro/bro.work8/scripts/base/frameworks/sumstats/./plugins /./std-dev.bro, line 21: arithmetic mixed with non-arithmetic (double and SumStats::rv$) }}} There are equivalently named branches in the test suite repositories, but I left some issues in those unfixed because I'm not sure why they changed. They were mostly related to conn.log changes. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Apr 22 12:58:04 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 22 Apr 2013 19:58:04 -0000 Subject: [Bro-Dev] #968: Add bytestring_to_uint16, uint32, uint64 functions In-Reply-To: <041.91bbf7ca939f2dab16b849fee5136a33@tracker.bro.org> References: <041.91bbf7ca939f2dab16b849fee5136a33@tracker.bro.org> Message-ID: <056.97e9dd118a82da9b8bc9cb1dc752e450@tracker.bro.org> #968: Add bytestring_to_uint16, uint32, uint64 functions --------------------+------------------------ Reporter: yun | Owner: Type: Patch | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: bytestring --------------------+------------------------ Comment (by yun): Replying to [comment:33 jsiwek]: > Replying to [comment:31 yun]: > > Replying to [comment:30 robin]: > > > Thinking about it, would it be better to switch the boolean to > > > indicate host- vs. network-byte order, rather than big-endian vs > > > low-endian? > > > > I think that would not work as that would make the bro script platform dependent. So for example using the bro script on a little endian platform would read the bytestream differently on a big endian platform if you specified host order. > > If a bytestring is parsed from some network traffic and delivered to the script-layer, it's either going to be in host order or network order and usually no one has to care the endianness that's behind those if they've got access to `hton*/ntoh*` functions that will hide any of the byte swapping that may be needed. > > So I think it makes more sense for a `bytestring_to_count` function to have a boolean flag for network vs. host order (instead of big vs. little endian). > > Or has endianness explicitly been exposed to the script-layer for some reason in your case? The problem is that Bro does not have access to {{{hton*/ntoh*}}} functions, I did add the {{{reverse()}}} function that could be used for that. That's why the initial function was only {{{bytestring_to_count()}}} without any boolean flag. But indeed if Bro had access to these functions then the boolean could go away. It's true that I don't care in what byte order it's stored on my machine but I do care when I read it from a packet. Most applications send bytes as big-ending/network-order, but in some cases it's sent as little-endian. A good example is Metasploit. Which when it sends the Meterpreter payload to the victim it will first send a 32bit unsigned integer (4 bytes) of the payload size before sending the actual payload. Normally you would expect it to be in network-order/big-endian, but Metasploit sends the size in little-endian.. So if I read this from the packet, I definitely want to ensure that I read it from the known order. If the payload size was 255 bytes long I could have read 0xFF000000 instead of 0x000000FF. I hope this make sense, it's been a long discussion :) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Apr 22 14:36:43 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 22 Apr 2013 21:36:43 -0000 Subject: [Bro-Dev] #968: Add bytestring_to_uint16, uint32, uint64 functions In-Reply-To: <041.91bbf7ca939f2dab16b849fee5136a33@tracker.bro.org> References: <041.91bbf7ca939f2dab16b849fee5136a33@tracker.bro.org> Message-ID: <056.e7a71e0fc22cada56ee56bb413a419dc@tracker.bro.org> #968: Add bytestring_to_uint16, uint32, uint64 functions ----------------------------+------------------------ Reporter: yun | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: bytestring ----------------------------+------------------------ Changes (by jsiwek): * type: Patch => Merge Request Comment: > Normally you would expect it to be in network-order/big-endian, but Metasploit sends the size in little-endian.. So if I read this from the packet, I definitely want to ensure that I read it from the known order. Ok, makes sense to me now for the flag to indicate LE vs. BE so that we can handle either case as the canonical network form. The patch seemed ok, so changing to a merge request just to draw more attention from Robin. -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Tue Apr 23 00:00:04 2013 From: noreply at bro.org (Merge Tracker) Date: Tue, 23 Apr 2013 00:00:04 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201304230700.r3N704Qa022869@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 927 [1] | seth | robin | Normal | topic/seth/metrics-merge: Metrics framework updates [2] Bro | 968 [3] | yun | | Low | Add bytestring_to_uint16, uint32, uint64 functions Bro | 974 [4] | dmandelb | | Low | segmentation fault with schedule statement [1] #927: http://tracker.bro.org/bro/ticket/927 [2] metrics-merge:: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/metrics-merge: [3] #968: http://tracker.bro.org/bro/ticket/968 [4] #974: http://tracker.bro.org/bro/ticket/974 From bro at tracker.bro.org Tue Apr 23 08:03:47 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 23 Apr 2013 15:03:47 -0000 Subject: [Bro-Dev] #973: bro crash with default in record in record In-Reply-To: <046.1a0df40a8237343b6b804d9ccefb7c73@tracker.bro.org> References: <046.1a0df40a8237343b6b804d9ccefb7c73@tracker.bro.org> Message-ID: <061.3c1a010afe3140e3e7f6551fba453d8d@tracker.bro.org> #973: bro crash with default in record in record -----------------------+------------------------ Reporter: dmandelb | Owner: Type: Problem | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by jsiwek): In [changeset:7069f679c3205d9667daa28cc3e86c078f054e67/bro]: {{{ #!CommitTicketReference repository="bro" revision="7069f679c3205d9667daa28cc3e86c078f054e67" Fix record coercion for default inner record fields (addresses #973). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Apr 23 08:04:33 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 23 Apr 2013 15:04:33 -0000 Subject: [Bro-Dev] #973: bro crash with default in record in record In-Reply-To: <046.1a0df40a8237343b6b804d9ccefb7c73@tracker.bro.org> References: <046.1a0df40a8237343b6b804d9ccefb7c73@tracker.bro.org> Message-ID: <061.339bd5ad31f8872718a6254b1ece1d45@tracker.bro.org> #973: bro crash with default in record in record ----------------------------+------------------------ Reporter: dmandelb | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: Fix in `topic/jsiwek/973` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Apr 23 21:07:51 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 24 Apr 2013 04:07:51 -0000 Subject: [Bro-Dev] #968: Add bytestring_to_uint16, uint32, uint64 functions In-Reply-To: <041.91bbf7ca939f2dab16b849fee5136a33@tracker.bro.org> References: <041.91bbf7ca939f2dab16b849fee5136a33@tracker.bro.org> Message-ID: <056.fa58af9a97b043f1d6159067c0f4bf4d@tracker.bro.org> #968: Add bytestring_to_uint16, uint32, uint64 functions ----------------------------+------------------------ Reporter: yun | Owner: robin Type: Merge Request | Status: closed Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: bytestring ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [changeset:71591d706ea447dab000868a3d388284d26415b0/bro]: {{{ #!CommitTicketReference repository="bro" revision="71591d706ea447dab000868a3d388284d26415b0" Small tweaks for bytestring_to_count(). Closes #968. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Apr 23 21:07:51 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 24 Apr 2013 04:07:51 -0000 Subject: [Bro-Dev] #973: bro crash with default in record in record In-Reply-To: <046.1a0df40a8237343b6b804d9ccefb7c73@tracker.bro.org> References: <046.1a0df40a8237343b6b804d9ccefb7c73@tracker.bro.org> Message-ID: <061.c03d2cb16949e01b35d455ec436dba2e@tracker.bro.org> #973: bro crash with default in record in record ----------------------------+------------------------ Reporter: dmandelb | Owner: robin Type: Merge Request | Status: closed Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [changeset:f6f00924fcb8fff6512b79ca105aeeaaba80d0ae/bro]: {{{ #!CommitTicketReference repository="bro" revision="f6f00924fcb8fff6512b79ca105aeeaaba80d0ae" Merge remote-tracking branch 'origin/topic/jsiwek/973' Closes #973. * origin/topic/jsiwek/973: Fix record coercion for default inner record fields (addresses #973). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Apr 23 21:07:51 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 24 Apr 2013 04:07:51 -0000 Subject: [Bro-Dev] #974: segmentation fault with schedule statement In-Reply-To: <046.d01e8d84d5ec2c07b9d0bdadc4b2feb5@tracker.bro.org> References: <046.d01e8d84d5ec2c07b9d0bdadc4b2feb5@tracker.bro.org> Message-ID: <061.1564b6619212cd64c3235f55d560b7a3@tracker.bro.org> #974: segmentation fault with schedule statement ----------------------------+------------------------ Reporter: dmandelb | Owner: robin Type: Merge Request | Status: closed Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [changeset:e986247ff2fd304a2237aed39edfb190fb1d3b00/bro]: {{{ #!CommitTicketReference repository="bro" revision="e986247ff2fd304a2237aed39edfb190fb1d3b00" Merge remote-tracking branch 'origin/topic/jsiwek/974' Closes #974. * origin/topic/jsiwek/974: Fix schedule statements used outside event handlers (addresses #974). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Wed Apr 24 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Wed, 24 Apr 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201304240700.r3O702kE027975@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 927 [1] | seth | robin | Normal | topic/seth/metrics-merge: Metrics framework updates [2] [1] #927: http://tracker.bro.org/bro/ticket/927 [2] metrics-merge:: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/metrics-merge: From bro at tracker.bro.org Wed Apr 24 08:46:59 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 24 Apr 2013 15:46:59 -0000 Subject: [Bro-Dev] #982: topic/jsiwek/file-analysis Message-ID: <044.62a7d3a7b87d3c82b17137642aa083a2@tracker.bro.org> #982: topic/jsiwek/file-analysis ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This branch is in `bro`, `cmake`, `bro-testing`, and `bro-testing-private` repos. I'm claiming it's "ready" in the sense that "file analysis" functionality has parity with what was previously offered, but in a more general/robust way instead of the old method of concentrating logic around protocol specific events (mime/http entity data). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Apr 24 09:02:05 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 24 Apr 2013 16:02:05 -0000 Subject: [Bro-Dev] #982: topic/jsiwek/file-analysis In-Reply-To: <044.62a7d3a7b87d3c82b17137642aa083a2@tracker.bro.org> References: <044.62a7d3a7b87d3c82b17137642aa083a2@tracker.bro.org> Message-ID: <059.777be327810afcc126dff99f5077d0ee@tracker.bro.org> #982: topic/jsiwek/file-analysis ----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by aashish): None Filename None could not be saved, problem: [Errno 13] Permission denied: '/da/trac/bro/attachments/ticket/982'\Very very cool! Can't wait to play with this. Thanks for your excellent work Jon (and everyone else too) On Wed, Apr 24, 2013 at 03:46:59PM -0000, Bro Tracker wrote: > #982: topic/jsiwek/file-analysis > ---------------------------+------------------------ > Reporter: jsiwek | Owner: > Type: Merge Request | Status: new > Priority: Low | Milestone: Bro2.2 > Component: Bro | Version: git/master > Keywords: | > ---------------------------+------------------------ > This branch is in `bro`, `cmake`, `bro-testing`, and `bro-testing- private` > repos. > > I'm claiming it's "ready" in the sense that "file analysis" functionality > has parity with what was previously offered, but in a more general/robust > way instead of the old method of concentrating logic around protocol > specific events (mime/http entity data). > > -- > Ticket URL: > Bro Tracker > Bro Issue Tracker > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev [attachment:"None"] -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Apr 24 09:11:59 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 24 Apr 2013 16:11:59 -0000 Subject: [Bro-Dev] #982: topic/jsiwek/file-analysis In-Reply-To: <044.62a7d3a7b87d3c82b17137642aa083a2@tracker.bro.org> References: <044.62a7d3a7b87d3c82b17137642aa083a2@tracker.bro.org> Message-ID: <059.1ecddaab510e4879a7c8540347c2507d@tracker.bro.org> #982: topic/jsiwek/file-analysis ----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by seth): > Thanks for your excellent work Jon (and everyone else too) Yes, I agree. Really nice job Jon. Now let's get this merged so we can start making it do even more! -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Apr 24 09:35:54 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 24 Apr 2013 16:35:54 -0000 Subject: [Bro-Dev] #983: Deep typing bug Message-ID: <042.b93ebf52960db1e042b5f1f39a5358cc@tracker.bro.org> #983: Deep typing bug ---------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro2.2 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ This looks like an issue where the automatic typing is digging down one layer (beyond the set to the first record) but then doesn't auto type the record a layer deeper and gives an error. This is kind of a major usability issue for the 2.2 release because of how the vulnerable software version script works. Example script: {{{ @load frameworks/software/vulnerable redef Software::vulnerable_versions += { ["Java"] = set([$max=[$major=1,$minor=5,$minor2=999]], [$min=[$major=1,$minor=6], $max=[$major=1,$minor=6,$minor2=0,$minor3=43]], [$min=[$major=1,$minor=7], $max=[$major=1,$minor=7,$minor2=0,$minor3=20]]) }; }}} Gives the output: {{{ error in ./test-vuln.bro, line 3: type error in initialization ([$max=[$major=1, $minor=5, $minor2=999]], [$min=[$major=1, $minor=6], $max=[$major=1, $minor=6, $minor2=0, $minor3=43]], [$min=[$major=1, $minor=7], $max=[$major=1, $minor=7, $minor2=0, $minor3=20]]) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Apr 24 09:41:26 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 24 Apr 2013 16:41:26 -0000 Subject: [Bro-Dev] #983: Deep typing bug In-Reply-To: <042.b93ebf52960db1e042b5f1f39a5358cc@tracker.bro.org> References: <042.b93ebf52960db1e042b5f1f39a5358cc@tracker.bro.org> Message-ID: <057.69a0ac10a1585798ae496e24bd6be83d@tracker.bro.org> #983: Deep typing bug ----------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Description changed by seth: Old description: > This looks like an issue where the automatic typing is digging down one > layer (beyond the set to the first record) but then doesn't auto type the > record a layer deeper and gives an error. This is kind of a major > usability issue for the 2.2 release because of how the vulnerable > software version script works. > > Example script: > {{{ > @load frameworks/software/vulnerable > redef Software::vulnerable_versions += { > ["Java"] = set([$max=[$major=1,$minor=5,$minor2=999]], > [$min=[$major=1,$minor=6], > $max=[$major=1,$minor=6,$minor2=0,$minor3=43]], > [$min=[$major=1,$minor=7], > $max=[$major=1,$minor=7,$minor2=0,$minor3=20]]) > }; > }}} > > Gives the output: > {{{ > error in ./test-vuln.bro, line 3: type error in initialization > ([$max=[$major=1, $minor=5, $minor2=999]], [$min=[$major=1, $minor=6], > $max=[$major=1, $minor=6, $minor2=0, $minor3=43]], [$min=[$major=1, > $minor=7], $max=[$major=1, $minor=7, $minor2=0, $minor3=20]]) > }}} New description: This looks like an issue where the automatic typing is digging down one layer (beyond the set to the first record) but then doesn't auto type the record a layer deeper and gives an error. This is kind of a major usability issue for the 2.2 release because of how the vulnerable software version script works. Example script: {{{ @load frameworks/software/vulnerable redef Software::vulnerable_versions += { ["Java"] = set([$max=[$major=1,$minor=6,$minor2=0,$minor3=43]], [$min=[$major=1,$minor=7], $max=[$major=1,$minor=7,$minor2=0,$minor3=20]]) }; }}} Gives the output: {{{ error in ./test-vuln.bro, line 3: type error in initialization ([$max=[$major=1, $minor=6, $minor2=0, $minor3=43]], [$min=[$major=1, $minor=7], $max=[$major=1, $minor=7, $minor2=0, $minor3=20]])}}} -- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Apr 24 12:51:49 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 24 Apr 2013 19:51:49 -0000 Subject: [Bro-Dev] #982: topic/jsiwek/file-analysis In-Reply-To: <044.62a7d3a7b87d3c82b17137642aa083a2@tracker.bro.org> References: <044.62a7d3a7b87d3c82b17137642aa083a2@tracker.bro.org> Message-ID: <059.bb2e5a9fb1c0c111f136fba9a20f648b@tracker.bro.org> #982: topic/jsiwek/file-analysis ----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by seth): There is an oversight in the file_analysis.log right now. The "analyzers" field is redundant and a huge log space hog. The best analogy to connections is the "service" field in conn.log. Differences are that it uses short names and only represents analyzers that successfully analyzed the file. Here's an example from a file_analysis.log I have right now: {{{ FileAnalysis::ANALYZER_PE,FileAnalysis::ANALYZER_EXTRACT,FileAnalysis::ANALYZER_MD5,FileAnalysis::ANALYZER_SHA1 }}} I don't know what the field should be named, but I think it should only have "pe" in it in this case. I'm not terribly concerned that the hash analyzers and the extract analyzer were attached. Currently the hash analyzers and the extract analyzer add fields to the file_analysis.log already so you actually know when those were used anyway. This is where we're starting to bump into potential DFD (dynamic file detection) features too since the analyzer being present would indicate some degree of successful analysis. It's also likely that we may need to add FileConfirmation and FileViolation methods to the file analyzer base class. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Apr 24 13:39:36 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 24 Apr 2013 20:39:36 -0000 Subject: [Bro-Dev] #984: inconsistent scoping rules with events Message-ID: <046.c176e269b719e43cae396216d88f7660@tracker.bro.org> #984: inconsistent scoping rules with events ------------------------+--------------------- Reporter: dmandelb | Type: Problem Status: new | Priority: Low Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- This works as expected: {{{ module Foo; event foo() { print("foo!"); } event bro_init() { event foo(); } }}} But this doesn't print anything: {{{ module Foo; global foo: event(); event foo() { print("foo!"); } event bro_init() { event foo(); } }}} This does though: {{{ module Foo; global foo: event(); event foo() { print("foo!"); } event bro_init() { event Foo::foo(); } }}} I think I understand how it works now, but the scoping rules seem a bit inconsistent for events versus other data types. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Apr 24 15:53:44 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 24 Apr 2013 22:53:44 -0000 Subject: [Bro-Dev] #970: broctl stop/restart eating logs? In-Reply-To: <043.5d6412f7ce4509d8495a47e06660809a@tracker.bro.org> References: <043.5d6412f7ce4509d8495a47e06660809a@tracker.bro.org> Message-ID: <058.eb784ab6efef6ca97b5fd0ab1d16906b@tracker.bro.org> #970: broctl stop/restart eating logs? -----------------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by dnthayer): * component: Bro => BroControl -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Apr 24 15:57:32 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 24 Apr 2013 22:57:32 -0000 Subject: [Bro-Dev] #970: broctl stop/restart eating logs? In-Reply-To: <043.5d6412f7ce4509d8495a47e06660809a@tracker.bro.org> References: <043.5d6412f7ce4509d8495a47e06660809a@tracker.bro.org> Message-ID: <058.b4bfa392f8d35e7aa5919d8c1f1f27b3@tracker.bro.org> #970: broctl stop/restart eating logs? -----------------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Comment (by dnthayer): I found another problem: if a "broctl stop" happens while a normal log rotation/archive is happening, then there will be two background processes that are trying to write to the same file (conn-summary.log) in the same directory at the same time. As a result, only one conn-summary.log will be created, instead of two. -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Thu Apr 25 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Thu, 25 Apr 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201304250700.r3P702Wl010882@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 927 [1] | seth | robin | Normal | topic/seth/metrics-merge: Metrics framework updates [2] Bro | 982 [3] | jsiwek | | Low | topic/jsiwek/file-analysis [4] > Unmerged Fastpath Commits > ========================= Component | Revision | Committer | Date | Summary ------------------------------------------------------------------------------------------------------------------ binpac | af74234 | Jon Siwek | 2013-04-24 | Fix an exception slicing issue in binpac generated cleanup code. [5] [1] #927: http://tracker.bro.org/bro/ticket/927 [2] metrics-merge:: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/metrics-merge: [3] #982: http://tracker.bro.org/bro/ticket/982 [4] file-analysis: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/file-analysis [5] fastpath: http://tracker.bro.org/bro/changeset/af74234d216b5a845cfe8708ea82d8cb23bee68c/binpac From bro at tracker.bro.org Thu Apr 25 12:10:52 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 25 Apr 2013 19:10:52 -0000 Subject: [Bro-Dev] #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces In-Reply-To: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> References: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> Message-ID: <057.24a250d1ac26ddf8f9eb870a9bea555d@tracker.bro.org> #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces ----------------------------+------------------------ Reporter: seth | Owner: dnthayer Type: Merge Request | Status: new Priority: High | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by dnthayer): * type: Problem => Merge Request Comment: Has anyone had a chance to look at this? (broctl branch topic/dnthayer/ticket943) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Apr 25 12:29:18 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 25 Apr 2013 19:29:18 -0000 Subject: [Bro-Dev] #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces In-Reply-To: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> References: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> Message-ID: <057.b0c9ebf098035fb4b858d531ab2135fc@tracker.bro.org> #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces ----------------------------+------------------------ Reporter: seth | Owner: dnthayer Type: Merge Request | Status: new Priority: High | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by seth): From visually inspecting it, it seems ok to me. Did you test it enough to see if the cluster id's are set correctly? I'd say we can go ahead and merge it if you did. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Apr 25 12:41:58 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 25 Apr 2013 19:41:58 -0000 Subject: [Bro-Dev] #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces In-Reply-To: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> References: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> Message-ID: <057.1c0f88c1ac48934e8000c80d602b73dc@tracker.bro.org> #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces ----------------------------+------------------------ Reporter: seth | Owner: dnthayer Type: Merge Request | Status: new Priority: High | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by dnthayer): Replying to [comment:4 seth]: > From visually inspecting it, it seems ok to me. Did you test it enough to see if the cluster id's are set correctly? I'd say we can go ahead and merge it if you did. Yes, I tested it. I was just wondering if we need anything else, or if it is done for now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Apr 25 12:57:01 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 25 Apr 2013 19:57:01 -0000 Subject: [Bro-Dev] #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces In-Reply-To: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> References: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> Message-ID: <057.50679fa15758a3f6e75f1aab84a25c70@tracker.bro.org> #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces ----------------------------+------------------------ Reporter: seth | Owner: dnthayer Type: Merge Request | Status: new Priority: High | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by seth): > Yes, I tested it. I was just wondering if we need anything else, or if it > is done for now. I think it's done for now. Thanks for following up on it. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Apr 25 13:04:07 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 25 Apr 2013 20:04:07 -0000 Subject: [Bro-Dev] #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces In-Reply-To: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> References: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> Message-ID: <057.8d4d3abaed65af3debb54b3e92296498@tracker.bro.org> #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces ----------------------------+------------------------ Reporter: seth | Owner: dnthayer Type: Merge Request | Status: new Priority: High | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by seth): Hm, not sure who's responsible for merging in the broctl repository. I'll go ahead and do this one. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Apr 25 13:20:09 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 25 Apr 2013 20:20:09 -0000 Subject: [Bro-Dev] #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces In-Reply-To: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> References: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> Message-ID: <057.b87fe9f74054db63bebdbb461903accc@tracker.bro.org> #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces -----------------------------+------------------------ Reporter: seth | Owner: dnthayer Type: Merge Request | Status: closed Priority: High | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Merged in 058e66ce20604e2c4f45da1f06910c3528d89ec3 -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Fri Apr 26 00:00:05 2013 From: noreply at bro.org (Merge Tracker) Date: Fri, 26 Apr 2013 00:00:05 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201304260700.r3Q705Oj027240@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 927 [1] | seth | robin | Normal | topic/seth/metrics-merge: Metrics framework updates [2] Bro | 982 [3] | jsiwek | | Low | topic/jsiwek/file-analysis [4] > Unmerged Fastpath Commits > ========================= Component | Revision | Committer | Date | Summary ------------------------------------------------------------------------------------------------------------------ binpac | af74234 | Jon Siwek | 2013-04-24 | Fix an exception slicing issue in binpac generated cleanup code. [5] bro | 4bddcd2 | Seth Hall | 2013-04-25 | Fixed a bug in the vulnerable software script and added a test. [6] broctl | ac0dfa8 | Daniel Thayer | 2013-04-25 | Update main broctl doc (no new changes) [7] [1] #927: http://tracker.bro.org/bro/ticket/927 [2] metrics-merge:: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/metrics-merge: [3] #982: http://tracker.bro.org/bro/ticket/982 [4] file-analysis: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/file-analysis [5] fastpath: http://tracker.bro.org/bro/changeset/af74234d216b5a845cfe8708ea82d8cb23bee68c/binpac [6] fastpath: http://tracker.bro.org/bro/changeset/4bddcd23794bfcdd43eabd5da5e33d4227b117eb/bro [7] fastpath: http://tracker.bro.org/bro/changeset/ac0dfa8015e82fde8e4df5f6019f5979df519b1c/broctl From jones at tacc.utexas.edu Fri Apr 26 10:08:40 2013 From: jones at tacc.utexas.edu (William Jones) Date: Fri, 26 Apr 2013 17:08:40 +0000 Subject: [Bro-Dev] #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces In-Reply-To: <057.1c0f88c1ac48934e8000c80d602b73dc@tracker.bro.org> References: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> <057.1c0f88c1ac48934e8000c80d602b73dc@tracker.bro.org> Message-ID: I read from multiple interfaces per worker, a consequence of of using taps to monitor a two port 10 GigE LACP pair. The net I can't use PF_ring sense bro does not synchronizes the start up of each worker. Sterilized the startup it would allow a single work to get same has function for each interface. The a good chanes that a worker could end up with hash function in pf_Ring that are not the same. Here is what my worker config look like: # [worker-1] type=worker host=ids.tacc.utexas.edu. interface=p1p1.667 -ip1p2.667 -ip2p1.667 -ip2p2.667 aux_scripts=q1of8 # [worker-2] type=worker host=ids.tacc.utexas.edu. interface=p1p1.667 -ip1p2.667 -ip2p1.667 -ip2p2.667 aux_scripts=q2of8 I partition the ip space across 8 workers using pcap fitlers. I use PF_Ring if bro supported for multiple interfaces. -----Original Message----- From: bro-dev-bounces at bro.org [mailto:bro-dev-bounces at bro.org] On Behalf Of Bro Tracker Sent: Thursday, April 25, 2013 2:42 PM To: seth at icir.org; dnthayer at ncsa.illinois.edu Cc: bro-dev at bro.org Subject: Re: [Bro-Dev] #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces ----------------------------+------------------------ Reporter: seth | Owner: dnthayer Type: Merge Request | Status: new Priority: High | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by dnthayer): Replying to [comment:4 seth]: > From visually inspecting it, it seems ok to me. Did you test it enough to see if the cluster id's are set correctly? I'd say we can go ahead and merge it if you did. Yes, I tested it. I was just wondering if we need anything else, or if it is done for now. -- Ticket URL: Bro Tracker Bro Issue Tracker _______________________________________________ bro-dev mailing list bro-dev at bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From seth at icir.org Fri Apr 26 10:40:03 2013 From: seth at icir.org (Seth Hall) Date: Fri, 26 Apr 2013 13:40:03 -0400 Subject: [Bro-Dev] #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces In-Reply-To: References: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> <057.1c0f88c1ac48934e8000c80d602b73dc@tracker.bro.org> Message-ID: <6F86CEBC-7F45-4247-A10E-563074F25C7C@icir.org> On Apr 26, 2013, at 1:08 PM, William Jones wrote: > I read from multiple interfaces per worker, a consequence of of using taps to monitor a two port 10 GigE LACP pair. The net This is fixing a different problem. People have been having trouble monitoring two separate links that don't see split routing. The problem you're encountering is something that most people have been fixing by merging the traffic streams before sending them into the analysis box with a separate piece of hardware (it would typically get load balanced at the same time too). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jones at tacc.utexas.edu Fri Apr 26 11:23:10 2013 From: jones at tacc.utexas.edu (William Jones) Date: Fri, 26 Apr 2013 18:23:10 +0000 Subject: [Bro-Dev] #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces In-Reply-To: <6F86CEBC-7F45-4247-A10E-563074F25C7C@icir.org> References: <042.7b1cf3e72e5824853933bb4b823c68ca@tracker.bro.org> <057.1c0f88c1ac48934e8000c80d602b73dc@tracker.bro.org> <6F86CEBC-7F45-4247-A10E-563074F25C7C@icir.org> Message-ID: I understand what problem was fixed. I was hoping that some in the bro group would recognize that there are more problems with pf_ring and bro that the current set of problems being talked about. I merged packet streams before and found that method didn't solve my drop packet problems. What did was allocating enough packet space in the kernel per interface and having bro read from each interface. Right now I am monitoring 2 10 GigE lacp pair. I about to put a system so that I can monitor a 4 10 GigE lacp set up. You really should investigate what it takes keep up with multiple 10 GigE interfaces lacp interaces. You might come to the different conclusion the usefulness merging interface in the kernel kernel. -----Original Message----- From: Seth Hall [mailto:seth at icir.org] Sent: Friday, April 26, 2013 12:40 PM To: William Jones Cc: bro at tracker.bro.org; seth at icir.org; dnthayer at ncsa.illinois.edu; bro-dev at bro.org Subject: Re: [Bro-Dev] #943: PF_Ring plugin to support load balancing while sniffing multiple interfaces On Apr 26, 2013, at 1:08 PM, William Jones wrote: > I read from multiple interfaces per worker, a consequence of of using taps to monitor a two port 10 GigE LACP pair. The net This is fixing a different problem. People have been having trouble monitoring two separate links that don't see split routing. The problem you're encountering is something that most people have been fixing by merging the traffic streams before sending them into the analysis box with a separate piece of hardware (it would typically get load balanced at the same time too). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From bro at tracker.bro.org Fri Apr 26 12:20:02 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Fri, 26 Apr 2013 19:20:02 -0000 Subject: [Bro-Dev] #985: 'tail -f' functionality for file reading in input framework Message-ID: <047.b23ffeaa4950cac11fbb46a0ad111ed0@tracker.bro.org> #985: 'tail -f' functionality for file reading in input framework ------------------------+----------------------------- Reporter: scampbell | Type: Feature Request Status: new | Priority: Low Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+----------------------------- With the current input framework, file data -> event translation requires that the entire data file be read at bro start time. This can be prohibitive when the file sizes become large ( > 1GB ). It would be great to see a file open option that would start reading at the end of the file. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Fri Apr 26 14:34:49 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Fri, 26 Apr 2013 21:34:49 -0000 Subject: [Bro-Dev] #986: Bro master is leaking memory in OpaqueVal Message-ID: <044.843da895e468d6733a55ce9227f8d7b7@tracker.bro.org> #986: Bro master is leaking memory in OpaqueVal ---------------------+------------------------ Reporter: amannb | Owner: Type: Problem | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ The current bro master is apparently leaking strings when creating OpaqueVals. pprof output attached; compiled with gcc 4.4.2 (on landfill). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Fri Apr 26 18:20:56 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Sat, 27 Apr 2013 01:20:56 -0000 Subject: [Bro-Dev] #987: Bro core leak caused by sumstats framework Message-ID: <044.e2ef98f4f0c43d873d8ca1e04fe99e82@tracker.bro.org> #987: Bro core leak caused by sumstats framework ---------------------+------------------------ Reporter: amannb | Owner: Type: Problem | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ the sumstats framework might cause a memory leak in the Bro core. On landfill, I get the following pprof output when running a pretty minimal script (attached) using the topic/seth/metrics-merge branch. You need to run the script on a trace with connections - I used the very short ftp-ipv4.trace trace from testing. The leak seems to become worse over time. I am not entirely sure if it really is a leak or just some? artifact of the google perf tools. But on a first glance, it might be one. -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Sat Apr 27 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Sat, 27 Apr 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201304270700.r3R702Z0026241@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 927 [1] | seth | robin | Normal | topic/seth/metrics-merge: Metrics framework updates [2] Bro | 982 [3] | jsiwek | | Low | topic/jsiwek/file-analysis [4] > Unmerged Fastpath Commits > ========================= Component | Revision | Committer | Date | Summary ------------------------------------------------------------------------------------------------------------------ binpac | af74234 | Jon Siwek | 2013-04-24 | Fix an exception slicing issue in binpac generated cleanup code. [5] bro | 4bddcd2 | Seth Hall | 2013-04-25 | Fixed a bug in the vulnerable software script and added a test. [6] broctl | ac0dfa8 | Daniel Thayer | 2013-04-25 | Update main broctl doc (no new changes) [7] [1] #927: http://tracker.bro.org/bro/ticket/927 [2] metrics-merge:: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/metrics-merge: [3] #982: http://tracker.bro.org/bro/ticket/982 [4] file-analysis: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/file-analysis [5] fastpath: http://tracker.bro.org/bro/changeset/af74234d216b5a845cfe8708ea82d8cb23bee68c/binpac [6] fastpath: http://tracker.bro.org/bro/changeset/4bddcd23794bfcdd43eabd5da5e33d4227b117eb/bro [7] fastpath: http://tracker.bro.org/bro/changeset/ac0dfa8015e82fde8e4df5f6019f5979df519b1c/broctl From scampbell at lbl.gov Sat Apr 27 12:39:15 2013 From: scampbell at lbl.gov (Scott Campbell) Date: Sat, 27 Apr 2013 14:39:15 -0500 Subject: [Bro-Dev] #985: 'tail -f' functionality for file reading in input framework In-Reply-To: <047.b23ffeaa4950cac11fbb46a0ad111ed0@tracker.bro.org> References: <047.b23ffeaa4950cac11fbb46a0ad111ed0@tracker.bro.org> Message-ID: <517C2963.6090508@lbl.gov> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 4/26/13 2:20 PM, Bro Tracker wrote: > #985: 'tail -f' functionality for file reading in input framework > ------------------------+----------------------------- Reporter: > scampbell | Type: Feature Request Status: new | > Priority: Low Milestone: Bro2.2 | Component: Bro Version: > git/master | Keywords: > ------------------------+----------------------------- With the > current input framework, file data -> event translation requires > that the entire data file be read at bro start time. This can be > prohibitive when the file sizes become large ( > 1GB ). > > It would be great to see a file open option that would start > reading at the end of the file. > I tried to update the ticket, but there seems to be issues with the bug tracker. The patch to support this functionality is attached - it is only a few lines. An example of using this looks like: > Input::add_event([$source=data_file, $reader=Input::READER_RAW, > $mode=Input::TSTREAM, $name="issh", $fields=lineVals, > $ev=sshLine]); thanks! scott -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iD8DBQFRfCliK2Plq8B7ZBwRAsmKAJ963D3BOpuhFKxCFAZFLEUuJAxBKQCeOW+5 WOneupNCMl2ewnwbsJ00Oqs= =NkuM -----END PGP SIGNATURE----- -------------- next part -------------- *** /tmp/bro/src/input/Manager.cc 2013-04-27 12:13:56.009913562 -0700 --- bro/src/input/Manager.cc 2013-04-27 10:31:31.723948100 -0700 *************** *** 315,320 **** --- 315,324 ---- rinfo->mode = MODE_STREAM; break; + case 3: + rinfo->mode = MODE_TSTREAM; + break; + default: reporter->InternalError("unknown reader mode"); } *** /tmp/bro/src/input/ReaderBackend.h 2013-04-27 12:13:56.009913562 -0700 --- bro/src/input/ReaderBackend.h 2013-04-27 12:19:07.533907736 -0700 *************** *** 36,41 **** --- 36,48 ---- */ MODE_STREAM, + /** + * Streaming read tail mode. The reader should monitor the data source + * for new appended data. When new data is appended is has to be sent + * using the Put api functions. Reading begins at the *end) of the file. + */ + MODE_TSTREAM, + /** Internal dummy mode for initialization. */ MODE_NONE }; *** /tmp/bro/src/input/readers/Raw.cc 2013-04-27 12:13:56.011918627 -0700 --- bro/src/input/readers/Raw.cc 2013-04-27 12:21:48.121927531 -0700 *************** *** 63,72 **** } } // This is defined in input/fdstream.h in = new boost::fdistream(fileno(file)); ! if ( execute && Info().mode == MODE_STREAM ) fcntl(fileno(file), F_SETFL, O_NONBLOCK); return true; --- 63,76 ---- } } + // If looking at file tail, reset fd to end + if ( Info().mode == MODE_TSTREAM ) + fseek(file, 0, SEEK_END); + // This is defined in input/fdstream.h in = new boost::fdistream(fileno(file)); ! if ( execute && ((Info().mode == MODE_STREAM) || (Info().mode == MODE_TSTREAM)) ) fcntl(fileno(file), F_SETFL, O_NONBLOCK); return true; *************** *** 211,217 **** case MODE_MANUAL: case MODE_STREAM: ! if ( Info().mode == MODE_STREAM && file != NULL && in != NULL ) { //fpurge(file); in->clear(); // remove end of file evil bits --- 215,222 ---- case MODE_MANUAL: case MODE_STREAM: ! case MODE_TSTREAM: ! if ( ((Info().mode == MODE_STREAM) || (Info().mode == MODE_TSTREAM)) && file != NULL && in != NULL ) { //fpurge(file); in->clear(); // remove end of file evil bits *************** *** 261,266 **** --- 266,272 ---- case MODE_REREAD: case MODE_STREAM: + case MODE_TSTREAM: #ifdef DEBUG Debug(DBG_INPUT, "Starting Heartbeat update"); #endif *** /tmp/bro/src/types.bif 2013-04-27 12:13:56.022986957 -0700 --- bro/src/types.bif 2013-04-27 12:26:39.779955302 -0700 *************** *** 208,213 **** --- 208,214 ---- MANUAL = 0, REREAD = 1, STREAM = 2, + TSTREAM = 3, %} module GLOBAL; -------------- next part -------------- A non-text attachment was scrubbed... Name: PATCH.sig Type: application/octet-stream Size: 65 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130427/836dfc54/attachment.obj From robin at icir.org Sat Apr 27 12:56:46 2013 From: robin at icir.org (Robin Sommer) Date: Sat, 27 Apr 2013 12:56:46 -0700 Subject: [Bro-Dev] Tracker down Message-ID: <20130427195646.GB26957@icir.org> The tracker is currently down, seems an update broken something. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From robin at icir.org Sat Apr 27 13:22:12 2013 From: robin at icir.org (Robin Sommer) Date: Sat, 27 Apr 2013 13:22:12 -0700 Subject: [Bro-Dev] Tracker down In-Reply-To: <20130427195646.GB26957@icir.org> References: <20130427195646.GB26957@icir.org> Message-ID: <20130427202212.GA59376@icir.org> Fixed. Robin On Sat, Apr 27, 2013 at 12:56 -0700, I wrote: > > The tracker is currently down, seems an update broken something. > > Robin > -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From bro at tracker.bro.org Sat Apr 27 14:30:26 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Sat, 27 Apr 2013 21:30:26 -0000 Subject: [Bro-Dev] #985: 'tail -f' functionality for file reading in input framework In-Reply-To: <047.b23ffeaa4950cac11fbb46a0ad111ed0@tracker.bro.org> References: <047.b23ffeaa4950cac11fbb46a0ad111ed0@tracker.bro.org> Message-ID: <062.2d25ea0811fb389d2a81c0cd84bf976a@tracker.bro.org> #985: 'tail -f' functionality for file reading in input framework ------------------------------+------------------------ Reporter: scampbell | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------------+------------------------ Comment (by scampbell): The attached patch is run against 2.1-386. It adds an additional input framework option - TSTREAM to the raw reader type. sample usage: Input::add_event([$source=data_file, $reader=Input::READER_RAW, $mode=Input::TSTREAM, $name="issh", $fields=lineVals, $ev=sshLine]); -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Sun Apr 28 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Sun, 28 Apr 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201304280700.r3S702IQ016278@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 927 [1] | seth | robin | Normal | topic/seth/metrics-merge: Metrics framework updates [2] Bro | 982 [3] | jsiwek | | Low | topic/jsiwek/file-analysis [4] [1] #927: http://tracker.bro.org/bro/ticket/927 [2] metrics-merge:: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/metrics-merge: [3] #982: http://tracker.bro.org/bro/ticket/982 [4] file-analysis: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/file-analysis From bro at tracker.bro.org Sun Apr 28 16:16:28 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Sun, 28 Apr 2013 23:16:28 -0000 Subject: [Bro-Dev] #988: Bug in HTTP body extraction Message-ID: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> #988: Bug in HTTP body extraction ----------------------+-------------------- Reporter: matthias | Owner: seth Type: Problem | Status: new Priority: High | Milestone: Bro2.2 Component: Bro | Version: 2.1 Keywords: | ----------------------+-------------------- There exists a bug in HTTP body extraction that prevents certain bodies from being dumped, even though having set {{{ redef extract_file_types = /.*/; }}} This happens presumably because Bro does not figure out the correct MIME type and does not set `c$http$mime_type`. It results in this check failing: {{{ if ( c$http?$mime_type && extract_file_types in c$http$mime_type ) { c$http$extract_file = T; } }}} On a related note, I also find missing responses to HTTP POST requests which I assume come from the same issues. I have a trace that I could attach, but wanted to make sure it's worth the effort in face of the upcoming file analysis framework, or if we plan on pushing a 2.1 hotfix for this. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Sun Apr 28 16:16:49 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Sun, 28 Apr 2013 23:16:49 -0000 Subject: [Bro-Dev] #988: Bug in HTTP body extraction In-Reply-To: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> References: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> Message-ID: <061.14a14ffa4dd68ce24f75bbc725ed4941@tracker.bro.org> #988: Bug in HTTP body extraction -----------------------+-------------------- Reporter: matthias | Owner: seth Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: 2.1 Resolution: | Keywords: -----------------------+-------------------- Changes (by matthias): * priority: High => Medium -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Sun Apr 28 17:48:37 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 29 Apr 2013 00:48:37 -0000 Subject: [Bro-Dev] #987: Bro core leak caused by sumstats framework In-Reply-To: <044.e2ef98f4f0c43d873d8ca1e04fe99e82@tracker.bro.org> References: <044.e2ef98f4f0c43d873d8ca1e04fe99e82@tracker.bro.org> Message-ID: <059.8c1ea24c02e31ac003f45a4c6e3ed676@tracker.bro.org> #987: Bro core leak caused by sumstats framework ----------------------+------------------------ Reporter: amannb | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by robin): * priority: Low => High Comment: I see our leaks tests triggering too when merging the sumstats framework. -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Mon Apr 29 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Mon, 29 Apr 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201304290700.r3T703JX021582@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 927 [1] | seth | robin | Normal | topic/seth/metrics-merge: Metrics framework updates [2] Bro | 982 [3] | jsiwek | | Low | topic/jsiwek/file-analysis [4] [1] #927: http://tracker.bro.org/bro/ticket/927 [2] metrics-merge:: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/metrics-merge: [3] #982: http://tracker.bro.org/bro/ticket/982 [4] file-analysis: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/file-analysis From bro at tracker.bro.org Mon Apr 29 07:24:09 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 29 Apr 2013 14:24:09 -0000 Subject: [Bro-Dev] #988: Bug in HTTP body extraction In-Reply-To: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> References: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> Message-ID: <061.7bf1aa7826c9b70d2275c8d41df04604@tracker.bro.org> #988: Bug in HTTP body extraction -----------------------+-------------------- Reporter: matthias | Owner: seth Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: 2.1 Resolution: | Keywords: -----------------------+-------------------- Comment (by jsiwek): Replying to [ticket:988 matthias]: > I have a trace that I could attach, but wanted to make sure it's worth the effort in face of the upcoming file analysis framework, or if we plan on pushing a 2.1 hotfix for this. I don't have an opinion on whether to hotfix in 2.1, but if you want to see if we'll have nice behavior with the FAF, can you checkout `topic/jsiwek/file-analysis` (in `bro` and `cmake` repos) and use the same redef you had before to see if it works better? If doesn't, leave a comment in #982 and attach the pcap. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Apr 29 09:46:11 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 29 Apr 2013 16:46:11 -0000 Subject: [Bro-Dev] #927: topic/seth/metrics-merge: Metrics framework updates In-Reply-To: <042.a8a353a2e7de5a01b629d7caf36d7acc@tracker.bro.org> References: <042.a8a353a2e7de5a01b629d7caf36d7acc@tracker.bro.org> Message-ID: <057.fdde41a6766b89dad3aec679e9d271ce@tracker.bro.org> #927: topic/seth/metrics-merge: Metrics framework updates ----------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): Working on merging this in topic/robin/metrics-merge. TODOs: - memory leaks - coverage.bare-mode-errors - high CPU usage Also sent a list of smaller things to Seth. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Apr 29 12:21:51 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 29 Apr 2013 19:21:51 -0000 Subject: [Bro-Dev] #988: Bug in HTTP body extraction In-Reply-To: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> References: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> Message-ID: <061.9bc3a025485006d941eafa309ffca56b@tracker.bro.org> #988: Bug in HTTP body extraction -----------------------+-------------------- Reporter: matthias | Owner: seth Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: 2.1 Resolution: | Keywords: -----------------------+-------------------- Comment (by matthias): It seems that the FAF resolves this problem, I can now see the HTTP bodies for both POST requests and the text/plain MIME type. However, I do had trouble finding the corresponding files. For example, for a given HTTP session the `extraction_file` column shows `http-item- ku9xiCY0bg9-37.dat`, which includes the HTTP response body. The corresponding request body sits in `http-item-oh0kb6JHiM5-36.dat` but is not referenced in the `http.log`. Some options: 1. Extend the `extraction_file` column in the `http.log` to also include the file name of the request, say separated by comma (or whatever is the separator for sets). 2. Have a separate column for request and response. 3. Change the extraction file name to include also the connection UID. We need address issue (1) in any case because the current `extraction_file` approach does not match the reality where we can have multiple files per HTTP request-response pair. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Apr 29 12:29:32 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 29 Apr 2013 19:29:32 -0000 Subject: [Bro-Dev] #988: Bug in HTTP body extraction In-Reply-To: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> References: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> Message-ID: <061.800620f82fb0a2df0e6032d5319c7f93@tracker.bro.org> #988: Bug in HTTP body extraction -----------------------+-------------------- Reporter: matthias | Owner: seth Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: 2.1 Resolution: | Keywords: -----------------------+-------------------- Comment (by seth): > I have a trace that I could attach, but wanted to make sure it's worth the > effort in face of the upcoming file analysis framework, or if we plan on > pushing a 2.1 hotfix for this. I don't see any reason to fix it in Bro since the code would be outdated so quickly. You can set the c$http$extract_file field on your if you want to work around this issue for now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Apr 29 12:50:42 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 29 Apr 2013 19:50:42 -0000 Subject: [Bro-Dev] #988: Bug in HTTP body extraction In-Reply-To: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> References: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> Message-ID: <061.7debbd1ddac5835ed20b933e5ab7a608@tracker.bro.org> #988: Bug in HTTP body extraction -----------------------+-------------------- Reporter: matthias | Owner: seth Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: 2.1 Resolution: | Keywords: -----------------------+-------------------- Comment (by matthias): Replying to [comment:4 seth]: > I don't see any reason to fix it in Bro since the code would be outdated so quickly. You can set the c$http$extract_file field on your if you want to work around this issue for now. Yeah, I'm fine with leaving 2.1 as is. But the issues I raised in my last comment actually apply to Jon's `file-analysis` branch. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Apr 29 21:30:01 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 30 Apr 2013 04:30:01 -0000 Subject: [Bro-Dev] #987: Bro core leak caused by sumstats framework In-Reply-To: <044.e2ef98f4f0c43d873d8ca1e04fe99e82@tracker.bro.org> References: <044.e2ef98f4f0c43d873d8ca1e04fe99e82@tracker.bro.org> Message-ID: <059.6a7368bc9c2601dd4831bd24db0eb3c2@tracker.bro.org> #987: Bro core leak caused by sumstats framework ----------------------+------------------------ Reporter: amannb | Owner: robin Type: Problem | Status: closed Priority: High | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [changeset:1a41bfa0ef0bf2a8fc1829388a350609f98a6a42/bro]: {{{ #!CommitTicketReference repository="bro" revision="1a41bfa0ef0bf2a8fc1829388a350609f98a6a42" Fixing memory leak in CompHash. Amazing what code still has memory leaks ... Closes #987. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Apr 29 21:30:42 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 30 Apr 2013 04:30:42 -0000 Subject: [Bro-Dev] #987: Bro core leak caused by sumstats framework In-Reply-To: <044.e2ef98f4f0c43d873d8ca1e04fe99e82@tracker.bro.org> References: <044.e2ef98f4f0c43d873d8ca1e04fe99e82@tracker.bro.org> Message-ID: <059.f51423176192f9e51c641a29809baea0@tracker.bro.org> #987: Bro core leak caused by sumstats framework ----------------------+------------------------ Reporter: amannb | Owner: robin Type: Problem | Status: closed Priority: High | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------+------------------------ Comment (by robin): In [changeset:95cf662ff59513f16949425675344e89bf4a8ee0/bro]: {{{ #!CommitTicketReference repository="bro" revision="95cf662ff59513f16949425675344e89bf4a8ee0" Fixing memory leak in CompHash. Amazing what code still has memory leaks ... Closes #987. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Apr 29 22:12:10 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 30 Apr 2013 05:12:10 -0000 Subject: [Bro-Dev] #988: Bug in HTTP body extraction In-Reply-To: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> References: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> Message-ID: <061.e493540a6b93849fc200edcce1d48843@tracker.bro.org> #988: Bug in HTTP body extraction -----------------------+--------------------------- Reporter: matthias | Owner: seth Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: file-analysis -----------------------+--------------------------- Changes (by matthias): * keywords: => file-analysis * version: 2.1 => git/master -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Tue Apr 30 00:00:01 2013 From: noreply at bro.org (Merge Tracker) Date: Tue, 30 Apr 2013 00:00:01 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201304300700.r3U701qO021571@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 927 [1] | seth | robin | Normal | topic/seth/metrics-merge: Metrics framework updates [2] Bro | 982 [3] | jsiwek | | Low | topic/jsiwek/file-analysis [4] [1] #927: http://tracker.bro.org/bro/ticket/927 [2] metrics-merge:: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/metrics-merge: [3] #982: http://tracker.bro.org/bro/ticket/982 [4] file-analysis: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/file-analysis From bro at tracker.bro.org Tue Apr 30 09:30:06 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 30 Apr 2013 16:30:06 -0000 Subject: [Bro-Dev] #989: topic/dnthayer/cleanup2 Message-ID: <046.349f77fa6a6607106b6258a9a77624ab@tracker.bro.org> #989: topic/dnthayer/cleanup2 ---------------------------+------------------------ Reporter: dnthayer | Owner: Type: Merge Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: BroControl | Version: git/master Keywords: | ---------------------------+------------------------ This branch fixes various issues with broctl, including a few critical fixes for running broctl on OS X, and more error checks have been added to help prevent broctl cron from crashing. Here are the one-line summaries of all commits in this branch: Check exit status of croncmd Improve error checking of top helper output Improve error checking of capstats output Add more error reporting to broctl cron Fix a bug when the time command is not found Fix the broctl top and cron commands on OS X Fix a couple of bugs in the broctl ps plugin Remove unused broctl scripts Improve the check-pid helper script -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Apr 30 11:48:38 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 30 Apr 2013 18:48:38 -0000 Subject: [Bro-Dev] #983: Deep typing bug In-Reply-To: <042.b93ebf52960db1e042b5f1f39a5358cc@tracker.bro.org> References: <042.b93ebf52960db1e042b5f1f39a5358cc@tracker.bro.org> Message-ID: <057.00f56e21dd20825cd761d595dc960a67@tracker.bro.org> #983: Deep typing bug ----------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): Replying to [ticket:983 seth]: > > This looks like an issue where the automatic typing is digging down one layer (beyond the set to the first record) but then doesn't auto type the record a layer deeper and gives an error. It's less a problem with nested records as it is a problem with inlining any record ctor as indices within a set/table ctor. E.g.: {{{ print set([$max=5], [$min=2, $max=10]); }}} gives a "type error in initialization" since two different types are used to index the set and there's not any code yet to resolve the ambiguity of what to use for the overall type of the set. I think maybe an aggregate record type should be used that's made from all fields of all indices, but I've got some issues getting that to work right now. A second problem comes from something like: {{{ type MyRec: record { min: count &optional; max: count; }; local myset: set[MyRec] = set([$max=5], [$max=2]); }}} Here, the set ctor is fine since the index types agree, but there's a "type clash in assignment" since the type of the rhs doesn't match. I think implementing coercion for non-empty tables/sets would fix this. > This is kind of a major usability issue for the 2.2 release because of how the vulnerable software version script works. > > Example script: > {{{ > @load frameworks/software/vulnerable > redef Software::vulnerable_versions += { > ["Java"] = set([$max=[$major=1,$minor=6,$minor2=0,$minor3=43]], > [$min=[$major=1,$minor=7], $max=[$major=1,$minor=7,$minor2=0,$minor3=20]]) > }; > }}} A workaround for now would be: {{{ const one: Software::VulnerableVersionRange = [$max=[$major=1,$minor=6,$minor2=0,$minor3=43]]; const two: Software::VulnerableVersionRange = [$min=[$major=1,$minor=7], $max=[$major=1,$minor=7,$minor2=0,$minor3=20]]; redef Software::vulnerable_versions += { ["Java"] = set(one, two) }; }}} -- Ticket URL: Bro Tracker Bro Issue Tracker