From noreply at bro.org Sat Jun 1 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Sat, 1 Jun 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306010700.r51702r5002896@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 983 [1] | seth | | High | Deep typing bug [1] #983: http://tracker.bro.org/bro/ticket/983 From noreply at bro.org Sun Jun 2 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Sun, 2 Jun 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306020700.r52702QR006019@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 983 [1] | seth | | High | Deep typing bug [1] #983: http://tracker.bro.org/bro/ticket/983 From bro at tracker.bro.org Sun Jun 2 20:16:25 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 03 Jun 2013 03:16:25 -0000 Subject: [Bro-Dev] #983: Deep typing bug In-Reply-To: <042.b93ebf52960db1e042b5f1f39a5358cc@tracker.bro.org> References: <042.b93ebf52960db1e042b5f1f39a5358cc@tracker.bro.org> Message-ID: <057.4f0a2df5c61baaeda5878635c685660e@tracker.bro.org> #983: Deep typing bug ----------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: High | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [changeset:d3d14e10cf7ce10f4bf9b8272b4c5174cfdd0cc8/bro]: {{{ #!CommitTicketReference repository="bro" revision="d3d14e10cf7ce10f4bf9b8272b4c5174cfdd0cc8" Merge remote-tracking branch 'origin/topic/jsiwek/983' Closes #983. * origin/topic/jsiwek/983: Add named constructor examples to docs. Allow named vector constructors. Addresses #983. Allow named table constructors. Addresses #983. Improve set constructor argument coercion. Allow named set constructors. Addresses #983. Allow named record constructors. Addresses #983. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Sun Jun 2 20:18:41 2013 From: robin at icir.org (Robin Sommer) Date: Sun, 2 Jun 2013 20:18:41 -0700 Subject: [Bro-Dev] Plugin branch status In-Reply-To: References: <20130518011717.GB67280@icir.org> <996FF1A9-929E-4C00-A6F8-D2E0126BDDB8@icir.org> <20130523231111.GV8596@icir.org> <20130530223315.GI75431@icir.org> Message-ID: <20130603031841.GA44801@icir.org> On Fri, May 31, 2013 at 14:22 +0000, you wrote: > We'd probably hear the least complaints if 2.6.4 was the minimum requirement. Thanks for the summary. I've pushed a work-around that gets things to work with static archives. It's not nice but should do for now until we can up the cmake version more easily. The plugin branch is pretty much ready now. I still see one test failure however that I don't fully understand yet. If you're still reviewing the branch, could you take a look at this one: core.tunnels.teredo-known-services ... failed % 'btest-diff known_services.log' failed unexpectedly (exit code 1) % cat .diag btest-diff: input known_services.log does not exist. >From the test description I'm not sure if known_services.log can legitimately be missing in the 2nd case. rRobin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From jsiwek at illinois.edu Mon Jun 3 09:39:49 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Mon, 3 Jun 2013 16:39:49 +0000 Subject: [Bro-Dev] Plugin branch status In-Reply-To: <20130603031841.GA44801@icir.org> References: <20130518011717.GB67280@icir.org> <996FF1A9-929E-4C00-A6F8-D2E0126BDDB8@icir.org> <20130523231111.GV8596@icir.org> <20130530223315.GI75431@icir.org> <20130603031841.GA44801@icir.org> Message-ID: On Jun 2, 2013, at 10:18 PM, Robin Sommer wrote: > > core.tunnels.teredo-known-services ? failed There's a subtle change to the test in this branch: it no longer does `bro -b`. The reason that ends up mattering for the test is that the pcap has a connection for which both Teredo and DNS analyzers get attached and the Teredo analyzer does this thing where it won't emit a protocol_confirmation if some other analyzer on the same connection has already. When doing `bro -b`, the DNS analyzer doesn't get attached since the associated scripts aren't loaded, but the Teredo analyzer does since it has a signature that matches and so it will emit a protocol_confirmation which causes the known_service.log. > From the test description I'm not sure if known_services.log can > legitimately be missing in the 2nd case. Seems fine. Or you can add the -b back. - Jon From jsiwek at illinois.edu Mon Jun 3 14:31:54 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Mon, 3 Jun 2013 21:31:54 +0000 Subject: [Bro-Dev] Plugin branch status In-Reply-To: <20130603031841.GA44801@icir.org> References: <20130518011717.GB67280@icir.org> <996FF1A9-929E-4C00-A6F8-D2E0126BDDB8@icir.org> <20130523231111.GV8596@icir.org> <20130530223315.GI75431@icir.org> <20130603031841.GA44801@icir.org> Message-ID: > The plugin branch is pretty much ready now. Just finished reading all the changes; looks pretty great. I put some misc. doc and typo fixes in topic/jsiwek/plugins-cleanup if you want to merge it to your branch (or I'll make a merge ticket later if it's easier). A couple questions: Was there anything that makes use of analyzer::Tag::subtype yet? Wasn't sure I understood how to use that or what cases would be candidates for using it. Maybe put some initial calls to BRO_PLUGIN_VERSION(1) for everything? Just feels like that feature is going to be underused/forgotten unless people see things are already explicitly using it. - Jon From robin at icir.org Mon Jun 3 20:11:17 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 3 Jun 2013 20:11:17 -0700 Subject: [Bro-Dev] Plugin branch status In-Reply-To: References: <20130518011717.GB67280@icir.org> <996FF1A9-929E-4C00-A6F8-D2E0126BDDB8@icir.org> <20130523231111.GV8596@icir.org> <20130530223315.GI75431@icir.org> <20130603031841.GA44801@icir.org> Message-ID: <20130604031117.GB23573@icir.org> On Mon, Jun 03, 2013 at 16:39 +0000, you wrote: > Seems fine. Or you can add the -b back. Did that, not sure how that got lost in the first place. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From robin at icir.org Mon Jun 3 20:27:10 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 3 Jun 2013 20:27:10 -0700 Subject: [Bro-Dev] Plugin branch status In-Reply-To: References: <20130518011717.GB67280@icir.org> <996FF1A9-929E-4C00-A6F8-D2E0126BDDB8@icir.org> <20130523231111.GV8596@icir.org> <20130530223315.GI75431@icir.org> <20130603031841.GA44801@icir.org> Message-ID: <20130604032710.GC23573@icir.org> On Mon, Jun 03, 2013 at 21:31 +0000, you wrote: > I put some misc. doc and typo fixes in topic/jsiwek/plugins-cleanup Merged, thanks! > Was there anything that makes use of analyzer::Tag::subtype yet? That's indeed not used yet. It's in preparation for BinPAC++ where a single class will provide a set of analyzers. > Maybe put some initial calls to BRO_PLUGIN_VERSION(1) for everything? > Just feels like that feature is going to be underused/forgotten unless > people see things are already explicitly using it. Generally I'm thinking for statically compiled plugins versions aren't really useful; starting to version them would probably be more confusing than helpful. I'm seeing versions primarily for later when we have shared libraries and people may start updating plugins independently of Bro itself. That said, the idea was to always set the version of static plugins to BRO_PLUGIN_VERSION_BUILTIN. That doesn't happen right now if I'm not mistaken. Need to take another look. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From bro at tracker.bro.org Tue Jun 4 09:00:37 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 04 Jun 2013 16:00:37 -0000 Subject: [Bro-Dev] #869: Mechanism for discovering current script name In-Reply-To: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> References: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> Message-ID: <057.eb821b51b4f79b511e28cde837bcfe68@tracker.bro.org> #869: Mechanism for discovering current script name ------------------------------+------------------------ Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: language ------------------------------+------------------------ Comment (by jsiwek): In [changeset:307fc187c000c95044aa3f61d891b4878e5552c1/bro]: {{{ #!CommitTicketReference repository="bro" revision="307fc187c000c95044aa3f61d891b4878e5552c1" Add @PATH bro script macro. Addresses #869. The macro expands to a string value containing the file system path in which the script lives. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Jun 4 09:10:24 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 04 Jun 2013 16:10:24 -0000 Subject: [Bro-Dev] #869: Mechanism for discovering current script name In-Reply-To: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> References: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> Message-ID: <057.3f38bdd091a95f835ff930754b67c0c0@tracker.bro.org> #869: Mechanism for discovering current script name ----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: language ----------------------------+------------------------ Changes (by jsiwek): * type: Feature Request => Merge Request Comment: In `topic/jsiwek/869`. Also forgot to mention it doesn't necessarily expand to an absolute path, it may be relative to the working directory (e.g. if running standalone command-line bro for testing stuff). But either way, it still ends up as a valid way to reference other files/paths from within a script. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Jun 4 11:04:03 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 04 Jun 2013 18:04:03 -0000 Subject: [Bro-Dev] #869: Mechanism for discovering current script name In-Reply-To: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> References: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> Message-ID: <057.c10fae2bc57c3d401819c660db2aa63f@tracker.bro.org> #869: Mechanism for discovering current script name ----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: language ----------------------------+------------------------ Comment (by seth): > Also forgot to mention it doesn't necessarily expand to an absolute path, > it may be relative to the working directory (e.g. if running standalone > command-line bro for testing stuff). Hm, I guess this begs the question of if the input framework uses the working directory when it goes to load files. Bernhard? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Jun 4 11:35:36 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 04 Jun 2013 18:35:36 -0000 Subject: [Bro-Dev] #869: Mechanism for discovering current script name In-Reply-To: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> References: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> Message-ID: <057.a40b6dbf10a39954f7181cb5a2b860f5@tracker.bro.org> #869: Mechanism for discovering current script name ----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: language ----------------------------+------------------------ Comment (by robin): Why not always make it an absolute path so that we avoid any ambiguity? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Jun 4 12:20:21 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 04 Jun 2013 19:20:21 -0000 Subject: [Bro-Dev] #869: Mechanism for discovering current script name In-Reply-To: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> References: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> Message-ID: <057.ff71cd2e172e54212e3cd400d5b72a54@tracker.bro.org> #869: Mechanism for discovering current script name ----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: language ----------------------------+------------------------ Comment (by jsiwek): In [changeset:7e8b504305f8606246b52fb7a92ad151552aac11/bro]: {{{ #!CommitTicketReference repository="bro" revision="7e8b504305f8606246b52fb7a92ad151552aac11" Make @PATH always return absolute path. Addresses #869. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Tue Jun 4 15:24:46 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 4 Jun 2013 15:24:46 -0700 Subject: [Bro-Dev] Bloom filter interface Message-ID: I'm the process of devising Bro's new Bloom filter interface and would like to have your feedback on the BiFs. This is what I came up with so far: (1) bloomfilter_init(fp: double, capacity: count, max: count): opaque of bloomfilter (2) bloomfilter_add(bf: opaque of bloomfilter, x: any) (3) bloomfilter_lookup(bf: opaque of bloomfilter, x: any) : count (4) bloomfilter_merge(bf1: opaque of bloomfilter, bf2: opaque of bloomfilter) : bool Initialization involves (1), where "fp" represents the desired false-positive rate and "capacity" the maximum number of unique elements the bloom filter supports until "fp" can no longer be guaranteed. The "max" argument will default to 1 and represents the maximum counter value one can attach to each element. To add an element, one uses (2). The first use of this function "locks" the Bloom filter type to the value type of "x". Subsequent invocations of this function require that the type of "x" matches the value from the very first invocation. Lookup using (3) works analogous to (2). The return value represents the counter associated with "x". For basic membership queries (max=1) the return value takes on only 0 and 1. One can also merge related Bloom filters using (4). This requires that "bf1" and "bf2" have the internal types as well as parameterization, as the union of two Bloom filters needs to equi-sized underlying bit vectors. Matthias From bro at tracker.bro.org Tue Jun 4 15:39:29 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 04 Jun 2013 22:39:29 -0000 Subject: [Bro-Dev] #1013: redef'ing tables overwrites unrelated values Message-ID: <046.70c89aaf401c58227cd7d8cc363e89c1@tracker.bro.org> #1013: redef'ing tables overwrites unrelated values ------------------------+--------------------- Reporter: dmandelb | Type: Problem Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- This code: {{{ const foo: table[string] of double = {} &redef; redef foo["abc"] = 42.0; redef foo["def"] = -42.0; print(foo); }}} Prints this: {{{ { [def] = -42.0 } }}} But I expected it to print something like this: {{{ { [abc] = 42.0, [def] = -42.0 } }}} Did I misuse redef or is this a bug? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Jun 4 20:09:23 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 05 Jun 2013 03:09:23 -0000 Subject: [Bro-Dev] #845: PF_RING+DNA In-Reply-To: <046.fd6f0f04e887db155402c1572d02fff0@tracker.bro.org> References: <046.fd6f0f04e887db155402c1572d02fff0@tracker.bro.org> Message-ID: <061.1d1a85f372dd0fa432805fd3fae6496c@tracker.bro.org> #845: PF_RING+DNA ------------------------------+------------------------ Reporter: dnthayer | Owner: dnthayer Type: Feature Request | Status: assigned Priority: Normal | Milestone: Bro2.3 Component: BroControl | Version: git/master Resolution: | Keywords: ------------------------------+------------------------ Comment (by scampbell): I did some testing for this and made a couple of changes. First the pfdnacluster_master needs to fork away on run or it will block the starting of the worker nodes. Second, the worker nodes will need to use the pfdnacluster_master interfaces, rather than (ex.) dna0. The pfdnacluster_master interface is called dnacluster:nn w/ nn being the pf_ring id. To run as (not root), just use: setcap cap_net_raw,cap_net_admin=eip /usr/bin/pfdnacluster_master -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Jun 4 20:59:37 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 05 Jun 2013 03:59:37 -0000 Subject: [Bro-Dev] #869: Mechanism for discovering current script name In-Reply-To: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> References: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> Message-ID: <057.4da11954884bdc35844780d286187b8d@tracker.bro.org> #869: Mechanism for discovering current script name ----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: language ----------------------------+------------------------ Comment (by robin): Thanks. One more thing (I've already argued with Seth about this :) For me "path" suggests *full* path including the script's file, while this returns only the directory where the script is located in. Would you object to changing that? Alternatively, how about using {{{@DIR}}} instead of {{{@PATH}}}? (or maybe {{{@CWD}}}?) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Jun 4 21:16:09 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 05 Jun 2013 04:16:09 -0000 Subject: [Bro-Dev] #869: Mechanism for discovering current script name In-Reply-To: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> References: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> Message-ID: <057.9254e2ef244df7dca4e371093b5fcc3a@tracker.bro.org> #869: Mechanism for discovering current script name ----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: language ----------------------------+------------------------ Comment (by seth): > Thanks. One more thing (I've already argued with Seth about this :) For me > "path" suggests *full* path including the script's file, while this > returns only the directory where the script is located in. Would you > object to changing that? Alternatively, how about using {{{@DIR}}} instead > of {{{@PATH}}}? (or maybe {{{@CWD}}}?) I don't like @CWD since that conflates with cwd for Bro which will frequently be something different. I could get behind @DIR, but I still that @PATH indicates just the path and not the path+file to me. Not a big deal either way for me. -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Wed Jun 5 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Wed, 5 Jun 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306050700.r55702KA023249@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 869 [1] | seth | | Normal | Mechanism for discovering current script name [1] #869: http://tracker.bro.org/bro/ticket/869 From robin at icir.org Wed Jun 5 08:03:33 2013 From: robin at icir.org (Robin Sommer) Date: Wed, 5 Jun 2013 08:03:33 -0700 Subject: [Bro-Dev] Bloom filter interface In-Reply-To: References: Message-ID: <20130605150333.GJ3330@icir.org> I already talked to Matthias but for the record, this looks good to me. One more question: On Tue, Jun 04, 2013 at 15:24 -0700, you wrote: > (4) bloomfilter_merge(bf1: opaque of bloomfilter, bf2: opaque of > bloomfilter) : bool Is this merging bf2 into bf1? Would it make sense to return a new "opaque of bloomfilter" instead? Alternatively, we might need a bloomfilter_copy() for the case that one wants to merge without destroying any of the original filters. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From bro at tracker.bro.org Wed Jun 5 08:19:27 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 05 Jun 2013 15:19:27 -0000 Subject: [Bro-Dev] #1013: redef'ing tables overwrites unrelated values In-Reply-To: <046.70c89aaf401c58227cd7d8cc363e89c1@tracker.bro.org> References: <046.70c89aaf401c58227cd7d8cc363e89c1@tracker.bro.org> Message-ID: <061.7c24e198aca7f4b0cdbc887c6d28a93c@tracker.bro.org> #1013: redef'ing tables overwrites unrelated values -----------------------+------------------------ Reporter: dmandelb | Owner: Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by jsiwek): This is typically how tables can be redef'd: {{{ const foo: table[string] of double = {} &redef; # full (re)initialization redef foo = { ["nope"] = 37.0 }; # full (re)initialization, discards "nope" index redef foo = { ["abc"] = 42.0 }; # add elements redef foo += { ["def"] = -42.0, ["ghi"] = 7.0 }; # remove elements from LHS based on indices shared with RHS redef foo -= { ["ghi"] = 0.0 }; # RHS is just a table value, the syntax of how it's represented shouldn't matter redef foo += table(["cool"] = 5.0); print(foo); }}} Haven't yet looked in to whether what you tried (redef'ing an index expression) can actually be used in a meaningful way or if it should result in some error message. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Jun 5 08:32:22 2013 From: seth at icir.org (Seth Hall) Date: Wed, 5 Jun 2013 11:32:22 -0400 Subject: [Bro-Dev] Bloom filter interface In-Reply-To: <20130605150333.GJ3330@icir.org> References: <20130605150333.GJ3330@icir.org> Message-ID: <8530FF6B-ADA1-4FD1-A2CB-D4E8FDA12D8E@icir.org> On Jun 5, 2013, at 11:03 AM, Robin Sommer wrote: > On Tue, Jun 04, 2013 at 15:24 -0700, you wrote: > >> (4) bloomfilter_merge(bf1: opaque of bloomfilter, bf2: opaque of >> bloomfilter) : bool > > Is this merging bf2 into bf1? Would it make sense to return a new > "opaque of bloomfilter" instead? >From a clarity perspective I think that returning a bloom filter makes the most sense. From a performance perspective I can see the desire to do a merge to the left approach that Matthias is proposing. > Alternatively, we might need a > bloomfilter_copy() for the case that one wants to merge without > destroying any of the original filters.  Could we just make the existing copy bif work with it? Now that I know how that's implemented internally though I'm already skeptical of the idea. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From bro at tracker.bro.org Wed Jun 5 08:57:46 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 05 Jun 2013 15:57:46 -0000 Subject: [Bro-Dev] #869: Mechanism for discovering current script name In-Reply-To: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> References: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> Message-ID: <057.78b54576bc50a3e2fe4b6d90664d9bc9@tracker.bro.org> #869: Mechanism for discovering current script name ----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: language ----------------------------+------------------------ Comment (by jsiwek): Replying to [comment:6 robin]: > Thanks. One more thing (I've already argued with Seth about this :) For me "path" suggests *full* path including the script's file, while this returns only the directory where the script is located in. Would you object to changing that? Alternatively, how about using {{{@DIR}}} instead of {{{@PATH}}}? (or maybe {{{@CWD}}}?) `@CWD` is confusing since it can expand to something that's actually not the working dir of the process. I see how `@PATH` can be ambiguous, I'll change it to `@DIR`. That can only be interpreted one way. I'll add `@FILENAME` to expand to the file name (no path), so you can do `@DIR + "/" + @FILENAME` for an absolute file path if that's needed. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jun 5 09:05:46 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 05 Jun 2013 16:05:46 -0000 Subject: [Bro-Dev] #869: Mechanism for discovering current script name In-Reply-To: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> References: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> Message-ID: <057.d9c5d445ca82bfbcb6df1fd9c34e5264@tracker.bro.org> #869: Mechanism for discovering current script name ----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: language ----------------------------+------------------------ Comment (by jsiwek): In [changeset:022ce2505f3423378d193e2b16fda873cb325c3c/bro]: {{{ #!CommitTicketReference repository="bro" revision="022ce2505f3423378d193e2b16fda873cb325c3c" Change @PATH to @DIR for clarity. Add @FILENAME. Addresses #869. @DIR expands to directory path of the script, @FILENAME expands to just the script file name without path. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jun 5 09:09:45 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 05 Jun 2013 16:09:45 -0000 Subject: [Bro-Dev] #1014: Weirds could be unique Message-ID: <046.1c55d17db3c1a29c73a338c0948e23d3@tracker.bro.org> #1014: Weirds could be unique ------------------------+----------------------------- Reporter: thorkill | Type: Feature Request Status: new | Priority: Low Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+----------------------------- It would be nice to have unique wireds for each "test" which generates it. "An explanation what each of those 'weirds' would represent would be helpful" Most iteresting for me are: fragment_protocol_inconsistency, fragment_size_inconsistency and truncated_IP. This will give you an overview about current names: {{{ grep Weird *.cc | cut -d'"' -f2 | sort | uniq -c | sort -n }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Wed Jun 5 09:15:52 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Wed, 5 Jun 2013 09:15:52 -0700 Subject: [Bro-Dev] Bloom filter interface In-Reply-To: <20130605150333.GJ3330@icir.org> References: <20130605150333.GJ3330@icir.org> Message-ID: > Is this merging bf2 into bf1? Would it make sense to return a new > "opaque of bloomfilter" instead? I was aiming for non-mutable semantics: the function leaves both bf1 and bf2 untouched, but merely returns bf1 + bf2, where "+" means union of the underlying bit vectors. If this turns out to be a memory problem after profiling, we could still add a void-returning in-place merge function. That said, we could also use the name "bloomfilter_unify" or "bloomfilter_union", as we may end up with intersection at some point (although I do not have a use case for it yet). Matthias From bro at tracker.bro.org Wed Jun 5 09:32:20 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 05 Jun 2013 16:32:20 -0000 Subject: [Bro-Dev] #1015: weird.log contains binpac exceptions Message-ID: <044.df28939464f2108a59444758e09065b9@tracker.bro.org> #1015: weird.log contains binpac exceptions ------------------------+--------------------- Reporter: kraigu | Type: Problem Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- Contained in weird.log. We're running a git version from 26 February. {{{ 1370368807.759955 EEH6KdrWzNj 10.0.0.181 17926 76.192.60.215 514 binpac exception: string mismatch at /usr/local/src/bro-git/src/syslog-protocol.pac:8: \x0aexpected pattern: "[[:digit:]]+"\x0aactual data: "\x02\x0ch1\x80]T" - F worker-5 1370368811.151104 EEH6KdrWzNj 10.0.0.181 17926 76.192.60.215 514 binpac exception: string mismatch at /usr/local/src/bro-git/src/syslog-protocol.pac:8: \x0aexpected pattern: "[[:digit:]]+"\x0aactual data: "\x02\x0ch1\xb4\x1e\xca" - F worker-5 1370370685.869123 bgl21hARmjf 10.0.0.229 29263 190.23.207.127 514 binpac exception: string mismatch at /usr/local/src/bro-git/src/syslog-protocol.pac:8: \x0aexpected pattern: "[[:digit:]]+"\x0aactual data: "`\xdaf\xaa6!\xf3\x94\x11<,p\x9e9^\xdc\x8f\x88D\x906F\xf0&r\xd2\xc0\x8b\xc3\xff.-\x0f" - F worker-1 1370371495.244206 NcgBSHJfqP1 10.0.1.165 2471 201.145.199.174 514 binpac exception: string mismatch at /usr/local/src/bro-git/src/syslog-protocol.pac:8: \x0aexpected pattern: "[[:digit:]]+"\x0aactual data: "`\x88\x87sP\x8c\x92\xd6`\xce\xb1\xc8J{\xa9\x98%\xad79\xd8\xa2Bm\xa2\x02\xfa%\x1e0\x9c\x0e\x97" - F worker-6 1370371495.257415 NcgBSHJfqP1 10.0.1.165 2471 201.145.199.174 514 binpac exception: string mismatch at /usr/local/src/bro-git/src/syslog-protocol.pac:8: \x0aexpected pattern: "[[:digit:]]+"\x0aactual data: "a\x88\x87sP\x8c\x92\xd6`\xce\xb1\xc8J{\xa9\x98%\x02\xd5\x80\x97P5\x18@\xe3\xdej\x84\xe3\xa9"HK\xbd\x90E\x05\x9f@\x9f@\x01\x9a\xf5\xf8QW\xe1\x80\xc1`ym\x04\x8f\xea\xbd\xad\xc9\xfa:L\x0d \x0d " - F worker-6 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Jun 5 09:35:35 2013 From: seth at icir.org (Seth Hall) Date: Wed, 5 Jun 2013 12:35:35 -0400 Subject: [Bro-Dev] Bloom filter interface In-Reply-To: References: <20130605150333.GJ3330@icir.org> Message-ID: <1E090D7A-4D5C-4DC4-9147-42B217F2F558@icir.org> On Jun 5, 2013, at 12:15 PM, Matthias Vallentin wrote: >> Is this merging bf2 into bf1? Would it make sense to return a new >> "opaque of bloomfilter" instead? > > I was aiming for non-mutable semantics: the function leaves both bf1 > and bf2 untouched, but merely returns bf1 + bf2, where "+" means union > of the underlying bit vectors. So it sounds like having the merge function returning bool was an accident? Seems like we're all on the same page. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From vallentin at icir.org Wed Jun 5 09:51:16 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Wed, 5 Jun 2013 09:51:16 -0700 Subject: [Bro-Dev] Bloom filter interface In-Reply-To: <1E090D7A-4D5C-4DC4-9147-42B217F2F558@icir.org> References: <20130605150333.GJ3330@icir.org> <1E090D7A-4D5C-4DC4-9147-42B217F2F558@icir.org> Message-ID: > So it sounds like having the merge function returning bool was an accident? Exactly, well-spotted! Matthias From bro at tracker.bro.org Wed Jun 5 12:12:25 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 05 Jun 2013 19:12:25 -0000 Subject: [Bro-Dev] #1016: Option to extend uids to 128 bit Message-ID: <043.aebd5292dd16602cbdbf4057776e2ca2@tracker.bro.org> #1016: Option to extend uids to 128 bit ------------------------+----------------------------- Reporter: rhave | Type: Feature Request Status: new | Priority: Low Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+----------------------------- Bro's uids are currently 64 bits, which makes them collide with a 50% chance after 5.1 x 10^9^ different uids (see http://en.wikipedia.org/wiki/Birthday_problem#Probability_table). I'm currently generating uuids of 128 bit to replace the native uids in bro, as I'm using them as keys in a database, but this requires rewriting of the bro-logs. I suspect that more people could benefit from an option to extend the uids to 128 bit. I've made a quick and dirty patch to change most of the uids to 128 bit (file_analysis uids are missing). The patch is ugly, and is only to show some of the functionality I would like: http://pastebin.com/GkaGejNc -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jun 5 12:28:06 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 05 Jun 2013 19:28:06 -0000 Subject: [Bro-Dev] #1016: Option to extend uids to 128 bit In-Reply-To: <043.aebd5292dd16602cbdbf4057776e2ca2@tracker.bro.org> References: <043.aebd5292dd16602cbdbf4057776e2ca2@tracker.bro.org> Message-ID: <058.0e3d352044e50f71c1eb85c10e40f41d@tracker.bro.org> #1016: Option to extend uids to 128 bit ------------------------------+------------------------ Reporter: rhave | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------------+------------------------ Comment (by seth): I told him to go ahead file this ticket here so we could discuss it. I'm kind of on the fence about the idea. I kind of like it because it feels like something I'd propose, but I don't want to create unbearable uid's either. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at illinois.edu Wed Jun 5 14:36:56 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Wed, 5 Jun 2013 21:36:56 +0000 Subject: [Bro-Dev] Plugin branch status In-Reply-To: <20130518011717.GB67280@icir.org> References: <20130518011717.GB67280@icir.org> Message-ID: On May 17, 2013, at 8:17 PM, Robin Sommer wrote: > what we'll need to do is switching from a purely file-based model to > documenting semantic groups, like a specific analyzer. What would be the minimum we could get away with for now? I'm thinking: 1) Replace the "Built-In Functions (BIFs)" bullet under "Script Reference" with a "Plugin Reference" bullet. (better name to use here?) 2) The Plugin Reference page will have a section for each plugin. 3) Each plugin section lists its components (e.g. protocol analyzer w/ tag declaration) and BIF-items. 4) For the BIF-items, the reST markup would just use some directive to duplicate the content from where they're originally declared (the per-file documentation). Alternatively it could just be a link to that declaration; I'll probably try that first since it's easier, but I feel like it won't be enough. I want to avoid doing too much that I'd just throw away when I do the Broxygen overhaul (config files + sphinx directives), but I think at least this much will be re-usable as a sort of "plugin index" target. If there's more we need to do than have a new "plugin index", I think I should just start working on the Broxygen overhaul/rewrite. I'll probably be doing that soon anyway, but didn't expect to fit it in before next release. - Jon From bro at tracker.bro.org Wed Jun 5 22:37:27 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 06 Jun 2013 05:37:27 -0000 Subject: [Bro-Dev] #1017: MPLS in VLAN not considered/stripped Message-ID: <045.033cb3125cc0ae255b9de5f12aaec4c6@tracker.bro.org> #1017: MPLS in VLAN not considered/stripped ------------------------+--------------------- Reporter: ckanich | Type: Problem Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- PktSrc.cc doesn't seem to consider mpls packets in vlan. Here's a quick patch. I couldn't figure a straightforward way to incorporate L2 stripping so there's some code duplication. -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Thu Jun 6 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Thu, 6 Jun 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306060700.r56703HD030362@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 869 [1] | seth | | Normal | Mechanism for discovering current script name [1] #869: http://tracker.bro.org/bro/ticket/869 From robin at icir.org Thu Jun 6 10:23:14 2013 From: robin at icir.org (Robin Sommer) Date: Thu, 6 Jun 2013 10:23:14 -0700 Subject: [Bro-Dev] Plugin branch status In-Reply-To: References: <20130518011717.GB67280@icir.org> Message-ID: <20130606172314.GI16176@icir.org> On Wed, Jun 05, 2013 at 21:36 +0000, you wrote: > 1) Replace the "Built-In Functions (BIFs)" bullet under "Script Reference" with a "Plugin Reference" bullet. (better name to use here?) We could just call it "Analyzer Reference" for now because there aren't any other plugins. I'm not quite sure how it should look longer term once we have different component types, in particular because a single plugin can provide different ones (like a reader and writer). We probably want to group them by component type eventually, plus a separate plugin index listing all of a plugin's components. > 2) The Plugin Reference page will have a section for each plugin. Ack. > 3) Each plugin section lists its components (e.g. protocol analyzer w/ tag declaration) and BIF-items. Ack. (For later: we should add support for a README.rst coming with a plugin that would go to the top of its section; and probably also additional README.rst for each component it provides). > 4) For the BIF-items, the reST markup would just use some directive to > duplicate the content from where they're originally declared (the > per-file documentation). Alternatively it could just be a link to > that declaration; I'll probably try that first since it's easier, but > I feel like it won't be enough. It would be nice to have the content in there directly; duplicating is fine for now if it doesn't mess up indexing/linking etc? So, yes, I think this a good short-term compromise. I certainly want avoid doing something now that we'd just be throwing away soon, and I see that doing the full Broxygen overhaul will take more time. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From jsiwek at illinois.edu Thu Jun 6 11:14:30 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Thu, 6 Jun 2013 18:14:30 +0000 Subject: [Bro-Dev] Plugin branch status In-Reply-To: <20130606172314.GI16176@icir.org> References: <20130518011717.GB67280@icir.org> <20130606172314.GI16176@icir.org> Message-ID: >> 1) Replace the "Built-In Functions (BIFs)" bullet under "Script Reference" with a "Plugin Reference" bullet. (better name to use here?) > > We could just call it "Analyzer Reference" for now because there > aren't any other plugins. I expect there be File Analyzer plugins real soon. Making another page for that shouldn't be difficult in the short-term, but... > I'm not quite sure how it should look longer term once we have > different component types, in particular because a single plugin can > provide different ones (like a reader and writer). We probably want to > group them by component type eventually, plus a separate plugin index > listing all of a plugin's components. Separating by "plugin type" feels awkward because the type distinction doesn't actually exist. It's implicit by peeking at the components that belong to a plugin, but they aren't guaranteed to be all the same component type in the future. But we can probably just wait and see what component mixtures are there in the future and decide how to categorize them then. Another thing is that the bif-items are associated with a plugin, not with particular components. So if we want docs-per-component, it's not clear what bif-items go with what component. - Jon From robin at icir.org Thu Jun 6 11:48:59 2013 From: robin at icir.org (Robin Sommer) Date: Thu, 6 Jun 2013 11:48:59 -0700 Subject: [Bro-Dev] Plugin branch status In-Reply-To: References: <20130518011717.GB67280@icir.org> <20130606172314.GI16176@icir.org> Message-ID: <20130606184859.GM16176@icir.org> On Thu, Jun 06, 2013 at 18:14 +0000, you wrote: > I expect there be File Analyzer plugins real soon. Looking forward to them. :) > Separating by "plugin type" feels awkward because the type distinction > doesn't actually exist. It's implicit by peeking at the components > that belong to a plugin, but they aren't guaranteed to be all the same > component type in the future. That's why I thought to eventually group by component type, e.g.: Analyzers ========= HTTP DNS ... Readers ======= ... Writers ======= ... Independent of which plugin the components are coming from, they'd all be listed in the corresponding section. I imagine this to be the main view users will use for the documention (e.g., "what writers can I use?", "what functionality does the ascii writer provide?") And then we'd add a separate plugin list that links to there: Plugins ======= SQLite ------ Reader -> [reader page] Writer -> [writer page] You're bringing up a good point regarding the bifs though, as they are indeed associdated with the plugin, not the components. Not sure where to fit them in. Anyways, I agree with this: > But we can probably just wait and see what component mixtures are > there in the future and decide how to categorize them then. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From bro at tracker.bro.org Thu Jun 6 12:44:34 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 06 Jun 2013 19:44:34 -0000 Subject: [Bro-Dev] #869: Mechanism for discovering current script name In-Reply-To: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> References: <042.36d037e133f1d0dfc1048d3dd1033190@tracker.bro.org> Message-ID: <057.436abb2d6d6ca25b01bbaae611128e29@tracker.bro.org> #869: Mechanism for discovering current script name ----------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: language ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [changeset:203df4fa6b310e63ea311921208c18838975324e/bro]: {{{ #!CommitTicketReference repository="bro" revision="203df4fa6b310e63ea311921208c18838975324e" Merge remote-tracking branch 'origin/topic/jsiwek/869' * origin/topic/jsiwek/869: Change @PATH to @DIR for clarity. Add @FILENAME. Addresses #869. Make @PATH always return absolute path. Addresses #869. Add @PATH bro script macro. Addresses #869. Closes #869. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Jun 6 13:53:10 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 06 Jun 2013 20:53:10 -0000 Subject: [Bro-Dev] #1018: Invalid free() in H3 destructor Message-ID: <046.a563e71004816d99b1654a25a456d8d9@tracker.bro.org> #1018: Invalid free() in H3 destructor ----------------------+------------------------ Reporter: matthias | Owner: matthias Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Keywords: | ----------------------+------------------------ The class `H3` exhibits a bug in the destructor which free non-allocated memory. This went unnoticed for quite a while, presumably due to the singleton nature of this class. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Jun 6 13:54:10 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 06 Jun 2013 20:54:10 -0000 Subject: [Bro-Dev] #1018: Invalid free() in H3 destructor In-Reply-To: <046.a563e71004816d99b1654a25a456d8d9@tracker.bro.org> References: <046.a563e71004816d99b1654a25a456d8d9@tracker.bro.org> Message-ID: <061.e041b13ee5cbf34669d231c12e050542@tracker.bro.org> #1018: Invalid free() in H3 destructor -----------------------+------------------------ Reporter: matthias | Owner: matthias Type: Problem | Status: closed Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: -----------------------+------------------------ Changes (by matthias): * status: new => closed * resolution: => fixed Comment: In [changeset:fde081c30f6b1a5c13b2cd5a53872f9b2241bffb/bro]: {{{ #!CommitTicketReference repository="bro" revision="fde081c30f6b1a5c13b2cd5a53872f9b2241bffb" Remove invalid free on non-allocated pointer. The byte_lookup member is a fixed-size 2D array and should not be freed in the destructor. Fixes #1018. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Jun 6 13:54:43 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 06 Jun 2013 20:54:43 -0000 Subject: [Bro-Dev] #1018: Invalid free() in H3 destructor In-Reply-To: <046.a563e71004816d99b1654a25a456d8d9@tracker.bro.org> References: <046.a563e71004816d99b1654a25a456d8d9@tracker.bro.org> Message-ID: <061.140f154a7edf439b284e1465cb9a3542@tracker.bro.org> #1018: Invalid free() in H3 destructor -----------------------+------------------------ Reporter: matthias | Owner: matthias Type: Problem | Status: reopened Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Changes (by matthias): * status: closed => reopened * resolution: fixed => -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Jun 6 13:55:52 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 06 Jun 2013 20:55:52 -0000 Subject: [Bro-Dev] #1018: Invalid free() in H3 destructor In-Reply-To: <046.a563e71004816d99b1654a25a456d8d9@tracker.bro.org> References: <046.a563e71004816d99b1654a25a456d8d9@tracker.bro.org> Message-ID: <061.4cc1edd865eb082a91c3e150f56dbfff@tracker.bro.org> #1018: Invalid free() in H3 destructor ----------------------------+------------------------ Reporter: matthias | Owner: matthias Type: Merge Request | Status: reopened Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by matthias): * type: Problem => Merge Request Old description: > The class `H3` exhibits a bug in the destructor which free non-allocated > memory. This went unnoticed for quite a while, presumably due to the > singleton nature of this class. New description: The class `H3` exhibits a bug in the destructor which free non-allocated memory. This went unnoticed for quite a while, presumably due to the singleton nature of this class. Fixed and ready to merge from `topic/matthias/h3-dtor-fix`. -- -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Thu Jun 6 14:05:48 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Thu, 6 Jun 2013 14:05:48 -0700 Subject: [Bro-Dev] Maximum number of bytes users should be able to allocate Message-ID: When constructing a Bloom filter, users specify a desired false-positive rate and a capacity (max number of elements which guarantee the FP rate), which the implementation uses to allocate internal storage. Imprudent parametrization can lead to values beyond the machines memory limits, which would immediately crash Bro. Should we expect that users only do reasonable parametrizations by clearly documenting the effects of the parameters? Or should we create hard caps in the implementation to avoid users shooting themselves in the foot? Matthias From david at mandelberg.org Thu Jun 6 15:39:24 2013 From: david at mandelberg.org (David Mandelberg) Date: Thu, 06 Jun 2013 18:39:24 -0400 Subject: [Bro-Dev] Maximum number of bytes users should be able to allocate In-Reply-To: References: Message-ID: <27f3d4899d31d19805a2a5de74ce08bd@mail.mandelberg.org> Hi, On Thu, 6 Jun 2013 14:05:48 -0700, Matthias Vallentin wrote: > When constructing a Bloom filter, users specify a desired > false-positive rate and a capacity (max number of elements which > guarantee the FP rate), which the implementation uses to allocate > internal storage. Imprudent parametrization can lead to values beyond > the machines memory limits, which would immediately crash Bro. Should > we expect that users only do reasonable parametrizations by clearly > documenting the effects of the parameters? Or should we create hard > caps in the implementation to avoid users shooting themselves in the > foot? If you do decide to have hard caps, could those caps be easily configurable? An amount that might crash on one machine today could be entirely reasonable on another in a few years (or even on a more expensive machine today). -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ From noreply at bro.org Fri Jun 7 00:00:08 2013 From: noreply at bro.org (Merge Tracker) Date: Fri, 7 Jun 2013 00:00:08 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306070700.r577087Z008714@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1018 [1] | matthias | matthias | Medium | Invalid free() in H3 destructor [1] #1018: http://tracker.bro.org/bro/ticket/1018 From bro at tracker.bro.org Fri Jun 7 11:33:00 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Fri, 07 Jun 2013 18:33:00 -0000 Subject: [Bro-Dev] #1019: topic/jsiwek/plugin-docs Message-ID: <044.37226bc69187528df953997207fe87e7@tracker.bro.org> #1019: topic/jsiwek/plugin-docs ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This adds a protocol analyzer reference page to the script reference docs in place of the index page that used to list links to docs for each individual .bif file. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Fri Jun 7 16:40:09 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Fri, 07 Jun 2013 23:40:09 -0000 Subject: [Bro-Dev] #1018: Invalid free() in H3 destructor In-Reply-To: <046.a563e71004816d99b1654a25a456d8d9@tracker.bro.org> References: <046.a563e71004816d99b1654a25a456d8d9@tracker.bro.org> Message-ID: <061.405bba1f6ce7979285337d787570ccfe@tracker.bro.org> #1018: Invalid free() in H3 destructor ----------------------------+------------------------ Reporter: matthias | Owner: matthias Type: Merge Request | Status: closed Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * status: reopened => closed * resolution: => fixed Comment: In [changeset:b426040ccf73a9f771fd196bfb160e0f7b8a8cf3/bro]: {{{ #!CommitTicketReference repository="bro" revision="b426040ccf73a9f771fd196bfb160e0f7b8a8cf3" Merge remote-tracking branch 'origin/topic/matthias/h3-dtor-fix' * origin/topic/matthias/h3-dtor-fix: Remove invalid free on non-allocated pointer. Closes #1018. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Sat Jun 8 00:00:06 2013 From: noreply at bro.org (Merge Tracker) Date: Sat, 8 Jun 2013 00:00:06 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306080700.r587066h018434@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1019 [1] | jsiwek | | Medium | topic/jsiwek/plugin-docs [2] [1] #1019: http://tracker.bro.org/bro/ticket/1019 [2] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs From bro at tracker.bro.org Sat Jun 8 17:09:18 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Sun, 09 Jun 2013 00:09:18 -0000 Subject: [Bro-Dev] #985: 'tail -f' functionality for file reading in input framework In-Reply-To: <047.b23ffeaa4950cac11fbb46a0ad111ed0@tracker.bro.org> References: <047.b23ffeaa4950cac11fbb46a0ad111ed0@tracker.bro.org> Message-ID: <062.d65508e04958a68e7b4321a9663d9fd7@tracker.bro.org> #985: 'tail -f' functionality for file reading in input framework ------------------------------+------------------------ Reporter: scampbell | Owner: amannb Type: Feature Request | Status: assigned Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------------+------------------------ Changes (by amannb): * owner: => amannb * status: new => assigned Comment: I pondered about this a bit - I think it would be a good idea to offer this functionality, but not as a new input mode but instead as a new configuration option for the config.map. -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Sun Jun 9 00:00:05 2013 From: noreply at bro.org (Merge Tracker) Date: Sun, 9 Jun 2013 00:00:05 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306090700.r59705QV009061@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1019 [1] | jsiwek | | Medium | topic/jsiwek/plugin-docs [2] [1] #1019: http://tracker.bro.org/bro/ticket/1019 [2] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs From bro at tracker.bro.org Sun Jun 9 10:16:45 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Sun, 09 Jun 2013 17:16:45 -0000 Subject: [Bro-Dev] #1016: Option to extend uids to 128 bit In-Reply-To: <043.aebd5292dd16602cbdbf4057776e2ca2@tracker.bro.org> References: <043.aebd5292dd16602cbdbf4057776e2ca2@tracker.bro.org> Message-ID: <058.6484faec676eff4fcd7edcb613f7c280@tracker.bro.org> #1016: Option to extend uids to 128 bit ------------------------------+------------------------ Reporter: rhave | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------------+------------------------ Comment (by grigorescu): Just to kick off the discussion: I've been thinking about this one. So, I can understand the desire to move off of 64-bit UIDs, but I don't understand the immediate jump to 128-bit. Currently, UIDs are 11 characters long. Jumping to 128 bits would require doubling it, to 22 characters. I think 128 bits is overkill. Assuming a *very* busy network that's doing 100,000 connections/second, and a retention period of a year, we would get 3.2 x 10^12^ connections. Let's say we want a .1% chance of collision - we would only need 92 bits for that number of connections, which would be a 16 character UID. While poking around on this, it seems like current implementation could be more efficient. Specifically: {{{ // util.cc line 501 do { str[i++] = dig[v % base]; v /= base; } while ( v && i < n - 1 ); }}} To convert each uid from numeric into text format, we're repeatedly dividing by 62 (26 upper/lower case letters + 10 digits make up the string representation). Could we just add 2 characters to the string representation, to make it bit-shift friendly division by 64? Maybe '@' and '%' ? If we do find 2 more characters that we can add, I think it'd make sense to use 96 bits for the UID, as it would also fit into 16 characters. Personally, I would support extending the UID field to 16 characters (less than a 50% increase over what it currently is), so we can say with some confidence that UIDs actually are unique. -- Ticket URL: Bro Tracker Bro Issue Tracker From vladg at cmu.edu Sun Jun 9 12:55:48 2013 From: vladg at cmu.edu (Vlad Grigorescu) Date: Sun, 9 Jun 2013 19:55:48 +0000 Subject: [Bro-Dev] Should Bro Ignore PCAP Checksums by Default? Message-ID: <1202BE242E080642B0CD0AD0A03E8552C83B64@PGH-MSGMB-03.andrew.ad.cmu.edu> Just wanted to offer this up for discussion: Someone recently asked me if there were any "gotchas" to trying Bro. The only thing that I could think of is that if you're reading a PCAP with incorrect checksums, you need to use the -C flag. Having to point this out got me thinking - should this not be the default behavior? Bro already logs a weird for incorrect checksums; does it really make sense to have it ignore those packets? Should the option be flipped, to "enable strict checksum verification," or something like that? --Vlad From bro at tracker.bro.org Sun Jun 9 13:59:49 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Sun, 09 Jun 2013 20:59:49 -0000 Subject: [Bro-Dev] #1020: TLSv1.2 Support Message-ID: <049.6691fc15a068a27b0f83b980af157585@tracker.bro.org> #1020: TLSv1.2 Support -------------------------+----------------------------- Reporter: liamrandall | Type: Feature Request Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: -------------------------+----------------------------- Starting to see this in production as sites migrate to TLSv1.2; TLS Record Fragmentation also not implemented (not in wireshark either). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Sun Jun 9 16:34:07 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Sun, 09 Jun 2013 23:34:07 -0000 Subject: [Bro-Dev] #1021: merge topic/bernhard/input-update Message-ID: <044.9e7ba994ecc2870deb40b091909ee1d1@tracker.bro.org> #1021: merge topic/bernhard/input-update ---------------------------+------------------------ Reporter: amannb | Owner: Type: Merge Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ The branch contains a rewrite of the raw input reader which is much more stable and offers more features. All tests still seem to pass. -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Mon Jun 10 00:00:01 2013 From: noreply at bro.org (Merge Tracker) Date: Mon, 10 Jun 2013 00:00:01 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306100700.r5A7012C019167@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1019 [1] | jsiwek | | Medium | topic/jsiwek/plugin-docs [2] Bro | 1021 [3] | amannb | | Medium | merge topic/bernhard/input-update [4] [1] #1019: http://tracker.bro.org/bro/ticket/1019 [2] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [3] #1021: http://tracker.bro.org/bro/ticket/1021 [4] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update From bro at tracker.bro.org Mon Jun 10 07:51:22 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 10 Jun 2013 14:51:22 -0000 Subject: [Bro-Dev] #1017: MPLS in VLAN not considered/stripped In-Reply-To: <045.033cb3125cc0ae255b9de5f12aaec4c6@tracker.bro.org> References: <045.033cb3125cc0ae255b9de5f12aaec4c6@tracker.bro.org> Message-ID: <060.76ce1ae81bb3d56d09a7efcfe967ff25@tracker.bro.org> #1017: MPLS in VLAN not considered/stripped ----------------------+------------------------ Reporter: ckanich | Owner: Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by robin): Do you have a small trace we use for the test suite? -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon Jun 10 07:51:59 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 10 Jun 2013 07:51:59 -0700 Subject: [Bro-Dev] #1016: Option to extend uids to 128 bit In-Reply-To: <058.6484faec676eff4fcd7edcb613f7c280@tracker.bro.org> References: <043.aebd5292dd16602cbdbf4057776e2ca2@tracker.bro.org> <058.6484faec676eff4fcd7edcb613f7c280@tracker.bro.org> Message-ID: <20130610145159.GC72604@icir.org> On Sun, Jun 09, 2013 at 17:16 -0000, you wrote: > I've been thinking about this one. So, I can understand the desire to move > off of 64-bit UIDs, but I don't understand the immediate jump to 128-bit. That was my initial thought too when seeing the ticket: why not find some middle-ground (except that I didn't do the math :) > While poking around on this, it seems like current implementation could be > more efficient. Specifically: (It's not great, but I don't think this should have much of a performance impact.) -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From bro at tracker.bro.org Mon Jun 10 07:56:08 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 10 Jun 2013 14:56:08 -0000 Subject: [Bro-Dev] #1016: Option to extend uids to 128 bit In-Reply-To: <043.aebd5292dd16602cbdbf4057776e2ca2@tracker.bro.org> References: <043.aebd5292dd16602cbdbf4057776e2ca2@tracker.bro.org> Message-ID: <058.e635b198586e80016e9054184c6d27a3@tracker.bro.org> #1016: Option to extend uids to 128 bit ------------------------------+------------------------ Reporter: rhave | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------------+------------------------ Comment (by robin): On Sun, Jun 09, 2013 at 17:16 -0000, you wrote: > I've been thinking about this one. So, I can understand the desire to move > off of 64-bit UIDs, but I don't understand the immediate jump to 128-bit. That was my initial thought too when seeing the ticket: why not find some middle-ground (except that I didn't do the math :) > While poking around on this, it seems like current implementation could be > more efficient. Specifically: (It's not great, but I don't think this should have much of a performance impact.) -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon Jun 10 07:58:55 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 10 Jun 2013 07:58:55 -0700 Subject: [Bro-Dev] Should Bro Ignore PCAP Checksums by Default? In-Reply-To: <1202BE242E080642B0CD0AD0A03E8552C83B64@PGH-MSGMB-03.andrew.ad.cmu.edu> References: <1202BE242E080642B0CD0AD0A03E8552C83B64@PGH-MSGMB-03.andrew.ad.cmu.edu> Message-ID: <20130610145855.GD72604@icir.org> On Sun, Jun 09, 2013 at 19:55 +0000, you wrote: > with incorrect checksums, you need to use the -C flag. Having to point > this out got me thinking - should this not be the default behavior? An argument for enabling the checksum check by default is that if a checksum is broken, one can't trust the content of the packet anymore, it could be just garbage, or truncated, and hence cause havoc later at protocol decoding. However, a counter argument to that is that Bro should be robust against broken packets anyways, even if the checksum is correct. Current git gives a warning when Bro believes that your packets generally have incorrect checksums and you should hence use -C. I'm hoping that will point people into the right direction more quickly. However, I think I also wouldn't object to changing the default, as it indeed has become a very common problem these days. > Bro already logs a weird for incorrect checksums; But if the input generally doesn't have correct checksums, we also don't really want all those logged as wierds. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From robin at icir.org Mon Jun 10 08:09:58 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 10 Jun 2013 08:09:58 -0700 Subject: [Bro-Dev] Maximum number of bytes users should be able to allocate In-Reply-To: References: Message-ID: <20130610150958.GF72604@icir.org> On Thu, Jun 06, 2013 at 14:05 -0700, you wrote: > Should we expect that users only do reasonable parametrizations by > clearly documenting the effects of the parameters? That would be my preference. It's hard to protect against users doing something unreasonable, and we have plenty other opportunities for that already. :) Caps tend to eventually cause problems somewhere. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From bro at tracker.bro.org Mon Jun 10 12:04:12 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 10 Jun 2013 19:04:12 -0000 Subject: [Bro-Dev] #1022: HTTP bogus events Message-ID: <046.82a78e00b753fcfb7fd3f94395204d3e@tracker.bro.org> #1022: HTTP bogus events ----------------------+--------------------- Reporter: thorkill | Type: Problem Status: new | Priority: High Milestone: Bro2.2 | Component: Bro Version: 2.1 | Keywords: http ----------------------+--------------------- I am using attached script to watch for suspected activity in http- connections. This happens a lot in our network: > 2013-06-10-16:32:00 HTTP::HTTP_strange_event 87.139.xxx.2xx:3916/tcp -> xx.xx.xx.xx:80/tcp (uid ngRQOFjBgsg) unknown_HTTP_method={Accept: text/*} (0 missed bytes) # 87.139.xxx.2xx = p57xxx4xx.dip0.t-ipconnect.de xx.xx.xx.xx = I can not find out what the problem is. httpd logs tell me that everything was just fine. In most cases it happens after some POST request but not all the time. I will provide a pcap if I catch it somehow. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Jun 10 14:08:41 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 10 Jun 2013 21:08:41 -0000 Subject: [Bro-Dev] #1001: File analysis framework tasks In-Reply-To: <043.9c013e245588fe246d6f8a529a01a055@tracker.bro.org> References: <043.9c013e245588fe246d6f8a529a01a055@tracker.bro.org> Message-ID: <058.49f02ed7a5a75562fc2cf11a36225904@tracker.bro.org> #1001: File analysis framework tasks ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Task => Merge Request Comment: All the points in this ticket addressed in `topic/jsiwek/faf-cleanup`, plus also #988 and added a fix for HTTP multipart messages. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Jun 10 14:29:55 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 10 Jun 2013 21:29:55 -0000 Subject: [Bro-Dev] #1001: File analysis framework tasks In-Reply-To: <043.9c013e245588fe246d6f8a529a01a055@tracker.bro.org> References: <043.9c013e245588fe246d6f8a529a01a055@tracker.bro.org> Message-ID: <058.0c1d32e0975c0b0e76703b080bb80242@tracker.bro.org> #1001: File analysis framework tasks ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): Forgot to mention the branch is also in `bro-testing` and `bro-testing- private` repos. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Jun 10 15:16:09 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 10 Jun 2013 22:16:09 -0000 Subject: [Bro-Dev] #1017: MPLS in VLAN not considered/stripped In-Reply-To: <045.033cb3125cc0ae255b9de5f12aaec4c6@tracker.bro.org> References: <045.033cb3125cc0ae255b9de5f12aaec4c6@tracker.bro.org> Message-ID: <060.b3868dc4def12edd30a4a9217ed52429@tracker.bro.org> #1017: MPLS in VLAN not considered/stripped ----------------------+------------------------ Reporter: ckanich | Owner: Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by ckanich): I'll try to cook something up with scapy when I get a chance. -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Mon Jun 10 21:23:03 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 10 Jun 2013 21:23:03 -0700 Subject: [Bro-Dev] Bloom filter merging Message-ID: I noticed a slight complication while implementing the Bloom filter merge functionality: in order for Bloom filters to be mergeable, they must have been filled with the same hash functions. Consequently, Bloom filters need to share their internal hash functions for meaningful results. This means that we need to preallocate them because their seed depends on calls to bro_random(), that is, if we instantiate a hash function later in the game, we may not get the same one across all cluster nodes. It's also unclear how many hash functions we need, I've seen corner cases of over 40 instances depending on poor parametrization, though the common case is around 4-8. The hash function H3 takes on space o(2304B), so maintaining a global pool of, say, 100 hash functions to be on the safe side would cost us o(2MB). For example, if a Bloom filter needs 5 hash functions, it will always grab the first 5 from the pool to ensure a deterministic setup. The pool needs to be instantiated directly after the seed has been initialized. Instead of allocating a fixed number of hash functions, we could also set aside a fixed number of seeds and use them to instantiate hash functions as we need them. Does that sound like a good strategy to pursue? Matthias From vallentin at icir.org Mon Jun 10 21:27:50 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 10 Jun 2013 21:27:50 -0700 Subject: [Bro-Dev] Maximum number of bytes users should be able to allocate In-Reply-To: <20130610150958.GF72604@icir.org> References: <20130610150958.GF72604@icir.org> Message-ID: > That would be my preference. It's hard to protect against users doing > something unreasonable, and we have plenty other opportunities for > that already. 'K, I'll go down this path. Matthias From noreply at bro.org Tue Jun 11 00:00:04 2013 From: noreply at bro.org (Merge Tracker) Date: Tue, 11 Jun 2013 00:00:04 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306110700.r5B704GA004663@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1019 [2] | jsiwek | | Medium | topic/jsiwek/plugin-docs [3] Bro | 1021 [4] | amannb | | Medium | merge topic/bernhard/input-update [5] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1019: http://tracker.bro.org/bro/ticket/1019 [3] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [4] #1021: http://tracker.bro.org/bro/ticket/1021 [5] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update From noreply at bro.org Wed Jun 12 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Wed, 12 Jun 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306120700.r5C70206001267@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1019 [2] | jsiwek | | Medium | topic/jsiwek/plugin-docs [3] Bro | 1021 [4] | amannb | | Medium | merge topic/bernhard/input-update [5] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1019: http://tracker.bro.org/bro/ticket/1019 [3] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [4] #1021: http://tracker.bro.org/bro/ticket/1021 [5] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update From bro at tracker.bro.org Wed Jun 12 13:29:46 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 12 Jun 2013 20:29:46 -0000 Subject: [Bro-Dev] #1013: redef'ing tables overwrites unrelated values In-Reply-To: <046.70c89aaf401c58227cd7d8cc363e89c1@tracker.bro.org> References: <046.70c89aaf401c58227cd7d8cc363e89c1@tracker.bro.org> Message-ID: <061.e51db8575b939fa531005b97e1c4bca6@tracker.bro.org> #1013: redef'ing tables overwrites unrelated values -----------------------+------------------------ Reporter: dmandelb | Owner: Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by jsiwek): In [changeset:ae5a75bad9ce7b774c286391203d220d87079a2d/bro]: {{{ #!CommitTicketReference repository="bro" revision="ae5a75bad9ce7b774c286391203d220d87079a2d" Fix redef of table index from clearing table. Addresses #1013. `redef foo["x"] = 1` now acts like `redef foo += { ["x"] = 1 }` instead of `redef foo = { ["x"] = 1 }`. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jun 12 13:31:41 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 12 Jun 2013 20:31:41 -0000 Subject: [Bro-Dev] #1013: redef'ing tables overwrites unrelated values In-Reply-To: <046.70c89aaf401c58227cd7d8cc363e89c1@tracker.bro.org> References: <046.70c89aaf401c58227cd7d8cc363e89c1@tracker.bro.org> Message-ID: <061.c7d21dbe10c67a49725d501b9b5ebab0@tracker.bro.org> #1013: redef'ing tables overwrites unrelated values ----------------------------+------------------------ Reporter: dmandelb | Owner: Type: Merge Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: `topic/jsiwek/1013` -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Thu Jun 13 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Thu, 13 Jun 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306130700.r5D702vb017316@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update From bro at tracker.bro.org Thu Jun 13 07:54:02 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 13 Jun 2013 14:54:02 -0000 Subject: [Bro-Dev] #948: add bif for URI -> binary decoding In-Reply-To: <047.c6024f01bb6ff791d93bf4e0a7d3022f@tracker.bro.org> References: <047.c6024f01bb6ff791d93bf4e0a7d3022f@tracker.bro.org> Message-ID: <062.83728c15f8f68106f0c7e94fc3bc1632@tracker.bro.org> #948: add bif for URI -> binary decoding ------------------------------+------------------------ Reporter: scampbell | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------------+------------------------ Comment (by jsiwek): Replying to [comment:1 scampbell]: > http://code.google.com/p/auditing- sshd/source/browse/trunk/bro_policy_2.0/bifmodd I'm having trouble seeing how that's different from `unescape_URI`? The return value of `unescape_URI` doesn't have non-ascii data escaped, but it will get rendered with it escaped in /x format when printing/logging: {{{ local f: file = open("test"); local f2: file = open("test2"); local s: string = unescape_URI("%FC"); print s; print f2, s; enable_raw_output(f); print f, s; }}} Prints `\xfc` to stdout, and the files have: {{{ $ hexdump test 0000000 fc 0000001 $ hexdump test2 0000000 5c 78 66 63 0a 0000005 }}} So I think the actual data in the return value is what you want, but certain usages may provide the escaping. Can you give an example of what you need? -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Thu Jun 13 20:45:33 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Thu, 13 Jun 2013 20:45:33 -0700 Subject: [Bro-Dev] Bloom filter merging In-Reply-To: References: Message-ID: I had a chat with Vern about this earlier today and the discussion yielded the following idea: Rather than reserving a fixed number of seeds at startup, we use a keyed hashing scheme that computes a seed to construct the i'th hash function as follows: seed_i = h(initial_seed || name || i) One can fix "initial_seed" at startup via an environment variable. The new element "name" represents a parameter that the user provide (optionally?) as part of the BiF bloomfilter_init. The counter "i" allows for generating an arbitrary number of hash functions. The function "h" shall be a consistent hash function, e.g., H3 seeded with the initial seed. Feedback welcome, Matthias From noreply at bro.org Fri Jun 14 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Fri, 14 Jun 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306140700.r5E703jm010209@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update From noreply at bro.org Sat Jun 15 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Sat, 15 Jun 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306150700.r5F702o4030961@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update From noreply at bro.org Sun Jun 16 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Sun, 16 Jun 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306160700.r5G702GG023369@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update From gc355804 at ohio.edu Sun Jun 16 09:51:21 2013 From: gc355804 at ohio.edu (Clark, Gilbert) Date: Sun, 16 Jun 2013 12:51:21 -0400 Subject: [Bro-Dev] Bloom filter merging In-Reply-To: References: , Message-ID: <1D657054C5E2994C98FE68E14DD10A2B36FCF09625@EXMAIL1.ohio.edu> What about: seed_i = h( ((name.length() == 0) ? initial_seed : name) || i) In my mind, it'd be cool if identical names would produce identical results (in this case) without the need to set an environment variable. Maybe there's a reason that's a bad idea, though? --Gilbert ________________________________________ From: bro-dev-bounces at bro.org [bro-dev-bounces at bro.org] On Behalf Of Matthias Vallentin [vallentin at icir.org] Sent: Thursday, June 13, 2013 11:45 PM To: bro-dev at bro.org Subject: Re: [Bro-Dev] Bloom filter merging I had a chat with Vern about this earlier today and the discussion yielded the following idea: Rather than reserving a fixed number of seeds at startup, we use a keyed hashing scheme that computes a seed to construct the i'th hash function as follows: seed_i = h(initial_seed || name || i) One can fix "initial_seed" at startup via an environment variable. The new element "name" represents a parameter that the user provide (optionally?) as part of the BiF bloomfilter_init. The counter "i" allows for generating an arbitrary number of hash functions. The function "h" shall be a consistent hash function, e.g., H3 seeded with the initial seed. Feedback welcome, Matthias _______________________________________________ bro-dev mailing list bro-dev at bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From noreply at bro.org Mon Jun 17 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Mon, 17 Jun 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306170700.r5H702ma028479@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update From vallentin at icir.org Mon Jun 17 22:54:30 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 17 Jun 2013 22:54:30 -0700 Subject: [Bro-Dev] Bloom filter merging In-Reply-To: <1D657054C5E2994C98FE68E14DD10A2B36FCF09625@EXMAIL1.ohio.edu> References: <1D657054C5E2994C98FE68E14DD10A2B36FCF09625@EXMAIL1.ohio.edu> Message-ID: > seed_i = h( ((name.length() == 0) ? initial_seed : name) || i) This is how I actually implemented it internally currently. However, I do not think it will make a difference because CompHash is also seeded by initial_seed [1], and hashing the Bloom filter implementation uses CompHash to hash Val instances. To generate k different hash values I just stuff the CompHash output into k H3 instances that are seeded according to the formula above. Let x be an instance of a Val. Then the i'th hash value is: h_i(x) = H3_i(CompHash(x)) > In my mind, it'd be cool if identical names would produce identical results (in this case) without the need to set an environment variable. Maybe there's a reason that's a bad idea, though? Soumya pointed out the problem: there's a trade-off between making Bloom filters shareable on the one hand, and robust against attackers on the other hand. If we want to distribute Bloom filters at some point, we'll face the problem of running into different seeds in different deployments. Conceptually, we have two tuning knobs: the Bloom filter name, which is unique to the Bloom filter, and the environment variable BRO_SEED_FILE that determines initial_seed. Ideally our uses will not have to mess with their seed to use a Bloom filter someone wants to share. At this point, however, it is impossible to get rid of the initial_seed dependency due to the problem noted above. If there's a straight-forward way to hash an arbitrary Val in a different way, please let me know. Matthias [1] According to the documentation in Hash.cc, hashing a Val either uses H3 for short data and HMAC MD5 for arbitrary long data. Both the seed of the H3 instance as well as the HMAC key are a function of initial_seed. From noreply at bro.org Tue Jun 18 00:00:06 2013 From: noreply at bro.org (Merge Tracker) Date: Tue, 18 Jun 2013 00:00:06 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306180700.r5I706qT003865@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update From soumyabasu8 at gmail.com Mon Jun 17 20:50:27 2013 From: soumyabasu8 at gmail.com (Soumya Basu) Date: Mon, 17 Jun 2013 20:50:27 -0700 Subject: [Bro-Dev] Bloom filter merging In-Reply-To: <1D657054C5E2994C98FE68E14DD10A2B36FCF09625@EXMAIL1.ohio.edu> References: <1D657054C5E2994C98FE68E14DD10A2B36FCF09625@EXMAIL1.ohio.edu> Message-ID: I'm not entirely sure how much of a concern this is, but hashing solely based on the name would make the bloom filters vulnerable to attack since it's more predictable what the hash values are. Ideally, the names would be something simple and easily identifies what the filter is counting (from how I picture the names being used). That would put pressure on the user to choose names that are random or salted to keep the filters secure. I do like the idea of not setting an environment variable, but I'm not sure how much of a benefit it is. Thanks, Soumya Basu On Sun, Jun 16, 2013 at 9:51 AM, Clark, Gilbert wrote: > What about: > > seed_i = h( ((name.length() == 0) ? initial_seed : name) || i) > > In my mind, it'd be cool if identical names would produce identical results (in this case) without the need to set an environment variable. Maybe there's a reason that's a bad idea, though? > > --Gilbert > ________________________________________ > From: bro-dev-bounces at bro.org [bro-dev-bounces at bro.org] On Behalf Of Matthias Vallentin [vallentin at icir.org] > Sent: Thursday, June 13, 2013 11:45 PM > To: bro-dev at bro.org > Subject: Re: [Bro-Dev] Bloom filter merging > > I had a chat with Vern about this earlier today and the discussion > yielded the following idea: > > Rather than reserving a fixed number of seeds at startup, we use a > keyed hashing scheme that computes a seed to construct the i'th hash > function as follows: > > seed_i = h(initial_seed || name || i) > > One can fix "initial_seed" at startup via an environment variable. The > new element "name" represents a parameter that the user provide > (optionally?) as part of the BiF bloomfilter_init. The counter "i" > allows for generating an arbitrary number of hash functions. The > function "h" shall be a consistent hash function, e.g., H3 seeded with > the initial seed. > > Feedback welcome, > > Matthias > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From seth at icir.org Tue Jun 18 07:10:25 2013 From: seth at icir.org (Seth Hall) Date: Tue, 18 Jun 2013 10:10:25 -0400 Subject: [Bro-Dev] Should Bro Ignore PCAP Checksums by Default? In-Reply-To: <20130610145855.GD72604@icir.org> References: <1202BE242E080642B0CD0AD0A03E8552C83B64@PGH-MSGMB-03.andrew.ad.cmu.edu> <20130610145855.GD72604@icir.org> Message-ID: On Jun 10, 2013, at 10:58 AM, Robin Sommer wrote: > Current git gives a warning when Bro believes that your packets > generally have incorrect checksums and you should hence use -C. I'm > hoping that will point people into the right direction more quickly. > > However, I think I also wouldn't object to changing the default, as it > indeed has become a very common problem these days. I think we should keep the default with strict checksum checking, especially now that we have the new script that tells users if they seem to have invalid checksums. I would rather push people down the right path as much as possible. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From noreply at bro.org Wed Jun 19 00:00:05 2013 From: noreply at bro.org (Merge Tracker) Date: Wed, 19 Jun 2013 00:00:05 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306190700.r5J705WM020940@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update From bro at tracker.bro.org Wed Jun 19 10:03:23 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 19 Jun 2013 17:03:23 -0000 Subject: [Bro-Dev] #1016: Option to extend uids to 128 bit In-Reply-To: <043.aebd5292dd16602cbdbf4057776e2ca2@tracker.bro.org> References: <043.aebd5292dd16602cbdbf4057776e2ca2@tracker.bro.org> Message-ID: <058.701a21cf0d5f559358d459bca87b8df7@tracker.bro.org> #1016: Option to extend uids to 128 bit ------------------------------+------------------------ Reporter: rhave | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------------+------------------------ Comment (by seth): > That was my initial thought too when seeing the ticket: why not find > some middle-ground (except that I didn't do the math :) I'm fine with moving to 92-bit uids, we just need to make sure we do it for conn uids and file uids. It could be interesting to reduce the occurrences of uid reuse. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jun 19 10:04:30 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 19 Jun 2013 17:04:30 -0000 Subject: [Bro-Dev] #1017: MPLS in VLAN not considered/stripped In-Reply-To: <045.033cb3125cc0ae255b9de5f12aaec4c6@tracker.bro.org> References: <045.033cb3125cc0ae255b9de5f12aaec4c6@tracker.bro.org> Message-ID: <060.95b7b10836839b268d44eeab037186dd@tracker.bro.org> #1017: MPLS in VLAN not considered/stripped ----------------------+------------------------ Reporter: ckanich | Owner: Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): > I'll try to cook something up with scapy when I get a chance. Is this something you see on a real network though? I *really* prefer to see real traffic from deployed equipment rather than crafted packets. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jun 19 10:07:57 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 19 Jun 2013 17:07:57 -0000 Subject: [Bro-Dev] #1022: HTTP bogus events In-Reply-To: <046.82a78e00b753fcfb7fd3f94395204d3e@tracker.bro.org> References: <046.82a78e00b753fcfb7fd3f94395204d3e@tracker.bro.org> Message-ID: <061.8843b67872e49e357ad7cbd12fe90675@tracker.bro.org> #1022: HTTP bogus events -----------------------+-------------------- Reporter: thorkill | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro2.2 Component: Bro | Version: 2.1 Resolution: | Keywords: http -----------------------+-------------------- Comment (by seth): > xx.xx.xx.xx:80/tcp (uid ngRQOFjBgsg) > unknown_HTTP_method={Accept: text/*} (0 missed bytes) > # 87.139.xxx.2xx = p57xxx4xx.dip0.t-ipconnect.de xx.xx.xx.xx = Can you include the line from conn.log for one of these connections? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jun 19 10:32:36 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 19 Jun 2013 17:32:36 -0000 Subject: [Bro-Dev] #1023: const vector can be modified Message-ID: <046.59011b7d6bb58706b07e11af31bcf6e2@tracker.bro.org> #1023: const vector can be modified ------------------------+--------------------- Reporter: dmandelb | Type: Problem Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- The following code should probably be an error: {{{ const foo: vector of double = vector() &redef; foo[|foo|] = 42.0; print(foo); }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jun 19 10:52:52 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 19 Jun 2013 17:52:52 -0000 Subject: [Bro-Dev] #1017: MPLS in VLAN not considered/stripped In-Reply-To: <045.033cb3125cc0ae255b9de5f12aaec4c6@tracker.bro.org> References: <045.033cb3125cc0ae255b9de5f12aaec4c6@tracker.bro.org> Message-ID: <060.7b5b75b14ed06e21d08ae21dc05e33d1@tracker.bro.org> #1017: MPLS in VLAN not considered/stripped ----------------------+------------------------ Reporter: ckanich | Owner: Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by ckanich): It really is, I just don't have permission to move the packets (even anonymized) so simulated packets are my next best bet. On Jun 19, 2013, at 10:04 AM, "Bro Tracker" wrote: > #1017: MPLS in VLAN not considered/stripped > ----------------------+------------------------ > Reporter: ckanich | Owner: > Type: Problem | Status: new > Priority: Medium | Milestone: Bro2.2 > Component: Bro | Version: git/master > Resolution: | Keywords: > ----------------------+------------------------ > > Comment (by seth): > >> I'll try to cook something up with scapy when I get a chance. > > Is this something you see on a real network though? I *really* prefer to > see real traffic from deployed equipment rather than crafted packets. > > -- > Ticket URL: > Bro Tracker > Bro Issue Tracker -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jun 19 11:00:08 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 19 Jun 2013 18:00:08 -0000 Subject: [Bro-Dev] #1017: MPLS in VLAN not considered/stripped In-Reply-To: <045.033cb3125cc0ae255b9de5f12aaec4c6@tracker.bro.org> References: <045.033cb3125cc0ae255b9de5f12aaec4c6@tracker.bro.org> Message-ID: <060.755fcea95d5c3231d96978df2f0b750e@tracker.bro.org> #1017: MPLS in VLAN not considered/stripped ----------------------+------------------------ Reporter: ckanich | Owner: Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): > It really is, I just don't have permission to move the packets (even > anonymized) so simulated packets are my next best bet. Ok. Just promise me that your crafted packets are effectively the same as your real traffic. :) -- Ticket URL: Bro Tracker Bro Issue Tracker From ckanich at uic.edu Wed Jun 19 10:48:37 2013 From: ckanich at uic.edu (Chris Kanich) Date: Wed, 19 Jun 2013 10:48:37 -0700 Subject: [Bro-Dev] #1017: MPLS in VLAN not considered/stripped In-Reply-To: <060.95b7b10836839b268d44eeab037186dd@tracker.bro.org> References: <045.033cb3125cc0ae255b9de5f12aaec4c6@tracker.bro.org> <060.95b7b10836839b268d44eeab037186dd@tracker.bro.org> Message-ID: <8D43A8B7-A65B-4E1C-A16B-A7DAB97D134A@uic.edu> It really is, I just don't have permission to move the packets (even anonymized) so simulated packets are my next best bet. On Jun 19, 2013, at 10:04 AM, "Bro Tracker" wrote: > #1017: MPLS in VLAN not considered/stripped > ----------------------+------------------------ > Reporter: ckanich | Owner: > Type: Problem | Status: new > Priority: Medium | Milestone: Bro2.2 > Component: Bro | Version: git/master > Resolution: | Keywords: > ----------------------+------------------------ > > Comment (by seth): > >> I'll try to cook something up with scapy when I get a chance. > > Is this something you see on a real network though? I *really* prefer to > see real traffic from deployed equipment rather than crafted packets. > > -- > Ticket URL: > Bro Tracker > Bro Issue Tracker From bro at tracker.bro.org Wed Jun 19 15:10:08 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 19 Jun 2013 22:10:08 -0000 Subject: [Bro-Dev] #1024: make it possible to use redef to append to vectors Message-ID: <046.0bbc08c176f214d1179ab50d7527ead3@tracker.bro.org> #1024: make it possible to use redef to append to vectors ------------------------+----------------------------- Reporter: dmandelb | Type: Feature Request Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+----------------------------- On 2013-06-19 18:00, Siwek, Jonathan Luke wrote: > On Jun 19, 2013, at 11:14 AM, David Mandelberg wrote: > >> What's the recommended way to append to a vector? The documentation >> says vectors are like tables, so I tried the below code, but it gives >> some errors. >> >> const foo: vector of double = vector() &redef; >> >> redef foo += { >> [|foo|] = 42.0 >> }; > > Appending to a vector can't currently be done w/ redef, but I don't > think it would be difficult to implement if you want to add a ticket > to the tracker. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jun 19 15:55:33 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 19 Jun 2013 22:55:33 -0000 Subject: [Bro-Dev] #1025: internal error: over-ran key in CompositeHash::RecoverVals Message-ID: <046.0de19f29390d8892a4977149491e02ab@tracker.bro.org> #1025: internal error: over-ran key in CompositeHash::RecoverVals ------------------------+--------------------- Reporter: dmandelb | Type: Problem Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- {{{ $ bro bbn-correlator internal error: over-ran key in CompositeHash::RecoverVals Aborted }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Thu Jun 20 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Thu, 20 Jun 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306200700.r5K7038Q015009@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update From bro at tracker.bro.org Thu Jun 20 02:08:02 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 20 Jun 2013 09:08:02 -0000 Subject: [Bro-Dev] #1022: HTTP bogus events In-Reply-To: <046.82a78e00b753fcfb7fd3f94395204d3e@tracker.bro.org> References: <046.82a78e00b753fcfb7fd3f94395204d3e@tracker.bro.org> Message-ID: <061.02c96c1d5d915ce59dc8b159883aad72@tracker.bro.org> #1022: HTTP bogus events -----------------------+-------------------- Reporter: thorkill | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro2.2 Component: Bro | Version: 2.1 Resolution: | Keywords: http -----------------------+-------------------- Comment (by thorkill): It seems, that this connections does have missed_bytes, then why is this value at the time of the event 0? I have no idea, may be the content_gap occured later. {{{ 1370881915.933901 ngRQOFjBgsg 87.139.xxx.2xx 3916 xx.xx.xx.xx 80 tcp - 106.864390 4534779 8095 SF F 12064 ShADadfF 10077 14023089 5226 234741 (empty) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Jun 20 06:11:08 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 20 Jun 2013 13:11:08 -0000 Subject: [Bro-Dev] #1022: HTTP bogus events In-Reply-To: <046.82a78e00b753fcfb7fd3f94395204d3e@tracker.bro.org> References: <046.82a78e00b753fcfb7fd3f94395204d3e@tracker.bro.org> Message-ID: <061.7fc7af40b920fe810ffc330399893c40@tracker.bro.org> #1022: HTTP bogus events -----------------------+-------------------- Reporter: thorkill | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro2.2 Component: Bro | Version: 2.1 Resolution: | Keywords: http -----------------------+-------------------- Comment (by seth): Ohh, well you're using the c$conn$missed_bytes value to determined missed bytes but I don't know the order that the content_gap event is generated (which is where that data comes from). Ultimately the best thing would be if you could catch some of this traffic and get a tracefile to us. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Jun 20 10:02:12 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 20 Jun 2013 17:02:12 -0000 Subject: [Bro-Dev] #1024: make it possible to use redef to append to vectors In-Reply-To: <046.0bbc08c176f214d1179ab50d7527ead3@tracker.bro.org> References: <046.0bbc08c176f214d1179ab50d7527ead3@tracker.bro.org> Message-ID: <061.914de5aab778d648e47758f3c8602c51@tracker.bro.org> #1024: make it possible to use redef to append to vectors ------------------------------+------------------------ Reporter: dmandelb | Owner: Type: Feature Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------------+------------------------ Comment (by jsiwek): In [changeset:2f5c9b83ea92024b45fb33bc48f7079c686b717d/bro]: {{{ #!CommitTicketReference repository="bro" revision="2f5c9b83ea92024b45fb33bc48f7079c686b717d" Allow redef to append to vectors. Addresses #1024. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Jun 20 10:06:39 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 20 Jun 2013 17:06:39 -0000 Subject: [Bro-Dev] #1024: make it possible to use redef to append to vectors In-Reply-To: <046.0bbc08c176f214d1179ab50d7527ead3@tracker.bro.org> References: <046.0bbc08c176f214d1179ab50d7527ead3@tracker.bro.org> Message-ID: <061.1c37d95fce71ccf5da61a1f81ccb2e1a@tracker.bro.org> #1024: make it possible to use redef to append to vectors ----------------------------+------------------------ Reporter: dmandelb | Owner: Type: Merge Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Feature Request => Merge Request Comment: In `topic/jsiwek/1024`. Note the vector ctors/initializers don't use the "index = val" syntax like tables do: {{{ const foo: vector of double = vector( 21.0, 11.0, 8.0 ) &redef; redef foo += vector(42.0, 44.0, 50.0); redef foo += { 88.0, 99.0, 101.0 }; }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Jun 20 15:10:36 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 20 Jun 2013 22:10:36 -0000 Subject: [Bro-Dev] #1025: internal error: over-ran key in CompositeHash::RecoverVals In-Reply-To: <046.0de19f29390d8892a4977149491e02ab@tracker.bro.org> References: <046.0de19f29390d8892a4977149491e02ab@tracker.bro.org> Message-ID: <061.52e5b74cd1fcb490fba300edc672c07f@tracker.bro.org> #1025: internal error: over-ran key in CompositeHash::RecoverVals -----------------------+------------------------ Reporter: dmandelb | Owner: Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by jsiwek): If you're using git/master you can try: {{{ --- bbn-correlator/correlators/default.bro 2013-06-19 17:49:08.000000000 -0500 +++ bbn-correlator/correlators/default.new.bro 2013-06-20 16:48:20.000000000 -0500 @@ -12,10 +12,10 @@ export { const default_correlator: Correlator = [ $actions=set( - [ + Action( $begin_threshold=10.0, $begin_action=notice - ] + ) ) ] &redef; }}} That at least doesn't crash immediately. This is probably a better style of constructing records going forward, but if it does work for you still leave the ticket open since the composite table/set indexing for anonymous records definitely has a bug. -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Fri Jun 21 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Fri, 21 Jun 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306210700.r5L703rX001182@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] Bro | 1024 [7] | dmandelb | | Medium | make it possible to use redef to append to vectors [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update [7] #1024: http://tracker.bro.org/bro/ticket/1024 From bro at tracker.bro.org Fri Jun 21 14:21:30 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Fri, 21 Jun 2013 21:21:30 -0000 Subject: [Bro-Dev] #1026: runtime error with local set of record with optional fields Message-ID: <046.0182f9931d95c81f844bd188211c8963@tracker.bro.org> #1026: runtime error with local set of record with optional fields ------------------------+--------------------- Reporter: dmandelb | Type: Problem Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- This code: {{{ type Foo: record { a: count &optional; b: count &optional; }; event bro_init() { local foos: set[Foo] = {}; add foos[[$a=0]]; print(foos); } }}} Gives this output: {{{ error in and /home/dmandelb/reps/test.bro, line 7: index type doesn't match table ([a=0, b=] and list of any) { } }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Fri Jun 21 14:53:40 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Fri, 21 Jun 2013 21:53:40 -0000 Subject: [Bro-Dev] #1026: runtime error with local set of record with optional fields In-Reply-To: <046.0182f9931d95c81f844bd188211c8963@tracker.bro.org> References: <046.0182f9931d95c81f844bd188211c8963@tracker.bro.org> Message-ID: <061.7648ed6eeac3e3e2196162caa513bf4b@tracker.bro.org> #1026: runtime error with local set of record with optional fields -----------------------+------------------------ Reporter: dmandelb | Owner: Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by jsiwek): That's odd, seems like the assignment of `{}` messed up the type of `foos`. For now you can declare it like: {{{ local foos: set[Foo] = set(); }}} Or maybe better, since it's already implicitly an empty set, just omit the assignment of a different empty set. {{{ local foos: set[Foo]; }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From vern at icir.org Fri Jun 21 23:33:22 2013 From: vern at icir.org (Vern Paxson) Date: Fri, 21 Jun 2013 23:33:22 -0700 Subject: [Bro-Dev] Should Bro Ignore PCAP Checksums by Default? In-Reply-To: (Tue, 18 Jun 2013 10:10:25 EDT). Message-ID: <20130622063322.E2FD82C4008@rock.ICSI.Berkeley.EDU> > I think we should keep the default with strict checksum checking, especially now that we have the new script that tells users if they seem to have invalid checksums. I would rather push people down the right path as much as possible. My thoughts too. Vern From noreply at bro.org Sat Jun 22 00:00:04 2013 From: noreply at bro.org (Merge Tracker) Date: Sat, 22 Jun 2013 00:00:04 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306220700.r5M704A1003412@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] Bro | 1024 [7] | dmandelb | | Medium | make it possible to use redef to append to vectors [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update [7] #1024: http://tracker.bro.org/bro/ticket/1024 From noreply at bro.org Sun Jun 23 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Sun, 23 Jun 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306230700.r5N703sN023263@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] Bro | 1024 [7] | dmandelb | | Medium | make it possible to use redef to append to vectors [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update [7] #1024: http://tracker.bro.org/bro/ticket/1024 From noreply at bro.org Mon Jun 24 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Mon, 24 Jun 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306240700.r5O703TF021295@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] Bro | 1024 [7] | dmandelb | | Medium | make it possible to use redef to append to vectors [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update [7] #1024: http://tracker.bro.org/bro/ticket/1024 From noreply at bro.org Tue Jun 25 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Tue, 25 Jun 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306250700.r5P702SU015363@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] Bro | 1024 [7] | dmandelb | | Medium | make it possible to use redef to append to vectors [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update [7] #1024: http://tracker.bro.org/bro/ticket/1024 From seth at icir.org Tue Jun 25 07:40:47 2013 From: seth at icir.org (Seth Hall) Date: Tue, 25 Jun 2013 10:40:47 -0400 Subject: [Bro-Dev] file analysis extraction analyzer Message-ID: <341756C8-8284-4BAC-A058-324EA4D77442@icir.org> This is mostly intended for Jon, but I thought it'd be nice for everyone to see it. Jon, what do you think about adding extraction events for when an extraction begins and ends? They could be events like this? event file_extract_begin(f: fa_file, tag: Analyzer, args: AnalyzerArgs) event file_extract_end(f: fa_file, tag: Analyzer, args: AnalyzerArgs) I know that the events don't match how the core works (by splitting $tag out of args) but that's more along the lines of how I'm making the script land API look so I think it makes sense to split the event arguments out that way. This makes it much easier to write some of the scripts and should generally provide good feedback from the file extraction "analyzer". :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130625/67fc9a73/attachment.bin From jsiwek at illinois.edu Tue Jun 25 10:35:03 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Tue, 25 Jun 2013 17:35:03 +0000 Subject: [Bro-Dev] file analysis extraction analyzer In-Reply-To: <341756C8-8284-4BAC-A058-324EA4D77442@icir.org> References: <341756C8-8284-4BAC-A058-324EA4D77442@icir.org> Message-ID: On Jun 25, 2013, at 9:40 AM, Seth Hall wrote: > Jon, what do you think about adding extraction events for when an extraction begins and ends? They could be events like this? > > event file_extract_begin(f: fa_file, tag: Analyzer, args: AnalyzerArgs) > event file_extract_end(f: fa_file, tag: Analyzer, args: AnalyzerArgs) Generally sounds fine, but why is the tag needed? Unless there's plans to be different kinds of file extraction analyzers that re-use those events, won't it always be the same tag? Similarly, do you need the full args since the only relevant part of it is the file/path name? - Jon From noreply at bro.org Wed Jun 26 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Wed, 26 Jun 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306260700.r5Q702YK000788@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] Bro | 1024 [7] | dmandelb | | Medium | make it possible to use redef to append to vectors [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update [7] #1024: http://tracker.bro.org/bro/ticket/1024 From seth at icir.org Wed Jun 26 06:39:45 2013 From: seth at icir.org (Seth Hall) Date: Wed, 26 Jun 2013 09:39:45 -0400 Subject: [Bro-Dev] file analysis extraction analyzer In-Reply-To: References: <341756C8-8284-4BAC-A058-324EA4D77442@icir.org> Message-ID: <60B59F7F-C878-43CD-8E43-BBACC5F757BA@icir.org> On Jun 25, 2013, at 1:35 PM, "Siwek, Jonathan Luke" wrote: > Generally sounds fine, but why is the tag needed? Unless there's plans to be different kinds of file extraction analyzers that re-use those events, won't it always be the same tag? Similarly, do you need the full args since the only relevant part of it is the file/path name? After talking yesterday we both came to the conclusion that we should just add a single event that looks like this? event file_extract_end(f: fa_file, filename: string) This way we'll be able to spool currently downloading files into a temporary location and move them to their final location when the extraction is finished (and perhaps most importantly, do it all at script land). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Wed Jun 26 06:47:48 2013 From: seth at icir.org (Seth Hall) Date: Wed, 26 Jun 2013 09:47:48 -0400 Subject: [Bro-Dev] file analysis extraction analyzer In-Reply-To: <60B59F7F-C878-43CD-8E43-BBACC5F757BA@icir.org> References: <341756C8-8284-4BAC-A058-324EA4D77442@icir.org> <60B59F7F-C878-43CD-8E43-BBACC5F757BA@icir.org> Message-ID: <3B2BCA76-B58E-45F7-8B0A-106A9335F731@icir.org> On Jun 26, 2013, at 9:39 AM, Seth Hall wrote: > event file_extract_end(f: fa_file, filename: string) Sigh, I'm already reconsidering this. There is no functional difference between this and the file_state_remove event except for the filename which I should have available elsewhere within the fa_file record. Let me stew on this a bit more. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From noreply at bro.org Thu Jun 27 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Thu, 27 Jun 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306270700.r5R702Ww022572@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] Bro | 1024 [7] | dmandelb | | Medium | make it possible to use redef to append to vectors [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update [7] #1024: http://tracker.bro.org/bro/ticket/1024 From bro at tracker.bro.org Thu Jun 27 10:32:55 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 27 Jun 2013 17:32:55 -0000 Subject: [Bro-Dev] #1006: topic/dnthayer/broctl-testing In-Reply-To: <046.8bdcf826e8fe87f2956d3815443d8a31@tracker.bro.org> References: <046.8bdcf826e8fe87f2956d3815443d8a31@tracker.bro.org> Message-ID: <061.d8f3401dc895e6f1f66ac40201b8720d@tracker.bro.org> #1006: topic/dnthayer/broctl-testing -------------------------+------------------------ Reporter: dnthayer | Owner: dnthayer Type: Problem | Status: assigned Priority: Medium | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: | Keywords: -------------------------+------------------------ Comment (by dnthayer): Replying to [comment:1 robin]: > Good job on this! I've tweaked some things a little bit, mostly moving stuff around, but other than that it looks good to me. I've merged it in. > > I do see 3 tests failing though: > > {{{ > command.print-cluster > command.status-cluster > command.peerstatus-cluster > }}} > > They all fail with a time-out problem. I believe I remember a discussion of that but don't recall the details. Is that the result of some other change? Do you see that too? > Yes, those failures are being caused by a problem in the sumstats framework. Seth mentioned that he's working on fixing that. -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Fri Jun 28 00:00:05 2013 From: noreply at bro.org (Merge Tracker) Date: Fri, 28 Jun 2013 00:00:05 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306280700.r5S705vc026350@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] Bro | 1024 [7] | dmandelb | | Medium | make it possible to use redef to append to vectors [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update [7] #1024: http://tracker.bro.org/bro/ticket/1024 From bro at tracker.bro.org Fri Jun 28 08:46:39 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Fri, 28 Jun 2013 15:46:39 -0000 Subject: [Bro-Dev] #1027: topic/seth/ssl-remove-log-queue Message-ID: <042.68150ac2b7b87dea8be623c2e7907f6b@tracker.bro.org> #1027: topic/seth/ssl-remove-log-queue ---------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ I removed the log queueing that was included with the log delay mechanism in the SSL script. It could lead to high memory use and there was at least one case where a suspected bug was leading to complete log loss. -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Sat Jun 29 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Sat, 29 Jun 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306290700.r5T703A7004061@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] Bro | 1024 [7] | dmandelb | | Medium | make it possible to use redef to append to vectors Bro | 1027 [8] | seth | | Medium | topic/seth/ssl-remove-log-queue [9] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update [7] #1024: http://tracker.bro.org/bro/ticket/1024 [8] #1027: http://tracker.bro.org/bro/ticket/1027 [9] ssl-remove-log-queue: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/ssl-remove-log-queue From noreply at bro.org Sun Jun 30 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Sun, 30 Jun 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201306300700.r5U702dg020435@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] Bro | 1024 [7] | dmandelb | | Medium | make it possible to use redef to append to vectors Bro | 1027 [8] | seth | | Medium | topic/seth/ssl-remove-log-queue [9] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update [7] #1024: http://tracker.bro.org/bro/ticket/1024 [8] #1027: http://tracker.bro.org/bro/ticket/1027 [9] ssl-remove-log-queue: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/ssl-remove-log-queue