From seth at icir.org Mon Aug 1 03:34:39 2011 From: seth at icir.org (Seth Hall) Date: Mon, 1 Aug 2011 06:34:39 -0400 Subject: [Bro-Dev] 0MQ security considerations In-Reply-To: <4E3621B5.8010208@ohio.edu> References: <20110705045853.GD17340@icir.org> <1104165816.2780.1310062895959.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110707193836.GI66084@icir.org> <4E16155A.9050102@ohio.edu> <20110707214303.GD77684@icir.org> <20110707215133.GF77684@icir.org> <4E163B71.6050804@ohio.edu> <20110708132228.GA84600@icir.org> <1310914521-sup-1714@samurai.icir.org> <20110722042053.GB32586@icir.org> <1311788385-sup-6149@samurai.icir.org> <4E308532.9090205@ohio.edu> <1311840515-sup-4852@samurai.icir.org> <4E31A172.6010600@ohio.edu> <1311878972-sup-3675@samurai.icir.org> <4E32152C.1000407@ohio.edu> <1312066147-sup-5217@samurai.icir.org> <4E34E873.70109@ohio.edu> <05B3721C-8B12-4CA3-83DA-F97C8AD0E0AF@icir.org> <4E3621B5.8010208@ohio.edu> Message-ID: On Jul 31, 2011, at 11:47 PM, Gilbert Clark wrote: > Internally, filters are assigned to streams (e.g. seem to be part of a stream). They *are* discrete components, though, so maybe it's not a good idea to lump the two together like I did? Yeah, it's mostly just semantics. I'm glad to know that there isn't a mix of terminology between the core and scripting layer. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Mon Aug 1 08:47:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 01 Aug 2011 15:47:23 -0000 Subject: [Bro-Dev] #527: --with-perftools broken? In-Reply-To: <046.8cf6b9c3acfe750a6f4cb75681b3b0ed@tracker.bro-ids.org> References: <046.8cf6b9c3acfe750a6f4cb75681b3b0ed@tracker.bro-ids.org> Message-ID: <061.294bed60b817021082fb76add37f1e02@tracker.bro-ids.org> #527: --with-perftools broken? ----------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------+------------------------ Changes (by jsiwek): * status: new => closed * resolution: => fixed Comment: In [c0c8b515f9f705e5cf945ef235f0019f22af68b8/bro]: {{{ #!CommitTicketReference repository="bro" revision="c0c8b515f9f705e5cf945ef235f0019f22af68b8" --with-perftools configure option now assumes --enable-perftools (closes #527) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 1 09:20:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 01 Aug 2011 16:20:40 -0000 Subject: [Bro-Dev] #528: Python 3 compatibility Message-ID: <047.5418f300a382e1c09535f6855df979b9@tracker.bro-ids.org> #528: Python 3 compatibility ------------------------+-------------------- Reporter: robin | Owner: robin Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | ------------------------+-------------------- We should make sure that BroControl (and other Pytjon pieces we ship run fine with Python 3.x). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 1 09:25:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 01 Aug 2011 16:25:14 -0000 Subject: [Bro-Dev] #528: Python 3 compatibility In-Reply-To: <047.5418f300a382e1c09535f6855df979b9@tracker.bro-ids.org> References: <047.5418f300a382e1c09535f6855df979b9@tracker.bro-ids.org> Message-ID: <062.7242da1cd67a8aeb4902087fd63bc596@tracker.bro-ids.org> #528: Python 3 compatibility -------------------------+-------------------- Reporter: robin | Owner: robin Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: -------------------------+-------------------- Comment (by slagell): Are any of the current NMI B&T target platforms using that by default? If so, we may already be testing that. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon Aug 1 09:49:19 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 1 Aug 2011 09:49:19 -0700 Subject: [Bro-Dev] Script reorg proposal Message-ID: <20110801164919.GA2509@icir.org> Seems we are still not quite happy with the layout of the new scripts. The current directory structure can be a bit confusing to follow, and also for users, to find out what to load. Seth and I were kicking around another idea last week. It's somewhat radical compared to "Bro tradition" but I think it makes sense. The proposal is to move more scripts into the the set loaded by default. While currently, Bro essentially does nothing when no further scripts are specified, we would change things so that by default, Bro now loads all the basic scripts that do just logging and state building (but not more extensive/expensive kinds of analysis or detection). So when users just run Bro on a link/trace, they'll immediately get a bunch of high-value log files, without needing to figure out anything else (and some may just want to stop right there in terms of what to learn about Bro, which is fine). In addition, however, as there clearly is value in running Bro with a minimal bare-bones config (for fine-tuned trace analysis, research, or debugging), we'd also provide an option (i.e., redefable script variable and/or environemnt variable), that brings back the old behaviour of loading just bro.init. With that, one could still cherry-pick what to load. With this scheme, we'd organize the script installation like this, assuming --prefix=/usr: /usr/share/bro/base/ - All the scripts that are loaded by default (large parts of the current protocols and frameworks directories). Most users wouldn't need to know much about these scripts (but they'll be documented and can be extended). /usr/share/bro/policy/ - All the scripts that the user can load in addition. I.e., much like the 1.5 policy/ directory, but with less stuff in it. /usr/share/bro/site/ - Local site-policies. Depending on file system standards, this may go somehwere else as well. /usr/share/bro/contrib/ - At some point, we'll add the set of contributed scripts here. Will be externally managed in some way we haven't figured out yet. BROAPTH would include all these four directories. What do you guys think? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Mon Aug 1 09:47:44 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 01 Aug 2011 09:47:44 -0700 Subject: [Bro-Dev] Directly committing to master In-Reply-To: <20110731034933.GA24247@icir.org> References: <4E3469E4.5010304@icir.org> <20110731034933.GA24247@icir.org> Message-ID: <4E36D8B0.6000309@icir.org> On 7/30/11 20:49 , Robin Sommer wrote: > Gregor, what's the original problem you ran into? I actually didn't run into a problem I just noticed the direct commits to master. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From robin at icir.org Mon Aug 1 10:10:36 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 1 Aug 2011 10:10:36 -0700 Subject: [Bro-Dev] Directly committing to master In-Reply-To: <4E36D8B0.6000309@icir.org> References: <4E3469E4.5010304@icir.org> <20110731034933.GA24247@icir.org> <4E36D8B0.6000309@icir.org> Message-ID: <20110801171036.GB2509@icir.org> On Mon, Aug 01, 2011 at 09:47 -0700, you wrote: > I actually didn't run into a problem I just noticed the direct commits > to master. I'm trying to figure out what it is about the recent commits that you didn't like. Is it because it may break things more easily, or because it is harder to track what's going into master, or ... ? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Mon Aug 1 10:49:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 01 Aug 2011 17:49:47 -0000 Subject: [Bro-Dev] #520: trace-summary can't parse new conn.log In-Reply-To: <048.1a6560990d4e6ee8693add3a3cac1151@tracker.bro-ids.org> References: <048.1a6560990d4e6ee8693add3a3cac1151@tracker.bro-ids.org> Message-ID: <063.5baf8125aad036d4949bd2b0cc248ff8@tracker.bro-ids.org> #520: trace-summary can't parse new conn.log ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: trace-summary | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * status: reopened => closed * resolution: => fixed Comment: In [d97e7da9e3675627782169d561611f4599995fdc/trace-summary]: {{{ #!CommitTicketReference repository="trace-summary" revision="d97e7da9e3675627782169d561611f4599995fdc" trace-summary now parses both Bro 1.x and 2.x conn.log formats The default setting is to make an educated guess at the format, but can be explicitly set via the new --conn-version switch. Closes #520 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Mon Aug 1 11:01:34 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 01 Aug 2011 11:01:34 -0700 Subject: [Bro-Dev] Directly committing to master In-Reply-To: <20110801171036.GB2509@icir.org> References: <4E3469E4.5010304@icir.org> <20110731034933.GA24247@icir.org> <4E36D8B0.6000309@icir.org> <20110801171036.GB2509@icir.org> Message-ID: <4E36E9FE.9030106@icir.org> >> I actually didn't run into a problem I just noticed the direct commits >> to master. > > I'm trying to figure out what it is about the recent commits that you > didn't like. Is it because it may break things more easily, or because > it is harder to track what's going into master, or ... ? A combination of all of these I guess (none of these has happened yet but I think we risk them by committing directly to master). It may break things more easily and it lacks the 4-eyes approach. Ultimately, one might get a bit more "sloppy" when committing to master. (*) It's harder to track what goes into master. If we do regular merges we can combine a bunch of commits and but in a nice quick summary in the CHANGES files describing the merge. It may break things for people using master if the happen to get an in-between or checkpoint commit. Similar for developers branching off master. It might make it harder to pinpoint bugs in master reported by users. Maybe the bug was just an (unknown) side effect of a commit that has been fixed by a later one. For us that's going to be harder to track down without knowing exactly what revision the user was running. We've spend a lot of time last year thinking about and working out a development process that "works best" for Bro with a lot of discussion. So we should stick to it unless we have good reasons to not do so. (*) E.g., consider some of the older code that is/was in Bro that maybe shouldn't have been released because it was quite limited or buggy and has never been tested extensively or regularly (e.g., old NFS analyzer, SMB analyzer, ConnStats analyzer) just my 2ct Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From slagell at ncsa.illinois.edu Mon Aug 1 11:14:30 2011 From: slagell at ncsa.illinois.edu (Adam J. Slagell) Date: Mon, 1 Aug 2011 13:14:30 -0500 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <20110801164919.GA2509@icir.org> References: <20110801164919.GA2509@icir.org> Message-ID: <51963CF3-FB4D-4956-A959-231AA5C57BF0@ncsa.illinois.edu> On Aug 1, 2011, at 11:49 AM, Robin Sommer wrote: > > BROAPTH would include all these four directories. > > > What do you guys think? Makes sense and addresses some of the noob complaints. As long as there is a way to run barebones, I think it is fine. ------ Adam J. Slagell, CISSP Sr. Security Engineer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info 217.244.8965 From jsiwek at ncsa.illinois.edu Mon Aug 1 11:23:46 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 1 Aug 2011 13:23:46 -0500 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <20110801164919.GA2509@icir.org> References: <20110801164919.GA2509@icir.org> Message-ID: > With this scheme, we'd organize the script installation like this, > assuming --prefix=/usr: > > /usr/share/bro/base/ > > /usr/share/bro/policy/ > > /usr/share/bro/site/ > > /usr/share/bro/contrib/ > > > BROAPTH would include all these four directories. > If they're all in BROPATH, that means it's possible for scripts/packages in one directory to overshadow equivalently named ones in another directory. e.g. this makes it hard for one to create an "protocols/http" package that exists in all 4 dirs, which seems like a reasonable use to me. Is there a specific path ordering or naming convention that can deal with this? I think I'd probably like for the default BROPATH to just contain the $prefix/share/bro. - Jon From scampbell at lbl.gov Mon Aug 1 11:27:27 2011 From: scampbell at lbl.gov (Scott Campbell) Date: Mon, 01 Aug 2011 13:27:27 -0500 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <20110801164919.GA2509@icir.org> References: <20110801164919.GA2509@icir.org> Message-ID: <4E36F00F.2020705@lbl.gov> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I think that this is an excellent idea since it addresses the new user (!RTFM)/first impression problem. cheers, scott On 8/1/11 11:49 AM, Robin Sommer wrote: > Seems we are still not quite happy with the layout of the new > scripts. The current directory structure can be a bit confusing to > follow, and also for users, to find out what to load. > > Seth and I were kicking around another idea last week. It's somewhat > radical compared to "Bro tradition" but I think it makes sense. > > The proposal is to move more scripts into the the set loaded by > default. While currently, Bro essentially does nothing when no > further scripts are specified, we would change things so that by > default, Bro now loads all the basic scripts that do just logging and > state building (but not more extensive/expensive kinds of analysis > or detection). > > So when users just run Bro on a link/trace, they'll immediately get > a bunch of high-value log files, without needing to figure out > anything else (and some may just want to stop right there in terms of > what to learn about Bro, which is fine). > > In addition, however, as there clearly is value in running Bro with > a minimal bare-bones config (for fine-tuned trace analysis, research, > or debugging), we'd also provide an option (i.e., redefable script > variable and/or environemnt variable), that brings back the old > behaviour of loading just bro.init. With that, one could still > cherry-pick what to load. > > With this scheme, we'd organize the script installation like this, > assuming --prefix=/usr: > > /usr/share/bro/base/ > > - All the scripts that are loaded by default (large parts of the > current protocols and frameworks directories). Most users wouldn't > need to know much about these scripts (but they'll be documented and > can be extended). > > /usr/share/bro/policy/ > > - All the scripts that the user can load in addition. I.e., much like > the 1.5 policy/ directory, but with less stuff in it. > > /usr/share/bro/site/ > > - Local site-policies. Depending on file system standards, this may > go somehwere else as well. > > /usr/share/bro/contrib/ > > - At some point, we'll add the set of contributed scripts here. Will > be externally managed in some way we haven't figured out yet. > > > BROAPTH would include all these four directories. > > > What do you guys think? > > Robin > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFONvAPK2Plq8B7ZBwRAivvAKCnmP+WpKdsq9WUP4zLY3qVX2Di2QCfX5uM cOGBe6/zV0409ZuTNmL+uhI= =EgID -----END PGP SIGNATURE----- From robin at icir.org Mon Aug 1 11:32:38 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 1 Aug 2011 11:32:38 -0700 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: References: <20110801164919.GA2509@icir.org> Message-ID: <20110801183238.GA8583@icir.org> On Mon, Aug 01, 2011 at 13:23 -0500, you wrote: > If they're all in BROPATH, that means it's possible for > scripts/packages in one directory to overshadow equivalently named > ones in another directory. Good catch, and actually this is my mistake when writing the summary, sorry. While I don't have my notes with me right now, I think what Seth and I actually discussed is that BROPATH would include only $prefix/share/bro and $prefix/share/bro/policy. That allows to pull in the optional stuff from policy/ directly without needing the prefix (which is the common case); but all the other stuff does require the base/contrib/site prefixes, and we don't get conflicts with directories overshadowin each other. Does that sound better? Perhaps site/ should be in BROPATH as well though, in particiular if it ends up being moved to somewhere else than share/bro/. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Mon Aug 1 11:48:23 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 01 Aug 2011 11:48:23 -0700 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <20110801164919.GA2509@icir.org> References: <20110801164919.GA2509@icir.org> Message-ID: <4E36F4F7.5090404@icir.org> I like the idea as well. However I'm wondering whether it would make sense to put the stuff that's loaded by default into specific policy file, e.g., default.bro and load that file by default. This would make it easier to selectively remove some parts of the analysis instead of having to go from-all-to-nothing. Then we could also just use a single (base) directory for all the scripts instead of bro/base and bro/policy (which kinda irks me). The default.bro file could actually live in the site directory, so it's readily editable for users. This way users get a nice set of default analyzers and logs and it's easy for slightly advanced users to disable some analyzers. I think that's important since for a bunch of analyzers or framework features I expect it to be a close call whether it should be loaded on default or not..... In any case I think it's important to have a nice documentation briefly describing what each of the policy files or packages does. Basically a quick guide to: "if I want X I have to load a,b,c". Don't know whether this readily comes out of the new documentation framework or not. Even if it does it might make sense to manually structure it thematically. In any case the document should link to the generated documentation of the individual scripts/packages. Might also be nice if these brief descriptions would also make it into the default.bro file as comments. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From vallentin at icir.org Mon Aug 1 12:19:02 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 01 Aug 2011 12:19:02 -0700 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <20110801164919.GA2509@icir.org> References: <20110801164919.GA2509@icir.org> Message-ID: <1312225098-sup-5662@samurai.icir.org> I like this rather simple scheme. > /usr/share/bro/site/ > > - Local site-policies. Depending on file system standards, > this may go somehwere else as well. It always bothered me that the site scripts were in the same place as the distribution policy scripts. This made it more difficult to keep local customixations in git or other VCS. What about ~/.bro/site? (see below) > BROAPTH would include all these four directories. What about a ~/.bro directory in addition to BROPATH? This would allow users to customize/override default script versions and facilite script hacking. For example, say a user wants to replace PREFIX/bro/base/foo.bro with a custom version. This would simply require creating ~/.bro/base/foo.bro. We might raise a (suppressable) warning that the file in the home directory is shadowed in this case. In general, I could imagine that a ~/.bro directory makes it easier for UNIX-folks to get-it-up-and-running by simply creating policy scripts in ~/.bro. Thoughts? Matthias From slagell at ncsa.illinois.edu Mon Aug 1 12:22:49 2011 From: slagell at ncsa.illinois.edu (Adam J. Slagell) Date: Mon, 1 Aug 2011 14:22:49 -0500 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <1312225098-sup-5662@samurai.icir.org> References: <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> Message-ID: <4E76EE33-5BBE-4BDD-8DCE-66877DCD26C7@ncsa.illinois.edu> On Aug 1, 2011, at 2:19 PM, Matthias Vallentin wrote: > I like this rather simple scheme. > >> /usr/share/bro/site/ >> >> - Local site-policies. Depending on file system standards, >> this may go somehwere else as well. > > It always bothered me that the site scripts were in the same place as > the distribution policy scripts. This made it more difficult to keep > local customixations in git or other VCS. What about ~/.bro/site? (see > below) Does this make an implicit assumption that only one user is configuring the Bro policy for a site or system? Or does bro run as root and hence this would almost always be in /root/.bro ? ------ Adam J. Slagell, CISSP Sr. Security Engineer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info 217.244.8965 From vallentin at icir.org Mon Aug 1 12:36:38 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 01 Aug 2011 12:36:38 -0700 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <4E76EE33-5BBE-4BDD-8DCE-66877DCD26C7@ncsa.illinois.edu> References: <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> <4E76EE33-5BBE-4BDD-8DCE-66877DCD26C7@ncsa.illinois.edu> Message-ID: <1312227012-sup-6206@samurai.icir.org> > Does this make an implicit assumption that only one user is > configuring the Bro policy for a site or system? No, I did not mean to imply a single "Bro admin" per system, although this is probably common practice. > Or does bro run as root and hence this would almost always be in > /root/.bro ? On many UNIX flavors [1], Bro will probably need to run as root in order to access the network interfaces. But supporting ~/.bro has also benefits for users who simply want to do trace analysis (i.e., no root privileges required) and customize "their" Bro. Another plus is that rolling Bro updates system-wide or uninstalling Bro is independent of a user's configuration. Matthias [1] Some BSDs support access control via groups, and IIRC Robin wrote a patch for Linux. From jsiwek at ncsa.illinois.edu Mon Aug 1 12:52:51 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 1 Aug 2011 14:52:51 -0500 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <20110801183238.GA8583@icir.org> References: <20110801164919.GA2509@icir.org> <20110801183238.GA8583@icir.org> Message-ID: > While I don't have my notes with me right now, I think what Seth and I > actually discussed is that BROPATH would include only > $prefix/share/bro and $prefix/share/bro/policy. Sounds better. > Perhaps site/ should be in BROPATH as well though, in particiular if > it ends up being moved to somewhere else than share/bro/. Yes, I think it's ok to have site/ in the default BROPATH (now that I realize $prefix/share/bro will also be there). - Jon From jsiwek at ncsa.illinois.edu Mon Aug 1 13:25:08 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 1 Aug 2011 15:25:08 -0500 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <1312225098-sup-5662@samurai.icir.org> References: <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> Message-ID: <4F893448-08DB-4056-82ED-63F1A76B6E6E@ncsa.illinois.edu> > What about a ~/.bro directory in addition to BROPATH? I could see that being a good default place to search for custom scripts. > This would allow > users to customize/override default script versions and facilite script > hacking. For example, say a user wants to replace > PREFIX/bro/base/foo.bro with a custom version. This would simply require > creating ~/.bro/base/foo.bro. I don't think intentional script overshadowing is a use-case that will work as nicely as you want it to if, after the reorganization, most scripts are still as part of a script "package" and referred to internally by a relative path. e.g. the "base/foo/" package might have a "base/foo/bar.bro" script that's loaded internally via its "__load__.bro" that does "@load ./bar". That means you should never be able to overshadow that bar.bro. - Jon From bro at tracker.bro-ids.org Mon Aug 1 14:50:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 01 Aug 2011 21:50:34 -0000 Subject: [Bro-Dev] #500: PktSrc::Statistics generates bogus stats when reading a trace In-Reply-To: <046.d0332ebd9070627807d30aec450eea2e@tracker.bro-ids.org> References: <046.d0332ebd9070627807d30aec450eea2e@tracker.bro-ids.org> Message-ID: <061.68354f3a6f3228a5d29ceb44a1da3dfc@tracker.bro-ids.org> #500: PktSrc::Statistics generates bogus stats when reading a trace ----------------------+--------------------- Reporter: vern | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: Preview ----------------------+--------------------- Changes (by jsiwek): * owner: => jsiwek * status: new => closed * resolution: => fixed Comment: In [d6d34f3a1fc11546a88f457a42e136a22d1e37b0/bro]: {{{ #!CommitTicketReference repository="bro" revision="d6d34f3a1fc11546a88f457a42e136a22d1e37b0" Fix PktSrc::Statistics returning bogus stats. When reading from trace files, 'dropped' and 'link' fields are now just zeroed. When reading from an interface, the values filled in by pcap_stats() are now only used when that function indicates success. Closes #500. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 1 15:09:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 01 Aug 2011 22:09:54 -0000 Subject: [Bro-Dev] #509: SMTP rework In-Reply-To: <046.35777c113d149585390ec44632cce689@tracker.bro-ids.org> References: <046.35777c113d149585390ec44632cce689@tracker.bro-ids.org> Message-ID: <061.d12b466f04b6f36a405e9f7f5e34c7fe@tracker.bro-ids.org> #509: SMTP rework ---------------------+------------------------ Reporter: seth | Owner: seth Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ---------------------+------------------------ Changes (by jsiwek): * owner: jsiwek => seth Comment: This is in `topic/jsiwek/smtp-refactor`. Leaving it up to Seth to give me comments on it or merge if it checks out ok. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 1 17:46:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 02 Aug 2011 00:46:43 -0000 Subject: [Bro-Dev] #529: Support for DLT IEEE802_11_RADIO linktype Message-ID: <048.1d684594da18918960f9b44782f56de8@tracker.bro-ids.org> #529: Support for DLT IEEE802_11_RADIO linktype -----------------------------+------------------------ Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Component: Bro | Version: git/master Keywords: | -----------------------------+------------------------ {{{ #!rst Add support for DLT IEEE802_11_RADIO to Bro. It appears this linktype adds a bunch of info from the WLAN radio in front of the actual ethernet header. Unfortunately, it appears to have variable length headers, to adding support to Bro is not trivial. Many (all?) wlan interface can create pcap captures with this DLT. E.g, one can use * ``tcpdump -I ....`` or * ``tcpdump -y IEEE802_11_RADIO`` (depending on OS and tcpdump version used) On my Mac OS ``tcpdump -I`` works. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Tue Aug 2 07:04:15 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 02 Aug 2011 07:04:15 -0700 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <4F893448-08DB-4056-82ED-63F1A76B6E6E@ncsa.illinois.edu> References: <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> <4F893448-08DB-4056-82ED-63F1A76B6E6E@ncsa.illinois.edu> Message-ID: <1312293542-sup-3790@samurai.icir.org> > I don't think intentional script overshadowing is a use-case that will > work as nicely as you want it to if, after the reorganization, most > scripts are still as part of a script "package" and referred to > internally by a relative path. e.g. the "base/foo/" package might > have a "base/foo/bar.bro" script that's loaded internally via its > "__load__.bro" that does "@load ./bar". That means you should never > be able to overshadow that bar.bro. Sorry, I'm not sure if I get it. Are you saying that the "@load ./bar" directive in __load__.bro is problematic, because, if we search in $HOME before the current directory, the wrong bar.bro (in the users home directory) is picked rather than the one in base/foo? Matthias From robin at icir.org Tue Aug 2 07:58:27 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 2 Aug 2011 07:58:27 -0700 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <4E36F4F7.5090404@icir.org> References: <20110801164919.GA2509@icir.org> <4E36F4F7.5090404@icir.org> Message-ID: <20110802145827.GB44696@icir.org> On Mon, Aug 01, 2011 at 11:48 -0700, you wrote: > having to go from-all-to-nothing. Then we could also just use a single > (base) directory for all the scripts instead of bro/base and bro/policy > (which kinda irks me). One of the main motivations for the proposed reorg is actually to split the two apart. Currently, they are all inside one directory hierarchy, but that makes it hard to get a good picture of what's there. With a default.bro that would get even worse becauese now one would need to check that file to see what's already loaded. With the two dirs split, there'll probably be some top-level.bro in base/ pulling in all the other stuff. To remove only some parts one could either copy-and-edit that file, or (better where it works) @unload stuff. But in any case, that's something only "the experts" would be doing anyway. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Tue Aug 2 08:01:21 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 2 Aug 2011 08:01:21 -0700 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <1312225098-sup-5662@samurai.icir.org> References: <4F893448-08DB-4056-82ED-63F1A76B6E6E@ncsa.illinois.edu> <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> <4E76EE33-5BBE-4BDD-8DCE-66877DCD26C7@ncsa.illinois.edu> <1312227012-sup-6206@samurai.icir.org> <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> <4E76EE33-5BBE-4BDD-8DCE-66877DCD26C7@ncsa.illinois.edu> <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> Message-ID: <20110802150121.GC44696@icir.org> A few thoughts regarding where to put site policies: - I think the default for all pre-built packages should be whereever the target system's file system standard wants such stuff. - A user can always pick a different place. With BroCtl, it's single config option; and otherwise one just points BROPATH to that new location. - That said, we still need a default for the source install of course. I'm not sure I like ~/.bro for that, it's not where I'd intuitivelely look for local scripts, in particular when working as root. Does anybody have a good idea where the different OSs/distros want such local scripts files to be located? I'd say let's just pick one of those as the default for the src install as well. - I don't think we should rely (or "approve") the overshadowing. It will work to some degree (but not always, per Jon's mail), but in any case let's not make that the official way of extending Bro. :) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at ncsa.illinois.edu Tue Aug 2 08:09:59 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 2 Aug 2011 10:09:59 -0500 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <1312293542-sup-3790@samurai.icir.org> References: <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> <4F893448-08DB-4056-82ED-63F1A76B6E6E@ncsa.illinois.edu> <1312293542-sup-3790@samurai.icir.org> Message-ID: >> I don't think intentional script overshadowing is a use-case that will >> work as nicely as you want it to if, after the reorganization, most >> scripts are still as part of a script "package" and referred to >> internally by a relative path. e.g. the "base/foo/" package might >> have a "base/foo/bar.bro" script that's loaded internally via its >> "__load__.bro" that does "@load ./bar". That means you should never >> be able to overshadow that bar.bro. > > Sorry, I'm not sure if I get it. Are you saying that the "@load ./bar" > directive in __load__.bro is problematic, because, if we search in $HOME > before the current directory, the wrong bar.bro (in the users home > directory) is picked rather than the one in base/foo? No, it's that relative script loading (e.g. @load ./bar) is currently implemented such that it first always searches the directory-of-the-current-script-being-loaded (e.g. the place where __load__.bro lives; $PWD may differ from that). I guess we could change that, but to me it seems like increasing complexity and decreasing intuitiveness for a situation that will be run into more often by accident rather than intention. Instead we should probably recommend extending/modifying script behavior through the public API that it advertises. I think what you're trying to do with script overshadowing is workaround the lack of OO in the language, and if we wanted to solve that, it would be better to address it directly instead of hacking around it? - Jon From vallentin at icir.org Tue Aug 2 08:11:57 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 02 Aug 2011 08:11:57 -0700 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <20110802150121.GC44696@icir.org> References: <4F893448-08DB-4056-82ED-63F1A76B6E6E@ncsa.illinois.edu> <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> <4E76EE33-5BBE-4BDD-8DCE-66877DCD26C7@ncsa.illinois.edu> <1312227012-sup-6206@samurai.icir.org> <20110802150121.GC44696@icir.org> Message-ID: <1312297474-sup-4558@samurai.icir.org> > - A user can always pick a different place. With BroCtl, it's > single config option; and otherwise one just points BROPATH to > that new location. This makes me wonder whether BROPATH="/path/to/foo:/here/is/bar" are considered like prefixes, i.e., standard sub-directories like site, policy, etc. are also included? > - That said, we still need a default for the source install of > course. I'm not sure I like ~/.bro for that, it's not where I'd > intuitivelely look for local scripts, in particular when working > as root. I agree that *installing* local scripts into ~/.bro is not the best choice. Rather, I was proposing ~/.bro in addition to the base script installation. > Does anybody have a good idea where the different OSs/distros want > such local scripts files to be located? I'd say let's just pick > one of those as the default for the src install as well. /var could be a good choice since the site policies are subject to modification. Perhaps /var/bro/site? > - I don't think we should rely (or "approve") the overshadowing. > It will work to some degree (but not always, per Jon's mail), but > in any case let's not make that the official way of extending Bro. :) ACK ;-). I thought of overshadowing rather a bug than feature that would have to be solved when scanning the script include paths. Matthias From vallentin at icir.org Tue Aug 2 08:21:50 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 02 Aug 2011 08:21:50 -0700 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: References: <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> <4F893448-08DB-4056-82ED-63F1A76B6E6E@ncsa.illinois.edu> <1312293542-sup-3790@samurai.icir.org> Message-ID: <1312298148-sup-3626@samurai.icir.org> > No, it's that relative script loading (e.g. @load ./bar) is currently > implemented such that it first always searches the > directory-of-the-current-script-being-loaded (e.g. the place where > __load__.bro lives; $PWD may differ from that). Got it, thanks for clarifying. > I think what you're trying to do with script overshadowing is > workaround the lack of OO in the language, and if we wanted to solve > that, it would be better to address it directly instead of hacking > around it? Substituting scripts is indeed a hack at best to replace or extend existing code. Because the language itself used to have little customization points (data redefs, callback tables for a few structures), I found myself often replacing entire chunks of the base code by means of substituting implementations. Maybe this strategy is not apt anymore after the script reorganization and the new logging framework?! Matthias From seth at icir.org Tue Aug 2 08:22:29 2011 From: seth at icir.org (Seth Hall) Date: Tue, 2 Aug 2011 11:22:29 -0400 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: References: <20110801164919.GA2509@icir.org> Message-ID: <5874C3B8-4B80-4314-A4F6-F82DC1C711F2@icir.org> On Aug 1, 2011, at 2:23 PM, Jonathan Siwek wrote: > I think I'd probably like for the default BROPATH to just contain the $prefix/share/bro. I'd actually like the path to have bro/, bro/policy/, and bro/site/. The things that people are typically going to be loading will come from those two directories. I'd like people to understand they're loading stuff from contrib when they do so i'd like to leave that out of the path. (This is all with the caveat that I just started reading this thread and this may have been addressed already). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Tue Aug 2 08:29:33 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 2 Aug 2011 08:29:33 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Async DNS lookups may cause memleaks under certain conditions. (6c806b0) In-Reply-To: <201108021508.p72F8lEV018586@bro-ids.icir.org> References: <201108021508.p72F8lEV018586@bro-ids.icir.org> Message-ID: <20110802152933.GF44696@icir.org> On Tue, Aug 02, 2011 at 08:08 -0700, Seth Hall wrote: > Author: Seth Hall > > Async DNS lookups may cause memleaks under certain conditions. I belive I already have perftools output for that, just need to track it down! Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Tue Aug 2 08:46:02 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 02 Aug 2011 08:46:02 -0700 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <20110802145827.GB44696@icir.org> References: <20110801164919.GA2509@icir.org> <4E36F4F7.5090404@icir.org> <20110802145827.GB44696@icir.org> Message-ID: <4E381BBA.4030005@icir.org> On 8/2/11 7:58 , Robin Sommer wrote: >> having to go from-all-to-nothing. Then we could also just use a single >> (base) directory for all the scripts instead of bro/base and bro/policy >> (which kinda irks me). > > One of the main motivations for the proposed reorg is actually to > split the two apart. Currently, they are all inside one directory > hierarchy, but that makes it hard to get a good picture of what's > there. I think that needs to be solved by documentation. Have some (short) doc that lists everything that's @load'able and briefly describes what it does.(1) > With a default.bro that would get even worse becauese now one > would need to check that file to see what's already loaded. One option would be to put everything that's @load'able into the default.bro file and comment everything that we don't load per default. We could also include the descriptions I mentioned above as comments. > With the two dirs split, there'll probably be some top-level.bro in > base/ pulling in all the other stuff. To remove only some parts one > could either copy-and-edit that file, or (better where it works) > @unload stuff. But in any case, that's something only "the experts" > would be doing anyway. So this top-level.bro is basically the same as my default.bro except it lives in base/, right? So what's the advantage? That it's easy to see what's loaded by default based on the directory where the script is? If that's indeed the case and you want to do the split into two directories, I would still advocate to do top-level.bro the way I described my default.bro, i.e., but it in /site and add comments to make it "user serviceable" :-) Another question would be whether one would split protocol analysis between base and policy? E.g., is there going to be "base/http/" and "policy/http" and when I load the first as package I get the basic and when I also load "policy/http" as package I get the heavier analysis or would I also cherry-pick additional features in "policy/http/*"? cu Gregor (1) This list of @load'able things should be sorted thematically which shouldn't be so hard given that we already have the directory structure. We can proably also get the descriptions from the autodoc feature. -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From jsiwek at ncsa.illinois.edu Tue Aug 2 09:38:07 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 2 Aug 2011 11:38:07 -0500 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <20110802150121.GC44696@icir.org> References: <4F893448-08DB-4056-82ED-63F1A76B6E6E@ncsa.illinois.edu> <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> <4E76EE33-5BBE-4BDD-8DCE-66877DCD26C7@ncsa.illinois.edu> <1312227012-sup-6206@samurai.icir.org> <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> <4E76EE33-5BBE-4BDD-8DCE-66877DCD26C7@ncsa.illinois.edu> <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> <20110802150121.GC44696@icir.org> Message-ID: <853923D8-FCDD-4AA9-8F8E-583E3ADF5154@ncsa.illinois.edu> > - That said, we still need a default for the source install of > course. I'm not sure I like ~/.bro for that, it's not where I'd > intuitivelely look for local scripts, in particular when working > as root. Yeah, that's not a good place to install anything by default, but my understanding was it would just be an additional place to search by default. So it doesn't hurt anything, but it's also not extremely useful seeing as a BROPATH can just be customized to include it. I don't have a strong opinion either way. > Does anybody have a good idea where the different OSs/distros want > such local scripts files to be located? I'd say let's just pick > one of those as the default for the src install as well. All the binary packages use an install prefix of "/opt/bro" as per FHS[1]. And except for Mac, variable/run-time related data (BroControl's spool and log dirs) are set to go into "/var/opt/bro" (for Mac it seemed reasonable to think users expect stuff to be installed in a single self-contained place -- the "app" mentality). Currently the site-specific scripts would go in "/opt/bro/share/bro/site". According to the FHS, "No other package files may exist outside the /opt, /var/opt, and /etc/opt", so that limits the options for where local scripts would go. /etc/opt would be for static, host-specific config files, which doesn't seem entirely suitable. /var/opt is also not an exact match because that tends to be more for data that varies *during* operation. My interpretation is that we're not really violating any rule with the current placement of them, although in a similar case[2], people more familiar with packaging would probably recommend something under "/var/opt". The standard recommendation for a manual build/install is to put stuff in /usr/local, but the specific place that's appropriate for local scripts is again ambiguous. Or maybe this issue is just outside the scope of FHS, which says "Local placement of local files is a local issue, so FHS does not attempt to usurp system administrators." Our current approach might be adequate since the admin has an easy way (change default BROPATH) to choose the best place for their local files. [1] http://www.pathname.com/fhs/pub/fhs-2.3.html [2] http://ldn.linuxfoundation.org/forum/lsb-general-forum/topic/fhs-type-question-where-put-modifiable-data-shared-all-users From jsiwek at ncsa.illinois.edu Tue Aug 2 09:45:18 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 2 Aug 2011 11:45:18 -0500 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <1312298148-sup-3626@samurai.icir.org> References: <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> <4F893448-08DB-4056-82ED-63F1A76B6E6E@ncsa.illinois.edu> <1312293542-sup-3790@samurai.icir.org> <1312298148-sup-3626@samurai.icir.org> Message-ID: > structures), I found myself often replacing entire chunks of the base > code by means of substituting implementations. Maybe this strategy is > not apt anymore after the script reorganization and the new logging > framework?! I don't have enough experience with the old scripts to compare, but I find that Seth did a great job at creating lots of points in the new scripts for extensibility. - Jon From jsiwek at ncsa.illinois.edu Tue Aug 2 10:48:55 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 2 Aug 2011 12:48:55 -0500 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <4E381BBA.4030005@icir.org> References: <20110801164919.GA2509@icir.org> <4E36F4F7.5090404@icir.org> <20110802145827.GB44696@icir.org> <4E381BBA.4030005@icir.org> Message-ID: >>> having to go from-all-to-nothing. Then we could also just use a single >>> (base) directory for all the scripts instead of bro/base and bro/policy >>> (which kinda irks me). >> >> One of the main motivations for the proposed reorg is actually to >> split the two apart. Currently, they are all inside one directory >> hierarchy, but that makes it hard to get a good picture of what's >> there. > > I think that needs to be solved by documentation. Have some (short) doc > that lists everything that's @load'able and briefly describes what it > does.(1) My preference would be for the "optionality" (or probably even functionality) of a script/package to be implied by where it lives in a directory hierarchy. The documentation would naturally also reflect this without extra work (probably). The difference is that if everything lives under a common hierarchy, I 'm forced to consult some additional documentation/manifest in order to understand what parts are optional instead of just being able to know when I'm poking around in a shell. > So this top-level.bro is basically the same as my default.bro except it > lives in base/, right? So what's the advantage? That it's easy to see > what's loaded by default based on the directory where the script is? Yeah, that's my impression of what the main advantage is. > If that's indeed the case and you want to do the split into two > directories, I would still advocate to do top-level.bro the way I > described my default.bro, i.e., but it in /site and add comments to make > it "user serviceable" :-) I think this is what the site/local.bro currently does, and I generally like that way of doing it. > Another question would be whether one would split protocol analysis > between base and policy? E.g., is there going to be "base/http/" and > "policy/http" and when I load the first as package I get the basic and > when I also load "policy/http" as package I get the heavier analysis or > would I also cherry-pick additional features in "policy/http/*"? Yes, both packages could exist, base/http for basic analysis and policy/http for advanced. In the later case, my feeling is that cherry-picking is always allowed/encouraged, but if we allow policy/http to be loaded as a whole package, we might need some convention that makes that behavior well-defined or inferable from the naming/hierarchy for the cases where a package contains related-but-conflicting scripts that shouldn't be loaded together and thus require cherry-picking. If that situation occurs, my suggestion would be to use an opt/ subdir of a given package to reflect such add-on scripts that require the user to do a little thinking before loading it. - Jon From vallentin at icir.org Tue Aug 2 12:23:53 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 02 Aug 2011 12:23:53 -0700 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <853923D8-FCDD-4AA9-8F8E-583E3ADF5154@ncsa.illinois.edu> References: <4F893448-08DB-4056-82ED-63F1A76B6E6E@ncsa.illinois.edu> <20110801164919.GA2509@icir.org> <1312225098-sup-5662@samurai.icir.org> <4E76EE33-5BBE-4BDD-8DCE-66877DCD26C7@ncsa.illinois.edu> <1312227012-sup-6206@samurai.icir.org> <20110802150121.GC44696@icir.org> <853923D8-FCDD-4AA9-8F8E-583E3ADF5154@ncsa.illinois.edu> Message-ID: <1312310328-sup-240@samurai.icir.org> (Thanks for the elaborate summary on filesystem paths! I am on the same page.) > Our current approach might be adequate since the admin has an easy way > (change default BROPATH) to choose the best place for their local > files. Yup, the current approach appeals to me as well. Rethinking the introduction of ~/.bro, I am not sure whether it adds more complexity or helps newcomers. When in doubt, it's always good to avoid feature creep. Matthias From gregor at icir.org Tue Aug 2 12:55:34 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 02 Aug 2011 12:55:34 -0700 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: References: <20110801164919.GA2509@icir.org> <4E36F4F7.5090404@icir.org> <20110802145827.GB44696@icir.org> <4E381BBA.4030005@icir.org> Message-ID: <4E385636.7060309@icir.org> >> Another question would be whether one would split protocol analysis >> between base and policy? E.g., is there going to be "base/http/" and >> "policy/http" and when I load the first as package I get the basic and >> when I also load "policy/http" as package I get the heavier analysis or >> would I also cherry-pick additional features in "policy/http/*"? > > Yes, both packages could exist, base/http for basic analysis and policy/http for advanced. In the later case, my feeling is that cherry-picking is always allowed/encouraged, but if we allow policy/http to be loaded as a whole package, we might need some convention that makes that behavior well-defined or inferable from the naming/hierarchy for the cases where a package contains related-but-conflicting scripts that shouldn't be loaded together and thus require cherry-picking. If that situation occurs, my suggestion would be to use an opt/ subdir of a given package to reflect such add-on scripts that require the user to do a little thinking before loading it. Yeah. I think users should have to cherry-pick individual scripts form policy/* and not be able to load them via packages. cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From robin at icir.org Tue Aug 2 13:08:47 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 2 Aug 2011 13:08:47 -0700 Subject: [Bro-Dev] Directly committing to master In-Reply-To: <4E36E9FE.9030106@icir.org> References: <4E3469E4.5010304@icir.org> <20110731034933.GA24247@icir.org> <4E36D8B0.6000309@icir.org> <20110801171036.GB2509@icir.org> <4E36E9FE.9030106@icir.org> Message-ID: <20110802200847.GC50722@icir.org> Thanks for the summary. That all makes sense, but also note that our process always had the "maintainer exception" in there for direct commits. Talking with Seth, we said that at least until the beta gets out, he will keep making direct commits to the scripts. He knows them best by far, and at the moment I'd be merging them pretty much blindly anyway. We'll keep doing the merges for the core, and we will switch back to doing merges for the scripts as well once things have settled down. The most important part is that master remains stable, so we all need to be careful not to break things. I also still haven't given up my hope to make use of the devel branch eventually btw (don't laugh :) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Tue Aug 2 13:09:34 2011 From: seth at icir.org (Seth Hall) Date: Tue, 2 Aug 2011 16:09:34 -0400 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <4E385636.7060309@icir.org> References: <20110801164919.GA2509@icir.org> <4E36F4F7.5090404@icir.org> <20110802145827.GB44696@icir.org> <4E381BBA.4030005@icir.org> <4E385636.7060309@icir.org> Message-ID: On Aug 2, 2011, at 3:55 PM, Gregor Maier wrote: > Yeah. I think users should have to cherry-pick individual scripts form > policy/* and not be able to load them via packages. That's the plan. This is nice, everyone that has commented on this thread really seems to either be in complete agreement or very nearly complete agreement across the board. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From slagell at ncsa.illinois.edu Tue Aug 2 13:17:46 2011 From: slagell at ncsa.illinois.edu (Adam J. Slagell) Date: Tue, 2 Aug 2011 15:17:46 -0500 Subject: [Bro-Dev] Directly committing to master In-Reply-To: <20110802200847.GC50722@icir.org> References: <4E3469E4.5010304@icir.org> <20110731034933.GA24247@icir.org> <4E36D8B0.6000309@icir.org> <20110801171036.GB2509@icir.org> <4E36E9FE.9030106@icir.org> <20110802200847.GC50722@icir.org> Message-ID: <7FFAFE10-651D-4697-82FF-8F8BD382CAF3@ncsa.illinois.edu> On Aug 2, 2011, at 3:08 PM, Robin Sommer wrote: > I also still haven't given up my hope to make use of the devel branch > eventually btw (don't laugh :) I wasn't going to mention the devel branch. :-) ------ Adam J. Slagell, CISSP Sr. Security Engineer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info 217.244.8965 From bro at tracker.bro-ids.org Tue Aug 2 13:33:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 02 Aug 2011 20:33:46 -0000 Subject: [Bro-Dev] #530: Change to hashing code Message-ID: <046.25b83da963246725d453aa2425e3bc37@tracker.bro-ids.org> #530: Change to hashing code ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: beta | ---------------------+-------------------- This is probably fairly inconsequential, but it seemed like an oversight to me. Should the following change be applied? {{{ diff --git a/src/Hash.cc b/src/Hash.cc index ffabe36..1cee679 100644 --- a/src/Hash.cc +++ b/src/Hash.cc @@ -170,7 +170,7 @@ hash_t HashKey::HashBytes(const void* bytes, int size) { const uint8* b = reinterpret_cast(bytes); // H3 doesn't check if size is zero - return ( size == 0 ) ? 0 : (*h3)(bytes, size); + return ( size == 0 ) ? 0 : (*h3)(b, size); } // Fall back to HMAC/MD5 for longer data (which is usually rare). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 2 15:32:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 02 Aug 2011 22:32:31 -0000 Subject: [Bro-Dev] #531: Handle IPv6 protocol chains Message-ID: <048.a1036225e7e49822f38fb91c4b12ccff@tracker.bro-ids.org> #531: Handle IPv6 protocol chains ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Keywords: IPv6 | ---------------------+------------------------ Bro currently expects TCP/UDP/ICMP headers to follow immediately after the first IPv6 header. Bro should support header chains. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 2 17:08:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 03 Aug 2011 00:08:37 -0000 Subject: [Bro-Dev] #532: IP_Hdr class constructor possible problem Message-ID: <048.0fe3b43b12692907fc98a90a54e00944@tracker.bro-ids.org> #532: IP_Hdr class constructor possible problem ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Keywords: IPv6 | ---------------------+------------------------ {{{ #!rst The IP_Hdr() is used as an abstract version for an IP header than handles IPv4 and IPv6 (i.e., struct ip* and struct ip6*). The constructors look like this:: IP_Hdr(struct ip* arg_ip4) IP_Hdr(const struct ip* arg_ip4) (similar for ip6). The class then stores this pointer internally. If the constructor without the const pointer is used then the IP_Hdr class will ``delete`` the pointer in its destructor, otherwise it won't. IMHO this is a very dangerous behavior, since it's very easy to pass the "wrong" pointer to the constructor. We should probably change the constructor to explicitly require a flag indicating whether the pointer should be ``delete``'ed upon destruction. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 2 17:24:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 03 Aug 2011 00:24:13 -0000 Subject: [Bro-Dev] #68: BroV6 support -- memory usage In-Reply-To: <048.a7ea7982ce037ed963ed20f3871939b0@tracker.bro-ids.org> References: <048.a7ea7982ce037ed963ed20f3871939b0@tracker.bro-ids.org> Message-ID: <063.7d05867fbb084e321a8d869d74128d8c@tracker.bro-ids.org> #68: BroV6 support -- memory usage ---------------------+------------------------------------------ Reporter: gregor | Owner: Type: Task | Status: seen Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: 1.5.2 Resolution: | Keywords: BroV6, binpac++, HILTI, IPv6 ---------------------+------------------------------------------ Changes (by gregor): * keywords: BroV6, binpac++, HILTI => BroV6, binpac++, HILTI, IPv6 -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 3 09:06:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 03 Aug 2011 16:06:37 -0000 Subject: [Bro-Dev] #474: &raw_output turns null values into \0 In-Reply-To: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> References: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> Message-ID: <061.da787fbdbf58cf641bbd5b70e8b16aac@tracker.bro-ids.org> #474: &raw_output turns null values into \0 ----------------------+--------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ----------------------+--------------------- Comment (by jsiwek): In [648e1bda267244dc3f785a9718044b71d73f427b/bro]: {{{ #!CommitTicketReference repository="bro" revision="648e1bda267244dc3f785a9718044b71d73f427b" Fix &raw_output and enable_raw_output interpretation of NUL characters When using a `print` statement to write to a file that has raw output enabled, NUL characters in string are no longer interpreted into "\0", no newline is appended afterwards, and each argument to `print` is written to the file without any additional separation. (Re)Assigning to identifiers with the &raw_output attribute should also now correctly apply the attribute to the file value being assigned. Note that the write_file BiF should already be capable of raw string data to a file, expect it bypasses the print_hook event. Addresses #474 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 3 09:09:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 03 Aug 2011 16:09:32 -0000 Subject: [Bro-Dev] #474: &raw_output turns null values into \0 In-Reply-To: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> References: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> Message-ID: <061.3d22f36e2ff8522155d3a1725ad9801e@tracker.bro-ids.org> #474: &raw_output turns null values into \0 ----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------------+------------------------ Changes (by jsiwek): * version: => git/master * type: Problem => Merge Request Comment: In `topic/jsiwek/raw_output` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 3 11:03:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 03 Aug 2011 18:03:15 -0000 Subject: [Bro-Dev] #533: Support STARTTLS in various other protocols Message-ID: <046.eb31182326c73895e8a74c14690577fd@tracker.bro-ids.org> #533: Support STARTTLS in various other protocols -----------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Keywords: | -----------------------------+-------------------- Now that master has a functional and (apparently) stable SSL/TLS analyzer we need to add support for other protocols to pass traffic into it once they upgrade to SSL. Protocols I would like to see support this.. SMTP POP3 XMPP - http://xmpp.org/rfcs/rfc3920.html#tls (once we have an XMPP analyzer) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 3 11:09:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 03 Aug 2011 18:09:50 -0000 Subject: [Bro-Dev] #488: Bug in HTTP parser In-Reply-To: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> References: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> Message-ID: <062.c28645480fc1e25fda1cedc5ac58dca9@tracker.bro-ids.org> #488: Bug in HTTP parser ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): I think this might be done? But I'm confused by the `policy/protocols/http/http-header-crlf.bro` test that currently fails for me. It looks like it's accidentally testing the opposite of what was solved here -- is the `grep -q` exit status supposed to be inverted and also the comment, "not report that weird", might have typo'd "now" ? -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Wed Aug 3 13:15:01 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 03 Aug 2011 13:15:01 -0700 Subject: [Bro-Dev] enable-debug doesn't work on FreeBSD Message-ID: <4E39AC45.3030701@icir.org> Hi, when I run ./configure --enable-debug on FreeBSD it doesn't use the -g flag for gcc: CC: /usr/bin/gcc CFLAGS: -Wall -Wno-unused -DDEBUG CXX: /usr/bin/c++ CXXFLAGS: -Wall -Wno-unused -DDEBUG CPP: /usr/bin/c++ It works fine on Linux and Mac OS X: CC: /usr/bin/gcc CFLAGS: -Wall -Wno-unused -DDEBUG -g CXX: /usr/bin/c++ CXXFLAGS: -Wall -Wno-unused -DDEBUG -g CPP: /usr/bin/c++ Both my Mac OS and the Free BSD have gcc 4.2.1 and cmake 2.8.4. Any ideas what's wrong? (and BTW, just wondering whether -O0 should be added as well (I know it's the default)) cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Wed Aug 3 14:07:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 03 Aug 2011 21:07:20 -0000 Subject: [Bro-Dev] #488: Bug in HTTP parser In-Reply-To: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> References: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> Message-ID: <062.9c78166a70afbc6c66d6415e3407998e@tracker.bro-ids.org> #488: Bug in HTTP parser ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by robin): On Wed, Aug 03, 2011 at 18:09 -0000, you wrote: > I think this might be done? But I'm confused by the > `policy/protocols/http/http-header-crlf.bro` test that currently fails for > me. It looks like it's accidentally testing the opposite of what was > solved here -- is the `grep -q` exit status supposed to be inverted and > also the comment, "not report that weird", might have typo'd "now" ? No, the patch that was committed is just a work-around to stop Bro from crashing, it doesn't fix the problem correctly. And the failing test is a reminder for that. :) Iirc, the problem is that the HTTP session is actually just fine, but the HTTP parser still got confused and reported an assert. Now it reports a weird instead, but it's still not parsing the session right. So what we need to do is track down why it believes there's somethign wrong with the session, and then fix that. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Wed Aug 3 15:30:42 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 3 Aug 2011 17:30:42 -0500 Subject: [Bro-Dev] enable-debug doesn't work on FreeBSD In-Reply-To: <4E39AC45.3030701@icir.org> References: <4E39AC45.3030701@icir.org> Message-ID: > when I run ./configure --enable-debug on FreeBSD it doesn't use the -g > flag for gcc: It works for me on FreeBSD 8.2, gcc 4.2.1, CMake 2.8.4. What's your build/CMakeCache.txt say for these variables: ENABLE_DEBUG CMAKE_CXX_FLAGS_DEBUG CMAKE_C_FLAGS_DEBUG - Jon From gregor at icir.org Wed Aug 3 15:43:55 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 03 Aug 2011 15:43:55 -0700 Subject: [Bro-Dev] enable-debug doesn't work on FreeBSD In-Reply-To: References: <4E39AC45.3030701@icir.org> Message-ID: <4E39CF2B.8060607@icir.org> On 8/3/11 15:30 , Jonathan Siwek wrote: >> when I run ./configure --enable-debug on FreeBSD it doesn't use the -g >> flag for gcc: > > It works for me on FreeBSD 8.2, gcc 4.2.1, CMake 2.8.4. Weird. Mine is FreeBSD 8.2-RELEASE-p2 amd64.... > What's your build/CMakeCache.txt say for these variables: > ENABLE_DEBUG:BOOL=true CMAKE_CXX_FLAGS_DEBUG:STRING= CMAKE_C_FLAGS_DEBUG:STRING= so I guess those two are the problems. Were do they come from? thx Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Wed Aug 3 15:44:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 03 Aug 2011 22:44:14 -0000 Subject: [Bro-Dev] #474: &raw_output turns null values into \0 In-Reply-To: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> References: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> Message-ID: <061.8d7ce12b79fd0dd681e8ff4bd5589cc5@tracker.bro-ids.org> #474: &raw_output turns null values into \0 ----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------------+------------------------ Comment (by robin): > (Re)Assigning to identifiers with the &raw_output attribute should also > now correctly apply the attribute to the file value being assigned. I'm somewhat reluctant to apply this part. Bro is already inconsistent with what happens with attributes on assigment, and I don't want to make it worse. Here, the attribute would now be transfered from the target to the value being assigned, which doesn't sound right (it makes sense in terms of what one wants to achieve, but still?) Can we leave this part out and use the enable_raw_output() function instead? Otherwise, nice patch! -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 3 15:44:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 03 Aug 2011 22:44:27 -0000 Subject: [Bro-Dev] #474: &raw_output turns null values into \0 In-Reply-To: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> References: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> Message-ID: <061.bea330978ddba860bccb893d57a6af91@tracker.bro-ids.org> #474: &raw_output turns null values into \0 ----------------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------------+------------------------ Changes (by robin): * owner: => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 3 15:45:41 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 03 Aug 2011 22:45:41 -0000 Subject: [Bro-Dev] #474: &raw_output turns null values into \0 In-Reply-To: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> References: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> Message-ID: <061.7be6a6b11b89f0e3c65d5e39694d4435@tracker.bro-ids.org> #474: &raw_output turns null values into \0 ----------------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------------+------------------------ Comment (by robin): | Can we leave this part out and use the enable_raw_output() function instead? That's just the part in NameExpr, correct? I'll commit without that for now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 3 16:08:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 03 Aug 2011 23:08:13 -0000 Subject: [Bro-Dev] #511: Misc distribution cleanup In-Reply-To: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> References: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> Message-ID: <062.2ec0fb076b664868938eb45e0c2a7967@tracker.bro-ids.org> #511: Misc distribution cleanup ---------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ---------------------+--------------------- Description changed by robin: Old description: > There are some things that we need to clean up across all modules: > > - Many files still have Subversion's $Id$ in them. Delete. > - Source files aren't consistent in how they mention their license. I'm > thinking we should just a single line statement that refers to the > COPYING file. > - Update and unifty READMEs New description: There are some things that we need to clean up across all modules: - Many files still have Subversion's $Id$ in them. Delete. - Source files aren't consistent in how they mention their license. I'm thinking we should just a single line statement that refers to the COPYING file. - Update and unify READMEs - Some READMEs still give old download locations. -- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 3 17:05:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 04 Aug 2011 00:05:37 -0000 Subject: [Bro-Dev] #527: --with-perftools broken? In-Reply-To: <046.8cf6b9c3acfe750a6f4cb75681b3b0ed@tracker.bro-ids.org> References: <046.8cf6b9c3acfe750a6f4cb75681b3b0ed@tracker.bro-ids.org> Message-ID: <061.62341cfd9fb3262d3cfabe5a239f9cc9@tracker.bro-ids.org> #527: --with-perftools broken? ----------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------+------------------------ Comment (by robin): In [652ec272d6307ec16c6efc99beb0ead05545ff49/bro]: {{{ #!CommitTicketReference repository="bro" revision="652ec272d6307ec16c6efc99beb0ead05545ff49" Merge remote-tracking branch 'origin/fastpath' * origin/fastpath: Fix genDocSourcesList script lack of explicit sorting Fix PktSrc::Statistics returning bogus stats. --with-perftools configure option now assumes --enable-perftools (closes #527) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 3 17:06:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 04 Aug 2011 00:06:05 -0000 Subject: [Bro-Dev] #520: trace-summary can't parse new conn.log In-Reply-To: <048.1a6560990d4e6ee8693add3a3cac1151@tracker.bro-ids.org> References: <048.1a6560990d4e6ee8693add3a3cac1151@tracker.bro-ids.org> Message-ID: <063.b4f4a5597fe08b183aba5b9443eb9f04@tracker.bro-ids.org> #520: trace-summary can't parse new conn.log ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: trace-summary | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Comment (by robin): In [cb6d946c901f0195775546f77eb07f7148280690/trace-summary]: {{{ #!CommitTicketReference repository="trace-summary" revision="cb6d946c901f0195775546f77eb07f7148280690" Merge remote-tracking branch 'origin/fastpath' * origin/fastpath: trace-summary now parses both Bro 1.x and 2.x conn.log formats Teach trace-summary to parse Bro's new conn.log format (closes #520) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 3 17:25:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 04 Aug 2011 00:25:28 -0000 Subject: [Bro-Dev] #534: Leak in DNS resolver Message-ID: <047.a6cec0e639c569fb09138a1ce7c3bfb4@tracker.bro-ids.org> #534: Leak in DNS resolver ------------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: Bro1.6 | Component: Bro Version: git/master | Keywords: BETA ------------------------+--------------------- I saw the following reported by perftools with the small trace in the public testsuite: {{{ Welcome to pprof! For help, type 'help'. (pprof) top Total: 14070 objects 4512 32.1% 32.1% 4512 32.1% BroObj::SetLocationInfo Obj.cc:173 4500 32.0% 64.1% 8975 63.8% ListExpr::ListExpr Expr.cc:4839 4475 31.8% 95.9% 4475 31.8% safe_malloc util.h:281 166 1.2% 97.0% 330 2.3% StringVal::StringVal Val.cc:1372 164 1.2% 98.2% 164 1.2% BroString::Set BroString.cc:147 97 0.7% 98.9% 291 2.1% LookupHostCallback::Timeout bro.bif:2890 69 0.5% 99.4% 205 1.5% LookupHostCallback::Resolved bro.bif:2876 45 0.3% 99.7% 45 0.3% DNS_Mgr::IssueAsyncRequests DNS_Mgr.cc:955 42 0.3% 100.0% 42 0.3% DNS_Mgr::AddResult DNS_Mgr.cc:695 0 0.0% 100.0% 414 2.9% EventHandler::Call EventHandler.cc:73 0 0.0% 100.0% 449 3.2% Trigger::EvaluatePending Trigger.cc:281 }}} Seth has been running into DNS resolver leaks as well, problem the same one. Haven't been able to track this down yet, and unfortunately perftools doesn't report the leaks consistnetly. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 3 19:47:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 04 Aug 2011 02:47:43 -0000 Subject: [Bro-Dev] #474: &raw_output turns null values into \0 In-Reply-To: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> References: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> Message-ID: <061.88c8120dcbab3e329a33ecf11f5138c1@tracker.bro-ids.org> #474: &raw_output turns null values into \0 ----------------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------------+------------------------ Comment (by jsiwek): Replying to [comment:4 robin]: > > (Re)Assigning to identifiers with the &raw_output attribute should also > > now correctly apply the attribute to the file value being assigned. > > Can we leave this part out and use the enable_raw_output() function instead? Yeah. Besides &raw_output, there's &encrypt, &disable_print_hook, &rotate_interval, and &rotate_size that `BroFile` is supposed to know about, but in most cases won't work because they don't get transferred from the identifier to assigned values. Would it be better to remove them all and just provide equivalent BIFs for ones that don't already have them? > That's just the part in NameExpr, correct? I had the test for &raw_output in language/raw_output_attr.test (the test for the BIF was in the bif/ dir) -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Wed Aug 3 20:06:28 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 3 Aug 2011 22:06:28 -0500 (CDT) Subject: [Bro-Dev] enable-debug doesn't work on FreeBSD In-Reply-To: <4E39CF2B.8060607@icir.org> Message-ID: <2054937021.5690.1312427188812.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > CMAKE_CXX_FLAGS_DEBUG:STRING= > CMAKE_C_FLAGS_DEBUG:STRING= > > so I guess those two are the problems. Were do they come from? It looks like they might get set from $cmake_install_prefix/share/cmake/Modules/Compiler/GNU.cmake Maybe you're missing some modules that are supposed to come with CMake? Or maybe it's just not detecting the compiler correctly. (If you're desperate, a temporary workaround would be: `CFLAGS=-g CXXFLAGS=-g ./configure --enable-debug`) - Jon From gregor at icir.org Wed Aug 3 21:15:23 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 03 Aug 2011 21:15:23 -0700 Subject: [Bro-Dev] enable-debug doesn't work on FreeBSD In-Reply-To: <2054937021.5690.1312427188812.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <2054937021.5690.1312427188812.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <4E3A1CDB.6040506@icir.org> On 8/3/11 20:06 , Jonathan Siwek wrote: >> CMAKE_CXX_FLAGS_DEBUG:STRING= >> CMAKE_C_FLAGS_DEBUG:STRING= >> >> so I guess those two are the problems. Were do they come from? > > It looks like they might get set from $cmake_install_prefix/share/cmake/Modules/Compiler/GNU.cmake > > Maybe you're missing some modules that are supposed to come with CMake? Or maybe it's just not detecting the compiler correctly. Ok. Now I found it. Looks like the port maintainer in FreeBSD didn't like the fact that "-g" is in there and just removed it: http://www.freebsd.org/cgi/cvsweb.cgi/ports/devel/cmake/files/patch-Modules_Compiler_GNU.cmake I guess that's to for making port builds work better but it obviously breaks other source builds. Somebody more involved / knowledgeable about ports might want to flag that with the port maintainer... This change is from Apr 2011 so eventually it will be a problem we'll be facing on all FreeBSD systems. So we might have to work around it. Grrrr. > (If you're desperate, a temporary workaround would be: `CFLAGS=-g CXXFLAGS=-g ./configure --enable-debug`) yeah. That's what I ended up doing. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From jsiwek at ncsa.illinois.edu Thu Aug 4 08:44:58 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 4 Aug 2011 10:44:58 -0500 Subject: [Bro-Dev] enable-debug doesn't work on FreeBSD In-Reply-To: <4E3A1CDB.6040506@icir.org> References: <2054937021.5690.1312427188812.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <4E3A1CDB.6040506@icir.org> Message-ID: > Ok. Now I found it. Looks like the port maintainer in FreeBSD didn't like the fact that "-g" is in there and just removed it: > > http://www.freebsd.org/cgi/cvsweb.cgi/ports/devel/cmake/files/patch-Modules_Compiler_GNU.cmake > > I guess that's to for making port builds work better but it obviously breaks other source builds. Somebody more involved / knowledgeable about ports might want to flag that with the port maintainer... > > This change is from Apr 2011 so eventually it will be a problem we'll be facing on all FreeBSD systems. So we might have to work around it. Grrrr. I'd like to keep Bro's use of use of CMAKE_BUILD_TYPE profiles to automatically determine the right flags for the job based on the compiler being used. Adding a hack, exclusive to FreeBSD, to set default compiler flags isn't great since this isn't a general platform problem, just a problem with the port of CMake. I think I understand why they did it -- it makes it easier for port maintainers to arrest full control of compile flags without patching upstream sources of projects that use build profiles. But that approach doesn't seem to play nice when users are trying to build software outside the port system. Taking the issue up with the CMake port maintainer might be best for now. - Jon From bro at tracker.bro-ids.org Thu Aug 4 08:45:02 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 04 Aug 2011 15:45:02 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. Message-ID: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> #535: Get rid of the "net" type. --------------------+-------------------- Reporter: seth | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Keywords: | --------------------+-------------------- This type isn't used in any of the new scripts (and I don't think it was used in any of the old scripts either) and isn't really recommended for use. If we decide to go ahead and remove it, references will need to be removed from the scanner and parser, some BiFs, broccoli, and the broccoli bindings. Anywhere else? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 4 07:38:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 04 Aug 2011 14:38:09 -0000 Subject: [Bro-Dev] #474: &raw_output turns null values into \0 In-Reply-To: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> References: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> Message-ID: <061.d7c8593a480e772710c27af8b055a61c@tracker.bro-ids.org> #474: &raw_output turns null values into \0 ----------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------+------------------------ Changes (by jsiwek): * type: Merge Request => Problem -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 4 08:17:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 04 Aug 2011 15:17:21 -0000 Subject: [Bro-Dev] #515: NMI: Redhat type conversion errors in cxx wrapper made by swig In-Reply-To: <050.cc968c6768b0f08bb69c74b43963844c@tracker.bro-ids.org> References: <050.cc968c6768b0f08bb69c74b43963844c@tracker.bro-ids.org> Message-ID: <065.d70723019438a008508b6bd249e3cbc4@tracker.bro-ids.org> #515: NMI: Redhat type conversion errors in cxx wrapper made by swig -----------------------------+----------------------------------- Reporter: appleman | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: pysubnettree | Version: git/master Resolution: Solved/Applied | Keywords: nmi swig pysubnettree -----------------------------+----------------------------------- Changes (by appleman): * status: new => closed * resolution: => Solved/Applied Comment: NMI build for x86_64_rhap_5.3-updated now uses python-2.3.5 for compatibility between Python & Swig -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 4 12:53:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 04 Aug 2011 19:53:05 -0000 Subject: [Bro-Dev] #511: Misc distribution cleanup In-Reply-To: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> References: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> Message-ID: <062.9e543901174b806f9d98801d5482ebbe@tracker.bro-ids.org> #511: Misc distribution cleanup ---------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ---------------------+--------------------- Comment (by jsiwek): I'm going to start doing this in `topic/dist-cleanup` branches across the repos. A couple more things I'll do: * Remove the old policy script directory, "policy.old" * Remove bro-aux scripts per #472 -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Aug 4 15:07:34 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 04 Aug 2011 15:07:34 -0700 Subject: [Bro-Dev] Generate bro documentation Message-ID: <4E3B1826.6080207@icir.org> Hi, how can I generate the autodocs from Bro. I guess running bro with -Z and loading all.bro, right? This gives me the rst files. How can I then run sphinx to get the HTLM output? Would be great if you could also add that info to: http://www.bro-ids.org/development/autodoc.html And/or provide a shell-script that will do all that. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Thu Aug 4 15:09:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 04 Aug 2011 22:09:09 -0000 Subject: [Bro-Dev] #472: Cleanup scripts in bro-aux In-Reply-To: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> References: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> Message-ID: <062.88d3a902dcd6c954a1bbd95e0f71acb0@tracker.bro-ids.org> #472: Cleanup scripts in bro-aux ---------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ---------------------+--------------------- Comment (by jsiwek): In [156a4b19faab1e60b872cb8c5ef91988bdccce11/bro-aux]: {{{ #!CommitTicketReference repository="bro-aux" revision="156a4b19faab1e60b872cb8c5ef91988bdccce11" Remove all bro-aux scripts, addresses #511, #472 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 4 15:09:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 04 Aug 2011 22:09:09 -0000 Subject: [Bro-Dev] #511: Misc distribution cleanup In-Reply-To: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> References: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> Message-ID: <062.d0db1523d23d835c376dedab5d834927@tracker.bro-ids.org> #511: Misc distribution cleanup ---------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ---------------------+--------------------- Comment (by jsiwek): In [156a4b19faab1e60b872cb8c5ef91988bdccce11/bro-aux]: {{{ #!CommitTicketReference repository="bro-aux" revision="156a4b19faab1e60b872cb8c5ef91988bdccce11" Remove all bro-aux scripts, addresses #511, #472 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Thu Aug 4 15:14:47 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 4 Aug 2011 17:14:47 -0500 Subject: [Bro-Dev] Generate bro documentation In-Reply-To: <4E3B1826.6080207@icir.org> References: <4E3B1826.6080207@icir.org> Message-ID: <4C066D50-DCE0-46AD-9F19-A7E11E94428B@ncsa.illinois.edu> > how can I generate the autodocs from Bro. I guess running bro with -Z > and loading all.bro, right? This gives me the rst files. How can I then > run sphinx to get the HTLM output? There are two Makefile targets `restdoc` and `doc`. The former generates just reST and the later renders it to HTML using sphinx. See doc/scripts/README > > Would be great if you could also add that info to: > http://www.bro-ids.org/development/autodoc.html Will add something. - Jon From bro at tracker.bro-ids.org Thu Aug 4 15:17:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 04 Aug 2011 22:17:50 -0000 Subject: [Bro-Dev] #472: Cleanup scripts in bro-aux In-Reply-To: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> References: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> Message-ID: <062.f303e6b4ef3ef514f683c3b15ef72097@tracker.bro-ids.org> #472: Cleanup scripts in bro-aux -----------------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: preview -----------------------------+--------------------- Changes (by jsiwek): * status: new => closed * resolution: => Solved/Applied Comment: This is going to be finished as part of #511 -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Aug 4 16:29:46 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 04 Aug 2011 16:29:46 -0700 Subject: [Bro-Dev] Generate bro documentation In-Reply-To: <4C066D50-DCE0-46AD-9F19-A7E11E94428B@ncsa.illinois.edu> References: <4E3B1826.6080207@icir.org> <4C066D50-DCE0-46AD-9F19-A7E11E94428B@ncsa.illinois.edu> Message-ID: <4E3B2B6A.9080407@icir.org> On 8/4/11 15:14 , Jonathan Siwek wrote: >> how can I generate the autodocs from Bro. I guess running bro with -Z >> and loading all.bro, right? This gives me the rst files. How can I then >> run sphinx to get the HTLM output? > > There are two Makefile targets `restdoc` and `doc`. The former generates just reST and the later renders it to HTML using sphinx. See doc/scripts/README > Cool, thanks. >> Would be great if you could also add that info to: >> http://www.bro-ids.org/development/autodoc.html > > Will add something. Thanks. Just the above sentence is already pretty good I think. cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From gregor at icir.org Thu Aug 4 17:07:11 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 04 Aug 2011 17:07:11 -0700 Subject: [Bro-Dev] Generate bro documentation In-Reply-To: <4C066D50-DCE0-46AD-9F19-A7E11E94428B@ncsa.illinois.edu> References: <4E3B1826.6080207@icir.org> <4C066D50-DCE0-46AD-9F19-A7E11E94428B@ncsa.illinois.edu> Message-ID: <4E3B342F.5030305@icir.org> On 8/4/11 15:14 , Jonathan Siwek wrote: >> how can I generate the autodocs from Bro. I guess running bro with -Z >> and loading all.bro, right? This gives me the rst files. How can I then >> run sphinx to get the HTLM output? > > There are two Makefile targets `restdoc` and `doc`. The former generates just reST and the later renders it to HTML using sphinx. See doc/scripts/README BTW, there are some problems with sphinx-build using Mac OS X with MacPorts. Probably because I'm doing something wrong with MacPorts. When I just install the "py-sphinx" port I get a sphinx-build binary, but the version of sphinx is so ancient, that it's not compatible with the Makefile (doesn't have a -c option). I can also install one of the many "py2.x-sphinx" ports (e.g., 2.6). Then I get a sphinx-build binary that works with the Makefile but the port only installs it as "sphinx-build-2.6". I.e., I have to manually create a symlink. Grrr. So question is: should cmake try to figure out if there's an awkwardly named sphinx-build binary so it works with MacPorts? Or is there a way that I can tell MacPorts to create a symlink for the binary. Thx, Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Thu Aug 4 17:25:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 00:25:56 -0000 Subject: [Bro-Dev] #536: protocols/ssh and geoip problems Message-ID: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> #536: protocols/ssh and geoip problems ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ I got the following problems when running the ssh analyzer protocols/ssh on a FreeBSD. Geo-IP installed via ports and is: {{{ pkg-info | grep GeoIP GeoIP-1.4.7 Find the country that any IP address or hostname originates }}} Here are the error messages (on stderr) {{{ Error Opening file /usr/local/share/GeoIP/GeoIPCity.dat 1312300855.632467 error in /home/gregor/projects/bro- master/policy/protocols/ssh/base.bro, line 123: can't initialize GeoIP City database.. trying Country version (lookup_location(SSH::c$id$orig_h)) Invalid database type GeoIP Country Edition, expected GeoIP City Edition, Rev 1 Invalid database type GeoIP Country Edition, expected GeoIP City Edition, Rev 1 Invalid database type GeoIP Country Edition, expected GeoIP City Edition, Rev 1 [last line repeated many times] }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 4 17:42:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 00:42:05 -0000 Subject: [Bro-Dev] #537: Default policy script overhead due to DNS lookups in extend-email/hostnames.bro Message-ID: <048.931166b0a66e88c700755af557fbbd31@tracker.bro-ids.org> #537: Default policy script overhead due to DNS lookups in extend- email/hostnames.bro ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ {{{ #!rst Bro's new default set of policy-scripts include loading ``extend- email/hostnames.bro`` (via ``bro.init -> packet-filter -> frameworks/notice``) This script does many async DNS lookups for notices. Doing so will significantly degrade performance. According to Robin probably due to when statements. Using a full trace with "normal" traffic runtime increase by a factor of 1.35 and memory usage by 3.4 (!). When using a SYN/FIN/RST only trace it's a lot lot lot worse (runtime even exceeds the runtime of the full trace and so does memory usage!) We should definitely remove the script from the default config. And, given its overhead, put a warning in the comments/docstrings so that users know what they are getting when they load the script. We might also want to consider removing other, potentially high volume DNS lookups from the default scripts (there's one in protocols/ssh but I don't think it's that high volume, but YMMV). -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Thu Aug 4 17:53:53 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 4 Aug 2011 19:53:53 -0500 Subject: [Bro-Dev] Generate bro documentation In-Reply-To: <4E3B342F.5030305@icir.org> References: <4E3B1826.6080207@icir.org> <4C066D50-DCE0-46AD-9F19-A7E11E94428B@ncsa.illinois.edu> <4E3B342F.5030305@icir.org> Message-ID: > So question is: should cmake try to figure out if there's an awkwardly named sphinx-build binary so it works with MacPorts? Or is there a way that I can tell MacPorts to create a symlink for the binary. I don't know about getting MacPorts to do that, but the CMake script could be made more configurable about finding the right "sphinx-build" -- the typical Find* package macros could be used instead of hardcoding it like it is now. - Jon From bro at tracker.bro-ids.org Thu Aug 4 18:00:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 01:00:48 -0000 Subject: [Bro-Dev] #536: protocols/ssh and geoip problems In-Reply-To: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> References: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> Message-ID: <063.1ec7136b213f2ae1240dc7d66795c8bb@tracker.bro-ids.org> #536: protocols/ssh and geoip problems ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): Looks like that's because it's expecting to use a City database, but not finding it? The GeoLiteCity database usually has to be downloaded manually, more info here: http://www.bro-ids.org/documentation/geoip.html -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Aug 4 19:53:21 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 04 Aug 2011 19:53:21 -0700 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <20110801164919.GA2509@icir.org> References: <20110801164919.GA2509@icir.org> Message-ID: <4E3B5B21.7040609@icir.org> Another comment probably related to this: When looking through the automatically generated script documentation I was at first confused because I clicked on say: protocols/conn, when I was actually interested in protocols/conn/base. I guess the reorg should help with that too. Otherwise we might want to consider if we can make this easier for users.... cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Thu Aug 4 19:16:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 02:16:22 -0000 Subject: [Bro-Dev] #538: lookup_location() should handle missing database gracefully. Message-ID: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> #538: lookup_location() should handle missing database gracefully. ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ If the city level GeoIP database isn't found lookup_location() prints error message on stderr. Some of these appear to come from libgeoip itself. Missing databases should be handled more gracefully. {{{ Error Opening file /usr/local/share/GeoIP/GeoIPCity.dat 1312300855.632467 error in /home/gregor/projects/bro- master/policy/protocols/ssh/base.bro, line 123: can't initialize GeoIP City database.. trying Country version (lookup_location(SSH::c$id$orig_h)) }}} The first one is probably from libgeoip. We should try to prevent libgeoip from priting stuff on stderr.... It appears that lookup_location() has a fallback for using the country database if the city database isn't found. In my case opening the country DB worked, but queries against it didn't. Error messages on stderr presumably from libgeoip: {{{ Invalid database type GeoIP Country Edition, expected GeoIP City Edition, Rev 1 Invalid database type GeoIP Country Edition, expected GeoIP City Edition, Rev 1 Invalid database type GeoIP Country Edition, expected GeoIP City Edition, Rev 1 [tons more of those] }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 4 19:26:26 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 02:26:26 -0000 Subject: [Bro-Dev] #536: protocols/ssh and geoip problems In-Reply-To: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> References: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> Message-ID: <063.5cc3ba9b6f1f56df72e822c82480c972@tracker.bro-ids.org> #536: protocols/ssh and geoip problems ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by gregor): {{{ #!rst yeah. I guess the ticket should more accurately have said that lookup_location() should handle this case more gracefully. I've opened #538 for that. However, I'll leave this ticket open for now. Since ssh seems to require GeoIP support but the documentation says GeoIP is optional. Either protocol/ssh should handle missing GeoIP gracefully or GeoIP should be required. (``lookup_location()`` will issue a ``builtin_error()`` if it's called without GeoIP supprt) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 4 19:33:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 02:33:16 -0000 Subject: [Bro-Dev] #536: protocols/ssh and geoip problems In-Reply-To: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> References: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> Message-ID: <063.97bc104d63021ff8d999b84090f854f2@tracker.bro-ids.org> #536: protocols/ssh and geoip problems ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): > Looks like that's because it's expecting to use a City database, but not > finding it? The GeoLiteCity database usually has to be downloaded > manually, more info here: I've been meaning to get to that, it's fine to not have the city database but we need to make sure that we don't try to capture city information if we didn't end up loading the city database. We just need to check the loaded database with the GeoIP_database_edition function. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 4 19:33:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 02:33:16 -0000 Subject: [Bro-Dev] #536: protocols/ssh and geoip problems In-Reply-To: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> References: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> Message-ID: <063.97bc104d63021ff8d999b84090f854f2@tracker.bro-ids.org> #536: protocols/ssh and geoip problems ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): > Looks like that's because it's expecting to use a City database, but not > finding it? The GeoLiteCity database usually has to be downloaded > manually, more info here: I've been meaning to get to that, it's fine to not have the city database but we need to make sure that we don't try to capture city information if we didn't end up loading the city database. We just need to check the loaded database with the GeoIP_database_edition function. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Aug 4 20:42:32 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 04 Aug 2011 20:42:32 -0700 Subject: [Bro-Dev] New http policy scripts Message-ID: <4E3B66A8.6090508@icir.org> Hi, I've looked at the new http policy scripts and have some comments (first: thanks Seth for this re-organization, looks great): It appears that per default protocols/http now does analyzer HTTP-headers and HTTP-payload. Both of them are quite expensive in terms of CPU time. Particularly body analysis. I would opt to *not* include those if just protocols/http is loaded (which will always be loaded by default in the future). HTTP is usually going to be the most expensive analysis (due to traffic volume) anyway, so we should give users and easy way to adjust the load according to their traffic and available hardware. So, I would opt to only do http request and http reply analysis by default and provide users with an easy option to a) load HTTP-header analysis. E.g., protocols/http/headers b) load HTTP-body analysis. E.g., protocols/http/body (or name a) and b) http/medium and http/heavy respectively) (I can see the value of always doing header analysis, so I think I could accept HTTP header analysis by default if others really want this, but I really think body analysis should not be done by default) Also note that there is an http_headers event (note the "s" at the end). This event gives you all the headers with one event call. You will loose the order of events, and you'll only get the headers after the header is done (i.e., there's an empty line). In my experience it's a *lot* faster, if you only use the http_headers event instead of relying on individual http_header() events (*). I would therefore opt to use the http_headers() event whenever possible!! (Particularly for everything that gets loaded by default) cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Thu Aug 4 20:22:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 03:22:11 -0000 Subject: [Bro-Dev] #539: Autodoc: document redefined record fields Message-ID: <048.98188c725f77b0d6fb55caeb04eb3de1@tracker.bro-ids.org> #539: Autodoc: document redefined record fields ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: preview | ---------------------+------------------------ If a record is redef'ed and extened with +=, then this extension will only show up as "Redefined" in the documentation, but the actual fields that have been added will ''not'' show up in the documentation. I think we really want this, in particular, because this will / might be the prime way to document log file formats. Related to #510, IMHO -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 4 20:47:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 03:47:07 -0000 Subject: [Bro-Dev] #536: protocols/ssh and geoip problems In-Reply-To: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> References: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> Message-ID: <063.c22f656ca22616afe6f6da900fa36441@tracker.bro-ids.org> #536: protocols/ssh and geoip problems ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by gregor): > > Looks like that's because it's expecting to use a City database, but not > > finding it? The GeoLiteCity database usually has to be downloaded > > manually, more info here: > > I've been meaning to get to that, it's fine to not have the city database > but we need to make sure that we don't try to capture city information if > we didn't end up loading the city database. We just need to check the > loaded database with the GeoIP_database_edition function. > It would also be great if we could prevent libgeoip from printing stuff directly on stderr. That's annoying. And, calling lookup_location() without geoip support enabled will yield an error. If we use lookup_location() in any of the new default policy scripts, we should suppress this error. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Thu Aug 4 21:16:05 2011 From: seth at icir.org (Seth Hall) Date: Fri, 5 Aug 2011 00:16:05 -0400 Subject: [Bro-Dev] Script reorg proposal In-Reply-To: <4E3B5B21.7040609@icir.org> References: <20110801164919.GA2509@icir.org> <4E3B5B21.7040609@icir.org> Message-ID: <137E6B5A-3BF3-4867-9E59-1DE58DEEFE9B@icir.org> On Aug 4, 2011, at 10:53 PM, Gregor Maier wrote: > because I clicked on say: protocols/conn, when I was actually interested > in protocols/conn/base. I guess the reorg should help with that too. Yep, it should help quite a bit. I think we can probably start collecting feedback on the optimal way to organize the logs for Jon after the reorg is finished (hopefully very soon!). I agree that it's important and needs to be user focused. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Thu Aug 4 21:43:36 2011 From: seth at icir.org (Seth Hall) Date: Fri, 5 Aug 2011 00:43:36 -0400 Subject: [Bro-Dev] New http policy scripts In-Reply-To: <4E3B66A8.6090508@icir.org> References: <4E3B66A8.6090508@icir.org> Message-ID: On Aug 4, 2011, at 11:42 PM, Gregor Maier wrote: > It appears that per default protocols/http now does analyzer > HTTP-headers and HTTP-payload. Both of them are quite expensive in terms > of CPU time. Particularly body analysis. > > I would opt to *not* include those if just protocols/http is loaded I knew that someone would make this argument eventually. :) There are two parts to my argument in favor of including both by default. By removing some of the analysis like that by default, you are basically taking a runtime optimization step as you pointed out since it does certain cause overhead to do everything. The problem is that it would make the usage of Bro more obtuse for users since it would be a singular optimization specifically for a type of traffic that just happens to be prevalent on most networks. I think that in 99% of cases, people want everything anyway (people running Bro on live traffic for operational security purposes at least). That has been my experience. The other side of this is the http_body events. I don't like how I'm doing that either, but it's a stopgap until we have the more general file analyzer that would do everything i'm doing in the base http analysis scripts internally (identifications, hashing, extraction, etc). I do agree that I'm doing some pretty egregious stuff in some of those scripts from an optimization perspective, but I think that optimization attempts in Bro scripts have led to incredibly convoluted scripts and dependency chains. I'm going to press instead for optimizations that allow the scripts to remain well structured. For instance, what about just disabling the http_header or http_data events if you don't want those done? This should already be do-able with the disable_event_group like this: disable_event_group("http-body"); disable_event_group("http-header"); .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Thu Aug 4 21:58:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 04:58:25 -0000 Subject: [Bro-Dev] #536: protocols/ssh and geoip problems In-Reply-To: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> References: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> Message-ID: <063.5a7bdb0921aa30efdaf314a7167df63a@tracker.bro-ids.org> #536: protocols/ssh and geoip problems ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): > However, I'll leave this ticket open for now. Since ssh seems to require > GeoIP support but the documentation says GeoIP is optional. Either > protocol/ssh should handle missing GeoIP gracefully or GeoIP should be > required. My plan is to remove that from the base ssh scripts actually. This reorganization is making functionality lines much clearer and this is definitely a case where the functionality should be moved out of the base protocol support. -- Ticket URL: Bro Tracker Bro Issue Tracker From hlin33 at illinois.edu Thu Aug 4 22:19:12 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Thu, 4 Aug 2011 22:19:12 -0700 Subject: [Bro-Dev] Hui Lin_Confusion in Dynamic Protocol Detection Message-ID: Hi, When I read http://www.bro-ids.org/development/dpd.html about DPD, I always have confusion in its wording. 1. From the "Class Layout" picture, every analyzer is derived from class "Analyzer", but the wording also says that "The root node must always be of type TransportLayerAnalyzer." So which one is the real root in the Bro's code. yzer directly derived by "Analyzer") are located in this analyzer tree structure. 2. In the section "Determining Analyzer Activation", I am also confused about the method to activate the analyzer on all connections. Foo_Analyzer is derived TCP_ApplicationAnalyzer, but why this Foo_Analyzer is added as the child of TCP_Analyzer. tcp->AddChildAnalyzer(new Foo_Analyzer(conn)); So what is the differences between TCP_ApplicationAnalyzer and TCP_Analyzer. Hui Lin -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110804/53513c29/attachment.html From bro at tracker.bro-ids.org Fri Aug 5 07:50:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 14:50:17 -0000 Subject: [Bro-Dev] #511: Misc distribution cleanup In-Reply-To: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> References: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> Message-ID: <062.fccd41a05a9fd2c416704200c559e6a1@tracker.bro-ids.org> #511: Misc distribution cleanup ---------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ---------------------+--------------------- Comment (by jsiwek): I did most of the cleanup mentioned in the ticket in `topic/dist-cleanup`; here's the remainder for someone to pickup: 1) I did a pass over READMEs to make sure all info was relevant/updated, but I didn't know how to define how they should be "unified" 2) I didn't touch the `broctl` README; I thought there's probably enough unmerged stuff that I should wait to see what the final master looks like, there's #503 already to make sure we don't forget to update it 3) I didn't change or add any license/copyright/COPYING stuff -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 5 07:53:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 14:53:12 -0000 Subject: [Bro-Dev] #540: Method of initializing communication subsystem Message-ID: <046.71e2fbd6fd1c296226d9add95702e199@tracker.bro-ids.org> #540: Method of initializing communication subsystem ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: beta | ---------------------+-------------------- The way the communications system is enabled internally prevents us from making the script-level communications framework part of the base script load. From Expr.cc: {{{ // ### This is a hack. We check whether one of the remote serializer's // built-in functions is referenced. If so, we activate the serializer. // A better solution would be to either (1) a generic mechanism in // which have (internal) attributes associated with identifiers and // as we see references to the identifiers, we do bookkeeping // associated with their attribute (so in this case the attribute // would be "flag that inter-Bro communication is being used"), // or (2) after the parse is done, we'd query whether these // particular identifiers were seen, rather than doing the test // here for every NameExpr we create. }}} As soon as the parser sees one of the communication BiFs in a script it initializes the internal remote serializer (which still leads to performance issues on lightly loaded links). Would it easy/possible to delay initializing the remote serializer until a more deliberate action has been taken at the scripting layer? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 5 07:57:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 14:57:17 -0000 Subject: [Bro-Dev] #540: Method of initializing communication subsystem In-Reply-To: <046.71e2fbd6fd1c296226d9add95702e199@tracker.bro-ids.org> References: <046.71e2fbd6fd1c296226d9add95702e199@tracker.bro-ids.org> Message-ID: <061.4feb90c99320bb7b9aa5c0c07ad2d4c1@tracker.bro-ids.org> #540: Method of initializing communication subsystem ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by robin): > performance issues on lightly loaded links). Would it easy/possible to > delay initializing the remote serializer until a more deliberate action > has been taken at the scripting layer? Would the "more deliberate action" be an explicit bif call like ``enable_communication``? If so, when would you call it? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 5 08:10:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 15:10:20 -0000 Subject: [Bro-Dev] #540: Method of initializing communication subsystem In-Reply-To: <046.71e2fbd6fd1c296226d9add95702e199@tracker.bro-ids.org> References: <046.71e2fbd6fd1c296226d9add95702e199@tracker.bro-ids.org> Message-ID: <061.3d678d8d3a8c49919422f77ceab31d69@tracker.bro-ids.org> #540: Method of initializing communication subsystem ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by seth): > Would the "more deliberate action" be an explicit bif call like > ``enable_communication``? Yes. > If so, when would you call it? Probably at the end of communication/main.bro in this section: {{{ event bro_init() &priority = -10 # let others modify nodes { for ( tag in nodes ) { if ( ! nodes[tag]$connect ) next; connect_peer(tag); } } }}} It would get called if there is anything in the nodes table. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 5 08:21:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 15:21:01 -0000 Subject: [Bro-Dev] #539: Autodoc: document redefined record fields In-Reply-To: <048.98188c725f77b0d6fb55caeb04eb3de1@tracker.bro-ids.org> References: <048.98188c725f77b0d6fb55caeb04eb3de1@tracker.bro-ids.org> Message-ID: <063.a8743c8e43ecd73e2407d6d47423de66@tracker.bro-ids.org> #539: Autodoc: document redefined record fields ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------+------------------------ Comment (by jsiwek): Replying to [ticket:539 gregor]: > If a record is redef'ed and extened with +=, then this extension will only show up as "Redefined" in the documentation, but the actual fields that have been added will ''not'' show up in the documentation. I might be misunderstanding, but I think they are already there. Maybe you are looking at the entry in the "Overview/Summary" section? There's another "Redefinitions" section in the "Public Interface" that shows what fields were added via the `redef`, with xref links that can be followed to find more docs about the type of each field. Let me know if you're talking about something else, or if you think of something to make it more intuitive. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 5 08:36:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 15:36:01 -0000 Subject: [Bro-Dev] #517: NMI: MacOS unit test failures because of missing 'setsid' In-Reply-To: <050.3adeffdd175bfbc733a2c922a71c552a@tracker.bro-ids.org> References: <050.3adeffdd175bfbc733a2c922a71c552a@tracker.bro-ids.org> Message-ID: <065.48a3bc52c1380abc551beef53083167a@tracker.bro-ids.org> #517: NMI: MacOS unit test failures because of missing 'setsid' -----------------------------+------------------------------------- Reporter: appleman | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: nmi btest testing macos -----------------------------+------------------------------------- Changes (by appleman): * status: new => closed * resolution: => Solved/Applied Comment: Fixed -- the submodule update suggested by Jon was sufficient to get this to run to completion. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 5 08:36:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 15:36:40 -0000 Subject: [Bro-Dev] #541: Proposal for default filter $path names Message-ID: <046.4f90fad6a8b6c0a92033571067d81ead@tracker.bro-ids.org> #541: Proposal for default filter $path names ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: beta | ---------------------+-------------------- The way that default $paths are named for logging filters is a bit of a mess right now. My plan is to rename most of the Log::IDs like this: SSH::LOG, SSL::LOG, Syslog::LOG. The problem (as I see it) with the way things are done now is that if you are outside of the module namespace and you want to install a filter you need to use the Log::IDs full name which currently looks like this: SSH::SSH, SSL::SSL, Syslog::SYSLOG which is very non-obvious. When the logging framework comes up with the $path value for the default filter it names it by the ID name without the module and this works great with the current but confusing naming scheme. I want to propose that the logging framework use the module name for the default path if it's a ::LOG value, and otherwise use the value itself without the module. One case in particular where I'm thinking this would already work is in the notice framework. The notice framework will have Notice::LOG and Notice::NOTICE_POLICY which should result in default $paths of "notice" and "notice_policy" respectively. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 5 10:20:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 17:20:28 -0000 Subject: [Bro-Dev] #539: Autodoc: document redefined record fields In-Reply-To: <048.98188c725f77b0d6fb55caeb04eb3de1@tracker.bro-ids.org> References: <048.98188c725f77b0d6fb55caeb04eb3de1@tracker.bro-ids.org> Message-ID: <063.496f0687f9fef1c5efaaa0e923641a54@tracker.bro-ids.org> #539: Autodoc: document redefined record fields ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Invalid | Keywords: preview ----------------------+------------------------ Changes (by gregor): * status: new => closed * resolution: => Invalid Comment: Indeed that's what I was looking for. I just didn't see it. Sorry for that. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 5 11:12:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 18:12:46 -0000 Subject: [Bro-Dev] #541: Proposal for default filter $path names In-Reply-To: <046.4f90fad6a8b6c0a92033571067d81ead@tracker.bro-ids.org> References: <046.4f90fad6a8b6c0a92033571067d81ead@tracker.bro-ids.org> Message-ID: <061.cadb84e001d63e5a8389e73484dd8573@tracker.bro-ids.org> #541: Proposal for default filter $path names ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by seth): I forgot to mention that this proposal sucks and alternates or improvements would be gladly accepted. :) -- Ticket URL: Bro Tracker Bro Issue Tracker From gc355804 at ohio.edu Fri Aug 5 11:30:17 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Fri, 05 Aug 2011 11:30:17 -0700 Subject: [Bro-Dev] trace-summary question Message-ID: <4E3C36B9.5000201@ohio.edu> From trace-summary: if duration and payload_resp * 8 / (1024 * 1024 * duration) > 700: # Bandwith exceed due to Bro bug. if Options.conn_version == 1: print >>sys.stderr, "%.6f originator exceeds bandwith" % time else: print >>sys.stderr, "UID %s originator exceeds bandwith" % f[uid_idx] payload_resp = 0 Just curious: is there a good reason for '700'? --Gilbert From robin at icir.org Fri Aug 5 11:51:16 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 5 Aug 2011 11:51:16 -0700 Subject: [Bro-Dev] enable-debug doesn't work on FreeBSD In-Reply-To: References: <2054937021.5690.1312427188812.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <4E3A1CDB.6040506@icir.org> Message-ID: <20110805185116.GA97850@icir.org> On Thu, Aug 04, 2011 at 10:44 -0500, you wrote: > I think I understand why they did it -- it makes it easier for port > maintainers to arrest full control of compile flags without patching > upstream sources of projects that use build profiles. FreeBSD can be a pain in this regard. For the autotools, they used to have their own version for the ports, which however didn't work well for external software. As a work-around they offered an additional port gnu-autotools that would install an unmodified version into /usr/local/gnu/bin/ ... Sigh. This seems to be along the same lines: optimizing for the ports, but not considering external software using the same tools. That said, I think it would be good for us to work around this. While it's easy to fix once one knows what's going on, it's of these little things that will be annoying every single time; and everybody on recent FreeBSD will have that problem. Can we just generally add a "-g" to debug builds if it's not there yet (or even if it may be, if we can't check easily?). Seem there's not much harm to that? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Fri Aug 5 15:10:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 05 Aug 2011 22:10:34 -0000 Subject: [Bro-Dev] #378: Test for problem with optional record fields. In-Reply-To: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> References: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> Message-ID: <062.c60416f0850455cc91ed657a30f53242@tracker.bro-ids.org> #378: Test for problem with optional record fields. --------------------------------+-------------------- Reporter: robin | Owner: Type: Test Case Missing | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: --------------------------------+-------------------- Comment (by jsiwek): This test case was still failing so I looked into it a bit. This is the essential part of the bro script that triggers it: {{{ type FOO: record { a: count; b: count &optional; }; local set_of_foo: set[FOO] = set(); add set_of_foo[[$a=4, $b=5]]; print set_of_foo; }}} What happens is when the value is assigned to the set in `TableVal::Assign`, part of the `HashKey` computed from that `add`ed `RecordVal` index value takes into consideration whether the fields in the value's associated `RecordType` have the `&optional` attribute. In this case, the hash is computed thinking there's no optional fields (due to the implicit/coerced typing?). Now when using that `HashKey` to try to recover an item during the `print` statement, it thinks the size of the key is wacky because it's suddenly taking into account the `&optional` attribute of the `FOO` `RecordType` used as the set's index type. I can get everything to work if the computation and recovery both agree to use the set's index type like this: {{{ diff --git a/src/CompHash.cc b/src/CompHash.cc index 605949b..9c06656 100644 --- a/src/CompHash.cc +++ b/src/CompHash.cc @@ -169,7 +169,7 @@ char* CompositeHash::SingleValHash(int type_check, char* kp0, { char* kp = kp0; RecordVal* rv = v->AsRecordVal(); - RecordType* rt = v->Type()->AsRecordType(); + RecordType* rt = bt->AsRecordType(); int num_fields = rt->NumFields(); for ( int i = 0; i < num_fields; ++i ) }}} But I'm wondering if before the `ComputeHash` part happens, the `add`ed index value's type should have already been coerced into the actual record type that's being used for the set, including the attributes of all the fields? Because right now it looks like the type of the added value includes all the right fields, just not the attributes. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Fri Aug 5 15:40:29 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Fri, 5 Aug 2011 17:40:29 -0500 Subject: [Bro-Dev] enable-debug doesn't work on FreeBSD In-Reply-To: <20110805185116.GA97850@icir.org> References: <2054937021.5690.1312427188812.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <4E3A1CDB.6040506@icir.org> <20110805185116.GA97850@icir.org> Message-ID: <21DA6FD8-1048-43DA-950C-B5D0975C1187@ncsa.illinois.edu> > Can we just generally add a "-g" to debug builds if it's not there yet > (or even if it may be, if we can't check easily?). Seem there's not > much harm to that? Ok. I'm imagining that a Bro port might have to workaround my workaround. But I guess that would be fine since we'd be back where we started before the CMake port did their workaround. - Jon From bro at tracker.bro-ids.org Sat Aug 6 18:39:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 07 Aug 2011 01:39:31 -0000 Subject: [Bro-Dev] #542: SHA-1 and SHA-256 Message-ID: <046.6edfaf522dfff4dedd615707b9c858f1@tracker.bro-ids.org> #542: SHA-1 and SHA-256 -----------------------------+----------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Component: Bro | Version: Keywords: language | -----------------------------+----------------- It would be nice to have the primitives in Bro to calculate SHA-1 and SHA-256 hashes in addition to MD5. Additionally it would be nice to have these extended into script-land similarly to the MD5 calculation BiFs. No timeline on this, I just wanted to make sure I filed it away somewhere. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Sat Aug 6 21:58:51 2011 From: gregor at icir.org (Gregor Maier) Date: Sat, 06 Aug 2011 21:58:51 -0700 Subject: [Bro-Dev] Command line VS. environmnet variable (was: Re: [Bro-Commits] [git/bro] master: Hopefully the last major script reorganization. (597a4d6)) In-Reply-To: <201108060447.p764l9QV022156@bro-ids.icir.org> References: <201108060447.p764l9QV022156@bro-ids.icir.org> Message-ID: <4E3E1B8B.9090708@icir.org> > - All of scripts/base/ is loaded by main.cc > - Can be disabled by setting $BRO_NO_BASE_SCRIPTS > - Scripts in scripts/base/ don't use relative path loading to ease use of BRO_NO_BASE_SCRIPTS (to copy and paste that script). I really, really, *really* think this should be a command line option! After all, that's what command line options are there for: to enable/disable/modify features of the executed process. Environment variables are a pain when used to set options. They are nice/good to set paths, etc. It's easy to think an env. variable is set (or not set) when in fact it's the other way round. Similarly, it's hard if a user has an issue: It's now not enough to ask for a the command line, one also needs the environment. If one really wants to set this up through the environment, one can use a wrapper script or an alias. For the same reason I also think that FAKE_DNS should be a command line option (in particular because two other DNS modes are settable via the command line). Maybe make a single DNS command line option that takes a parameter. We can still honor the environment variable or issue a warning/error so that user's aren't surprised. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From gregor at icir.org Sun Aug 7 10:07:06 2011 From: gregor at icir.org (Gregor Maier) Date: Sun, 07 Aug 2011 10:07:06 -0700 Subject: [Bro-Dev] New script organization Message-ID: <4E3EC63A.2070005@icir.org> Hi, I really like the new script layout. Having just poked at it, it really is a lot easier to find stuff in there :-) I do however have some quick comments: * I would rename "all.bro" to "default.bro". I think "all" is misleading, since it doesn't load everything, it rather loads the default config. * I would also rename use the extension ".init" for the file, since its automatically loaded by the core without script intervention. I.e., call it "default.init". * I would move the test check whether "default.init" should be loaded into the core and not handle that at the script level. * Does it make sense to do some BROPATH magic? For each path in BROPATH, we could automatically add "/policy", "/site", "/base"? But YMMV. I'd like to see the first three points. Not so sure about the last one though. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From robin at icir.org Sun Aug 7 11:50:51 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 7 Aug 2011 11:50:51 -0700 Subject: [Bro-Dev] Command line VS. environmnet variable (was: Re: [Bro-Commits] [git/bro] master: Hopefully the last major script reorganization. (597a4d6)) In-Reply-To: <4E3E1B8B.9090708@icir.org> References: <201108060447.p764l9QV022156@bro-ids.icir.org> <4E3E1B8B.9090708@icir.org> Message-ID: <20110807185051.GC52919@icir.org> On Sat, Aug 06, 2011 at 21:58 -0700, you wrote: > I really, really, *really* think this should be a command line option! Yeah, I like a command flag better too. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From slagell at ncsa.illinois.edu Sun Aug 7 11:51:49 2011 From: slagell at ncsa.illinois.edu (Adam Slagell) Date: Sun, 7 Aug 2011 13:51:49 -0500 (CDT) Subject: [Bro-Dev] Command line VS. environmnet variable (was: Re: [Bro-Commits] [git/bro] master: Hopefully the last major script reorganization. (597a4d6)) In-Reply-To: <20110807185051.GC52919@icir.org> References: <201108060447.p764l9QV022156@bro-ids.icir.org> <4E3E1B8B.9090708@icir.org> <20110807185051.GC52919@icir.org> Message-ID: <3F44C077-FF4F-482F-84EA-3924B99C50CC@ncsa.illinois.edu> Sent from my mobile On Aug 7, 2011, at 1:51 PM, Robin Sommer wrote: > > On Sat, Aug 06, 2011 at 21:58 -0700, you wrote: > >> I really, really, *really* think this should be a command line option! > > Yeah, I like a command flag better too. > Ditto > Robin > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From robin at icir.org Sun Aug 7 11:52:37 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 7 Aug 2011 11:52:37 -0700 Subject: [Bro-Dev] New script organization In-Reply-To: <4E3EC63A.2070005@icir.org> References: <4E3EC63A.2070005@icir.org> Message-ID: <20110807185237.GD52919@icir.org> On Sun, Aug 07, 2011 at 10:07 -0700, you wrote: > * I would rename "all.bro" to "default.bro". I think "all" is > * I would also rename use the extension ".init" for the file, since its > * I would move the test check whether "default.init" should be loaded > into the core and not handle that at the script level. I like these (the final one only if we did a command line option though). > * Does it make sense to do some BROPATH magic? For each path in > BROPATH, we could automatically add "/policy", "/site", "/base"? > But YMMV. I don't like this at all. :) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Sun Aug 7 12:58:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 07 Aug 2011 19:58:44 -0000 Subject: [Bro-Dev] #541: Proposal for default filter $path names In-Reply-To: <046.4f90fad6a8b6c0a92033571067d81ead@tracker.bro-ids.org> References: <046.4f90fad6a8b6c0a92033571067d81ead@tracker.bro-ids.org> Message-ID: <061.dae44122304e70071da0e103c3c04bdd@tracker.bro-ids.org> #541: Proposal for default filter $path names ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by robin): On Fri, Aug 05, 2011 at 15:36 -0000, you wrote: > with the current but confusing naming scheme. I want to propose that the > logging framework use the module name for the default path if it's a ::LOG > value, and otherwise use the value itself without the module. This is getting quite complex for something done internally. But: I think we can just add a default_path_func to the scripting layer that kicks in if a filter doesn't define its own, and there you can do whatever you want. I'll add that. (We might need a another bif to get you access to the module name though.) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Aug 7 13:28:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 07 Aug 2011 20:28:12 -0000 Subject: [Bro-Dev] #540: Method of initializing communication subsystem In-Reply-To: <046.71e2fbd6fd1c296226d9add95702e199@tracker.bro-ids.org> References: <046.71e2fbd6fd1c296226d9add95702e199@tracker.bro-ids.org> Message-ID: <061.709e8900711783d7c46d21f72505d4ca@tracker.bro-ids.org> #540: Method of initializing communication subsystem ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by robin): On Fri, Aug 05, 2011 at 15:10 -0000, you wrote: > > ``enable_communication``? Ok, I have a patch for this. Will come once I have switched to the reorged policy scripts locally. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Sun Aug 7 13:54:23 2011 From: gregor at icir.org (Gregor Maier) Date: Sun, 07 Aug 2011 13:54:23 -0700 Subject: [Bro-Dev] Connection Compressor Message-ID: <4E3EFB7F.10603@icir.org> Hi, since my tunnel decapsulation code can't propagate the identity of the tunnel's parent and since the connection compressor doesn't work with IPv6 yet, I was wondering whether it actually still saves a significant amount of memory or not. So, I've done some benchmarks and it appears that the connection compressor doesn't help much more these days given that we accept all packets by default. * 107GB trace, 5M conns, with http and conn: memory and runtime unchanged. same trace with bro-1.5, conn, http-request, http-response, filter that * accepts all packets: runtime unchanged. Memory 202MB vs. 215MB * same trace, only SYN,FIN,RST packets: memory and runtime unchanged * pure SYN trace, 34M conns (==SYNs): that's the only case were I saw a difference: CC: 1202s, 128MB, no-CC: 1613s, 276MB So, all in all, it appears that the connection compressor doesn't help much anymore these days and given that has been in pain in the past and that we'd have to extend it to support IPv6 as well, I would opt for removing it. (BTW, I've briefly talked to Robin about that before I did the benchmark and the thought was, that just disabling the connection compressor by default is not a good idea, since it would almost certainly fall in disuse and would succumb to bit-rot. So we should either keep it and leave it on per default, or remove the code) cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Sun Aug 7 15:29:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 07 Aug 2011 22:29:54 -0000 Subject: [Bro-Dev] #543: Bro autodoc after script reorganization Message-ID: <048.a8604e1df88ab44815b110740aa391a4@tracker.bro-ids.org> #543: Bro autodoc after script reorganization ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: autodoc | ---------------------+------------------------ Hi, Doc generation (make doc) is currently broken in master, presumably since the identity of the base/, policy/, and site/ is lost. As Seth noted in his commit mail for d223637460988419ff "Small tweaks for doc generation. - There is still a bug in that scripts loaded by core scripts don't get their path correctly figured out by Bro's documentation generation. " It would be great if we could get that fixed soon, because I actually wanted to finish my tunnel mode documentation by checking whether the cross references work as expected, but I guess I'll have to wait for that. -- Ticket URL: Bro Tracker Bro Issue Tracker From hlin33 at illinois.edu Sun Aug 7 18:18:42 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Sun, 7 Aug 2011 18:18:42 -0700 Subject: [Bro-Dev] Another Commit problem Message-ID: HI, I met another problem when committing. Actually, when I am committing CMakeLists.txt, the following error happens. error: insufficient permission for adding an object to repository database .git/objects error: src/CMakeLists.txt: failed to insert into database error: unable to index file src/CMakeLists.txt fatal: updating files failed -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110807/1a73257d/attachment.html From robin at icir.org Sun Aug 7 19:23:49 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 7 Aug 2011 19:23:49 -0700 Subject: [Bro-Dev] Another Commit problem In-Reply-To: References: Message-ID: <20110808022349.GI52919@icir.org> On Sun, Aug 07, 2011 at 18:18 -0700, you wrote: > I met another problem when committing. Actually, when I am committing > CMakeLists.txt, the following error happens. Is this indeed happening with "commit" (and not "push")? If yes, it looks like a local problem with your copy of the repository I would say. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From hlin33 at illinois.edu Sun Aug 7 19:29:29 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Sun, 7 Aug 2011 19:29:29 -0700 Subject: [Bro-Dev] Another Commit problem In-Reply-To: <20110808022349.GI52919@icir.org> References: <20110808022349.GI52919@icir.org> Message-ID: Hi, Robin, Thanks for pointing out. I have to sudo to use git commit for some files. I have not met this situation before. Anyway, thanks. Best, Hui On Sun, Aug 7, 2011 at 7:23 PM, Robin Sommer wrote: > > On Sun, Aug 07, 2011 at 18:18 -0700, you wrote: > > > I met another problem when committing. Actually, when I am committing > > CMakeLists.txt, the following error happens. > > Is this indeed happening with "commit" (and not "push")? If yes, it > looks like a local problem with your copy of the repository I would > say. > > Robin > > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org > -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110807/4b9891f0/attachment.html From bro at tracker.bro-ids.org Sun Aug 7 21:26:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 08 Aug 2011 04:26:47 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.1bbbed7bdebb021f8650ed7081cd8f6b@tracker.bro-ids.org> #535: Get rid of the "net" type. ---------------------+-------------------- Reporter: seth | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ---------------------+-------------------- Changes (by robin): * milestone: Bro1.7 => Bro1.6 Comment: On Thu, Aug 04, 2011 at 15:45 -0000, you wrote: > #535: Get rid of the "net" type. This is a good idea. I'd actually like to do it sooner than 1.7 if we can. Getting rid of stuff is good, and this only confuses people when they run into it. So setting milestone to 1.6, we can always bump it. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Sun Aug 7 21:33:58 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 7 Aug 2011 21:33:58 -0700 Subject: [Bro-Dev] trace-summary question In-Reply-To: <4E3C36B9.5000201@ohio.edu> References: <4E3C36B9.5000201@ohio.edu> Message-ID: <20110808043358.GA77167@icir.org> On Fri, Aug 05, 2011 at 11:30 -0700, you wrote: > Just curious: is there a good reason for '700'? It's "good enough". :-) The problem here is that Bro sometimes reports outrageously high volume: it computes the volume from TCP sequence numbers and gets utterly confused if they wrap around. So anything that looks like an unrealistic bandwidth will do. (Unfortunately, wrap around is more likely to occur for larger connections, and by excluding those we may miss actually a signficiant chunk for the summary. But there's not much to do about that with a given summary; garbage in, garbage out. :-) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Sun Aug 7 21:39:26 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 7 Aug 2011 21:39:26 -0700 Subject: [Bro-Dev] Hui Lin_Confusion in Dynamic Protocol Detection In-Reply-To: References: Message-ID: <20110808043926.GE75899@icir.org> On Thu, Aug 04, 2011 at 22:19 -0700, you wrote: > 1. From the "Class Layout" picture, every analyzer is derived from class > "Analyzer", but the wording also says that "The root node must always be of > type TransportLayerAnalyzer." So which one is the real root in the Bro's > code. yzer directly derived by "Analyzer") are located in this analyzer tree > structure. There are two different trees here: (1) the class hierarchy, which is shown on the Wiki page and in which the Analyzer class is the root; (2) the tree of analyzer *instances* instantiated for each connection at runtime. In the latter, a TransportLayerAnalyzer instance must be the root. The paper may help: http://www.icir.org/robin/papers/usenix06.pdf > So what is the differences between TCP_ApplicationAnalyzer and > TCP_Analyzer. The TCP_Analyzer analyzes TCP itself, while a TCP_ApplicationAnalyzer analyzes an application-layer protocol that's running on top of TCP. The former passes payload data on to the latter that's why they are linked in the analyzer tree. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From hlin33 at illinois.edu Sun Aug 7 21:56:01 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Sun, 7 Aug 2011 21:56:01 -0700 Subject: [Bro-Dev] Hui Lin_Confusion in Dynamic Protocol Detection In-Reply-To: <20110808043926.GE75899@icir.org> References: <20110808043926.GE75899@icir.org> Message-ID: For 1, I am OK. For 2, I still confused, please see the inline comment. On Sun, Aug 7, 2011 at 9:39 PM, Robin Sommer wrote: > > On Thu, Aug 04, 2011 at 22:19 -0700, you wrote: > > > 1. From the "Class Layout" picture, every analyzer is derived from class > > "Analyzer", but the wording also says that "The root node must always be > of > > type TransportLayerAnalyzer." So which one is the real root in the Bro's > > code. yzer directly derived by "Analyzer") are located in this analyzer > tree > > structure. > > There are two different trees here: (1) the class hierarchy, which is > shown on the Wiki page and in which the Analyzer class is the root; > (2) the tree of analyzer *instances* instantiated for each connection > at runtime. In the latter, a TransportLayerAnalyzer instance must be > the root. The paper may help: > http://www.icir.org/robin/papers/usenix06.pdf > > > So what is the differences between TCP_ApplicationAnalyzer and > > TCP_Analyzer. > > The TCP_Analyzer analyzes TCP itself, while a TCP_ApplicationAnalyzer > analyzes an application-layer protocol that's running on top of TCP. > The former passes payload data on to the latter that's why they are > linked in the analyzer tree. > So it seems that TCP_ApplicationAnalyzer behave like a helping interface between TCP protocol and other application-over-TCP protocol. I would also like to learn how TCP_Analyzer passes payload to TCP_AppliationAnalyzer in implementation. For the DNP3 protocol, I actually have to write two application level analyzer and one passes the payload to the other one to do some further parsing. I would like to refer TCP's implementation. > Robin > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org > -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110807/f0a804d2/attachment.html From vern at icir.org Sun Aug 7 22:01:23 2011 From: vern at icir.org (Vern Paxson) Date: Sun, 07 Aug 2011 22:01:23 -0700 Subject: [Bro-Dev] trace-summary question In-Reply-To: <20110808043358.GA77167@icir.org> (Sun, 07 Aug 2011 21:33:58 PDT). Message-ID: <20110808050123.5ACF136A376@taffy.ICSI.Berkeley.EDU> > (Unfortunately, wrap around is more likely to occur for larger > connections, and by excluding those we may miss actually a signficiant > chunk for the summary. But there's not much to do about that with a > given summary; garbage in, garbage out. :-) (Well, there used to be something to do - use large-conns.bro, until it was unceremoniously dumped) From robin at icir.org Sun Aug 7 22:04:53 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 7 Aug 2011 22:04:53 -0700 Subject: [Bro-Dev] trace-summary question In-Reply-To: <20110808050123.5ACF136A376@taffy.ICSI.Berkeley.EDU> References: <20110808043358.GA77167@icir.org> <20110808050123.5ACF136A376@taffy.ICSI.Berkeley.EDU> Message-ID: <20110808050453.GF75899@icir.org> On Sun, Aug 07, 2011 at 22:01 -0700, you wrote: > (Well, there used to be something to do - use large-conns.bro That's what the part 'with a given summary' was aiming at: if you feed trace-summary a conn.log that already has that problme, there's nothing it can do about it. That's not saying there aren't ways to get the conn.log right in the first place. :) > , until it was unceremoniously dumped) Was it dumped, or is it just not moved over yet? I don't recall. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Mon Aug 8 07:28:00 2011 From: seth at icir.org (Seth Hall) Date: Mon, 8 Aug 2011 10:28:00 -0400 Subject: [Bro-Dev] Command line VS. environmnet variable (was: Re: [Bro-Commits] [git/bro] master: Hopefully the last major script reorganization. (597a4d6)) In-Reply-To: <4E3E1B8B.9090708@icir.org> References: <201108060447.p764l9QV022156@bro-ids.icir.org> <4E3E1B8B.9090708@icir.org> Message-ID: <7B0D767E-3F90-4BE2-81BE-336AA8A6537F@icir.org> On Aug 7, 2011, at 12:58 AM, Gregor Maier wrote: > >> - All of scripts/base/ is loaded by main.cc >> - Can be disabled by setting $BRO_NO_BASE_SCRIPTS >> - Scripts in scripts/base/ don't use relative path loading to ease use of BRO_NO_BASE_SCRIPTS (to copy and paste that script). > > I really, really, *really* think this should be a command line option! After all, that's what command line options are there for: to enable/disable/modify features of the executed process. > Environment variables are a pain when used to set options. They are nice/good to set paths, etc. Good to hear that you think so too. :) That's the plan but while I was doing the script-level development I just made it an environment variable. I did forget to mention that in the message. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Mon Aug 8 07:33:43 2011 From: seth at icir.org (Seth Hall) Date: Mon, 8 Aug 2011 10:33:43 -0400 Subject: [Bro-Dev] trace-summary question In-Reply-To: <20110808050453.GF75899@icir.org> References: <20110808043358.GA77167@icir.org> <20110808050123.5ACF136A376@taffy.ICSI.Berkeley.EDU> <20110808050453.GF75899@icir.org> Message-ID: On Aug 8, 2011, at 1:04 AM, Robin Sommer wrote: > That's what the part 'with a given summary' was aiming at: if you feed > trace-summary a conn.log that already has that problme, there's > nothing it can do about it. That's not saying there aren't ways to get > the conn.log right in the first place. :) My plan was to enable Gregor's ConnSize analyzer by default. Does it make sense to use the values acquired from that in place of the existing values? >> , until it was unceremoniously dumped) > > Was it dumped, or is it just not moved over yet? I don't recall. Not moved over yet. I didn't dump anything, it's all just waiting to regain it's status in the sun. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Mon Aug 8 07:38:08 2011 From: seth at icir.org (Seth Hall) Date: Mon, 8 Aug 2011 10:38:08 -0400 Subject: [Bro-Dev] New script organization In-Reply-To: <4E3EC63A.2070005@icir.org> References: <4E3EC63A.2070005@icir.org> Message-ID: On Aug 7, 2011, at 1:07 PM, Gregor Maier wrote: > * I would rename "all.bro" to "default.bro". I think "all" is > * I would also rename use the extension ".init" for the file, since its > * I would move the test check whether "default.init" should be loaded Agreed on all points. I'll make sure those happen soon. > * Does it make sense to do some BROPATH magic? For each path in > BROPATH, we could automatically add "/policy", "/site", "/base"? > But YMMV. I agree with Robin here. If we start adding lots of directories to the BROPATH it would get confusing to tell where something is loading from when you see "@load xxx" in a script. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Mon Aug 8 07:39:39 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 08 Aug 2011 14:39:39 -0000 Subject: [Bro-Dev] #541: Proposal for default filter $path names In-Reply-To: <046.4f90fad6a8b6c0a92033571067d81ead@tracker.bro-ids.org> References: <046.4f90fad6a8b6c0a92033571067d81ead@tracker.bro-ids.org> Message-ID: <061.b96fde41aadd9ccee1c713b8e56fb56e@tracker.bro-ids.org> #541: Proposal for default filter $path names ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by seth): > This is getting quite complex for something done internally. But: I > think we can just add a default_path_func to the scripting layer that > kicks in if a filter doesn't define its own, and there you can do > whatever you want. I'll add that. Sounds great. As I was writing the proposal I kept hating it more and more too. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 8 07:40:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 08 Aug 2011 14:40:25 -0000 Subject: [Bro-Dev] #540: Method of initializing communication subsystem In-Reply-To: <046.71e2fbd6fd1c296226d9add95702e199@tracker.bro-ids.org> References: <046.71e2fbd6fd1c296226d9add95702e199@tracker.bro-ids.org> Message-ID: <061.643dc15a5dbc5b6caf7387f1ff88f4b0@tracker.bro-ids.org> #540: Method of initializing communication subsystem ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by seth): > Ok, I have a patch for this. Will come once I have switched to the > reorged policy scripts locally. Awesome, I'm looking forward to getting rid of that ENABLE_COMMUNICATION environment variable. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Mon Aug 8 07:46:09 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 8 Aug 2011 09:46:09 -0500 Subject: [Bro-Dev] New script organization In-Reply-To: <4E3EC63A.2070005@icir.org> References: <4E3EC63A.2070005@icir.org> Message-ID: > * I would also rename use the extension ".init" for the file, since its > automatically loaded by the core without script intervention. I.e., > call it "default.init". I don't like the use of ".init" for anything. To me, that's implying it might not be the same language as the other scripts. Is there an advantage of using the exceptional extension? Maybe it was originally meant to imply it's an internal script, but now anything in "base/" could almost be considered "internal". Just throwing out an idea: use a "default.bro" and then change "bro.init" => "core.bro". - Jon From seth at icir.org Mon Aug 8 07:52:12 2011 From: seth at icir.org (Seth Hall) Date: Mon, 8 Aug 2011 10:52:12 -0400 Subject: [Bro-Dev] New script organization In-Reply-To: References: <4E3EC63A.2070005@icir.org> Message-ID: On Aug 8, 2011, at 10:46 AM, Jonathan Siwek wrote: > I don't like the use of ".init" for anything. To me, that's implying it might not be the same language as the other scripts. Is there an advantage of using the exceptional extension? Maybe it was originally meant to imply it's an internal script, but now anything in "base/" could almost be considered "internal". That's actually a really good point. I think I could go for that. The first time I saw the bro.init script I was pretty confused since it's just a plain Bro script. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Mon Aug 8 08:17:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 08 Aug 2011 15:17:11 -0000 Subject: [Bro-Dev] #543: Bro autodoc after script reorganization In-Reply-To: <048.a8604e1df88ab44815b110740aa391a4@tracker.bro-ids.org> References: <048.a8604e1df88ab44815b110740aa391a4@tracker.bro-ids.org> Message-ID: <063.8d9b0473fa12bf509dc83192f010e015@tracker.bro-ids.org> #543: Bro autodoc after script reorganization ----------------------+------------------------ Reporter: gregor | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: autodoc ----------------------+------------------------ Changes (by jsiwek): * owner: => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon Aug 8 08:20:03 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 8 Aug 2011 08:20:03 -0700 Subject: [Bro-Dev] trace-summary question In-Reply-To: References: <20110808043358.GA77167@icir.org> <20110808050123.5ACF136A376@taffy.ICSI.Berkeley.EDU> <20110808050453.GF75899@icir.org> Message-ID: <20110808152003.GD60443@icir.org> On Mon, Aug 08, 2011 at 10:33 -0400, you wrote: > My plan was to enable Gregor's ConnSize analyzer by default. Does it make sense to use the values acquired from that in place of the existing values? We should offer both, they have different sematnics. However we could include only the ConnSize one in base (as that's probably what people intuively expect) and offer the current one as an option to add additionally? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon Aug 8 08:25:12 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 8 Aug 2011 08:25:12 -0700 Subject: [Bro-Dev] New script organization In-Reply-To: References: <4E3EC63A.2070005@icir.org> Message-ID: <20110808152512.GE60443@icir.org> On Mon, Aug 08, 2011 at 09:46 -0500, you wrote: > it might not be the same language as the other scripts. Is there an > advantage of using the exceptional extension? It indicates that loading them is hard-coded into the core, there's nothign the script-level can do about that. But you're right in that the .init extension can be misleading. > Maybe it was originally meant to imply it's an internal script, but > now anything in "base/" could almost be considered "internal". These scripts are still different than the other base/* stuff, which is loaded from other scripts. I'd like to keep that distinction somewhow. How about using a prefix, like "init-bare.bro" and "init-default.bro" instead? > "bro.init" => "core.bro". (Let's avoid the term "core", that's already overloaded.) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at ncsa.illinois.edu Mon Aug 8 08:33:45 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 8 Aug 2011 10:33:45 -0500 Subject: [Bro-Dev] New script organization In-Reply-To: <20110808152512.GE60443@icir.org> References: <4E3EC63A.2070005@icir.org> <20110808152512.GE60443@icir.org> Message-ID: <4693A34A-4475-49CC-BD04-A7A15F1E81DE@ncsa.illinois.edu> > These scripts are still different than the other base/* stuff, which > is loaded from other scripts. I'd like to keep that distinction > somewhow. How about using a prefix, like "init-bare.bro" and > "init-default.bro" instead? I like it. - Jon From robin at icir.org Mon Aug 8 08:49:33 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 8 Aug 2011 08:49:33 -0700 Subject: [Bro-Dev] Hui Lin_Confusion in Dynamic Protocol Detection In-Reply-To: References: <20110808043926.GE75899@icir.org> Message-ID: <20110808154932.GC94652@icir.org> On Sun, Aug 07, 2011 at 21:56 -0700, you wrote: > So it seems that TCP_ApplicationAnalyzer behave like a helping interface > between TCP protocol and other application-over-TCP protocol. I would also > like to learn how TCP_Analyzer passes payload to TCP_AppliationAnalyzer in > implementation. For the DNP3 protocol, I actually have to write two > application level analyzer and one passes the payload to the other one to do > some further parsing. I would like to refer TCP's implementation. TCP's data flow is more complex than you need (I believe) because the TCP reassembler is potentially involved too. In your case, the first analyzer would call its ForwardStream(), and the data will then show up in the second's DeliverStream() method. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Mon Aug 8 09:08:58 2011 From: seth at icir.org (Seth Hall) Date: Mon, 8 Aug 2011 12:08:58 -0400 Subject: [Bro-Dev] trace-summary question In-Reply-To: <20110808152003.GD60443@icir.org> References: <20110808043358.GA77167@icir.org> <20110808050123.5ACF136A376@taffy.ICSI.Berkeley.EDU> <20110808050453.GF75899@icir.org> <20110808152003.GD60443@icir.org> Message-ID: On Aug 8, 2011, at 11:20 AM, Robin Sommer wrote: > On Mon, Aug 08, 2011 at 10:33 -0400, you wrote: > >> My plan was to enable Gregor's ConnSize analyzer by default. Does it make sense to use the values acquired from that in place of the existing values? > > We should offer both, they have different sematnics. However we could > include only the ConnSize one in base (as that's probably what people > intuively expect) and offer the current one as an option to add > additionally? I think that sounds optimal. We even have the perfect place to put the script that adds that information now (after the latest reorg). :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Mon Aug 8 09:12:38 2011 From: seth at icir.org (Seth Hall) Date: Mon, 8 Aug 2011 12:12:38 -0400 Subject: [Bro-Dev] New script organization In-Reply-To: <4693A34A-4475-49CC-BD04-A7A15F1E81DE@ncsa.illinois.edu> References: <4E3EC63A.2070005@icir.org> <20110808152512.GE60443@icir.org> <4693A34A-4475-49CC-BD04-A7A15F1E81DE@ncsa.illinois.edu> Message-ID: <3FC03F17-CD79-41F5-9E6A-F7A694A8BD84@icir.org> On Aug 8, 2011, at 11:33 AM, Jonathan Siwek wrote: >> These scripts are still different than the other base/* stuff, which >> is loaded from other scripts. I'd like to keep that distinction >> somewhow. How about using a prefix, like "init-bare.bro" and >> "init-default.bro" instead? > > I like it. So do I. I'll go that route. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Mon Aug 8 10:01:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 08 Aug 2011 17:01:01 -0000 Subject: [Bro-Dev] #68: BroV6 support -- memory usage In-Reply-To: <048.a7ea7982ce037ed963ed20f3871939b0@tracker.bro-ids.org> References: <048.a7ea7982ce037ed963ed20f3871939b0@tracker.bro-ids.org> Message-ID: <063.91a1f6a0879db5ef8702f22e43e36923@tracker.bro-ids.org> #68: BroV6 support -- memory usage ---------------------+------------------------------------------ Reporter: gregor | Owner: Type: Task | Status: seen Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: 1.5.2 Resolution: | Keywords: BroV6, binpac++, HILTI, IPv6 ---------------------+------------------------------------------ Comment (by gregor): See also http://www.bro-ids.org/development/projects/ipv6.html -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Mon Aug 8 10:02:28 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 08 Aug 2011 10:02:28 -0700 Subject: [Bro-Dev] Tracker's favicon Message-ID: <4E4016A4.1050409@icir.org> Hi, looks like the tracker still uses the its default favicon. Since we've already tuned the layout to look like the Bro website, we might also want to use Bro's favicon (if somebody feels like doing that) cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From seth at icir.org Mon Aug 8 10:11:20 2011 From: seth at icir.org (Seth Hall) Date: Mon, 8 Aug 2011 13:11:20 -0400 Subject: [Bro-Dev] Tracker's favicon In-Reply-To: <4E4016A4.1050409@icir.org> References: <4E4016A4.1050409@icir.org> Message-ID: On Aug 8, 2011, at 1:02 PM, Gregor Maier wrote: > looks like the tracker still uses the its default favicon. Since we've > already tuned the layout to look like the Bro website, we might also > want to use Bro's favicon (if somebody feels like doing that) Done. I have no idea how long browsers cache the old favicon though. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Mon Aug 8 13:32:37 2011 From: seth at icir.org (Seth Hall) Date: Mon, 8 Aug 2011 16:32:37 -0400 Subject: [Bro-Dev] vnc+websockets Message-ID: >From twitter: VNC client using HTML5 (Web Sockets, Canvas) with encryption (wss://) support. github.com/kanaka/noVNC I doubt it's too far off until we have to write a Web Sockets analyzer. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From vern at icir.org Mon Aug 8 16:11:06 2011 From: vern at icir.org (Vern Paxson) Date: Mon, 08 Aug 2011 16:11:06 -0700 Subject: [Bro-Dev] vnc+websockets In-Reply-To: (Mon, 08 Aug 2011 16:32:37 EDT). Message-ID: <20110808231106.A2A0836A38F@taffy.ICSI.Berkeley.EDU> > VNC client using HTML5 (Web Sockets, Canvas) with encryption (wss://) support. github.com/kanaka/noVNC > > I doubt it's too far off until we have to write a Web Sockets analyzer. :) And how about a VNC analyzer? The ICSI admins were just asking for one (or at least a recognizer, to spot non-tunneled VNC). Vern From seth at icir.org Mon Aug 8 17:07:22 2011 From: seth at icir.org (Seth Hall) Date: Mon, 8 Aug 2011 20:07:22 -0400 Subject: [Bro-Dev] vnc+websockets In-Reply-To: <20110808231106.A2A0836A38F@taffy.ICSI.Berkeley.EDU> References: <20110808231106.A2A0836A38F@taffy.ICSI.Berkeley.EDU> Message-ID: On Aug 8, 2011, at 7:11 PM, Vern Paxson wrote: > And how about a VNC analyzer? The ICSI admins were just asking for one > (or at least a recognizer, to spot non-tunneled VNC). We have the very slightest beginnings of one in the tracker from Justin Azoff. It probably needs a lot added to it before being integrated though. http://tracker.bro-ids.org/bro/ticket/194 Anyone up for working on that more? Of course, it sounds like this analyzer would probably satisfy the ICSI admins as-is. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Mon Aug 8 17:52:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 00:52:16 -0000 Subject: [Bro-Dev] #544: scan.bro needs updating Message-ID: <047.737dcb7555d7c558c1757654428e87e0@tracker.bro-ids.org> #544: scan.bro needs updating ---------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: beta | ---------------------+------------------------ This is a reminder that scan.bro needs updating for the new scripts. When done, uncomment it in test-all-policy.bro. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 8 17:53:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 00:53:05 -0000 Subject: [Bro-Dev] #544: scan.bro and hot.conn.bro need updating (was: scan.bro needs updating) In-Reply-To: <047.737dcb7555d7c558c1757654428e87e0@tracker.bro-ids.org> References: <047.737dcb7555d7c558c1757654428e87e0@tracker.bro-ids.org> Message-ID: <062.9bb9dd77dbec6ee4bdd7776381a9b235@tracker.bro-ids.org> #544: scan.bro and hot.conn.bro need updating ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Description changed by robin: Old description: > This is a reminder that scan.bro needs updating for the new scripts. When > done, uncomment it in test-all-policy.bro. New description: This is a reminder that scan.bro and hot.conn.bro need updating for the new scripts. When done, uncomment them in test-all-policy.bro. -- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 8 17:58:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 00:58:37 -0000 Subject: [Bro-Dev] #544: scan.bro and hot.conn.bro need updating In-Reply-To: <047.737dcb7555d7c558c1757654428e87e0@tracker.bro-ids.org> References: <047.737dcb7555d7c558c1757654428e87e0@tracker.bro-ids.org> Message-ID: <062.367cb958dff8dd309ab5b714e7d47109@tracker.bro-ids.org> #544: scan.bro and hot.conn.bro need updating ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by seth): Will do with scan.bro. hot.conn.bro is going away. It's going to become protocols/conn/detect-intel.bro, I just left it around to remind me and to pull code from. I'm finishing up most of the lower level things now so I'm back to doing final improvements to scripts and finishing the metrics and intel frameworks. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 8 18:08:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 01:08:36 -0000 Subject: [Bro-Dev] #543: Bro autodoc after script reorganization In-Reply-To: <048.a8604e1df88ab44815b110740aa391a4@tracker.bro-ids.org> References: <048.a8604e1df88ab44815b110740aa391a4@tracker.bro-ids.org> Message-ID: <063.f6d550bb58d47d9ceef92edbd84386d9@tracker.bro-ids.org> #543: Bro autodoc after script reorganization ----------------------+------------------------ Reporter: gregor | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: autodoc ----------------------+------------------------ Comment (by jsiwek): In [4ac6d0ae2e4ffa9397bff62f42483f13dac4bb6f/bro]: {{{ #!CommitTicketReference repository="bro" revision="4ac6d0ae2e4ffa9397bff62f42483f13dac4bb6f" Fixes for script auto-documentation. - Fixing the parts of the `make restdoc` and `make doc` process that were broken by the last Bro script re-organization - Generated documentation for Bro scripts derived from BiFs now use the original BiF source file as the "original source file" link - Renaming of the internal POLICYDEST definition and other misc places that refer to "policy" scripts; that terminology doesn't make total sense now - Added a documentation blacklist reminder test that will fail if there's scripts that are blacklisted from being documentated because they're still in progress - Some minor Bro script changes to fix small @load dependency errors Addresses #543 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 8 18:12:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 01:12:49 -0000 Subject: [Bro-Dev] #543: Bro autodoc after script reorganization In-Reply-To: <048.a8604e1df88ab44815b110740aa391a4@tracker.bro-ids.org> References: <048.a8604e1df88ab44815b110740aa391a4@tracker.bro-ids.org> Message-ID: <063.30ffb4e1295220846ffa69716119a113@tracker.bro-ids.org> #543: Bro autodoc after script reorganization ----------------------------+------------------------ Reporter: gregor | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: autodoc ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: In `topic/jsiwek/autodoc-fixes` -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Mon Aug 8 18:16:56 2011 From: seth at icir.org (Seth Hall) Date: Mon, 8 Aug 2011 21:16:56 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/autodoc-fixes: Fixes for script auto-documentation. (4ac6d0a) In-Reply-To: <201108090108.p7918dZK006803@bro-ids.icir.org> References: <201108090108.p7918dZK006803@bro-ids.icir.org> Message-ID: <1C64E654-C654-4FD3-B87E-D9098EC2F92B@icir.org> On Aug 8, 2011, at 9:08 PM, Jonathan Siwek wrote: > - Fixing the parts of the `make restdoc` and `make doc` process that were > broken by the last Bro script re-organization Awesome! Thanks. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Mon Aug 8 19:11:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 02:11:46 -0000 Subject: [Bro-Dev] #545: More script reorg fallout Message-ID: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> #545: More script reorg fallout ---------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ - in general, it seems scripts may be leaving out `@load` directives based on the assumption that everything in the `base/` directory is loaded by default. Scripts should always explicitly `@load` all dependencies as that will be more pleasant for a user trying to run with `--bare-mode` enabled. - the `testing/btest/policy` directory should be renamed/reorganized according to the new script organization - the `policy/misc/testing-coverage.bro` unit test is falsely succeeding - the `policy/misc/check-test-all.bro` unit test and accompanying `test- all.bro` are inadequate. What I'd like to see it do is a separate invocation of bro using `--bare-mode` for every script under the `scripts/` directory. The manifest of scripts should be auto-generated by the test and doesn't need to be under revision control, meaning `test- all.bro` can be removed. Performing the test like this will uncover dependency issues that are otherwise hidden and always provide complete coverage, meaning the `testing-coverage.bro` test mentioned above can just be removed. - the `make restdoc` target should also enable `--bare-mode` for creating documentation for individual scripts -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 8 19:25:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 02:25:12 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.21153344290f0e38f8b3636e8ad2e2f2@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): > - in general, it seems scripts may be leaving out `@load` directives based > on the assumption that everything in the `base/` directory is loaded by > default. Scripts should always explicitly `@load` all dependencies as > that will be more pleasant for a user trying to run with `--bare-mode` > enabled. Yes, I left that out with an eye toward discussing it a bit more. Do you think scripts within the base/ directory should do relative loading or they should be doing full path loading? I suppose it doesn't matter in any real sense, but I feel like we may run into side effect going with one way compared to another. Presumably the scripts in policy/ would be doing full path loading, but that suddenly makes leaving base/ out of the path a little nasty because people are going to end up prepending that to most of the @loads they do. > - the `testing/btest/policy` directory should be renamed/reorganized > according to the new script organization Agreed. > - the `policy/misc/testing-coverage.bro` unit test is falsely succeeding I'd rather that worked the way to propose anyway. > - the `make restdoc` target should also enable `--bare-mode` for creating > documentation for individual scripts Good point. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 8 19:31:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 02:31:56 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.13bce1b149aa83e7d29805043eec2177@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by robin): Funny, I was just working on this stuff. I've addressed a few things, and collected some notes too: - core.load-prefixes now fails and I don't immediately see why. I believe that was passing before, right? - bare mode still loads the logging framework. Do we want that? - I was trying to load listen-{clear,ssl) in testing-all.bro, but that failed complaining that Communication::default_port_* aren't defined. But it seem they should be available because the comm framework is pulled in as well as part of the base. Any idea? - There are some files in base that aren't loaded by default. A new test, policy.misc.init-default-coverage, shows them. - I think it should work to run "bro -b" and then load files from policy/* without having to manually pull in dependencies from base/*. Currrently however, when I do that I get something quite nasty: bro -b test-all-policy error in /home/robin/bro/master/scripts/policy/frameworks/dpd/detect- protocols.bro, line 6: unknown identifier (Notice::Type) bro: parse.y:717: int yyparse(): Assertion `cur_enum_type' failed. I've added a test for it (which currently fails of course) - *.bif.bro install into the top-level share/bro directory. I think they should go into base now. - There are actually more broctl things going into the top-level share/bro, it would be nice if that directory were empty except for the sub-directories. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Mon Aug 8 21:30:02 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 08 Aug 2011 21:30:02 -0700 Subject: [Bro-Dev] New http policy scripts In-Reply-To: References: <4E3B66A8.6090508@icir.org> Message-ID: <4E40B7CA.7030903@icir.org> > I do agree that I'm doing some pretty egregious stuff in some of those scripts from an optimization perspective, but I think that optimization attempts in Bro scripts have led to incredibly convoluted scripts and dependency chains. I'm going to press instead for optimizations that allow the scripts to remain well structured. For instance, what about just disabling the http_header or http_data events if you don't want those done? This should already be do-able with the disable_event_group like this: > > disable_event_group("http-body"); > disable_event_group("http-header"); after chatting with Seth I actually start to like this idea. Maybe add policy scripts that, when loaded, will disable these groups to shed load. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From hlin33 at illinois.edu Mon Aug 8 22:16:05 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Mon, 8 Aug 2011 22:16:05 -0700 Subject: [Bro-Dev] Hui Lin_Confusion in Dynamic Protocol Detection In-Reply-To: <20110808154932.GC94652@icir.org> References: <20110808043926.GE75899@icir.org> <20110808154932.GC94652@icir.org> Message-ID: After checking Bro's code (especially Analyzer.h, Analyzer.cc), I think the logic is like this (please point out if I am wrong): TCP_Analyzer will parse the TCP protocol and extract the palyload (input to application level protocol analyzer), this payload will be passed up to Analyzer class (how to pass is not clear to me). After Analyzer knows this stream of TCP payload and this is the input of the ForwardStream. ForwardStream then call Analyzer's children to use their own DeliverStream. In each DeliverStream implementation, the Binpac Conn function is used to parse the stream. So in my opinion, from Analyzer, TCP_ApplicationAnalyzer to the application-level analyzer, the stream actually does not change. But for my situation, I have two application-level protocols, p1 and p2. p1 derive from TCP_ApplicationAnalyzer and p1 needs to parse and reconstruct stream from TCP level, not directly pass the stream to p2. So I think what I can do is to let p2 derives from p1. And then define a event handler in p1 to reconstruct stream as it parse its protocol, in this event handler, we can have the reconstructed stream and use it as the input to call ForwardStream of p1. p2 still defines the DeliverStream as usual, and in this way, p2's protocol analyzer should be able get those reconstructed stream. Best, On Mon, Aug 8, 2011 at 8:49 AM, Robin Sommer wrote: > > On Sun, Aug 07, 2011 at 21:56 -0700, you wrote: > > > So it seems that TCP_ApplicationAnalyzer behave like a helping interface > > between TCP protocol and other application-over-TCP protocol. I would > also > > like to learn how TCP_Analyzer passes payload to TCP_AppliationAnalyzer > in > > implementation. For the DNP3 protocol, I actually have to write two > > application level analyzer and one passes the payload to the other one to > do > > some further parsing. I would like to refer TCP's implementation. > > TCP's data flow is more complex than you need (I believe) because the > TCP reassembler is potentially involved too. In your case, the first > analyzer would call its ForwardStream(), and the data will then show > up in the second's DeliverStream() method. > > Robin > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org > -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110808/5008ae58/attachment.html From bro at tracker.bro-ids.org Tue Aug 9 06:52:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 13:52:36 -0000 Subject: [Bro-Dev] #546: topic/seth/piped_exec-update Message-ID: <046.aa8796da7721564d8d12d7a1dcc172d5@tracker.bro-ids.org> #546: topic/seth/piped_exec-update ---------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ Fixed piped_exec to support null characters in the to_write argument. Binary data wasn't working if you tried to pass it to an external program. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 08:18:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 15:18:58 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.324f2fda396ea45eca5bd346100525b4@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): Replying to [comment:0 seth]: > > default. Scripts should always explicitly `@load` all dependencies as > > that will be more pleasant for a user trying to run with `--bare-mode` > > enabled. > > Yes, I left that out with an eye toward discussing it a bit more. Do you think scripts within the base/ directory should do relative loading or they should be doing full path loading? Here's a possible rule to follow: if you need to @load across "package" boundaries use the full path, but if you need to @load something that's intra-package you can use a path relative to the current script. And I'm defining "package" to be any directory of related scripts one level below `base/`, `policy/`, etc. directories. I forget why exactly we implemented relative path loading in the first place (!), but doing it like this should mean you can move packages around as a whole without having to rename dependencies. e.g. say you want to promote a package in `policy/` to now live in `base/`, that should just involve a move of the directory. Replying to [comment:1 robin]: > - core.load-prefixes now fails and I don't immediately see why. It's fixed, pending merge request in #543. > - I was trying to load listen-{clear,ssl) in testing-all.bro, but that > failed complaining that Communication::default_port_* aren't > defined. But it seem they should be available because the comm > framework is pulled in as well as part of the base. Any idea? Didn't verify, but I was looking yesterday at similar errors and there's a TODO in `base/frameworks/communication/__load__.bro` that was waiting on the enable_communication BIF stuff; can probably be fixed now. > - I think it should work to run "bro -b" and then load files from > policy/* without having to manually pull in dependencies from > base/*. Yeah, but I'd go further than that (one of my points above): for any given script anywhere in the `scripts/` tree you should be able to load it by itself with `bro -b` without having to figure out and manually specify dependencies. > bro -b test-all-policy > /home/robin/bro/master/scripts/policy/frameworks/dpd/detect- protocols.bro, > line 6: unknown identifier (Notice::Type) > bro: parse.y:717: int yyparse(): Assertion `cur_enum_type' failed. > > I've added a test for it (which currently fails of course) I don't think relying on the `test-all-policy.bro` script for this coverage test is good enough because 1) we also need to check stuff in `base/` can be loaded individually 2) scripts that are loaded further down in the file might succeed when they otherwise wouldn't just because the right dependency happened to be loaded earlier on > - *.bif.bro install into the top-level share/bro directory. I think they should go into base now. I was wondering that, too. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 08:30:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 15:30:09 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.61b239e7e5c99f2899ed087b3bed6a20@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by robin): On Tue, Aug 09, 2011 at 15:18 -0000, you wrote: > Yeah, but I'd go further than that (one of my points above): for any given > script anywhere in the `scripts/` tree you should be able to load it by > itself with `bro -b` without having to figure out and manually specify > dependencies. That makes sense, want to add a test for that? But we'll still need the test-all-default-coverage.bro test (as it's now called) because test-all-default.bro is loaded by other tests that want to have all scripts included, and we should make sure everything is indeed in there. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 08:34:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 15:34:49 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.b07604b87a8c0823342093adae7255e4@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): > I forget why exactly we implemented > relative path loading in the first place (!), but doing it like this > should mean you can move packages around as a whole without having to > rename dependencies. e.g. say you want to promote a package in `policy/` > to now live in `base/`, that should just involve a move of the directory. That was the basic idea. It was more necessary when we had the and optional stuff in the same directory, but it's still helpful for cases where a module needs to be split across multiple files internally. > Didn't verify, but I was looking yesterday at similar errors and there's a > TODO in `base/frameworks/communication/__load__.bro` that was waiting on > the enable_communication BIF stuff; can probably be fixed now. Fixed in master. > Yeah, but I'd go further than that (one of my points above): for any given > script anywhere in the `scripts/` tree you should be able to load it by > itself with `bro -b` without having to figure out and manually specify > dependencies. Agreed. >> - *.bif.bro install into the top-level share/bro directory. I think they > should go into base now. > > I was wondering that, too. I think makes sense. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 08:59:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 15:59:40 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.9bf397b3f2849e8bcb2d37e396259502@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): Replying to [comment:4 robin]:> > > Yeah, but I'd go further than that (one of my points above): for any given > > script anywhere in the `scripts/` tree you should be able to load it by > > itself with `bro -b` without having to figure out and manually specify > > dependencies. > > That makes sense, want to add a test for that? Ok. I could also work on the other issues mentioned in this ticket, except I don't know what to do about organizing the broctl scripts that get installed at the top-level `share/bro`. I'm not up-to-speed with the current status of broctl; Seth do you want to take that task and I'll do the rest? > But we'll still need the test-all-default-coverage.bro test (as it's > now called) because test-all-default.bro is loaded by other tests that > want to have all scripts included, and we should make sure everything > is indeed in there. Ah, forgot that was one purpose of that script. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 09:13:10 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 16:13:10 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.bd172a64d755551ed71e1e99816844a1@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): > Ok. I could also work on the other issues mentioned in this ticket, > except I don't know what to do about organizing the broctl scripts that > get installed at the top-level `share/bro`. I'm not up-to-speed with the > current status of broctl; Seth do you want to take that task and I'll do > the rest? Already done. I'm just making sure that everything still works correctly before pushing it out. It looks like those files were accidentally readded during the big merge recently. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Tue Aug 9 10:53:53 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 09 Aug 2011 10:53:53 -0700 Subject: [Bro-Dev] Child analyzer question Message-ID: <4E417431.6010507@icir.org> Hi, I want to add a child analyzer to my ConnSizeAnalyzer. I want to do this statically based on the child's Available() class method. The code will look something like: if (ChildAnalyzerClass::Available()) AddChildanalyzer(ChildAnalyzer'sTag) The question now is: do I add the child in the parent's constructor or the parent's Init() method? (I think it should work in either case and that Init() would be the right place to do it, but I'm not sure) thx Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From robin at icir.org Tue Aug 9 10:56:47 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 9 Aug 2011 10:56:47 -0700 Subject: [Bro-Dev] Child analyzer question In-Reply-To: <4E417431.6010507@icir.org> References: <4E417431.6010507@icir.org> Message-ID: <20110809175647.GA50021@icir.org> On Tue, Aug 09, 2011 at 10:53 -0700, you wrote: > that Init() would be the right place to do it, but I'm not sure) Yes, Init(). I don't remembmer the specifics but there are some potential order-of-execution issues with doing it in the ctor. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Tue Aug 9 11:37:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 18:37:51 -0000 Subject: [Bro-Dev] #547: topic/jsiwek/debug-flags Message-ID: <048.644ec20dbcc0abbef35013f8bbaced72@tracker.bro-ids.org> #547: topic/jsiwek/debug-flags ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This branch is in the `capstats`, `pysubnettree`, `broccoli-python`, `broccoli`, `bro-aux`, `binpac`, and `bro` repos. It works around the FreeBSD port of CMake's removal of the `-g` flag when using `CMAKE_BUILD_TYPE`=`Debug` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 12:19:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 19:19:14 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.27c71db340b6104081e59b4d34ace619@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by jsiwek): * owner: => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 13:58:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 09 Aug 2011 20:58:43 -0000 Subject: [Bro-Dev] #548: broctl brokenness multi-ticket Message-ID: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> #548: broctl brokenness multi-ticket ------------------------+-------------------- Reporter: seth | Owner: robin Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Keywords: beta | ------------------------+-------------------- My updates to fix bro/broctl integration are in topic/various-updates. Those updates do not fix these issues however (and any help would be greatly appreciated).... - "check" command is broken. The Bro instance that the check-config script starts up doesn't die. - Potential problem with node naming. Names like this "nids-27A" would start, but broctl wouldn't know afterward that the instance was running (even though it was). - File rotation needs to be disabled when not running on live traffic (for the new process command primarily). - "scripts" command needs to be fixed. Maybe it should run bro and print the loaded_scripts.log file to the output? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 19:36:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 02:36:37 -0000 Subject: [Bro-Dev] #309: Work with Endace to get their code back in In-Reply-To: <047.b48356ad4c0bc5e98558c47d09be301b@tracker.bro-ids.org> References: <047.b48356ad4c0bc5e98558c47d09be301b@tracker.bro-ids.org> Message-ID: <062.e5c81b0782321fd7effc808beaf8f1e4@tracker.bro-ids.org> #309: Work with Endace to get their code back in ---------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: ---------------------+-------------------- Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: Pushing this back since we definitely aren't getting to it yet. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 19:40:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 02:40:31 -0000 Subject: [Bro-Dev] #228: inacitvity.bro defaults should be changed In-Reply-To: <047.4cb06ff9ad63809e037a6b5543762605@tracker.bro-ids.org> References: <047.4cb06ff9ad63809e037a6b5543762605@tracker.bro-ids.org> Message-ID: <062.936b75cdc1cd922ff686599e2b7fedfa@tracker.bro-ids.org> #228: inacitvity.bro defaults should be changed -----------------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.1 Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: inactivity.bro has been ported. http & smtp have been removed since they aren't typically interactive services. User's can add them back in their local config if they desire. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 19:41:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 02:41:28 -0000 Subject: [Bro-Dev] #304: Unit tests for default scripts In-Reply-To: <046.85a1e34c46d45dd021703c2fb461d3bd@tracker.bro-ids.org> References: <046.85a1e34c46d45dd021703c2fb461d3bd@tracker.bro-ids.org> Message-ID: <061.8543116eab8aa3234884304ef7cd5a18@tracker.bro-ids.org> #304: Unit tests for default scripts -----------------------------+-------------------- Reporter: seth | Owner: Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Let's close this. We are in progress on this and will be for quite some time. This ticket is just taking up space. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 19:43:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 02:43:16 -0000 Subject: [Bro-Dev] #487: Bug and missing feature in syslog analyzer In-Reply-To: <046.39317bf8c8ad376fb1f4afd399768ef7@tracker.bro-ids.org> References: <046.39317bf8c8ad376fb1f4afd399768ef7@tracker.bro-ids.org> Message-ID: <061.17ab732c77b1ee458ee1c4d1144ccbd1@tracker.bro-ids.org> #487: Bug and missing feature in syslog analyzer ----------------------+------------------------ Reporter: seth | Owner: seth Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): It turns out that some syslog implementations include the final newline in the actual message and some include an actual end of string null. I'm thinking that we should just strip either of these bytes from the end of syslog messages since they aren't exactly correct even though they are in the actual message on the wire. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 19:44:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 02:44:53 -0000 Subject: [Bro-Dev] #524: Bro fuzz testing In-Reply-To: <048.a8b2fd1e8661b692c679d3046e408dc7@tracker.bro-ids.org> References: <048.a8b2fd1e8661b692c679d3046e408dc7@tracker.bro-ids.org> Message-ID: <063.85a807b6b4a094ba2612ab5ae49e6441@tracker.bro-ids.org> #524: Bro fuzz testing ---------------------+------------------------ Reporter: gregor | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: Bumping this. I don't think we're going to approach this right now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 19:46:55 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 02:46:55 -0000 Subject: [Bro-Dev] #536: protocols/ssh and geoip problems In-Reply-To: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> References: <048.70d4a04b39b9d3f5005cc5da9f166c31@tracker.bro-ids.org> Message-ID: <063.9a55fd023ad6cf3dfcca56f0593ef374@tracker.bro-ids.org> #536: protocols/ssh and geoip problems -----------------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Closing this ticket since I'm mostly done with the SSH reworking that this ticket spawned. #538 can be referenced for the core work that still needs to be done. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 19:48:02 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 02:48:02 -0000 Subject: [Bro-Dev] #540: Method of initializing communication subsystem In-Reply-To: <046.71e2fbd6fd1c296226d9add95702e199@tracker.bro-ids.org> References: <046.71e2fbd6fd1c296226d9add95702e199@tracker.bro-ids.org> Message-ID: <061.2b6020340140597f3efedb763f9efecf@tracker.bro-ids.org> #540: Method of initializing communication subsystem -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: beta -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Done and done. Committed in 103396f6d32fa92e4087fe700b10854aa4488979 -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 19:49:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 02:49:37 -0000 Subject: [Bro-Dev] #473: Remove or redirect "bro_logger" messages. In-Reply-To: <046.ac0045106dd553365c25462db1b26a3d@tracker.bro-ids.org> References: <046.ac0045106dd553365c25462db1b26a3d@tracker.bro-ids.org> Message-ID: <061.9af3489d2234cd2d067bfdffca8e4db5@tracker.bro-ids.org> #473: Remove or redirect "bro_logger" messages. -----------------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: You almost closed it. :) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 19:52:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 02:52:43 -0000 Subject: [Bro-Dev] #253: Can't bind to port 47760, Address already in use In-Reply-To: <056.21a044ffe4916939ea16cb28b2316505@tracker.bro-ids.org> References: <056.21a044ffe4916939ea16cb28b2316505@tracker.bro-ids.org> Message-ID: <071.1ff089851262289cce468f09b856b6bc@tracker.bro-ids.org> #253: Can't bind to port 47760, Address already in use ------------------------------+-------------------- Reporter: Tyler.Schoenke | Owner: robin Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: BroControl | Version: 1.5.2 Resolution: | Keywords: ------------------------------+-------------------- Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: Sorry, I think we're going to have to bump this back. We should be addressing some more of these debugging type issues before long in broctl though. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 19:59:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 02:59:09 -0000 Subject: [Bro-Dev] #460: Problem with extended records and empty container types In-Reply-To: <046.e32eb4f1b2af66ecfcc23a676b3b6332@tracker.bro-ids.org> References: <046.e32eb4f1b2af66ecfcc23a676b3b6332@tracker.bro-ids.org> Message-ID: <061.48a1813a6834e09bbe73597da6273411@tracker.bro-ids.org> #460: Problem with extended records and empty container types ----------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): I have a number of places where this bug has cropped up in the current scripts. Anyone know what's causing it? Definitely helpful to have it fixed. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:01:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:01:53 -0000 Subject: [Bro-Dev] #498: Efficiency problem with remote log flushing In-Reply-To: <047.837129432a53b3b6b387597dd321a7c9@tracker.bro-ids.org> References: <047.837129432a53b3b6b387597dd321a7c9@tracker.bro-ids.org> Message-ID: <062.1c5bedf03903d80c04716154c90d0bb5@tracker.bro-ids.org> #498: Efficiency problem with remote log flushing ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------+------------------------ Comment (by seth): I know you did a bit of work on remote logging lately. Did this get fixed? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:02:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:02:37 -0000 Subject: [Bro-Dev] #273: "read" command in broctl In-Reply-To: <046.a0d547edf0846538616e3f0c2c4ac223@tracker.bro-ids.org> References: <046.a0d547edf0846538616e3f0c2c4ac223@tracker.bro-ids.org> Message-ID: <061.56bfae0058f507cb703c9b64f33248c4@tracker.bro-ids.org> #273: "read" command in broctl ------------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: 1.5.1 Resolution: Solved/Applied | Keywords: ------------------------------+-------------------- Changes (by seth): * status: testing => closed * resolution: => Solved/Applied Comment: This is in broctl master now. Let's just say it's done. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:03:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:03:16 -0000 Subject: [Bro-Dev] #370: Plugin interface for BroControl In-Reply-To: <046.a1bdb89c0c6469205ba5da84eb36c88e@tracker.bro-ids.org> References: <046.a1bdb89c0c6469205ba5da84eb36c88e@tracker.bro-ids.org> Message-ID: <061.c118e59bbc7cf2da86d18d09c6e75ac3@tracker.bro-ids.org> #370: Plugin interface for BroControl ------------------------------+-------------------- Reporter: seth | Owner: robin Type: Feature Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: Solved/Applied | Keywords: ------------------------------+-------------------- Changes (by seth): * status: testing => closed * resolution: => Solved/Applied Comment: And now it's in master so I'm closing the ticket. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:04:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:04:45 -0000 Subject: [Bro-Dev] #461: internal_error called for async DNS failures In-Reply-To: <046.adc7e841ed311111a107267ffb4a852b@tracker.bro-ids.org> References: <046.adc7e841ed311111a107267ffb4a852b@tracker.bro-ids.org> Message-ID: <061.dad82fe614d431a99900aee8f4ad29f3@tracker.bro-ids.org> #461: internal_error called for async DNS failures ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): Any possibility someone could pick up the torch on this for the next release? It's definitely an annoying "crash". -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:06:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:06:42 -0000 Subject: [Bro-Dev] #378: Test for problem with optional record fields. In-Reply-To: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> References: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> Message-ID: <062.dc268ac47d224cebe73d7df4b89f7b43@tracker.bro-ids.org> #378: Test for problem with optional record fields. --------------------------------+-------------------- Reporter: robin | Owner: Type: Test Case Missing | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: --------------------------------+-------------------- Comment (by seth): Robin? Thoughts? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:07:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:07:33 -0000 Subject: [Bro-Dev] #490: Segfault in logging framework In-Reply-To: <046.aea7f8c7ca241603b6858d80236ddc86@tracker.bro-ids.org> References: <046.aea7f8c7ca241603b6858d80236ddc86@tracker.bro-ids.org> Message-ID: <061.a548d2f837c6e44f715ed862d43ac93d@tracker.bro-ids.org> #490: Segfault in logging framework -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: I believe this was fixed and I haven't seen crashes from the logging framework in a while now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:14:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:14:14 -0000 Subject: [Bro-Dev] #470: print helpful info when os.remove() fails In-Reply-To: <047.203d22df6ee6795bf7dd48222840bfe7@tracker.bro-ids.org> References: <047.203d22df6ee6795bf7dd48222840bfe7@tracker.bro-ids.org> Message-ID: <062.810dfc931d7590d8ec7b968a8942f625@tracker.bro-ids.org> #470: print helpful info when os.remove() fails -----------------------------+-------------------- Reporter: leres | Owner: robin Type: Patch | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: 1.5.3 Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: accepted => closed * resolution: => Solved/Applied Comment: This patch is applied to broctl's topic/various-updates branch and will be merged into master along with that. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:18:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:18:12 -0000 Subject: [Bro-Dev] #549: Expose broccoli file handle through pybroccoli Message-ID: <046.1adc4393e81172e99fdad54fd5737163@tracker.bro-ids.org> #549: Expose broccoli file handle through pybroccoli -----------------------------+-------------------- Reporter: seth | Owner: robin Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: broccoli-python | Version: Keywords: | -----------------------------+-------------------- I know there is a patch for this already, it just needs to be applied. :) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:19:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:19:24 -0000 Subject: [Bro-Dev] #347: Problem with df helper script In-Reply-To: <046.ad3146495c6bd8cde499bd4955032353@tracker.bro-ids.org> References: <046.ad3146495c6bd8cde499bd4955032353@tracker.bro-ids.org> Message-ID: <061.b7e540934bd34fffc0b626c9420315c9@tracker.bro-ids.org> #347: Problem with df helper script -----------------------------+------------------------ Reporter: seth | Owner: robin Type: Problem | Status: closed Priority: High | Milestone: Component: BroControl | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: I believe this is working everywhere now. Feel free to reopen if I am incorrect. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:21:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:21:29 -0000 Subject: [Bro-Dev] #205: Cluster code doesn't always startup dead nodes In-Reply-To: <046.e8744337c34a0d40aa8d8db42a122afa@tracker.bro-ids.org> References: <046.e8744337c34a0d40aa8d8db42a122afa@tracker.bro-ids.org> Message-ID: <061.88ad04571de09674348e67af39fdb3d5@tracker.bro-ids.org> #205: Cluster code doesn't always startup dead nodes ---------------------------+-------------------- Reporter: seth | Owner: robin Type: Problem | Status: closed Priority: High | Milestone: Component: BroControl | Version: 1.5.2 Resolution: Works for Me | Keywords: ---------------------------+-------------------- Changes (by seth): * status: assigned => closed * resolution: => Works for Me Comment: I'm going to close this ticket for now. I still haven't heard of others running into this trouble and I haven't seen evidence of it in a long time. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:22:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:22:15 -0000 Subject: [Bro-Dev] #467: irc_dcc_message event is broken. In-Reply-To: <046.2cf508bb8daba167ac058271391d11c0@tracker.bro-ids.org> References: <046.2cf508bb8daba167ac058271391d11c0@tracker.bro-ids.org> Message-ID: <061.749edeeaaac87847196c1a5ab34069af@tracker.bro-ids.org> #467: irc_dcc_message event is broken. -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Jon fixed this several weeks ago. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:23:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:23:33 -0000 Subject: [Bro-Dev] #446: git search is busted In-Reply-To: <046.874deb8b32820528ed90392ca481bd2f@tracker.bro-ids.org> References: <046.874deb8b32820528ed90392ca481bd2f@tracker.bro-ids.org> Message-ID: <061.7472360507c35056e4cf3034e3495d50@tracker.bro-ids.org> #446: git search is busted -----------------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: This was fixed. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:40:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:40:34 -0000 Subject: [Bro-Dev] #378: Test for problem with optional record fields. In-Reply-To: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> References: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> Message-ID: <062.bdc879bf6e3daf4537d768c3569d075c@tracker.bro-ids.org> #378: Test for problem with optional record fields. --------------------------------+---------------------- Reporter: robin | Owner: robin Type: Test Case Missing | Status: accepted Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: --------------------------------+---------------------- Changes (by robin): * owner: => robin * status: new => accepted Comment: (Assigning to me is more reliable than just asking :) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:44:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:44:00 -0000 Subject: [Bro-Dev] #376: Parse git commits to change ticket status In-Reply-To: <047.a5be61122edbaad387e4988bc4831ee8@tracker.bro-ids.org> References: <047.a5be61122edbaad387e4988bc4831ee8@tracker.bro-ids.org> Message-ID: <062.ad5e6471fdd2b6c8f72042bf641be3c8@tracker.bro-ids.org> #376: Parse git commits to change ticket status ------------------------------+-------------------- Reporter: robin | Owner: seth Type: Feature Request | Status: closed Priority: Normal | Milestone: Component: TicketTracker | Version: Resolution: Solved/Applied | Keywords: ------------------------------+-------------------- Changes (by seth): * status: reopened => closed * resolution: => Solved/Applied Comment: Done. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:49:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:49:25 -0000 Subject: [Bro-Dev] #366: Crash with optional table and record ctors In-Reply-To: <047.24f56aef53ab5507c9e96064f32f4268@tracker.bro-ids.org> References: <047.24f56aef53ab5507c9e96064f32f4268@tracker.bro-ids.org> Message-ID: <062.f9509383529a4edf75952c4f9e581af0@tracker.bro-ids.org> #366: Crash with optional table and record ctors -----------------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: This crash is gone now and semantics have changed to reflect what I wrote in my comment. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:50:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:50:29 -0000 Subject: [Bro-Dev] #455: Infrastructure for event engine message In-Reply-To: <047.6dad6de1fba965c77a29aba9353e1865@tracker.bro-ids.org> References: <047.6dad6de1fba965c77a29aba9353e1865@tracker.bro-ids.org> Message-ID: <062.44b73e03c2e6b9cbf00ccdded6d57324@tracker.bro-ids.org> #455: Infrastructure for event engine message -----------------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: closed Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Resolution: Solved/Applied | Keywords: cleanup -----------------------------+--------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: This was done with the reporter framework. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 20:56:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 03:56:22 -0000 Subject: [Bro-Dev] #407: broctl/scripts/send-mail hardwires hostnames, subject prefixes, etc. In-Reply-To: <047.61618ad4a939392bff91a8eeffca0812@tracker.bro-ids.org> References: <047.61618ad4a939392bff91a8eeffca0812@tracker.bro-ids.org> Message-ID: <062.bbfe70a449b678bb06163d824bc7becc@tracker.bro-ids.org> #407: broctl/scripts/send-mail hardwires hostnames, subject prefixes, etc. -----------------------------+-------------------- Reporter: leres | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Component: BroControl | Version: 1.5.1 Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Fixed. closing this ticket. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 21:18:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 04:18:31 -0000 Subject: [Bro-Dev] #528: Python 3 compatibility In-Reply-To: <047.5418f300a382e1c09535f6855df979b9@tracker.bro-ids.org> References: <047.5418f300a382e1c09535f6855df979b9@tracker.bro-ids.org> Message-ID: <062.975b33e70a81d295e0493232bce1ef43@tracker.bro-ids.org> #528: Python 3 compatibility -------------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: -------------------------+-------------------- Changes (by robin): * owner: robin => -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 21:18:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 04:18:48 -0000 Subject: [Bro-Dev] #516: NMI: Output formatting issues in the istate.pybroccoli unit test (Fedora 14) In-Reply-To: <050.8a31ed622b642173bdfa95bbe6b94396@tracker.bro-ids.org> References: <050.8a31ed622b642173bdfa95bbe6b94396@tracker.bro-ids.org> Message-ID: <065.1861b46f533117705b97c0c3444e0487@tracker.bro-ids.org> #516: NMI: Output formatting issues in the istate.pybroccoli unit test (Fedora 14) ------------------------------+-------------------------------------- Reporter: appleman | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: broccoli-python | Version: git/master Resolution: | Keywords: nmi btest testing fedora ------------------------------+-------------------------------------- Changes (by robin): * owner: robin => -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 21:19:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 04:19:14 -0000 Subject: [Bro-Dev] #503: Update broctl README In-Reply-To: <047.1a0db4ec338e9735ce78c840cd80bbc3@tracker.bro-ids.org> References: <047.1a0db4ec338e9735ce78c840cd80bbc3@tracker.bro-ids.org> Message-ID: <062.7d1025277dcc11a44065dd9b6bc17c76@tracker.bro-ids.org> #503: Update broctl README -------------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: | Keywords: preview -------------------------+------------------------ Changes (by robin): * owner: robin => -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 9 21:19:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 04:19:28 -0000 Subject: [Bro-Dev] #548: broctl brokenness multi-ticket In-Reply-To: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> References: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> Message-ID: <061.b935143f8f06a3da7745d56150791550@tracker.bro-ids.org> #548: broctl brokenness multi-ticket -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta -------------------------+-------------------- Changes (by robin): * owner: robin => -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Tue Aug 9 21:22:04 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 9 Aug 2011 21:22:04 -0700 Subject: [Bro-Dev] [Fwd] Re: #548: broctl brokenness multi-ticket Message-ID: <20110810042204.GA66666@icir.org> Fyi, I noticed that Trac was still setting owners automatically for some components, which isn't that useful anymore I think. So I removed that, and also added a new action "unassign" to the modify sectio as I didn't find a way to revert my auto assignemtns. :-) Robin ----- Forwarded message from Bro Tracker ----- Date: Wed, 10 Aug 2011 04:19:28 -0000 From: Bro Tracker Subject: Re: [Bro-Dev] #548: broctl brokenness multi-ticket Message-ID: <061.b935143f8f06a3da7745d56150791550 at tracker.bro-ids.org> Reply-To: bro at tracker.bro-ids.org #548: broctl brokenness multi-ticket -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta -------------------------+-------------------- Changes (by robin): * owner: robin => ----- End forwarded message ----- -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gc355804 at ohio.edu Tue Aug 9 22:03:29 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Wed, 10 Aug 2011 01:03:29 -0400 Subject: [Bro-Dev] #528: Python 3 compatibility In-Reply-To: <062.975b33e70a81d295e0493232bce1ef43@tracker.bro-ids.org> References: <047.5418f300a382e1c09535f6855df979b9@tracker.bro-ids.org> <062.975b33e70a81d295e0493232bce1ef43@tracker.bro-ids.org> Message-ID: <4E421121.7020401@ohio.edu> Should this issue be pushed back to a later release? I think Python 3 is still relatively rare in the wild, and incompatibilities with Python 2 means that there'll be two versions of Python tools to maintain. --Gilbert On 8/10/2011 12:18 AM, Bro Tracker wrote: > #528: Python 3 compatibility > -------------------------+-------------------- > Reporter: robin | Owner: > Type: Task | Status: new > Priority: Normal | Milestone: Bro1.6 > Component: BroControl | Version: > Resolution: | Keywords: > -------------------------+-------------------- > Changes (by robin): > > * owner: robin => > > From bro at tracker.bro-ids.org Tue Aug 9 22:28:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 05:28:23 -0000 Subject: [Bro-Dev] #548: broctl brokenness multi-ticket In-Reply-To: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> References: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> Message-ID: <061.50e0051c8d32b721861ce64833640a15@tracker.bro-ids.org> #548: broctl brokenness multi-ticket -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta -------------------------+-------------------- Description changed by seth: Old description: > My updates to fix bro/broctl integration are in topic/various-updates. > Those updates do not fix these issues however (and any help would be > greatly appreciated).... > > - "check" command is broken. The Bro instance that the check-config > script starts up doesn't die. > > - Potential problem with node naming. Names like this "nids-27A" would > start, but broctl wouldn't know afterward that the instance was running > (even though it was). > > - File rotation needs to be disabled when not running on live traffic > (for the new process command primarily). > > - "scripts" command needs to be fixed. Maybe it should run bro and print > the loaded_scripts.log file to the output? New description: My updates to fix bro/broctl integration are in topic/various-updates. Those updates do not fix these issues however (and any help would be greatly appreciated).... - "check" command is broken. The Bro instance that the check-config script starts up doesn't die. - Potential problem with node naming. Names like this "nids-27A" would start, but broctl wouldn't know afterward that the instance was running (even though it was). - File rotation needs to be disabled when not running on live traffic (for the new process command primarily). - "scripts" command needs to be fixed. Maybe it should run bro and print the loaded_scripts.log file to the output? - Problem with file rotation - There are three different styles of file names being created right now depending on if Bro crashed and was restarted or if files were rotated normally, and another file name in an unknown case. Here are the example files names: - syslog.20:00:00-21:00:00.gz - syslog.2011-08-09-13-00-00.log.12:29:45-13:44:59.gz - syslog.log.12:20:22-12:28:42.gz -- -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Aug 10 07:19:37 2011 From: seth at icir.org (Seth Hall) Date: Wed, 10 Aug 2011 10:19:37 -0400 Subject: [Bro-Dev] #528: Python 3 compatibility In-Reply-To: <4E421121.7020401@ohio.edu> References: <047.5418f300a382e1c09535f6855df979b9@tracker.bro-ids.org> <062.975b33e70a81d295e0493232bce1ef43@tracker.bro-ids.org> <4E421121.7020401@ohio.edu> Message-ID: On Aug 10, 2011, at 1:03 AM, Gilbert Clark wrote: > Should this issue be pushed back to a later release? I think Python 3 > is still relatively rare in the wild, and incompatibilities with Python > 2 means that there'll be two versions of Python tools to maintain. I agree for two reasons. I don't think anyone feels like doing this now and I haven't actually seen python 3 installed anywhere. :) I'll bump the ticket. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Wed Aug 10 07:21:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 14:21:24 -0000 Subject: [Bro-Dev] #528: Python 3 compatibility In-Reply-To: <047.5418f300a382e1c09535f6855df979b9@tracker.bro-ids.org> References: <047.5418f300a382e1c09535f6855df979b9@tracker.bro-ids.org> Message-ID: <062.d58988a202059fb457e40db846f1c273@tracker.bro-ids.org> #528: Python 3 compatibility -------------------------+----------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Component: BroControl | Version: Resolution: | Keywords: -------------------------+----------------- Changes (by seth): * milestone: Bro1.6 => Comment: From Gilbert: > Should this issue be pushed back to a later release? I think Python 3 is still relatively rare in the wild, and incompatibilities with Python 2 means that there'll be two versions of Python tools to maintain. I agree. I'm just going to remove any milestone from the ticket since this doesn't provide us much until python 3 starts showing up in the wild. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 10 07:23:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 14:23:24 -0000 Subject: [Bro-Dev] #474: &raw_output turns null values into \0 In-Reply-To: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> References: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> Message-ID: <061.b6c18208f1ef35cdc4e93499418b0b85@tracker.bro-ids.org> #474: &raw_output turns null values into \0 ----------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------+------------------------ Comment (by seth): Jon, I think that the updates for the other file attributes can wait until later since none of the shipped scripts do logs directly with print anymore. What do you think? We can just bump this ticket off the milestone and save it as a clean up task for later. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 10 07:42:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 14:42:58 -0000 Subject: [Bro-Dev] #474: &raw_output turns null values into \0 In-Reply-To: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> References: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> Message-ID: <061.7f11e3f90b60bee9b7d6ca7570ce1f65@tracker.bro-ids.org> #474: &raw_output turns null values into \0 ----------------------+------------------------ Reporter: seth | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------+------------------------ Changes (by jsiwek): * milestone: Bro1.6 => Bro1.7 Comment: Replying to [comment:9 seth]: > Jon, I think that the updates for the other file attributes can wait until later since none of the shipped scripts do logs directly with print anymore. Ok. As a reminder for when we figure out what to do with file attributes: the current `language/raw_output_attr` unit test should replace use of the `enable_raw_output` BIF with `&raw_output` and also needs to check application of the attribute to global variables since they get handled slightly differently than locals. (There's already a test for the BIF in `bifs/enable_raw_output`) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 10 10:39:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 17:39:14 -0000 Subject: [Bro-Dev] #509: SMTP rework In-Reply-To: <046.35777c113d149585390ec44632cce689@tracker.bro-ids.org> References: <046.35777c113d149585390ec44632cce689@tracker.bro-ids.org> Message-ID: <061.3c92dfeeb22fbdcbcc67f6757b502331@tracker.bro-ids.org> #509: SMTP rework ---------------------+------------------------ Reporter: seth | Owner: seth Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ---------------------+------------------------ Comment (by jsiwek): In [17d74c23dbb60bcf2917d3f1b8ce2d8dc46adbd2/bro]: {{{ #!CommitTicketReference repository="bro" revision="17d74c23dbb60bcf2917d3f1b8ce2d8dc46adbd2" SMTP script refactor. (addresses #509) - message header state tracking is now done by handling mime_one_header instead of parsing the data in the smtp_data event - changed the logging point to be when an smtp_reply is seen in response to the end of a DATA section - the smtp package now uses it's own mime script and logging stream for logging entities, extraction, etc. - fixes for mime file extraction: now logs the extracted file name, and the count of extracted files needed to be maintained in the State record }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 10 10:39:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 17:39:14 -0000 Subject: [Bro-Dev] #509: SMTP rework In-Reply-To: <046.35777c113d149585390ec44632cce689@tracker.bro-ids.org> References: <046.35777c113d149585390ec44632cce689@tracker.bro-ids.org> Message-ID: <061.27676de0f29088a29607994b2805aca9@tracker.bro-ids.org> #509: SMTP rework ---------------------+------------------------ Reporter: seth | Owner: seth Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ---------------------+------------------------ Comment (by seth): In [adc486c673d11f46caaec27da5c731aaaa3996ec/bro]: {{{ #!CommitTicketReference repository="bro" revision="adc486c673d11f46caaec27da5c731aaaa3996ec" Merge remote-tracking branch 'origin/topic/jsiwek/smtp-refactor' - While updating, I did some further work on the branch. - New function in the base/utils/files for extracting filenames from content-dispositions. - New script for entity excerpt extraction if you aren't interested in full extraction. The data goes a log field too. - Some renaming and reorganization of types. - Updated tests to work with new code. * origin/topic/jsiwek/smtp-refactor: Make the doc.coverage test happy. SMTP script refactor. (addresses #509) Conflicts: doc/scripts/DocSourcesList.cmake policy/protocols/smtp/__load__.bro policy/protocols/smtp/base/__load__.bro }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 10 10:43:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 17:43:16 -0000 Subject: [Bro-Dev] #509: SMTP rework In-Reply-To: <046.35777c113d149585390ec44632cce689@tracker.bro-ids.org> References: <046.35777c113d149585390ec44632cce689@tracker.bro-ids.org> Message-ID: <061.b67f0c84e0f3b111a8abbef3d2a4eb5b@tracker.bro-ids.org> #509: SMTP rework -----------------------------+------------------------ Reporter: seth | Owner: seth Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: preview -----------------------------+------------------------ Changes (by seth): * status: assigned => closed * resolution: => Solved/Applied Comment: This is finished, thanks Jon! -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 10 10:47:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 17:47:12 -0000 Subject: [Bro-Dev] #550: libmagic not found on RHELS 6.1 Message-ID: <046.6e538485994b3809dd5eb6e3495872aa@tracker.bro-ids.org> #550: libmagic not found on RHELS 6.1 ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: beta | ---------------------+-------------------- magic.h from the "file-libs" package is installed in /usr/include/linux/magic.h which is a pain to say the least. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 10 10:56:10 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 17:56:10 -0000 Subject: [Bro-Dev] #550: libmagic not found on RHELS 6.1 In-Reply-To: <046.6e538485994b3809dd5eb6e3495872aa@tracker.bro-ids.org> References: <046.6e538485994b3809dd5eb6e3495872aa@tracker.bro-ids.org> Message-ID: <061.f2a2869fde53468c765a8202402b6bac@tracker.bro-ids.org> #550: libmagic not found on RHELS 6.1 ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by jsiwek): Replying to [ticket:550 seth]: > magic.h from the "file-libs" package is installed in /usr/include/linux/magic.h which is a pain to say the least. Is there a "file-devel" package? -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Wed Aug 10 11:17:58 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 10 Aug 2011 11:17:58 -0700 Subject: [Bro-Dev] Autodoc: how to link to another script? Message-ID: <4E42CB56.5070609@icir.org> Hi, what's the best way to link from one script to another? I know I can link to identifiers and types, but linking to a full script would be nice..... Or maybe a more general question for best practices: In my tunnel branch I've added an optional column to conn.log showing to tunnel information. In the autodoc for this column it would be nice to link to the redef'able consts that control tunnel decapsulation. My idea was to link to the script that redef's these const to enable the tunnel decapsulation. But it also be nice (or better) to be able to link to where the consts directly as a "block" instead of linking to them individually. They are in bro.init in a module called Tunnel. cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From seth at icir.org Wed Aug 10 11:22:53 2011 From: seth at icir.org (Seth Hall) Date: Wed, 10 Aug 2011 14:22:53 -0400 Subject: [Bro-Dev] Autodoc: how to link to another script? In-Reply-To: <4E42CB56.5070609@icir.org> References: <4E42CB56.5070609@icir.org> Message-ID: <6C784B54-A8A8-4BC2-8F7E-58C835ECF9D5@icir.org> On Aug 10, 2011, at 2:17 PM, Gregor Maier wrote: > Or maybe a more general question for best practices: In my tunnel branch > I've added an optional column to conn.log showing to tunnel information. Where have you done this extension? One of the rules of thumb I've been trying to follow is that frameworks don't depend on protocols. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Wed Aug 10 11:36:31 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 10 Aug 2011 13:36:31 -0500 Subject: [Bro-Dev] Autodoc: how to link to another script? In-Reply-To: <4E42CB56.5070609@icir.org> References: <4E42CB56.5070609@icir.org> Message-ID: <6689D3AF-735C-479B-834C-D72EF8C60384@ncsa.illinois.edu> > what's the best way to link from one script to another? The :doc: role: http://sphinx.pocoo.org/markup/inline.html That can be embedded in any of the doc-style comments: e.g. This renders a link (in the summary section) to the documentation for conn's main.bro: ##! :doc:`/scripts/base/protocols/conn/main` And this would render a link to an index of all the script docs in that base/protocols/conn directory: ##! :doc:`/scripts/base/protocols/conn/index` - Jon From gregor at icir.org Wed Aug 10 11:41:00 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 10 Aug 2011 11:41:00 -0700 Subject: [Bro-Dev] Autodoc: how to link to another script? In-Reply-To: <6C784B54-A8A8-4BC2-8F7E-58C835ECF9D5@icir.org> References: <4E42CB56.5070609@icir.org> <6C784B54-A8A8-4BC2-8F7E-58C835ECF9D5@icir.org> Message-ID: <4E42D0BC.70207@icir.org> On 8/10/11 11:22 , Seth Hall wrote: > > On Aug 10, 2011, at 2:17 PM, Gregor Maier wrote: > >> Or maybe a more general question for best practices: In my tunnel branch >> I've added an optional column to conn.log showing to tunnel information. > > Where have you done this extension? One of the rules of thumb I've been trying to follow is that frameworks don't depend on protocols. > The connection record type has an optional field tunnel_parent. I actually added that in bro.init, since I forgot that we can now extend records with +=. from bro.init: ----8<------ module Tunnel; export { ## Records the identity of a the parent of a tunneled connection. type parent_t: record { ## The 4-tuple of the tunnel "connection". cid: conn_id; ## The type of tunnel. tunnel_type: tunneltype_t; } &log; } # end export module GLOBAL; type connection: record { [the other fields] tunnel_parent: Tunnel::parent_t &optional; }; ----8<------ I could actually add this tunnel_parent field in my tunnel.bro script that logs child-conn-id <--> parent_t However, today I thought it might be handy to at least add the tunneltype to conn.log as and indicator whether a particular connection was tunneled or not. That's why I added a "tunnel_type" field to the Conn::Info record (in conn.bro). So, while I directly modified the conn/main.bro script, it doesn't depend on the code in the framework part. Not sure what you mean by "depend" though. I guess I could also try to extend Conn::Info in tunnel.bro, right. However, if multiple scripts to this, then the order of columns in conn.log would depend on the order in which these scripts are loaded..... Thinking of this some more, I think that the http-scripts already do this, so the order of columns isn't well-defined anyways, right? And if somebody writes a script to parse a bro log file, then one has to check the header, right? cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Wed Aug 10 11:46:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 18:46:50 -0000 Subject: [Bro-Dev] #550: libmagic not found on RHELS 6.1 In-Reply-To: <046.6e538485994b3809dd5eb6e3495872aa@tracker.bro-ids.org> References: <046.6e538485994b3809dd5eb6e3495872aa@tracker.bro-ids.org> Message-ID: <061.b9fff60ac9d4c0662310a067e989f447@tracker.bro-ids.org> #550: libmagic not found on RHELS 6.1 ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by seth): > Is there a "file-devel" package? Nope. I think Debian is the only distro that makes that distinction. magic.h is definitely there, just in a weird place. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 10 12:16:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 19:16:54 -0000 Subject: [Bro-Dev] #550: libmagic not found on RHELS 6.1 In-Reply-To: <046.6e538485994b3809dd5eb6e3495872aa@tracker.bro-ids.org> References: <046.6e538485994b3809dd5eb6e3495872aa@tracker.bro-ids.org> Message-ID: <061.4f06b9e0c0e4b4f23511faba92a57f40@tracker.bro-ids.org> #550: libmagic not found on RHELS 6.1 ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Invalid | Keywords: beta ----------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Invalid Comment: Ugh... I was wrong in so many ways here. I did need to install file-devel and /usr/include/linux/magic.h was something related to the kernel. Cheerfully closing this ticket. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Aug 10 12:31:16 2011 From: seth at icir.org (Seth Hall) Date: Wed, 10 Aug 2011 15:31:16 -0400 Subject: [Bro-Dev] Autodoc: how to link to another script? In-Reply-To: <4E42D0BC.70207@icir.org> References: <4E42CB56.5070609@icir.org> <6C784B54-A8A8-4BC2-8F7E-58C835ECF9D5@icir.org> <4E42D0BC.70207@icir.org> Message-ID: On Aug 10, 2011, at 2:41 PM, Gregor Maier wrote: > The connection record type has an optional field tunnel_parent. I actually added that in bro.init, since I forgot that we can now extend records with +=. Haha, get that out of there! :P > type parent_t: record { You may want to check out: http://www.bro-ids.org/development/script-conventions.html It certainly makes the code easier to read if everyone names things consistently. > I could actually add this tunnel_parent field in my tunnel.bro script that logs child-conn-id <--> parent_t I'm actually starting to wonder if tunnel.bro should go in base/protocols/conn/. That actually seems like the appropriate place since it has to do with connections. It's just using extra core support to find and log these tunnels. I would still extend the Conn::Info type in the tunnels.bro script though. What do you think? > However, today I thought it might be handy to at least add the tunneltype to conn.log as and indicator whether a particular connection was tunneled or not. That's why I added a "tunnel_type" field to the Conn::Info record (in conn.bro). > So, while I directly modified the conn/main.bro script, it doesn't depend on the code in the framework part. Not sure what you mean by > "depend" though. Nevermind. I was sort of still thinking along the lines of a tunnel framework or something. I just don't want to see frameworks @load-ing anything out of protocols/. If integration needs to happen I think it should happen the other way (the protocol loading the framework and pulling data from it or something). It irrelevant if you put the tunnels.bro file in base/protocols/conn/. > However, if multiple scripts to this, then the order of columns in conn.log would depend on the order in which these scripts are loaded..... Thinking of this some more, I think that the http-scripts already do this, so the order of columns isn't well-defined anyways, right? And if somebody writes a script to parse a bro log file, then one has to check the header, right? You've got it. I'm hoping to get everyone away from the notion of column numbers even. Once we get binary logging added it will really be inconsequential because you will essentially load a log (or logs) and request specifically named fields from the log since the binary log doesn't have a notion of column ordering anyway. For the ascii logs, looking at the headers certainly works though. In most cases if people just use the default loaded scripts as-is we should maintain pretty steady column ordering for most columns. If you can tell, I'm tired of file format parsing ever getting in the way of doing actual analysis. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Wed Aug 10 12:55:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 10 Aug 2011 19:55:44 -0000 Subject: [Bro-Dev] #551: Potential alternate signature loading method? Message-ID: <046.8fa46d7f399c279d6a4c4e9463558fac@tracker.bro-ids.org> #551: Potential alternate signature loading method? -----------------------------+----------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Component: Bro | Version: Keywords: | -----------------------------+----------------- The way signature file loading happens is annoying and inconsistent. At some point it would be nice to be able to do this: {{{ @load-sigs mysigs # or... @load-sigs mysigs.sig # and even better (since the lack of relative loading for sigs is now annoying)... @load-sigs ./mysigs }}} This is opposed to the current method of: {{{ redef signature_files += "mysigs.sig"; }}} In this case mysigs.sig would have to be located in the BROPATH somewhere which breaks the "module" model of being able to distribute small chunks of functionality independently of where it's placed in the BROPATH. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Wed Aug 10 13:13:06 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 10 Aug 2011 13:13:06 -0700 Subject: [Bro-Dev] Autodoc: how to link to another script? In-Reply-To: References: <4E42CB56.5070609@icir.org> <6C784B54-A8A8-4BC2-8F7E-58C835ECF9D5@icir.org> <4E42D0BC.70207@icir.org> Message-ID: <4E42E652.705@icir.org> > You may want to check out: http://www.bro-ids.org/development/script-conventions.html ooops. will do. >> I could actually add this tunnel_parent field in my tunnel.bro script that logs child-conn-id<--> parent_t > > I'm actually starting to wonder if tunnel.bro should go in base/protocols/conn/. That actually seems like the appropriate place since it has to do with connections. It's just using extra core support to find and log these tunnels. I would still extend the Conn::Info type in the tunnels.bro script though. What do you think? sounds good to me. However, I wouldn't put it in base. I think the default should be to not decapsulate tunnels! [snip] > You've got it. I'm hoping to get everyone away from the notion of column numbers even. Once we get binary logging added it will really be inconsequential because you will essentially load a log (or logs) and request specifically named fields from the log since the binary log doesn't have a notion of column ordering anyway. For the ascii logs, looking at the headers certainly works though. Yeah. People using awk will hate that.... (But I'm using mostly python these days anyways) > In most cases if people just use the default loaded scripts as-is we should maintain pretty steady column ordering for most columns. That's what's worrying me: people assuming a fixed ordering when writing analysis scripts. We should probably just mention this somewhere in the Getting Starting and From 1.5 to 2.0 HowTo. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From seth at icir.org Wed Aug 10 13:22:45 2011 From: seth at icir.org (Seth Hall) Date: Wed, 10 Aug 2011 16:22:45 -0400 Subject: [Bro-Dev] Autodoc: how to link to another script? In-Reply-To: <4E42E652.705@icir.org> References: <4E42CB56.5070609@icir.org> <6C784B54-A8A8-4BC2-8F7E-58C835ECF9D5@icir.org> <4E42D0BC.70207@icir.org> <4E42E652.705@icir.org> Message-ID: On Aug 10, 2011, at 4:13 PM, Gregor Maier wrote: > sounds good to me. However, I wouldn't put it in base. I think the default should be to not decapsulate tunnels! I agree. I think we should have a configuration variable to enable it, but the support for *how* it's actually accomplished and logged seems like something that should be in the base. > Yeah. People using awk will hate that.... > (But I'm using mostly python these days anyways) Actually they won't! I'm not sure how it will look in the end, but it's going to be something like this (Gilbert can give more and better detail): ds2txt -s $'\t' -f host,uri,referrer http.ds.* | awk -F $'\t' '{if ($1 == "www.google.com") print}' You get to dynamically create your own column ordering which will stay consistent since you're defining it at search time. The awk use-case is one that I'm trying to make sure is really nice because I use awk for a lot of stuff too. :) > That's what's worrying me: people assuming a fixed ordering when writing analysis scripts. We should probably just mention this somewhere in the Getting Starting and From 1.5 to 2.0 HowTo. Good point. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Wed Aug 10 13:29:10 2011 From: seth at icir.org (Seth Hall) Date: Wed, 10 Aug 2011 16:29:10 -0400 Subject: [Bro-Dev] Autodoc: how to link to another script? In-Reply-To: References: <4E42CB56.5070609@icir.org> <6C784B54-A8A8-4BC2-8F7E-58C835ECF9D5@icir.org> <4E42D0BC.70207@icir.org> <4E42E652.705@icir.org> Message-ID: <62FAC82B-0E5D-4A15-8A73-5BA10BCD1430@icir.org> On Aug 10, 2011, at 4:22 PM, Seth Hall wrote: > On Aug 10, 2011, at 4:13 PM, Gregor Maier wrote: > >> sounds good to me. However, I wouldn't put it in base. I think the default should be to not decapsulate tunnels! > > I agree. I think we should have a configuration variable to enable it, but the support for *how* it's actually accomplished and logged seems like something that should be in the base. Continuing this thought... outside of base/ (in policy/protocols/conn) it might make sense to do things that actually "detect" something. I consider non-obfuscated tunnel decapsulation very similarly to normal protocol analysis. The rule of thumb is that the scripts in base/ should only be doing protocol logging and state building which is exactly what it sounds like your tunnel.bro script is doing. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gregor at icir.org Wed Aug 10 13:40:16 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 10 Aug 2011 13:40:16 -0700 Subject: [Bro-Dev] Autodoc: how to link to another script? In-Reply-To: <62FAC82B-0E5D-4A15-8A73-5BA10BCD1430@icir.org> References: <4E42CB56.5070609@icir.org> <6C784B54-A8A8-4BC2-8F7E-58C835ECF9D5@icir.org> <4E42D0BC.70207@icir.org> <4E42E652.705@icir.org> <62FAC82B-0E5D-4A15-8A73-5BA10BCD1430@icir.org> Message-ID: <4E42ECB0.7080507@icir.org> On 8/10/11 13:29 , Seth Hall wrote: > > On Aug 10, 2011, at 4:22 PM, Seth Hall wrote: > >> On Aug 10, 2011, at 4:13 PM, Gregor Maier wrote: >> >>> sounds good to me. However, I wouldn't put it in base. I think the default should be to not decapsulate tunnels! >> >> I agree. I think we should have a configuration variable to enable it, but the support for *how* it's actually accomplished and logged seems like something that should be in the base. > > > Continuing this thought... outside of base/ (in policy/protocols/conn) it might make sense to do things that actually "detect" something. I consider non-obfuscated tunnel decapsulation very similarly to normal protocol analysis. The rule of thumb is that the scripts in base/ should only be doing protocol logging and state building which is exactly what it sounds like your tunnel.bro script is doing. :) Well it depends. The script does two--three things: 1) enable tunnel decapsulation by redef'ing the appropriate consts 2) create a tunnel.log file that logs all tunneled connections (c$id, c$uid) and the parent connection. 3) provide a single point were the tunnel stuff is documented (what it does, how to tune it, its limitations). (I love the new autodoc features!!) (1) and (3) are kinda related. I always found it very hard to know and understand what all the 100's of redef'able consts in bro.init did. I think doing it this way is nice way of putting the documentation together and giving users and easy way to access the functionality (load the tunnel script, look at it's documentation for details) We can probably split it up and put (1) in policy/ and (2) in base/. However, (2) only works if the connection_compressor is disabled (otherwise the identity of the tunnel is lost), so this makes it more problematic to put it in base (at least while the connection_compressor remains on by default) I've attached the current version. Might be easier to just look at it than explaining it via email ;-) cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: tunnel.bro Url: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110810/30dae8a7/attachment.ksh From seth at icir.org Wed Aug 10 13:56:11 2011 From: seth at icir.org (Seth Hall) Date: Wed, 10 Aug 2011 16:56:11 -0400 Subject: [Bro-Dev] Autodoc: how to link to another script? In-Reply-To: <4E42ECB0.7080507@icir.org> References: <4E42CB56.5070609@icir.org> <6C784B54-A8A8-4BC2-8F7E-58C835ECF9D5@icir.org> <4E42D0BC.70207@icir.org> <4E42E652.705@icir.org> <62FAC82B-0E5D-4A15-8A73-5BA10BCD1430@icir.org> <4E42ECB0.7080507@icir.org> Message-ID: On Aug 10, 2011, at 4:40 PM, Gregor Maier wrote: > (1) and (3) are kinda related. I always found it very hard to know and understand what all the 100's of redef'able consts in bro.init did. I think doing it this way is nice way of putting the documentation together and giving users and easy way to access the functionality (load the tunnel script, look at it's documentation for details) > > We can probably split it up and put (1) in policy/ and (2) in base/. However, (2) only works if the connection_compressor is disabled (otherwise the identity of the tunnel is lost), so this makes it more problematic to put it in base (at least while the connection_compressor remains on by default) Ah, ok. That makes sense. That's probably the right way to do it. Make it disabled by default (with scripts using it loaded in base) and create a file named policy/protocols/conn/decapsulate-tunnels.bro (or something like that). In the decapsulate-tunnels.bro script, you could document that it relies on the connection compressor being disabled. If people don't read the docs for scripts before @load-ing them it's hard to blame the script author. :) > I've attached the current version. Might be easier to just look at it than explaining it via email ;-) Cool, thanks. I'll take a look later. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gregor at icir.org Wed Aug 10 14:25:52 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 10 Aug 2011 14:25:52 -0700 Subject: [Bro-Dev] Autodoc: how to link to another script? In-Reply-To: References: <4E42CB56.5070609@icir.org> <6C784B54-A8A8-4BC2-8F7E-58C835ECF9D5@icir.org> <4E42D0BC.70207@icir.org> <4E42E652.705@icir.org> <62FAC82B-0E5D-4A15-8A73-5BA10BCD1430@icir.org> <4E42ECB0.7080507@icir.org> Message-ID: <4E42F760.1010807@icir.org> On 8/10/11 13:56 , Seth Hall wrote: > > On Aug 10, 2011, at 4:40 PM, Gregor Maier wrote: > >> (1) and (3) are kinda related. I always found it very hard to know and understand what all the 100's of redef'able consts in bro.init did. I think doing it this way is nice way of putting the documentation together and giving users and easy way to access the functionality (load the tunnel script, look at it's documentation for details) >> >> We can probably split it up and put (1) in policy/ and (2) in base/. However, (2) only works if the connection_compressor is disabled (otherwise the identity of the tunnel is lost), so this makes it more problematic to put it in base (at least while the connection_compressor remains on by default) > > Ah, ok. That makes sense. That's probably the right way to do it. Make it disabled by default (with scripts using it loaded in base) and create a file named policy/protocols/conn/decapsulate-tunnels.bro (or something like that). In the decapsulate-tunnels.bro script, you could document that it relies on the connection compressor being disabled. If people don't read the docs for scripts before @load-ing them it's hard to blame the script author. :) I was just about to implement it this way, when the following occurred to me: If I split it up, then conn.log will always contain a column "tunneltype" (since (2) is in base) even if the tunnel decapsulation isn't enabled. This might be counter intuitive for users, since the presence of the column would suggest that something with tunnels is happening (esp. now that we have extendable log files) (Just for the tunnels this whole discussion is definitely overblown, but IMHO we'll face such questions more often further down the road, so I think we should figure out what the best way to do it is) cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From gregor at icir.org Wed Aug 10 15:17:18 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 10 Aug 2011 15:17:18 -0700 Subject: [Bro-Dev] Document final "state" of redef'ed records Message-ID: <4E43036E.8060408@icir.org> Hi, given a bunch of scripts, that extend record types (I mostly thinking about the Info records used for logging) if there a way to get a autodoc generated file that shows the final version of the record? The idea would be, that this "final version" is exactly what actually gets logged, so it would be a nice-to-have for users who want to understand all the columns in a log file. Now, they have to know all the scripts that are loaded that extend a particular record and look at their documentation to find out the meaning of columns / fields. This is fine for scripts in policy, but for the base scripts it would be nice to have a single place to go too. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From jsiwek at ncsa.illinois.edu Wed Aug 10 16:12:40 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 10 Aug 2011 18:12:40 -0500 Subject: [Bro-Dev] Document final "state" of redef'ed records In-Reply-To: <4E43036E.8060408@icir.org> References: <4E43036E.8060408@icir.org> Message-ID: <7B7FC268-0850-43AF-83EC-27FBA1FB41FA@ncsa.illinois.edu> > given a bunch of scripts, that extend record types (I mostly thinking > about the Info records used for logging) if there a way to get a autodoc > generated file that shows the final version of the record? I don't think there's a way to do that; everything tries to auto-document from the point-of-view of what effects a script has when loaded by itself. > The idea would be, that this "final version" is exactly what actually > gets logged, so it would be a nice-to-have for users who want to > understand all the columns in a log file. Yeah, I can see that being useful. You'll have to file a feature request, though; I'm not coming up with an easy method to auto collect that info off the top of my head, I'd need to think about it. - Jon From seth at icir.org Wed Aug 10 18:38:41 2011 From: seth at icir.org (Seth Hall) Date: Wed, 10 Aug 2011 21:38:41 -0400 Subject: [Bro-Dev] Autodoc: how to link to another script? In-Reply-To: <4E42F760.1010807@icir.org> References: <4E42CB56.5070609@icir.org> <6C784B54-A8A8-4BC2-8F7E-58C835ECF9D5@icir.org> <4E42D0BC.70207@icir.org> <4E42E652.705@icir.org> <62FAC82B-0E5D-4A15-8A73-5BA10BCD1430@icir.org> <4E42ECB0.7080507@icir.org> <4E42F760.1010807@icir.org> Message-ID: On Aug 10, 2011, at 5:25 PM, Gregor Maier wrote: > I was just about to implement it this way, when the following occurred to me: If I split it up, then conn.log will always contain a column "tunneltype" (since (2) is in base) even if the tunnel decapsulation isn't enabled. This might be counter intuitive for users, since the presence of the column would suggest that something with tunnels is happening (esp. now that we have extendable log files) You're right. I tend to go through the same flailing back and forth when making little decisions like this too. :) > (Just for the tunnels this whole discussion is definitely overblown, but IMHO we'll face such questions more often further down the road, so I think we should figure out what the best way to do it is) Yep, completely agreed. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Wed Aug 10 20:59:40 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 10 Aug 2011 20:59:40 -0700 Subject: [Bro-Dev] Autodoc: how to link to another script? In-Reply-To: <4E42E652.705@icir.org> References: <4E42CB56.5070609@icir.org> <6C784B54-A8A8-4BC2-8F7E-58C835ECF9D5@icir.org> <4E42D0BC.70207@icir.org> <4E42E652.705@icir.org> Message-ID: <20110811035940.GC1694@icir.org> On Wed, Aug 10, 2011 at 13:13 -0700, you wrote: > Yeah. People using awk will hate that.... Actually it's not really difficult in awk to use the header as index (perhaps a bit slower, but not sure how much). And Gilbert is working on a Python interface to logs. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From hlin33 at illinois.edu Wed Aug 10 22:28:13 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Wed, 10 Aug 2011 22:28:13 -0700 Subject: [Bro-Dev] How to Print string in Bro Message-ID: Hi, I am writing a bro event handler with one argument as "string" type. The string is actually treated as a byte stream but not ended with '\0' in Bro. I am wondering is there any way to print the hexadecimal value of each byte in this string (including how to index each byte within the string). I know that %s can print string, but it prints out the ASCII code, which sometimes is hard to decode. Best, -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110810/fc38e26d/attachment.html From seth at icir.org Wed Aug 10 22:41:32 2011 From: seth at icir.org (Seth Hall) Date: Thu, 11 Aug 2011 01:41:32 -0400 Subject: [Bro-Dev] How to Print string in Bro In-Reply-To: References: Message-ID: <65122355-2BF6-4286-8E9E-2BA05BF89AC9@icir.org> On Aug 11, 2011, at 1:28 AM, Hui Lin wrote: > I am writing a bro event handler with one argument as "string" type. The string is actually treated as a byte stream but not ended with '\0' in Bro. > I am wondering is there any way to print the hexadecimal value of each byte in this string (including how to index each byte within the string). I know that %s can print string, but it prints out the ASCII code, which sometimes is hard to decode. You have a couple of choices with this right now. local test_string = "this is\x00a test"; print string_to_ascii_hex(test_string); => 7468697320697300612074657374 print hexdump(test_string); => 0000 74 68 69 73 20 69 73 00 61 20 74 65 73 74 this is. a test^J Would either of those work for you? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Thu Aug 11 07:46:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 14:46:29 -0000 Subject: [Bro-Dev] #518: NMI: Use consistent version of CMake In-Reply-To: <050.2fd3ee8f00f4300b2ccd6054beddb68b@tracker.bro-ids.org> References: <050.2fd3ee8f00f4300b2ccd6054beddb68b@tracker.bro-ids.org> Message-ID: <065.f075bd7c6c3b7f1fec38f5016c403df4@tracker.bro-ids.org> #518: NMI: Use consistent version of CMake -----------------------------+----------------------------- Reporter: appleman | Owner: Type: Task | Status: closed Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: nmi build cmake -----------------------------+----------------------------- Changes (by appleman): * status: new => closed * resolution: => Solved/Applied Comment: Updated to use cmake-2.8.5 across all platforms except on x86_64_macos_10.5-updated where I use cmake-2.6.4 to confirm backward compatibility -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 07:59:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 14:59:58 -0000 Subject: [Bro-Dev] #518: NMI: Use consistent version of CMake In-Reply-To: <050.2fd3ee8f00f4300b2ccd6054beddb68b@tracker.bro-ids.org> References: <050.2fd3ee8f00f4300b2ccd6054beddb68b@tracker.bro-ids.org> Message-ID: <065.a0e1b769381b55c1402eb37653927434@tracker.bro-ids.org> #518: NMI: Use consistent version of CMake -----------------------+----------------------------- Reporter: appleman | Owner: Type: Task | Status: reopened Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: nmi build cmake -----------------------+----------------------------- Changes (by jsiwek): * status: closed => reopened * resolution: Solved/Applied => Comment: The scripts that do binary packaging currently require CMake 2.8.4 (due to particular bugs in past versions that break packages), I'll manually "vet" that 2.8.5 still works and then change the requirement. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 08:00:10 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 15:00:10 -0000 Subject: [Bro-Dev] #518: NMI: Use consistent version of CMake In-Reply-To: <050.2fd3ee8f00f4300b2ccd6054beddb68b@tracker.bro-ids.org> References: <050.2fd3ee8f00f4300b2ccd6054beddb68b@tracker.bro-ids.org> Message-ID: <065.e8bda802f541aacc3c61cd66709de143@tracker.bro-ids.org> #518: NMI: Use consistent version of CMake -----------------------+----------------------------- Reporter: appleman | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: nmi build cmake -----------------------+----------------------------- Changes (by jsiwek): * owner: => jsiwek * status: reopened => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Aug 11 08:22:52 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 11 Aug 2011 08:22:52 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Updates for SSH scripts. (9c2273b) In-Reply-To: <201108110544.p7B5i3Jm002262@bro-ids.icir.org> References: <201108110544.p7B5i3Jm002262@bro-ids.icir.org> Message-ID: <4E43F3CC.1030202@icir.org> > +##! Requires that :bro:id:`use_conn_size_analyzer` is set to T! The heuristic > +##! is not attempted if the connection size analyzer isn't enabled. > I looks like the code doesn't check yet, whether use_conn_size_analyzer is indeed on (or better, it should probably check c$resp?$num_ip_bytes, since num_ip_bytes is optional. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From seth at icir.org Thu Aug 11 08:39:28 2011 From: seth at icir.org (Seth Hall) Date: Thu, 11 Aug 2011 11:39:28 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Updates for SSH scripts. (9c2273b) In-Reply-To: <4E43F3CC.1030202@icir.org> References: <201108110544.p7B5i3Jm002262@bro-ids.icir.org> <4E43F3CC.1030202@icir.org> Message-ID: <7D718519-74EB-4C42-A183-3A4C44F9A8F5@icir.org> On Aug 11, 2011, at 11:22 AM, Gregor Maier wrote: >> +##! Requires that :bro:id:`use_conn_size_analyzer` is set to T! The heuristic >> +##! is not attempted if the connection size analyzer isn't enabled. > > I looks like the code doesn't check yet, whether use_conn_size_analyzer is indeed on (or better, it should probably check > c$resp?$num_ip_bytes, since num_ip_bytes is optional. It actually does that check further down in the file when the ssh_watcher event is initially scheduled. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gregor at icir.org Thu Aug 11 08:50:46 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 11 Aug 2011 08:50:46 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Updates for SSH scripts. (9c2273b) In-Reply-To: <7D718519-74EB-4C42-A183-3A4C44F9A8F5@icir.org> References: <201108110544.p7B5i3Jm002262@bro-ids.icir.org> <4E43F3CC.1030202@icir.org> <7D718519-74EB-4C42-A183-3A4C44F9A8F5@icir.org> Message-ID: <4E43FA56.9010403@icir.org> On 8/11/11 8:39 , Seth Hall wrote: > > On Aug 11, 2011, at 11:22 AM, Gregor Maier wrote: > >>> +##! Requires that :bro:id:`use_conn_size_analyzer` is set to T! The heuristic >>> +##! is not attempted if the connection size analyzer isn't enabled. >> >> I looks like the code doesn't check yet, whether use_conn_size_analyzer is indeed on (or better, it should probably check >> c$resp?$num_ip_bytes, since num_ip_bytes is optional. > > > It actually does that check further down in the file when the ssh_watcher event is initially scheduled. :) But you call check_ssh_connection() form connection_state_remove() as well and then you don't check. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Thu Aug 11 09:26:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 16:26:56 -0000 Subject: [Bro-Dev] #516: NMI: Output formatting issues in the istate.pybroccoli unit test (Fedora 14) In-Reply-To: <050.8a31ed622b642173bdfa95bbe6b94396@tracker.bro-ids.org> References: <050.8a31ed622b642173bdfa95bbe6b94396@tracker.bro-ids.org> Message-ID: <065.a150b5393f66203cce64f233d4af0fea@tracker.bro-ids.org> #516: NMI: Output formatting issues in the istate.pybroccoli unit test (Fedora 14) ------------------------------+-------------------------------------- Reporter: appleman | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: broccoli-python | Version: git/master Resolution: Works for Me | Keywords: nmi btest testing fedora ------------------------------+-------------------------------------- Changes (by appleman): * status: new => closed * resolution: => Works for Me Comment: It appears that this test is now passing, so Jon must be correct that it was fixed by #507. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 09:51:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 16:51:45 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.a64ef670a5206864f748bc9505f8c323@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): In [7b07a19cd6a0267ce889b22646ce013848fa8528/bro]: {{{ #!CommitTicketReference repository="bro" revision="7b07a19cd6a0267ce889b22646ce013848fa8528" bro scripts generated from bifs now install to $prefix/share/bro/base/ Addresses #545 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 09:51:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 16:51:45 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.5e3d098158d716ac1ba56bf5155817ea@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): In [c3fb0ea03567c7b82fb008fb42be2206eb1ed657/bro]: {{{ #!CommitTicketReference repository="bro" revision="c3fb0ea03567c7b82fb008fb42be2206eb1ed657" Reorganizing btest/policy directory to match new scripts/ organization Addresses #545 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 10:07:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 17:07:12 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.1e1a911c3f750d6bf24c06b6c766caa9@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): Most of this ticket is done in `topic/jsiwek/reorg-followup` in the `bro` and `bro-aux` repos, still left: * I'm waiting on #543 to merge so I can enable bare-mode for `make restdoc` and check everything still works. * I think the bare-mode-coverage.test will depend on `btest` commit 9141140, so might wait on that before requesting a merge of this * The init-default-coverage test could use Seth's touch to make it succeed -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 11:07:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 18:07:13 -0000 Subject: [Bro-Dev] #552: ASCII Log Writer precision of interval values Message-ID: <048.e754f597cc42f38ad361859f8fbad41c@tracker.bro-ids.org> #552: ASCII Log Writer precision of interval values -----------------------------+------------------------ Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | -----------------------------+------------------------ The ASCII Log Writer prints time interval values with full precision. However, since all our input time stamps are only 1us precision (and the additional precision is due to floating point rounding). I think we should hard code the precision to 6 digits, same as we do for time values (see #508) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 11:20:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 18:20:13 -0000 Subject: [Bro-Dev] #460: Problem with extended records and empty container types In-Reply-To: <046.e32eb4f1b2af66ecfcc23a676b3b6332@tracker.bro-ids.org> References: <046.e32eb4f1b2af66ecfcc23a676b3b6332@tracker.bro-ids.org> Message-ID: <061.5a3152921a54e79e0a14dc2f3dee66b2@tracker.bro-ids.org> #460: Problem with extended records and empty container types ----------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): In [142faa90c8bba3f312e19544a41b0abfdab47365/bro]: {{{ #!CommitTicketReference repository="bro" revision="142faa90c8bba3f312e19544a41b0abfdab47365" Fix redef'ing records with &default empty set fields. Attributes have state to track whether they're in a record and should apply to a record field, but this state wasn't being set for TypeDecls that are part of a redef'd record. Addresses #460 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 11:22:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 18:22:25 -0000 Subject: [Bro-Dev] #460: Problem with extended records and empty container types In-Reply-To: <046.e32eb4f1b2af66ecfcc23a676b3b6332@tracker.bro-ids.org> References: <046.e32eb4f1b2af66ecfcc23a676b3b6332@tracker.bro-ids.org> Message-ID: <061.fae04a217eca4ad38812a3f567e5fa5e@tracker.bro-ids.org> #460: Problem with extended records and empty container types ----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: This example works for me with the fix I did in `topic/jsiwek/record- redef-default-set`. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 11:37:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 18:37:14 -0000 Subject: [Bro-Dev] #255: internal error: NB-DNS error in DNS_Mgr::Process (recvfrom(): Connection refused) In-Reply-To: <056.417dfe2f09777f9f1b0a836ed664bcc7@tracker.bro-ids.org> References: <056.417dfe2f09777f9f1b0a836ed664bcc7@tracker.bro-ids.org> Message-ID: <071.8b671354396a9df575444a115ec273d2@tracker.bro-ids.org> #255: internal error: NB-DNS error in DNS_Mgr::Process (recvfrom(): Connection refused) -----------------------------+---------------------- Reporter: Tyler.Schoenke | Owner: robin Type: Problem | Status: accepted Priority: High | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: -----------------------------+---------------------- Changes (by seth): * priority: Normal => High -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 11:47:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 18:47:46 -0000 Subject: [Bro-Dev] #447: Bro snaplen In-Reply-To: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> References: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> Message-ID: <061.9bc57bb6a728a3b74cf8cb903af88faa@tracker.bro-ids.org> #447: Bro snaplen ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): I'm not sure this is something that could be auto tuned easily. I assume (i know, i know. assumptions and all) that the reason almost every tool exposes this setting to users is because there just isn't a really good way to tell if you are truncating packets besides testing higher snap lengths. Vern, do you have any fundamental objections with us going forward with creating some user accessible setting for the sake of expediency? We could even set 65535 as the default so that it should just work out of the box in most scenarios with no tuning with the downside that it could potentially cause performance or capacity problems in some installations. I actually ran into this problem yesterday and needed to modify Pktsrc.cc to make Bro see full packets. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 11:50:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 18:50:38 -0000 Subject: [Bro-Dev] #534: Leak in DNS resolver In-Reply-To: <047.a6cec0e639c569fb09138a1ce7c3bfb4@tracker.bro-ids.org> References: <047.a6cec0e639c569fb09138a1ce7c3bfb4@tracker.bro-ids.org> Message-ID: <062.e1a4650efbea248df48af8df21f12c6f@tracker.bro-ids.org> #534: Leak in DNS resolver ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: BETA ----------------------+------------------------ Changes (by seth): * priority: Normal => High -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 11:59:57 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 18:59:57 -0000 Subject: [Bro-Dev] #537: Default policy script overhead due to DNS lookups in extend-email/hostnames.bro In-Reply-To: <048.931166b0a66e88c700755af557fbbd31@tracker.bro-ids.org> References: <048.931166b0a66e88c700755af557fbbd31@tracker.bro-ids.org> Message-ID: <063.7a7174a8ccfa1a60838fd09715ceba77@tracker.bro-ids.org> #537: Default policy script overhead due to DNS lookups in extend- email/hostnames.bro -----------------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Adjusted and not "default" in SSH and the notice framework anymore. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 12:02:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 19:02:27 -0000 Subject: [Bro-Dev] #506: loggable functions In-Reply-To: <046.fd94fc1efd6f020fa77217ca380cf485@tracker.bro-ids.org> References: <046.fd94fc1efd6f020fa77217ca380cf485@tracker.bro-ids.org> Message-ID: <061.e64c4116a7e8a010637bc9dcc4916885@tracker.bro-ids.org> #506: loggable functions ----------------------+---------------------- Reporter: seth | Owner: robin Type: Problem | Status: assigned Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: preview ----------------------+---------------------- Changes (by seth): * priority: Normal => High -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 12:59:19 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 19:59:19 -0000 Subject: [Bro-Dev] #553: Script variable to set pcap's buffer size Message-ID: <048.fdc2e81d1331136042f212703cab1b0c@tracker.bro-ids.org> #553: Script variable to set pcap's buffer size -----------------------------+------------------------ Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Keywords: | -----------------------------+------------------------ Newer pcap versions have a pcap_set_buffer_size(). This call can be used to adjust the kernel level buffer size for captures. (E.g., Linux PF_RING's buffer size, which will be quite important if we start using 64KB snaplen) It's supported for all platforms we care about (Linux, BPF-based OSes), but we probably want to make sure that it obsoletes the manual tuning through: http://www.net.t-labs.tu-berlin.de/research/bpcs/ (It might well be that one still has to manually increase the max buffer sizes) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 13:40:59 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 20:40:59 -0000 Subject: [Bro-Dev] #364: login.bro should raise Notices when being confused In-Reply-To: <047.ea749bad321e14efd7466bee5fc315ad@tracker.bro-ids.org> References: <047.ea749bad321e14efd7466bee5fc315ad@tracker.bro-ids.org> Message-ID: <062.2f36f7d678843a6e10918dc58c7362e7@tracker.bro-ids.org> #364: login.bro should raise Notices when being confused ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: I don't think this is going to make it into the next release. I want to approach the notion of authentication process from a more generic approach (FTP, login, etc). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 13:42:04 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 11 Aug 2011 20:42:04 -0000 Subject: [Bro-Dev] #378: Test for problem with optional record fields. In-Reply-To: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> References: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> Message-ID: <062.6913f990d5fb97c6c964ccf90fc904da@tracker.bro-ids.org> #378: Test for problem with optional record fields. --------------------------------+---------------------- Reporter: robin | Owner: robin Type: Test Case Missing | Status: accepted Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: --------------------------------+---------------------- Comment (by seth): I tend to forget about assigning ticket to people :) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 18:30:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 12 Aug 2011 01:30:23 -0000 Subject: [Bro-Dev] #447: Bro snaplen In-Reply-To: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> References: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> Message-ID: <061.af7487cac052229fb3f5d0290e81133b@tracker.bro-ids.org> #447: Bro snaplen ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by vern): User accessible + 65535 default is okay with me for now. I note that netstat -rl, at least on MacOS, displays MTU per interface. OTOH, that maybe isn't useful, because one doesn't know at config time what interface will be used, and using the max across these will be pessimistic, since for example localhost on my Mac is 16384. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 11 21:45:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 12 Aug 2011 04:45:37 -0000 Subject: [Bro-Dev] #554: topic/gregor/script-polishing Message-ID: <048.cb2dd828f6587598fb24b58176d476c2@tracker.bro-ids.org> #554: topic/gregor/script-polishing ---------------------------+------------------------ Reporter: gregor | Owner: seth Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ Some tuning to the new policy scripts. You can cherry-pick each individual commit if desired. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Aug 11 22:11:38 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 11 Aug 2011 22:11:38 -0700 Subject: [Bro-Dev] Python script/module to parse new log files Message-ID: <4E44B60A.3070003@icir.org> Hi, has anybody already written a python script or module to easily parse the new Bro ASCII log files? cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From gc355804 at ohio.edu Thu Aug 11 23:05:12 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Fri, 12 Aug 2011 02:05:12 -0400 Subject: [Bro-Dev] Python script/module to parse new log files In-Reply-To: <4E44B60A.3070003@icir.org> References: <4E44B60A.3070003@icir.org> Message-ID: <4E44C298.90101@ohio.edu> I've got some code for this, but it's a work in progress; you're welcome to take a look at what's there, though, and do whatever you'd like with it. *** Note that this code demands a slightly modified ASCII header (which includes the bro path name and the separator char). Anyway, branch is: topic/gilbert/log-util and check: aux/log-util I think the usage is pretty straightforward: manager = BroLogManager() manager.load('/path/to/logs') #If this is a directory, *all files from all subdirectories* of this path will be loaded. # Otherwise, only the referenced files will be loaded. # .log, .log.gz, and .log.bz2 are treated as new-style bro logs print manager['conn'].get_stats('orig_bytes') # This prints some interesting statistics about the orig_bytes column (min / max / mean / std_dev). # Note that the above will actually load *all* relevant log files and perform the calculation, then cache # the results; the first get_stats() will be very slow, but the rest should work pretty quickly. for e in manager['conn'].entries(): print e['ts'] # Note that 'e.ts' will also work, but that this doesn't work for field names that don't map nicely to Python. See 'bro-logtool' for a toy script I've been using to play with the library. --Gilbert On 8/12/2011 1:11 AM, Gregor Maier wrote: > Hi, > > has anybody already written a python script or module to easily parse > the new Bro ASCII log files? > > cu > Gregor From gc355804 at ohio.edu Thu Aug 11 23:56:04 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Fri, 12 Aug 2011 02:56:04 -0400 Subject: [Bro-Dev] bro's dlmalloc Message-ID: <4E44CE84.8040609@ohio.edu> So I was testing the threaded code on FreeBSD, and discovered I'd forgotten to check to see if the malloc.c that shipped with bro was thread-safe. Oops. When I went to browse malloc.c's source, I found out that it was dlmalloc. Further, it appears that dlmalloc is not really meant to support threads [1]. With that in mind, I'm wondering if malloc.c is really necessary; most systems these days have a pretty good default allocator ([2], [3]). From what I've read, jemalloc in particular (default allocator in FreeBSD 7+) is competitive with the Linux default allocator (ptmalloc2 derivative) [4], and is used in one context or another by Firefox and Facebook to help with memory fragmentation issues [5]. For folks worried about performance, there's always tcmalloc [6]; this comes with google perftools, but would it be a good idea to just embed this in the project? With the above in mind, what would y'all think about either: *) Removing malloc.c entirely *) Replacing dlmalloc with e.g. ptmalloc3 (which I believe to be licensed under something BSD-like, but need to check on that) or jemalloc (BSD-ish license [7]) --Gilbert [1] From dlmalloc's source: "... When USE_LOCKS is defined, each public call to malloc, free, etc is surrounded with either a pthread mutex or a win32 spinlock (depending on WIN32). This is not especially fast, and can be a major bottleneck..." [2] http://people.freebsd.org/~jasone/jemalloc/bsdcan2006/BSDcan2006_slides.pdf [3] http://people.freebsd.org/~kris/scaling/ebizzy.html [4] http://www.sintel.org/development/memory-jemalloc/ [5] http://www.canonware.com/jemalloc/ [6] http://googlecode.blogspot.com/2006/12/tcmalloc-success.html [7] http://www.canonware.com/jemalloc/license.html From bro at tracker.bro-ids.org Fri Aug 12 07:03:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 12 Aug 2011 14:03:25 -0000 Subject: [Bro-Dev] #555: Logging framework path_func Message-ID: <046.097d89da8440145fd9e8f44dc674842e@tracker.bro-ids.org> #555: Logging framework path_func ---------------------+-------------------- Reporter: seth | Owner: robin Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- In the Log::Filter type, the $path_func record field is incomplete. The field looks like this now: {{{ path_func: function(id: ID, path: string): string &optional; }}} It needs to also receive a reference to the actual log line like $pred field (with the any type) so that the function prototype would look like this: {{{ path_func: function(id: ID, path: string, rec: any): string &optional; }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 12 08:03:18 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 12 Aug 2011 15:03:18 -0000 Subject: [Bro-Dev] #485: Error when defining vectors of records with optional fields In-Reply-To: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> References: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> Message-ID: <065.bbed66e780a392aa450f1b9a74120d04@tracker.bro-ids.org> #485: Error when defining vectors of records with optional fields -----------------------+------------------------ Reporter: vallenti | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: -----------------------+------------------------ Comment (by jsiwek): In [8c20d2051fe9016874015fb49b68583525ef9fbf/bro]: {{{ #!CommitTicketReference repository="bro" revision="8c20d2051fe9016874015fb49b68583525ef9fbf" Fix vector initialization for lists of records with optional types. If possible the list elements now get promoted to the yield type of the vector. There was also a problem with the value returned by the record constructor expression's eval being completely unref'd since the vector element assignment function doesn't ref the element -- so I changed it to ref values if they just constructed before assigning them to the vector. Addresses #485. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 12 08:06:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 12 Aug 2011 15:06:34 -0000 Subject: [Bro-Dev] #485: Error when defining vectors of records with optional fields In-Reply-To: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> References: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> Message-ID: <065.d0d2aca5f3cb92f6bba67c7e7a729a0b@tracker.bro-ids.org> #485: Error when defining vectors of records with optional fields ----------------------------+------------------------ Reporter: vallenti | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: Patch is in `topic/jsiwek/vector-list-init-records`. I'm not sure if what I did with the reference counting issue is best/correct (is there some documentation/guidelines to follow?). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 12 08:29:02 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 12 Aug 2011 15:29:02 -0000 Subject: [Bro-Dev] #556: Extended CA certificate information Message-ID: <046.8ebd0a63baadb617fd332f651cf1ba13@tracker.bro-ids.org> #556: Extended CA certificate information -----------------------------+----------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Component: bro-aux | Version: Keywords: | -----------------------------+----------------- At some point I'd like to include more information in the auto-generated script that currently only has Mozilla's CA certs. At the very least I'd like to include extended validation OIDs for the various approved EV certificate vendors. The extra data can be found in the XML file located here: https://www.mozilla.org/projects/security/certs/included/ -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 12 08:31:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 12 Aug 2011 15:31:34 -0000 Subject: [Bro-Dev] #556: Extended CA certificate information In-Reply-To: <046.8ebd0a63baadb617fd332f651cf1ba13@tracker.bro-ids.org> References: <046.8ebd0a63baadb617fd332f651cf1ba13@tracker.bro-ids.org> Message-ID: <061.dfda0940e4e34186bccf9e1c775be0d6@tracker.bro-ids.org> #556: Extended CA certificate information ------------------------------+----------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Component: bro-aux | Version: Resolution: | Keywords: ------------------------------+----------------- Description changed by seth: Old description: > At some point I'd like to include more information in the auto-generated > script that currently only has Mozilla's CA certs. At the very least I'd > like to include extended validation OIDs for the various approved EV > certificate vendors. > > The extra data can be found in the XML file located here: > https://www.mozilla.org/projects/security/certs/included/ New description: At some point I'd like to include more information in the auto-generated script that currently only has Mozilla's CA certs. At the very least I'd like to include extended validation OIDs for the various approved EV certificate vendors and OCSP URLs. The extra data can be found in the XML file located here: https://www.mozilla.org/projects/security/certs/included/ -- -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Fri Aug 12 09:14:13 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 12 Aug 2011 09:14:13 -0700 Subject: [Bro-Dev] bro's dlmalloc In-Reply-To: <4E44CE84.8040609@ohio.edu> References: <4E44CE84.8040609@ohio.edu> Message-ID: <20110812161413.GC71457@icir.org> On Fri, Aug 12, 2011 at 02:56 -0400, you wrote: > support threads [1]. With that in mind, I'm wondering if malloc.c is > really necessary; most systems these days have a pretty good default > allocator ([2], [3]). That's an excellent point. We added that many years ago when FreeBSD's malloc performed pretty badly for Bro, but I have no idea if that still applies. As you say, it seems reasonable to assume that things have changed. > *) Removing malloc.c entirely That sounds like the right thing to do, at least as long as we don't have any data to back up any of the other options. I'll file a ticket. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Fri Aug 12 09:17:04 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 12 Aug 2011 16:17:04 -0000 Subject: [Bro-Dev] #557: Remove malloc.c Message-ID: <047.d9416191ab6a77114a37dd25d0191abc@tracker.bro-ids.org> #557: Remove malloc.c ------------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: Bro1.6 | Component: Bro Version: git/master | Keywords: beta ------------------------+--------------------- Bro is still using its own malloc.c on FreeBSD, which was helpful many years ago, but likely won't be needed anymore these days. In fact it could even make things worse now. Let's remove it and use the system malloc everywhere (at least until we get some data convincing us that doing somethign custom is worth it). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 12 12:56:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 12 Aug 2011 19:56:53 -0000 Subject: [Bro-Dev] #298: Automatic build tests on NMI testbed. In-Reply-To: <046.52cb04f5dd135b7d304d18bd01deebaf@tracker.bro-ids.org> References: <046.52cb04f5dd135b7d304d18bd01deebaf@tracker.bro-ids.org> Message-ID: <061.f905fe0c8225935ba331a990a0f062db@tracker.bro-ids.org> #298: Automatic build tests on NMI testbed. ---------------------+-------------------- Reporter: seth | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: nmi ---------------------+-------------------- Changes (by jsiwek): * keywords: => nmi -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 12 23:30:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 13 Aug 2011 06:30:49 -0000 Subject: [Bro-Dev] #543: Bro autodoc after script reorganization In-Reply-To: <048.a8604e1df88ab44815b110740aa391a4@tracker.bro-ids.org> References: <048.a8604e1df88ab44815b110740aa391a4@tracker.bro-ids.org> Message-ID: <063.c27b375240a11534694713fa722a387b@tracker.bro-ids.org> #543: Bro autodoc after script reorganization ----------------------------+------------------------ Reporter: gregor | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: autodoc ----------------------------+------------------------ Comment (by robin): This gives me a number of conflicts when merging, and for some I'm not sure what the right version is now. Please merge master into the branch and adapt as necessary. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 07:47:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 13 Aug 2011 14:47:38 -0000 Subject: [Bro-Dev] #543: Bro autodoc after script reorganization In-Reply-To: <048.a8604e1df88ab44815b110740aa391a4@tracker.bro-ids.org> References: <048.a8604e1df88ab44815b110740aa391a4@tracker.bro-ids.org> Message-ID: <063.7652b423b8d3efa71402701ca09b351c@tracker.bro-ids.org> #543: Bro autodoc after script reorganization ----------------------------+------------------------ Reporter: gregor | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: autodoc ----------------------------+------------------------ Comment (by jsiwek): Replying to [comment:4 robin]: > This gives me a number of conflicts when merging, and for some I'm not sure what the right version is now. Please merge master into the branch and adapt as necessary. These are fixed in the branch now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 12:15:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 13 Aug 2011 19:15:34 -0000 Subject: [Bro-Dev] #492: Small bug in new reporter code In-Reply-To: <046.8bda721c6943df6332a38c8f506a2b5f@tracker.bro-ids.org> References: <046.8bda721c6943df6332a38c8f506a2b5f@tracker.bro-ids.org> Message-ID: <061.1cb52d94cf84ff7ae7e8eb244dadcbd4@tracker.bro-ids.org> #492: Small bug in new reporter code ----------------------+------------------------ Reporter: seth | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------+------------------------ Changes (by robin): * status: accepted => closed * resolution: => fixed Comment: In [33b064bdb2ab19f02be8b21a0a828a9c6922fd24/bro]: {{{ #!CommitTicketReference repository="bro" revision="33b064bdb2ab19f02be8b21a0a828a9c6922fd24" Fixing reporter's location tracking. Closes #492. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 12:15:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 13 Aug 2011 19:15:34 -0000 Subject: [Bro-Dev] #555: Logging framework path_func In-Reply-To: <046.097d89da8440145fd9e8f44dc674842e@tracker.bro-ids.org> References: <046.097d89da8440145fd9e8f44dc674842e@tracker.bro-ids.org> Message-ID: <061.01f57bc46b17ee04733856470040604c@tracker.bro-ids.org> #555: Logging framework path_func ----------------------+-------------------- Reporter: seth | Owner: robin Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: ----------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => fixed Comment: In [cb31fd3bb9345bf9594d24adce58788be75bec4f/bro]: {{{ #!CommitTicketReference repository="bro" revision="cb31fd3bb9345bf9594d24adce58788be75bec4f" Logging's path_func now receives the log record as argument. Closes #555. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 12:15:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 13 Aug 2011 19:15:34 -0000 Subject: [Bro-Dev] #506: loggable functions In-Reply-To: <046.fd94fc1efd6f020fa77217ca380cf485@tracker.bro-ids.org> References: <046.fd94fc1efd6f020fa77217ca380cf485@tracker.bro-ids.org> Message-ID: <061.26a499145b04f9484fbd7eeb00da2003@tracker.bro-ids.org> #506: loggable functions ----------------------+--------------------- Reporter: seth | Owner: robin Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: preview ----------------------+--------------------- Changes (by robin): * status: assigned => closed * resolution: => fixed Comment: In [c436930acfb0746d044ae666ac23bae52b41cdf2/bro]: {{{ #!CommitTicketReference repository="bro" revision="c436930acfb0746d044ae666ac23bae52b41cdf2" Functions can now be logged. The function's code is rendered as ASCII and included as a string. Closes #506. Note that I'm not sure if the formatting is as desired: should the LFs and tabs be rendered as \xXX or removed?. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 12:15:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 13 Aug 2011 19:15:34 -0000 Subject: [Bro-Dev] #255: internal error: NB-DNS error in DNS_Mgr::Process (recvfrom(): Connection refused) In-Reply-To: <056.417dfe2f09777f9f1b0a836ed664bcc7@tracker.bro-ids.org> References: <056.417dfe2f09777f9f1b0a836ed664bcc7@tracker.bro-ids.org> Message-ID: <071.008671d25e4fe2d607cb50d1e6061341@tracker.bro-ids.org> #255: internal error: NB-DNS error in DNS_Mgr::Process (recvfrom(): Connection refused) -----------------------------+-------------------- Reporter: Tyler.Schoenke | Owner: robin Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: fixed | Keywords: -----------------------------+-------------------- Changes (by robin): * status: accepted => closed * resolution: => fixed Comment: In [46d3570bf5af366170b408a04e94062f03fb28ff/bro]: {{{ #!CommitTicketReference repository="bro" revision="46d3570bf5af366170b408a04e94062f03fb28ff" Turning DNS errors into a warning. It seems these errors aren't Bro's fault, and in any case it's clearly not an internal error. This should finally solve the problem in #255. Closes #255. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 12:15:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 13 Aug 2011 19:15:35 -0000 Subject: [Bro-Dev] #485: Error when defining vectors of records with optional fields In-Reply-To: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> References: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> Message-ID: <065.7712f4c691286febbd5c100ef08ccfb2@tracker.bro-ids.org> #485: Error when defining vectors of records with optional fields ----------------------------+------------------------ Reporter: vallenti | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): In [6f060a58d6a5f8f38aa97f2ba94382a247956ec2/bro]: {{{ #!CommitTicketReference repository="bro" revision="6f060a58d6a5f8f38aa97f2ba94382a247956ec2" Fix vector initialization for lists of records with optional types. If possible the list elements now get promoted to the yield type of the vector. There was also a problem with the value returned by the record constructor expression's eval being completely unref'd since the vector element assignment function doesn't ref the element -- so I changed it to ref values if they just constructed before assigning them to the vector. Addresses #485. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 12:15:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 13 Aug 2011 19:15:35 -0000 Subject: [Bro-Dev] #460: Problem with extended records and empty container types In-Reply-To: <046.e32eb4f1b2af66ecfcc23a676b3b6332@tracker.bro-ids.org> References: <046.e32eb4f1b2af66ecfcc23a676b3b6332@tracker.bro-ids.org> Message-ID: <061.05b43e8b8a2bb0e7c02c60657e288311@tracker.bro-ids.org> #460: Problem with extended records and empty container types ----------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [560685f1c6527e7f83b5cdd947691f4eef14f737/bro]: {{{ #!CommitTicketReference repository="bro" revision="560685f1c6527e7f83b5cdd947691f4eef14f737" Fix redef'ing records with &default empty set fields. Attributes have state to track whether they're in a record and should apply to a record field, but this state wasn't being set for TypeDecls that are part of a redef'd record. Closes #460 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 12:26:19 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 13 Aug 2011 19:26:19 -0000 Subject: [Bro-Dev] #485: Error when defining vectors of records with optional fields In-Reply-To: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> References: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> Message-ID: <065.6792032683655baf947c4e241401a7b5@tracker.bro-ids.org> #485: Error when defining vectors of records with optional fields ----------------------------+------------------------ Reporter: vallenti | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => jsiwek * status: new => assigned Comment: I've removed the ref counting change as I believe it was correct in its original form. Eval(0) should always return a +1 ref count. What did you see that didn't work right? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 12:26:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 13 Aug 2011 19:26:52 -0000 Subject: [Bro-Dev] #543: Bro autodoc after script reorganization In-Reply-To: <048.a8604e1df88ab44815b110740aa391a4@tracker.bro-ids.org> References: <048.a8604e1df88ab44815b110740aa391a4@tracker.bro-ids.org> Message-ID: <063.efa41c9f4bf41240c0a9ad1ed7110496@tracker.bro-ids.org> #543: Bro autodoc after script reorganization -----------------------------+------------------------ Reporter: gregor | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: autodoc -----------------------------+------------------------ Changes (by robin): * status: assigned => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 12:27:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 13 Aug 2011 19:27:11 -0000 Subject: [Bro-Dev] #546: topic/seth/piped_exec-update In-Reply-To: <046.aa8796da7721564d8d12d7a1dcc172d5@tracker.bro-ids.org> References: <046.aa8796da7721564d8d12d7a1dcc172d5@tracker.bro-ids.org> Message-ID: <061.37e4a6e4f5f35ed786c4b56921d4a8a6@tracker.bro-ids.org> #546: topic/seth/piped_exec-update -----------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 12:27:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 13 Aug 2011 19:27:28 -0000 Subject: [Bro-Dev] #547: topic/jsiwek/debug-flags In-Reply-To: <048.644ec20dbcc0abbef35013f8bbaced72@tracker.bro-ids.org> References: <048.644ec20dbcc0abbef35013f8bbaced72@tracker.bro-ids.org> Message-ID: <063.36b9a380855cea90e9eb5a64d6701018@tracker.bro-ids.org> #547: topic/jsiwek/debug-flags -----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: closed Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 20:52:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 14 Aug 2011 03:52:20 -0000 Subject: [Bro-Dev] #506: loggable functions In-Reply-To: <046.fd94fc1efd6f020fa77217ca380cf485@tracker.bro-ids.org> References: <046.fd94fc1efd6f020fa77217ca380cf485@tracker.bro-ids.org> Message-ID: <061.76c04f9a06ecd02cf3aa83d172b7e8a7@tracker.bro-ids.org> #506: loggable functions ----------------------+--------------------- Reporter: seth | Owner: robin Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: preview ----------------------+--------------------- Comment (by seth): > Note that I'm not sure if the formatting is as desired: should the LFs > and tabs be rendered as \xXX or removed?. I think that if you want to remove it for the ascii output, that'd be ok. I'd pass the LFs on to the plugins though since it's possible that it could be sent to a database where the where strict "no linefeeds" rule needs to be enforced. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 21:37:26 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 14 Aug 2011 04:37:26 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header Message-ID: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header --------------------+--------------------------- Reporter: gclark | Type: Merge Request Status: new | Priority: Normal Milestone: Bro1.6 | Component: Bro Version: | Keywords: logging --------------------+--------------------------- Updated ASCII header to better support scripting. Exposes bro path, separator character, and bro type of each field in the log file. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 21:56:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 14 Aug 2011 04:56:50 -0000 Subject: [Bro-Dev] #548: broctl brokenness multi-ticket In-Reply-To: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> References: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> Message-ID: <061.74d67a75e473655222ea6552ce3ea2a0@tracker.bro-ids.org> #548: broctl brokenness multi-ticket -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta -------------------------+-------------------- Changes (by seth): * priority: Normal => High -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Aug 13 22:00:02 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 14 Aug 2011 05:00:02 -0000 Subject: [Bro-Dev] #461: internal_error called for async DNS failures In-Reply-To: <046.adc7e841ed311111a107267ffb4a852b@tracker.bro-ids.org> References: <046.adc7e841ed311111a107267ffb4a852b@tracker.bro-ids.org> Message-ID: <061.3b5858da8ce957bc62e3be70d7e22172@tracker.bro-ids.org> #461: internal_error called for async DNS failures ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): Is this ticket also addressed by changeset 46d3570bf5af366170b408a04e94062f03fb28ff ? That changeset is the one that addressed ticket #255 -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Aug 14 20:31:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 15 Aug 2011 03:31:56 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.c38e6f4d36c1fac210cfd0546cb3370d@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ----------------------------+---------------------- Reporter: gclark | Owner: robin Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ----------------------------+---------------------- Changes (by robin): * owner: => robin * status: new => assigned Comment: Thanks! Questions/comments: - I'd prefer the line with the types to have just those, i.e., `time` instread of `ts=time`. That makes it easier to parse in like awk, and the relationship to the field name is already there via the column. - Good idea to record the field separator as well. But for non-printable characters (like the default tab) it looks a bit odd, and the enclosing `'...'` are somewhat tricky to parse as well. How about ``# separator \t``, with whitespace separating the two and the separator itself having non-printable characters escaped. Note though that (1) I'm not sure whether it should be ``\t`` or for consistency ``\x09``, and (2) whether either of these can be easily converted back into the binary form in awk. Other ideas? - What's the use case for including the path? - I'm wondering whether we should make it easy to tell what kind of meta- line it is. How about this format: {{{ #fields ts uid id.orig_h id.orig_p ... #types time string addr port ... #separator \t #path conn }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Aug 14 21:33:04 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 15 Aug 2011 04:33:04 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.ecc08634ca495492a152752fe12ea49c@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ----------------------------+---------------------- Reporter: gclark | Owner: robin Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ----------------------------+---------------------- Comment (by gclark): > - I'd prefer the line with the types to have just those, i.e., `time` instread of `ts=time`. That makes it easier to parse in like awk, and the relationship to the field name is already there via the column. All right. > - What's the use case for including the path? This allows log processors to recognize log files that have been renamed to formats other than simple 'conn.log' or 'ftp.log' without having to wield complex file name regex voodoo. > - I'm wondering whether we should make it easy to tell what kind of meta-line it is. How about this format: > {{{ > #fields ts uid id.orig_h id.orig_p ... > #types time string addr port ... > #separator \t > #path conn > }}} I think separator needs to come first, as it's used while parsing the other header fields. I also think we need to be careful to ensure the separator line itself uses a constant separator (a space?), or the line won't be very useful :) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Aug 14 21:37:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 15 Aug 2011 04:37:07 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.9017cf10729a2e0dc7b69c65dd7c7b69@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ----------------------------+---------------------- Reporter: gclark | Owner: robin Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ----------------------------+---------------------- Comment (by gclark): Oh, I missed the 'whitespace' bit for the separator; never mind the 'constant separator' comment above. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon Aug 15 08:36:30 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 15 Aug 2011 08:36:30 -0700 Subject: [Bro-Dev] Connection Compressor In-Reply-To: <4E3EFB7F.10603@icir.org> References: <4E3EFB7F.10603@icir.org> Message-ID: <20110815153630.GK76421@icir.org> On Sun, Aug 07, 2011 at 13:54 -0700, you wrote: > So, all in all, it appears that the connection compressor doesn't help > much anymore these days I've done some measurements as well now and can confirm this. With master in default mode, I also see only tiny savings in time and memory. So I agree that we should remove the compressor code. I'm sad to see it go, but it's one of these pieces that complicate things considerably and that's only worth it if there's a clear gain coming out of it. Opinions? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From vern at icir.org Mon Aug 15 08:42:55 2011 From: vern at icir.org (Vern Paxson) Date: Mon, 15 Aug 2011 08:42:55 -0700 Subject: [Bro-Dev] Connection Compressor In-Reply-To: <20110815153630.GK76421@icir.org> (Mon, 15 Aug 2011 08:36:30 PDT). Message-ID: <20110815154255.BEB9A36A38A@taffy.ICSI.Berkeley.EDU> > I've done some measurements as well now and can confirm this. With > master in default mode, I also see only tiny savings in time and > memory. So just to confirm: for a high-speed SYN flooding attack, it's not much help? That was the original motivation, after all. Vern From robin at icir.org Mon Aug 15 08:53:59 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 15 Aug 2011 08:53:59 -0700 Subject: [Bro-Dev] Connection Compressor In-Reply-To: <20110815154255.BEB9A36A38A@taffy.ICSI.Berkeley.EDU> References: <20110815153630.GK76421@icir.org> <20110815154255.BEB9A36A38A@taffy.ICSI.Berkeley.EDU> Message-ID: <20110815155359.GM76421@icir.org> On Mon, Aug 15, 2011 at 08:42 -0700, you wrote: > So just to confirm: for a high-speed SYN flooding attack, it's not much > help? I didn't try that (only "normal" traiffc) and it probably still helps with that. But that was only part of the original motivation, which started out from the general observation of many connections not getting established. And another piece of the story was the separate flood detector that starts sampling traffic from specific sources or destinations (Seth, is that ported already?) So I think the compressor still helps in some (extreme) situations, and generally, performance-wise it certainly doesn't hurt to have it. But I'm not sure it's worth the complexity: we keep running into issues with the changes in semantics it introduces, it's on a separate code path that needs to be integrated with all packet-level stuff, and as Gregor said, it would also need more than just maintainance work in the future, like adding IPv6 support. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Mon Aug 15 09:05:19 2011 From: seth at icir.org (Seth Hall) Date: Mon, 15 Aug 2011 12:05:19 -0400 Subject: [Bro-Dev] Connection Compressor In-Reply-To: <20110815155359.GM76421@icir.org> References: <20110815153630.GK76421@icir.org> <20110815154255.BEB9A36A38A@taffy.ICSI.Berkeley.EDU> <20110815155359.GM76421@icir.org> Message-ID: On Aug 15, 2011, at 11:53 AM, Robin Sommer wrote: > And another piece of the story was the separate > flood detector that starts sampling traffic from specific sources or > destinations (Seth, is that ported already?) Nope, but that's a perfect use case for the metrics framework (which I just got working on cluster deployments!). I can work on it this afternoon as an example use for the metrics framework. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gregor at icir.org Mon Aug 15 09:10:51 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 15 Aug 2011 09:10:51 -0700 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <063.c38e6f4d36c1fac210cfd0546cb3370d@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> <063.c38e6f4d36c1fac210cfd0546cb3370d@tracker.bro-ids.org> Message-ID: <4E49450B.3020301@icir.org> > - Good idea to record the field separator as well. But for non-printable > characters (like the default tab) it looks a bit odd, and the enclosing > `'...'` are somewhat tricky to parse as well. How about ``# separator > \t``, with whitespace separating the two and the separator itself having > non-printable characters escaped. Note though that (1) I'm not sure > whether it should be ``\t`` or for consistency ``\x09``, and (2) whether > either of these can be easily converted back into the binary form in awk. I don't like using whitespace here. If the remainder of the file uses tabs, that's a mix of two whitespace characters :-(. How about using a colon? separator:\x09 With my darwin awk the following works: awk 'BEGIN { FS="\x09" } > Other ideas? > - What's the use case for including the path? Maybe "path" is misleading. Could try with "log" or "file" instead. > - I'm wondering whether we should make it easy to tell what kind of meta- > line it is. How about this format: > {{{ > #fields ts uid id.orig_h id.orig_p ... > #types time string addr port ... > #separator \t > #path conn > }}} ACK. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Mon Aug 15 09:10:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 15 Aug 2011 16:10:56 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.83c82b32439ead0018f6e28608bec608@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ----------------------------+---------------------- Reporter: gclark | Owner: robin Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ----------------------------+---------------------- Comment (by gregor): > - Good idea to record the field separator as well. But for non- printable > characters (like the default tab) it looks a bit odd, and the enclosing > `'...'` are somewhat tricky to parse as well. How about ``# separator > \t``, with whitespace separating the two and the separator itself having > non-printable characters escaped. Note though that (1) I'm not sure > whether it should be ``\t`` or for consistency ``\x09``, and (2) whether > either of these can be easily converted back into the binary form in awk. I don't like using whitespace here. If the remainder of the file uses tabs, that's a mix of two whitespace characters :-(. How about using a colon? separator:\x09 With my darwin awk the following works: awk 'BEGIN { FS="\x09" } > Other ideas? > - What's the use case for including the path? Maybe "path" is misleading. Could try with "log" or "file" instead. > - I'm wondering whether we should make it easy to tell what kind of meta- > line it is. How about this format: > {{{ > #fields ts uid id.orig_h id.orig_p ... > #types time string addr port ... > #separator \t > #path conn > }}} ACK. cu Gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 15 09:26:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 15 Aug 2011 16:26:48 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.d6c785e9cc89e605966fcfa39f288b94@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ----------------------------+---------------------- Reporter: gclark | Owner: robin Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ----------------------------+---------------------- Comment (by seth): >> Other ideas? >> - What's the use case for including the path? > > Maybe "path" is misleading. Could try with "log" or "file" instead. I think "stream" might be the correct term here since the written records are associated with the stream. Fields could be removed by the filter, but knowing the stream would at least help to know what's being dealt with since there is no 1-to-1 mapping between file name and file content anywhere (files don't necessarily have their stream named prepended). -- Ticket URL: Bro Tracker Bro Issue Tracker From jmellander at lbl.gov Mon Aug 15 10:09:19 2011 From: jmellander at lbl.gov (Jim Mellander) Date: Mon, 15 Aug 2011 10:09:19 -0700 Subject: [Bro-Dev] Connection Compressor In-Reply-To: <20110815155359.GM76421@icir.org> References: <20110815153630.GK76421@icir.org> <20110815154255.BEB9A36A38A@taffy.ICSI.Berkeley.EDU> <20110815155359.GM76421@icir.org> Message-ID: Hi: We like setting record_state_history=T to record the flags that were seen during the connection, and found that the connection compressor didn't play nicely with that, in some cases. On Mon, Aug 15, 2011 at 8:53 AM, Robin Sommer wrote: > > On Mon, Aug 15, 2011 at 08:42 -0700, you wrote: > > > So just to confirm: for a high-speed SYN flooding attack, it's not much > > help? > > I didn't try that (only "normal" traiffc) and it probably still helps > with that. But that was only part of the original motivation, which > started out from the general observation of many connections not > getting established. And another piece of the story was the separate > flood detector that starts sampling traffic from specific sources or > destinations (Seth, is that ported already?) > > So I think the compressor still helps in some (extreme) situations, > and generally, performance-wise it certainly doesn't hurt to have it. > But I'm not sure it's worth the complexity: we keep running into > issues with the changes in semantics it introduces, it's on a separate > code path that needs to be integrated with all packet-level stuff, and > as Gregor said, it would also need more than just maintainance work in > the future, like adding IPv6 support. > > Robin > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110815/b19d89bc/attachment-0001.html From seth at icir.org Mon Aug 15 10:29:19 2011 From: seth at icir.org (Seth Hall) Date: Mon, 15 Aug 2011 13:29:19 -0400 Subject: [Bro-Dev] Connection Compressor In-Reply-To: References: <20110815153630.GK76421@icir.org> <20110815154255.BEB9A36A38A@taffy.ICSI.Berkeley.EDU> <20110815155359.GM76421@icir.org> Message-ID: <42467A98-42DE-4EB6-947F-C536B67A999D@icir.org> On Aug 15, 2011, at 1:09 PM, Jim Mellander wrote: > We like setting record_state_history=T to record the flags that were seen during the connection, and found that the connection compressor didn't play nicely with that, in some cases. Sounds like more evidence for removing the conn compressor if possible. Also, state history is included in the new conn logs by default. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Mon Aug 15 10:35:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 15 Aug 2011 17:35:25 -0000 Subject: [Bro-Dev] #485: Error when defining vectors of records with optional fields In-Reply-To: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> References: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> Message-ID: <065.170847037b0435836c5e0642039c94d7@tracker.bro-ids.org> #485: Error when defining vectors of records with optional fields ----------------------------+------------------------ Reporter: vallenti | Owner: robin Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * owner: jsiwek => robin Comment: > I've removed the ref counting change as I believe it was correct in its original form. Eval(0) should always return a +1 ref count. Unless git is pulling a trick on me, that change I made looks like it's in master now and didn't get reverted? > What did you see that didn't work right? An eventual seg fault (e.g. in the unit test accompanying the changeset) I think because the Eval(0) is done on a record constructor expression, returning a record val with ref count of 1 which gets assigned as a vector element without adding a ref (VectorVal::Assign says it leaves that up to the caller), then it gets unref'd right after the assignment resulting in no more refs for that val. -- Ticket URL: Bro Tracker Bro Issue Tracker From sri at basam.org Mon Aug 15 10:48:22 2011 From: sri at basam.org (sridhar basam) Date: Mon, 15 Aug 2011 13:48:22 -0400 Subject: [Bro-Dev] Connection Compressor In-Reply-To: <42467A98-42DE-4EB6-947F-C536B67A999D@icir.org> References: <20110815153630.GK76421@icir.org> <20110815154255.BEB9A36A38A@taffy.ICSI.Berkeley.EDU> <20110815155359.GM76421@icir.org> <42467A98-42DE-4EB6-947F-C536B67A999D@icir.org> Message-ID: On Mon, Aug 15, 2011 at 1:29 PM, Seth Hall wrote: > > On Aug 15, 2011, at 1:09 PM, Jim Mellander wrote: > > > We like setting record_state_history=T to record the flags that were seen > during the connection, and found that the connection compressor didn't play > nicely with that, in some cases. > > Sounds like more evidence for removing the conn compressor if possible. > Also, state history is included in the new conn logs by default. Not to pile on... I disable the CC in some cases where i need better timing on connections. With CC enabled, it discards some of the duplicate packets. I may be in the minority though, I am not using Bro as an IDS. I use it as an analytical tool in my case. Sridhar -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110815/dbd31f42/attachment.html From bro at tracker.bro-ids.org Mon Aug 15 11:44:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 15 Aug 2011 18:44:25 -0000 Subject: [Bro-Dev] #447: Bro snaplen In-Reply-To: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> References: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> Message-ID: <061.ea112d3e682eba3adb8f2cdc30ec1a15@tracker.bro-ids.org> #447: Bro snaplen ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Changes (by seth): * priority: Normal => High -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 15 11:44:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 15 Aug 2011 18:44:46 -0000 Subject: [Bro-Dev] #538: lookup_location() should handle missing database gracefully. In-Reply-To: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> References: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> Message-ID: <063.9ea9500c06779477a04fd8d7a3152bf2@tracker.bro-ids.org> #538: lookup_location() should handle missing database gracefully. ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by seth): * priority: Normal => High -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 15 11:46:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 15 Aug 2011 18:46:40 -0000 Subject: [Bro-Dev] #541: Proposal for default filter $path names In-Reply-To: <046.4f90fad6a8b6c0a92033571067d81ead@tracker.bro-ids.org> References: <046.4f90fad6a8b6c0a92033571067d81ead@tracker.bro-ids.org> Message-ID: <061.12097c242a518722724df4565a8aff96@tracker.bro-ids.org> #541: Proposal for default filter $path names ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Changes (by seth): * priority: Normal => High -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 15 12:44:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 15 Aug 2011 19:44:50 -0000 Subject: [Bro-Dev] #538: lookup_location() should handle missing database gracefully. In-Reply-To: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> References: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> Message-ID: <063.6f74ef3283cebac38ba2ae357dcfec00@tracker.bro-ids.org> #538: lookup_location() should handle missing database gracefully. ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by gregor): > The first one is probably from libgeoip. We should try to prevent libgeoip > from priting stuff on stderr.... > It appears this is hardcoded in libgeoip's code. So in order to get rid of this error message, we have set stderr to something else before calling libgeoip code and then restoring stderr. Grrr. cu gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 15 12:52:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 15 Aug 2011 19:52:06 -0000 Subject: [Bro-Dev] #538: lookup_location() should handle missing database gracefully. In-Reply-To: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> References: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> Message-ID: <063.0a69213c51516f259b70890da2ddbce7@tracker.bro-ids.org> #538: lookup_location() should handle missing database gracefully. ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): > It appears this is hardcoded in libgeoip's code. So in order to get rid of > this error message, we have set stderr to something else before calling > libgeoip code and then restoring stderr. Grrr. If we just stop trying to access fields that aren't in the currently loaded database the errors should go away. I would much rather do that than try to catch messages that libgeoip is outputting. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 15 16:00:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 15 Aug 2011 23:00:53 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.ac468123f3b83c76926fcf4f42a44359@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: seth Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by jsiwek): * owner: jsiwek => seth Comment: > Most of this ticket is done in `topic/jsiwek/reorg-followup` in the `bro` and `bro-aux` repos Should be fully done now. Seth, can you look it over? You could probably also easily fix a few of the "coverage" type tests so they'll succeed. If I touched anything that's not directly policy script related that you're not comfortable merging, just change this to a merge request and leave for Robin. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Mon Aug 15 19:40:34 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 15 Aug 2011 19:40:34 -0700 Subject: [Bro-Dev] Connection Compressor In-Reply-To: <20110815154255.BEB9A36A38A@taffy.ICSI.Berkeley.EDU> References: <20110815154255.BEB9A36A38A@taffy.ICSI.Berkeley.EDU> Message-ID: <4E49D8A2.6090105@icir.org> On 8/15/11 8:42 , Vern Paxson wrote: >> I've done some measurements as well now and can confirm this. With >> master in default mode, I also see only tiny savings in time and >> memory. > > So just to confirm: for a high-speed SYN flooding attack, it's not much > help? That was the original motivation, after all. FYI I've done some tests with a trace that contains only SYNs. The trace has 10M syns and spans approx. 20min. I've also made the trace "faster" by changing the timestamps: original (20min) * 227s 100MB with CC * 354s 320MB without compressed to 2min * 221s 780MB with * 373s 2029MB without compressed to 1min * 219s 1915MB with * 404s 5308MB without compressed to 1min, tcp_SYN_timeout set to 1sec (instead of 5sec) * 219s 398MB with * 349s 714MB without So, the compressor helps for massive SYN storms but IMHO not enough to make it worthwhile keeping it considering all the other tradeoffs mentioned in this thread. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From seth at icir.org Mon Aug 15 20:02:17 2011 From: seth at icir.org (Seth Hall) Date: Mon, 15 Aug 2011 23:02:17 -0400 Subject: [Bro-Dev] newly found segfaults from metrics framework Message-ID: <153E37B6-3303-41A6-81DB-23482BCDCD42@icir.org> After I did more work on the metrics framework today I started seeing new segfaults but they appear to derive from memory corruption. I'm seeing them all over the place now when I load the frameworks/metrics/ssl-example.bro Would someone mind taking a look through base/frameworks/metrics/main.bro and base/frameworks/metrics/non-cluster.bro in addition to playing with the ssl-example.bro script to see if I'm doing something crazy that could lead to memory corruption? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Mon Aug 15 21:10:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 16 Aug 2011 04:10:20 -0000 Subject: [Bro-Dev] #485: Error when defining vectors of records with optional fields In-Reply-To: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> References: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> Message-ID: <065.488275492dc87607e4a6e255ee5b9134@tracker.bro-ids.org> #485: Error when defining vectors of records with optional fields ----------------------------+------------------------ Reporter: vallenti | Owner: robin Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): > Unless git is pulling a trick on me, that change I made looks like it's in > master now and didn't get reverted? Indeed. Don't ask me where my change went, I must have lost it somewhere. > the caller), then it gets unref'd right after the assignment resulting in > no more refs for that val. The right fix is to remove the Unref. Eval() returns the +1 refcount, which the Assign() takes ownership of, so no Unref() necessary. To answer your earlier question, there's no documentation on the ref-counting and it's generally also not consistent unfortunately whether caller or callee are expected to do the increments/decrements. If I were granted just one wish about Bro's C++ code, I'd get rid of the explicit reference counting ... -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 15 21:12:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 16 Aug 2011 04:12:09 -0000 Subject: [Bro-Dev] #461: internal_error called for async DNS failures In-Reply-To: <046.adc7e841ed311111a107267ffb4a852b@tracker.bro-ids.org> References: <046.adc7e841ed311111a107267ffb4a852b@tracker.bro-ids.org> Message-ID: <061.bb38348b255537a16c54e099b83561c9@tracker.bro-ids.org> #461: internal_error called for async DNS failures ----------------------+-------------------- Reporter: seth | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: ----------------------+-------------------- Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [63eac6c1745b5adcddf26d8413cf0aba0b83e11e/bro]: {{{ #!CommitTicketReference repository="bro" revision="63eac6c1745b5adcddf26d8413cf0aba0b83e11e" Reclassifying more DNS manager errors. Closes #461. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 15 21:12:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 16 Aug 2011 04:12:45 -0000 Subject: [Bro-Dev] #485: Error when defining vectors of records with optional fields In-Reply-To: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> References: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> Message-ID: <065.884639c859426fe5f6e5d6a3c5eb166b@tracker.bro-ids.org> #485: Error when defining vectors of records with optional fields -----------------------------+------------------------ Reporter: vallenti | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: assigned => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 15 21:43:41 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 16 Aug 2011 04:43:41 -0000 Subject: [Bro-Dev] #559: Remove conn compressor Message-ID: <047.44f48ecee506dc9ee47f21a82ca03af5@tracker.bro-ids.org> #559: Remove conn compressor --------------------+--------------------------- Reporter: robin | Owner: robin Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Bro1.7 Component: Bro | Version: git/master Keywords: beta | --------------------+--------------------------- See subject. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon Aug 15 22:16:25 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 15 Aug 2011 22:16:25 -0700 Subject: [Bro-Dev] No scripts Message-ID: <20110816051625.GA76421@icir.org> When one starts Bro with no scripts at all (e.g., "bro -i ix1"), it doesn't do anything because it will try to read from stdin. Seems we should change that now so that it will just start processing? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Mon Aug 15 22:22:42 2011 From: seth at icir.org (Seth Hall) Date: Tue, 16 Aug 2011 01:22:42 -0400 Subject: [Bro-Dev] No scripts In-Reply-To: <20110816051625.GA76421@icir.org> References: <20110816051625.GA76421@icir.org> Message-ID: <6CDFC105-E182-4B9B-830B-85B3BBD3599F@icir.org> On Aug 16, 2011, at 1:16 AM, Robin Sommer wrote: > When one starts Bro with no scripts at all (e.g., "bro -i ix1"), it > doesn't do anything because it will try to read from stdin. Seems we > should change that now so that it will just start processing? Oh, I didn't realize this was behaving differently when sniffing a live interface. It already does this when reading a tracefile. It seems reasonable to me to make this the global behavior. What was the reason for being able to provide Bro scripts on stdin? Does anyone use this functionality? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Tue Aug 16 07:29:54 2011 From: seth at icir.org (Seth Hall) Date: Tue, 16 Aug 2011 10:29:54 -0400 Subject: [Bro-Dev] inline record type definition Message-ID: <6123DA55-3933-4925-B3D2-3B3CF17B6FF6@icir.org> I'm writing some of the builtin path_funcs and I'm running into a problem that we identified a while ago but I don't remember if we came to a conclusion on how to deal with it. When I use the following function as a $path_func for the logging framework it fails because when the rec value is passed in, the first ordinal value is almost always a time field and it tries to turn the time field into a conn_id field which obviously fails. What we need it to do is make rec$id the actual value of the $id field which is normally the third field so it needs to be done by name instead. function path_with_direction(id: ID, path: string, rec: record {id: conn_id;}): string { if ( Site::is_local_addr(rec$id$orig_h) ) return fmt("%s-%s", path, "outbound"); else return fmt("%s-%s", path, "inbound"); } Robin, do you recall if we figured out a way to work around this or if I'm doing this wrong? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Tue Aug 16 07:45:13 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 16 Aug 2011 09:45:13 -0500 Subject: [Bro-Dev] newly found segfaults from metrics framework In-Reply-To: <153E37B6-3303-41A6-81DB-23482BCDCD42@icir.org> References: <153E37B6-3303-41A6-81DB-23482BCDCD42@icir.org> Message-ID: <2D06C4B7-A038-47D4-A8AB-B19D0C932722@ncsa.illinois.edu> > After I did more work on the metrics framework today I started seeing new segfaults but they appear to derive from memory corruption. I'm seeing them all over the place now when I load the frameworks/metrics/ssl-example.bro I was playing some last night with that and think I was seeing the same thing as you running with ssl-example.bro on the 2009-M57-day11-21.trace Trying to track it down, there was one line in base/protocols/ssl/main.bro that is either directly causing the problem or possibly related. In the ssl_extension handler: c$ssl$server_name = sub_bytes(val, 6, |val|); Changing that to assign a static dummy string instead made the crashes stop. What the sub_bytes BIF was doing looked ok -- seems to be confirmed by removing that and directly assigning 'val' to $server_name still causes crashes. Don't think anything looked wrong about how that 'val' StringVal was being created in the SSL analyzer either, so it seems like it may be clobbered somewhere in between the point that it's created and when the metrics framework wants to use it. I'll see if I can find out where that is today, but if you see the same thing as above, do you have any ideas? - Jon From seth at icir.org Tue Aug 16 07:54:39 2011 From: seth at icir.org (Seth Hall) Date: Tue, 16 Aug 2011 10:54:39 -0400 Subject: [Bro-Dev] newly found segfaults from metrics framework In-Reply-To: <2D06C4B7-A038-47D4-A8AB-B19D0C932722@ncsa.illinois.edu> References: <153E37B6-3303-41A6-81DB-23482BCDCD42@icir.org> <2D06C4B7-A038-47D4-A8AB-B19D0C932722@ncsa.illinois.edu> Message-ID: On Aug 16, 2011, at 10:45 AM, Jonathan Siwek wrote: > Don't think anything looked wrong about how that 'val' StringVal was being created in the SSL analyzer either, so it seems like it may be clobbered somewhere in between the point that it's created and when the metrics framework wants to use it. I'll see if I can find out where that is today, but if you see the same thing as above, do you have any ideas? I was seeing crashes happen all over the place. It seems like memory is getting corrupted by the call to the Metrics::reset function. If you comment out that one function (in base/frameworks/metrics/non-cluster.bro) the crashes should also stop but the metrics framework also doesn't work correctly anymore! We do probably need to find an open tracefile soon that exhibits these corruption crashes. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Tue Aug 16 07:58:47 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 16 Aug 2011 07:58:47 -0700 Subject: [Bro-Dev] newly found segfaults from metrics framework In-Reply-To: References: <153E37B6-3303-41A6-81DB-23482BCDCD42@icir.org> <2D06C4B7-A038-47D4-A8AB-B19D0C932722@ncsa.illinois.edu> Message-ID: <20110816145846.GB76421@icir.org> Have you guys tried valgrind? That can often pinpoint memory corruoption right at the source. It can be slow, but prefiltering the trace for SSL might help (if that's what's triggering it). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Tue Aug 16 08:05:15 2011 From: seth at icir.org (Seth Hall) Date: Tue, 16 Aug 2011 11:05:15 -0400 Subject: [Bro-Dev] newly found segfaults from metrics framework In-Reply-To: <20110816145846.GB76421@icir.org> References: <153E37B6-3303-41A6-81DB-23482BCDCD42@icir.org> <2D06C4B7-A038-47D4-A8AB-B19D0C932722@ncsa.illinois.edu> <20110816145846.GB76421@icir.org> Message-ID: <170B1C35-D698-454E-AC23-62801D56928B@icir.org> On Aug 16, 2011, at 10:58 AM, Robin Sommer wrote: > Have you guys tried valgrind? That can often pinpoint memory > corruoption right at the source. It can be slow, but prefiltering the > trace for SSL might help (if that's what's triggering it). I don't really think that SSL is causing it, I was seeing crashes in the mime analyzer, ssl analyzer, http analyzer, and when destructing connection records. :) Using valgrind is a good idea though. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Tue Aug 16 08:36:01 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 16 Aug 2011 08:36:01 -0700 Subject: [Bro-Dev] inline record type definition In-Reply-To: <6123DA55-3933-4925-B3D2-3B3CF17B6FF6@icir.org> References: <6123DA55-3933-4925-B3D2-3B3CF17B6FF6@icir.org> Message-ID: <20110816153601.GJ76421@icir.org> On Tue, Aug 16, 2011 at 10:29 -0400, you wrote: > field which obviously fails. What we need it to do is make rec$id the > actual value of the $id field which is normally the third field so it > needs to be done by name instead. I was pretty sure that it *is* done by name. Need to take a look. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Tue Aug 16 08:48:59 2011 From: seth at icir.org (Seth Hall) Date: Tue, 16 Aug 2011 11:48:59 -0400 Subject: [Bro-Dev] inline record type definition In-Reply-To: <20110816153601.GJ76421@icir.org> References: <6123DA55-3933-4925-B3D2-3B3CF17B6FF6@icir.org> <20110816153601.GJ76421@icir.org> Message-ID: On Aug 16, 2011, at 11:36 AM, Robin Sommer wrote: > I was pretty sure that it *is* done by name. Need to take a look. That's the conclusion that I thought we came to also. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Tue Aug 16 15:47:33 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 16 Aug 2011 17:47:33 -0500 (CDT) Subject: [Bro-Dev] newly found segfaults from metrics framework In-Reply-To: <20110816145846.GB76421@icir.org> Message-ID: <860067955.5718.1313534853172.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > Have you guys tried valgrind? That can often pinpoint memory > corruoption right at the source. It can be slow, but prefiltering the > trace for SSL might help (if that's what's triggering it). Yeah, valgrind is pointing out invalid accesses in CompHash.cc, reproducible for me by running the metrics/ssl-example.bro on the 2009-M57-day11-21.trace from bro-testing filtered for "tcp port 443". Looked like it tried to stuff a string value someplace inside the CompositeHash::key buffer when it hasn't been sized to hold that much: ==27445== Invalid write of size 4 ==27445== at 0x4027C3D: memcpy (mc_replace_strmem.c:635) ==27445== by 0x82229C5: CompositeHash::SingleValHash(int, char*, BroType*, Val*, bool) const (CompHash.cc:211) ... I didn't think it could be sized right in the CompositeHash ctor since it's only got Types and not Vals to work with at that point, so I changed the computation to allocate and use a buffer on the fly if the pre-allocated one from the ctor isn't large enough to hold the key computed from the index vals. $ git diff diff --git a/src/CompHash.cc b/src/CompHash.cc index 605949b..c3c65e3 100644 --- a/src/CompHash.cc +++ b/src/CompHash.cc @@ -241,16 +241,13 @@ HashKey* CompositeHash::ComputeHash(const Val* v, int type } char* k = key; + int sz = ComputeKeySize(v, type_check); + if ( sz == 0 ) + return 0; + type_check = 0; // no need to type-check again. - if ( ! k ) - { - int sz = ComputeKeySize(v, type_check); - if ( sz == 0 ) - return 0; - + if ( ! k || sz != size ) k = reinterpret_cast(new double[sz/sizeof(double) + 1]); - type_check = 0; // no need to type-check again. - } const type_list* tl = type->Types(); That made the metrics framework happy for me, but does that seem like a reasonable approach? - Jon From seth at icir.org Tue Aug 16 17:56:00 2011 From: seth at icir.org (Seth Hall) Date: Tue, 16 Aug 2011 20:56:00 -0400 Subject: [Bro-Dev] newly found segfaults from metrics framework In-Reply-To: <860067955.5718.1313534853172.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <860067955.5718.1313534853172.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <7245C037-24DF-461D-98B7-050330A5B8BA@icir.org> On Aug 16, 2011, at 6:47 PM, Jonathan Siwek wrote: > Looked like it tried to stuff a string value someplace inside the CompositeHash::key buffer when it hasn't been sized to hold that much: Arg! I'm creating indexes too large again! (not the first time I've done this). Robin, Vern, any guidance? I generally really like how the metrics framework is implemented now and I'd like to build on top of the current model but I obviously don't want to cause overly extensive performance problems. Thanks Jon! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gregor at icir.org Tue Aug 16 22:21:47 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 16 Aug 2011 22:21:47 -0700 Subject: [Bro-Dev] New http scripts Message-ID: <4E4B4FEB.1000000@icir.org> Hi, quick question. For the new http scripts, after logging a request in http_message_done, should the c$http record be deleted and re-populated? Otherwise if the new request that gets logged only fills part of these fields, the ones from the previous request will be filled out and thus get logged twice. (Don't know whether that's an issue for other protocols as well) event http_message_done(c: connection, is_orig: bool, stat: http_message_stat) &priority = -5 { # The reply body is done so we're ready to log. if ( ! is_orig ) { Log::write(HTTP, c$http); delete c$http_state$pending[c$http_state$current_response]; } } cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From jsiwek at ncsa.illinois.edu Wed Aug 17 08:17:59 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 17 Aug 2011 10:17:59 -0500 Subject: [Bro-Dev] newly found segfaults from metrics framework In-Reply-To: <7245C037-24DF-461D-98B7-050330A5B8BA@icir.org> References: <860067955.5718.1313534853172.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <7245C037-24DF-461D-98B7-050330A5B8BA@icir.org> Message-ID: <57E610D1-42D3-4E07-9A1A-4CB9EB366772@ncsa.illinois.edu> > Arg! I'm creating indexes too large again! (not the first time I've done this). Robin, Vern, any guidance? I generally really like how the metrics framework is implemented now and I'd like to build on top of the current model but I obviously don't want to cause overly extensive performance problems. >From my reading of the CompHash code, there's a general problem with it assuming it can use a fixed hash key size in situations where it can't and you just innocently found one of those situations. Performance consideration of the metrics indexing scheme is a second thing (depends somewhat on how the first thing is fixed). - Jon From bro at tracker.bro-ids.org Wed Aug 17 09:07:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 17 Aug 2011 16:07:28 -0000 Subject: [Bro-Dev] #560: Child analyzer Init() problem Message-ID: <048.34d603b1c85589c490200b63ec47eb7f@tracker.bro-ids.org> #560: Child analyzer Init() problem ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ {{{ #!rst I think there is an inherent problem in the way analyzers and child analyzers are initialized. If analyzers are added by BuildInitialAnalyzerTree() they are not initialized at first but in a batch by calling:: root->Init(); root->InitChildren(); If an analyzer wants to add a child in its Init(), the parent doesn't know whether it needs to init this child or not. If the parent was added by ``BuildInitialAnalyzerTree()``, it *must not* ``Init()`` the child, because ``BuildInitialAnalyzerTree()`` will do it. OTOH, if the parent was added dynamically, e.g., by DPD signatures, then it *must* ``Init()`` the child. What was the reason for ``BuildInitialAnalyzerTree()`` to defer initialization of the tree until the end of the function? Initializing when they are added would solve the problem but I guess there was a good reason to do it this way. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 17 10:29:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 17 Aug 2011 17:29:53 -0000 Subject: [Bro-Dev] #541: Proposal for default filter $path names In-Reply-To: <046.4f90fad6a8b6c0a92033571067d81ead@tracker.bro-ids.org> References: <046.4f90fad6a8b6c0a92033571067d81ead@tracker.bro-ids.org> Message-ID: <061.b0aff6d88a7e799a50b0117b71c29f1c@tracker.bro-ids.org> #541: Proposal for default filter $path names -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: beta -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 17 13:06:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 17 Aug 2011 20:06:25 -0000 Subject: [Bro-Dev] #561: Bro doesn't do anything without a script Message-ID: <047.098a735a9cea48d3e7cbeb90aa759173@tracker.bro-ids.org> #561: Bro doesn't do anything without a script ------------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: Bro1.6 | Component: Bro Version: git/master | Keywords: beta ------------------------+--------------------- When reading live, Bro doesn't do anything without a script (i.e., "bro -i ix1"). We need to change that as that's going to be a common way to start Bro with the new scripts. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 17 13:06:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 17 Aug 2011 20:06:25 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.24c79fbcea48b70977b8c0ebf0dbc43d@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ----------------------------+---------------------- Reporter: gclark | Owner: robin Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ----------------------------+---------------------- Comment (by robin): I just noticed that setting awk's FS only works inside BEGIN, not later. What I was hoping to use is something like this: {{{ echo -e "#separator ;\n1;2;3;4\n" | awk '/#separator/ {FS=$2} {print $1}' }}} But that doesn't work unfortinately. > With my darwin awk the following works: > awk 'BEGIN { FS="\x09" } Yeah, but the question is how to do it if "\x09" is part of the input (and i.e., not decoded as parsw time). Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 17 14:02:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 17 Aug 2011 21:02:17 -0000 Subject: [Bro-Dev] #530: Change to hashing code In-Reply-To: <046.25b83da963246725d453aa2425e3bc37@tracker.bro-ids.org> References: <046.25b83da963246725d453aa2425e3bc37@tracker.bro-ids.org> Message-ID: <061.3325abe1a2da127e438407b5f9d02039@tracker.bro-ids.org> #530: Change to hashing code ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by jsiwek): In [d412aa9d636b22e65b11a9c4308fff7d9387fca8/bro]: {{{ #!CommitTicketReference repository="bro" revision="d412aa9d636b22e65b11a9c4308fff7d9387fca8" Fix H3 assumption of an 8-bit byte/char. The hash function was internally casting the void* data argument into an unsigned char* and then using values from that to index another internal array that's dimensioned based on the assumption of 256 values possible for an unsigned char (8-bit chars/bytes). This is probably a correct assumption most of the time, but should be safer to use the limits as defined in standard headers to get it right for the particular system/compiler. There was an unused uint8* casted variable in HashKey::HashBytes that seemed like it might have been meant to be passed to H3's hash function as an unfinished attempt to solve the 8-bit byte assumption problem, but that doesn't seem as good as taking care of that internally in H3 so users of the API are only concerned with byte sizes as reported by `sizeof`. Removing the unused variable addresses #530. Also a minor tweak to an hmac_md5 call that was casting away const from one argument (which doesn't match the prototype). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 17 14:05:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 17 Aug 2011 21:05:16 -0000 Subject: [Bro-Dev] #530: Change to hashing code In-Reply-To: <046.25b83da963246725d453aa2425e3bc37@tracker.bro-ids.org> References: <046.25b83da963246725d453aa2425e3bc37@tracker.bro-ids.org> Message-ID: <061.60777de77f791a738eb361ad17c2d20c@tracker.bro-ids.org> #530: Change to hashing code ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Changes (by jsiwek): * type: Problem => Merge Request Comment: In `topic/jsiwek/h3-byte-size` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 17 15:59:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 17 Aug 2011 22:59:33 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.53958560e88502e7c58422cb29e0316d@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ----------------------------+---------------------- Reporter: gclark | Owner: robin Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ----------------------------+---------------------- Comment (by gclark): Replying to [comment:7 robin]: > I just noticed that setting awk's FS only works inside BEGIN, not > later. What I was hoping to use is something like this: > > {{{ > echo -e "#separator ;\n1;2;3;4\n" | awk '/#separator/ {FS=$2} {print $1}' > }}} > > But that doesn't work unfortinately. Are you sure that echo statement is doing what I think you think that echo statement is doing? The above does recognize / use FS when I leave -e out of the echo. Alternatively, there's always something like: {{{ echo `head -n 1 hdr_test.log | sed 's/^#separator\s*//'` }}} which is pretty close to what I'm doing in Python; it's not especially efficient, but it is relatively simple. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 17 17:13:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 00:13:11 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.15dfaaf54257e806bee03b016b77a7db@tracker.bro-ids.org> #535: Get rid of the "net" type. ---------------------+-------------------- Reporter: seth | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ---------------------+-------------------- Comment (by jsiwek): In [1730496d128984921abe7baa94b50d8f1e4b7a19/bro]: {{{ #!CommitTicketReference repository="bro" revision="1730496d128984921abe7baa94b50d8f1e4b7a19" Remove the 'net' type from Bro (addresses #535). Incremented the serialization data format version in the process. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 17 17:14:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 00:14:40 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.3f4476336fdfb7c2dc0c0889e67a581b@tracker.bro-ids.org> #535: Get rid of the "net" type. ---------------------+-------------------- Reporter: seth | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ---------------------+-------------------- Comment (by jsiwek): In [e8335dc9fc7df4aea72178eb49c958bd5be21898/broccoli]: {{{ #!CommitTicketReference repository="broccoli" revision="e8335dc9fc7df4aea72178eb49c958bd5be21898" Remove the 'net' type from Broccoli (addresses #535). Incremented the serialization data format version in the process. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 17 17:14:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 00:14:50 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.fcd185a507cadc0062fc2f4f249a5032@tracker.bro-ids.org> #535: Get rid of the "net" type. ---------------------+-------------------- Reporter: seth | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ---------------------+-------------------- Comment (by jsiwek): In [5fbe406ecd4b52f1458a87581fe5087030231360/broccoli-ruby]: {{{ #!CommitTicketReference repository="broccoli-ruby" revision="5fbe406ecd4b52f1458a87581fe5087030231360" Remove the 'net' type from Broccoli ruby bindings (addresses #535). Also removed swig-generated files from git control. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 17 17:15:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 00:15:03 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.6db53ddc7db0e36b07a3d23c6c191dcb@tracker.bro-ids.org> #535: Get rid of the "net" type. ---------------------+-------------------- Reporter: seth | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ---------------------+-------------------- Comment (by jsiwek): In [14ade5110df7c47fb7e4d4a40bc5bfd7caac62c9/broccoli-python]: {{{ #!CommitTicketReference repository="broccoli-python" revision="14ade5110df7c47fb7e4d4a40bc5bfd7caac62c9" Remove the 'net' type from Broccoli python bindings (addresses #535). Also removed swig-generated files from git control. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 17 17:18:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 00:18:00 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.298e230434b02d02580086016b04bcee@tracker.bro-ids.org> #535: Get rid of the "net" type. ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------------+-------------------- Changes (by jsiwek): * type: Task => Merge Request Comment: 'net' type removal is done in `topic/jsiwek/remove-net-type` branches in the `bro`, `broccoli`, `broccoli-python`, and `broccoli-ruby` repos. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 17 17:37:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 00:37:13 -0000 Subject: [Bro-Dev] #561: Bro doesn't do anything without a script In-Reply-To: <047.098a735a9cea48d3e7cbeb90aa759173@tracker.bro-ids.org> References: <047.098a735a9cea48d3e7cbeb90aa759173@tracker.bro-ids.org> Message-ID: <062.c230a701faa0b22237fd4ac2811ebca8@tracker.bro-ids.org> #561: Bro doesn't do anything without a script ----------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Changes (by jsiwek): * owner: => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 17 18:38:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 01:38:36 -0000 Subject: [Bro-Dev] #561: Bro doesn't do anything without a script In-Reply-To: <047.098a735a9cea48d3e7cbeb90aa759173@tracker.bro-ids.org> References: <047.098a735a9cea48d3e7cbeb90aa759173@tracker.bro-ids.org> Message-ID: <062.421141bfbd98381d1279f4da5ea33443@tracker.bro-ids.org> #561: Bro doesn't do anything without a script ----------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: beta ----------------------+------------------------ Changes (by jsiwek): * status: assigned => closed * resolution: => fixed Comment: In [bc1c3ea28a6977999c74a46e56bc6677c461796b/bro]: {{{ #!CommitTicketReference repository="bro" revision="bc1c3ea28a6977999c74a46e56bc6677c461796b" Allow reading from interface without additional script arguments. Reading from an interface like `bro -i en0` no longer expects to start reading stdin for a script to load. Explicitly passing in '-' as an additional command line argument still allows reading a script from stdin. Closes #561 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Thu Aug 18 07:43:50 2011 From: seth at icir.org (Seth Hall) Date: Thu, 18 Aug 2011 10:43:50 -0400 Subject: [Bro-Dev] "event" signature option. Message-ID: <047E5985-8EAD-4ABC-91B3-B1F29F2D55D1@icir.org> This question is mostly for Robin, but I thought that others could benefit from seeing the answer. Why does the "event" signature option work like it does? All it seems to do is give the supplied string to the signature_match event, but you essentially get that same functionality from the signature id that is given to the same event. It seems to me like you'd provide the name of an event handler to the event option which would then be triggered when the signature matches. That event handler would also have to take the same arguments as the signature_match event. It would provide a nice way to skip over the signature->notice support that the signature framework provides since we keep using signatures for things other than direct detection of malicious activity. Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Thu Aug 18 07:57:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 14:57:13 -0000 Subject: [Bro-Dev] #544: scan.bro and hot.conn.bro need updating In-Reply-To: <047.737dcb7555d7c558c1757654428e87e0@tracker.bro-ids.org> References: <047.737dcb7555d7c558c1757654428e87e0@tracker.bro-ids.org> Message-ID: <062.5a652dc0405b8e4296556fb2d008cf9b@tracker.bro-ids.org> #544: scan.bro and hot.conn.bro need updating ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by jsiwek): Another thing that should be done for this ticket is remove those scripts from the temporary auto-documentation blacklist in doc/scripts/genDocSourcesList.sh -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Aug 18 08:12:25 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 18 Aug 2011 08:12:25 -0700 Subject: [Bro-Dev] "event" signature option. In-Reply-To: <047E5985-8EAD-4ABC-91B3-B1F29F2D55D1@icir.org> References: <047E5985-8EAD-4ABC-91B3-B1F29F2D55D1@icir.org> Message-ID: <20110818151225.GD26033@icir.org> On Thu, Aug 18, 2011 at 10:43 -0400, you wrote: > It seems to me like you'd provide the name of an event handler to the > event option which would then be triggered when the signature > matches. Yeah, that would be the right way to do it. The current scheme was driven by the Snort-to-Bro conversion we were doing back then when other signatures uses weren't really on the radar. I'm fine changing that, but perhaps we should then add another keyword like "notice" that always triggers the signature_match event (or then perhaps signature_notice). That would make it clear which signatures are triggering a notice, vs. those which are for other stuff. The reason for passing the string is mainly convinience: without it, the script layer would need a mapping id->msg for givubg the user more context in the notice. I'd keep that with the signature_notice event if we go that way, but skip for other events. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Aug 18 08:20:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 15:20:44 -0000 Subject: [Bro-Dev] #557: Remove malloc.c In-Reply-To: <047.d9416191ab6a77114a37dd25d0191abc@tracker.bro-ids.org> References: <047.d9416191ab6a77114a37dd25d0191abc@tracker.bro-ids.org> Message-ID: <062.030188ecae35d797c0db636279c92ad5@tracker.bro-ids.org> #557: Remove malloc.c ----------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Changes (by jsiwek): * owner: => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 08:27:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 15:27:00 -0000 Subject: [Bro-Dev] #557: Remove malloc.c In-Reply-To: <047.d9416191ab6a77114a37dd25d0191abc@tracker.bro-ids.org> References: <047.d9416191ab6a77114a37dd25d0191abc@tracker.bro-ids.org> Message-ID: <062.3d924e478f6acd13f55ea214fd245433@tracker.bro-ids.org> #557: Remove malloc.c ----------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by robin): On Thu, Aug 18, 2011 at 15:20 -0000, you wrote: > * owner: => jsiwek > * status: new => assigned Gilbert is actually just doing some measurements to see if there's an impact on performance. He's also comparing with tcmalloc, which seems could make quite a difference on Linux at least. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 08:35:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 15:35:40 -0000 Subject: [Bro-Dev] #557: Remove malloc.c In-Reply-To: <047.d9416191ab6a77114a37dd25d0191abc@tracker.bro-ids.org> References: <047.d9416191ab6a77114a37dd25d0191abc@tracker.bro-ids.org> Message-ID: <062.d99d9f70a2e420dd723b11b370dc7dd4@tracker.bro-ids.org> #557: Remove malloc.c ----------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by jsiwek): > Gilbert is actually just doing some measurements to see if there's an > impact on performance. He's also comparing with tcmalloc, which seems > could make quite a difference on Linux at least. Ok, leaving it be for now, then. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Aug 18 08:46:29 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 18 Aug 2011 08:46:29 -0700 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <063.53958560e88502e7c58422cb29e0316d@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> <063.53958560e88502e7c58422cb29e0316d@tracker.bro-ids.org> Message-ID: <20110818154629.GL26033@icir.org> On Wed, Aug 17, 2011 at 22:59 -0000, you wrote: > Are you sure that echo statement is doing what I think you think that echo > statement is doing? Yes, I think it does what I think it does. :-) THis prints the separator as I would expect it to: echo -e "#separator ;\n1;2;3;4\n" | awk '/#separator/ {FS=$2; print $2}' But it doesn't apply the new FS, at least on my Mac awk. Anyway, let's just pick something and go ahead. I still prefer the whitespace here[1] so why don't we always make the first line look like this {{{ #seperator \t }}} (Indeed with the standard C escaping for non-printable characters). [1] Because the escaping makes sure there's no whitespace in the separator itself. -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Aug 18 08:46:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 15:46:30 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.0e7330272814cc58801f3d73089c0dfa@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ----------------------------+---------------------- Reporter: gclark | Owner: robin Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ----------------------------+---------------------- Comment (by robin): On Wed, Aug 17, 2011 at 22:59 -0000, you wrote: > Are you sure that echo statement is doing what I think you think that echo > statement is doing? Yes, I think it does what I think it does. :-) THis prints the separator as I would expect it to: echo -e "#separator ;\n1;2;3;4\n" | awk '/#separator/ {FS=$2; print $2}' But it doesn't apply the new FS, at least on my Mac awk. Anyway, let's just pick something and go ahead. I still prefer the whitespace here[1] so why don't we always make the first line look like this {{{ #seperator \t }}} (Indeed with the standard C escaping for non-printable characters). [1] Because the escaping makes sure there's no whitespace in the separator itself. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 09:43:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 16:43:44 -0000 Subject: [Bro-Dev] #562: SWIG version trouble Message-ID: <047.7373068dbc1c1c4f3cf10a4e7e65e5bc@tracker.bro-ids.org> #562: SWIG version trouble ------------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: 1.6 | Component: Bro Version: git/master | Keywords: beta ------------------------+--------------------- Looks like we need a "minimum SWIG version" check. With 1.3.29, I get the errors below. Updating to 1.3.40 solves it. {{{ .../aux/pysubnettree/SubnetTreePYTHON_wrap.cxx: In function ?int SWIG_Python_ConvertFunctionPtr(PyObject*, void**, swig_type_info*)?: .../aux/pysubnettree/SubnetTreePYTHON_wrap.cxx:2051: error: invalid conversion from ?const char*? to ?char*? .../aux/pysubnettree/SubnetTreePYTHON_wrap.cxx: In function ?int SWIG_AsCharPtrAndSize(PyObject*, char**, size_t*, int*)?: .../aux/pysubnettree/SubnetTreePYTHON_wrap.cxx:2575: error: cannot convert ?int*? to ?Py_ssize_t*? for argument ?3? to ?int PyString_AsStringAndSize(PyObject*, char**, Py_ssize_t*)? .../aux/pysubnettree/SubnetTreePYTHON_wrap.cxx: In function ?PyObject* SubnetTree___contains____SWIG_0(SubnetTree*, char*, int)?: .../aux/pysubnettree/SubnetTreePYTHON_wrap.cxx:2849: warning: dereferencing type-punned pointer will break strict-aliasing rules .../aux/pysubnettree/SubnetTreePYTHON_wrap.cxx:2849: warning: dereferencing type-punned pointer will break strict-aliasing rules .../aux/pysubnettree/SubnetTreePYTHON_wrap.cxx:2851: warning: dereferencing type-punned pointer will break strict-aliasing rules .../aux/pysubnettree/SubnetTreePYTHON_wrap.cxx:2851: warning: dereferencing type-punned pointer will break strict-aliasing rules .../aux/pysubnettree/SubnetTreePYTHON_wrap.cxx: In function ?void SWIG_Python_FixMethods(PyMethodDef*, swig_const_info*, swig_type_info**, swig_type_info**)?: .../aux/pysubnettree/SubnetTreePYTHON_wrap.cxx:4141: error: invalid conversion from ?const char*? to ?char*? make[2]: *** [aux/broctl/aux/pysubnettree/CMakeFiles/_SubnetTree.dir/SubnetTreePYTHON_wrap.cxx.o] Error 1 make[1]: *** [aux/broctl/aux/pysubnettree/CMakeFiles/_SubnetTree.dir/all] Error 2 make: *** [all] Error 2 > swig -version SWIG Version 1.3.29 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 10:04:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:04:36 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.41fe25f9e174485f1df666793ab0c51b@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ---------------------+---------------------- Reporter: gclark | Owner: robin Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ---------------------+---------------------- Changes (by robin): * type: Merge Request => Task Comment: Changing ticket type until finalized. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 10:07:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:07:33 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest Message-ID: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ------------------------+-------------------- Reporter: robin | Type: Task Status: new | Priority: Normal Milestone: 1.6 | Component: Bro Version: git/master | ------------------------+-------------------- Haven't gotten to this yet. If somebody could take a look if this looks good, that would be helpful. {{{ commit daa9662dd3082c82c552139ed53c87a03abb44d6 Author: Don Appleman Date: Thu Jul 21 12:20:10 2011 -0500 Initial push of reST doc format of broccoli-manual }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 10:16:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:16:47 -0000 Subject: [Bro-Dev] #562: SWIG version trouble In-Reply-To: <047.7373068dbc1c1c4f3cf10a4e7e65e5bc@tracker.bro-ids.org> References: <047.7373068dbc1c1c4f3cf10a4e7e65e5bc@tracker.bro-ids.org> Message-ID: <062.4f8c662c5f14ae7e7d6a97023565b450@tracker.bro-ids.org> #562: SWIG version trouble ----------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: 1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Changes (by jsiwek): * owner: => jsiwek * status: new => assigned Comment: That swig version will work, just not with some versions of python: http://permalink.gmane.org/gmane.comp.programming.swig.devel/17840 But yeah, I'll add a check for swig < 1.3.40 and python >= 2.5 and print a useful error message for that case. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 10:21:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:21:01 -0000 Subject: [Bro-Dev] #562: SWIG version trouble In-Reply-To: <047.7373068dbc1c1c4f3cf10a4e7e65e5bc@tracker.bro-ids.org> References: <047.7373068dbc1c1c4f3cf10a4e7e65e5bc@tracker.bro-ids.org> Message-ID: <062.187b9534164c7fdaea058cdf182b036e@tracker.bro-ids.org> #562: SWIG version trouble ----------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: 1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by robin): On Thu, Aug 18, 2011 at 17:16 -0000, you wrote: > But yeah, I'll add a check for swig < 1.3.40 and python >= 2.5 and print a > useful error message for that case. Python is 2.6 on that system. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 10:22:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:22:01 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.31d2777c9c2fc5bbf19da21aa480098e@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ---------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: 1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Changes (by jsiwek): * owner: => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 10:34:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:34:11 -0000 Subject: [Bro-Dev] #530: Change to hashing code In-Reply-To: <046.25b83da963246725d453aa2425e3bc37@tracker.bro-ids.org> References: <046.25b83da963246725d453aa2425e3bc37@tracker.bro-ids.org> Message-ID: <061.47556c68f57d8fdf63b020fe50c0c4fb@tracker.bro-ids.org> #530: Change to hashing code ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Comment (by jsiwek): In [d412aa9d636b22e65b11a9c4308fff7d9387fca8/bro]: {{{ #!CommitTicketReference repository="bro" revision="d412aa9d636b22e65b11a9c4308fff7d9387fca8" Fix H3 assumption of an 8-bit byte/char. The hash function was internally casting the void* data argument into an unsigned char* and then using values from that to index another internal array that's dimensioned based on the assumption of 256 values possible for an unsigned char (8-bit chars/bytes). This is probably a correct assumption most of the time, but should be safer to use the limits as defined in standard headers to get it right for the particular system/compiler. There was an unused uint8* casted variable in HashKey::HashBytes that seemed like it might have been meant to be passed to H3's hash function as an unfinished attempt to solve the 8-bit byte assumption problem, but that doesn't seem as good as taking care of that internally in H3 so users of the API are only concerned with byte sizes as reported by `sizeof`. Removing the unused variable addresses #530. Also a minor tweak to an hmac_md5 call that was casting away const from one argument (which doesn't match the prototype). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 10:34:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:34:11 -0000 Subject: [Bro-Dev] #530: Change to hashing code In-Reply-To: <046.25b83da963246725d453aa2425e3bc37@tracker.bro-ids.org> References: <046.25b83da963246725d453aa2425e3bc37@tracker.bro-ids.org> Message-ID: <061.515d254d2d978ec4869942089a410bad@tracker.bro-ids.org> #530: Change to hashing code ----------------------------+-------------------- Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: beta ----------------------------+-------------------- Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [ecaa05073e0432f9512e70467867043a9f8cdb2d/bro]: {{{ #!CommitTicketReference repository="bro" revision="ecaa05073e0432f9512e70467867043a9f8cdb2d" Merge remote-tracking branch 'origin/topic/jsiwek/h3-byte-size' * origin/topic/jsiwek/h3-byte-size: Fix H3 assumption of an 8-bit byte/char. Closes #530. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 10:34:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:34:11 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.69c635cdefa441b82a9cff3fe98bd7ed@tracker.bro-ids.org> #535: Get rid of the "net" type. ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------------+-------------------- Comment (by jsiwek): In [1730496d128984921abe7baa94b50d8f1e4b7a19/bro]: {{{ #!CommitTicketReference repository="bro" revision="1730496d128984921abe7baa94b50d8f1e4b7a19" Remove the 'net' type from Bro (addresses #535). Incremented the serialization data format version in the process. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 10:34:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:34:11 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.70e8e4f24ccec1a327666d605b442704@tracker.bro-ids.org> #535: Get rid of the "net" type. ----------------------------+-------------------- Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: ----------------------------+-------------------- Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [27ccc77463fe928430237efc5b2944d0b9953477/bro]: {{{ #!CommitTicketReference repository="bro" revision="27ccc77463fe928430237efc5b2944d0b9953477" Merge remote-tracking branch 'origin/topic/jsiwek/remove-net-type' * origin/topic/jsiwek/remove-net-type: Remove the 'net' type from Bro (addresses #535). Closes #535. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Aug 18 10:34:21 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 18 Aug 2011 10:34:21 -0700 Subject: [Bro-Dev] "event" signature option. In-Reply-To: <20110818151225.GD26033@icir.org> References: <047E5985-8EAD-4ABC-91B3-B1F29F2D55D1@icir.org> <20110818151225.GD26033@icir.org> Message-ID: <4E4D4D1D.6050100@icir.org> >> It seems to me like you'd provide the name of an event handler to the >> event option which would then be triggered when the signature >> matches. I like that idea a lot! > I'm fine changing that, but perhaps we should then add another keyword > like "notice" that always triggers the signature_match event (or then > perhaps signature_notice). That would make it clear which signatures > are triggering a notice, vs. those which are for other stuff. I think we should then aim for the next release, since it would break current signature files. So it seems appropriate to do something disruptive like this together with all the other disruptive changes. > The reason for passing the string is mainly convinience: without it, > the script layer would need a mapping id->msg for givubg the user more > context in the notice. I'd keep that with the signature_notice event > if we go that way, but skip for other events. ACK. It's a nice way to "group" signatures together by supplying the same string message to all of them. just my 2ct Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Thu Aug 18 10:34:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:34:28 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.a40942f481df39a363e3dda1286381ba@tracker.bro-ids.org> #535: Get rid of the "net" type. ----------------------------+-------------------- Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: ----------------------------+-------------------- Comment (by jsiwek): In [e8335dc9fc7df4aea72178eb49c958bd5be21898/broccoli]: {{{ #!CommitTicketReference repository="broccoli" revision="e8335dc9fc7df4aea72178eb49c958bd5be21898" Remove the 'net' type from Broccoli (addresses #535). Incremented the serialization data format version in the process. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 10:34:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:34:29 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.66c5e83f4f6e80ec82fccf89b6b811b7@tracker.bro-ids.org> #535: Get rid of the "net" type. ----------------------------+-------------------- Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: ----------------------------+-------------------- Comment (by robin): In [3ad85cadc0cb576d768c1cd03bd95287d3edbe0d/broccoli]: {{{ #!CommitTicketReference repository="broccoli" revision="3ad85cadc0cb576d768c1cd03bd95287d3edbe0d" Merge remote-tracking branch 'origin/topic/jsiwek/remove-net-type' * origin/topic/jsiwek/remove-net-type: Remove the 'net' type from Broccoli (addresses #535). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 10:34:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:34:36 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.6982631b9787c973c4acad382bdfe347@tracker.bro-ids.org> #535: Get rid of the "net" type. ----------------------------+-------------------- Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: ----------------------------+-------------------- Comment (by robin): In [22c41b38141f2c5158769cb337a8937a9fae40b2/broccoli-python]: {{{ #!CommitTicketReference repository="broccoli-python" revision="22c41b38141f2c5158769cb337a8937a9fae40b2" Merge remote-tracking branch 'origin/topic/jsiwek/remove-net-type' * origin/topic/jsiwek/remove-net-type: Remove the 'net' type from Broccoli python bindings (addresses #535). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 10:34:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:34:36 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.79b976d9496ef313da059f34ebf5057e@tracker.bro-ids.org> #535: Get rid of the "net" type. ----------------------------+-------------------- Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: ----------------------------+-------------------- Comment (by jsiwek): In [14ade5110df7c47fb7e4d4a40bc5bfd7caac62c9/broccoli-python]: {{{ #!CommitTicketReference repository="broccoli-python" revision="14ade5110df7c47fb7e4d4a40bc5bfd7caac62c9" Remove the 'net' type from Broccoli python bindings (addresses #535). Also removed swig-generated files from git control. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 10:44:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:44:00 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.3d86f991fec9cb90178fe4f9c489ddf1@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ---------------------+---------------------- Reporter: gclark | Owner: gclark Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ---------------------+---------------------- Changes (by robin): * owner: robin => gclark -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Aug 18 10:45:09 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 18 Aug 2011 10:45:09 -0700 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <20110818154629.GL26033@icir.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> <063.53958560e88502e7c58422cb29e0316d@tracker.bro-ids.org> <20110818154629.GL26033@icir.org> Message-ID: <4E4D4FA5.4000402@icir.org> > Yes, I think it does what I think it does. :-) THis prints the > separator as I would expect it to: > > echo -e "#separator ;\n1;2;3;4\n" | awk '/#separator/ {FS=$2; print $2}' > > But it doesn't apply the new FS, at least on my Mac awk. I don't know but it works on mine. What's the problem you are facing? The new FS will only apply to the following lines, not the current one. (And you'll also need a rule for lines without /#separator/): $ echo -e "#separator ;\n1;2;3;4\n" |\ awk '/#separator/ {FS=$2; print "sep",$2} !/#separator/ { print "field3",$3} ' sep ; field3 3 field3 $ uname -a Darwin tongariro 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386 i386 $ which awk /usr/bin/awk cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Thu Aug 18 10:50:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 17:50:58 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.7e0a8b9c1da348aa2d7a3c4026505f5b@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ---------------------+---------------------- Reporter: gclark | Owner: gclark Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ---------------------+---------------------- Comment (by robin): Uh, this is weird! The first example above is what I tried yesterday. It pretty certainly didn't work then, but now it does for me as well. I'm not quite sure anymore now *where* I tried it, but it's now indeed working on Mac, FSB, and Linux for me. Weird, I had just c&p'ed it. Any way, that's good news! :) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 11:54:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 18:54:50 -0000 Subject: [Bro-Dev] #562: SWIG version trouble In-Reply-To: <047.7373068dbc1c1c4f3cf10a4e7e65e5bc@tracker.bro-ids.org> References: <047.7373068dbc1c1c4f3cf10a4e7e65e5bc@tracker.bro-ids.org> Message-ID: <062.ee78afbb64bd591bdfd86ade9bde470f@tracker.bro-ids.org> #562: SWIG version trouble ----------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: 1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by jsiwek): In [43f8a69f1644ca63fbcd11b3b9164dfd2ab7295b/pysubnettree]: {{{ #!CommitTicketReference repository="pysubnettree" revision="43f8a69f1644ca63fbcd11b3b9164dfd2ab7295b" Add check for incompatible swig+python versions. (addresses #562) Swig versions less than 1.3.40 don't work with Python headers of versions greater than or equal to 2.5, so it now prints a more useful message at configure time rather than failing at build time for combinations that won't work. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 11:57:39 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 18:57:39 -0000 Subject: [Bro-Dev] #562: SWIG version trouble In-Reply-To: <047.7373068dbc1c1c4f3cf10a4e7e65e5bc@tracker.bro-ids.org> References: <047.7373068dbc1c1c4f3cf10a4e7e65e5bc@tracker.bro-ids.org> Message-ID: <062.912c14c018f201c8361cfd091b04c621@tracker.bro-ids.org> #562: SWIG version trouble ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: 1.6 Component: pysubnettree | Version: git/master Resolution: | Keywords: beta ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request * component: Bro => pysubnettree Comment: Fixed in `topic/jsiwek/swig-version-check` in `pysubnettree` repo (others that use swig to make bindings don't run into this swig bug). -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Thu Aug 18 13:32:29 2011 From: seth at icir.org (Seth Hall) Date: Thu, 18 Aug 2011 16:32:29 -0400 Subject: [Bro-Dev] New http scripts In-Reply-To: <4E4B4FEB.1000000@icir.org> References: <4E4B4FEB.1000000@icir.org> Message-ID: On Aug 17, 2011, at 1:21 AM, Gregor Maier wrote: > quick question. For the new http scripts, after logging a request in > http_message_done, should the c$http record be deleted and re-populated? Gregor and I talked about this and he wasn't seeing the line of code that does what he wanted. :) In almost all cases, when you handle http events and you don't supply a priority for the handler, the c$http record should be set correctly even in the face of pipelined requests. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Thu Aug 18 14:11:45 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 18 Aug 2011 14:11:45 -0700 Subject: [Bro-Dev] newly found segfaults from metrics framework In-Reply-To: <860067955.5718.1313534853172.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <20110816145846.GB76421@icir.org> <860067955.5718.1313534853172.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20110818211144.GF74177@icir.org> On Tue, Aug 16, 2011 at 17:47 -0500, you wrote: > I didn't think it could be sized right in the CompositeHash ctor since > it's only got Types and not Vals to work with at that point, so I > changed the computation to allocate and use a buffer on the fly if the > pre-allocated one from the ctor isn't large enough to hold the key > computed from the index vals. I looked at this a bit more now. You're right that it can't know the size, but in general the code already accounts for that: it preallocates the buffer only if the size is static independent of the value, and otherwise allocates on the fly already. The problem is however that this check-if-static test is not done right: for strings that are part of records and optional, it may say it's static when it in fact it's not. Your patch fixes that by double-checking that the supposedly static size is indeed correct, but I think I have found a better fix: the patch below gets rid of the crash for me. It also passes the test-suite so I'm going to commit. Please put some stress on the metrics framework everybody to see if we're fine now. The patch may look a bit odd but the problem is that the semantics of "v" are overloaded for SingleTypeKeySize(): v==0 can either mean we are computing the size of a potential static buffer, or we have an unset optional record field. Robin --- a/src/CompHash.cc +++ b/src/CompHash.cc @@ -393,7 +393,7 @@ int CompositeHash::SingleTypeKeySize(BroType* bt, const Val* v, sz = SingleTypeKeySize(rt->FieldType(i), rv ? rv->Lookup(i) : 0, - type_check, sz, optional); + type_check, sz, v && optional); if ( ! sz ) return 0; } -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Thu Aug 18 14:19:48 2011 From: seth at icir.org (Seth Hall) Date: Thu, 18 Aug 2011 17:19:48 -0400 Subject: [Bro-Dev] newly found segfaults from metrics framework In-Reply-To: <20110818211144.GF74177@icir.org> References: <20110816145846.GB76421@icir.org> <860067955.5718.1313534853172.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110818211144.GF74177@icir.org> Message-ID: <82E61632-DFFB-4110-80D8-0C2A9EEA5134@icir.org> On Aug 18, 2011, at 5:11 PM, Robin Sommer wrote: > Your patch fixes that by double-checking that the supposedly static > size is indeed correct, but I think I have found a better fix: the > patch below gets rid of the crash for me. It also passes the > test-suite so I'm going to commit. Please put some stress on the > metrics framework everybody to see if we're fine now. Oh, believe me, I will. Hopefully tonight I can finish intermediate metric updates for the metrics framework so that detection time can decrease for the metrics framework on clusters. You won't have to wait for the break interval to hit in order to detect threshold's being crossed globally. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Thu Aug 18 14:32:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 21:32:34 -0000 Subject: [Bro-Dev] #562: SWIG version trouble In-Reply-To: <047.7373068dbc1c1c4f3cf10a4e7e65e5bc@tracker.bro-ids.org> References: <047.7373068dbc1c1c4f3cf10a4e7e65e5bc@tracker.bro-ids.org> Message-ID: <062.77d09d136795d1015e01d64f1ca6af55@tracker.bro-ids.org> #562: SWIG version trouble ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: 1.6 Component: pysubnettree | Version: git/master Resolution: fixed | Keywords: beta ----------------------------+------------------------ Changes (by robin): * status: assigned => closed * resolution: => fixed Comment: In [8b37d9aa5dfc48d6ccc0a2994d713c5b8d80ebc2/pysubnettree]: {{{ #!CommitTicketReference repository="pysubnettree" revision="8b37d9aa5dfc48d6ccc0a2994d713c5b8d80ebc2" Merge remote-tracking branch 'origin/topic/jsiwek/swig-version-check' * origin/topic/jsiwek/swig-version-check: Add check for incompatible swig+python versions. (addresses #562) Closes #562. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Aug 18 15:08:29 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 18 Aug 2011 15:08:29 -0700 Subject: [Bro-Dev] inline record type definition In-Reply-To: <6123DA55-3933-4925-B3D2-3B3CF17B6FF6@icir.org> References: <6123DA55-3933-4925-B3D2-3B3CF17B6FF6@icir.org> Message-ID: <20110818220829.GC75286@icir.org> On Tue, Aug 16, 2011 at 10:29 -0400, you wrote: > to a conclusion on how to deal with it. When I use the following > function as a $path_func for the logging framework it fails because > when the rec value is passed in, the first ordinal value is almost > always a time field and it tries to turn the time field into a conn_id > field which obviously fails. I can't reproduce this. Below is a script which I believe does what you're describing (in three different ways!), but it runs just fine. Anything I'm missing? Robin --------- cut ------------------------------------------------------- type Info: record { ts: time &log; id: conn_id &log; }; function path_with_direction(id: Log::ID, path: string, rec: record {id: conn_id;}): string { print id, path, rec; return path; } redef Log::default_path_func = path_with_direction; redef enum Log::ID += { MyLog }; event connection_established(c: connection) { local i: Info = [$ts=network_time(), $id=c$id]; path_with_direction(HTTP::HTTP, "manual", i); Log::write(MyLog, [$ts=network_time(), $id=c$id]); } event bro_init() { Log::create_stream(MyLog, [$columns=Info]); Log::remove_default_filter(MyLog); Log::add_filter(MyLog, [$name="default", $path_func=path_with_direction]); } -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Aug 18 15:57:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 22:57:22 -0000 Subject: [Bro-Dev] #361: Copy Bro binary only when in NFS mode In-Reply-To: <047.942404a556cf0aae549398f3c78c6a7e@tracker.bro-ids.org> References: <047.942404a556cf0aae549398f3c78c6a7e@tracker.bro-ids.org> Message-ID: <062.9d51378208048960085c00e41a42651b@tracker.bro-ids.org> #361: Copy Bro binary only when in NFS mode -------------------------+------------------------ Reporter: robin | Owner: robin Type: Task | Status: reopened Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: | Keywords: -------------------------+------------------------ Changes (by jsiwek): * status: closed => reopened * resolution: Solved/Applied => Comment: It looks like the `run-bro` script still always copies the bro binary to the `TmpExecDir`, is there a fix still waiting to be merged somewhere or want me to change that? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 16:01:04 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 18 Aug 2011 23:01:04 -0000 Subject: [Bro-Dev] #361: Copy Bro binary only when in NFS mode In-Reply-To: <047.942404a556cf0aae549398f3c78c6a7e@tracker.bro-ids.org> References: <047.942404a556cf0aae549398f3c78c6a7e@tracker.bro-ids.org> Message-ID: <062.3ed45a75fc3015f2d0206cb521b9d5dd@tracker.bro-ids.org> #361: Copy Bro binary only when in NFS mode -------------------------+------------------------ Reporter: robin | Owner: robin Type: Task | Status: reopened Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: | Keywords: -------------------------+------------------------ Comment (by robin): If it still does that, the fix must have been lost during a merge, so yes, please change. Reminds me that we should also test NFS mode to make sure it still works. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 17:08:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 19 Aug 2011 00:08:49 -0000 Subject: [Bro-Dev] #548: broctl brokenness multi-ticket In-Reply-To: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> References: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> Message-ID: <061.163e8fa5b96298f0433637cc3080c1e0@tracker.bro-ids.org> #548: broctl brokenness multi-ticket -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta -------------------------+-------------------- Comment (by robin): I tried ``check`` in both standalone and cluster mode, and it's working fine for me. Not sure what to do ... -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 18 17:10:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 19 Aug 2011 00:10:17 -0000 Subject: [Bro-Dev] #548: broctl brokenness multi-ticket In-Reply-To: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> References: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> Message-ID: <061.48a555ddc14197b752e4ec799af50ac8@tracker.bro-ids.org> #548: broctl brokenness multi-ticket -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta -------------------------+-------------------- Comment (by robin): I get another broctl error in cluster mode with `check`: {{{ > broctl check manager failed. error in /xa/robin/bro/share/bro/base/frameworks/cluster/__load__.bro, line 24: can't open frameworks/communication/listen-clear proxy-1 failed. error in /xa/robin/bro/share/bro/base/frameworks/cluster/__load__.bro, line 24: can't open frameworks/communication/listen-clear worker-1 failed. error in /xa/robin/bro/share/bro/base/frameworks/cluster/__load__.bro, line 24: can't open frameworks/communication/listen-clear }}} This is after a fresh installation with just standard changes to broctl.cfg and nodes.cfg. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 19 12:07:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 19 Aug 2011 19:07:03 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.33066445199eaccc8b0d2609feebc300@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ---------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: 1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by jsiwek): In [51ea6aa1e687439a531b88e4549576a4f3c4d205/broccoli]: {{{ #!CommitTicketReference repository="broccoli" revision="51ea6aa1e687439a531b88e4549576a4f3c4d205" Changes to new, reST-format Broccoli manual. Addresses #563 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 19 12:21:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 19 Aug 2011 19:21:32 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.667df8f3f0cec6b7f67ae907ba8e76d1@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * priority: Normal => Low * type: Task => Merge Request * component: Bro => Broccoli * milestone: 1.6 => Bro1.6 Comment: Fixing the Broccoli API Reference stuff still seems like a task for the release after next since that depends on extending doc extraction to C/C++ sources. But the manual looks pretty good now in reST and can be pulled/linked from somewhere in the `www` repo. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 19 13:27:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 19 Aug 2011 20:27:27 -0000 Subject: [Bro-Dev] #361: Copy Bro binary only when in NFS mode In-Reply-To: <047.942404a556cf0aae549398f3c78c6a7e@tracker.bro-ids.org> References: <047.942404a556cf0aae549398f3c78c6a7e@tracker.bro-ids.org> Message-ID: <062.7acff1a5e6ef604aebda2c8db1373309@tracker.bro-ids.org> #361: Copy Bro binary only when in NFS mode -------------------------+------------------------ Reporter: robin | Owner: robin Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: fixed | Keywords: -------------------------+------------------------ Changes (by jsiwek): * status: reopened => closed * resolution: => fixed Comment: In [c7ccb637ae20923f41cf04c4762d2130575b34f9/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="c7ccb637ae20923f41cf04c4762d2130575b34f9" Copy bro binary only in NFS mode (fixes #361). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 19 13:28:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 19 Aug 2011 20:28:42 -0000 Subject: [Bro-Dev] #361: Copy Bro binary only when in NFS mode In-Reply-To: <047.942404a556cf0aae549398f3c78c6a7e@tracker.bro-ids.org> References: <047.942404a556cf0aae549398f3c78c6a7e@tracker.bro-ids.org> Message-ID: <062.739481383599cc4c9799e24accc9401a@tracker.bro-ids.org> #361: Copy Bro binary only when in NFS mode -------------------------+------------------------ Reporter: robin | Owner: robin Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: fixed | Keywords: -------------------------+------------------------ Comment (by jsiwek): (This will be merged as part of #548 now) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 19 14:44:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 19 Aug 2011 21:44:49 -0000 Subject: [Bro-Dev] #554: topic/gregor/script-polishing In-Reply-To: <048.cb2dd828f6587598fb24b58176d476c2@tracker.bro-ids.org> References: <048.cb2dd828f6587598fb24b58176d476c2@tracker.bro-ids.org> Message-ID: <063.324760530fa548894e81a2f67e81d80d@tracker.bro-ids.org> #554: topic/gregor/script-polishing ----------------------------+------------------------ Reporter: gregor | Owner: seth Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by gregor): In [89d5e01d3858e25e44fd8abe5c7b251baf5c0940/bro]: {{{ #!CommitTicketReference repository="bro" revision="89d5e01d3858e25e44fd8abe5c7b251baf5c0940" Tune when c$conn is set. After discussion with Seth we now set c$conn in connection_state_remove() with priority 5, and log it with priority -5. If users want to extend c$conn before connection_state_remove, they can just create c$conn and add custom fields. Addresses: #554 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 19 15:50:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 19 Aug 2011 22:50:12 -0000 Subject: [Bro-Dev] #548: broctl brokenness multi-ticket In-Reply-To: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> References: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> Message-ID: <061.775eeac2fa80aff16cfa7cbc72e855a2@tracker.bro-ids.org> #548: broctl brokenness multi-ticket -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta -------------------------+-------------------- Comment (by jsiwek): In [a33776f3e1ec7af7f6c48d0959cae4e482ca7e77/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="a33776f3e1ec7af7f6c48d0959cae4e482ca7e77" Fixes for broctl 'check' command (addresses #548). - The 'set-bro-path' script wasn't correctly looking at the $use_installed_policies variable passed from other scripts. - The 'check' command now passes the same parameters used to run bro via the 'start' command to the 'check-config' script. Without these, the Bro instance being used to check the configuration might wait for input on stdin indefinitely or the 'check' command might incorrectly return "success" even if there's an error in the site scripts. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 19 15:54:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 19 Aug 2011 22:54:33 -0000 Subject: [Bro-Dev] #548: broctl brokenness multi-ticket In-Reply-To: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> References: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> Message-ID: <061.9722513192da7b9bb4e953407e32940f@tracker.bro-ids.org> #548: broctl brokenness multi-ticket -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta -------------------------+-------------------- Comment (by jsiwek): Not sure if that fixed your problem, Robin; I was doing standalone. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Sat Aug 20 06:28:27 2011 From: robin at icir.org (Robin Sommer) Date: Sat, 20 Aug 2011 06:28:27 -0700 Subject: [Bro-Dev] newly found segfaults from metrics framework In-Reply-To: <20110818211144.GF74177@icir.org> References: <20110816145846.GB76421@icir.org> <860067955.5718.1313534853172.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20110818211144.GF74177@icir.org> Message-ID: <20110820132827.GA24404@icir.org> On Thu, Aug 18, 2011 at 14:11 -0700, I wrote: > The patch may look a bit odd but the problem is that the semantics of Well, turns out it didn't work. While everything passed on the Linux box I had tried it on, it crashes on my laptop. I'm going to push a new version of the fix that now passes the Mac (but can't try Linux right now). I hope this does the trick, but if it this still makes problem, feel free to revert. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Mon Aug 22 07:42:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 22 Aug 2011 14:42:16 -0000 Subject: [Bro-Dev] #334: Portmapper.bro documentation and script interaction In-Reply-To: <048.dbd999fcc379921b003073dcd16b3781@tracker.bro-ids.org> References: <048.dbd999fcc379921b003073dcd16b3781@tracker.bro-ids.org> Message-ID: <063.e954218c57d48fe09cfc71fc04ff7325@tracker.bro-ids.org> #334: Portmapper.bro documentation and script interaction ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: Pushing this ticket back to the next release. I'll will probably get around to nfs/portmapper/rpc integration for the release after the next. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 22 07:43:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 22 Aug 2011 14:43:51 -0000 Subject: [Bro-Dev] #411: Non-binpac analyzer generates incorrect weird In-Reply-To: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> References: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> Message-ID: <061.52bca4a48a7a866f0fa728400914fca0@tracker.bro-ids.org> #411: Non-binpac analyzer generates incorrect weird ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Changes (by seth): * priority: Normal => High -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 22 08:04:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 22 Aug 2011 15:04:05 -0000 Subject: [Bro-Dev] #411: Non-binpac analyzer generates incorrect weird In-Reply-To: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> References: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> Message-ID: <061.9ae5d41f8c30add7f6ab12c810798b3f@tracker.bro-ids.org> #411: Non-binpac analyzer generates incorrect weird ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by gregor): You might want to check that the first of the two replies is indeed a 1xx reply. All of them are considered temporary. Which reminds me, you might want to check whether the script level handles these 1xx replies correctly. IIRC, a while ago the script layer would finish the request- reply pair when it saw a 1xx reply and then got (silently) confused when the final reply for the request came in. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 22 08:29:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 22 Aug 2011 15:29:53 -0000 Subject: [Bro-Dev] #411: Non-binpac analyzer generates incorrect weird In-Reply-To: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> References: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> Message-ID: <061.afd08e71701cb7cb5587c9bc6e8e7396@tracker.bro-ids.org> #411: Non-binpac analyzer generates incorrect weird ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): > You might want to check that the first of the two replies is indeed a 1xx > reply. All of them are considered temporary. Which reminds me, you might > want to check whether the script level handles these 1xx replies > correctly. IIRC, a while ago the script layer would finish the request- > reply pair when it saw a 1xx reply and then got (silently) confused when > the final reply for the request came in. Yep, my plan was to deal with this completely in the scripts and you're right, the scripts don't currently account for this. Since the 1xx responses do not impact further parsing of the HTTP protocol it doesn't make sense to me to try and handle them specially within the core. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 22 12:32:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 22 Aug 2011 19:32:29 -0000 Subject: [Bro-Dev] #548: broctl brokenness multi-ticket In-Reply-To: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> References: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> Message-ID: <061.3c54ca0754c7f0b23c97da7a0b129728@tracker.bro-ids.org> #548: broctl brokenness multi-ticket -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta -------------------------+-------------------- Comment (by jsiwek): In [a8df0bc2020c84cfbce76c1268603ff0cc9fdc92/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="a8df0bc2020c84cfbce76c1268603ff0cc9fdc92" Fix Config.state key capitalization inconsistencies. Dynamic state variables from broctl.dat were always read/initialized into the state dictionary and normalized to use lowercase keys, but writing/querying the state didn't always normalize to lowercase. This addresses #548, where node names with capital letters could be started, but a later 'status' command would incorrectly show it as stopped. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 22 12:39:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 22 Aug 2011 19:39:12 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: jswaro Message-ID: <20110822193912.CB47536A00C@taffy.ICSI.Berkeley.EDU> New user registration for user jswaro -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 22 23:11:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 23 Aug 2011 06:11:48 -0000 Subject: [Bro-Dev] #564: Bro silently quicks when filter for -f does not compile Message-ID: <048.2e1452a19b89aeb611c2b003cd56db4f@tracker.bro-ids.org> #564: Bro silently quicks when filter for -f does not compile ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: BETA | ---------------------+------------------------ When an invalid filter is specified with the -f option, bro just silently quits without any error message. It should print an error message! I haven't tested what happens if one tries to specify an invalid filter via a script. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 23 12:48:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 23 Aug 2011 19:48:13 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: ord Message-ID: <20110823194813.8A7D436A37C@taffy.ICSI.Berkeley.EDU> New user registration for user ord -- Bro Tracker Bro Issue Tracker From gregor at icir.org Tue Aug 23 14:53:28 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 23 Aug 2011 14:53:28 -0700 Subject: [Bro-Dev] Valgrind suppression file for Bro Message-ID: <4E542158.6040106@icir.org> Hi, I was wondering whether somebody has a valgrind suppression file that will suppress memory leaks found during initialization, etc. I'm trying to track down a possible leak in the SSL analyzer and valgrind floods me with messages from policy language parser, which I really don't care about. cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Tue Aug 23 20:14:10 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 24 Aug 2011 03:14:10 -0000 Subject: [Bro-Dev] #565: Excessive memory usage of SSL analyzer -- topic/gregor/ssl-fix Message-ID: <048.9ce4169ca7983cbc7e743f299be329ef@tracker.bro-ids.org> #565: Excessive memory usage of SSL analyzer -- topic/gregor/ssl-fix ---------------------+------------------------ Reporter: gregor | Owner: robin Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: BETA | ---------------------+------------------------ I've discovered excessive memory usage when running master. After some digging it appears that the SSL analyzer is causing it on connections with content gaps. At first I suspected a memory leak but it appears that it's in fact not a leak but just excessive usage while the connection remains established. Since there are some very long-lived imaps connections, it actually looks like a leak. (I won't rule out that it might still be a leak though). A quick fix that works is to not deliver any more data to the binpac analyzer after a content gap, which this branch does. However, I think the root of the problem is either in binpac itself or in the SSL analyzer since binpac has a NewGap() method.... -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 23 20:17:57 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 24 Aug 2011 03:17:57 -0000 Subject: [Bro-Dev] #566: Binpac analyzers and content gaps Message-ID: <048.f660012aacff96470eebea84cb8c5530@tracker.bro-ids.org> #566: Binpac analyzers and content gaps ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: BinPAC | Version: git/master Keywords: | ---------------------+------------------------ Binpac analyzers generally do not handle content gaps. In some cases content gaps might even lead to excessive memory usage (See #565). It's possible to work around this by checking whether there was a gap and not delivering any more data to binpac if there was. However, I think that maybe binpac should handle this directly. After all, there's a NewGap() method in binpac.... (We might also want to not address this issue and wait for binpac++ to solve issues like that) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 23 20:18:19 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 24 Aug 2011 03:18:19 -0000 Subject: [Bro-Dev] #565: Excessive memory usage of SSL analyzer -- topic/gregor/ssl-fix In-Reply-To: <048.9ce4169ca7983cbc7e743f299be329ef@tracker.bro-ids.org> References: <048.9ce4169ca7983cbc7e743f299be329ef@tracker.bro-ids.org> Message-ID: <063.94d40e86f0f9a8ef9bd70b8d364c0ec3@tracker.bro-ids.org> #565: Excessive memory usage of SSL analyzer -- topic/gregor/ssl-fix ----------------------------+------------------------ Reporter: gregor | Owner: robin Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: BETA ----------------------------+------------------------ Changes (by gregor): * type: Problem => Merge Request -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 23 20:35:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 24 Aug 2011 03:35:48 -0000 Subject: [Bro-Dev] #552: ASCII Log Writer precision of interval values In-Reply-To: <048.e754f597cc42f38ad361859f8fbad41c@tracker.bro-ids.org> References: <048.e754f597cc42f38ad361859f8fbad41c@tracker.bro-ids.org> Message-ID: <063.c5f83fbf3f5884b2e8d86fada71a810a@tracker.bro-ids.org> #552: ASCII Log Writer precision of interval values ------------------------------+------------------------ Reporter: gregor | Owner: Type: Feature Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: ------------------------------+------------------------ Changes (by gregor): * status: new => closed * resolution: => Solved/Applied Comment: Fix commited in fastpath (28b417381c205) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 23 22:42:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 24 Aug 2011 05:42:27 -0000 Subject: [Bro-Dev] #567: Protect/secure format strings in Reporter.cc Message-ID: <048.2977795f08466e96df0bed12ea39f590@tracker.bro-ids.org> #567: Protect/secure format strings in Reporter.cc ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ {{{ #!rst Reporter.cc uses variable argument list for formatting error message. This is problematic if somebody doesn't specify a format and the passed string contains a % character. E.g.,:: char foo[] = "foo Bar %s"; // Not a good idea reporter->Error(foo); // That's the way to do it: reporter->Error("%s", foo); We should check the code an identify each such occurrence and fix it. However, this won't prevent future errors. Given the severity of such errors (e.g., if the error message contains packet data) I would actually opt for a way to make it harder for programmers to cause such errors. E.g., by removing the methods with variable arguments from Reporter.cc and only accepting fixed strings without formatting. If formatting is desired, one can then use ``fmt()``. See also commit d3e4d5b }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 23 22:49:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 24 Aug 2011 05:49:24 -0000 Subject: [Bro-Dev] #568: Syslog analyzer should catch binpac exceptions Message-ID: <048.fa82a0608170a3c7dde2449bcee5d6d3@tracker.bro-ids.org> #568: Syslog analyzer should catch binpac exceptions ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ The syslog analyzer should catch binpac exceptions and raise a ProtocolViolation() instead of a letting the exception bubble up to Analyzer.cc were it causes a Weird() -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 25 07:47:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 25 Aug 2011 14:47:07 -0000 Subject: [Bro-Dev] #569: topic/jsiwek/mail-alarms Message-ID: <048.62ebf3ef95adf0175e881664ce36a049@tracker.bro-ids.org> #569: topic/jsiwek/mail-alarms ---------------------------+------------------------ Reporter: jsiwek | Owner: seth Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ - Adds ability for bulk alarm emails to be sent at a configurable interval (default 24hrs) when there's a Notice::mail_dest defined - Adds a bro_version() BiF for getting Bro version string from scripts -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 25 08:10:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 25 Aug 2011 15:10:36 -0000 Subject: [Bro-Dev] #570: topic/jsiwek/ascii-log-rotate-fix Message-ID: <048.7a1abf54719746c4b4bb2aaf41579a96@tracker.bro-ids.org> #570: topic/jsiwek/ascii-log-rotate-fix ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ Change ASCII writer to delay creation of log after rotation until next write -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 25 10:19:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 25 Aug 2011 17:19:45 -0000 Subject: [Bro-Dev] #554: topic/gregor/script-polishing In-Reply-To: <048.cb2dd828f6587598fb24b58176d476c2@tracker.bro-ids.org> References: <048.cb2dd828f6587598fb24b58176d476c2@tracker.bro-ids.org> Message-ID: <063.bb2b77b09ac53be9a9158dedbacc96d1@tracker.bro-ids.org> #554: topic/gregor/script-polishing ----------------------------+------------------------ Reporter: gregor | Owner: seth Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by gregor): In [89d5e01d3858e25e44fd8abe5c7b251baf5c0940/bro]: {{{ #!CommitTicketReference repository="bro" revision="89d5e01d3858e25e44fd8abe5c7b251baf5c0940" Tune when c$conn is set. After discussion with Seth we now set c$conn in connection_state_remove() with priority 5, and log it with priority -5. If users want to extend c$conn before connection_state_remove, they can just create c$conn and add custom fields. Addresses: #554 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 25 10:26:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 25 Aug 2011 17:26:11 -0000 Subject: [Bro-Dev] #554: topic/gregor/script-polishing In-Reply-To: <048.cb2dd828f6587598fb24b58176d476c2@tracker.bro-ids.org> References: <048.cb2dd828f6587598fb24b58176d476c2@tracker.bro-ids.org> Message-ID: <063.d14fa92473af4338dfedbf7c47cc14aa@tracker.bro-ids.org> #554: topic/gregor/script-polishing -----------------------------+------------------------ Reporter: gregor | Owner: seth Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: All of your stuff looked good so I committed all of it. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 25 10:26:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 25 Aug 2011 17:26:42 -0000 Subject: [Bro-Dev] #406: Tracker not sending password reset emails In-Reply-To: <046.3240e55faaa1665217f9ac50c6ab1c89@tracker.bro-ids.org> References: <046.3240e55faaa1665217f9ac50c6ab1c89@tracker.bro-ids.org> Message-ID: <061.60278e0572eee2b4ab3db245aadb13db@tracker.bro-ids.org> #406: Tracker not sending password reset emails -----------------------------+-------------------- Reporter: seth | Owner: seth Type: Problem | Status: closed Priority: Normal | Milestone: Component: TicketTracker | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: I fixed this a while ago. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 25 10:58:59 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 25 Aug 2011 17:58:59 -0000 Subject: [Bro-Dev] #564: Bro silently quicks when filter for -f does not compile In-Reply-To: <048.2e1452a19b89aeb611c2b003cd56db4f@tracker.bro-ids.org> References: <048.2e1452a19b89aeb611c2b003cd56db4f@tracker.bro-ids.org> Message-ID: <063.7dbb8f9f0149491a589296b1f18b21a2@tracker.bro-ids.org> #564: Bro silently quicks when filter for -f does not compile ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: BETA ----------------------+------------------------ Comment (by jsiwek): I ran into this, too. It looks like an error is supposed to get reported through the reporter framework but `packet-filter/main.bro` uses the `exit()` BIF which exits Bro immediately so the event that would log the error never gets drained. Changing it to use `terminate()` would get the error logged in `reporter.log` when the reporter framework is loaded (it is by default now) or stderr otherwise, but still exit with 0 status. I think a better option is to expose `Reporter::FatalError` as a `Reporter::fatal` BIF that can be used from a script to exit with non-zero status and an stderr message. I'll commit a fix that does this, but I'm open to other ideas. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 25 11:18:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 25 Aug 2011 18:18:49 -0000 Subject: [Bro-Dev] #564: Bro silently quicks when filter for -f does not compile In-Reply-To: <048.2e1452a19b89aeb611c2b003cd56db4f@tracker.bro-ids.org> References: <048.2e1452a19b89aeb611c2b003cd56db4f@tracker.bro-ids.org> Message-ID: <063.4433b39de039711e2ca19512c4fb34bd@tracker.bro-ids.org> #564: Bro silently quicks when filter for -f does not compile ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: BETA ----------------------+------------------------ Comment (by jsiwek): In [31e70e447de976f58a07f29b46c99968d601fafd/bro]: {{{ #!CommitTicketReference repository="bro" revision="31e70e447de976f58a07f29b46c99968d601fafd" Add a Reporter::fatal BIF. This can be used from a Bro script to cause Bro to exit with non-zero status and a message to stderr. Uses of the exit() BIF in scripts for terminating Bro because of an error were replaced with this one, which addresses #564. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 25 11:21:08 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 25 Aug 2011 18:21:08 -0000 Subject: [Bro-Dev] #564: Bro silently quicks when filter for -f does not compile In-Reply-To: <048.2e1452a19b89aeb611c2b003cd56db4f@tracker.bro-ids.org> References: <048.2e1452a19b89aeb611c2b003cd56db4f@tracker.bro-ids.org> Message-ID: <063.013942f36eb2aaed456ba0796913a2ca@tracker.bro-ids.org> #564: Bro silently quicks when filter for -f does not compile ----------------------------+------------------------ Reporter: gregor | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: BETA ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: Fixed in `topic/jsiwek/reporter-fatal-bif` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 25 11:36:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 25 Aug 2011 18:36:00 -0000 Subject: [Bro-Dev] #548: broctl brokenness multi-ticket In-Reply-To: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> References: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> Message-ID: <061.c1361022f39e22b47400ccff64f701be@tracker.bro-ids.org> #548: broctl brokenness multi-ticket ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: High | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Changes (by jsiwek): * type: Problem => Merge Request Comment: Here's a commit, [516f22e8367dc3dd50925bdc51063981ad53bad4/broctl], that didn't automatically make it into the ticket: {{{ Broctl fixes addressing #548 - Log rotation is disabled when using the 'process' command to analyze trace files. - Fixed 'scripts' command to print out the loaded_scripts.log produced from a test run of bro with the same configuration as the 'check' command. Full paths to scripts are now always printed (this was optional before). - Fix some inconsistent rotated-log naming: part of this was because the 'post-terminate' script was manually calling the 'archive-log' script for log rotation/postprocess and passing it the rotated-log's filename instead of the base name (e.g. 'conn.log' instead of 'conn') - Changed the 'post-terminate' script to only do manual 'archive-log' log rotation/postprocess in the event of a crash. If bro terminates normally, it will have handled the rotation/postprocess. - Change the 'mail-log' postprocessor to mail alarm.log's instead of look for mail.log. }}} Looks like all the issues mentioned in the ticket are addressed in `topic /various-updates` now, so changing to a merge request. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 25 13:45:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 25 Aug 2011 20:45:52 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.54f291e0eb7dced2cb79ae897ab66d7f@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: seth Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): In [c3fb0ea03567c7b82fb008fb42be2206eb1ed657/bro]: {{{ #!CommitTicketReference repository="bro" revision="c3fb0ea03567c7b82fb008fb42be2206eb1ed657" Reorganizing btest/policy directory to match new scripts/ organization Addresses #545 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 25 13:45:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 25 Aug 2011 20:45:52 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.e3b9cbbc3dfbce7380203457d03286c9@tracker.bro-ids.org> #545: More script reorg fallout ----------------------+------------------------ Reporter: jsiwek | Owner: seth Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): In [7b07a19cd6a0267ce889b22646ce013848fa8528/bro]: {{{ #!CommitTicketReference repository="bro" revision="7b07a19cd6a0267ce889b22646ce013848fa8528" bro scripts generated from bifs now install to $prefix/share/bro/base/ Addresses #545 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Aug 25 13:47:02 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 25 Aug 2011 20:47:02 -0000 Subject: [Bro-Dev] #545: More script reorg fallout In-Reply-To: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> References: <048.b3fc45b5ba72b101cfa1e470fa387af8@tracker.bro-ids.org> Message-ID: <063.ba9212250fd4ba4c3e021e37abd28869@tracker.bro-ids.org> #545: More script reorg fallout -----------------------------+------------------------ Reporter: jsiwek | Owner: seth Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: assigned => closed * resolution: => Solved/Applied Comment: I think I merged this ok. :) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Aug 26 07:41:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 26 Aug 2011 14:41:17 -0000 Subject: [Bro-Dev] #544: scan.bro and hot.conn.bro need updating In-Reply-To: <047.737dcb7555d7c558c1757654428e87e0@tracker.bro-ids.org> References: <047.737dcb7555d7c558c1757654428e87e0@tracker.bro-ids.org> Message-ID: <062.3053a675a007a4fedbc067a9d7fed14b@tracker.bro-ids.org> #544: scan.bro and hot.conn.bro need updating ----------------------+------------------------ Reporter: robin | Owner: seth Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Changes (by seth): * owner: => seth * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From hlin33 at illinois.edu Fri Aug 26 09:46:41 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Fri, 26 Aug 2011 11:46:41 -0500 Subject: [Bro-Dev] Hui Lin_Out of Bound Exception from flowunit Message-ID: HI, when I am writing a flowunit for dnp3 protocol, I find a weird situation. Here is the record type for the flowunit: type Dnp3_Test = record { start: uint16; len: uint8; ctrl: uint8; dest_addr: uint16; src_addr: uint16; rest: bytestring &restofdata; } &byteorder = bigendian &length= (8 + len -5 - 1) ; I am writing this type for step by step debug. There is no compiling and linking error. But when you parse the traffic dump by this protocol analyzer, binpac will generate following exception 1217561494.208541 weird: binpac exception: out_of_bound: Dnp3_Test:src_addr: 8 > 3 8 is the size of all data before "rest" the bytestring, and 3 is the size of data "start" and "len". "len" is used to define the &length of this record. It seems that after "len", you can not define extra data, such as "ctrl", "dest_addr" and doing this will generate the above exception. However, if you change the type of all data after "len" into bytestring, then the exception will not happen. But I still want to keep those data as the "uint8". Any suggestion to solve this problem? Best, Hui -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110826/30ad19be/attachment.html From seth at icir.org Fri Aug 26 10:03:54 2011 From: seth at icir.org (Seth Hall) Date: Fri, 26 Aug 2011 13:03:54 -0400 Subject: [Bro-Dev] Hui Lin_Out of Bound Exception from flowunit In-Reply-To: References: Message-ID: <2E98634D-A34E-451A-A367-505E80A82547@icir.org> On Aug 26, 2011, at 12:46 PM, Hui Lin wrote: > 1217561494.208541 weird: binpac exception: out_of_bound: Dnp3_Test:src_addr: 8 > 3 > > 8 is the size of all data before "rest" the bytestring, and 3 is the size of data "start" and "len". "len" is used to define the &length of this record. It seems that after "len", you can not define extra data, such as "ctrl", "dest_addr" and doing this will generate the above exception. However, if you change the type of all data after "len" into bytestring, then the exception will not happen. But I still want to keep those data as the "uint8". Any suggestion to solve this problem? It looks like you probably want to do: &length=(8+len) You also forgot to explain what the "5" is for and it looks like binpac tried to parse 5 bytes too far (8>3). From a more broad perspective, if you have framing around this parse unit (&length applied to a parent unit) it probably makes more sense to define this record like this: type Dnp3_Test = record { start: uint16; len: uint8; ctrl: uint8; dest_addr: uint16; src_addr: uint16; rest: bytestring &length=len; } &byteorder = bigendian; Binpac shouldn't have any problems with that as long as it can calculate the fully parsed record size based on a parent record. (to avoid complaints about incremental parsing) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Fri Aug 26 12:09:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 26 Aug 2011 19:09:34 -0000 Subject: [Bro-Dev] #571: topic/jsiwek/nested-record-coerce-fix Message-ID: <048.5755952d6bb1d678d97b2365edf175e2@tracker.bro-ids.org> #571: topic/jsiwek/nested-record-coerce-fix ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This branch is my attempt at fixing the type clash error reported by the following test case: {{{ type Foo: record { i: interval &default=1hr; s: string &optional; }; type FooContainer: record { c: count; f: Foo &optional; }; function foo_func(fc: FooContainer) { print fc; } foo_func([$c=1, $f=[$i=2hrs]]); }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From hlin33 at illinois.edu Fri Aug 26 12:12:24 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Fri, 26 Aug 2011 14:12:24 -0500 Subject: [Bro-Dev] Hui Lin_Out of Bound Exception from flowunit In-Reply-To: <2E98634D-A34E-451A-A367-505E80A82547@icir.org> References: <2E98634D-A34E-451A-A367-505E80A82547@icir.org> Message-ID: Actually, -5 comes from the meaning of the "len" which is specified in the protocol itself. I also try to add 5 on the &length to the record type. It still generate same exception. So I guess it is not the overall length of the record, but the length before "rest". Your second method to put length on the bytestring instead of record actually generate the incremental input warning. Actually, I also consider about define "rest" as a uint8[]. But I just don't know how to declare the array type in event.bif. How can I pass the array of uint8 as the input to the event handler? On Fri, Aug 26, 2011 at 12:03 PM, Seth Hall wrote: > > On Aug 26, 2011, at 12:46 PM, Hui Lin wrote: > > > 1217561494.208541 weird: binpac exception: out_of_bound: > Dnp3_Test:src_addr: 8 > 3 > > > > 8 is the size of all data before "rest" the bytestring, and 3 is the size > of data "start" and "len". "len" is used to define the &length of this > record. It seems that after "len", you can not define extra data, such as > "ctrl", "dest_addr" and doing this will generate the above exception. > However, if you change the type of all data after "len" into bytestring, > then the exception will not happen. But I still want to keep those data as > the "uint8". Any suggestion to solve this problem? > > It looks like you probably want to do: &length=(8+len) > > You also forgot to explain what the "5" is for and it looks like binpac > tried to parse 5 bytes too far (8>3). From a more broad perspective, if you > have framing around this parse unit (&length applied to a parent unit) it > probably makes more sense to define this record like this: > > type Dnp3_Test = record { > start: uint16; > len: uint8; > ctrl: uint8; > dest_addr: uint16; > src_addr: uint16; > rest: bytestring &length=len; > } &byteorder = bigendian; > > Binpac shouldn't have any problems with that as long as it can calculate > the fully parsed record size based on a parent record. (to avoid complaints > about incremental parsing) > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110826/0e6fc2dd/attachment.html From bro at tracker.bro-ids.org Fri Aug 26 12:48:41 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 26 Aug 2011 19:48:41 -0000 Subject: [Bro-Dev] #572: topic/jsiwek/filter-rotation Message-ID: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> #572: topic/jsiwek/filter-rotation ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This branch adds an optional `Log::RotationControl` to `Log::Filter` records so that defining a filter's rotation behavior can be done at the time of the filter creation/addition instead of separately redef'ing `Log::rotation_control`. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Mon Aug 29 09:57:15 2011 From: seth at icir.org (Seth Hall) Date: Mon, 29 Aug 2011 12:57:15 -0400 Subject: [Bro-Dev] Hui Lin_Out of Bound Exception from flowunit In-Reply-To: References: <2E98634D-A34E-451A-A367-505E80A82547@icir.org> Message-ID: <48B1D17C-376A-4543-A49C-CA9FE690ED0B@icir.org> On Aug 26, 2011, at 3:12 PM, Hui Lin wrote: > Actually, -5 comes from the meaning of the "len" which is specified in the protocol itself. I also try to add 5 on the &length to the record type. It still generate same exception. So I guess it is not the overall length of the record, but the length before "rest". > Your second method to put length on the bytestring instead of record actually generate the incremental input warning. Ah, ok. So this is your "top level" data structure? It just looks to me like you might be doing your field length calculation wrong. I'd try thinking about it a bit more. Alternately, if all of the messages start with "start" and "len" like you have in the record that you sent, you could make a higher level container and apply the length there to provide yourself a framing unit. Like this... type Dnp3_Head = record { start: uint16; len: uint8; # len-3 could very well be wrong since I'm probably misunderstanding the protocol. body: Dnp3_Test &length = len-3; } &byteorder=bigendian; type Dnp3_Test = record { ctrl: uint8; dest_addr: uint16; src_addr: uint16; # applying &length to the parent unit should allow us to use &restofdata rest: bytestring &restofdata; } &byteorder = bigendian; > Actually, I also consider about define "rest" as a uint8[]. But I just don't know how to declare the array type in event.bif. How can I pass the array of uint8 as the input to the event handler? I would probably try to avoid doing that unless the data makes sense as an array of ints. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Mon Aug 29 10:42:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 29 Aug 2011 17:42:58 -0000 Subject: [Bro-Dev] #573: topic/jsiwek/scp-log-pp Message-ID: <048.1cf0a773bf8d6f980a0ae32e58845a8b@tracker.bro-ids.org> #573: topic/jsiwek/scp-log-pp ---------------------------+------------------------ Reporter: jsiwek | Owner: seth Type: Merge Request | Status: new Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This branch has postprocessing function that can be used to SCP logs to a configurable set of remote hosts. This shouldn't depend on #571 and #572, but those do help make using this postprocessor more easy/intuitive. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 29 10:45:57 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 29 Aug 2011 17:45:57 -0000 Subject: [Bro-Dev] #574: More exessive memory usage Message-ID: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> #574: More exessive memory usage ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ It appears that there are still cases in which Bro uses excessive amounts of memory and doesn't release them when using the SSL-anlyzer (which is enabled by default). I couldn't pinpoint the exact reason, but it looks very similar to #565. My guess is that for some there is some dynamic length field so something in the SSL-analyzer that allocates large amounts of buffer space and which is particularly "expensive" for long-lived connections. The memory usage doesn't suddenly jump. Memory usage grows in small increments over longer periods of time (e.g., 500MB--2GB in 30--60min) and then stays high. (In addition I also noticed that Bro's memory footprint with SSL analyzer is 10%--20% higher pretty soon after startup) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 29 10:46:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 29 Aug 2011 17:46:40 -0000 Subject: [Bro-Dev] #569: topic/jsiwek/mail-alarms In-Reply-To: <048.62ebf3ef95adf0175e881664ce36a049@tracker.bro-ids.org> References: <048.62ebf3ef95adf0175e881664ce36a049@tracker.bro-ids.org> Message-ID: <063.d24511f48f7f4253c351a23adf6af565@tracker.bro-ids.org> #569: topic/jsiwek/mail-alarms ----------------------------+------------------------ Reporter: jsiwek | Owner: seth Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): I originally added this log postprocessor function inside the Notice framework/module, but while doing #573, had the idea that it might be better to generally organize postprocessors in the Log framework/module. -- Ticket URL: Bro Tracker Bro Issue Tracker From hlin33 at illinois.edu Mon Aug 29 11:13:23 2011 From: hlin33 at illinois.edu (Hui Lin) Date: Mon, 29 Aug 2011 13:13:23 -0500 Subject: [Bro-Dev] Hui Lin_Out of Bound Exception from flowunit In-Reply-To: <48B1D17C-376A-4543-A49C-CA9FE690ED0B@icir.org> References: <2E98634D-A34E-451A-A367-505E80A82547@icir.org> <48B1D17C-376A-4543-A49C-CA9FE690ED0B@icir.org> Message-ID: On Mon, Aug 29, 2011 at 11:57 AM, Seth Hall wrote: > > On Aug 26, 2011, at 3:12 PM, Hui Lin wrote: > > > Actually, -5 comes from the meaning of the "len" which is specified in > the protocol itself. I also try to add 5 on the &length to the record type. > It still generate same exception. So I guess it is not the overall length of > the record, but the length before "rest". > > Your second method to put length on the bytestring instead of record > actually generate the incremental input warning. > > Ah, ok. So this is your "top level" data structure? > > It just looks to me like you might be doing your field length calculation > wrong. I'd try thinking about it a bit more. > > Alternately, if all of the messages start with "start" and "len" like you > have in the record that you sent, you could make a higher level container > and apply the length there to provide yourself a framing unit. Like this... > > type Dnp3_Head = record { > start: uint16; > len: uint8; > # len-3 could very well be wrong since I'm probably misunderstanding > the protocol. > body: Dnp3_Test &length = len-3; > } &byteorder=bigendian; > I was doing this before actually. There is still problem when you put a uint8 data type after this high level record. However, I find that right after the int data type, you have to set a "bytestring" to eliminate this problem. I don't know why. So what I am doing is that I actually defined a dump variable which is of type bytestring with length 0 and it works. > > type Dnp3_Test = record { > ctrl: uint8; > dest_addr: uint16; > src_addr: uint16; > # applying &length to the parent unit should allow us to use > &restofdata > rest: bytestring &restofdata; > } &byteorder = bigendian; > > > Actually, I also consider about define "rest" as a uint8[]. But I just > don't know how to declare the array type in event.bif. How can I pass the > array of uint8 as the input to the event handler? > > I would probably try to avoid doing that unless the data makes sense as an > array of ints. > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > -- Hui Lin Research Assistant DEPEND Research Group, ECE Department University of Illinois at Urbana-Champaign hlin33 at illinois.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110829/9c18a8bd/attachment.html From bro at tracker.bro-ids.org Mon Aug 29 12:12:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 29 Aug 2011 19:12:36 -0000 Subject: [Bro-Dev] #562: SWIG version trouble In-Reply-To: <047.7373068dbc1c1c4f3cf10a4e7e65e5bc@tracker.bro-ids.org> References: <047.7373068dbc1c1c4f3cf10a4e7e65e5bc@tracker.bro-ids.org> Message-ID: <062.5966f8b3517a396cf2574d637901b2b7@tracker.bro-ids.org> #562: SWIG version trouble ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: 1.6 Component: pysubnettree | Version: git/master Resolution: fixed | Keywords: beta ----------------------------+------------------------ Comment (by jsiwek): In [b309bf8e63e9ab7f773f1c5338be03fe51a27a53/pysubnettree]: {{{ #!CommitTicketReference repository="pysubnettree" revision="b309bf8e63e9ab7f773f1c5338be03fe51a27a53" Fixing SWIG minimum version for Python >= 2.5 compat. The problem with Python.h switching const-ness was fixed in SWIG 1.3.30, not 1.3.40. Fixes #562 (again). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 29 13:54:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 29 Aug 2011 20:54:30 -0000 Subject: [Bro-Dev] #447: Bro snaplen In-Reply-To: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> References: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> Message-ID: <061.5908e0ecabe0923b89793da6256e2fc2@tracker.bro-ids.org> #447: Bro snaplen ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by jsiwek): In [0ae1d8a35980eacb183d058e00d8ab59786bd47e/bro]: {{{ #!CommitTicketReference repository="bro" revision="0ae1d8a35980eacb183d058e00d8ab59786bd47e" Change default snaplen to 65535, add -l/--snaplen command line option. Addresses #447. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 29 13:56:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 29 Aug 2011 20:56:09 -0000 Subject: [Bro-Dev] #447: Bro snaplen In-Reply-To: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> References: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> Message-ID: <061.75ddf4e7e6ece83c044f916835ae4867@tracker.bro-ids.org> #447: Bro snaplen ----------------------------+-------------------- Reporter: vern | Owner: Type: Merge Request | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------------+-------------------- Changes (by jsiwek): * type: Problem => Merge Request Comment: In `topic/jsiwek/snaplen` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Aug 29 14:56:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 29 Aug 2011 21:56:06 -0000 Subject: [Bro-Dev] #538: lookup_location() should handle missing database gracefully. In-Reply-To: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> References: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> Message-ID: <063.b2f682ac513c1a491a6557b3073c287f@tracker.bro-ids.org> #538: lookup_location() should handle missing database gracefully. ----------------------+------------------------ Reporter: gregor | Owner: jsiwek Type: Problem | Status: assigned Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by jsiwek): * owner: => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Tue Aug 30 07:19:17 2011 From: seth at icir.org (Seth Hall) Date: Tue, 30 Aug 2011 10:19:17 -0400 Subject: [Bro-Dev] Impact of timers/scheduled events? Message-ID: I'm not exactly sure of all the impacts of scheduling events. Are there any known issues with scheduling very large numbers of them? I know there could be the issue of memory exhaustion if you are creating references to large data structures, but assuming that I'm passing reasonably small data structures into events is it reasonable to create something on the order of 100,000 scheduled timers that expire at various times over a three hour interval? Could that cause any performance or resource exhaustion problems? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Tue Aug 30 11:21:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 30 Aug 2011 18:21:34 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.d11552fff07fb9426a67407863d474ec@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ---------------------+---------------------- Reporter: gclark | Owner: gclark Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ---------------------+---------------------- Comment (by Gilbert Clark gc355804@?>): In [3b411b69f3536bccaf22c3aba84a4c27c02226bd/bro]: {{{ #!CommitTicketReference repository="bro" revision="3b411b69f3536bccaf22c3aba84a4c27c02226bd" Updated header format (see #558) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 30 11:46:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 30 Aug 2011 18:46:49 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.6c8d1a17530fa519788b04f3eeb13f9a@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ---------------------+---------------------- Reporter: gclark | Owner: robin Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ---------------------+---------------------- Changes (by gclark): * owner: gclark => robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 30 15:18:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 30 Aug 2011 22:18:43 -0000 Subject: [Bro-Dev] #538: lookup_location() should handle missing database gracefully. In-Reply-To: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> References: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> Message-ID: <063.660116b1bbfc53a7d0b7f0c0df2b3148@tracker.bro-ids.org> #538: lookup_location() should handle missing database gracefully. ----------------------+------------------------ Reporter: gregor | Owner: jsiwek Type: Problem | Status: assigned Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): In [13a09aa488e21ec581047249460425d338afe3c4/bro]: {{{ #!CommitTicketReference repository="bro" revision="13a09aa488e21ec581047249460425d338afe3c4" Fixes for GeoIP support (addresses #538). - Missing GeoIP databases now generate warnings/errors that go through the reporter framework instead of hitting GeoIP's internal use of stderr - lookup_location now just queries for country code if the city database was not loaded, which gets rid of invalid database type errors. - lookup_location now leaves missing fields uninitialized in the returned geo_location record value. Updated existing scripts to check for initialized fields in geo_location records before use. - Fixed support for GeoIP's IPv6 API and databases }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 30 15:19:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 30 Aug 2011 22:19:29 -0000 Subject: [Bro-Dev] #538: lookup_location() should handle missing database gracefully. In-Reply-To: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> References: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> Message-ID: <063.a67f701f40a4b7313dbf9b66415d7b8c@tracker.bro-ids.org> #538: lookup_location() should handle missing database gracefully. ----------------------------+------------------------ Reporter: gregor | Owner: jsiwek Type: Merge Request | Status: assigned Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: In `topic/jsiwek/geoip-fixes` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 30 20:13:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 03:13:29 -0000 Subject: [Bro-Dev] #575: PySubnetTree does not support IPv6 prefixes Message-ID: <048.85047f5d342118b53753a8939f6f2ee0@tracker.bro-ids.org> #575: PySubnetTree does not support IPv6 prefixes -----------------------------+-------------------- Reporter: gregor | Owner: robin Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: pysubnettree | Version: Keywords: | -----------------------------+-------------------- summary says it all. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 30 21:32:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 04:32:00 -0000 Subject: [Bro-Dev] #576: Conn.log does not use well known ports for service field anymore Message-ID: <048.db8d9c5b5dbcaeb9188fac26e3c86ea5@tracker.bro-ids.org> #576: Conn.log does not use well known ports for service field anymore -----------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: BETA | -----------------------------+-------------------- The new conn.log does not use well known ports for the service field anymore. I actually found this feature quite convenient to have. Can we get it back? Maybe by adding an additional column that specifies whether the service field is derived from DPD or port based. Or we have a "dpd_service" column and a "port_service" column. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 30 21:38:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 04:38:47 -0000 Subject: [Bro-Dev] #577: DNS analyzer doesn't use ProtocolViolation / ProtocolConfirmation Message-ID: <048.29ecdb2d548b5d4be16bd03c4228b3f7@tracker.bro-ids.org> #577: DNS analyzer doesn't use ProtocolViolation / ProtocolConfirmation ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: BETA | ---------------------+------------------------ ... and thus the service field in conn.log never gets set for DNS connections. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Aug 30 21:53:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 04:53:23 -0000 Subject: [Bro-Dev] #542: SHA-1 and SHA-256 In-Reply-To: <046.6edfaf522dfff4dedd615707b9c858f1@tracker.bro-ids.org> References: <046.6edfaf522dfff4dedd615707b9c858f1@tracker.bro-ids.org> Message-ID: <061.1ad7f4b914d154f602652c58bd6abaf3@tracker.bro-ids.org> #542: SHA-1 and SHA-256 ------------------------------+---------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: language ------------------------------+---------------------- Changes (by seth): * milestone: => Bro1.7 Comment: We need this if we want to support Google's Certificate Catalog. http://googleonlinesecurity.blogspot.com/2011/04/improving-ssl- certificate-security.html -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 31 06:20:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 13:20:36 -0000 Subject: [Bro-Dev] #569: topic/jsiwek/mail-alarms In-Reply-To: <048.62ebf3ef95adf0175e881664ce36a049@tracker.bro-ids.org> References: <048.62ebf3ef95adf0175e881664ce36a049@tracker.bro-ids.org> Message-ID: <063.2ba33a33ba93a7e099514c7e658b68a3@tracker.bro-ids.org> #569: topic/jsiwek/mail-alarms -----------------------------+------------------------ Reporter: jsiwek | Owner: seth Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Merged. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 31 06:28:26 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 13:28:26 -0000 Subject: [Bro-Dev] #573: topic/jsiwek/scp-log-pp In-Reply-To: <048.1cf0a773bf8d6f980a0ae32e58845a8b@tracker.bro-ids.org> References: <048.1cf0a773bf8d6f980a0ae32e58845a8b@tracker.bro-ids.org> Message-ID: <063.b6391caff420e17f0420074fb28fddf8@tracker.bro-ids.org> #573: topic/jsiwek/scp-log-pp -----------------------------+------------------------ Reporter: jsiwek | Owner: seth Type: Merge Request | Status: closed Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Done -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 31 07:23:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 14:23:17 -0000 Subject: [Bro-Dev] #565: Excessive memory usage of SSL analyzer -- topic/gregor/ssl-fix In-Reply-To: <048.9ce4169ca7983cbc7e743f299be329ef@tracker.bro-ids.org> References: <048.9ce4169ca7983cbc7e743f299be329ef@tracker.bro-ids.org> Message-ID: <063.9a91d36dad006b200c7c6429a896e144@tracker.bro-ids.org> #565: Excessive memory usage of SSL analyzer -- topic/gregor/ssl-fix -----------------------------+------------------------ Reporter: gregor | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: BETA -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Merged, thanks. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 31 07:26:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 14:26:22 -0000 Subject: [Bro-Dev] #576: Conn.log does not use well known ports for service field anymore In-Reply-To: <048.db8d9c5b5dbcaeb9188fac26e3c86ea5@tracker.bro-ids.org> References: <048.db8d9c5b5dbcaeb9188fac26e3c86ea5@tracker.bro-ids.org> Message-ID: <063.821d4b653ebbfdc56d9f467970663d43@tracker.bro-ids.org> #576: Conn.log does not use well known ports for service field anymore ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: BETA ------------------------------+-------------------- Comment (by seth): > The new conn.log does not use well known ports for the service field > anymore. I actually found this feature quite convenient to have. Can we > get it back? Maybe by adding an additional column that specifies whether > the service field is derived from DPD or port based. Or we have a > "dpd_service" column and a "port_service" column. Would it work for you if that was just included as an extension script instead of being included in the base script? -- Ticket URL: Bro Tracker Bro Issue Tracker From mcholste at gmail.com Wed Aug 31 07:52:12 2011 From: mcholste at gmail.com (Martin Holste) Date: Wed, 31 Aug 2011 09:52:12 -0500 Subject: [Bro-Dev] Update on log management Message-ID: As per Gilbert's request, I'm moving this discussion over to the dev list. Last night I sent out a small set of scripts to the list which I've been using to help leverage Bro logs with other systems, namely by sending them to our central syslog as well as logging the data to a DB and putting a basic web query interface on the data. Gilbert, you said that you've got a similar project for log snarfing going, but I am a bit of a n00b on your git system and I can't find your topic branch. Can you provide a link to it? Can you guys please lay out what the basic goals, roadmap, and timeline are for all Bro output? Please be clear about things you would *like* to do versus things which will actually be production ready in the next few months. Thanks, Martin From seth at icir.org Wed Aug 31 08:29:36 2011 From: seth at icir.org (Seth Hall) Date: Wed, 31 Aug 2011 11:29:36 -0400 Subject: [Bro-Dev] Update on log management In-Reply-To: References: Message-ID: <12DFE60E-F86E-40CE-8A24-6F3AE0C56193@icir.org> On Aug 31, 2011, at 10:52 AM, Martin Holste wrote: > Can you guys please lay out what the basic goals, roadmap, and > timeline are for all Bro output? Please be clear about things you > would *like* to do versus things which will actually be production > ready in the next few months. Right now we have the logging framework's writer plugin API ready to go. We don't have any concrete plans for when or what writer plugins we are going to write but personally I'm very interested in a PostgreSQL plugin and you've gotten me interested in getting a syslog writer created if TCP syslog does in fact allow for extremely long messages. To be honest it pained me a little bit when I saw your code to insert the ascii logs into a database (however I can't fault anyone for doing things out of operational necessity). The benefits to outputting logs directly to a database from Bro is that you get to take advantage of logging framework features. You can filter out log lines that you don't want to go to certain outputs or remove certain fields. For instance, if we had a PostgreSQL writer plugin and you only wanted SSL sessions that didn't successfully validate put into the database, you could do this (keeping in mind that due to the lack of a postgresql writer, this code very much does not work).... event bro_init() { Log::add_filter(SSL::SSL, [$name="failed_to_pg", $writer=POSTGRESQL_WRITER, $path="ssl_invalid_cert_session", # This equates to the table name in postgresql $pred(rec: SSL::Info) = { return (rec$validation_status != "ok"); }]); } All of the SSL logs would still be written to the normal (ascii by default) logs but all of the sessions using invalid certificates would also be sent to postgresql. In terms of future plans we're really just at the point where we need more writer plugins, most of the rest of the code is finished. Does that answer your questions? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From mcholste at gmail.com Wed Aug 31 08:40:39 2011 From: mcholste at gmail.com (Martin Holste) Date: Wed, 31 Aug 2011 10:40:39 -0500 Subject: [Bro-Dev] validate-certs.bro error Message-ID: Fatal from git rev 005b150: 1314803689.614709 internal error in /usr/local/bro-005b150/share/bro/policy/protocols/ssl/validate-certs.bro, line 20: field value missing (SSL::c$ssl) Aborted Looks like a basic case where a sub-hash key is being accessed without verify the parent hash key exists. I think I fixed with: if ( !c?$ssl || !c$ssl?$cert || !c$ssl?$cert_chain ) From seth at icir.org Wed Aug 31 08:54:22 2011 From: seth at icir.org (Seth Hall) Date: Wed, 31 Aug 2011 11:54:22 -0400 Subject: [Bro-Dev] validate-certs.bro error In-Reply-To: References: Message-ID: On Aug 31, 2011, at 11:40 AM, Martin Holste wrote: > Fatal from git rev 005b150: > 1314803689.614709 internal error in > /usr/local/bro-005b150/share/bro/policy/protocols/ssl/validate-certs.bro, > line 20: field value missing (SSL::c$ssl) Sorry about that, I've had it fixed locally for a few days but I'm trying to finish up some automatic notice deduplication code before I commit it because I'm modifying some other SSL scripts to use the notice dedup-ing. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From mcholste at gmail.com Wed Aug 31 09:10:56 2011 From: mcholste at gmail.com (Martin Holste) Date: Wed, 31 Aug 2011 11:10:56 -0500 Subject: [Bro-Dev] Update on log management In-Reply-To: <12DFE60E-F86E-40CE-8A24-6F3AE0C56193@icir.org> References: <12DFE60E-F86E-40CE-8A24-6F3AE0C56193@icir.org> Message-ID: > To be honest it pained me a little bit when I saw your code to insert the ascii logs into a database (however I can't fault anyone for doing things out of operational necessity). LOL! Yes, this is very much out of operational necessity. I would love to hear how other incident response teams are using Bro in production (beyond PoC setups), as right now, it doesn't integrate with anything enterprise, and therefore becomes much more limited than it should be given all of its sophisticated abilities. > The benefits to outputting logs directly to a database from Bro is that you get to take advantage of logging framework features. Yes, we all want this to be integrated inside Bro. The reason that my script snarfer works efficiently, ugly and inelegant as it is, is because you are guaranteed that the effort and errors of dealing with the output from Bro is not Bro's problem. This is critical for things like database errors, network lag, and CPU overhead. Please, please make sure that you make these proposed writers async and in their own thread. Do not make the same mistakes Snort made by having the output in the main thread. Output is a very easy task to delegate to a child thread or fork, and because Bro does an incredible amount of logging, you don't want to be wasting precious CPU cycles in the network detection thread on output. My single Bro instance is overwhelmed on any traffic over 200 Mb/sec on a decent machine. > > event bro_init() > ? ? ? ?{ > ? ? ? ?Log::add_filter(SSL::SSL, [$name="failed_to_pg", > ? ? ? ? ? ? ? ?$writer=POSTGRESQL_WRITER, > ? ? ? ? ? ? ? ?$path="ssl_invalid_cert_session", ? # This equates to the table name in postgresql > ? ? ? ? ? ? ? ?$pred(rec: SSL::Info) = { return (rec$validation_status != "ok"); }]); > ? ? ? ?} > > All of the SSL logs would still be written to the normal (ascii by default) logs but all of the sessions using invalid certificates would also be sent to postgresql. > Yep, this is good stuff. I'd settle for this working on files, as I can limp along on my scripts until the Pg plugin is ready. > In terms of future plans we're really just at the point where we need more writer plugins, most of the rest of the code is finished. ?Does that answer your questions? > Almost. The next question is: and then what? So let's say you've got all of this data in Pg. Do you have anyone working on a frontend for this? Or is accessing the data Bro puts out considered wholly out-of-scope for the Bro project? Again, I'd love to hear from other IR team leads as to how they are using this data to either trigger or supplement investigations. From bro at tracker.bro-ids.org Wed Aug 31 09:14:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 16:14:13 -0000 Subject: [Bro-Dev] #576: Conn.log does not use well known ports for service field anymore In-Reply-To: <048.db8d9c5b5dbcaeb9188fac26e3c86ea5@tracker.bro-ids.org> References: <048.db8d9c5b5dbcaeb9188fac26e3c86ea5@tracker.bro-ids.org> Message-ID: <063.2ec1a583e81a25212dc4eae8740428f0@tracker.bro-ids.org> #576: Conn.log does not use well known ports for service field anymore ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: BETA ------------------------------+-------------------- Comment (by gregor): works for me. But that would be something to add to the "1.5 to 2.0 transition HOWTO" (do we have one already or do we have at least a list with bullet points for the HOWTO?) -- Ticket URL: Bro Tracker Bro Issue Tracker From mcholste at gmail.com Wed Aug 31 09:14:25 2011 From: mcholste at gmail.com (Martin Holste) Date: Wed, 31 Aug 2011 11:14:25 -0500 Subject: [Bro-Dev] PF_RING cluster ID Message-ID: PF_RING's greatest feature is the ability to do per-flow clustering so that many sniffers can run in parallel on their own CPU and each receive a portion of the traffic which has been load-balanced. Snort and Suricata both support this, and we've come to rely on it as we don't have access to hardware load balancing. Is there a way to do this in Bro? From seth at icir.org Wed Aug 31 09:23:39 2011 From: seth at icir.org (Seth Hall) Date: Wed, 31 Aug 2011 12:23:39 -0400 Subject: [Bro-Dev] Update on log management In-Reply-To: References: <12DFE60E-F86E-40CE-8A24-6F3AE0C56193@icir.org> Message-ID: On Aug 31, 2011, at 12:10 PM, Martin Holste wrote: >> The benefits to outputting logs directly to a database from Bro is that you get to take advantage of logging framework features. > > Yes, we all want this to be integrated inside Bro. The reason that my > script snarfer works efficiently, ugly and inelegant as it is, is > because you are guaranteed that the effort and errors of dealing with > the output from Bro is not Bro's problem. The "normal" Bro deployment is as a cluster at this point anyway where the manager is dedicated to notice handling and logging (as you've noticed, standalone instances basically suck for anything over 80Mbps). Also, Gilbert has been spending the summer threading the logging framework and I *think* his branch is probably close to being integrated. We basically planned on threading the logging framework from the start for all of the reasons that you mentioned. :) I started inserting Bro logs into PostgreSQL a long time ago at OSU too with my bro-dblogger project... https://github.com/sethhall/bro-dblogger If you read the README for that project closely, you can see way back then that I was already heading down the path that we ended up going down with the new logging framework. ;) > Yep, this is good stuff. I'd settle for this working on files, as I > can limp along on my scripts until the Pg plugin is ready. The rest of the filter actually works already except for the $writer. :) >> In terms of future plans we're really just at the point where we need more writer plugins, most of the rest of the code is finished. Does that answer your questions? > > Almost. The next question is: and then what? So let's say you've got > all of this data in Pg. Do you have anyone working on a frontend for > this? Or is accessing the data Bro puts out considered wholly > out-of-scope for the Bro project? We've been poking around at various people and places trying to figure out what a Bro interface would look like and do. I suspect we aren't too far off from movement in this area, but we have no plans yet. > Again, I'd love to hear from other > IR team leads as to how they are using this data to either trigger or > supplement investigations. I'm hoping to have a couple of people talk about this specifically at the Bro workshop. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Wed Aug 31 09:28:19 2011 From: seth at icir.org (Seth Hall) Date: Wed, 31 Aug 2011 12:28:19 -0400 Subject: [Bro-Dev] PF_RING cluster ID In-Reply-To: References: Message-ID: On Aug 31, 2011, at 12:14 PM, Martin Holste wrote: > PF_RING's greatest feature is the ability to do per-flow clustering so > that many sniffers can run in parallel on their own CPU and each > receive a portion of the traffic which has been load-balanced. Snort > and Suricata both support this, and we've come to rely on it as we > don't have access to hardware load balancing. Is there a way to do > this in Bro? Use the cluster deployment in BroControl and assign each of the virtual interfaces to a separate worker node. Since you're running the git master though, keep in mind that the broctl master that is currently checked out has some problems. I'll work on merging in our fixed branch today and I'll reply again once it's updated. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Wed Aug 31 09:35:10 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 16:35:10 -0000 Subject: [Bro-Dev] #538: lookup_location() should handle missing database gracefully. In-Reply-To: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> References: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> Message-ID: <063.8c3cf9050c785c3ff8be0424ea137e61@tracker.bro-ids.org> #538: lookup_location() should handle missing database gracefully. ----------------------------+------------------------ Reporter: gregor | Owner: jsiwek Type: Merge Request | Status: assigned Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by seth): In [c67c1e8ba6ebc5f67361bda589cdb2d197da51d1/bro]: {{{ #!CommitTicketReference repository="bro" revision="c67c1e8ba6ebc5f67361bda589cdb2d197da51d1" Merge remote-tracking branch 'origin/topic/jsiwek/geoip-fixes' * origin/topic/jsiwek/geoip-fixes: Fixes for GeoIP support (addresses #538). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 31 09:35:10 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 16:35:10 -0000 Subject: [Bro-Dev] #538: lookup_location() should handle missing database gracefully. In-Reply-To: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> References: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> Message-ID: <063.5712a12a043eb1d8e5f88e4fe43f3211@tracker.bro-ids.org> #538: lookup_location() should handle missing database gracefully. ----------------------------+------------------------ Reporter: gregor | Owner: jsiwek Type: Merge Request | Status: assigned Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): In [13a09aa488e21ec581047249460425d338afe3c4/bro]: {{{ #!CommitTicketReference repository="bro" revision="13a09aa488e21ec581047249460425d338afe3c4" Fixes for GeoIP support (addresses #538). - Missing GeoIP databases now generate warnings/errors that go through the reporter framework instead of hitting GeoIP's internal use of stderr - lookup_location now just queries for country code if the city database was not loaded, which gets rid of invalid database type errors. - lookup_location now leaves missing fields uninitialized in the returned geo_location record value. Updated existing scripts to check for initialized fields in geo_location records before use. - Fixed support for GeoIP's IPv6 API and databases }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 31 09:37:19 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 16:37:19 -0000 Subject: [Bro-Dev] #538: lookup_location() should handle missing database gracefully. In-Reply-To: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> References: <048.0c35cda9db50886179465ab453bb7386@tracker.bro-ids.org> Message-ID: <063.52cfe19051f0eb22371661a0bd82618c@tracker.bro-ids.org> #538: lookup_location() should handle missing database gracefully. -----------------------------+------------------------ Reporter: gregor | Owner: jsiwek Type: Merge Request | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: assigned => closed * resolution: => Solved/Applied Comment: Merged. -- Ticket URL: Bro Tracker Bro Issue Tracker From gc355804 at ohio.edu Wed Aug 31 09:44:17 2011 From: gc355804 at ohio.edu (G. Clark) Date: Wed, 31 Aug 2011 09:44:17 -0700 Subject: [Bro-Dev] Update on log management In-Reply-To: References: <12DFE60E-F86E-40CE-8A24-6F3AE0C56193@icir.org> Message-ID: <4E5E64E1.4020806@ohio.edu> Martin: http://git.bro-ids.org/bro.git/tree/refs/heads/topic/gilbert/log-util:/aux/log-util should take you to the python library when opened in a browser. Keep in mind, though, that this code won't be practically usable until we finish fixing up the ASCII header and I revise that code accordingly; I used a prototype header format to write / test this code. See bro-logtool in that directory for a simple script I've been using to play with the library. Also, re: > The "normal" Bro deployment is as a cluster at this point anyway where the manager is dedicated to notice handling and logging (as you've noticed, standalone instances basically suck for anything over 80Mbps). Also, Gilbert has been spending the summer threading the logging framework and I *think* his branch is probably close to being integrated. We basically planned on threading the logging framework from the start for all of the reasons that you mentioned. :) Working revision is in topic/gilbert/log-threads. There's some work left to do here, but the current code does seem to pass basic testing. --Gilbert -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6010 bytes Desc: S/MIME Cryptographic Signature Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110831/ab84909a/attachment-0001.bin From bro at tracker.bro-ids.org Wed Aug 31 09:49:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 16:49:36 -0000 Subject: [Bro-Dev] #361: Copy Bro binary only when in NFS mode In-Reply-To: <047.942404a556cf0aae549398f3c78c6a7e@tracker.bro-ids.org> References: <047.942404a556cf0aae549398f3c78c6a7e@tracker.bro-ids.org> Message-ID: <062.07dda56159976edd54ecff030ac8712f@tracker.bro-ids.org> #361: Copy Bro binary only when in NFS mode -------------------------+------------------------ Reporter: robin | Owner: robin Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: fixed | Keywords: -------------------------+------------------------ Comment (by jsiwek): In [c7ccb637ae20923f41cf04c4762d2130575b34f9/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="c7ccb637ae20923f41cf04c4762d2130575b34f9" Copy bro binary only in NFS mode (fixes #361). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 31 09:49:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 16:49:36 -0000 Subject: [Bro-Dev] #548: broctl brokenness multi-ticket In-Reply-To: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> References: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> Message-ID: <061.e9599624be55c8baf8563d776cd61472@tracker.bro-ids.org> #548: broctl brokenness multi-ticket ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: High | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Comment (by jsiwek): In [a33776f3e1ec7af7f6c48d0959cae4e482ca7e77/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="a33776f3e1ec7af7f6c48d0959cae4e482ca7e77" Fixes for broctl 'check' command (addresses #548). - The 'set-bro-path' script wasn't correctly looking at the $use_installed_policies variable passed from other scripts. - The 'check' command now passes the same parameters used to run bro via the 'start' command to the 'check-config' script. Without these, the Bro instance being used to check the configuration might wait for input on stdin indefinitely or the 'check' command might incorrectly return "success" even if there's an error in the site scripts. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 31 09:49:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 16:49:36 -0000 Subject: [Bro-Dev] #548: broctl brokenness multi-ticket In-Reply-To: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> References: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> Message-ID: <061.0db205b3429789725c9c3b9ba72464ec@tracker.bro-ids.org> #548: broctl brokenness multi-ticket ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: High | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Comment (by jsiwek): In [a8df0bc2020c84cfbce76c1268603ff0cc9fdc92/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="a8df0bc2020c84cfbce76c1268603ff0cc9fdc92" Fix Config.state key capitalization inconsistencies. Dynamic state variables from broctl.dat were always read/initialized into the state dictionary and normalized to use lowercase keys, but writing/querying the state didn't always normalize to lowercase. This addresses #548, where node names with capital letters could be started, but a later 'status' command would incorrectly show it as stopped. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From mcholste at gmail.com Wed Aug 31 09:52:35 2011 From: mcholste at gmail.com (Martin Holste) Date: Wed, 31 Aug 2011 11:52:35 -0500 Subject: [Bro-Dev] PF_RING cluster ID In-Reply-To: References: Message-ID: > Use the cluster deployment in BroControl and assign each of the virtual interfaces to a separate worker node. ?Since you're running the git master though, keep in mind that the broctl master that is currently checked out has some problems. ?I'll work on merging in our fixed branch today and I'll reply again once it's updated. What I'm confused about is where the docs refer to a "node." Can this be just another instance of Bro, with all instances on the same machine? I don't want to run Bro on separate machines, I want to run a lot of Bros on the same machine, and I want each instance to see only some of the flows. From gregor at icir.org Wed Aug 31 09:54:43 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 31 Aug 2011 09:54:43 -0700 Subject: [Bro-Dev] PF_RING cluster ID In-Reply-To: References: Message-ID: <4E5E6753.7090404@icir.org> On 8/31/11 9:28 , Seth Hall wrote: > > On Aug 31, 2011, at 12:14 PM, Martin Holste wrote: > >> PF_RING's greatest feature is the ability to do per-flow clustering so >> that many sniffers can run in parallel on their own CPU and each >> receive a portion of the traffic which has been load-balanced. Snort >> and Suricata both support this, and we've come to rely on it as we >> don't have access to hardware load balancing. Is there a way to do >> this in Bro? > > Use the cluster deployment in BroControl and assign each of the virtual interfaces to a separate worker node. Since you're running the git master though, keep in mind that the broctl master that is currently checked out has some problems. I'll work on merging in our fixed branch today and I'll reply again once it's updated. I think it doesn't work like that :-( It appears that PF_RINGs clustering doesn't use virtual interfaces for its clustering. Bro currently uses libpcap exclusively for packet capture and AFAIK libpcap doesn't use PF_RING clustering. I guess there are two possible ways to get this support to Bro: a) directly: add a new PktSrc() that uses the PF_RING API directly and add the command line argument to specify the clustering information. BroControl will likely need to be updated as well. If somebody does that, than the same framework can probably be used for Endace's load-balancing packet capture as well. b) use a modified libpcap: modify libpcap so that it uses PF_RING clusters and add some magic to the way libpcap parses interface names passed to it. Say that the monitoring interface it eth0, then libpcap could interpret an interface like "eth0::1" (*) as the first cluster stream from eth0 and setup PF_RING accordingly. You'd probably need a small external program that sets up the cluster then. Although b) is a little hacky, it would also enable any libpcap based application to make use of PF_RING and it likely quicker to integrate with Bro and BroControl (since neither Bro nor BroControl need to be changed for that) (*) I guess one could even encode even more information into this "virtual" interface name, like the cluster id: "eth0::CLUSTERID::STREAM" cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Wed Aug 31 09:58:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 16:58:34 -0000 Subject: [Bro-Dev] #578: Add ICMPv6 support to Bro Message-ID: <048.1e6e618d9d8a166299c6c8582e9c8511@tracker.bro-ids.org> #578: Add ICMPv6 support to Bro ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Keywords: IPv6 | ---------------------+------------------------ Bro currently does not handle ICMPv6 at all (one reason being that Bro ignore IP protocol 58 which is ipv6-icmp) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 31 10:00:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 17:00:40 -0000 Subject: [Bro-Dev] #576: Conn.log does not use well known ports for service field anymore In-Reply-To: <048.db8d9c5b5dbcaeb9188fac26e3c86ea5@tracker.bro-ids.org> References: <048.db8d9c5b5dbcaeb9188fac26e3c86ea5@tracker.bro-ids.org> Message-ID: <063.a23298ee3c1b6e58c3b04c074c8bf52f@tracker.bro-ids.org> #576: Conn.log does not use well known ports for service field anymore ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: BETA ------------------------------+-------------------- Comment (by jsiwek): > But that would be something to add to the "1.5 to 2.0 transition HOWTO" > (do we have one already or do we have at least a list with bullet points for the HOWTO?) We have ticket #510 as a reminder and separately people brainstormed bullet points on the bro-blue wiki. I'll consolidate that into the ticket shortly. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 31 10:02:26 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 17:02:26 -0000 Subject: [Bro-Dev] #548: broctl brokenness multi-ticket In-Reply-To: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> References: <046.3b1116424af43925bf614843d6166160@tracker.bro-ids.org> Message-ID: <061.5054bb8f37a616b22c3d7dbc57af434f@tracker.bro-ids.org> #548: broctl brokenness multi-ticket -----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: closed Priority: High | Milestone: Bro1.6 Component: BroControl | Version: Resolution: Solved/Applied | Keywords: beta -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Merged. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Aug 31 10:03:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 17:03:25 -0000 Subject: [Bro-Dev] #578: Add ICMPv6 support to Bro In-Reply-To: <048.1e6e618d9d8a166299c6c8582e9c8511@tracker.bro-ids.org> References: <048.1e6e618d9d8a166299c6c8582e9c8511@tracker.bro-ids.org> Message-ID: <063.03a11adaaed5a001c3293b86a0e7ee26@tracker.bro-ids.org> #578: Add ICMPv6 support to Bro ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Resolution: | Keywords: IPv6 ----------------------+------------------------ Comment (by seth): > Bro currently does not handle ICMPv6 at all (one reason being that Bro > ignore IP protocol 58 which is ipv6-icmp) What is the status of Matti's branch that he was working on? -- Ticket URL: Bro Tracker Bro Issue Tracker From mcholste at gmail.com Wed Aug 31 10:44:20 2011 From: mcholste at gmail.com (Martin Holste) Date: Wed, 31 Aug 2011 12:44:20 -0500 Subject: [Bro-Dev] Update on log management In-Reply-To: References: <12DFE60E-F86E-40CE-8A24-6F3AE0C56193@icir.org> Message-ID: > The "normal" Bro deployment is as a cluster at this point anyway where the manager is dedicated to notice handling and logging (as you've noticed, standalone instances basically suck for anything over 80Mbps). ?Also, Gilbert has been spending the summer threading the logging framework and I *think* his branch is probably close to being integrated. ?We basically planned on threading the logging framework from the start for all of the reasons that you mentioned. :) > Cool, got it. > The rest of the filter actually works already except for the $writer. :) Excellent! > We've been poking around at various people and places trying to figure out what a Bro interface would look like and do. ?I suspect we aren't too far off from movement in this area, but we have no plans yet. Ok. > I'm hoping to have a couple of people talk about this specifically at the Bro workshop. Yes, I'm still hoping management will let me attend. In the meantime, if there are any incident responders on the list who wouldn't mind sharing with me off-list in what capacity they use Bro, it would be appreciated! From seth at icir.org Wed Aug 31 10:45:42 2011 From: seth at icir.org (Seth Hall) Date: Wed, 31 Aug 2011 13:45:42 -0400 Subject: [Bro-Dev] PF_RING cluster ID In-Reply-To: References: Message-ID: <93B5D67A-5C76-4AE0-BECE-0A55CC2478E2@icir.org> On Aug 31, 2011, at 12:28 PM, Seth Hall wrote: > Use the cluster deployment in BroControl and assign each of the virtual interfaces to a separate worker node. Since you're running the git master though, keep in mind that the broctl master that is currently checked out has some problems. I'll work on merging in our fixed branch today and I'll reply again once it's updated. I just got this branch merged so you'll have to do: git pull git submodule update --init --recursive Also, there seems to be some problem with installing this update overtop of an old install. You will probably want to backup any config and scripts files you have and delete your entire installation directory. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From mcholste at gmail.com Wed Aug 31 10:46:50 2011 From: mcholste at gmail.com (Martin Holste) Date: Wed, 31 Aug 2011 12:46:50 -0500 Subject: [Bro-Dev] PF_RING cluster ID In-Reply-To: <93B5D67A-5C76-4AE0-BECE-0A55CC2478E2@icir.org> References: <93B5D67A-5C76-4AE0-BECE-0A55CC2478E2@icir.org> Message-ID: But will this work if I don't have virtual interfaces? From seth at icir.org Wed Aug 31 10:47:51 2011 From: seth at icir.org (Seth Hall) Date: Wed, 31 Aug 2011 13:47:51 -0400 Subject: [Bro-Dev] PF_RING cluster ID In-Reply-To: <4E5E6753.7090404@icir.org> References: <4E5E6753.7090404@icir.org> Message-ID: <5C301778-F050-4B88-9831-504AA9788A6C@icir.org> On Aug 31, 2011, at 12:54 PM, Gregor Maier wrote: > I think it doesn't work like that :-( > It appears that PF_RINGs clustering doesn't use virtual interfaces for its clustering. Arg. Well that's annoying. > a) directly: add a new PktSrc() that uses the PF_RING API directly and add the command line argument to specify the clustering information. BroControl will likely need to be updated as well. If somebody does that, than the same framework can probably be used for Endace's load-balancing packet capture as well. I keep thinking that we should something along these lines eventually. I'd like to support the upcoming FreeBSD netmap API directly as well since it seems like there are some performance benefits to that over using their libpcap wrapper. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From mcholste at gmail.com Wed Aug 31 10:48:49 2011 From: mcholste at gmail.com (Martin Holste) Date: Wed, 31 Aug 2011 12:48:49 -0500 Subject: [Bro-Dev] PF_RING cluster ID In-Reply-To: <5C301778-F050-4B88-9831-504AA9788A6C@icir.org> References: <4E5E6753.7090404@icir.org> <5C301778-F050-4B88-9831-504AA9788A6C@icir.org> Message-ID: The PF_RING API is incredibly simple. I bet you could get this working in just a few hours, and that would be an enormous win for the community. On Wed, Aug 31, 2011 at 12:47 PM, Seth Hall wrote: > > On Aug 31, 2011, at 12:54 PM, Gregor Maier wrote: > >> I think it doesn't work like that :-( >> It appears that PF_RINGs clustering doesn't use virtual interfaces for its clustering. > > Arg. ?Well that's annoying. > >> a) directly: add a new PktSrc() that uses the PF_RING API directly and add the command line argument to specify the clustering information. BroControl will likely need to be updated as well. If somebody does that, than the same framework can probably be used for Endace's load-balancing packet capture as well. > > I keep thinking that we should something along these lines eventually. ?I'd like to support the upcoming FreeBSD netmap API directly as well since it seems like there are some performance benefits to that over using their libpcap wrapper. > > ?.Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > From slagell at ncsa.illinois.edu Wed Aug 31 10:50:18 2011 From: slagell at ncsa.illinois.edu (Adam J. Slagell) Date: Wed, 31 Aug 2011 12:50:18 -0500 Subject: [Bro-Dev] PF_RING cluster ID In-Reply-To: References: <4E5E6753.7090404@icir.org> <5C301778-F050-4B88-9831-504AA9788A6C@icir.org> Message-ID: On Aug 31, 2011, at 12:48 PM, Martin Holste wrote: > The PF_RING API is incredibly simple. I bet you could get this > working in just a few hours, and that would be an enormous win for the > community. Unless you are using myricom cards :-) It hates those. ------ Adam J. Slagell, CISSP Sr. Security Engineer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info 217.244.8965 From bro at tracker.bro-ids.org Wed Aug 31 11:17:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 18:17:42 -0000 Subject: [Bro-Dev] #510: Upgrade How-To In-Reply-To: <047.e793bc8325e1f7ba35cd0d2c4f23760e@tracker.bro-ids.org> References: <047.e793bc8325e1f7ba35cd0d2c4f23760e@tracker.bro-ids.org> Message-ID: <062.aa036f1735bc3f2171036af81428f929@tracker.bro-ids.org> #510: Upgrade How-To ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------+------------------------ Comment (by jsiwek): Transferring the following list of topics from the wiki for the upgrade guide, and expanding on some: - Explain new script organization's hierarchy - top-level source dir name changed from `policy` to `scripts` - the new `base` subdir differs from the new `policy` subdir in that `base` is all scripts that are loaded by Bro by default (unless the `-b` option is given) - default/builtin BROPATH only looks in `policy` and `site`, not `base` (important to know for those running in `-b` mode) - Scripting-layer API changes - The `@prefixes` directive works differently, see [d97003892bf1e3f6d642ca93bf992488f0c9bb7b/bro] - `mask_addr` BIF returns `subnet` vs. `addr` - `net` type has been removed - `remote.bro` has evolved into the communication framework - `Remote` module renamed to `Communication` - `Remote::destinations` renamed to `Communication::nodes` (the table of peers) - `Remote::Destination` renamed to `Communication::Node` (the type defining a remote peer) - The default packet filter now examines all packets instead of dynamically building a filter based on which protocol analysis scripts are loaded. See `PacketFilter::all_packets`. - Dynamic Protocol Detection (DPD) is now enabled/loaded by default - The "default" conn.log does not use well known ports for the service field, may be an added field in an extension script, see #576 - The majority of shipped Bro scripts have been overhauled to take advantage of modern Bro features. This includes using a new logging framework to standardize the format of logs across scripts. - The "notice" framework has been overhauled to make it easier for a user to define their site's policy. Might need more details/examples here. - The development process has moved from using SVN to Git for revision control and from Autotools to CMake for the build system. -- Ticket URL: Bro Tracker Bro Issue Tracker From jones at tacc.utexas.edu Wed Aug 31 12:24:48 2011 From: jones at tacc.utexas.edu (William Jones) Date: Wed, 31 Aug 2011 19:24:48 +0000 Subject: [Bro-Dev] PF_RING cluster ID In-Reply-To: References: Message-ID: You can divide the work up by using a pcap filters and specifying a different aux script for each work node. I have set up one node to monitor a tagged 10 GigE VLAN that I use to peer with I2, NLR and the internet using a TAP. Each worker mentors 1/8 the traffic on the I2, NLR and Internet connections. The second node monitors traffic from a mirrored 10 GigE interface. Each works monitors 1/4 the traffic. I haven't see any drops out of bro. I use the following filter: q1of8.bro: redef restrict_filters += { ["q1"] = "(ip[12:4] + ip[16:4]) & 7 == 1" }; q2of8.bro: redef restrict_filters += { ["q1"] = "(ip[12:4] + ip[16:4]) & 7 == 2" }; .. q8of8.bro: redef restrict_filters += { ["q1"] = "(ip[12:4] + ip[16:4]) & 7 == 7" }; Here the node file: # # Node configuration # [manager] type=manager host=homey.tacc.utexas.edu [proxy-1] type=proxy host=homey.tacc.utexas.edu [worker-1] type=worker host=mojo1.tacc.utexas.edu interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 -C aux_scripts=q0of8 [worker-2] type=worker host=mojo1.tacc.utexas.edu interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 -C aux_scripts=q1of8 [worker-3] type=worker host=mojo1.tacc.utexas.edu interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 -C aux_scripts=q2of8 [worker-4] type=worker host=mojo1.tacc.utexas.edu interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 -C aux_scripts=q3of8 [worker-5] type=worker host=mojo1.tacc.utexas.edu interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 -C aux_scripts=q4of8 [worker-6] type=worker host=mojo1.tacc.utexas.edu interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 -C aux_scripts=q5of8 [worker-7] type=worker host=mojo1.tacc.utexas.edu interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 -C aux_scripts=q6of8 [worker-8] type=worker host=mojo1.tacc.utexas.edu interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 -C aux_scripts=q7of8 [worker-9] type=worker host=homey1.tacc.utexas.edu interface=eth3.2 -C aux_scripts=q1 [worker-10] type=worker host=homey1.tacc.utexas.edu interface=eth3.2 -C aux_scripts=q2 [worker-11] type=worker host=homey1.tacc.utexas.edu interface=eth3.2 -C aux_scripts=q3 [worker-12] type=worker host=homey1.tacc.utexas.edu interface=eth3.2 -C aux_scripts=q0 -----Original Message----- From: bro-dev-bounces at bro-ids.org [mailto:bro-dev-bounces at bro-ids.org] On Behalf Of Seth Hall Sent: Wednesday, August 31, 2011 11:28 AM To: Martin Holste Cc: bro-dev Subject: Re: [Bro-Dev] PF_RING cluster ID On Aug 31, 2011, at 12:14 PM, Martin Holste wrote: > PF_RING's greatest feature is the ability to do per-flow clustering so > that many sniffers can run in parallel on their own CPU and each > receive a portion of the traffic which has been load-balanced. Snort > and Suricata both support this, and we've come to rely on it as we > don't have access to hardware load balancing. Is there a way to do > this in Bro? Use the cluster deployment in BroControl and assign each of the virtual interfaces to a separate worker node. Since you're running the git master though, keep in mind that the broctl master that is currently checked out has some problems. I'll work on merging in our fixed branch today and I'll reply again once it's updated. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ _______________________________________________ bro-dev mailing list bro-dev at bro-ids.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From seth at icir.org Wed Aug 31 12:43:09 2011 From: seth at icir.org (Seth Hall) Date: Wed, 31 Aug 2011 15:43:09 -0400 Subject: [Bro-Dev] PF_RING cluster ID In-Reply-To: References: Message-ID: On Aug 31, 2011, at 3:24 PM, William Jones wrote: > [worker-1] > type=worker > host=mojo1.tacc.utexas.edu > interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 -C Heh, the hack rises it's head again (including command line stuff in the 'interface' option). We really need to get multiple interface support in broctl soon. :) I do actually have a question though, why have you included -C? Using the -C flag when running in production on live traffic is actually pretty bad to do since you leave yourself open to trivial evasion. You should be seeing valid checksums anywhere you'd be sniffing anyway, the checksum problem only happens when a host offloads the checksum calculation to the NIC and you're sniffing traffic on your local machine. Let me know when you are ready to upgrade to the next release when we get it out too because there is an easier way to do your per-node script. I *think* that what you're currently doing should continue to work though. Thanks for documenting the filter trick for the list. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Wed Aug 31 12:53:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 19:53:43 -0000 Subject: [Bro-Dev] #579: Syslog logging writer Message-ID: <046.4e6efad585e65a2ccb68427348524651@tracker.bro-ids.org> #579: Syslog logging writer ---------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ Martin has completely convinced me of the need for this. I don't know about timeline we should put on it though. The one thought I have about it is that it needs to use TCP due to extremely long lines that Bro logs tend to have. I think it would be ok for it to have the same output rendering that the LogAscii writer has. -- Ticket URL: Bro Tracker Bro Issue Tracker From jones at tacc.utexas.edu Wed Aug 31 13:15:50 2011 From: jones at tacc.utexas.edu (William Jones) Date: Wed, 31 Aug 2011 20:15:50 +0000 Subject: [Bro-Dev] PF_RING cluster ID In-Reply-To: References: Message-ID: Good point about the -C. I set it because I was getting checksum errors. I now know that the source was due to the small snap len that bro uses in combination with a standard optimization in the new 10 GiGE card that will present several continues packets as one to pcap. So 10 1500 packets could turn into one 15000 buy packet. I patched bro server weeks ago. I forgot remove the -C parameter. I just removed on my bro cluster are it not complaining about checksum. I don't mind fixing my configs if you give me a more standard way to do something. When the next version coming out? -----Original Message----- From: Seth Hall [mailto:seth at icir.org] Sent: Wednesday, August 31, 2011 2:43 PM To: William Jones Cc: Martin Holste; bro-dev Subject: Re: [Bro-Dev] PF_RING cluster ID On Aug 31, 2011, at 3:24 PM, William Jones wrote: > [worker-1] > type=worker > host=mojo1.tacc.utexas.edu > interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 -C Heh, the hack rises it's head again (including command line stuff in the 'interface' option). We really need to get multiple interface support in broctl soon. :) I do actually have a question though, why have you included -C? Using the -C flag when running in production on live traffic is actually pretty bad to do since you leave yourself open to trivial evasion. You should be seeing valid checksums anywhere you'd be sniffing anyway, the checksum problem only happens when a host offloads the checksum calculation to the NIC and you're sniffing traffic on your local machine. Let me know when you are ready to upgrade to the next release when we get it out too because there is an easier way to do your per-node script. I *think* that what you're currently doing should continue to work though. Thanks for documenting the filter trick for the list. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Wed Aug 31 13:20:46 2011 From: seth at icir.org (Seth Hall) Date: Wed, 31 Aug 2011 16:20:46 -0400 Subject: [Bro-Dev] PF_RING cluster ID In-Reply-To: References: Message-ID: <7027C56D-AED2-43F6-B962-CBDD18DF9DD8@icir.org> On Aug 31, 2011, at 4:15 PM, William Jones wrote: > I now know that the source was due to the small snap len that bro uses in combination with a standard optimization in the new 10 GiGE card that will present several continues packets as one to pcap. So 10 1500 packets could turn into one 15000 buy packet. We'll be integrating the permanent-ish fix for this soon. The default snaplen is changing to 65535 along with having a -s option in case you need to shorten the snaplen. I think a lot of people have been running into that packet merging thing lately. > I don't mind fixing my configs if you give me a more standard way to do something. When the next version coming out? Our aim right now is a beta next month. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Wed Aug 31 14:02:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 21:02:40 -0000 Subject: [Bro-Dev] #580: pf_ring support Message-ID: <046.86c1866acdc92a7dc2ecdcca2b1a14ad@tracker.bro-ids.org> #580: pf_ring support -----------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Keywords: | -----------------------------+-------------------- Martin has convinced me that it's worthwhile to go ahead and get pf_ring support added sooner rather than later since it's a very common way of doing single interface load balancing on Linux. Eventually we should move toward a more generic PktSrc model (I don't think what we have is quite abstracted enough) but since the pf_ring code would be surrounded by #ifdefs anyway it should be fairly clean to rip it out later when we have a more generic way of grabbing packets and the code that would need to be added to support it should be fairly minimal. Like the syslog writer, I'm not sure of a timeline on this either. I'm setting it to the release after the next, but if we could squeeze it into the next release I'm sure that everyone would benefit. -- Ticket URL: Bro Tracker Bro Issue Tracker From gc355804 at ohio.edu Wed Aug 31 14:40:59 2011 From: gc355804 at ohio.edu (G. Clark) Date: Wed, 31 Aug 2011 14:40:59 -0700 Subject: [Bro-Dev] #580: pf_ring support In-Reply-To: <046.86c1866acdc92a7dc2ecdcca2b1a14ad@tracker.bro-ids.org> References: <046.86c1866acdc92a7dc2ecdcca2b1a14ad@tracker.bro-ids.org> Message-ID: <4E5EAA6B.70801@ohio.edu> The PF_RING user land library seems to be LGPL; the code itself seems to disagree with itself as to which version applies (pfring.h says 2 or later, pfring.c says 3 or later). COPYING is LGPL v3, so I'd assume that was the intended license. --Gilbert On 8/31/11 2:02 PM, Bro Tracker wrote: > #580: pf_ring support > -----------------------------+-------------------- > Reporter: seth | Owner: > Type: Feature Request | Status: new > Priority: Normal | Milestone: Bro1.7 > Component: Bro | Version: > Keywords: | > -----------------------------+-------------------- > Martin has convinced me that it's worthwhile to go ahead and get pf_ring > support added sooner rather than later since it's a very common way of > doing single interface load balancing on Linux. Eventually we should move > toward a more generic PktSrc model (I don't think what we have is quite > abstracted enough) but since the pf_ring code would be surrounded by > #ifdefs anyway it should be fairly clean to rip it out later when we have > a more generic way of grabbing packets and the code that would need to be > added to support it should be fairly minimal. > > Like the syslog writer, I'm not sure of a timeline on this either. I'm > setting it to the release after the next, but if we could squeeze it into > the next release I'm sure that everyone would benefit. > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6010 bytes Desc: S/MIME Cryptographic Signature Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110831/dee33620/attachment-0001.bin From mcholste at gmail.com Wed Aug 31 14:45:51 2011 From: mcholste at gmail.com (Martin Holste) Date: Wed, 31 Aug 2011 16:45:51 -0500 Subject: [Bro-Dev] PF_RING cluster ID In-Reply-To: References: Message-ID: Ah, very cool! This may work for me--I will have to play around with it and learn the cluster setup. Vlan interfaces often have wrong checksums on them, depending on the setup, so you may initially have needed that and then changed something. On Wed, Aug 31, 2011 at 2:24 PM, William Jones wrote: > You can divide the work up by using a pcap filters and specifying a different aux script for each work node. ?I have set up one node to monitor a tagged 10 GigE VLAN that I use to peer with I2, NLR and the internet using a TAP. ? Each worker mentors 1/8 the traffic on the I2, NLR and Internet connections. > > The second node monitors traffic from a mirrored 10 GigE interface. ?Each works monitors 1/4 the traffic. > > I haven't see any drops out of bro. > > > I use the following filter: > > q1of8.bro: > ? ?redef restrict_filters += { ["q1"] = "(ip[12:4] + ip[16:4]) & 7 == 1" }; > > q2of8.bro: > ? ?redef restrict_filters += { ["q1"] = "(ip[12:4] + ip[16:4]) & 7 == 2" }; > .. > q8of8.bro: > ? ?redef restrict_filters += { ["q1"] = "(ip[12:4] + ip[16:4]) & 7 == 7" }; > > Here the node file: > > # > # Node configuration > # > > [manager] > type=manager > host=homey.tacc.utexas.edu > > [proxy-1] > type=proxy > host=homey.tacc.utexas.edu > > [worker-1] > type=worker > host=mojo1.tacc.utexas.edu > interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 ?-C > aux_scripts=q0of8 > > > [worker-2] > type=worker > host=mojo1.tacc.utexas.edu > interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 ?-C > aux_scripts=q1of8 > > > [worker-3] > type=worker > host=mojo1.tacc.utexas.edu > interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 ?-C > aux_scripts=q2of8 > > > [worker-4] > type=worker > host=mojo1.tacc.utexas.edu > interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 ?-C > aux_scripts=q3of8 > > [worker-5] > type=worker > host=mojo1.tacc.utexas.edu > interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 ?-C > aux_scripts=q4of8 > > [worker-6] > type=worker > host=mojo1.tacc.utexas.edu > interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 ?-C > aux_scripts=q5of8 > > [worker-7] > type=worker > host=mojo1.tacc.utexas.edu > interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 ?-C > aux_scripts=q6of8 > > [worker-8] > type=worker > host=mojo1.tacc.utexas.edu > interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 ?-C > aux_scripts=q7of8 > > [worker-9] > type=worker > host=homey1.tacc.utexas.edu > interface=eth3.2 ?-C > aux_scripts=q1 > > [worker-10] > type=worker > host=homey1.tacc.utexas.edu > interface=eth3.2 ?-C > aux_scripts=q2 > > [worker-11] > type=worker > host=homey1.tacc.utexas.edu > interface=eth3.2 ?-C > aux_scripts=q3 > > [worker-12] > type=worker > host=homey1.tacc.utexas.edu > interface=eth3.2 ?-C > aux_scripts=q0 > > > > > > > > > > -----Original Message----- > From: bro-dev-bounces at bro-ids.org [mailto:bro-dev-bounces at bro-ids.org] On Behalf Of Seth Hall > Sent: Wednesday, August 31, 2011 11:28 AM > To: Martin Holste > Cc: bro-dev > Subject: Re: [Bro-Dev] PF_RING cluster ID > > > On Aug 31, 2011, at 12:14 PM, Martin Holste wrote: > >> PF_RING's greatest feature is the ability to do per-flow clustering so >> that many sniffers can run in parallel on their own CPU and each >> receive a portion of the traffic which has been load-balanced. ?Snort >> and Suricata both support this, and we've come to rely on it as we >> don't have access to hardware load balancing. ?Is there a way to do >> this in Bro? > > Use the cluster deployment in BroControl and assign each of the virtual interfaces to a separate worker node. ?Since you're running the git master though, keep in mind that the broctl master that is currently checked out has some problems. ?I'll work on merging in our fixed branch today and I'll reply again once it's updated. > > ?.Seth > > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From bro at tracker.bro-ids.org Wed Aug 31 14:49:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 31 Aug 2011 21:49:01 -0000 Subject: [Bro-Dev] #510: Upgrade How-To In-Reply-To: <047.e793bc8325e1f7ba35cd0d2c4f23760e@tracker.bro-ids.org> References: <047.e793bc8325e1f7ba35cd0d2c4f23760e@tracker.bro-ids.org> Message-ID: <062.767ef63240558e2836db054c878dd706@tracker.bro-ids.org> #510: Upgrade How-To ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------+------------------------ Comment (by jsiwek): I've started making the guide in the `www` repo now and I've got all of the above points explained with some TODOs for logging/notice frameworks. If anyone can expand it with documentation for new topics, feel free, else just make a note of new things that need to be added in the ticket here. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Aug 31 17:43:41 2011 From: seth at icir.org (Seth Hall) Date: Wed, 31 Aug 2011 20:43:41 -0400 Subject: [Bro-Dev] #580: pf_ring support In-Reply-To: <4E5EAA6B.70801@ohio.edu> References: <046.86c1866acdc92a7dc2ecdcca2b1a14ad@tracker.bro-ids.org> <4E5EAA6B.70801@ohio.edu> Message-ID: <4450FC1B-786F-453B-B51D-47693BEC3D4B@icir.org> On Aug 31, 2011, at 5:40 PM, G. Clark wrote: > The PF_RING user land library seems to be LGPL; the code itself seems to disagree with itself as to which version applies (pfring.h says 2 or later, pfring.c says 3 or later). COPYING is LGPL v3, so I'd assume that was the intended license. Ugh. Oh well, at least we will only need to link with it. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/