From robin at icir.org Fri Oct 1 08:33:34 2010 From: robin at icir.org (Robin Sommer) Date: Fri, 1 Oct 2010 08:33:34 -0700 Subject: [Bro-Dev] Book outline Message-ID: <20101001153334.GB76965@icir.org> Below's a first shot at a chapter outline. This is clearly not perfect yet, but let me know what you think. In particular, what's missing? Robin --------- cut ------------------------------------------------------- 1. Introduction Philosophy (aka "Bro is not Snort") Features 2. Getting Started System Requirements Installing Bro Running Bro from the Command Line Using Bro Control 3. Using Bro Understanding Bro's Output Notices and Alarms Activity Logs Weird Activity Customizing Scripts Building a Site Policy Notice Policy Tuning Standard Policy Files Behind the Curtain: Capture Filters Dynamic Protocol Detection Log Rotation and Post-Processing Active Response Offline Analysis System Tuning 4. Writing Bro Scripts Language Overview Event Handlers State Management Inter-Bro Communication Signatures Profiling and Debugging 5. Scripting Idioms/Patterns TODO: Collect. 6. Bro Control 7. Operating a Bro Cluster 8. Interfacing with the External World Broccoli Time Machine 9. Bro in Operation 10. Summary Getting More Information Contributing Back -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro-dev at bro-ids.org Sun Oct 3 14:22:31 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Sun, 03 Oct 2010 21:22:31 -0000 Subject: [Bro-Dev] #279: file_opened event is now consistlely raised In-Reply-To: <044.d9eb4450143797224b27df8d2303bfc0@bro-ids.org> References: <044.d9eb4450143797224b27df8d2303bfc0@bro-ids.org> Message-ID: <053.e3bfc26b3aef43370ee817c46b6f519a@bro-ids.org> #279: file_opened event is now consistlely raised ------------------------------+--------------------------------------------- Reporter: robin | Owner: vern Type: Patch | Status: closed Priority: Normal | Component: Bro Version: 1.5.2-devel (svn) | Resolution: Solved Keywords: | ------------------------------+--------------------------------------------- Changes (by vern): * status: new => closed * resolution: => Solved Comment: Integrated into 1.5.2.8. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Sun Oct 3 14:22:56 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Sun, 03 Oct 2010 21:22:56 -0000 Subject: [Bro-Dev] #280: Rotation post-processors get an additional argument In-Reply-To: <044.0b97a13fa07a3fec45200c399b017ea0@bro-ids.org> References: <044.0b97a13fa07a3fec45200c399b017ea0@bro-ids.org> Message-ID: <053.38667e05ee8f3488afbdc95dea39263a@bro-ids.org> #280: Rotation post-processors get an additional argument ------------------------------+--------------------------------------------- Reporter: robin | Owner: vern Type: Patch | Status: closed Priority: Normal | Component: Bro Version: 1.5.2-devel (svn) | Resolution: Solved Keywords: | ------------------------------+--------------------------------------------- Changes (by vern): * status: new => closed * resolution: => Solved Comment: Integrated into 1.5.2.8. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Sun Oct 3 14:23:18 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Sun, 03 Oct 2010 21:23:18 -0000 Subject: [Bro-Dev] #281: Make ct ignore leading t= prefix In-Reply-To: <044.dc5aa7e49680ca14d15b207f25bb2fae@bro-ids.org> References: <044.dc5aa7e49680ca14d15b207f25bb2fae@bro-ids.org> Message-ID: <053.19d9e409426db12dec39c5bceaab6f25@bro-ids.org> #281: Make ct ignore leading t= prefix ------------------------------+--------------------------------------------- Reporter: robin | Owner: vern Type: Patch | Status: closed Priority: Normal | Component: Bro Version: 1.5.2-devel (svn) | Resolution: Solved Keywords: | ------------------------------+--------------------------------------------- Changes (by vern): * status: new => closed * resolution: => Solved Comment: Integrated into 1.5.2.8. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Sun Oct 3 14:23:28 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Sun, 03 Oct 2010 21:23:28 -0000 Subject: [Bro-Dev] #284: Making email_notice_to customizable In-Reply-To: <044.c221231e947578bb3ce0938f4c8d9cb5@bro-ids.org> References: <044.c221231e947578bb3ce0938f4c8d9cb5@bro-ids.org> Message-ID: <053.7bf642db39fbc6d7b1b0b5817f407296@bro-ids.org> #284: Making email_notice_to customizable ------------------------------+--------------------------------------------- Reporter: robin | Owner: vern Type: Patch | Status: closed Priority: Normal | Component: Bro Version: 1.5.2-devel (svn) | Resolution: Solved Keywords: | ------------------------------+--------------------------------------------- Changes (by vern): * status: new => closed * resolution: => Solved Comment: Integrated into 1.5.2.8. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Sun Oct 3 20:07:49 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Mon, 04 Oct 2010 03:07:49 -0000 Subject: [Bro-Dev] #277: broctl install failed In-Reply-To: <072.a24d4ef9528d99012116b5ef1806c557@bro-ids.org> References: <072.a24d4ef9528d99012116b5ef1806c557@bro-ids.org> Message-ID: <081.1adee046245997986c1128c3afbdb571@bro-ids.org> #277: broctl install failed ----------------------------------------------+----------------------------- Reporter: Ar Kar Oo | Owner: robin Type: Problem | Status: new Priority: Normal | Component: BroControl Version: 1.5.1 | Keywords: ----------------------------------------------+----------------------------- Comment(by naurinramay@?): Its quite old entry and I am not sure you are still looking for an answer but try ./broctl install. I had the same problem solved using that. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Sun Oct 3 20:08:40 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Mon, 04 Oct 2010 03:08:40 -0000 Subject: [Bro-Dev] #277: broctl install failed In-Reply-To: <072.a24d4ef9528d99012116b5ef1806c557@bro-ids.org> References: <072.a24d4ef9528d99012116b5ef1806c557@bro-ids.org> Message-ID: <081.d770c6a2a96fa20c3b90d2a2736d2e0f@bro-ids.org> #277: broctl install failed ----------------------------------------------+----------------------------- Reporter: Ar Kar Oo | Owner: robin Type: Problem | Status: new Priority: Normal | Component: BroControl Version: 1.5.1 | Keywords: ----------------------------------------------+----------------------------- Comment(by anonymous): Replying to [ticket:277 Ar Kar Oo ]: > No command 'broctl' found, did you mean: ..... > > Although make install-broctl finished successfully I got this error when I ran this command. > > I read from http://svn.icir.org/bro/releases/release_1_5/bro/aux/broctl/README.html but it still isn't helpful. Its quite old entry and I am not sure you are still looking for an answer but try ./broctl install. I had the same problem solved using that. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Mon Oct 4 08:37:47 2010 From: seth at icir.org (Seth Hall) Date: Mon, 4 Oct 2010 11:37:47 -0400 Subject: [Bro-Dev] first try at a skeleton script In-Reply-To: <20100929222357.GB20718@icir.org> References: <790F4138-BAD8-483B-BE02-0B4F3D373FCF@icir.org> <20100929222357.GB20718@icir.org> Message-ID: <15929E68-67CC-4470-9BEF-587FEBF211B4@icir.org> Finally coming back around to this... On Sep 29, 2010, at 6:23 PM, Robin Sommer wrote: > - Another thought, considering our goal of building "configuration > wizards" Yes! That would be awesome and very helpful for the users that get nervous when they look at a programming language. I don't know if it would be loading too much functionality into broctl or not, but I can imagine having the wizard-ish interface there. I agree with all of your other points though (thanks), and I'll do another revision sometime this week although I know you won't see it for a couple of week still. (I feel like I'm talking to myself). .Seth From gregor at icir.org Mon Oct 4 14:08:44 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 04 Oct 2010 14:08:44 -0700 Subject: [Bro-Dev] first try at a skeleton script In-Reply-To: <20100929222705.GA21337@icir.org> References: <790F4138-BAD8-483B-BE02-0B4F3D373FCF@icir.org> <20100929222357.GB20718@icir.org> <20100929222705.GA21337@icir.org> Message-ID: <4CAA425C.6040501@icir.org> On 9/29/10 15:27 , Robin Sommer wrote: > - As a general rule, I guess a comment block should always be > associated with the next language element. ACK. > - The script itslef should start with a block describing the > overall purpose of the script and perhaps potential > caveats/things to keep in mind etc. ACK. Question: how to distinguish scripts' doc section from the doc section of the first language element, if somebody choose to only use one of them.... > - Each indidivual doc section should start with a single short > sentence summarizing the documented element, followed by further > text going into more detail. ACK. > - Generally, all doc text is formatted in reST. However, one > thing I always hate when writing documentation strings is adding > all kinds of markers for things like parameters, return values, > etc. (e.g., "@param", or ":param:"). > [cut] > - "Returns: texttextext" is the magic to mark the description > of the return value. > Thoughts? I'm just wondering whether not using markers might end up exploding in our faces once enough people write doc strings .... -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Mon Oct 4 14:20:43 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 04 Oct 2010 14:20:43 -0700 Subject: [Bro-Dev] first try at a skeleton script In-Reply-To: <20100929222357.GB20718@icir.org> References: <790F4138-BAD8-483B-BE02-0B4F3D373FCF@icir.org> <20100929222357.GB20718@icir.org> Message-ID: <4CAA452B.1000202@icir.org> > - The explanatory text is very helpful when writing a script, > however we should make it clear that once a script is done, this > should be removed, as otherwise we'll end up with many scripts > having the same boilerplate in them. So, perhaps it makes sense to > start anything that's just explaining something with an explicit > marker, like > > # REMOVE-ME: The export section contains the external ... > # REMOVE-ME: ...... > > - Also, some elements have a documentation string (i.e., something > that will end up in the auto-generated documentation later) while > others don't. I think that generally at least everything in the > export section should have documentation. > - Taking the two previous points together, it doesn't yet become > clear what is documentation for a script's *users*, vs. what is help > for a script *writer*. Is a user somebody who just needs to use the "interface" (globals, events, etc.) the script defines or somebody who might want to modify the script according to their needs. I guess the question is: where do we draw the line? Maybe it would help if the per-script doc-block would specify the "public" interface that users have to know about..... Or maybe the per-element doc-comments can have a flag indicating whether they are public interface or internal functionality. Then the doc generator can separate these elements for the output. Then a user can quickly see, what's relevant (which events, functions, globals) and somebody who needs to understand the script (to enhance, modify, whatever) also has all the information. > - As an alternative, we could keep the writer help inside a script > to a minimum, but have an additional external document giving more > details. Not sure which approach is better. I think that everything that documents the actual script should go into the script. If we separate code from documentation, we will probably face the problem that the documentation and the code don't match further down the road. Documentation about language elements can maybe go into a different document. > - The Notice enums should be documented as well. Maybe more general: can we come up with a nice way to document the redef of the various "special purpose" globals that are edited by many scripts? E.g., Notice enums, capture filters, analyzer configurations, etc. cu gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From slagell at ncsa.illinois.edu Mon Oct 4 18:28:18 2010 From: slagell at ncsa.illinois.edu (Adam Slagell) Date: Mon, 4 Oct 2010 20:28:18 -0500 (CDT) Subject: [Bro-Dev] first try at a skeleton script In-Reply-To: <4CAA452B.1000202@icir.org> References: <790F4138-BAD8-483B-BE02-0B4F3D373FCF@icir.org> <20100929222357.GB20718@icir.org> <4CAA452B.1000202@icir.org> Message-ID: Probably a good idea. If they are messing with internal interface, probably easier to look at source code than bring up a separate documentation manual. It would be easy to flood the documentation with so much information that it serves no purpose beyond just reading comments in source directly On Oct 4, 2010, at 4:20 PM, Gregor Maier wrote: > > Or maybe the per-element doc-comments can have a flag indicating whether > they are public interface or internal functionality. Then the doc > generator can separate these elements for the output. Then a user can > quickly see, what's relevant (which events, functions, globals) and > somebody who needs to understand the script (to enhance, modify, > whatever) also has all the information. From bro-dev at bro-ids.org Thu Oct 7 13:48:21 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Thu, 07 Oct 2010 20:48:21 -0000 Subject: [Bro-Dev] #286: Script level varargs Message-ID: <043.372ee0e82def24fae201e5ed5565ae49@bro-ids.org> #286: Script level varargs ------------------------------+--------------------------------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Component: Bro Version: 1.5.2-devel (svn) | Keywords: ------------------------------+--------------------------------------------- I'd like to be able to use something akin to varargs in the scripting language. The syntax I'm using in the example is pretty bad. :) i.e. {{{ function do_something(a: string, ...:set[string[): bool { for ( s in VARARGS) { print s; } return t; } }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Thu Oct 7 13:48:38 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Thu, 07 Oct 2010 20:48:38 -0000 Subject: [Bro-Dev] #286: Script level varargs In-Reply-To: <043.372ee0e82def24fae201e5ed5565ae49@bro-ids.org> References: <043.372ee0e82def24fae201e5ed5565ae49@bro-ids.org> Message-ID: <052.b012a7d640bccb443240378e576dfc8d@bro-ids.org> #286: Script level varargs ------------------------------+--------------------------------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Component: Bro Version: 1.5.2-devel (svn) | Keywords: ------------------------------+--------------------------------------------- Description changed by seth: Old description: > I'd like to be able to use something akin to varargs in the scripting > language. The syntax I'm using in the example is pretty bad. :) > > i.e. > {{{ > function do_something(a: string, ...:set[string[): bool > { > for ( s in VARARGS) > { > print s; > } > return t; > } > }}} New description: I'd like to be able to use something akin to varargs in the scripting language. The syntax I'm using in the example is pretty bad. :) i.e. {{{ function do_something(a: string, ...:set[string]): bool { for ( s in VARARGS) { print s; } return t; } }}} -- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Wed Oct 13 08:14:23 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Wed, 13 Oct 2010 15:14:23 -0000 Subject: [Bro-Dev] #272: Patch to fix do_split function In-Reply-To: <043.1e378c2cf2d2f272b054d3bd5b2f44d9@bro-ids.org> References: <043.1e378c2cf2d2f272b054d3bd5b2f44d9@bro-ids.org> Message-ID: <052.a6d26b05c36afd62d0c9f1bbf24ec1d0@bro-ids.org> #272: Patch to fix do_split function ------------------------------+--------------------------------------------- Reporter: seth | Owner: vern Type: Patch | Status: assigned Priority: Normal | Component: Bro Version: 1.5.2-devel (svn) | Keywords: ------------------------------+--------------------------------------------- Changes (by seth): * owner: => vern * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Oct 13 12:08:46 2010 From: seth at icir.org (Seth Hall) Date: Wed, 13 Oct 2010 15:08:46 -0400 Subject: [Bro-Dev] git and bash-completions Message-ID: <02F663D6-6C79-410D-A99D-F8C153283184@icir.org> I don't know if anyone else uses the bash-completions script, but if you do it appears that there is git support available for it. It makes things much, much easier. Long topic branch names especially become much easier to work with because you can just tab-complete them. Here's a site I came across where a guy was talking about using bash-completions with git. http://www.simplicidade.org/notes/archives/2008/02/git_bash_comple.html For anyone that uses a Mac and *really* doesn't want to go to any effort... ======= sudo port install bash-completion echo "if [ -f /opt/local/etc/bash_completion ]; then . /opt/local/etc/bash_completion . ~/.git-completion.sh export PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ ' fi" >> ~/.profile curl http://repo.or.cz/w/git.git/blob_plain/HEAD:/contrib/completion/git-completion.bash > ~/.git-completion.sh ======= If you add the PS1 line into your profile, it will show you which branch you currently have checked out whenever you are inside a git repository. Here's my prompt: [seth at Blake bro.git (devel)]$ .Seth From bro-dev at bro-ids.org Wed Oct 13 12:11:22 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Wed, 13 Oct 2010 19:11:22 -0000 Subject: [Bro-Dev] #272: Patch to fix do_split function In-Reply-To: <043.1e378c2cf2d2f272b054d3bd5b2f44d9@bro-ids.org> References: <043.1e378c2cf2d2f272b054d3bd5b2f44d9@bro-ids.org> Message-ID: <052.d9f3543e3de6cc29c96ecd6951dbb34f@bro-ids.org> #272: Patch to fix do_split function ------------------------------+--------------------------------------------- Reporter: seth | Owner: seth Type: Patch | Status: assigned Priority: Normal | Component: Bro Version: 1.5.2-devel (svn) | Keywords: ------------------------------+--------------------------------------------- Changes (by seth): * owner: vern => seth Comment: Ignore the patch on this ticket. It is now merged into the topic/seth /strings-without-checkstring branch. I'll use this as the tracking ticket for that branch now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Thu Oct 14 09:53:46 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Thu, 14 Oct 2010 16:53:46 -0000 Subject: [Bro-Dev] #287: BroControl should warn if changes aren't "install"ed Message-ID: <043.e2f467e027cfc6e405f4e889a9cd7ff2@bro-ids.org> #287: BroControl should warn if changes aren't "install"ed ------------------------------+--------------------------------------------- Reporter: seth | Owner: robin Type: Feature Request | Status: new Priority: Normal | Component: BroControl Version: 1.5.2-devel (svn) | Keywords: ------------------------------+--------------------------------------------- At times it can be difficult to remember to "install" after making changes in the site directory before restarting with BroControl. BroControl should check to see if there are any files that are updated in the site directory and put up a warning to the user if they are restarting or starting without installing. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Thu Oct 14 10:54:31 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Thu, 14 Oct 2010 17:54:31 -0000 Subject: [Bro-Dev] #288: Requesting BroControl feature to force flush log files. Message-ID: <043.368bf33fa8f3cb21440a87dbcf314a7e@bro-ids.org> #288: Requesting BroControl feature to force flush log files. ------------------------------+--------------------------------------------- Reporter: seth | Owner: robin Type: Feature Request | Status: new Priority: Normal | Component: BroControl Version: 1.5.2-devel (svn) | Keywords: ------------------------------+--------------------------------------------- If would be handy for BroControl to have the ability to force flush log files as a command. In the default case of running "flush-logs" (or whatever it would be named) with a node name, it could flush only the manager (or standalone) since the other nodes wouldn't make much sense to flush and it would make the process take longer. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Thu Oct 14 12:04:31 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Thu, 14 Oct 2010 19:04:31 -0000 Subject: [Bro-Dev] #289: Crash in broctl due to missing lockfile. Message-ID: <043.917fec1d31d25ef8a59f31d1c7cfc5d2@bro-ids.org> #289: Crash in broctl due to missing lockfile. ------------------------------+--------------------------------------------- Reporter: seth | Owner: robin Type: Problem | Status: new Priority: Normal | Component: BroControl Version: 1.5.2-devel (svn) | Keywords: ------------------------------+--------------------------------------------- I suppose the right way to handle this would just be to ignore it? Here's the stack trace from the python unhandled crash. {{{ waiting for lock ...........warning: removing stale lock Traceback (most recent call last): File "/home/bro/bin/broctl", line 726, in loop.onecmd(line) File "/usr/local/lib/python2.5/cmd.py", line 219, in onecmd return func(arg) File "/home/bro/bin/broctl", line 180, in do_start self.lock() File "/home/bro/bin/broctl", line 96, in lock if not util.lock(): File "/bro/lib/broctl/BroControl/util.py", line 187, in lock while not _aquireLock(): File "/bro/lib/broctl/BroControl/util.py", line 138, in _aquireLock if _breakLock(): File "/bro/lib/broctl/BroControl/util.py", line 104, in _breakLock os.unlink(config.Config.lockfile) OSError: [Errno 2] No such file or directory: '/bro/spool/lock' }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Thu Oct 14 12:33:27 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Thu, 14 Oct 2010 19:33:27 -0000 Subject: [Bro-Dev] #289: Crash in broctl due to missing lockfile. In-Reply-To: <043.917fec1d31d25ef8a59f31d1c7cfc5d2@bro-ids.org> References: <043.917fec1d31d25ef8a59f31d1c7cfc5d2@bro-ids.org> Message-ID: <052.9b6f34c1991305e01b57ed60aadaa843@bro-ids.org> #289: Crash in broctl due to missing lockfile. ------------------------------+--------------------------------------------- Reporter: seth | Owner: robin Type: Problem | Status: new Priority: Normal | Component: BroControl Version: 1.5.2-devel (svn) | Keywords: ------------------------------+--------------------------------------------- Comment(by seth): This seems to just be a race condition problem that likely isn't worth fixing. The reason it isn't being handled in the try block correctly is that the 'except' catches IOError but the lockfile missing is an OSError. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Sun Oct 17 21:36:48 2010 From: robin at icir.org (Robin Sommer) Date: Sun, 17 Oct 2010 21:36:48 -0700 Subject: [Bro-Dev] first try at a skeleton script In-Reply-To: <4CAA425C.6040501@icir.org> References: <790F4138-BAD8-483B-BE02-0B4F3D373FCF@icir.org> <20100929222357.GB20718@icir.org> <4CAA452B.1000202@icir.org> <790F4138-BAD8-483B-BE02-0B4F3D373FCF@icir.org> <20100929222357.GB20718@icir.org> <20100929222705.GA21337@icir.org> <4CAA425C.6040501@icir.org> Message-ID: <20101018043648.GP17675@icir.org> On Mon, Oct 04, 2010 at 14:08 -0700, Gregor Maier wrote: > I'm just wondering whether not using markers might end up exploding in > our faces once enough people write doc strings .... "Exploding" in which sense? (Note that I'm not too strong on not using markers; I personally find them cumbersome, but I can also configure my editor to do it for me :) On Mon, Oct 04, 2010 at 14:20 -0700, Gregor Maier wrote: > Is a user somebody who just needs to use the "interface" (globals, > events, etc.) the script defines or somebody who might want to modify > the script according to their needs. I meant the former. I don't think we should formalize documenting a script's internals. As Adam wrote, for that it's often easier to read the source directly, and where it isn't, a few informal comments (which don't go into the reference manual) are fine, just as it's now. My point was a different one: Seth's initial skeleton had some instructions for the person writing a script *initially* (e.g., how to structure things etc.). I found that confusing as there was no separation between these and things which will be left in the final script for people later needing to understand the interface. > Or maybe the per-element doc-comments can have a flag indicating whether > they are public interface or internal functionality. Can't we just use the export section? Everything in there is by definition part of the interface and should be documented. > Maybe more general: can we come up with a nice way to document the redef > of the various "special purpose" globals that are edited by many > scripts? E.g., Notice enums, capture filters, analyzer configurations, etc. Good point. For these we need two different types of documentation: one defining the purpose of the global, and one which states how a script modifies the global when loaded. However, there aren't that many of these so we might be fine special-casing them (i.e., just listing which notices, filters, etc. a script defines). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Mon Oct 18 07:29:53 2010 From: seth at icir.org (Seth Hall) Date: Mon, 18 Oct 2010 10:29:53 -0400 Subject: [Bro-Dev] first try at a skeleton script In-Reply-To: <20101018043648.GP17675@icir.org> References: <790F4138-BAD8-483B-BE02-0B4F3D373FCF@icir.org> <20100929222357.GB20718@icir.org> <4CAA452B.1000202@icir.org> <790F4138-BAD8-483B-BE02-0B4F3D373FCF@icir.org> <20100929222357.GB20718@icir.org> <20100929222705.GA21337@icir.org> <4CAA425C.6040501@icir.org> <20101018043648.GP17675@icir.org> Message-ID: On Oct 18, 2010, at 12:36 AM, Robin Sommer wrote: >> Maybe more general: can we come up with a nice way to document the redef >> of the various "special purpose" globals that are edited by many >> scripts? E.g., Notice enums, capture filters, analyzer configurations, etc. > > Good point. For these we need two different types of documentation: > one defining the purpose of the global, and one which states how a > script modifies the global when loaded. However, there aren't that > many of these so we might be fine special-casing them (i.e., just > listing which notices, filters, etc. a script defines). I'll rework the skeleton script sometime this week to account for the documentation aimed at script users and at script developers. Maybe the right way to do it is for me to remove the embedded documentation for the script developers and move all of that documentation into a separate document. The skeleton script will only contain what the script developer would write. .Seth From robin at icir.org Mon Oct 18 09:37:20 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 18 Oct 2010 09:37:20 -0700 Subject: [Bro-Dev] FreeBSD Ringmap Message-ID: <20101018163720.GM55971@icir.org> I received a mail from Alexander Fiveg, who is working in Berlin on the FreeBSD zero-copy ringmap for high-speed packet capturing, see http://code.google.com/p/ringmap. He worked on this as a Google SoC and presented his work at the recent FreeBSD development summit, as he was hoping to attract some funding from the FreeBSD foundation to continue the work. Alexander got quite some good feedback at the summit but unfortunately no major network developer was at there who could have sponsored the work with the foundation. Robert Watson was quite interested in this in the past, but apparently he's taking a break right now to finish his Ph.D. It now looks like Alexander won't be able to continue the project, which would be a pity. Does anybody have an idea who else might be interested in funding such work? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon Oct 18 10:04:09 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 18 Oct 2010 10:04:09 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <17474768.101.1287418425950.JavaMail.jsiwek@tangent.ncsa.illinois.edu> References: <20101018043815.GQ17675@icir.org> <17474768.101.1287418425950.JavaMail.jsiwek@tangent.ncsa.illinois.edu> Message-ID: <20101018170408.GO55971@icir.org> (Moving this to bro-dev, let's keep dev discussions there). On Mon, Oct 18, 2010 at 11:13 -0500, you wrote: > I can see that happening. I'll plan for that to happen unless we > find some other CMake projects that are doing something more > brilliant. Ok, I think this is my favorite option so far, it will also solve the --help problem. > 1) --enable-shippedpcap, "use the shipped version of libpcap" Please get rid of that, we don't want to ship a pcap anymore at all. > 2) bison vs. byacc I think the main reason for supporting both was that some systems used to come with the former, and others with the latter. I'm not sure how that's today, but I'd be fine supporting just bison. Vern, what do you think? Just looked at the other (Bro-specific) configure options, below are my thoughts. Anybody, please check whether you agree (active mapping and DFA cache expiration I'm most unsure about). Robin --------- cut ------------------------------------------------------- --enable-brov6 enable IPV6 processing -> Keep; we should eventually turn this always on and remove the configure option, but we aren't quire there yet. --enable-int64 enable use of int64 (long long) for integers -> Remove; let's always turns this on and remove the configure option. --enable-activemapping enable active mapping processing -> Do we want to keep active mapping in the code? Is anybody using that? --enable-expire-dfa-states enable DFA state expiration -> Do we want to keep this in code? Nobody is probably using it, and it complicates the DFA code quite a bit. --enable-debug no compiler optimizations -> Keep. --disable-select-loop disable select-based main loop -> Remove; no longer needed. --enable-perftools use Google's perftools -> Keep. --enable-shippedpcap use the shipped version of libpcap -> Remove. --disable-nbdns Disable non-blocking DNS support -> Remove; don't think anybody uses this. --disable-largefile omit support for large files -> Remove. --disable-broccoli Do not build/package Broccoli -> Remove; broccoli will be its own package. --disable-broctl Do not build/package broctl framework -> Remove; BroControl will be its own package. --enable-cluster Configure broctl for cluster usage -> Remove; that's an option for BroControl. --with-openssl=PATH path to OpenSSL (needed for SSL analyzer and secure communication) -> Keep, but I'm wondering whether we should make OpenSSL a mandatory dependency to get rid of all the "#if openssl" conditions. --with-perl=PATH path/name of the Perl interpreter -> Remove; I don't think we're using Perl anywhere anymore. --with-dag=PATH path to the DAG library (for native support for Endace Tech.'s DAG monitoring cards) -> Remove; nobody has been using the native DAG support in ages. --with-binpac=PATH path to a binpac executable for compiling analyzer code -> Keep. -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Mon Oct 18 10:22:17 2010 From: seth at icir.org (Seth Hall) Date: Mon, 18 Oct 2010 13:22:17 -0400 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101018170408.GO55971@icir.org> References: <20101018043815.GQ17675@icir.org> <17474768.101.1287418425950.JavaMail.jsiwek@tangent.ncsa.illinois.edu> <20101018170408.GO55971@icir.org> Message-ID: On Oct 18, 2010, at 1:04 PM, Robin Sommer wrote: > --enable-activemapping enable active mapping processing > -> Do we want to keep active mapping in the code? Is anybody > using that? What is active mapping? I've seen that for a long time but never asked what it is. > --disable-nbdns Disable non-blocking DNS support > -> Remove; don't think anybody uses this. Isn't the nonblocking dns support used for lookups in "when" blocks? > --with-openssl=PATH path to OpenSSL (needed for SSL analyzer and secure communication) > -> Keep, but I'm wondering whether we should make OpenSSL a > mandatory dependency to get rid of all the "#if openssl" > conditions. I vote for making it a mandatory dependency. At the very least, it's another assumption we can make when developing scripts (that SSL support exists) and is there anything that doesn't include OpenSSL in the base distribution? .Seth From vern at icir.org Mon Oct 18 10:31:15 2010 From: vern at icir.org (Vern Paxson) Date: Mon, 18 Oct 2010 10:31:15 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101018170408.GO55971@icir.org> (Mon, 18 Oct 2010 10:04:09 PDT). Message-ID: <20101018173115.ACF1A3137E4@taffy.ICSI.Berkeley.EDU> > I think the main reason for supporting both was that some systems > used to come with the former, and others with the latter. That's my memory too. > I'm not > sure how that's today, but I'd be fine supporting just bison. Vern, > what do you think? Let's give that a try. In general, I agree with your config suggestions. Some comments: > --enable-activemapping enable active mapping processing > -> Do we want to keep active mapping in the code? Is anybody > using that? I don't believe anyone uses this, so it makes sense to remove it. (Seth: this is an alternative to in-line normalization for resisting evasion attacks. It works by building up a map of how different end systems resolve ambiguities. You can read more about it if you want at http://www.icir.org/vern/papers/activemap-oak03.pdf .) > --enable-expire-dfa-states enable DFA state expiration > -> Do we want to keep this in code? Nobody is probably using > it, and it complicates the DFA code quite a bit. Seems okay to remove it. > --disable-nbdns Disable non-blocking DNS support > -> Remove; don't think anybody uses this. We can get the equivalent via setenv BRO_DNS_FAKE, right? (Seth: this is to *avoid* using non-blocking DNS because it won't locally build.) > --with-openssl=PATH path to OpenSSL (needed for SSL analyzer and secure communication) > -> Keep, but I'm wondering whether we should make OpenSSL a > mandatory dependency to get rid of all the "#if openssl" > conditions. I like that notion. I guess we'll see whether it leads to portability problems, similar to bison. > --with-dag=PATH path to the DAG library (for native support for Endace Tech.'s DAG monitoring cards) > -> Remove; nobody has been using the native DAG support in > ages. Are we pretty sure about that? Vern From jsiwek at ncsa.illinois.edu Mon Oct 18 11:52:35 2010 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 18 Oct 2010 13:52:35 -0500 (CDT) Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101018170408.GO55971@icir.org> Message-ID: <21243955.115.1287427954690.JavaMail.jsiwek@tangent.ncsa.illinois.edu> > --with-perl=PATH path/name of the Perl interpreter > -> Remove; I don't think we're using Perl anywhere anymore. So far there are two places I found dependent on Perl: 1) bro/src/make_parser.pl Because we decided to require bison over byacc, this can be removed since in the bison case it just does: cp $srcdir/parse.in $builddir/parse.y" 2) bro/src/make_dbg_constants.pl I didn't see an immediate way out of this one. Do you have thoughts on if something can be done with it? - Jon From vern at icir.org Mon Oct 18 11:56:24 2010 From: vern at icir.org (Vern Paxson) Date: Mon, 18 Oct 2010 11:56:24 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <21243955.115.1287427954690.JavaMail.jsiwek@tangent.ncsa.illinois.edu> (Mon, 18 Oct 2010 13:52:35 CDT). Message-ID: <20101018185624.C4D613137F2@taffy.ICSI.Berkeley.EDU> > 2) bro/src/make_dbg_constants.pl > > I didn't see an immediate way out of this one. Do you have thoughts on > if something can be done with it? I would think it's not hard to rewrite in another scripting language, if we decide to standardize on say Python. It's just generating a bunch of header files from the info in DebugCmdInfoConstants.in . This is used for Bro's interactive debugger, which has fallen into disrepair - but I don't think we should just stop supporting the debugger entirely, as it's potentially really quite handy functionality. Vern From robin at icir.org Mon Oct 18 12:20:00 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 18 Oct 2010 12:20:00 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101018185624.C4D613137F2@taffy.ICSI.Berkeley.EDU> References: <21243955.115.1287427954690.JavaMail.jsiwek@tangent.ncsa.illinois.edu> <20101018185624.C4D613137F2@taffy.ICSI.Berkeley.EDU> Message-ID: <20101018192000.GV55971@icir.org> On Mon, Oct 18, 2010 at 11:56 -0700, you wrote: > for Bro's interactive debugger, which has fallen into disrepair Do you have a sense how much work it will take to get it back into shape? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon Oct 18 12:21:35 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 18 Oct 2010 12:21:35 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101018173115.ACF1A3137E4@taffy.ICSI.Berkeley.EDU> References: <20101018170408.GO55971@icir.org> <20101018173115.ACF1A3137E4@taffy.ICSI.Berkeley.EDU> Message-ID: <20101018192135.GW55971@icir.org> On Mon, Oct 18, 2010 at 10:31 -0700, you wrote: > Are we pretty sure about that? I don't think I've ever heard from anybody (except me :) using it. People are using Endace's libpcap wrapper though. Let me ask on the mailing list. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Mon Oct 18 12:23:59 2010 From: seth at icir.org (Seth Hall) Date: Mon, 18 Oct 2010 15:23:59 -0400 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101018192000.GV55971@icir.org> References: <21243955.115.1287427954690.JavaMail.jsiwek@tangent.ncsa.illinois.edu> <20101018185624.C4D613137F2@taffy.ICSI.Berkeley.EDU> <20101018192000.GV55971@icir.org> Message-ID: <8442C7BF-37E6-48B0-8259-F5A3B8A7C659@icir.org> On Oct 18, 2010, at 3:20 PM, Robin Sommer wrote: > > On Mon, Oct 18, 2010 at 11:56 -0700, you wrote: > >> for Bro's interactive debugger, which has fallen into disrepair > > Do you have a sense how much work it will take to get it back into > shape? I'm definitely interested in how broken it is. I vaguely remember using it at one point a while ago, but maybe I didn't use the broken parts? .Seth From bro-dev at bro-ids.org Mon Oct 18 12:24:31 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Mon, 18 Oct 2010 19:24:31 -0000 Subject: [Bro-Dev] #290: Argument validation of open() Message-ID: <047.4517c0cc530374661c33fed23814b17d@bro-ids.org> #290: Argument validation of open() ---------------------+------------------------------------------------------ Reporter: vallenti | Owner: Type: Problem | Status: new Priority: Normal | Component: Bro Version: 1.5.1 | Keywords: ---------------------+------------------------------------------------------ The script function open() can overwrite important system files if called with the wrong arguments. For example, {{{ redef notice_file = open("/dev/null"); }}} Overwrites `/dev/null` with an ASCII file, which can have detrimental effects on the system integrity. Some ideas for mitigation: (1) Ensure that the file type is ASCII text (and not, say, a character special). (2) Do not allow log files to be created in /dev. (3) Do not overwrite existing files. (4) Make `open("")` be equivalent to writing into `/dev/null`. (5) Allow for log file deactivation with a new mechanism. Matthias -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Mon Oct 18 14:58:15 2010 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 18 Oct 2010 16:58:15 -0500 (CDT) Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101018170408.GO55971@icir.org> Message-ID: <22746924.183.1287439094864.JavaMail.jsiwek@tangent.ncsa.illinois.edu> First a general question: for the configure options we decide to remove, are we talking about just removing them from the user-facing configure script or additionally cleaning out the preprocessor blocks from the code that implements them? > --disable-broccoli Do not build/package Broccoli > -> Remove; broccoli will be its own package. > > --disable-broctl Do not build/package broctl framework > -> Remove; BroControl will be its own package. > > --enable-cluster Configure broctl for cluster usage > -> Remove; that's an option for BroControl. I'm not fully understanding how the new packaging and build system are going to cooperate. It seems like this will make each sub-module's build independent from the main Bro "core". That would be fine (and probably easiest thing to implement), but I'm not sure that's what we want? I thought that conflicts with the goal of providing an easy method of retrieve/configure/build/install for all desired sub-modules in one shot. In order to do that, the top-level Bro configure script needs to be able to enable sub-modules and be aware of their config options. CMake is fine with recursing on sub-projects and git can do sub-modules, so I was expecting to leverage that functionality. - Jon From vern at icir.org Mon Oct 18 16:07:20 2010 From: vern at icir.org (Vern Paxson) Date: Mon, 18 Oct 2010 16:07:20 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <22746924.183.1287439094864.JavaMail.jsiwek@tangent.ncsa.illinois.edu> (Mon, 18 Oct 2010 16:58:15 CDT). Message-ID: <20101018230720.512FB3137F2@taffy.ICSI.Berkeley.EDU> > for the configure options we decide to remove, > are we talking about just removing them from the user-facing configure > script or additionally cleaning out the preprocessor blocks from the code > that implements them? I was thinking the latter. We can dig the goop out of the revision archives if we decide to resurrect one of them. > I thought that conflicts with the goal of providing an easy method of > retrieve/configure/build/install for all desired sub-modules in one shot. Well, I'm wondering about this too. But it sounds like other folks have a model in mind whereby this isn't much of a hassle. Vern From gregor at icir.org Mon Oct 18 16:56:11 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 18 Oct 2010 16:56:11 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101018192135.GW55971@icir.org> References: <20101018170408.GO55971@icir.org> <20101018173115.ACF1A3137E4@taffy.ICSI.Berkeley.EDU> <20101018192135.GW55971@icir.org> Message-ID: <4CBCDE9B.3020107@icir.org> On 10/18/10 12:21 , Robin Sommer wrote: > > On Mon, Oct 18, 2010 at 10:31 -0700, you wrote: > >> Are we pretty sure about that? > > I don't think I've ever heard from anybody (except me :) using it. > People are using Endace's libpcap wrapper though. > > Let me ask on the mailing list. I agree. I think using the libpcap wrapper is the way to got. That's also what the TimeMachine uses. cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Mon Oct 18 17:11:16 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 18 Oct 2010 17:11:16 -0700 Subject: [Bro-Dev] first try at a skeleton script In-Reply-To: <20101018043648.GP17675@icir.org> References: <790F4138-BAD8-483B-BE02-0B4F3D373FCF@icir.org> <20100929222357.GB20718@icir.org> <4CAA452B.1000202@icir.org> <790F4138-BAD8-483B-BE02-0B4F3D373FCF@icir.org> <20100929222357.GB20718@icir.org> <20100929222705.GA21337@icir.org> <4CAA425C.6040501@icir.org> <20101018043648.GP17675@icir.org> Message-ID: <4CBCE224.8060501@icir.org> On 10/17/10 21:36 , Robin Sommer wrote: >> I'm just wondering whether not using markers might end up exploding in >> our faces once enough people write doc strings .... > > "Exploding" in which sense? > > (Note that I'm not too strong on not using markers; I personally > find them cumbersome, but I can also configure my editor to do it > for me :) I'm worrying that if we don't require markers (but infer the parameters) somebody will write a doc string that confuses the auto-detection and we get incorrect / weird output. Don't know how much of a problem this is going to be though. >> Is a user somebody who just needs to use the "interface" (globals, >> events, etc.) the script defines or somebody who might want to modify >> the script according to their needs. > > I meant the former. I don't think we should formalize documenting a > script's internals. As Adam wrote, for that it's often easier to > read the source directly, and where it isn't, a few informal > comments (which don't go into the reference manual) are fine, just > as it's now. Ok. OTOH, I think it's nice if functions and parameters are described or documented briefly, even if they are internal. However, If somebody has to modify the policy script's code (as I had to do for my http-analysis), it's helpful to have such documentation in the src code. But yes, they don't need to go into the reference manual. > My point was a different one: Seth's initial skeleton had some > instructions for the person writing a script *initially* (e.g., how > to structure things etc.). I found that confusing as there was no > separation between these and things which will be left in the final > script for people later needing to understand the interface. ACK. >> Or maybe the per-element doc-comments can have a flag indicating whether >> they are public interface or internal functionality. > > Can't we just use the export section? Everything in there is by > definition part of the interface and should be documented. Yes. Didn't think about that actually. >> Maybe more general: can we come up with a nice way to document the redef >> of the various "special purpose" globals that are edited by many >> scripts? E.g., Notice enums, capture filters, analyzer configurations, etc. > > Good point. For these we need two different types of documentation: > one defining the purpose of the global, and one which states how a > script modifies the global when loaded. However, there aren't that > many of these so we might be fine special-casing them (i.e., just > listing which notices, filters, etc. a script defines). Yeah. And maybe have a tool, script that, given the names of the "special globals", goes through the policy files and lists how each policy script modifies them. cu gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at ICSI.Berkeley.EDU Mon Oct 18 16:54:22 2010 From: gregor at ICSI.Berkeley.EDU (Gregor Maier) Date: Mon, 18 Oct 2010 16:54:22 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101018170408.GO55971@icir.org> References: <20101018043815.GQ17675@icir.org> <17474768.101.1287418425950.JavaMail.jsiwek@tangent.ncsa.illinois.edu> <20101018170408.GO55971@icir.org> Message-ID: <4CBCDE2E.8030704@icsi.berkeley.edu> >> 2) bison vs. byacc > > I think the main reason for supporting both was that some systems > used to come with the former, and others with the latter. I'm not > sure how that's today, but I'd be fine supporting just bison. Vern, > what do you think? IIRC, somebody also suggested to ship Bro with the generated scanner and parser so that a default build wouldn't need to run flex/bison/yacc. Don't know whether that's a good idea though. cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From robin at icir.org Mon Oct 18 17:21:05 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 18 Oct 2010 17:21:05 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <22746924.183.1287439094864.JavaMail.jsiwek@tangent.ncsa.illinois.edu> References: <20101018170408.GO55971@icir.org> <22746924.183.1287439094864.JavaMail.jsiwek@tangent.ncsa.illinois.edu> Message-ID: <20101019002105.GK77396@icir.org> On Mon, Oct 18, 2010 at 16:58 -0500, you wrote: > I'm not fully understanding how the new packaging and build system are going to cooperate. Frankly, it's not totally clear to me yet either ... My current assumption was that all the packages are first of all independent, and need to be downloaded/configured/installed separately. Then, we could go from there and see how we can make things easier for people working from sources, for example by providing a meta-distribution bro-all which comes with all parts and has some glue/magic to make their joint installation easier. But not sure how that would look like, suggestions welcome. > CMake is fine with recursing on sub-projects and git can do sub-modules, so I was expecting to leverage that functionality. If we can make this work, I'd be happy to do so. How would this look like, say for passing options to the sub-project's configure? And how would building separate packages for each module work then? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon Oct 18 17:21:18 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 18 Oct 2010 17:21:18 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101018230720.512FB3137F2@taffy.ICSI.Berkeley.EDU> References: <22746924.183.1287439094864.JavaMail.jsiwek@tangent.ncsa.illinois.edu> <20101018230720.512FB3137F2@taffy.ICSI.Berkeley.EDU> Message-ID: <20101019002118.GL77396@icir.org> On Mon, Oct 18, 2010 at 16:07 -0700, you wrote: > I was thinking the latter. Me too. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro-dev at bro-ids.org Mon Oct 18 17:54:25 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Tue, 19 Oct 2010 00:54:25 -0000 Subject: [Bro-Dev] #160: Output totaled information with the capstats command In-Reply-To: <043.5bafdfcbd01cdd316c13bc99cefb62f6@bro-ids.org> References: <043.5bafdfcbd01cdd316c13bc99cefb62f6@bro-ids.org> Message-ID: <052.dfba9d4e968493f5145d6cfa66eb4481@bro-ids.org> #160: Output totaled information with the capstats command ------------------------------+--------------------------------------------- Reporter: seth | Owner: robin Type: Feature Request | Status: closed Priority: Low | Component: BroControl Version: 1.5.2-devel (svn) | Resolution: fixed Keywords: | ------------------------------+--------------------------------------------- Comment(by robin): (In [7098]) - A larger BroControl update: o Increasing default timeouts for scan detector significantly. o Increasing the manager's max_remote_events_processed to something large, as it would slow down the process too much otherwise and there's no other work to be interleaved with it anyway. o Adding debug output to cluster's part of catch-and-release (extends the debugging already present in policy/debug.bro) o Fixing typo in util.py. Closes #223. o Added note to README pointing to HTML version. o Disabling print_hook for proxies' remote.log. o broctl's capstats now reports a total as well, and stats.log tracks these totals. Closes #160. o Avoiding spurious "waiting for lock" messages in cron mode. Closes #206. o Bug fixes for installation on NFS. o Bug fix for top command on FreeBSD 8. o crash-diag now checks whether gdb is available. o trace-summary reports the sample factor in use in its output, and now also applies it to the top-local-networks output (not doing the latter was a bug). o Removed the default twice-a-day rotation for conn.log. The default rotation for conn.log now is now once every 24h, just like for all other logs with the exception of mail.log (which is still rotated twice a day, and thus the alarms are still mailed out twice a day). o Fixed the problem of logs sometimes being filed into the wrong directory (see the (now gone) FAQ entry in the README). o One can now customize the archive naming scheme. See the corresponding FAQ entry in the README. o Cleaned up, and extended, collection of cluster statistics. ${logdir}/stats now looks like this: drwxr-xr-x 4 bro wheel 59392 Apr 5 17:55 . drwxr-xr-x 96 bro wheel 2560 Apr 6 12:00 .. -rw-r--r-- 1 bro wheel 576 Apr 6 16:40 meta.dat drwxr-xr-x 2 bro wheel 2048 Apr 6 16:40 profiling -rw-r--r-- 1 bro wheel 771834825 Apr 6 16:40 stats.log drwxr-xr-x 2 bro wheel 2048 Apr 6 16:25 www stats.log accumulates cluster statistics collected every time "cron" is called. - profiling/ keeps the nodes' prof.logs. - www/ keeps a subset of stats.log in CSV format for easy plotting. - meta.dat contains meta information about the current cluster state (in particular which nodes we have, and when the last stats update was done). Note that there is not Web setup yet to actually plot the data in www/. o BroControl now automatically maintains links inside today's log archive directory pointing to the current live version of the corresponding log file (if Bro is running). For example: smtp.log.11:52:18-current -> /usr/local/cluster/spool/manager/smtp.log o Alarms mailed out by BroControl now (1) have the notice msg in the subject; and (2) come with the full mail.log entry in the body. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Mon Oct 18 17:54:25 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Tue, 19 Oct 2010 00:54:25 -0000 Subject: [Bro-Dev] #223: Patch for small bug in BroControl In-Reply-To: <043.51955a462e7dfea77098f0843cedca5f@bro-ids.org> References: <043.51955a462e7dfea77098f0843cedca5f@bro-ids.org> Message-ID: <052.83c334f40c71a5a4303e9831cf793371@bro-ids.org> #223: Patch for small bug in BroControl ------------------------------+--------------------------------------------- Reporter: seth | Owner: robin Type: Patch | Status: closed Priority: Normal | Component: BroControl Version: 1.5.2-devel (svn) | Resolution: fixed Keywords: | ------------------------------+--------------------------------------------- Changes (by robin): * status: new => closed * resolution: => fixed Comment: (In [7098]) - A larger BroControl update: o Increasing default timeouts for scan detector significantly. o Increasing the manager's max_remote_events_processed to something large, as it would slow down the process too much otherwise and there's no other work to be interleaved with it anyway. o Adding debug output to cluster's part of catch-and-release (extends the debugging already present in policy/debug.bro) o Fixing typo in util.py. Closes #223. o Added note to README pointing to HTML version. o Disabling print_hook for proxies' remote.log. o broctl's capstats now reports a total as well, and stats.log tracks these totals. Closes #160. o Avoiding spurious "waiting for lock" messages in cron mode. Closes #206. o Bug fixes for installation on NFS. o Bug fix for top command on FreeBSD 8. o crash-diag now checks whether gdb is available. o trace-summary reports the sample factor in use in its output, and now also applies it to the top-local-networks output (not doing the latter was a bug). o Removed the default twice-a-day rotation for conn.log. The default rotation for conn.log now is now once every 24h, just like for all other logs with the exception of mail.log (which is still rotated twice a day, and thus the alarms are still mailed out twice a day). o Fixed the problem of logs sometimes being filed into the wrong directory (see the (now gone) FAQ entry in the README). o One can now customize the archive naming scheme. See the corresponding FAQ entry in the README. o Cleaned up, and extended, collection of cluster statistics. ${logdir}/stats now looks like this: drwxr-xr-x 4 bro wheel 59392 Apr 5 17:55 . drwxr-xr-x 96 bro wheel 2560 Apr 6 12:00 .. -rw-r--r-- 1 bro wheel 576 Apr 6 16:40 meta.dat drwxr-xr-x 2 bro wheel 2048 Apr 6 16:40 profiling -rw-r--r-- 1 bro wheel 771834825 Apr 6 16:40 stats.log drwxr-xr-x 2 bro wheel 2048 Apr 6 16:25 www stats.log accumulates cluster statistics collected every time "cron" is called. - profiling/ keeps the nodes' prof.logs. - www/ keeps a subset of stats.log in CSV format for easy plotting. - meta.dat contains meta information about the current cluster state (in particular which nodes we have, and when the last stats update was done). Note that there is not Web setup yet to actually plot the data in www/. o BroControl now automatically maintains links inside today's log archive directory pointing to the current live version of the corresponding log file (if Bro is running). For example: smtp.log.11:52:18-current -> /usr/local/cluster/spool/manager/smtp.log o Alarms mailed out by BroControl now (1) have the notice msg in the subject; and (2) come with the full mail.log entry in the body. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Mon Oct 18 17:54:26 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Tue, 19 Oct 2010 00:54:26 -0000 Subject: [Bro-Dev] #206: cron 'waiting for lock issue' In-Reply-To: <045.08113708059b9093d8f9434f8f95a4e8@bro-ids.org> References: <045.08113708059b9093d8f9434f8f95a4e8@bro-ids.org> Message-ID: <054.4e028df723fdbf735c5edab00a4939d4@bro-ids.org> #206: cron 'waiting for lock issue' ------------------------------+--------------------------------------------- Reporter: justin | Owner: robin Type: Problem | Status: closed Priority: Normal | Component: BroControl Version: 1.5.2-devel (svn) | Resolution: fixed Keywords: broctl cron | ------------------------------+--------------------------------------------- Comment(by robin): (In [7098]) - A larger BroControl update: o Increasing default timeouts for scan detector significantly. o Increasing the manager's max_remote_events_processed to something large, as it would slow down the process too much otherwise and there's no other work to be interleaved with it anyway. o Adding debug output to cluster's part of catch-and-release (extends the debugging already present in policy/debug.bro) o Fixing typo in util.py. Closes #223. o Added note to README pointing to HTML version. o Disabling print_hook for proxies' remote.log. o broctl's capstats now reports a total as well, and stats.log tracks these totals. Closes #160. o Avoiding spurious "waiting for lock" messages in cron mode. Closes #206. o Bug fixes for installation on NFS. o Bug fix for top command on FreeBSD 8. o crash-diag now checks whether gdb is available. o trace-summary reports the sample factor in use in its output, and now also applies it to the top-local-networks output (not doing the latter was a bug). o Removed the default twice-a-day rotation for conn.log. The default rotation for conn.log now is now once every 24h, just like for all other logs with the exception of mail.log (which is still rotated twice a day, and thus the alarms are still mailed out twice a day). o Fixed the problem of logs sometimes being filed into the wrong directory (see the (now gone) FAQ entry in the README). o One can now customize the archive naming scheme. See the corresponding FAQ entry in the README. o Cleaned up, and extended, collection of cluster statistics. ${logdir}/stats now looks like this: drwxr-xr-x 4 bro wheel 59392 Apr 5 17:55 . drwxr-xr-x 96 bro wheel 2560 Apr 6 12:00 .. -rw-r--r-- 1 bro wheel 576 Apr 6 16:40 meta.dat drwxr-xr-x 2 bro wheel 2048 Apr 6 16:40 profiling -rw-r--r-- 1 bro wheel 771834825 Apr 6 16:40 stats.log drwxr-xr-x 2 bro wheel 2048 Apr 6 16:25 www stats.log accumulates cluster statistics collected every time "cron" is called. - profiling/ keeps the nodes' prof.logs. - www/ keeps a subset of stats.log in CSV format for easy plotting. - meta.dat contains meta information about the current cluster state (in particular which nodes we have, and when the last stats update was done). Note that there is not Web setup yet to actually plot the data in www/. o BroControl now automatically maintains links inside today's log archive directory pointing to the current live version of the corresponding log file (if Bro is running). For example: smtp.log.11:52:18-current -> /usr/local/cluster/spool/manager/smtp.log o Alarms mailed out by BroControl now (1) have the notice msg in the subject; and (2) come with the full mail.log entry in the body. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Mon Oct 18 19:30:55 2010 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 18 Oct 2010 21:30:55 -0500 (CDT) Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101019002105.GK77396@icir.org> Message-ID: <1437900296.9683.1287455455605.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > > CMake is fine with recursing on sub-projects and git can do > > sub-modules, so I was expecting to leverage that functionality. > > If we can make this work, I'd be happy to do so. How would this look > like, say for passing options to the sub-project's configure? Take git out of the equation for now and say we downloaded the 'bro-all' source bundle containing the Bro super-project and all sub-projects. The super-project's configure wrapper will have to be able to recursively look for additional options provided by any existing sub-projects, but only has to display and maybe validate them. Setting a sub-project option via the super-project's configure wrapper will end up as a globally scoped CMake variable although only maybe a single sub-project cares about it. If you consider how this works with git sub-modules, the only thing that's different is how you obtain the source bundle. Here, cloning the Bro super-project will give you empty sub-module directories that one can init/update as desired: git submodule init git submodule update The super-project just maintains sub-module pointers to a particular commit in their repository (probably a release tag). > how would building separate packages for each module work then? That stumped me. After reading around a bit, it seems like CPack does not currently play nice with sub-projects (although it's on someone's TODO list). So it looks like we can't just do a `make package` at the super-project level and expect it to spit out source & binary packages for independent sub-projects. But if each module as a standalone works fine with CPack, I don't see why we can't just make a package generating script that iterates over modules, doing an independent out-of-source build and package creation for each. The 'bro-all' package should still be possible via CPack if sub-projects had some extra logic to disable their own CPack settings when they detect that they're part of a super-project. Thoughts on those approaches? - Jon From robin at icir.org Mon Oct 18 20:10:05 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 18 Oct 2010 20:10:05 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <1437900296.9683.1287455455605.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <20101019002105.GK77396@icir.org> <1437900296.9683.1287455455605.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20101019031005.GA22106@icir.org> I like this, sounds like it should work. On Mon, Oct 18, 2010 at 21:30 -0500, you wrote: > The super-project's configure wrapper will have to be able to > recursively look for additional options provided by any existing > sub-projects, but only has to display and maybe validate them. The wrapper should also make clear which options apply to which sub-package so that one doesn't get confused. Generally, I expect the number of options to remain small though, so this shouldn't be much of a problem. > the Bro super-project will give you empty sub-module directories > that one can init/update as desired: > > git submodule init > git submodule update Sounds like we could provide a script which does this for each submodule. > standalone works fine with CPack, I don't see why we can't just make > a package generating script that iterates over modules, doing an > independent out-of-source build and package creation for each. That sounds fine. > The 'bro-all' package should still be possible via CPack if I don't think we necessarily need a bro-all package. I see bro-all mainly as for those folks building/developing from source. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon Oct 18 20:29:48 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 18 Oct 2010 20:29:48 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <1437900296.9683.1287455455605.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <20101019002105.GK77396@icir.org> <1437900296.9683.1287455455605.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20101019032948.GE22106@icir.org> One more thing: broctl has three further aux/* sub-modules: capstats, pysubnettree, and trace-summary. These are actually all usable without broctl so I think we should make them separate modules as well (I'm already maintaining them in their own Subversion repositories, which so far has been a pain because I had to apply all changes both to these repos and to the Bro repository. Once in git as separate modules, I'll switch over to using just those). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From vern at icir.org Mon Oct 18 22:48:23 2010 From: vern at icir.org (Vern Paxson) Date: Mon, 18 Oct 2010 22:48:23 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101018192000.GV55971@icir.org> (Mon, 18 Oct 2010 12:20:00 PDT). Message-ID: <20101019054823.AFAEC36A421@taffy.ICSI.Berkeley.EDU> > > for Bro's interactive debugger, which has fallen into disrepair > > Do you have a sense how much work it will take to get it back into > shape? I don't think it should be all that hard. At one point it worked pretty well, so my guess is that the bit-rot that broke it isn't all that much work to repair. Vern From vern at icir.org Mon Oct 18 22:49:03 2010 From: vern at icir.org (Vern Paxson) Date: Mon, 18 Oct 2010 22:49:03 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <8442C7BF-37E6-48B0-8259-F5A3B8A7C659@icir.org> (Mon, 18 Oct 2010 15:23:59 EDT). Message-ID: <20101019054903.4C07836A421@taffy.ICSI.Berkeley.EDU> > I vaguely remember using it at one point a while ago, but maybe I didn't > use the broken parts? I suspect you used it a number of years ago, pre-brokenness. Because right now, it essentially doesn't work at all, I believe. Vern From bro-dev at bro-ids.org Wed Oct 20 16:20:02 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Wed, 20 Oct 2010 23:20:02 -0000 Subject: [Bro-Dev] #291: lookup_addr() only supports IPv4 addresses Message-ID: <044.ac2fb4cb11fcfbae75d1e9c96c3f1a02@bro-ids.org> #291: lookup_addr() only supports IPv4 addresses -------------------+-------------------------------------------------------- Reporter: leres | Type: Problem Status: new | Priority: Normal Component: Bro | Version: 1.5.1 Keywords: | -------------------+-------------------------------------------------------- We've started deploying IPv6 on one subnet at LBL and bro is missing some IPv6 support; see attached crash report. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Oct 21 11:05:26 2010 From: robin at icir.org (Robin Sommer) Date: Thu, 21 Oct 2010 11:05:26 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101018192135.GW55971@icir.org> References: <20101018170408.GO55971@icir.org> <20101018173115.ACF1A3137E4@taffy.ICSI.Berkeley.EDU> <20101018192135.GW55971@icir.org> Message-ID: <20101021180526.GA10592@icir.org> On Mon, Oct 18, 2010 at 12:21 -0700, I wrote: > I don't think I've ever heard from anybody (except me :) using it. > People are using Endace's libpcap wrapper though. > > Let me ask on the mailing list. Nobody said they are using it, some are using the pcap wrapper. So, let's remove the native Endace/DAG support. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at ncsa.illinois.edu Thu Oct 21 18:08:13 2010 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 21 Oct 2010 20:08:13 -0500 (CDT) Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <20101021180526.GA10592@icir.org> Message-ID: <854530195.7563.1287709693236.JavaMail.root@zimbra-1.ncsa.uiuc.edu> I noticed that ClamAV support is always disabled because Bro has yet to migrate to a newer API. How should this be handled for the CMake port? I already implemented it as an always disabled and unadvertised feature, but maybe that was too much? - Jon From seth at icir.org Thu Oct 21 18:45:37 2010 From: seth at icir.org (Seth Hall) Date: Thu, 21 Oct 2010 21:45:37 -0400 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <854530195.7563.1287709693236.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <854530195.7563.1287709693236.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <2049B405-32F2-41C5-ACBF-061609E15ABA@icir.org> On Oct 21, 2010, at 9:08 PM, Jonathan Siwek wrote: > I noticed that ClamAV support is always disabled because Bro has yet to migrate to a newer API. How should this be handled for the CMake port? They removed the support for virus scanning buffers instead of files and when I checked a few days ago, that API was still gone. I'd say the clamav support is officially dead at this point. .Seth From robin at icir.org Thu Oct 21 21:02:53 2010 From: robin at icir.org (Robin Sommer) Date: Thu, 21 Oct 2010 21:02:53 -0700 Subject: [Bro-Dev] Cmake for Bro In-Reply-To: <2049B405-32F2-41C5-ACBF-061609E15ABA@icir.org> References: <854530195.7563.1287709693236.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <2049B405-32F2-41C5-ACBF-061609E15ABA@icir.org> Message-ID: <20101022040252.GL17674@icir.org> On Thu, Oct 21, 2010 at 21:45 -0400, you wrote: > and when I checked a few days ago, that API was still gone. I'd say > the clamav support is officially dead at this point. Ack. Thanks for checking, let's remove it. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From martin.arlitt at ucalgary.ca Fri Oct 22 15:44:37 2010 From: martin.arlitt at ucalgary.ca (Martin Arlitt) Date: Fri, 22 Oct 2010 16:44:37 -0600 Subject: [Bro-Dev] DataSeries for Bro In-Reply-To: <20101019154033.GB72825@icir.org> References: <20100825155558.GI50273@icir.org> <4C753E43.1090300@ucalgary.ca> <20100825160611.GN50273@icir.org> <4C755103.30402@ucalgary.ca> <20100827041452.GH49514@icir.org> <4C77B100.1000609@ucalgary.ca> <20100929211256.GA18993@icir.org> <5e936a2be758e596bc77b51bbd70a550.squirrel@webmail.ucalgary.ca> <20100929215853.GD17368@icir.org> <4CBD91BB.50401@ucalgary.ca> <20101019154033.GB72825@icir.org> Message-ID: <4CC213D5.6070502@ucalgary.ca> hi Robin the technical report is now available at: http://www.hpl.hp.com/techreports/2010/HPL-2010-164.html The work has more focus on Apache than Bro, primarily because I couldn't get Sergey access to Bro on a production network. However, he did integrate DataSeries with Bro and ran some tests. I think his work does show that DataSeries has clear benefits for log collection and analysis with these types of applications. There is at least one thing we would do differently if we started over again, and that is to use an in-memory buffer for log entries before writing an extent to disk. Sergey used a temporary file because he was concerned about messing up Apache's memory management, and then followed the same approach when he added DataSeries logging to Bro. Obviously for those familiar with the Bro source, this shouldn't be an issue. if you or anyone else has questions about this, please let me know. thanks Martin From seth at icir.org Fri Oct 22 19:21:48 2010 From: seth at icir.org (Seth Hall) Date: Fri, 22 Oct 2010 22:21:48 -0400 Subject: [Bro-Dev] DataSeries for Bro In-Reply-To: <4CC213D5.6070502@ucalgary.ca> References: <20100825155558.GI50273@icir.org> <4C753E43.1090300@ucalgary.ca> <20100825160611.GN50273@icir.org> <4C755103.30402@ucalgary.ca> <20100827041452.GH49514@icir.org> <4C77B100.1000609@ucalgary.ca> <20100929211256.GA18993@icir.org> <5e936a2be758e596bc77b51bbd70a550.squirrel@webmail.ucalgary.ca> <20100929215853.GD17368@icir.org> <4CBD91BB.50401@ucalgary.ca> <20101019154033.GB72825@icir.org> <4CC213D5.6070502@ucalgary.ca> Message-ID: On Oct 22, 2010, at 6:44 PM, Martin Arlitt wrote: > if you or anyone else has questions about this, please let me know. Hi Martin! I'm really excited about the prospects of integrating DataSeries into Bro. Are the changes to Bro available anywhere? For the benefit of anyone that hasn't found it yet, here's the link to where you can download the DataSeries source code: http://tesla.hpl.hp.com/opensource/ .Seth From robin at icir.org Mon Oct 25 08:49:17 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 25 Oct 2010 08:49:17 -0700 Subject: [Bro-Dev] Opinions on trace rewriter? Message-ID: <20101025154917.GC63553@icir.org> What are the opinions on completely removing the trace rewriter from the Bro code base? I think I vaguely remember us discussing this, with a conclusion that removing is the way to go, but I wanted to check. Pros of removing: - Rewriter is broken right now anyway - Not used by many people (just the "occasional researcher" :-) - Nobody knows the internals well. - It's quite intrusive to the code, and removing it will get rid of quite a bit of stuff sprinkled across the source. Cons: - It's pretty cool functionality and nobody else has it. I really like the rewriting but I'm thinking it would be better done in an external tool than inside Bro itself. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon Oct 25 08:53:21 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 25 Oct 2010 08:53:21 -0700 Subject: [Bro-Dev] DataSeries for Bro In-Reply-To: <4CC213D5.6070502@ucalgary.ca> References: <20100825160611.GN50273@icir.org> <4C755103.30402@ucalgary.ca> <20100827041452.GH49514@icir.org> <4C77B100.1000609@ucalgary.ca> <20100929211256.GA18993@icir.org> <5e936a2be758e596bc77b51bbd70a550.squirrel@webmail.ucalgary.ca> <20100929215853.GD17368@icir.org> <4CBD91BB.50401@ucalgary.ca> <20101019154033.GB72825@icir.org> <4CC213D5.6070502@ucalgary.ca> Message-ID: <20101025155321.GE63553@icir.org> On Fri, Oct 22, 2010 at 16:44 -0600, you wrote: > the technical report is now available at: Thanks, Martin! Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From martin.arlitt at ucalgary.ca Mon Oct 25 09:41:05 2010 From: martin.arlitt at ucalgary.ca (Martin Arlitt) Date: Mon, 25 Oct 2010 10:41:05 -0600 Subject: [Bro-Dev] DataSeries for Bro In-Reply-To: References: <20100825155558.GI50273@icir.org> <4C753E43.1090300@ucalgary.ca> <20100825160611.GN50273@icir.org> <4C755103.30402@ucalgary.ca> <20100827041452.GH49514@icir.org> <4C77B100.1000609@ucalgary.ca> <20100929211256.GA18993@icir.org> <5e936a2be758e596bc77b51bbd70a550.squirrel@webmail.ucalgary.ca> <20100929215853.GD17368@icir.org> <4CBD91BB.50401@ucalgary.ca> <20101019154033.GB72825@icir.org> <4CC213D5.6070502@ucalgary.ca> Message-ID: <4CC5B321.6030308@ucalgary.ca> hi Seth the modified source should be available at http://www.sfu.ca/~sba70/files/dataseries/ if there are any issues, please let me know. thanks Martin Seth Hall wrote: > On Oct 22, 2010, at 6:44 PM, Martin Arlitt wrote: > > >> if you or anyone else has questions about this, please let me know. >> > > Hi Martin! I'm really excited about the prospects of integrating DataSeries into Bro. Are the changes to Bro available anywhere? > > For the benefit of anyone that hasn't found it yet, here's the link to where you can download the DataSeries source code: > http://tesla.hpl.hp.com/opensource/ > > .Seth > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20101025/25e07399/attachment.html From vern at icir.org Mon Oct 25 10:31:16 2010 From: vern at icir.org (Vern Paxson) Date: Mon, 25 Oct 2010 10:31:16 -0700 Subject: [Bro-Dev] Opinions on trace rewriter? In-Reply-To: <20101025154917.GC63553@icir.org> (Mon, 25 Oct 2010 08:49:17 PDT). Message-ID: <20101025173116.88DDE3137E4@taffy.ICSI.Berkeley.EDU> > What are the opinions on completely removing the trace rewriter from > the Bro code base? I would like to keep it. I hear you on the benefits of removing. However: > Cons: > > - It's pretty cool functionality and nobody else has it. Yep. It remains a striking demonstration of what's possible, IMHO, plus it has for me a flavor of potentially leading to serendipitous uses. Also: > I really like the rewriting but I'm thinking it would be better done > in an external tool than inside Bro itself. It doesn't make sense as an external tool. It uses the coupling between Bro event generation and Bro policy scripts to do the rewriting. You'd have to write Bro all over again. Now, if you're willing to pledge that BinPAC++ will support rewriting functionality .... then I could see removing the existing code. :-) Vern From robin at icir.org Mon Oct 25 10:41:13 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 25 Oct 2010 10:41:13 -0700 Subject: [Bro-Dev] Opinions on trace rewriter? In-Reply-To: <20101025173116.88DDE3137E4@taffy.ICSI.Berkeley.EDU> References: <20101025154917.GC63553@icir.org> <20101025173116.88DDE3137E4@taffy.ICSI.Berkeley.EDU> Message-ID: <20101025174113.GB17795@icir.org> On Mon, Oct 25, 2010 at 10:31 -0700, you wrote: > Now, if you're willing to pledge that BinPAC++ will support > rewriting functionality .... That was indeed a thought I had, but because I don't want to *pledge* it, I didn't say it; but you see where the "separate tool" idea is coming from. :-) The problem with the rewriter is that either (a) we leave it broken or (b) we need to spend significant time fixing it (and make sure it doesn't immediately break again with future changes ...). The former doesn't sound like a great option, but I'm not sure that the latter is worth the effort. We could do a compromise: Remove it, but turn all the code into a big patch that applies cleanly to the next release. We wouldn't fix anything for now, and we wouldn't maintain the patch for future releases, but we'll keep the patch around if somebody comes along who has an interest in doing (b) (which might just as well by ourselves if at some point we want to spend our resources on that; I don't see that near-term though). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From slagell at ncsa.illinois.edu Mon Oct 25 10:50:29 2010 From: slagell at ncsa.illinois.edu (Adam J. Slagell) Date: Mon, 25 Oct 2010 12:50:29 -0500 Subject: [Bro-Dev] Opinions on trace rewriter? In-Reply-To: <20101025174113.GB17795@icir.org> References: <20101025154917.GC63553@icir.org> <20101025173116.88DDE3137E4@taffy.ICSI.Berkeley.EDU> <20101025174113.GB17795@icir.org> Message-ID: On Oct 25, 2010, at 12:41 PM, Robin Sommer wrote: > We could do a compromise: Remove it, but turn all the code into a > big patch that applies cleanly to the next release. We wouldn't fix > anything for now, and we wouldn't maintain the patch for future > releases, but we'll keep the patch around if somebody comes along > who has an interest in doing (b) (which might just as well by > ourselves if at some point we want to spend our resources on that; I > don't see that near-term though). Not sure what I am stepping into, but here I go. I agree strongly with Robin. It seems if it is broken now, then you don't really have a user community that you would disappoint. I don't see that (b) was really in line with any proposed work in the latest grant and would likely need a champion with funding to lead the effort to resurrect it. If the only reason to keep it is to show what can be done with Bro, the papers already do that and a broken piece of software doesn't help anyway. ------ Adam J. Slagell, CISSP Sr. Security Engineer National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info 217.244.8965 From vern at icir.org Mon Oct 25 22:48:01 2010 From: vern at icir.org (Vern Paxson) Date: Mon, 25 Oct 2010 22:48:01 -0700 Subject: [Bro-Dev] Opinions on trace rewriter? In-Reply-To: <20101025174113.GB17795@icir.org> (Mon, 25 Oct 2010 10:41:13 PDT). Message-ID: <20101026054801.0BC4536A413@taffy.ICSI.Berkeley.EDU> > That was indeed a thought I had, but because I don't want to > *pledge* it, I didn't say it; but you see where the "separate tool" > idea is coming from. :-) Well, in terms of "separate tool" that's still just half of it, though. One needs a Bro interpreter, too. > We could do a compromise: Remove it, but turn all the code into a > big patch that applies cleanly to the next release. Okay, that works for me. I like the notion of capturing the necessary pieces as a coherent whole. Vern From vern at icir.org Mon Oct 25 22:51:25 2010 From: vern at icir.org (Vern Paxson) Date: Mon, 25 Oct 2010 22:51:25 -0700 Subject: [Bro-Dev] Opinions on trace rewriter? In-Reply-To: (Mon, 25 Oct 2010 12:50:29 CDT). Message-ID: <20101026055125.C737D36A3D9@taffy.ICSI.Berkeley.EDU> > Not sure what I am stepping into, but here I go. I agree strongly with Robin. Yeah, I can appreciate that viewpoint. The counterpoint is; > I don't see that (b) was really in line with > any proposed work in the latest grant Sure, but Robin is proposing actively yanking out code; I wasn't arguing for putting further work into it, just leaving it be. > If the only reason to keep it is to show what can be done with Bro, the > papers already do that Only somewhat, because for this sort of capability (which I do get inquiries about time-to-time), telling folks that we removed it smacks of "it never really worked", which is more harsh than I think is deserved. In any case, I'm won over with the notion of having a coherent patch that captures what's been removed. Vern From robin at icir.org Tue Oct 26 09:07:26 2010 From: robin at icir.org (Robin Sommer) Date: Tue, 26 Oct 2010 09:07:26 -0700 Subject: [Bro-Dev] Opinions on trace rewriter? In-Reply-To: <20101026055125.C737D36A3D9@taffy.ICSI.Berkeley.EDU> References: <20101026055125.C737D36A3D9@taffy.ICSI.Berkeley.EDU> Message-ID: <20101026160726.GA67227@icir.org> On Mon, Oct 25, 2010 at 22:51 -0700, you wrote: > Only somewhat, because for this sort of capability (which I do get inquiries > about time-to-time), telling folks that we removed it smacks of "it never > really worked", which is more harsh than I think is deserved. (At some point I was trying to figure out what the last version was in which the rewriter "just worked" out of the box. I don't remember the details anymore but I had to go back *very* far, and even then there was still some trouble I can't recall right now.) > In any case, I'm won over with the notion of having a coherent patch that > captures what's been removed. Ok. If we do the yanking in a separate branch, it should be pretty easy to make a big patch for putting the stuff back in; we just need to reverse the direction of the branch's diff I would think. Once the next release is out, we can also create a tagged branch 1.6-with-rewriter that has the patch already applied. Then we can just point people to it. That's even easier than keeping the patch itself around.. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro-dev at bro-ids.org Wed Oct 27 07:41:27 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Wed, 27 Oct 2010 14:41:27 -0000 Subject: [Bro-Dev] #292: Cluttering of alarm.log with serialization messages Message-ID: <047.7dd189330e344b30c6d647cacb8d3cd8@bro-ids.org> #292: Cluttering of alarm.log with serialization messages ---------------------+------------------------------------------------------ Reporter: vallenti | Owner: robin Type: Problem | Status: new Priority: Normal | Component: BroControl Version: 1.5.1 | Keywords: ---------------------+------------------------------------------------------ In a standard setup with broctl, I find that `alarm.log` gets cluttered with a lot of messages like: {{{ 1288164542.559074 Starting incremental serialization... 1288164542.682096 Finished incremental serialization. 1288165442.559114 Starting incremental serialization... 1288165442.680014 Finished incremental serialization. 1288166342.559481 Starting incremental serialization... 1288166342.679497 Finished incremental serialization. ... }}} I don't remember the exact mechanism to tweak these messages (notice policies?), but since they don't seem critical, I suggest removing them from `alarm.log` in the default setup. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Wed Oct 27 07:51:48 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Wed, 27 Oct 2010 14:51:48 -0000 Subject: [Bro-Dev] #293: scripts/local-interfaces cannot find ifconfig Message-ID: <047.ac9986b0dbc785e69d293483a6ce99e7@bro-ids.org> #293: scripts/local-interfaces cannot find ifconfig ---------------------+------------------------------------------------------ Reporter: vallenti | Owner: robin Type: Problem | Status: new Priority: Normal | Component: BroControl Version: 1.5.1 | Keywords: ---------------------+------------------------------------------------------ I find the following error in `debug.log`: {{{ 28 Oct 01:25:01 [local] /usr/local/share/broctl/scripts/local- interfaces 28 Oct 01:25:01 [local] exit code 0 28 Oct 01:25:01 Local IPs: /usr/local/share/broctl/scripts/local- interfaces: line 9: ifconfig: command not found 28 Oct 01:25:01 [local] sh }}} If the log entries reflect serial execution, this is a little weird because `local-interfaces` first terminates cleanly with exit code 0 but then cannot find `ifconfig`. Since `local-interfaces` merely invokes `ifconfig` from bash, I wonder how this can happen. The system I am working with is a vanilla CentOs 5.5 Linux where `ifconfig` is located in `/sbin`. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Wed Oct 27 19:06:11 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Thu, 28 Oct 2010 02:06:11 -0000 Subject: [Bro-Dev] #292: Cluttering of alarm.log with serialization messages In-Reply-To: <047.7dd189330e344b30c6d647cacb8d3cd8@bro-ids.org> References: <047.7dd189330e344b30c6d647cacb8d3cd8@bro-ids.org> Message-ID: <056.b14ed18dd486b1846b01b6264a26af79@bro-ids.org> #292: Cluttering of alarm.log with serialization messages ---------------------+------------------------------------------------------ Reporter: vallenti | Owner: robin Type: Problem | Status: new Priority: Normal | Component: BroControl Version: 1.5.1 | Keywords: ---------------------+------------------------------------------------------ Comment(by robin): Yepp, we need come up with a way to log stuff that's just informational but not related to any specific analysis (and thus would have its own log file). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Wed Oct 27 19:08:00 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Thu, 28 Oct 2010 02:08:00 -0000 Subject: [Bro-Dev] #293: scripts/local-interfaces cannot find ifconfig In-Reply-To: <047.ac9986b0dbc785e69d293483a6ce99e7@bro-ids.org> References: <047.ac9986b0dbc785e69d293483a6ce99e7@bro-ids.org> Message-ID: <056.1b47e7ddf9128aafb1c164632b22370d@bro-ids.org> #293: scripts/local-interfaces cannot find ifconfig ---------------------+------------------------------------------------------ Reporter: vallenti | Owner: robin Type: Problem | Status: new Priority: Normal | Component: BroControl Version: 1.5.1 | Keywords: ---------------------+------------------------------------------------------ Comment(by robin): The reason for the "exit 0" is probably that the helper script fails to recognize that there's a problem. Regarding ifconfig, it is in the PATH? (On Linux, often it's not for users) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro-dev at bro-ids.org Wed Oct 27 19:54:20 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Thu, 28 Oct 2010 02:54:20 -0000 Subject: [Bro-Dev] #293: scripts/local-interfaces cannot find ifconfig In-Reply-To: <047.ac9986b0dbc785e69d293483a6ce99e7@bro-ids.org> References: <047.ac9986b0dbc785e69d293483a6ce99e7@bro-ids.org> Message-ID: <056.7515daae13c5bfcb09a14735a6b2dadc@bro-ids.org> #293: scripts/local-interfaces cannot find ifconfig ---------------------+------------------------------------------------------ Reporter: vallenti | Owner: robin Type: Problem | Status: new Priority: Normal | Component: BroControl Version: 1.5.1 | Keywords: ---------------------+------------------------------------------------------ Comment(by vallenti): Replying to [comment:1 robin]: > The reason for the "exit 0" is probably that the helper script fails to recognize that there's a problem. Is `local-interfaces` used somewhere other than in `debug.log`? Or put differently, does it matter if this command fails? > Regarding ifconfig, it is in the PATH? (On Linux, often it's not for users) Annoyingly, this is the case here. On Linux, `/usr/bin/env bash` is not a login shell and hence does not source `/etc/profile`, which usually extends `$PATH` with the various `**/sbin` directories. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Thu Oct 28 11:54:38 2010 From: seth at icir.org (Seth Hall) Date: Thu, 28 Oct 2010 14:54:38 -0400 Subject: [Bro-Dev] Local MPLS coding support In-Reply-To: <4CC99666.9000807@iu.edu> References: <4CC99666.9000807@iu.edu> Message-ID: Hi Keith! I left the full message you sent me for the benefit of everyone on the bro-dev mailing list. I just pushed a branch to our git repository that has more complete support for MPLS. I'd appreciate it if you could try it. I added some functionality to it that the patch in the tracker doesn't have. * Supports MPLS over Ethernet (which is likely what you're sniffing). * Builds the appropriate BPF filter so that MPLS encapsulated and non-encapsulated traffic is sent through. You just need to make sure and load the mpls.bro script. It *should* just work from that point (no filters to add or anything). If you define your own filter on the command line (with the -f flag), then the auto mpls support will not work. You will have to define the appropriate mpls filter in that case. If this works for you, then you can have your intern work on something else. :P Let us know if you have staff time needing projects to work on, I'm sure we can figure something out. git clone git://git.icir.org/bro git checkout origin/topic/seth/mpls (ignore all of the warnings) .Seth On Oct 28, 2010, at 11:27 AM, Keith Lehigh wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi Seth, > Hope all is going well for you with getting your head wrapped around > plans for the future of Bro. > We may have some hours available from an intern here to put to work > on solving our MPLS problem with Bro. This is one of a couple different > projects I have in mind, but before I get too far into considering this, > I wanted to get a little input from you. > I would aim to have this support added in such a way that it is > generic to any flavor of encapsulation and could be contributed back as > patches. Given that this person won't be working for us long term, I'm > not interested in having him make some local hacks which quickly > become unmaintainable. As such, do you think the internals of Bro will > change enough in the near future to make this irrelevant? > This question may be more directed at Robin. Am I being overly > naive in the amount of effort and rewriting that would need to be done > to make this work? I'm pretty sure the intern will need to get some > time in just getting up to speed on Bro, but I think he doesn't have to > be a subject matter expert on Bro. Again, am I being naive? > If we go with this project, I'll direct our intern to put questions > to the Bro list so the community gains from the discussion and have him > make comments/submissions on the tracker ticket for this issue. > Thanks! > > - - Keith > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.5 (Darwin) > > iD8DBQFMyZZmW5AQrvjB4mcRAuEUAJ9Fj7Ti+YuvfvXNzFd9uMAqN7x8OQCeNfUG > a1TeCo9z20D+hjHx5868oYE= > =7VFU > -----END PGP SIGNATURE----- From seth at icir.org Thu Oct 28 13:04:02 2010 From: seth at icir.org (Seth Hall) Date: Thu, 28 Oct 2010 16:04:02 -0400 Subject: [Bro-Dev] Local MPLS coding support In-Reply-To: <4CC99666.9000807@iu.edu> References: <4CC99666.9000807@iu.edu> Message-ID: Hi Keith! I left the full message you sent me for the benefit of everyone on the bro-dev mailing list. I just pushed a branch to our git repository that has more complete support for MPLS. I'd appreciate it if you could try it. I added some functionality to it that the patch in the tracker doesn't have. * Supports MPLS over Ethernet (which is likely what you're sniffing). * Builds the appropriate BPF filter so that MPLS encapsulated and non-encapsulated traffic is sent through. You just need to make sure and load the mpls.bro script. It *should* just work from that point (no filters to add or anything). If you define your own filter on the command line (with the -f flag), then the auto mpls support will not work. You will have to define the appropriate mpls filter in that case. If this works for you, then you can have your intern work on something else. :P Let us know if you have staff time needing projects to work on, I'm sure we can figure something out. git clone git://git.icir.org/bro git checkout origin/topic/seth/mpls (ignore all of the warnings) .Seth On Oct 28, 2010, at 11:27 AM, Keith Lehigh wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi Seth, > Hope all is going well for you with getting your head wrapped around > plans for the future of Bro. > We may have some hours available from an intern here to put to work > on solving our MPLS problem with Bro. This is one of a couple different > projects I have in mind, but before I get too far into considering this, > I wanted to get a little input from you. > I would aim to have this support added in such a way that it is > generic to any flavor of encapsulation and could be contributed back as > patches. Given that this person won't be working for us long term, I'm > not interested in having him make some local hacks which quickly > become unmaintainable. As such, do you think the internals of Bro will > change enough in the near future to make this irrelevant? > This question may be more directed at Robin. Am I being overly > naive in the amount of effort and rewriting that would need to be done > to make this work? I'm pretty sure the intern will need to get some > time in just getting up to speed on Bro, but I think he doesn't have to > be a subject matter expert on Bro. Again, am I being naive? > If we go with this project, I'll direct our intern to put questions > to the Bro list so the community gains from the discussion and have him > make comments/submissions on the tracker ticket for this issue. > Thanks! > > - - Keith > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.5 (Darwin) > > iD8DBQFMyZZmW5AQrvjB4mcRAuEUAJ9Fj7Ti+YuvfvXNzFd9uMAqN7x8OQCeNfUG > a1TeCo9z20D+hjHx5868oYE= > =7VFU > -----END PGP SIGNATURE----- From bro-dev at bro-ids.org Sat Oct 30 13:25:14 2010 From: bro-dev at bro-ids.org (Bro Tracker) Date: Sat, 30 Oct 2010 20:25:14 -0000 Subject: [Bro-Dev] #293: scripts/local-interfaces cannot find ifconfig In-Reply-To: <047.ac9986b0dbc785e69d293483a6ce99e7@bro-ids.org> References: <047.ac9986b0dbc785e69d293483a6ce99e7@bro-ids.org> Message-ID: <056.94c9577bf483f6bf9518c4184399f11e@bro-ids.org> #293: scripts/local-interfaces cannot find ifconfig ---------------------+------------------------------------------------------ Reporter: vallenti | Owner: robin Type: Problem | Status: new Priority: Normal | Component: BroControl Version: 1.5.1 | Keywords: ---------------------+------------------------------------------------------ Comment(by robin): > Is `local-interfaces` used somewhere other than in `debug.log`? Or put differently, does it matter if this command fails? It's used to figure out what the local IPs are, which in turn is used to determine whether the script is running on the manager. If you see nothing that's obviously failing, you should be fine. > Annoyingly, this is the case here. On Linux, `/usr/bin/env bash` is not a login shell and hence does not source `/etc/profile`, which usually extends `$PATH` with the various `**/sbin` directories. Hmmm?. Perhaps broctl should just add */sbin to its standard PATH in any case. -- Ticket URL: Bro Tracker Bro Issue Tracker