From bro at tracker.icir.org Sun May 1 16:32:29 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Sun, 01 May 2011 23:32:29 -0000 Subject: [Bro-Dev] #446: git search is busted Message-ID: <043.69dbc72c406eb11707912b1db7deefd7@tracker.icir.org> #446: git search is busted ---------------------+----------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Keywords: | ---------------------+----------------- I just did a search for "snaplen" and got the output I've attached. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Sun May 1 16:34:07 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Sun, 01 May 2011 23:34:07 -0000 Subject: [Bro-Dev] #447: Bro snaplen Message-ID: <043.c751a05001e39b08cc735efcea989468@tracker.icir.org> #447: Bro snaplen ---------------------+----------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Keywords: | ---------------------+----------------- PktSrc.cc sets snaplen to 8192. That used to be ample to ensure full- packet capture, but isn't any longer for links with Jumbo frames. It should be set to either 65535 or (better, if indeed generally supported), 0, which says "entire packets please". -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Sun May 1 21:38:16 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 02 May 2011 04:38:16 -0000 Subject: [Bro-Dev] #446: git search is busted In-Reply-To: <043.69dbc72c406eb11707912b1db7deefd7@tracker.icir.org> References: <043.69dbc72c406eb11707912b1db7deefd7@tracker.icir.org> Message-ID: <058.74b2fbb6867b6b5740da565821269991@tracker.icir.org> #446: git search is busted ----------------------+----------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Resolution: | Keywords: ----------------------+----------------- Comment (by robin): Yeah, it's actually a Trac error caused by git having moved to the new server. Trac will move over soon as well, and this will be solved. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Sun May 1 21:38:34 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 02 May 2011 04:38:34 -0000 Subject: [Bro-Dev] #446: git search is busted In-Reply-To: <043.69dbc72c406eb11707912b1db7deefd7@tracker.icir.org> References: <043.69dbc72c406eb11707912b1db7deefd7@tracker.icir.org> Message-ID: <058.e2b94c7c55b71849d375a182c97f7762@tracker.icir.org> #446: git search is busted ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Changes (by robin): * milestone: => Bro1.6 -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Sun May 1 21:46:59 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 1 May 2011 21:46:59 -0700 Subject: [Bro-Dev] cmake/Python problem Message-ID: <20110502044659.GA66149@icir.org> Hi Jon, I've run into an issue with the CMake setup for python-broccoli and pysubnetree. The problem occurs on a machine that has Python 2.4 installed system-wide, but where I have installed a self-compiled 2.6 into my home. CMake does detect the 2.6 binary correctly (it's in the PATH), but its PythonLibs still goes for the 2.4 libraries. I think the underlying problem is that PythonLibs searches for the libraries itself, while it should better ask the "python-config" that corresponds to what PythonInterp found. I'm attaching a patch that runs python-config and then adapts CMAKE_LIBRARY_PATH accordingly before it does PythonLibs. Patching both python-broccoli and pysubnettree with this solves my problem. However, I'm not sure this is indeed the cleanest way of doing it. Could you take a look? Thanks, Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.icir.org Sun May 1 21:47:27 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 02 May 2011 04:47:27 -0000 Subject: [Bro-Dev] #446: git search is busted In-Reply-To: <043.69dbc72c406eb11707912b1db7deefd7@tracker.icir.org> References: <043.69dbc72c406eb11707912b1db7deefd7@tracker.icir.org> Message-ID: <058.3ce139636d0cf44f579724700dfc209e@tracker.icir.org> #446: git search is busted ----------------------+----------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Resolution: | Keywords: ----------------------+----------------- Changes (by robin): * milestone: Bro1.6 => -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Sun May 1 21:47:42 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 02 May 2011 04:47:42 -0000 Subject: [Bro-Dev] #447: Bro snaplen In-Reply-To: <043.c751a05001e39b08cc735efcea989468@tracker.icir.org> References: <043.c751a05001e39b08cc735efcea989468@tracker.icir.org> Message-ID: <058.3c5c1b5afc4ef19ac718769bd34bbde5@tracker.icir.org> #447: Bro snaplen ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Changes (by robin): * milestone: => Bro1.6 -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Sun May 1 22:20:28 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 1 May 2011 22:20:28 -0700 Subject: [Bro-Dev] cmake/Python problem In-Reply-To: <20110502044659.GA66149@icir.org> References: <20110502044659.GA66149@icir.org> Message-ID: <20110502052028.GA86768@icir.org> And now the attachment. :) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org -------------- next part -------------- diff --git a/aux/broctl/BroControl/options.py b/aux/broctl/BroControl/options.py index 7d851c3..931e9ab 100644 --- a/aux/broctl/BroControl/options.py +++ b/aux/broctl/BroControl/options.py @@ -30,6 +30,8 @@ options = [ "True to let backends capture short-term traces via '-w'. These are not archived but might be helpful for debugging."), Option("DevMode", "0", "bool", Option.USER, False, "Enable development mode, which changes how things are installed by the _install_ command."), + Option("RemoveState", "0", "bool", Option.USER, False, + "True if past persistent state should be removed before starting Bro."), Option("LogDir", "${BroBase}/logs", "string", Option.USER, False, "Directory for archived log files."), diff --git a/aux/broctl/bin/run-bro.in b/aux/broctl/bin/run-bro.in index d95a8da..c6cf6ef 100755 --- a/aux/broctl/bin/run-bro.in +++ b/aux/broctl/bin/run-bro.in @@ -56,12 +56,12 @@ else echo -1 >.pid exit 1 fi - -# ulimit -a -# echo - tmpbro=${tmpexecdir}/`basename ${bro}` +if [ "${removestate}" != "0" ]; then + rm -rf .state +fi + rm -f $tmpbro cp -p ${bro} $tmpbro sleep 1 From leres at ee.lbl.gov Sun May 1 22:35:03 2011 From: leres at ee.lbl.gov (Craig Leres) Date: Sun, 01 May 2011 22:35:03 -0700 Subject: [Bro-Dev] cmake/Python problem In-Reply-To: <20110502044659.GA66149@icir.org> References: <20110502044659.GA66149@icir.org> Message-ID: <4DBE4287.90704@ee.lbl.gov> > I think the underlying problem is that PythonLibs searches for the > libraries itself, while it should better ask the "python-config" that > corresponds to what PythonInterp found. I didn't look at the patch but I agree this is the right way to do this. Craig From bro at tracker.icir.org Mon May 2 07:36:58 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 02 May 2011 14:36:58 -0000 Subject: [Bro-Dev] #442: Hardcode Python path In-Reply-To: <044.75c677aee0c63a0465103d3162763279@tracker.icir.org> References: <044.75c677aee0c63a0465103d3162763279@tracker.icir.org> Message-ID: <059.c8be3372ace9b2584d6fd26f06c30f90@tracker.icir.org> #442: Hardcode Python path -------------------------+-------------------- Reporter: robin | Owner: jsiwek Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: -------------------------+-------------------- Comment (by jsiwek): > BroControl currently uses `#! /usr/bin/env` to find the Python binary at runtime. Craig suggested this instead: > > > I think the best way to go is to find python on the users path at the > > time bro is built and use that full path in all python scripts (e.g. at > > the point where the broctl.in template is transformed into broctl). > > Jon, is that something we can teach cmake to do? It's possible, but can you explain the underlying issue with `#! /usr/bin/env python` ? By using that in the shebang, the user selects which python they want by re-arranging their `PATH`. If an absolute path to a python interpreter is set in the shebang at configure/build time, then to change the python interpreter that is used, the user has to either (1) edit the script(s) or (2) re- configure/build/install `broctl`. These seem like the more "difficult" options to me. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon May 2 08:12:55 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 2 May 2011 08:12:55 -0700 Subject: [Bro-Dev] cmake/Python problem In-Reply-To: <20110502052028.GA86768@icir.org> References: <20110502044659.GA66149@icir.org> <20110502052028.GA86768@icir.org> Message-ID: <20110502151255.GA3985@icir.org> On Sun, May 01, 2011 at 22:20 -0700, I wrote: > And now the attachment. :) Grmpf. Now the the *right* attachment. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org -------------- next part -------------- index c1880b9..11ac74c 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -60,9 +60,20 @@ include(FindRequiredPackage) FindRequiredPackage(Broccoli) FindRequiredPackage(SWIG) -FindRequiredPackage(PythonLibs) FindRequiredPackage(PythonInterp) +execute_process(COMMAND ${PYTHON_EXECUTABLE}-config --prefix + OUTPUT_VARIABLE PYTHON_PREFIX + OUTPUT_STRIP_TRAILING_WHITESPACE + ERROR_QUIET + ) + +if ( PYTHON_PREFIX ) + set(CMAKE_LIBRARY_PATH ${PYTHON_PREFIX}/lib ${PYTHON_PREFIX}/lib ${CMAKE_LIBRARY_PATH}) +endif () + +FindRequiredPackage(PythonLibs) + if (MISSING_PREREQS) foreach (prereq ${MISSING_PREREQ_DESCS}) message(SEND_ERROR ${prereq}) From jsiwek at ncsa.illinois.edu Mon May 2 09:58:33 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 2 May 2011 11:58:33 -0500 (CDT) Subject: [Bro-Dev] cmake/Python problem In-Reply-To: <20110502151255.GA3985@icir.org> Message-ID: <27069479.4.1304355510455.JavaMail.jsiwek@tangent.ncsa.illinois.edu> > Grmpf. Now the the *right* attachment. Seems like a good method to use (I didn't see any innate/obvious way to get FindPythonLibs.cmake to be aware of the result from FindPythonInterp.cmake). The patch seems to add two "${PYTHON_PREFIX}/lib" paths, though, which I don't understand: set(CMAKE_LIBRARY_PATH ${PYTHON_PREFIX}/lib ${PYTHON_PREFIX}/lib ${CMAKE_LIBRARY_PATH}) Is that just a typo? - Jon From bro at tracker.icir.org Mon May 2 10:14:00 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 02 May 2011 17:14:00 -0000 Subject: [Bro-Dev] #448: Broccoli does not support IPv6 addresses Message-ID: <044.c83bbd0bcfd549facc28fff71c9b2918@tracker.icir.org> #448: Broccoli does not support IPv6 addresses ----------------------+---------------------- Reporter: robin | Owner: kreibich Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Broccoli | Version: Keywords: ipv6 | ----------------------+---------------------- Need to add support, and adapt python-broccoli. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon May 2 11:10:03 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 2 May 2011 11:10:03 -0700 Subject: [Bro-Dev] cmake/Python problem In-Reply-To: <27069479.4.1304355510455.JavaMail.jsiwek@tangent.ncsa.illinois.edu> References: <20110502151255.GA3985@icir.org> <27069479.4.1304355510455.JavaMail.jsiwek@tangent.ncsa.illinois.edu> Message-ID: <20110502181003.GA9443@icir.org> On Mon, May 02, 2011 at 11:58 -0500, you wrote: > Is that just a typo? Yes and no. I was wondering whether I need to add both lib64 and lib, and seems I didn't make up my mind. :) But I guess it should have both. Ok, I'll go ahead and commit this. Thanks, Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.icir.org Mon May 2 14:27:47 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 02 May 2011 21:27:47 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.00dea6917fe9998614ad65daa827a594@tracker.icir.org> #435: topic/jsiwek/doc-framework ---------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Task | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by jsiwek): This branch now has changes that we've talked about: * move `doc/` to `doc/scripts/` * delete `.state` dir before/after generating docs for a given script * fix `@load`s from subdirectories, e.g. `@load foo/bar` in a way that avoid conflicts with `@load baz/bar` * I chose to use a directory hierarchy in the Sphinx setup -- which should automatically pick up on subdirs that one uses when defining a script-to-document in the `CMakeLists.txt` * bro parse errors now generate a hint to check `##` comments when in doc mode * add a btest that tests success of `make restdoc` (generates reST for all scripts defined in `CMakeLists.txt`, but doesn't do the Sphinx part) * I know we said getting incremental builds wasn't a requirement, but it should now work for the `make restdoc` part of it. The other half of it, Sphinx creating HTML from reST, is still not incremental. From my effort so far, it was difficult to figure out the right way to bridge the two build systems (CMake and Sphinx). I think the new logging stuff is stable enough for me to try to start tracking it in the documentation right? Want me to try to get that working in some manner before you merge this branch again? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Mon May 2 14:55:37 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 02 May 2011 21:55:37 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.5e47d13e9bbaedc946805bfb70f3ba8f@tracker.icir.org> #435: topic/jsiwek/doc-framework ---------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Task | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by robin): On Mon, May 02, 2011 at 21:27 -0000, you wrote: > This branch now has changes that we've talked about: Cool, thanks! > I think the new logging stuff is stable enough for me to try to start > tracking it in the documentation right? Want me to try to get that > working in some manner before you merge this branch again? Yes to both. :) Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon May 2 15:35:10 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 2 May 2011 15:35:10 -0700 Subject: [Bro-Dev] dns.bro Message-ID: <20110502223510.GA19278@icir.org> As discussed, I'll start going through the scripts in the policy-scripts-new branch, starting with dns.bro. I really like the new dns.log, pretty neat! Two general, DNS-independent, questions first: - Does a policy/foo.bro script always load all of policy/foo/*.bro? Would be nice if that was consistent, and perhaps it already is. :-) - We should include new connection$uid into pretty much all relevant logs. dns/base.bro: - There are number of commented out "print" statements. Should we pass this into weird.bro? - The script activates the binpac analyzer. Do we want to remove "classic" C++ analyzer? - There's a TODO about the EDNS/TSIG. What's the problem? - The reply handlers check for "ans$answer_type == DNS_ANS", but there are also options dns_skip_all_auth/dns_skip_all_addl in bro.init? Can we get rid of one of the two ways (I'd say the latter)? - The reply handlers are all almost identical. How about refactoring that code into a function called by them all? - The comment in connection_state_remove() seems misleading: this is the only place that logs anything, right? Watislat_ctveusd or - om o te yps n ontsbr dn' semto Do we want to delete he? dns/passive-replication.bro - Can you remind me what the passive replication is for? I thought I knew but not sure that's matching with the script. :-) - Regarding the TODO: should "recent_requests" be a table[string] of set[string]? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon May 2 15:43:38 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 2 May 2011 15:43:38 -0700 Subject: [Bro-Dev] conn.bro Message-ID: <20110502224338.GB19278@icir.org> - ts: I'd prefer to keep this the timestamp of the first packet, that's more well defined. - Did we decide whether we want to add the raw byte volume via Gregor's patch? I'd say so. I'll see that that gets merged in. - I'd say let's include history by default, but not addl. - "logging" isn't used. - Rename "connection$log" to "connection$conn" for consistency? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon May 2 16:06:24 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 2 May 2011 16:06:24 -0700 Subject: [Bro-Dev] conn-size Message-ID: <20110502230624.GC19278@icir.org> Gregor, the following is part of your conn-size patch: - RecordVal* tmp_rc = resp_endp; - resp_endp = orig_endp; - orig_endp = tmp_rc; - + // Delete conn_val, will build a new one next time we need one + // XXX: breaks the assumption that conn_val never changes!! Why does a swap not work anymore? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.icir.org Mon May 2 16:39:13 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 02 May 2011 23:39:13 -0000 Subject: [Bro-Dev] #449: Move scripts tests from old test-suite over to btest Message-ID: <044.80f9fcd5de0ed7dbf1d4e580bd082da3@tracker.icir.org> #449: Move scripts tests from old test-suite over to btest --------------------+-------------------- Reporter: robin | Owner: robin Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | --------------------+-------------------- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Mon May 2 17:12:51 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 03 May 2011 00:12:51 -0000 Subject: [Bro-Dev] #429: Change vector indices to be zero-based In-Reply-To: <044.3f36644cc028ca5ce232deb62f75f6cb@tracker.icir.org> References: <044.3f36644cc028ca5ce232deb62f75f6cb@tracker.icir.org> Message-ID: <059.b7385126cf6ba88cfa4011f49f319365@tracker.icir.org> #429: Change vector indices to be zero-based -----------------------------+-------------------- Reporter: robin | Owner: robin Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Tue May 3 09:07:01 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 3 May 2011 09:07:01 -0700 Subject: [Bro-Dev] dns.bro In-Reply-To: <20110502223510.GA19278@icir.org> References: <20110502223510.GA19278@icir.org> Message-ID: <20110503160701.GA52202@icir.org> On Mon, May 02, 2011 at 15:35 -0700, I wrote: > Watislat_ctveusd or > - om o te yps n ontsbr dn' semto Do we > want to delete he? I have no idea what happened here. :) That's supposed to be: - What is last_active used for? - Some of the types in consts.bro don't seem to be used. Do we want to delete them? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Tue May 3 10:02:42 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 03 May 2011 10:02:42 -0700 Subject: [Bro-Dev] conn-size In-Reply-To: <20110502230624.GC19278@icir.org> References: <20110502230624.GC19278@icir.org> Message-ID: <4DC03532.9040802@icir.org> Hi, actually swapping still works. The code below is a left-over from an earlier iteration of the conn-size code in which the swapping did not work.... Want me to change it back or can you do so when you merge? thx gregor On 5/2/11 16:06 , Robin Sommer wrote: > Gregor, > > the following is part of your conn-size patch: > > - RecordVal* tmp_rc = resp_endp; > - resp_endp = orig_endp; > - orig_endp = tmp_rc; > - > + // Delete conn_val, will build a new one next time we need one > + // XXX: breaks the assumption that conn_val never changes!! > > > Why does a swap not work anymore? > > Robin > -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From robin at icir.org Tue May 3 10:06:32 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 3 May 2011 10:06:32 -0700 Subject: [Bro-Dev] conn-size In-Reply-To: <4DC03532.9040802@icir.org> References: <20110502230624.GC19278@icir.org> <4DC03532.9040802@icir.org> Message-ID: <20110503170632.GC7788@icir.org> On Tue, May 03, 2011 at 10:02 -0700, you wrote: > The code below is a left-over from an earlier iteration of the conn-size > code in which the swapping did not work.... Want me to change it back or > can you do so when you merge? Yes, please change it back then. Thanks, Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Tue May 3 10:17:49 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 03 May 2011 10:17:49 -0700 Subject: [Bro-Dev] conn-size In-Reply-To: <20110503170632.GC7788@icir.org> References: <20110502230624.GC19278@icir.org> <4DC03532.9040802@icir.org> <20110503170632.GC7788@icir.org> Message-ID: <4DC038BD.5040407@icir.org> will do. Is there a trace to test the swapping behavior? On 5/3/11 10:06 , Robin Sommer wrote: > > On Tue, May 03, 2011 at 10:02 -0700, you wrote: > >> The code below is a left-over from an earlier iteration of the conn-size >> code in which the swapping did not work.... Want me to change it back or >> can you do so when you merge? > > Yes, please change it back then. > > Thanks, > > Robin > -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From gregor at icir.org Tue May 3 10:31:02 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 03 May 2011 10:31:02 -0700 Subject: [Bro-Dev] conn-size In-Reply-To: <20110502230624.GC19278@icir.org> References: <20110502230624.GC19278@icir.org> Message-ID: <4DC03BD6.30108@icir.org> Actually there's a bit more to that. Even the original code did a Unref(conn_val); conn_val = 0; in Connection::FlipRoles(), i.e. the full conn_val had to be rebuild after a FlipRole() and only the endpoint pointers were reused as the original code also kept distinct pointers to the endp_vals. My code however doesn't keep pointers to the endp_vals, because I now use a root_analyzer->UpateConnVal() instead of two root_analyzer->UpdateEndpointVal() calls. As per our general discussion, I'm doing this so that any child analyzer could now update or extend the conn_val or the endpoint vals. So, swapping doesn't make sense anymore unless we keep conn_val (in which case conn_id and the endp_vals have to be replaced). (However, I just saw that the conn_size analyzer doesn't flip its counters. Will fix that.) (sorry for the first, incorrect e-mail. Should reply to e-mails before my first coffee) cu gregor On 5/2/11 16:06 , Robin Sommer wrote: > Gregor, > > the following is part of your conn-size patch: > > - RecordVal* tmp_rc = resp_endp; > - resp_endp = orig_endp; > - orig_endp = tmp_rc; > - > + // Delete conn_val, will build a new one next time we need one > + // XXX: breaks the assumption that conn_val never changes!! > > > Why does a swap not work anymore? > > Robin > -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From seth at icir.org Tue May 3 11:13:49 2011 From: seth at icir.org (Seth Hall) Date: Tue, 3 May 2011 14:13:49 -0400 Subject: [Bro-Dev] dns.bro In-Reply-To: <20110502223510.GA19278@icir.org> References: <20110502223510.GA19278@icir.org> Message-ID: On May 2, 2011, at 6:35 PM, Robin Sommer wrote: > - Does a policy/foo.bro script always load all of > policy/foo/*.bro? Would be nice if that was consistent, and > perhaps it already is. :-) It loads a subset that I believe are a mix between general utility and performance. More consistency would nice, but I'm not exactly sure how to get it. Any ideas? > - We should include new connection$uid into pretty much all > relevant logs. I'll start doing that. > - There are number of commented out "print" statements. Should we > pass this into weird.bro? Yes, I'll do that. I forgot to go back and do something with it. > - The script activates the binpac analyzer. Do we want to remove > "classic" C++ analyzer? The DNS analyzer would only be used if the --use-binpac flag is given to the bro binary. A long time ago I had segfaulting trouble with the DNS binpac analyzer though (which scares me a bit). We need to run it on live traffic for a while to see how it handles itself. By default the non-binpac analyzer will still be used so we should be good for the moment. > - There's a TODO about the EDNS/TSIG. What's the problem? I've never worked with those messages (and don't really know much about them). I keep meaning to look into how to handle them but I haven't gotten to it yet. > - The reply handlers check for "ans$answer_type == DNS_ANS", but > there are also options dns_skip_all_auth/dns_skip_all_addl in > bro.init? Can we get rid of one of the two ways (I'd say the > latter)? Do you think it could cause any performance problems due to the frequently high volume of DNS traffic? I suppose that's why the variables were implemented in the first place. I do like it better with the check in the handler though. We could always pass everything through and see if anyone is having trouble with performance. I actually just tested this a bit and my trace file a lot of DNS traffic didn't really take any longer to analyze with the auth and addl responses being passed through or not. I propose we remove all options from the analyzer related to not triggering on auth/addl RRs and handle everything at the scripting layer (it's less of a surprise that way at least). I added a script named dns/auth-addl that adds fields to store those responses in the logs, but it doesn't currently work because there seem to be a bug with the &priority attribute on script layer defined and called events. I can't influence the order that the events are handled. I left a note in the export section of dns/base.bro that comments on what may be a part of the problem since if I define the event prototype (like in events.bif.bro), I can't handle the at all. > - The reply handlers are all almost identical. How about > refactoring that code into a function called by them all? Done. > - The comment in connection_state_remove() seems misleading: this > is the only place that logs anything, right? Oops, it's not supposed to be misleading. I forgot to implement the logging action once all responses have been received in addition to the final logging at state removal time. > - What is last_active used for? Good question. It's removed now. I was thinking it would nice to know when the last response was received, but I wasn't even doing anything with it. > - Some of the types in consts.bro don't seem to be used. Do we > want to delete them? Let's leave them, most of them are being used now. :) > - Can you remind me what the passive replication is for? I thought > I knew but not sure that's matching with the script. :-) It's to log responses received for queries (so you can see what a query resolved to at that specific time). I'm not completely happy with it yet, but if you run it you can see that it usually logs many fewer lines and the total log file size is greatly reduced as well. I don't know if it's even useful enough to include in the base distribution but I liked it because it's a good example of using the logging framework to build a new log. > - Regarding the TODO: should "recent_requests" be a table[string] > of set[string]? Probably, but I didn't want to suck up too much memory for that script. That's yet another one of those hairy decisions that might not work in some high volume situations. Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Tue May 3 11:47:56 2011 From: seth at icir.org (Seth Hall) Date: Tue, 3 May 2011 14:47:56 -0400 Subject: [Bro-Dev] conn.bro In-Reply-To: <20110502224338.GB19278@icir.org> References: <20110502224338.GB19278@icir.org> Message-ID: On May 2, 2011, at 6:43 PM, Robin Sommer wrote: > - ts: I'd prefer to keep this the timestamp of the first packet, > that's more well defined. Isn't that what c$start_time is? > - Did we decide whether we want to add the raw byte volume via > Gregor's patch? I'd say so. I'll see that that gets merged in. This isn't merged yet then? If not, I'll add it once it's merged. > - I'd say let's include history by default, but not addl. Done. > - "logging" isn't used. Removed. I think I want to handle that more generically since the splitting and filtering is the same for all log files now. > - Rename "connection$log" to "connection$conn" for consistency? Agreed and done. This script was much faster to update than the dns script. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.icir.org Tue May 3 12:13:46 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 03 May 2011 19:13:46 -0000 Subject: [Bro-Dev] #450: Problem with newline character in logging framework text output plugin Message-ID: <043.4559da9fa4216fc67373c4bf0059cdae@tracker.icir.org> #450: Problem with newline character in logging framework text output plugin ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- Newlines need to be changed to byte representation value for the logging framework since all logging is line based. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Tue May 3 12:26:38 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 03 May 2011 19:26:38 -0000 Subject: [Bro-Dev] #451: Remove DNS options for skipping auth/addl events Message-ID: <044.d0c328379a210115138f0cde9b8717ec@tracker.icir.org> #451: Remove DNS options for skipping auth/addl events --------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | --------------------+-------------------- Let's remove `dns_skip_all_*`. The core should always raise the auth/addl events and let the script decide whether they are interesting. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Tue May 3 12:28:28 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 03 May 2011 19:28:28 -0000 Subject: [Bro-Dev] #452: Bug with event &priority Message-ID: <044.69f1c0ad97894f6d07475d190fd95148@tracker.icir.org> #452: Bug with event &priority ---------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- From Seth's mail > I added a script named dns/auth-addl that adds fields to store those > responses in the logs, but it doesn't currently work because there > seem to be a bug with the &priority attribute on script layer defined > and called events. I can't influence the order that the events are > handled. I left a note in the export section of dns/base.bro that > comments on what may be a part of the problem since if I define the > event prototype (like in events.bif.bro), I can't handle the at all. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Tue May 3 12:31:41 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 03 May 2011 19:31:41 -0000 Subject: [Bro-Dev] #451: Remove DNS options for skipping auth/addl events In-Reply-To: <044.d0c328379a210115138f0cde9b8717ec@tracker.icir.org> References: <044.d0c328379a210115138f0cde9b8717ec@tracker.icir.org> Message-ID: <059.21dbf39317252473530ad63bef766cd7@tracker.icir.org> #451: Remove DNS options for skipping auth/addl events ---------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ---------------------+-------------------- Comment (by seth): also remove the dns_skip_* options (they are set[addr]) -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Tue May 3 12:34:53 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 3 May 2011 12:34:53 -0700 Subject: [Bro-Dev] dns.bro In-Reply-To: References: <20110502223510.GA19278@icir.org> Message-ID: <20110503193453.GA55991@icir.org> On Tue, May 03, 2011 at 14:13 -0400, you wrote: > It loads a subset that I believe are a mix between general utility and > performance. More consistency would nice, but I'm not exactly sure > how to get it. Any ideas? Not really ... But the problem I see is that it's hard to figure out what a script is *not* loading and what thus can make sense to add manually. How about at least documenting this explicityly for the top-level script, something like: Further scripts that one might one to load in addition are: - foo/bar1.bro - foo/bar2.bro > DNS binpac analyzer though (which scares me a bit). We need to run it > on live traffic for a while to see how it handles itself. Ok, let's postone that for the more general analyzer overhaul at some point. > addl responses being passed through or not. I propose we remove all > options from the analyzer related to not triggering on auth/addl RRs > and handle everything at the scripting layer (it's less of a surprise > that way at least). Agreed, filed a ticket. > responses in the logs, but it doesn't currently work because there > seem to be a bug with the &priority attribute on script layer defined > and called events. I've filed a ticket for that as well. > It's to log responses received for queries (so you can see what a > query resolved to at that specific time). I'm not completely happy > with it yet, but if you run it you can see that it usually logs many > fewer lines and the total log file size is greatly reduced as well. I think one thing that confused me was the low timeout of 10secs; I was expecting something longer, perhaps an hour or so (but then listing all replies during that interval). But not sure my mental model on using this data is right. Also, including a timestamp by default could be helpful? > distribution but I liked it because it's a good example of using the > logging framework to build a new log. It definitly is! > Probably, but I didn't want to suck up too much memory for that > script. That's yet another one of those hairy decisions that might > not work in some high volume situations. Yeah, but I'd say let's first at least try to do these things "right", even if more expensive. If it turns out to be a problem, we can still change it change/optimize later. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Tue May 3 12:37:23 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 3 May 2011 12:37:23 -0700 Subject: [Bro-Dev] conn.bro In-Reply-To: References: <20110502224338.GB19278@icir.org> Message-ID: <20110503193722.GB55991@icir.org> On Tue, May 03, 2011 at 14:47 -0400, you wrote: > > - ts: I'd prefer to keep this the timestamp of the first packet, > > that's more well defined. > > Isn't that what c$start_time is? Sorry, I wasn't clear: I prefer to have c$start_time *logged*. > This isn't merged yet then? If not, I'll add it once it's merged. Correct, working on it. Thanks! Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Tue May 3 12:57:19 2011 From: seth at icir.org (Seth Hall) Date: Tue, 3 May 2011 15:57:19 -0400 Subject: [Bro-Dev] conn.bro In-Reply-To: <20110503193722.GB55991@icir.org> References: <20110502224338.GB19278@icir.org> <20110503193722.GB55991@icir.org> Message-ID: <0F85A669-C9CD-4236-989F-5573DC2FEABD@icir.org> On May 3, 2011, at 3:37 PM, Robin Sommer wrote: >> Isn't that what c$start_time is? > > Sorry, I wasn't clear: I prefer to have c$start_time *logged*. It is. Do you want the field named "start_time"? I was just trying to keep consistency among those first several fields for all of the logs with the assumption that in each case the $ts field is the earliest evident activity for whatever the logged data is (initial request for http, first packet for conn, etc). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Tue May 3 16:20:16 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 3 May 2011 16:20:16 -0700 Subject: [Bro-Dev] conn.bro In-Reply-To: <0F85A669-C9CD-4236-989F-5573DC2FEABD@icir.org> References: <20110502224338.GB19278@icir.org> <20110503193722.GB55991@icir.org> <0F85A669-C9CD-4236-989F-5573DC2FEABD@icir.org> Message-ID: <20110503232016.GI55991@icir.org> On Tue, May 03, 2011 at 15:57 -0400, you wrote: > It is. Ok, I see that now, but then the comment is wrong, isn't it? ## This is the time at which the connection was "fully established"; ts: time &log; Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Tue May 3 20:52:06 2011 From: seth at icir.org (Seth Hall) Date: Tue, 3 May 2011 23:52:06 -0400 Subject: [Bro-Dev] conn.bro In-Reply-To: <20110503232016.GI55991@icir.org> References: <20110502224338.GB19278@icir.org> <20110503193722.GB55991@icir.org> <0F85A669-C9CD-4236-989F-5573DC2FEABD@icir.org> <20110503232016.GI55991@icir.org> Message-ID: On May 3, 2011, at 7:20 PM, Robin Sommer wrote: > Ok, I see that now, but then the comment is wrong, isn't it? > > ## This is the time at which the connection was "fully established"; > ts: time &log; Oops. Fixed. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.icir.org Wed May 4 08:49:36 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 04 May 2011 15:49:36 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.0439cd4d5ff7d135a1b43daa3be4c618@tracker.icir.org> #435: topic/jsiwek/doc-framework ---------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Task | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by jsiwek): Is there any objection to merging this branch into `policy-scripts-new` and picking up work there? I'd like to be able to more closely test the doc generation against real examples of the new policy scripts and logging. I think there's going to be bugs to be worked out before things play nice together and it's hard to fix them when I'm off working in an isolated branch. Other choices: 1) merge `policy-scripts-new` into `doc-framework` and continue to work separately from everything in `doc-framework`, which would have to be merged to `master` *after* `policy-scripts-new` 2) merge current `doc-framework` into `master` for the final time and delete it; I'll pick up remaining doc-related work in `policy-scripts-new` afterwards -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed May 4 09:19:46 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 04 May 2011 16:19:46 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.0eb66547a1597a7fa7a47f188552ba0b@tracker.icir.org> #435: topic/jsiwek/doc-framework ---------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Task | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by seth): Replying to [comment:18 jsiwek]: > 1) merge `policy-scripts-new` into `doc-framework` and continue to work separately from everything in `doc-framework`, which would have to be merged to `master` *after* `policy-scripts-new` I'm inclined to go this direction. I understand the difficulty though, I've been running into it over and over again with all of the changes going on. :) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed May 4 09:29:16 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 04 May 2011 16:29:16 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.d14da1e8f5836f0bfde0a6f994159e14@tracker.icir.org> #435: topic/jsiwek/doc-framework ---------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Task | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by jsiwek): > > 1) merge `policy-scripts-new` into `doc-framework` and continue to work separately from everything in `doc-framework`, which would have to be merged to `master` *after* `policy-scripts-new` > > I'm inclined to go this direction. I understand the difficulty though, I've been running into it over and over again with all of the changes going on. :) Doing it this way seems like it's harder for people working in `policy- scripts-new` to get active feedback on what the documentation will look like -- they'd have to switch branches to `doc-framework` to generate documentation. It's also likely that I'm going to be making minor tweaks to policy scripts with respect to documentation and it's going to be tedious to commit those changes to `policy-scripts-new`, switch to `doc-framework`, merge, test, repeat... -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed May 4 09:38:31 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 04 May 2011 16:38:31 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.a3369a4aa45d333ba5c3298f80ff983f@tracker.icir.org> #435: topic/jsiwek/doc-framework ---------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Task | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by seth): On May 4, 2011, at 12:29 PM, Bro Tracker wrote: > Doing it this way seems like it's harder for people working in `policy- > scripts-new` to get active feedback on what the documentation will look like Oh, good point. Maybe it would be better to just do it all in the policy- scripts-new branch? I guess I don't really have any problem with doing it either way, but it does seem easier to just do it there like you're saying. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed May 4 09:45:30 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 04 May 2011 16:45:30 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.0b76c7fdfead570f1c000a0222997043@tracker.icir.org> #435: topic/jsiwek/doc-framework ---------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Task | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by robin): On Wed, May 04, 2011 at 16:38 -0000, you wrote: > Oh, good point. Maybe it would be better to just do it all in the policy- > scripts-new branch? Fine with me, but let me merge doc-framework once into master before we start doing changes to the doc stuff directly in policy-scripts-new. That reduces what will be merged later when policy-scripts-new is ready. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed May 4 10:10:14 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 04 May 2011 17:10:14 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.c82644db30fd8794f7789253cce2ab0a@tracker.icir.org> #435: topic/jsiwek/doc-framework ---------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Task | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by jsiwek): > Fine with me, but let me merge doc-framework once into master before > we start doing changes to the doc stuff directly in > policy-scripts-new. That reduces what will be merged later when > policy-scripts-new is ready. Ok, I can change this to a merge request. But first, one more worry I thought of: we don't really have a robust/stable test suite for policy-scripts-new, right? The current btest suite has 15 failures for me with a fresh checkout. What should I be using to make sure new doc changes in policy-scripts-new doesn't cause regressions? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed May 4 10:54:20 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 04 May 2011 17:54:20 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.3b36514414506551b1c147e08e03a625@tracker.icir.org> #435: topic/jsiwek/doc-framework ---------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Task | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by robin): On Wed, May 04, 2011 at 17:10 -0000, you wrote: > But first, one more worry I thought of: we don't really have a > robust/stable test suite for policy-scripts-new, right? Not yet, right. Seth is working on it. :) > The current btest suite has 15 failures for me with a fresh checkout. Hmmm ... Can you send me "btest -d" output? I see only a couple right now myself, which are still due to hashing inconsistencies permutating the output. Something's still not determistic there. > What should I be using to make sure new doc changes in > policy-scripts-new doesn't cause regressions? There's not much more we can do right now unfortuantely. That may actually be an argument for development in the doc branch: then I can merge into master and run the old test-suite ... I guess it depends on how risky you deem the changes you're doing to do in terms of non-doc stuff. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed May 4 12:02:47 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 04 May 2011 19:02:47 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.49fbc628a72ab78570059aeb132500ac@tracker.icir.org> #435: topic/jsiwek/doc-framework ---------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Task | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by jsiwek): > I guess it depends on how risky you deem the changes you're doing to do in terms of non-doc stuff. My plan for tracking documentation of logging streams shouldn't be that bad (if it actually works like I think it will). But doc mode seems to fail badly when scripts start using that new feature of adding new fields to record types. Let me see if I can reproduce/understand/fix this in the doc branch before we do any merging. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed May 4 12:07:14 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 04 May 2011 19:07:14 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.44c087c5f055289bb5d178de9bb9831e@tracker.icir.org> #435: topic/jsiwek/doc-framework ---------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Task | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by robin): On Wed, May 04, 2011 at 19:02 -0000, you wrote: > My plan for tracking documentation of logging streams shouldn't be that > bad (if it actually works like I think it will). The logging stuff is probably the best tested piece right now, so I'm not so worried about that specific part. :-) Btw, I'm looking forward to the stream documentation, that will be very helpful. > of adding new fields to record types. Let me see if I can > reproduce/understand/fix this in the doc branch before we do any merging. Ok, let me know whenever you're ready. -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Wed May 4 16:20:44 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 4 May 2011 16:20:44 -0700 Subject: [Bro-Dev] dns.bro In-Reply-To: <20110503193453.GA55991@icir.org> References: <20110502223510.GA19278@icir.org> <20110503193453.GA55991@icir.org> Message-ID: <20110504232044.GA12896@icir.org> On Tue, May 03, 2011 at 12:34 -0700, I wrote: > Not really ... But the problem I see is that it's hard to figure out > what a script is *not* loading and what thus can make sense to add > manually. Mulling over this a bit more, I'm wondering whether we should change tack and generally always load *everything* from the top-level file, even if that may be too much for some environments. These can always go ala carte and load the sub-files individually. By just loading everything, it's much easier to get an idea what's available. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.icir.org Fri May 6 09:31:27 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 16:31:27 -0000 Subject: [Bro-Dev] #337: BroCtl's top has trouble with large values In-Reply-To: <044.194bdf0be7a34bfafe07f70c1ae7871f@tracker.icir.org> References: <044.194bdf0be7a34bfafe07f70c1ae7871f@tracker.icir.org> Message-ID: <059.deb7e0d513e8af93b07b6ae0967ea919@tracker.icir.org> #337: BroCtl's top has trouble with large values -----------------------------+------------------------ Reporter: robin | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: reopened => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri May 6 09:32:10 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 16:32:10 -0000 Subject: [Bro-Dev] #337: BroCtl's top has trouble with large values In-Reply-To: <044.194bdf0be7a34bfafe07f70c1ae7871f@tracker.icir.org> References: <044.194bdf0be7a34bfafe07f70c1ae7871f@tracker.icir.org> Message-ID: <059.22c608959dd89c9091de4a36e12fc078@tracker.icir.org> #337: BroCtl's top has trouble with large values -----------------------------+------------------------ Reporter: robin | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Comment (by robin): Part of topic/robin/plugins branch. Will be merged along with it. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri May 6 10:11:20 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 17:11:20 -0000 Subject: [Bro-Dev] #361: Copy Bro binary only when in NFS mode In-Reply-To: <044.468176c485675ae94b1733335d79df5c@tracker.icir.org> References: <044.468176c485675ae94b1733335d79df5c@tracker.icir.org> Message-ID: <059.4147f2ade36ce7213a4fef365bbeabcd@tracker.icir.org> #361: Copy Bro binary only when in NFS mode -----------------------------+------------------------ Reporter: robin | Owner: robin Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: accepted => closed * resolution: => Solved/Applied Comment: Keeping NFS mode for now, but no longer copying in non-NFS mode. Part of topic/robin/plugins. Will be merged with it. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri May 6 10:35:02 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 17:35:02 -0000 Subject: [Bro-Dev] #370: Plugin interface for BroControl In-Reply-To: <043.95b2acb07df54963a84b633b546fafcb@tracker.icir.org> References: <043.95b2acb07df54963a84b633b546fafcb@tracker.icir.org> Message-ID: <058.a557fe4dd459ac27f702d04ad81f2094@tracker.icir.org> #370: Plugin interface for BroControl ------------------------------+--------------------- Reporter: seth | Owner: robin Type: Feature Request | Status: testing Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: ------------------------------+--------------------- Changes (by robin): * status: accepted => testing Comment: This is now implemented in branch ``topic/robin/plugins``. See README for documentation. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri May 6 10:38:57 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 17:38:57 -0000 Subject: [Bro-Dev] #390: Problem with _logStats method in BroControl In-Reply-To: <043.44051fe7dfe56717621df1195fa8a270@tracker.icir.org> References: <043.44051fe7dfe56717621df1195fa8a270@tracker.icir.org> Message-ID: <058.1818796279fd0501c2d15c47b616ea26@tracker.icir.org> #390: Problem with _logStats method in BroControl -------------------------+------------------------------- Reporter: seth | Owner: robin Type: Problem | Status: needs information Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: | Keywords: -------------------------+------------------------------- Changes (by robin): * status: accepted => needs information Comment: Can't tell what might be causing this. Any further specifics? I assume the cFlow stuff isn't activated in those installation,correct? -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Fri May 6 10:44:13 2011 From: gregor at icir.org (Gregor Maier) Date: Fri, 06 May 2011 10:44:13 -0700 Subject: [Bro-Dev] RPC and NFS Analyzers Message-ID: <4DC4336D.30002@icir.org> Hi, I was wondering what we should do with the RPC and NFS-Analyzers in Bro and the ones I've written. The analyzer that currently ship with Bro are rather incomplete (NFS supports only fstat, lookup, and getattr procedures) and the RPC analyzer doesn't log (only through conn$service and conn$addl) and has problems with re-syncing to streams with gaps. My RPC analyzer extends the stock one by adding a log-file (if desired) and doing the re-sync properly. So I think it makes sense the RPC analyzer into master (before or after 1.6) My NFS analyzer does not fully implement all procedures yet either but it has skeletons for all procedures (*), can track path- and filenames, reads/writes/creates, and extract and deliver file content to the script layer (e.g., for libmagic). I currently don't have the cycles to fully implement the missing procedures in the near future (the NFS analyzer does what I need for my analysis), however, I hope to do that some stage. The analyzer has, however, been tested with a ton of data (150GB+), is stable and works fine. So, even though it's not fully implemented yet its a huge improvement over the current one (which does pretty much nothing). I think it makes sense to merge it into master as well (alternatively we might consider removing the NFS analyzers altogether). (Also the NFS and RPC analyzers are in the same topic branch, so merging just one will require quite a bit of surgery) (*) it will report which not-implemented procedure has been called, its size and its success status. What are your thoughts? Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.icir.org Fri May 6 10:46:10 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 17:46:10 -0000 Subject: [Bro-Dev] #390: Problem with _logStats method in BroControl In-Reply-To: <043.44051fe7dfe56717621df1195fa8a270@tracker.icir.org> References: <043.44051fe7dfe56717621df1195fa8a270@tracker.icir.org> Message-ID: <058.1d3c7d3f90ece4e0c150728461a487c3@tracker.icir.org> #390: Problem with _logStats method in BroControl -------------------------+------------------------------- Reporter: seth | Owner: robin Type: Problem | Status: needs information Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: | Keywords: -------------------------+------------------------------- Comment (by seth): Replying to [comment:2 robin]: > I assume the cFlow stuff isn't activated in those installation,correct? At least in one case it wasn't active. I can't remember who all has had trouble with that (but I know that it's more than one place). -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Fri May 6 10:49:24 2011 From: seth at icir.org (Seth Hall) Date: Fri, 6 May 2011 13:49:24 -0400 Subject: [Bro-Dev] RPC and NFS Analyzers In-Reply-To: <4DC4336D.30002@icir.org> References: <4DC4336D.30002@icir.org> Message-ID: <5AB13CFC-C538-44A2-B577-36153D655014@icir.org> On May 6, 2011, at 1:44 PM, Gregor Maier wrote: > What are your thoughts? Let's integrate them and I'll start writing/adapting scripts in the policy-scripts-new branch! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.icir.org Fri May 6 11:11:15 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 18:11:15 -0000 Subject: [Bro-Dev] #401: broctl crash reports should display size of core dump In-Reply-To: <043.9bfa890ec9956919f0f88348b617384d@tracker.icir.org> References: <043.9bfa890ec9956919f0f88348b617384d@tracker.icir.org> Message-ID: <058.6627a831753bdc27cf704217c76f8941@tracker.icir.org> #401: broctl crash reports should display size of core dump -----------------------------+-------------------- Reporter: seth | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied Comment: This is now part of topic/robin/plugins and will be merged with that. -- Ticket URL: Bro Tracker Bro Issue Tracker From baxterw3232 at gmail.com Fri May 6 11:28:32 2011 From: baxterw3232 at gmail.com (Will) Date: Fri, 6 May 2011 14:28:32 -0400 Subject: [Bro-Dev] dns.bro In-Reply-To: <20110502223510.GA19278@icir.org> References: <20110502223510.GA19278@icir.org> Message-ID: Are these new scripts going to have Seth's 'dns-ext.bro' included/merged or is this re-write changing the base dns.bro altogether? On Mon, May 2, 2011 at 6:35 PM, Robin Sommer wrote: > As discussed, I'll start going through the scripts in the > policy-scripts-new branch, starting with dns.bro. > > I really like the new dns.log, pretty neat! > > Two general, DNS-independent, questions first: > > ? ?- Does a policy/foo.bro script always load all of > ? ?policy/foo/*.bro? Would be nice if that was consistent, and > ? ?perhaps it already is. :-) > > ? ?- We should include new connection$uid into pretty much all > ? relevant logs. > > dns/base.bro: > > ? ?- There are number of commented out "print" statements. Should we > ? ? ?pass this into weird.bro? > > ? ?- The script activates the binpac analyzer. Do we want to remove > ? ? ?"classic" C++ analyzer? > > ? ?- There's a TODO about the EDNS/TSIG. What's the problem? > > ? ?- The reply handlers check for "ans$answer_type == DNS_ANS", but > ? ? ?there are also options dns_skip_all_auth/dns_skip_all_addl in > ? ? ?bro.init? Can we get rid of one of the two ways (I'd say the > ? ? ?latter)? > > ? ?- The reply handlers are all almost identical. How about > ? ? ?refactoring that code into a function called by them all? > > ? ?- The comment in connection_state_remove() seems misleading: this > ? ? ?is the only place that logs anything, right? > > ? ?Watislat_ctveusd or > ?- om o te yps n ontsbr dn' semto Do we > ? ? ?want to delete he? > > dns/passive-replication.bro > > ? ?- Can you remind me what the passive replication is for? I thought > ? ?I knew but not sure that's matching with the script. :-) > > ? ?- Regarding the TODO: should "recent_requests" be a table[string] > ? ?of set[string]? > > Robin > > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL ? ?* Fax ? +1 (510) 666-2956 * ? www.icir.org > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From bro at tracker.icir.org Fri May 6 11:37:17 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 18:37:17 -0000 Subject: [Bro-Dev] #445: Split up cron command In-Reply-To: <044.981eef591b7c90a67b94edd89aad2930@tracker.icir.org> References: <044.981eef591b7c90a67b94edd89aad2930@tracker.icir.org> Message-ID: <059.96fa9af180a0ec8fd8c25556944040af@tracker.icir.org> #445: Split up cron command -----------------------------+-------------------- Reporter: robin | Owner: robin Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied Comment: The `cron` command now has an option `--no-watch` to disable watching/restart Bro processes. This is part of the `topic.robin/plugin branch` and will be merged with it. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Fri May 6 13:14:58 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 6 May 2011 13:14:58 -0700 Subject: [Bro-Dev] dns.bro In-Reply-To: References: <20110502223510.GA19278@icir.org> Message-ID: <20110506201458.GC6228@icir.org> On Fri, May 06, 2011 at 14:28 -0400, you wrote: > Are these new scripts going to have Seth's 'dns-ext.bro' > included/merged or is this re-write changing the base dns.bro > altogether? Both. :) Seth is rewriting/restructuring the base scripts and including the functionality of his *-ext scripts in the process (not only dns, but all). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Fri May 6 13:17:17 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 6 May 2011 13:17:17 -0700 Subject: [Bro-Dev] RPC and NFS Analyzers In-Reply-To: <5AB13CFC-C538-44A2-B577-36153D655014@icir.org> References: <4DC4336D.30002@icir.org> <5AB13CFC-C538-44A2-B577-36153D655014@icir.org> Message-ID: <20110506201717.GE6228@icir.org> On Fri, May 06, 2011 at 13:49 -0400, you wrote: > Let's integrate them and I'll start writing/adapting scripts in the policy-scripts-new branch! Sounds good to me too. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Fri May 6 14:20:07 2011 From: gregor at icir.org (Gregor Maier) Date: Fri, 06 May 2011 14:20:07 -0700 Subject: [Bro-Dev] RPC and NFS Analyzers In-Reply-To: <4DC4336D.30002@icir.org> References: <4DC4336D.30002@icir.org> Message-ID: <4DC46607.5030306@icir.org> just talked some more with Robin about that. The plan(TM) is that I'll prepare the analyzers so they can be merged into master and Seth can then pull from master into policy-scripts-new to polish the policy scripts. cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.icir.org Fri May 6 15:00:10 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 22:00:10 -0000 Subject: [Bro-Dev] #443: Print a message instead of a stack trace when debug.log can't be opened In-Reply-To: <044.4e043473c81c6bae9f3157b943593659@tracker.icir.org> References: <044.4e043473c81c6bae9f3157b943593659@tracker.icir.org> Message-ID: <059.558893a8ddc9a1f0c31f6741ea16d47b@tracker.icir.org> #443: Print a message instead of a stack trace when debug.log can't be opened ------------------------------+-------------------- Reporter: leres | Owner: robin Type: Feature Request | Status: closed Priority: Low | Milestone: Bro1.6 Component: BroControl | Version: 1.5.3 Resolution: Solved/Applied | Keywords: ------------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied Comment: Now part of `topic/robin/plugin`, will be merged with that. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri May 6 15:01:58 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 22:01:58 -0000 Subject: [Bro-Dev] #13: Run custom script with cluster In-Reply-To: <044.7005d6a40032ed7ee895439da6f4ae96@tracker.icir.org> References: <044.7005d6a40032ed7ee895439da6f4ae96@tracker.icir.org> Message-ID: <059.e4ae2c34a8b8b6fee53df4872adec96d@tracker.icir.org> #13: Run custom script with cluster ------------------------------+-------------------- Reporter: robin | Owner: robin Type: Feature Request | Status: closed Priority: Low | Milestone: Component: BroControl | Version: Resolution: Solved/Applied | Keywords: ------------------------------+-------------------- Changes (by robin): * status: accepted => closed * resolution: => Solved/Applied Comment: The new plugin interface provides this, see #370. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri May 6 15:10:32 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 22:10:32 -0000 Subject: [Bro-Dev] #226: Add print-filter to default standalone config In-Reply-To: <044.0af01e65a447bf0e5bd0f13833eb3cb4@tracker.icir.org> References: <044.0af01e65a447bf0e5bd0f13833eb3cb4@tracker.icir.org> Message-ID: <059.f89b76541403f788fb64ac0d4ac32781@tracker.icir.org> #226: Add print-filter to default standalone config ------------------------------+-------------------- Reporter: robin | Owner: robin Type: Feature Request | Status: closed Priority: Normal | Milestone: Component: BroControl | Version: 1.5.2 Resolution: Solved/Applied | Keywords: ------------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied Comment: In plugin branch now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri May 6 15:12:12 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 22:12:12 -0000 Subject: [Bro-Dev] #230: Update capstats In-Reply-To: <044.fc799bf1db46e4354fdad9f884fa59fb@tracker.icir.org> References: <044.fc799bf1db46e4354fdad9f884fa59fb@tracker.icir.org> Message-ID: <059.b158eaa8097daf8726bcd71e9be39e51@tracker.icir.org> #230: Update capstats -----------------------------+-------------------- Reporter: robin | Owner: robin Type: Task | Status: closed Priority: Normal | Milestone: Component: BroControl | Version: 1.5.2 Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied Comment: capstats is now maintained along with Bro and thus up-to-date by definition. :) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri May 6 15:12:48 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 22:12:48 -0000 Subject: [Bro-Dev] #232: installation error while running make In-Reply-To: <057.81303e5db49e40434f12697ecdc8b4da@tracker.icir.org> References: <057.81303e5db49e40434f12697ecdc8b4da@tracker.icir.org> Message-ID: <072.7aa7cb9028a72bd785193ac069247552@tracker.icir.org> #232: installation error while running make -----------------------------+------------------------------------------ Reporter: vestrada@? | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Component: BroControl | Version: 1.5.1 Resolution: Solved/Applied | Keywords: broctl install make Python.h -----------------------------+------------------------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied Comment: Inastallation has been redone completely, closing this one. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri May 6 15:17:19 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 22:17:19 -0000 Subject: [Bro-Dev] #243: Changes to broctl In-Reply-To: <044.307ed58dbf28e3c18c0ca0f890ff32c3@tracker.icir.org> References: <044.307ed58dbf28e3c18c0ca0f890ff32c3@tracker.icir.org> Message-ID: <059.ed0f70e07dad330911ff63e9d05e290c@tracker.icir.org> #243: Changes to broctl -----------------------------+-------------------- Reporter: robin | Owner: robin Type: Patch | Status: closed Priority: Normal | Milestone: Component: BroControl | Version: 1.5.2 Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: assigned => closed * resolution: => Solved/Applied Comment: Already applied. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri May 6 15:20:18 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 22:20:18 -0000 Subject: [Bro-Dev] #268: Patch to fix broctl's default display for the cronenabled setting In-Reply-To: <043.970d3cfe09faa3afc7de1549c2a475e1@tracker.icir.org> References: <043.970d3cfe09faa3afc7de1549c2a475e1@tracker.icir.org> Message-ID: <058.e0bd41b061adfd0e6f3b5908401fb7e0@tracker.icir.org> #268: Patch to fix broctl's default display for the cronenabled setting -----------------------------+-------------------- Reporter: seth | Owner: robin Type: Patch | Status: closed Priority: Normal | Milestone: Component: BroControl | Version: 1.5.2 Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied Comment: In plugin branch. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri May 6 16:52:19 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 06 May 2011 23:52:19 -0000 Subject: [Bro-Dev] #430: Build error in LogMgr.cc In-Reply-To: <047.b7495de40c3e3ce4a9d440149e22b540@tracker.icir.org> References: <047.b7495de40c3e3ce4a9d440149e22b540@tracker.icir.org> Message-ID: <062.3ae4cb6b1800a009ddecda02a22c4068@tracker.icir.org> #430: Build error in LogMgr.cc -----------------------+----------------------- Reporter: appleman | Owner: robin Type: Problem | Status: assigned Priority: Normal | Milestone: Component: Bro | Version: git/topic Resolution: | Keywords: -----------------------+----------------------- Comment (by gregor): I get this error from master. Please add the #include to master. thx gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Fri May 6 17:05:03 2011 From: gregor at icir.org (Gregor Maier) Date: Fri, 06 May 2011 17:05:03 -0700 Subject: [Bro-Dev] Where did the min() function go? Message-ID: <4DC48CAF.9020007@icir.org> Hi, I'm having a weird problem. The min() inline function is gone from master. It's still used by a bunch of files that still compile, so there must be a C++ builtin, macro, or whatever. I just can't find it where is it? I noticed that when trying to merge master into my rpc/nfs branch. I'm using min() to and it seem that the signature I use is not supported by the the current version of min..... BTW, the min() inlines where removed by this commit (in my actual patch I added overloaded min() functions but they didn't make it to master, presumably due to the cleanup) commit 714289bd1354f5ab62b87e7ae86592ad70414746 Author: Robin Sommer Date: Wed Apr 27 21:25:51 2011 -0700 Changing the HTTP's analyzers internals to use 64-bit integers. (Gregor Maier). This is the patch from #326, plus some cleanup. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From gregor at icir.org Fri May 6 17:12:40 2011 From: gregor at icir.org (Gregor Maier) Date: Fri, 06 May 2011 17:12:40 -0700 Subject: [Bro-Dev] Where did the min() function go? In-Reply-To: <4DC48CAF.9020007@icir.org> References: <4DC48CAF.9020007@icir.org> Message-ID: <4DC48E78.7000709@icir.org> well, of course I find out myself as soon as I hit send.... min() is an STL algorithm. But I guess that means that #include should now be added to util.h, right? (that's where the min functions used to live) cu gregor On 5/6/11 17:05 , Gregor Maier wrote: > Hi, > > I'm having a weird problem. The min() inline function is gone from > master. It's still used by a bunch of files that still compile, so there > must be a C++ builtin, macro, or whatever. I just can't find it where is > it? > > I noticed that when trying to merge master into my rpc/nfs branch. I'm > using min() to and it seem that the signature I use is not supported by > the the current version of min..... > > BTW, the min() inlines where removed by this commit (in my actual patch > I added overloaded min() functions but they didn't make it to master, > presumably due to the cleanup) > > commit 714289bd1354f5ab62b87e7ae86592ad70414746 > Author: Robin Sommer > Date: Wed Apr 27 21:25:51 2011 -0700 > > Changing the HTTP's analyzers internals to use 64-bit integers. > (Gregor Maier). > > This is the patch from #326, plus some cleanup. > > > cu > Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.icir.org Fri May 6 17:34:06 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Sat, 07 May 2011 00:34:06 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.30d188ed33ba9389d55042ddda5bd350@tracker.icir.org> #435: topic/jsiwek/doc-framework ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Task => Merge Request Comment: This can be merged again (for final time unless you find something) and I'll follow up with testing/tweaks in `policy-scripts-new` I think you had automatic doc generation on the web server, which will probably be broken by that change in source path from `doc/` to `doc/scripts` which got reflected in the build directory, too. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri May 6 17:39:51 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Sat, 07 May 2011 00:39:51 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.ba452bc79a0f6d51488bdee591503c7f@tracker.icir.org> #435: topic/jsiwek/doc-framework ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): Oh, and I guess since it's not mentioned in the ticket anywhere, I'll just note: for now, we seem to have agreed on manually documenting the new logging features because of the difficulty in programmatically tracking them. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri May 6 18:57:15 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Sat, 07 May 2011 01:57:15 -0000 Subject: [Bro-Dev] #273: "read" command in broctl In-Reply-To: <043.056559d2c20015e30c45a37dfa040284@tracker.icir.org> References: <043.056559d2c20015e30c45a37dfa040284@tracker.icir.org> Message-ID: <058.8bbf27c7f6dc50e06a48b52e74dec192@tracker.icir.org> #273: "read" command in broctl ------------------------------+--------------------- Reporter: seth | Owner: Type: Feature Request | Status: testing Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: 1.5.1 Resolution: | Keywords: ------------------------------+--------------------- Changes (by robin): * status: seen => testing * milestone: => Bro1.6 Comment: An experimental version of this is now part of the `plugin` branch. It's closer to Seth's version but allows to give additional Bro options and scripts so, e.g., execution tracing can be turned on. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Sat May 7 23:28:54 2011 From: seth at icir.org (Seth Hall) Date: Sun, 8 May 2011 02:28:54 -0400 Subject: [Bro-Dev] dns.bro In-Reply-To: <20110506201458.GC6228@icir.org> References: <20110502223510.GA19278@icir.org> <20110506201458.GC6228@icir.org> Message-ID: <8C076BE0-B3C8-4AD4-9F20-E371C1C33E69@icir.org> On May 6, 2011, at 4:14 PM, Robin Sommer wrote: > Both. :) Seth is rewriting/restructuring the base scripts and > including the functionality of his *-ext scripts in the process (not > only dns, but all). And that's not mentioning the new scripts that aren't even in my -ext scripts. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.icir.org Sun May 8 18:24:11 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 09 May 2011 01:24:11 -0000 Subject: [Bro-Dev] #430: Build error in LogMgr.cc In-Reply-To: <047.b7495de40c3e3ce4a9d440149e22b540@tracker.icir.org> References: <047.b7495de40c3e3ce4a9d440149e22b540@tracker.icir.org> Message-ID: <062.54ac45dfc7b88f43d56f70916630e259@tracker.icir.org> #430: Build error in LogMgr.cc -----------------------+----------------------- Reporter: appleman | Owner: robin Type: Problem | Status: assigned Priority: Normal | Milestone: Component: Bro | Version: git/topic Resolution: | Keywords: -----------------------+----------------------- Comment (by jsiwek): A commit now on fastpath will fix this: 35b7caf44fdb045e2e729cfbf6f84fde707985db -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Sun May 8 18:34:05 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Sun, 8 May 2011 20:34:05 -0500 (CDT) Subject: [Bro-Dev] Where did the min() function go? In-Reply-To: <4DC48E78.7000709@icir.org> Message-ID: <1736014250.337.1304904845349.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > min() is an STL algorithm. But I guess that means that > #include should now be added to util.h, right? (that's > where the min functions used to live) It might just be my personal style, but I like it when source files include all headers it requires and no more; not requiring some include cascade to get things to work. And I don't think util.h has any dependence on , so I'd say just add to files that generate compile errors because it's missing as you find them. There was another issue going on with using the templated std::min() with args of differing types that I address in commit 35b7caf44fdb045e2e729cfbf6f84fde707985db - Jon From jones at tacc.utexas.edu Sun May 8 19:11:57 2011 From: jones at tacc.utexas.edu (William Jones) Date: Sun, 8 May 2011 21:11:57 -0500 Subject: [Bro-Dev] Running bro with newer 10 GigE cards that Coalesce packeets in the hardware In-Reply-To: <20110506201717.GE6228@icir.org> References: <4DC4336D.30002@icir.org> <5AB13CFC-C538-44A2-B577-36153D655014@icir.org> <20110506201717.GE6228@icir.org> Message-ID: This takes many packets and presents them to bro as one very large packet 10K. MTU Throughput Mbyutes/sec Packet Rate Bro Packet Rate 1500 470 300K 24K 512 287 560K 21K There were no drops reported and no errors that I would normal see on the nics if I turned off coalescing. This does require increasing the pcap snap length. Can anyone think of any down side to running bro with packet coalescing nic's. -----Original Message----- From: bro-dev-bounces at bro-ids.org [mailto:bro-dev-bounces at bro-ids.org] On Behalf Of Robin Sommer Sent: Friday, May 06, 2011 3:17 PM To: Seth Hall Cc: Bro Dev Subject: Re: [Bro-Dev] RPC and NFS Analyzers On Fri, May 06, 2011 at 13:49 -0400, you wrote: > Let's integrate them and I'll start writing/adapting scripts in the policy-scripts-new branch! Sounds good to me too. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org _______________________________________________ bro-dev mailing list bro-dev at bro-ids.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From seth at icir.org Sun May 8 19:24:52 2011 From: seth at icir.org (Seth Hall) Date: Sun, 8 May 2011 22:24:52 -0400 Subject: [Bro-Dev] metrics framework Message-ID: (This is mostly for Robin and Jon, but I'm CC-ing bro-dev as well to solicit others thoughts about the topic) I'd appreciate if you guys took a look at the metrics framework and let me know what you think about it. There's a lot missing still, but it's a decent start. Running it on a tracefile with those metrics/*-example.bro scripts and taking a look at the output is probably one of the best ways to learn what it's doing. I'll document some features I'm still planning on adding to it here too. Currently, in the metrics framework a metric is just a key or keys that is connected to a number which is collected over some interval before being written to disk and reset. One metric that you could collect is the number of established TCP connections. Going further, you can imagine only wanting to collect the metric for local addresses and collecting the metric separately for every /24 or every known internally allocated subnet. Changing the break time for the metric could be useful too because you may only care about some value per hour while others you want to collect the metric every 15 seconds. A subnet is one aggregation technique and results in one of the possible indexes. A user could configure that they want all metrics aggregated into /24 netmask length instead of calculating the metric per individual IP address. The other index is a freeform string field which can represent anything. It's possible to use either, both, or no indexes to aggregate the data. Here are some thoughts about missing features: - Missing support for cluster deployment. The way this will work (I think) is that the manager will handle the break interval (the time between metrics collection/logging and reset) and it will send a metrics collection event to the workers which will send their metrics data to the manager where the data from each worker will be added together and logged. This would essentially act as a lazy synchronization technique. - Missing statistical support. I want to be able to define when notices should happen based on rate of change of a metric (per the break interval) increasing much faster than you think it should (SSH failed logins). There's probably a lot of other stuff in this area I haven't thought of. - Need a way to refine metrics to add things like netmask aggregation or aggregation table without defining it in the Metrics::create function so that we can ship a metrics collection script that the user can subsequently reconfigure. For example, maybe they want to track successful outbound TCP connections by department. They could supply a table like: const subnets_to_depts = table( [1.2.3.0/24] = "Accounting", [4.3.0.0/16] = "Sales", [4.3.2.0/24] = "Accounting" ); and using that as the address aggregation table would aggregate the metrics per department instead of just per /24 or some other arbitrary netmask length. - I need to write a command line tool to convert the log into something that Graphviz can understand because I'd like to be able to generate time-series graphs from these metrics really easily. Here's the list of metrics I'm working towards currently... 1. Youtube video views. 2. HTTPS requests per host header (using a new SSL analyzer that provides the information from the SSL establishment), this is an example of a non-IP address based metric too. 3. TCP connections originated and responded to. 4. HTTP SQL injection requests (raise a notice when too many attacks) 5. HTTP requests subindexed by returned status code from server. How many 404,200,500 status codes seen per client IP address? 6. SSH failed connections (too many looks like scanning obviously). 7. DNS requests (watch for spiking could possibly find DNS tunnels or DNS amplification attacks) Sorry for the rambling email, but my thoughts on this script are still mixing around a bit. :) Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Sun May 8 19:31:48 2011 From: seth at icir.org (Seth Hall) Date: Sun, 8 May 2011 22:31:48 -0400 Subject: [Bro-Dev] Running bro with newer 10 GigE cards that Coalesce packeets in the hardware In-Reply-To: References: <4DC4336D.30002@icir.org> <5AB13CFC-C538-44A2-B577-36153D655014@icir.org> <20110506201717.GE6228@icir.org> Message-ID: <261DABEE-0637-435E-95E8-9B93EB0E2548@icir.org> On May 8, 2011, at 10:11 PM, William Jones wrote: > This takes many packets and presents them to bro as one very large packet 10K. > Can anyone think of any down side to running bro with packet coalescing nic's. I think I'm lost. At first I thought you were talking about interrupt coalescing (which shouldn't cause any changes), but now I'm not so sure. Can you point me to documentation for your packet coalescing NIC and the setting you're changing in your operating system to enable it? Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Sun May 8 21:06:47 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 8 May 2011 21:06:47 -0700 Subject: [Bro-Dev] Where did the min() function go? In-Reply-To: <1736014250.337.1304904845349.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <4DC48E78.7000709@icir.org> <1736014250.337.1304904845349.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20110509040647.GB13587@icir.org> On Sun, May 08, 2011 at 20:34 -0500, you wrote: > any dependence on , so I'd say just add to > files that generate compile errors because it's missing as you find > them. Ack. > with args of differing types that I address in commit > 35b7caf44fdb045e2e729cfbf6f84fde707985db Merged, thanks. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gc355804 at ohio.edu Mon May 9 02:41:32 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Mon, 09 May 2011 05:41:32 -0400 Subject: [Bro-Dev] metrics framework In-Reply-To: References: Message-ID: <4DC7B6CC.5090100@ohio.edu> Maybe this is a long-term / out-of-scope feature, but what about adding an ability to bind OIDs to certain metrics and make them available via SNMP / NETCONF / etc.? This could integrate nicely with existing network monitoring / management infrastructure wherever bro might be deployed, and could also let existing tools handle the collection / graphing / visualization of metrics. --Gilbert Clark On 5/8/2011 10:24 PM, Seth Hall wrote: > (This is mostly for Robin and Jon, but I'm CC-ing bro-dev as well to solicit others thoughts about the topic) > > I'd appreciate if you guys took a look at the metrics framework and let me know what you think about it. There's a lot missing still, but it's a decent start. Running it on a tracefile with those metrics/*-example.bro scripts and taking a look at the output is probably one of the best ways to learn what it's doing. I'll document some features I'm still planning on adding to it here too. > > Currently, in the metrics framework a metric is just a key or keys that is connected to a number which is collected over some interval before being written to disk and reset. One metric that you could collect is the number of established TCP connections. Going further, you can imagine only wanting to collect the metric for local addresses and collecting the metric separately for every /24 or every known internally allocated subnet. Changing the break time for the metric could be useful too because you may only care about some value per hour while others you want to collect the metric every 15 seconds. A subnet is one aggregation technique and results in one of the possible indexes. A user could configure that they want all metrics aggregated into /24 netmask length instead of calculating the metric per individual IP address. The other index is a freeform string field which can represent anything. It's possible to use either, both, or no indexes to aggregate the data. > > Here are some thoughts about missing features: > > - Missing support for cluster deployment. The way this will work (I think) is that the manager will handle the break interval (the time between metrics collection/logging and reset) and it will send a metrics collection event to the workers which will send their metrics data to the manager where the data from each worker will be added together and logged. This would essentially act as a lazy synchronization technique. > - Missing statistical support. I want to be able to define when notices should happen based on rate of change of a metric (per the break interval) increasing much faster than you think it should (SSH failed logins). There's probably a lot of other stuff in this area I haven't thought of. > > - Need a way to refine metrics to add things like netmask aggregation or aggregation table without defining it in the Metrics::create function so that we can ship a metrics collection script that the user can subsequently reconfigure. For example, maybe they want to track successful outbound TCP connections by department. They could supply a table like: > const subnets_to_depts = table( [1.2.3.0/24] = "Accounting", [4.3.0.0/16] = "Sales", [4.3.2.0/24] = "Accounting" ); > and using that as the address aggregation table would aggregate the metrics per department instead of just per /24 or some other arbitrary netmask length. > > - I need to write a command line tool to convert the log into something that Graphviz can understand because I'd like to be able to generate time-series graphs from these metrics really easily. > > > Here's the list of metrics I'm working towards currently... > 1. Youtube video views. > 2. HTTPS requests per host header (using a new SSL analyzer that provides the information from the SSL establishment), this is an example of a non-IP address based metric too. > 3. TCP connections originated and responded to. > 4. HTTP SQL injection requests (raise a notice when too many attacks) > 5. HTTP requests subindexed by returned status code from server. How many 404,200,500 status codes seen per client IP address? > 6. SSH failed connections (too many looks like scanning obviously). > 7. DNS requests (watch for spiking could possibly find DNS tunnels or DNS amplification attacks) > > Sorry for the rambling email, but my thoughts on this script are still mixing around a bit. :) > > Thanks, > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From jsiwek at ncsa.illinois.edu Mon May 9 07:44:07 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 9 May 2011 09:44:07 -0500 (CDT) Subject: [Bro-Dev] metrics framework In-Reply-To: Message-ID: <1392176851.1500.1304952247573.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > - I need to write a command line tool to convert the log into > something that Graphviz can understand because I'd like to be able to > generate time-series graphs from these metrics really easily. Any particular reason for Graphviz? If not, I've frequently used matplotlib, a Python-based graphing/plotting library, to whip up stuff like this. Though, maybe we could get away with an even lighter weight library than either to do this. - Jon From bro at tracker.icir.org Mon May 9 08:38:04 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 09 May 2011 15:38:04 -0000 Subject: [Bro-Dev] #430: Build error in LogMgr.cc In-Reply-To: <047.b7495de40c3e3ce4a9d440149e22b540@tracker.icir.org> References: <047.b7495de40c3e3ce4a9d440149e22b540@tracker.icir.org> Message-ID: <062.248319883d325d99db35c33637583717@tracker.icir.org> #430: Build error in LogMgr.cc -----------------------------+----------------------- Reporter: appleman | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: git/topic Resolution: Solved/Applied | Keywords: -----------------------------+----------------------- Changes (by jsiwek): * status: assigned => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Mon May 9 09:14:06 2011 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 09 May 2011 09:14:06 -0700 Subject: [Bro-Dev] metrics framework In-Reply-To: References: Message-ID: <1304956189-sup-2546@samurai.local> > I'd appreciate if you guys took a look at the metrics framework and > let me know what you think about it. I'd love to do so, yet my cycles only allow for brief inline feedback. > Currently, in the metrics framework a metric is just a key or keys > that is connected to a number which is collected over some interval > before being written to disk and reset. If it is really just a sequence of number, why not calling it time series? The word metric (in networking) implies some sort of property of a path, and more generally, some sort of performance measure. This would also make more sense in the statistical context, where a time series analysis is well-defined field of its own. I prefer this term not only because I have taken a statistics course, but mainly because it is more neutral, maybe even more general, since it only describes the format of the data. > The way this will work (I think) is that the manager will handle the > break interval (the time between metrics collection/logging and reset) > and it will send a metrics collection event to the workers which will > send their metrics data to the manager where the data from each worker > will be added together and logged. This would essentially act as a > lazy synchronization technique. To preserve the temporal ordering, timestamps need to be part of the synchronization game. It looks like a mergeable table indexed by timestamp will do the trick. > - Missing statistical support. I want to be able to define when > notices should happen based on rate of change of a metric (per the > break interval) increasing much faster than you think it should (SSH > failed logins). There's probably a lot of other stuff in this area I > haven't thought of. There's a whole subfield of statistics waiting for you. The natural question is much of this should be in Bro versus offline log munging. Clearly you're talking Bro. It seems would you would like to have is a variance analysis on a detrended series (i.e., on the first-order differences between two data points). Other analyses would be to check for seasonal components. > - I need to write a command line tool to convert the log into > something that Graphviz can understand because I'd like to be able to > generate time-series graphs from these metrics really easily. Why not use R? It is brilliant time series support! (And there exist also scripting language bindings if you really want a separate tool. I tested the Ruby bindings once and they work well.) > 2. HTTPS requests per host header (using a new SSL analyzer that > provides the information from the SSL establishment), this is an > example of a non-IP address based metric too. Along those lines, one could (mis)use this new framework to count the number of unique certificates per host as crude way to identify TLS MITM attacks. Matthias From jones at tacc.utexas.edu Mon May 9 12:09:28 2011 From: jones at tacc.utexas.edu (William Jones) Date: Mon, 9 May 2011 14:09:28 -0500 Subject: [Bro-Dev] Running bro with newer 10 GigE cards that Coalesce packeets in the hardware In-Reply-To: <261DABEE-0637-435E-95E8-9B93EB0E2548@icir.org> References: <4DC4336D.30002@icir.org> <5AB13CFC-C538-44A2-B577-36153D655014@icir.org> <20110506201717.GE6228@icir.org> <261DABEE-0637-435E-95E8-9B93EB0E2548@icir.org> Message-ID: Gather scatter is the official term. It is on by default on the bmc 10 GigE cards. -----Original Message----- From: Seth Hall [mailto:seth at icir.org] Sent: Sunday, May 08, 2011 9:32 PM To: William Jones Cc: Bro Dev Subject: Re: [Bro-Dev] Running bro with newer 10 GigE cards that Coalesce packeets in the hardware On May 8, 2011, at 10:11 PM, William Jones wrote: > This takes many packets and presents them to bro as one very large packet 10K. > Can anyone think of any down side to running bro with packet coalescing nic's. I think I'm lost. At first I thought you were talking about interrupt coalescing (which shouldn't cause any changes), but now I'm not so sure. Can you point me to documentation for your packet coalescing NIC and the setting you're changing in your operating system to enable it? Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.icir.org Mon May 9 16:17:46 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 09 May 2011 23:17:46 -0000 Subject: [Bro-Dev] #453: Connection compressor doesn't handle IPv6 Message-ID: <044.0b812a286fbdbe138e8c086e80350301@tracker.icir.org> #453: Connection compressor doesn't handle IPv6 ---------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Keywords: ipv6 | ---------------------+------------------------ -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Mon May 9 17:50:02 2011 From: seth at icir.org (Seth Hall) Date: Mon, 9 May 2011 20:50:02 -0400 Subject: [Bro-Dev] metrics framework In-Reply-To: <1304956189-sup-2546@samurai.local> References: <1304956189-sup-2546@samurai.local> Message-ID: <62DE6891-3E78-449C-918D-102895DE599F@icir.org> On May 9, 2011, at 12:14 PM, Matthias Vallentin wrote: > I'd love to do so, yet my cycles only allow for brief inline feedback. All thoughts are welcomed. :) >> Currently, in the metrics framework a metric is just a key or keys >> that is connected to a number which is collected over some interval >> before being written to disk and reset. > > If it is really just a sequence of number, why not calling it time > series? The word metric (in networking) implies some sort of property of > a path, and more generally, some sort of performance measure. This would > also make more sense in the statistical context, where a time series > analysis is well-defined field of its own. I prefer this term not only > because I have taken a statistics course, but mainly because it is more > neutral, maybe even more general, since it only describes the format of > the data. I think I agree with this. I'll probably change the name at some point. > To preserve the temporal ordering, timestamps need to be part of the > synchronization game. It looks like a mergeable table indexed by > timestamp will do the trick. Hm... there isn't a whole lot of attention given to temporal ordering. The manager would just be asking for a particular measurement (conns originated for example) along with the index or indexes (possibly each local /24 where a connection was originated?) and their counts. Once the workers sent their values off to the manager, they would reset back to zero and start counting up until the manager asks for the numbers again. There is no actual attribute based synchronization (&synchronize) going on because workers don't care about the value on other workers and doing full synchronization would cause too much over head with variable synchronization. > There's a whole subfield of statistics waiting for you. The natural > question is much of this should be in Bro versus offline log munging. > Clearly you're talking Bro. It seems would you would like to have is a > variance analysis on a detrended series (i.e., on the first-order > differences between two data points). Other analyses would be to check > for seasonal components. You lost me at "variance analyzer on a detrended series". :) Anyway, what I'm searching for is just enough statistics (even if it's fake-ish pseudo-statistics) to be able to raise notices when statistically significant changes happen in the time series data. I just don't even know where to start with it. > Why not use R? It is brilliant time series support! (And there exist > also scripting language bindings if you really want a separate tool. I > tested the Ruby bindings once and they work well.) Someone mentioned to me recently that graphviz will even do ascii art based graphs in your terminal and it's already installed everywhere. This would just be a companion script so people would be free to write whatever works best for themselves. >> 2. HTTPS requests per host header (using a new SSL analyzer that >> provides the information from the SSL establishment), this is an >> example of a non-IP address based metric too. > > Along those lines, one could (mis)use this new framework to count the > number of unique certificates per host as crude way to identify TLS > MITM attacks. Oof. Yes. :) I'm trying to implement this in a way that people will be able to do things like that though, so I suppose it's all good. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Mon May 9 20:13:40 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 9 May 2011 22:13:40 -0500 (CDT) Subject: [Bro-Dev] metrics framework In-Reply-To: <62DE6891-3E78-449C-918D-102895DE599F@icir.org> Message-ID: <1718941059.5887.1304997220037.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > Someone mentioned to me recently that graphviz will even do ascii art > based graphs in your terminal and it's already installed everywhere. That would be cool if it did, but I'm not so sure.. >From what I remember graphviz's strength was more for (un)directed graphs. > This would just be a companion script so people would be free to write whatever works best for themselves. Ok, I think I understand better -- you just need some way to massage the logs into a form that's easily sucked up by some plotting software (user's choice). FWIW, gnuplot can just read a file of one "x y" datapoint per line. And I know it does do ASCII plots just as easily as X11 ones. R should also easily read files formatted like that. - Jon From bro at tracker.icir.org Tue May 10 07:19:05 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 10 May 2011 14:19:05 -0000 Subject: [Bro-Dev] #435: topic/jsiwek/doc-framework In-Reply-To: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> References: <045.b65563eef355461ecf3f82141e9643dd@tracker.icir.org> Message-ID: <060.261cedf8100d4921b3aa47c063583584@tracker.icir.org> #435: topic/jsiwek/doc-framework -----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: closed Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: assigned => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Tue May 10 10:02:46 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 10 May 2011 10:02:46 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master diff: Merge of Gregor's conn-size branch. (bd9855a) In-Reply-To: <201105100017.p4A0HMSO015706@bro-ids.icir.org> References: <201105100017.p4A0HMSO015706@bro-ids.icir.org> Message-ID: <4DC96FB6.5050001@icir.org> Robin, it appears that you've now completely removed the history logging from conn.bro, i.e.. state history is tracked but it can't be loggeed anymore. Is this on purpose? (To revert to the previous version, you need to add the following code back into conn.bro: if ( record_state_history ) log_msg = fmt("%s %s", log_msg, c$history == "" ? "X" : c$history); (and define record_state_history as const in conn.bro) cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From seth at icir.org Tue May 10 10:07:15 2011 From: seth at icir.org (Seth Hall) Date: Tue, 10 May 2011 13:07:15 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master diff: Merge of Gregor's conn-size branch. (bd9855a) In-Reply-To: <4DC96FB6.5050001@icir.org> References: <201105100017.p4A0HMSO015706@bro-ids.icir.org> <4DC96FB6.5050001@icir.org> Message-ID: <2464B542-F95F-4E56-B920-DC23C2B1D184@icir.org> On May 10, 2011, at 1:02 PM, Gregor Maier wrote: > it appears that you've now completely removed the history logging from > conn.bro, i.e.. state history is tracked but it can't be loggeed > anymore. Is this on purpose? This probably doesn't matter too much anyway since the conn.bro script was rewritten a while ago and logs the state history. The code that is there probably won't be in the next release. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Tue May 10 10:19:25 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 10 May 2011 10:19:25 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master diff: Merge of Gregor's conn-size branch. (bd9855a) In-Reply-To: <4DC96FB6.5050001@icir.org> References: <201105100017.p4A0HMSO015706@bro-ids.icir.org> <4DC96FB6.5050001@icir.org> Message-ID: <20110510171925.GK10525@icir.org> On Tue, May 10, 2011 at 10:02 -0700, you wrote: > it appears that you've now completely removed the history logging from > conn.bro, i.e.. state history is tracked but it can't be loggeed > anymore. Is this on purpose? Yeah, looks like I stripped out a bit too much. But I was indeed not really concerned about getting conn.bro right given that it's going to be replaced soon with Seth's version anyway. Still, I'll put this back in for the record before we'll merge in the new scripts. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Tue May 10 12:24:38 2011 From: seth at icir.org (Seth Hall) Date: Tue, 10 May 2011 15:24:38 -0400 Subject: [Bro-Dev] #453: test ticket In-Reply-To: <047.79662df6086b1299e450f36059cfa69d@tracker.bro-ids.org> References: <047.79662df6086b1299e450f36059cfa69d@tracker.bro-ids.org> Message-ID: <2FFD9F2B-84F0-4074-90A4-73BD1E423E47@icir.org> Whoops. Ignore this ticket. I'm setting up the tracker at the new domain but it's not quite ready to use yet. .Seth On May 10, 2011, at 3:23 PM, Bro Tracker wrote: > #453: test ticket > -------------------+--------------------- > Reporter: seth2 | Type: Problem > Status: new | Priority: Normal > Milestone: | Component: Bro > Version: | Keywords: > -------------------+--------------------- > > > -- > Ticket URL: > Bro Tracker > Bro Issue Tracker -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.icir.org Tue May 10 12:59:52 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 10 May 2011 19:59:52 -0000 Subject: [Bro-Dev] #340: Cleanup: unify where global consts are defined (access from policy layer and event engine) In-Reply-To: <045.e26a2c4cd7e7f98be9266f24419513fb@tracker.icir.org> References: <045.e26a2c4cd7e7f98be9266f24419513fb@tracker.icir.org> Message-ID: <060.9a2ba09bbec7b7028644d306d79aefb1@tracker.icir.org> #340: Cleanup: unify where global consts are defined (access from policy layer and event engine) ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by gregor): JFYI, all the plumbing for that is now in master (with the bif changes merged a while ago). So the task for this ticket is to go through NetVar.* and replace all the const, record, and table type "declarations" there with ones in types.bif and const.bif (cf http://www-new.bro-ids.org/devel/bif-doc/ ) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Tue May 10 13:52:10 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 10 May 2011 20:52:10 -0000 Subject: [Bro-Dev] #454: Handwritten HTTP analyzer fails in ambiguous case Message-ID: <043.ad64cc01c1dd584e508296ceea1cd02a@tracker.icir.org> #454: Handwritten HTTP analyzer fails in ambiguous case -------------------------------+----------------- Reporter: seth | Owner: Type: Test Case Missing | Status: new Priority: Low | Milestone: Component: Bro | Version: Keywords: strange | -------------------------------+----------------- The handwritten HTTP analyzer has trouble with the attached tracefile at the end of the response body. The server is returning "\n\r\n" instead of the expected "\r\n\r\n" and the first newline is passed into the http_entity_data event. According to the Content-Length header, this initial \n must be part of the end of data indicator and not a newline in the response body. I'm not filing this as something to be fixed (because browsers seem to have trouble with it too) but rather documenting it as a strange edge case that we may want to cope with in the future since there seem to be web servers actively behaving this way. To get a glimpse at the problem you can run the following command:: {{{ curl http://webcs.msg.yahoo.com/crossdomain.xml | hexdump }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Tue May 10 13:57:19 2011 From: seth at icir.org (Seth Hall) Date: Tue, 10 May 2011 16:57:19 -0400 Subject: [Bro-Dev] tracker move Message-ID: <8BEBA0FA-8416-4C0F-891B-5A6FE26EFFF3@icir.org> Hi all, I'm hoping to move the tracker over to it's new location tonight, but I'll make sure and take down the current tracker during the move so that no one is sitting and trying to edit tickets as I'm doing the migration. I'll send email when I start and when I finish. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Tue May 10 14:29:58 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 10 May 2011 14:29:58 -0700 Subject: [Bro-Dev] metrics framework In-Reply-To: References: <1304956189-sup-2546@samurai.local> <62DE6891-3E78-449C-918D-102895DE599F@icir.org> <1304956189-sup-2546@samurai.local> <1392176851.1500.1304952247573.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <4DC7B6CC.5090100@ohio.edu> Message-ID: <20110510212958.GI27664@icir.org> On Sun, May 08, 2011 at 22:24 -0400, you wrote: > I'd appreciate if you guys took a look at the metrics framework and > let me know what you think about it. Pretty neat. Thoughts: - I'd split configuration of the metrics framework from adding data. Currently the data producer also configures things via create(), but it seems that's something better left to the user of the metrics framework. Doing so would also answer your point on setting up aggregation without using the create() function. Can you just skip the create() function altogether? From the producer's perspecive, that function isn't really doing anything, right? You would then instead provide a configure() function that a user of the metrics framework calls to define aggregration/break_interval/etc, either globally or optionally on a per ID basis. In the absense of any call to configure(), just pick some default, like aggregation per /24 and 10s intervals, or whatever. - I'd move the $increment field out of DataPlug and make it a separate argument to add_data(). It has different semantics than the other fields, and you could then rename DataPlug to just Index. - When no subnet aggregation is set but $host is passed in, I think it won't work correctly. Your example for HTTP_REQUESTS_BY_HOST uses $index for per-host aggregration but looks like cheating. :-) - I'm wondering whether executing log_it() get expensive when it needs to iterate through too many entries. An alternative would be to schedule a number of more fine-granular timers (one per ID, or even one per aggregation unit); but then the log intervals would become desynchronized, which may not be desirable. > - Missing support for cluster deployment. Yeah, that's a tough one. Full &synchronize would be overkill, but sending the data via events, like you suggest, also sounds quite expensive if there are lots of entities for which something's counted. Here's an alternative idea: don't do any communication at all, and just let the workers log their metrics data separately (into the same log file but including a node id column). Then provide a script that postprocesses metrics.log by adding up all the worker's counts for the same unit/time interval. This might cause slight time desynchronizations, but not sure how much impact that would have if we set sufficiently large break intervals. Perhaps the manager could trigger logging by sending the log_it() events, and only then would all the worker go ahead and do their output. If the log_it() event comes with a unique interval ID, the worker can write that out as well and then offline aggregation will be really easy later (and if they in addition also log their local timestamps, one can see how well the timing matches). > - Missing statistical support. I'd leave that out for the first version. Or just do very a simple piece: static thresholds relative to the break intervals (i.e., provide a function add_threshold(id, value) that alarms if a counter for ID id exceeds value. > - I need to write a command line tool to convert the log into > something that Graphviz can understand because I'd like to be able to > enerate time-series graphs from these metrics really easily. As everybody is mentioning his favorite tools, let me throw in mine. :-) I also like matplotlib and R, in that order. But anything is fine with me. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Tue May 10 14:30:38 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 10 May 2011 14:30:38 -0700 Subject: [Bro-Dev] metrics framework In-Reply-To: <4DC7B6CC.5090100@ohio.edu> References: <4DC7B6CC.5090100@ohio.edu> Message-ID: <20110510213038.GJ27664@icir.org> On Mon, May 09, 2011 at 05:41 -0400, you wrote: > Maybe this is a long-term / out-of-scope feature, but what about adding > an ability to bind OIDs to certain metrics and make them available via > SNMP / NETCONF / etc.? Nice idea, I could see that as an external tool that takes the metrics.log as its input (or, at some point in the future, receives a real-time feed of the data going into metrics.log). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Tue May 10 14:35:38 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 10 May 2011 14:35:38 -0700 Subject: [Bro-Dev] metrics framework In-Reply-To: <20110510212958.GI27664@icir.org> References: <1304956189-sup-2546@samurai.local> <62DE6891-3E78-449C-918D-102895DE599F@icir.org> <1304956189-sup-2546@samurai.local> <1392176851.1500.1304952247573.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <4DC7B6CC.5090100@ohio.edu> <20110510212958.GI27664@icir.org> Message-ID: <20110510213538.GA31691@icir.org> On Tue, May 10, 2011 at 14:29 -0700, I wrote: > Thoughts: One more regarding the TODO: "create a metrics config logging stream": Indeed, having a record of how things were configured will be really helpful for building long-term archives. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.icir.org Tue May 10 17:12:23 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 11 May 2011 00:12:23 -0000 Subject: [Bro-Dev] #444: bug fix for POP3 analyzer In-Reply-To: <043.07292c738bf0c10d763add2815eb974d@tracker.icir.org> References: <043.07292c738bf0c10d763add2815eb974d@tracker.icir.org> Message-ID: <058.ecb370ad6090fb0af5fff66900e6effe@tracker.icir.org> #444: bug fix for POP3 analyzer -----------------------------+-------------------- Reporter: vern | Owner: robin Type: Patch | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: accepted => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Tue May 10 17:28:28 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 11 May 2011 00:28:28 -0000 Subject: [Bro-Dev] #449: Move scripts tests from old test-suite over to btest In-Reply-To: <044.80f9fcd5de0ed7dbf1d4e580bd082da3@tracker.icir.org> References: <044.80f9fcd5de0ed7dbf1d4e580bd082da3@tracker.icir.org> Message-ID: <059.c2b4b7930105d3f8d7c681dc55b3d4a5@tracker.icir.org> #449: Move scripts tests from old test-suite over to btest -----------------------------+-------------------- Reporter: robin | Owner: robin Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Tue May 10 18:15:08 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 11 May 2011 01:15:08 -0000 Subject: [Bro-Dev] #412: Port the istate tests to btes In-Reply-To: <044.325f74c6d09693a51bde1e009db18a42@tracker.icir.org> References: <044.325f74c6d09693a51bde1e009db18a42@tracker.icir.org> Message-ID: <059.72d0e1e9fe74b766dc96970e7a3b21b0@tracker.icir.org> #412: Port the istate tests to btes -----------------------------+-------------------- Reporter: robin | Owner: robin Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied Comment: Tests are ported. The SSL one doesn't pass yet, and there are failures on Mac, but that are other problems so I'm closing this one. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Tue May 10 18:19:55 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 11 May 2011 01:19:55 -0000 Subject: [Bro-Dev] #421: Initialize set/table field in records. In-Reply-To: <044.43058f6fba84734b712b1fcb46d48a68@tracker.icir.org> References: <044.43058f6fba84734b712b1fcb46d48a68@tracker.icir.org> Message-ID: <059.1d995e6dda05864c727a9fa78a9b690e@tracker.icir.org> #421: Initialize set/table field in records. -----------------------------+-------------------- Reporter: robin | Owner: robin Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Tue May 10 18:28:50 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 11 May 2011 01:28:50 -0000 Subject: [Bro-Dev] #455: Infrastructure for event engine message Message-ID: <044.f0934b12c60dbb8da9b159c19bafd0c4@tracker.icir.org> #455: Infrastructure for event engine message ---------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Keywords: cleanup | ---------------------+-------------------- We need to streamline how the event-engine reports run-time errors, warnings, and informational messages. These should go into a separate log file, but probably through a script and with additional filtering, like to suppress duplicates. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Tue May 10 18:30:55 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 11 May 2011 01:30:55 -0000 Subject: [Bro-Dev] #357: Only log missing support for libgeoip a single time. In-Reply-To: <043.5e3eefc4db47729fe7e887317f7619e6@tracker.icir.org> References: <043.5e3eefc4db47729fe7e887317f7619e6@tracker.icir.org> Message-ID: <058.77d688618b10706f1d9a98378ebb234b@tracker.icir.org> #357: Only log missing support for libgeoip a single time. -----------------------------+------------------------ Reporter: seth | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: sprint -----------------------------+------------------------ Changes (by robin): * status: accepted => closed * resolution: => Solved/Applied Comment: I've solved the GeoIP problem with a hack for now, and opened #455 for doing a more general solution later. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Tue May 10 18:48:26 2011 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 11 May 2011 01:48:26 -0000 Subject: [Bro-Dev] #340: Cleanup: unify where global consts are defined (access from policy layer and event engine) In-Reply-To: <045.e26a2c4cd7e7f98be9266f24419513fb@tracker.icir.org> References: <045.e26a2c4cd7e7f98be9266f24419513fb@tracker.icir.org> Message-ID: <060.30af23cdab209bc1509548e511fbf3a8@tracker.icir.org> #340: Cleanup: unify where global consts are defined (access from policy layer and event engine) ------------------------------+--------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: cleanup ------------------------------+--------------------- Changes (by robin): * keywords: => cleanup -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed May 11 07:07:40 2011 From: seth at icir.org (Seth Hall) Date: Wed, 11 May 2011 10:07:40 -0400 Subject: [Bro-Dev] updating tracker Message-ID: <7011E49E-2DAD-40B3-A205-FD6A5420D754@icir.org> I'm updating the tracker now, I'm hoping to be done in less than an hour. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Wed May 11 07:49:06 2011 From: seth at icir.org (Seth Hall) Date: Wed, 11 May 2011 10:49:06 -0400 Subject: [Bro-Dev] updating tracker In-Reply-To: <7011E49E-2DAD-40B3-A205-FD6A5420D754@icir.org> References: <7011E49E-2DAD-40B3-A205-FD6A5420D754@icir.org> Message-ID: <499A1BEF-8C59-4D4D-8735-F35BEC99B1C1@icir.org> On May 11, 2011, at 10:07 AM, Seth Hall wrote: > I'm updating the tracker now, I'm hoping to be done in less than an hour. The tracker update is done. For some reason that I can't figure out (and don't care too much about), existing passwords aren't working. When you go to login, you will have to do the password recovery thing on the login page. One change that has been made is that people can now register for their own accounts. The documentation regarding this has been updated on the main page of the tracker. If anyone notices anything broken please let me know! Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Wed May 11 07:51:25 2011 From: seth at icir.org (Seth Hall) Date: Wed, 11 May 2011 10:51:25 -0400 Subject: [Bro-Dev] updating tracker In-Reply-To: <499A1BEF-8C59-4D4D-8735-F35BEC99B1C1@icir.org> References: <7011E49E-2DAD-40B3-A205-FD6A5420D754@icir.org> <499A1BEF-8C59-4D4D-8735-F35BEC99B1C1@icir.org> Message-ID: On May 11, 2011, at 10:49 AM, Seth Hall wrote: > If anyone notices anything broken please let me know! Oops, I forgot to mention that the tracker can now be found at: http://tracker.bro-ids.org/ The links to old URLs for the tracker will redirect transparently so I don't expect people will notice anything in most cases. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From leres at ee.lbl.gov Wed May 11 09:54:34 2011 From: leres at ee.lbl.gov (Craig Leres) Date: Wed, 11 May 2011 09:54:34 -0700 Subject: [Bro-Dev] updating tracker In-Reply-To: <499A1BEF-8C59-4D4D-8735-F35BEC99B1C1@icir.org> References: <7011E49E-2DAD-40B3-A205-FD6A5420D754@icir.org> <499A1BEF-8C59-4D4D-8735-F35BEC99B1C1@icir.org> Message-ID: <4DCABF4A.7030903@ee.lbl.gov> When you login the first time with the temp pw, you must change it before verifying your email address or else the temp pw turns into a pumpkin and you will have to reset it again (ask me how I know). A bug I see is that when I click on "Logout" at the top of the page I end up here: http://tracker.bro-ids.org/bro/prefs/account with the error "Unknown preference panel" Craig From seth at icir.org Wed May 11 09:55:44 2011 From: seth at icir.org (Seth Hall) Date: Wed, 11 May 2011 12:55:44 -0400 Subject: [Bro-Dev] empty string and patterns Message-ID: <3EDFE04E-0F9A-42C9-961E-AB76103FA5E2@icir.org> Does it make sense that this expression would return F? if ( /.*/ in "" ) but then this one returns T: if ( /^.*$/ in "" ) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Wed May 11 10:12:52 2011 From: seth at icir.org (Seth Hall) Date: Wed, 11 May 2011 13:12:52 -0400 Subject: [Bro-Dev] updating tracker In-Reply-To: <4DCABF4A.7030903@ee.lbl.gov> References: <7011E49E-2DAD-40B3-A205-FD6A5420D754@icir.org> <499A1BEF-8C59-4D4D-8735-F35BEC99B1C1@icir.org> <4DCABF4A.7030903@ee.lbl.gov> Message-ID: <75226CE5-19CA-497C-B519-52279452FDD2@icir.org> On May 11, 2011, at 12:54 PM, Craig Leres wrote: > When you login the first time with the temp pw, you must change it > before verifying your email address or else the temp pw turns into a > pumpkin and you will have to reset it again (ask me how I know). Ah, your email address wasn't previously verified because we used to create everyone's accounts ourselves. You won't see that email verification thing again. I'm not sure what's going on there though. I noticed it was a little funny when I went through it, but I didn't change my password until I verified my email address so I didn't run into the same problem as you. I'll play with it some more when I get a chance. > A bug I see is that when I click on "Logout" at the top of the page I > end up here: > > http://tracker.bro-ids.org/bro/prefs/account It seems to only do that if you are in your preference panels when you click logout. Trac guys seem to be have been having some trouble in this area lately. Maybe it will be fixed in a future update. I'm not going to worry about it for now since it's really just a cosmetic issue (and one most people won't see). Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Wed May 11 15:20:09 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 11 May 2011 15:20:09 -0700 Subject: [Bro-Dev] empty string and patterns In-Reply-To: <3EDFE04E-0F9A-42C9-961E-AB76103FA5E2@icir.org> References: <3EDFE04E-0F9A-42C9-961E-AB76103FA5E2@icir.org> Message-ID: <20110511222009.GI10383@icir.org> On Wed, May 11, 2011 at 12:55 -0400, you wrote: > if ( /.*/ in "" ) > if ( /^.*$/ in "" ) Seems both should return True ... Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From vern at icir.org Thu May 12 00:01:07 2011 From: vern at icir.org (Vern Paxson) Date: Thu, 12 May 2011 00:01:07 -0700 Subject: [Bro-Dev] empty string and patterns In-Reply-To: <20110511222009.GI10383@icir.org> (Wed, 11 May 2011 15:20:09 PDT). Message-ID: <20110512070107.93289322C6F@taffy.ICSI.Berkeley.EDU> > > if ( /.*/ in "" ) > > if ( /^.*$/ in "" ) > > Seems both should return True ... Agreed. I'm surprised the first doesn't. Vern From jsiwek at ncsa.illinois.edu Thu May 12 08:52:17 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 12 May 2011 10:52:17 -0500 (CDT) Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> Message-ID: Can type notation like "vector of string" and "set[addr]" be parsed in BiF declarations? Doesn't work for me and I don't see any examples of this in current code. Looks like the current practice is to declare a type alias in bro.init and then use that alias in the BiF. - Jon From bro at tracker.bro-ids.org Thu May 12 04:50:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 12 May 2011 11:50:27 -0000 Subject: [Bro-Dev] #457: Signatures only triggering once per connection Message-ID: <046.b6548e12e0a5f2730674675ef51e14c8@tracker.bro-ids.org> #457: Signatures only triggering once per connection ---------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ A specific signature looks like it will only trigger once per connection. This causes problems with the HTTP file type identification if there are multiple pipelines requests in a connection and multiple of the same file type are transferred over the connection. Maybe there needs to be a keyword in the signature language to allow specific signatures to fire more than once per connection? Eventually this won't matter when we're doing generic file analysis because the file type identification will be done differently, but it's causing a problem for file type identification currently. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Thu May 12 09:29:21 2011 From: seth at icir.org (Seth Hall) Date: Thu, 12 May 2011 12:29:21 -0400 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: References: Message-ID: <979F7853-6629-49D1-B33A-B689DE4F4427@icir.org> On May 12, 2011, at 11:52 AM, Jonathan Siwek wrote: > Looks like the current practice is to declare a type alias in bro.init and then use that alias in the BiF. You got it right. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Thu May 12 10:09:33 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 12 May 2011 10:09:33 -0700 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> Message-ID: <20110512170933.GF47420@icir.org> On Thu, May 12, 2011 at 10:52 -0500, you wrote: > Can type notation like "vector of string" and "set[addr]" be parsed in > BiF declarations? D'oh! I never realized that that's reason for the type aliases. Seems something we should teach bifcl at some point. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Thu May 12 12:49:44 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 12 May 2011 12:49:44 -0700 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <20110512170933.GF47420@icir.org> References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> Message-ID: <4DCC39D8.4040302@icir.org> On 5/12/11 10:09 , Robin Sommer wrote: > > On Thu, May 12, 2011 at 10:52 -0500, you wrote: > >> Can type notation like "vector of string" and "set[addr]" be parsed in >> BiF declarations? > > D'oh! I never realized that that's reason for the type aliases. Seems > something we should teach bifcl at some point. having looked at bifcl recently, I might actually make sense to consider extending the Bro language parser to be able to also parse bifs (i.e., having a "bif-mode" but still compiling a bifcl binary). Type definitions in bro can be quite complex and one would have to completely duplicate that in bif. Similarly, when I was working on the bif syntax to make record types, enums, etc. available to C++ I thought that it would have been great if I could just specify the record in the BiF. Instead I have to specify the record in a bro file and put a simple "type foo: record;" in the BiF. Otherwise I would have had to copy all record-type-definition syntax. I think the more we use BiFs the more we will encounter such situations were the BiF-syntax is not powerful enough and were we would like to have elements of the Bro syntax available. Just my 2ct. gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From baxterw3232 at gmail.com Fri May 13 08:01:50 2011 From: baxterw3232 at gmail.com (Will) Date: Fri, 13 May 2011 11:01:50 -0400 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <4DCC39D8.4040302@icir.org> References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> Message-ID: This isn't likely in the same vein, but is the syntax below correct? It doesn't appear to be working properly. # ... unless the lookup comes from one of these hosts. const okay_to_lookup_sensitive_hosts: set[addr] = { [172.0.0.1], [172.0.0.2], } &redef; Thanks, -Will On Thu, May 12, 2011 at 3:49 PM, Gregor Maier wrote: > On 5/12/11 10:09 , Robin Sommer wrote: >> >> On Thu, May 12, 2011 at 10:52 -0500, you wrote: >> >>> Can type notation like "vector of string" and "set[addr]" be parsed in >>> BiF declarations? >> >> D'oh! I never realized that that's reason for the type aliases. Seems >> something we should teach bifcl at some point. > > having looked at bifcl recently, I might actually make sense to consider > extending the Bro language parser to be able to also parse bifs (i.e., > having a "bif-mode" but still compiling a bifcl binary). > > Type definitions in bro can be quite complex and one would have to > completely duplicate that in bif. > > Similarly, when I was working on the bif syntax to make record types, > enums, etc. available to C++ I thought that it would have been great if > I could just specify the record in the BiF. Instead I have to specify > the record in a bro file and put a simple "type foo: record;" in the > BiF. Otherwise I would have had to copy all record-type-definition syntax. > > I think the more we use BiFs the more we will encounter such situations > were the BiF-syntax is not powerful enough and were we would like to > have elements of the Bro syntax available. > > > Just my 2ct. > gregor > -- > Gregor Maier > ? > Int. Computer Science Institute (ICSI) > 1947 Center St., Ste. 600 > Berkeley, CA 94704, USA > http://www.icir.org/gregor/ > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From bro at tracker.bro-ids.org Fri May 13 12:22:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 13 May 2011 19:22:01 -0000 Subject: [Bro-Dev] #458: Debian Packaging via CPack Message-ID: <048.9b1834486ac88d67b57ae6dc4688573b@tracker.bro-ids.org> #458: Debian Packaging via CPack --------------------+-------------------- Reporter: jsiwek | Owner: jsiwek Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | --------------------+-------------------- This should be (re)investigated because, as of CMake 2.8.3, automatic dependency detection looks possible. This was a major reason why we didn't originally create scripts to generate DEBs -- they wouldn't automatically include the dependency meta-info. See http://public.kitware.com/Bug/view.php?id=10292 Main points of this task: * Add a `make-deb-packages` script analogous to the current mac/rpm ones * LD_LIBRARY_PATH might need to point to `./opt/bro/lib` during `make package` to workaround RPATH bug (see CMake ticket above) * In `cmake/ConfigurePackaging.cmake`'s `SetPackageGenerators` macro for "Linux" platform: * Add DEB to CPACK_GENERATOR * Set CPACK_DEBIAN_PACKAGE_SHLIBDEPS * Test that the pre/post install package scripts work -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Fri May 13 17:46:10 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 13 May 2011 17:46:10 -0700 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <4DCC39D8.4040302@icir.org> References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> Message-ID: <20110514004610.GN8623@icir.org> On Thu, May 12, 2011 at 12:49 -0700, you wrote: > Type definitions in bro can be quite complex and one would have to > completely duplicate that in bif. Does bifcf really need to parse the type fully? Could it just treat more complex types as opaque strings and carry them through into the output? > I could just specify the record in the BiF. Instead I have to specify > the record in a bro file and put a simple "type foo: record;" in the > BiF. Otherwise I would have had to copy all record-type-definition syntax. Likewise, would some simpler parsing rule work here that doesn't understand all the pieces going into the record? Haven't really thought this through admittedly. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Fri May 13 17:51:25 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 13 May 2011 17:51:25 -0700 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> Message-ID: <20110514005125.GO8623@icir.org> On Fri, May 13, 2011 at 11:01 -0400, you wrote: > It doesn't appear to be working properly. Normally you'd write: const okay_to_lookup_sensitive_hosts: set[addr] = { 172.0.0.1, 172.0.0.2, } &redef; But your version seems to work as well. With both versions, I get: # bro -e "print okay_to_lookup_sensitive_hosts" foo.bro { 172.0.0.1, 172.0.0.2 } What's not working? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Sat May 14 11:55:27 2011 From: gregor at icir.org (Gregor Maier) Date: Sat, 14 May 2011 11:55:27 -0700 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <20110514004610.GN8623@icir.org> References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> <20110514004610.GN8623@icir.org> Message-ID: <4DCED01F.40205@icir.org> On 5/13/11 17:46 , Robin Sommer wrote: >> Type definitions in bro can be quite complex and one would have to >> completely duplicate that in bif. > > Does bifcf really need to parse the type fully? Could it just treat > more complex types as opaque strings and carry them through into the > output? I thought about that too. However the problem with treating the types as opaque is that the parser still has to parse over them as they contain keyword, tokens, braces, etc. >> I could just specify the record in the BiF. Instead I have to specify >> the record in a bro file and put a simple "type foo: record;" in the >> BiF. Otherwise I would have had to copy all record-type-definition syntax. > > Likewise, would some simpler parsing rule work here that doesn't > understand all the pieces going into the record? Haven't really > thought this through admittedly. This has the same problem as the types in the header, however, it might be easier to solve: One solution might be to introduce a new kind of brackets/paranthese so that the scanner can skip over the record definition, e.g., type asdf: record @{ # opaque string comes here @} However, that's yet another brace-style.... (maybe one could also re-use another bracket or brace, have to look at the bif parser for that though). Still, one "problem" is that if there is a syntax error in the record definition, you will only see it during Bro-runtime in the generated .bro file. That implies that we probably have to add markers to the generated bro-file so that a user knows where the error actually came from. But that's more a cosmetic issue. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Sat May 14 13:23:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 14 May 2011 20:23:54 -0000 Subject: [Bro-Dev] #459: Test Ticket Message-ID: <047.0b888b07ab2792375ced235070fd3926@tracker.bro-ids.org> #459: Test Ticket -------------------+----------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Component: Bro | Version: git/master -------------------+----------------------- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat May 14 13:26:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 14 May 2011 20:26:51 -0000 Subject: [Bro-Dev] #459: Test Ticket In-Reply-To: <047.0b888b07ab2792375ced235070fd3926@tracker.bro-ids.org> References: <047.0b888b07ab2792375ced235070fd3926@tracker.bro-ids.org> Message-ID: <062.87e8bca0da1b5e8c5f0d51617e8e6765@tracker.bro-ids.org> #459: Test Ticket ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by robin): * status: new => closed Comment: On Sat, May 14, 2011 at 20:23 -0000, you wrote: > #459: Test Ticket Cool, email submission now works with the new tracker. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat May 14 13:31:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 14 May 2011 20:31:53 -0000 Subject: [Bro-Dev] #457: Signatures only triggering once per connection In-Reply-To: <046.b6548e12e0a5f2730674675ef51e14c8@tracker.bro-ids.org> References: <046.b6548e12e0a5f2730674675ef51e14c8@tracker.bro-ids.org> Message-ID: <061.47874c6aa84044f564e0a45dd9a832ed@tracker.bro-ids.org> #457: Signatures only triggering once per connection ----------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by robin): On Thu, May 12, 2011 at 11:50 -0000, you wrote: > A specific signature looks like it will only trigger once per connection. Yes, that's actually by design. I don't recall all the specifics right now, but I think changing that, even if only for selected signatures, wouldn't be straight-forward. And given that in the future the file analyzer will take care of this anyway, I'd prefer to leave things as they are. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Sun May 15 21:17:18 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 15 May 2011 21:17:18 -0700 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <4DCED01F.40205@icir.org> References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> <20110514004610.GN8623@icir.org> <4DCED01F.40205@icir.org> Message-ID: <20110516041717.GF1305@icir.org> On Sat, May 14, 2011 at 11:55 -0700, you wrote: > I thought about that too. However the problem with treating the types as > opaque is that the parser still has to parse over them as they contain > keyword, tokens, braces, etc. Wouldn't it only need to follow brackets and parentheses? That's pretty straight-forward, and everything read while doing so would just be concatenated into a single string. > However, that's yet another brace-style.... Yeah, I'm already confused by those we have. :-) But can't we get away here with a similar heuristic as above? > Still, one "problem" is that if there is a syntax error in the record > definition, you will only see it during Bro-runtime in the generated > .bro file. I could live with that as it's not something users are supposed to mess with anyway. :) And our testing should already make sure that this doesn't happen in released versions. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Mon May 16 11:56:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 16 May 2011 18:56:35 -0000 Subject: [Bro-Dev] #458: Debian Packaging via CPack In-Reply-To: <048.9b1834486ac88d67b57ae6dc4688573b@tracker.bro-ids.org> References: <048.9b1834486ac88d67b57ae6dc4688573b@tracker.bro-ids.org> Message-ID: <063.c7af09dda05487eda37fa4015e7a17a1@tracker.bro-ids.org> #458: Debian Packaging via CPack ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * version: => git/master * type: Task => Merge Request Comment: DEB packaging is now implemented via CMake/Cpack on branch `topic/jsiwek /deb-packaging` in the bro, broctl, and broccoli repositories. Extra points not covered in the original outline of how it was supposed to work: * configuration files had to be explicitly provided via a `conffiles` file (differs from how RPMs are done) * the `dpkg-shlibdeps` command is required to be installed (provided by dpkg-dev package) * CMake/CPack don't always include empty directories that are supposed to be installed by a package, so the post install script now ensures they get created -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Mon May 16 14:25:00 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 16 May 2011 23:25:00 +0200 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <20110516041717.GF1305@icir.org> References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> <20110514004610.GN8623@icir.org> <4DCED01F.40205@icir.org> <20110516041717.GF1305@icir.org> Message-ID: <4DD1962C.3060406@icir.org> >> I thought about that too. However the problem with treating the types as >> opaque is that the parser still has to parse over them as they contain >> keyword, tokens, braces, etc. > > Wouldn't it only need to follow brackets and parentheses? That's > pretty straight-forward, and everything read while doing so would just > be concatenated into a single string. I don't think that you can specify a yacc grammar that will work. E.g., consider this grammar: type_def: TOK_TYPE opt_ws TOK_ID opt_ws ':' opt_ws TOK_RECORD opt_ws ';' the trailing ';' would have to be replaced with something like .... TOK_RECORD opt_ws { THE_OPAQUE_STRING } but I don't see how you can specify THE_OPAQUE_STRING in a short way without reproducing a lot of Bro's grammar. But maybe I'm missing something. I'm not a yacc expert. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From baxterw3232 at gmail.com Tue May 17 08:16:43 2011 From: baxterw3232 at gmail.com (Will) Date: Tue, 17 May 2011 11:16:43 -0400 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <20110514005125.GO8623@icir.org> References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> <20110514005125.GO8623@icir.org> Message-ID: Thanks for the quick reply and sorry for the delayed response. I have been trying to troubleshoot a few other issues. On Fri, May 13, 2011 at 8:51 PM, Robin Sommer wrote: > > On Fri, May 13, 2011 at 11:01 -0400, you wrote: > >> It doesn't appear to be working properly. > > Normally you'd write: > > ? ?const okay_to_lookup_sensitive_hosts: set[addr] = { > ? ? ? ? ? ? ? ?172.0.0.1, > ? ? ? ? ? ? ? ?172.0.0.2, > ? ?} &redef; > > But your version seems to work as well. With both versions, I get: > > ? ?# bro -e "print okay_to_lookup_sensitive_hosts" foo.bro > ? ?{ > ? ?172.0.0.1, > ? ?172.0.0.2 > ? ?} > > What's not working? > So, this variable defines hosts in dns.bro that we intend to ignore sensitive DNS queries from (recursive DNS servers). A couple of weeks ago I installed and started using broctl on this box and with a few minor changes everything continued working fine except we are now getting alerts from the two hosts defined here. I wasn't aware of being able to specify and print a single variable from bro, as you did above, but ecstatic about how much easier that will make things when troubleshooting. On my first attempt, I think I broke something else. I actually have scan.bro commented out, so I'm assuming I this indicates a bigger issue? /usr/local/bro/share/bro/site]# bro -e "print okay_to_lookup_sensitive_hosts" local.bro /usr/local/bro/share/bro/scan.bro, line 117: internal error: NB-DNS error in DNS_Mgr::WaitForReplies (ns_initparse(): Message too long) Abort trap: 6 (core dumped) After rebuilding bro and broctl, I created a skeleton script that just had the statements above and it worked just as you showed. Unfortunately, we are still getting notices from the hosts defined in that variable. Like I said, the only change (that I am aware I made) is upgrading bro from 1.5.1 to 1.5.3 and installing broctl. This may be something that should be on the user list than dev? Thanks again! Will > Robin > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL ? ?* Fax ? +1 (510) 666-2956 * ? www.icir.org > From robin at icir.org Tue May 17 20:50:14 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 17 May 2011 20:50:14 -0700 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <4DD1962C.3060406@icir.org> References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> <20110514004610.GN8623@icir.org> <4DCED01F.40205@icir.org> <20110516041717.GF1305@icir.org> <4DD1962C.3060406@icir.org> Message-ID: <20110518035014.GC95709@icir.org> On Mon, May 16, 2011 at 23:25 +0200, you wrote: > .... TOK_RECORD opt_ws { THE_OPAQUE_STRING } How about something like this: .... TOK_RECORD opt_ws { opaque_string } opaque_string : TOK_NO_BRACKET_STRING opaque_string | '[' opaque_string ']' | '{' opaque_string '}' | '(' opaque_string ')' | TOK_NO_BRACKET_STRING is something like '[^{}[]()]+'. And for each rule, we append the $1,...,$n to a global current_opaque_string. This is probably too simplistic, I'm sure I'm missing something (and it doesn't preserve white-space; we'd need switch the lexer into a separate mode). But some hack like that. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Tue May 17 21:01:00 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 17 May 2011 21:01:00 -0700 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> <20110514005125.GO8623@icir.org> Message-ID: <20110518040100.GD95709@icir.org> On Tue, May 17, 2011 at 11:16 -0400, you wrote: > I wasn't aware of being able to specify and print a single variable > from bro, as you did above, but ecstatic about how much easier that > will make things when troubleshooting. Are you aware of broctl's "print" command? That shows you the value of variable at runtime. Try running that with "okay_to_lookup_sensitive_hosts" to see if the broctl configuration gets it right. > /usr/local/bro/share/bro/scan.bro, line 117: internal error: NB-DNS > error in DNS_Mgr::WaitForReplies (ns_initparse(): Message too long) > Abort trap: 6 (core dumped) These kind of errors usually indicate trouble with the system's DNS setup. However, I don't think I've ever seen the "message too long" message. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From vern at icir.org Wed May 18 02:04:33 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 18 May 2011 02:04:33 -0700 Subject: [Bro-Dev] metrics framework In-Reply-To: <20110510212958.GI27664@icir.org> (Tue, 10 May 2011 14:29:58 PDT). Message-ID: <20110518090433.D104F369FC8@taffy.ICSI.Berkeley.EDU> I like where this is going in general. > > - Missing support for cluster deployment. > > Yeah, that's a tough one. Full &synchronize would be overkill, but > sending the data via events, like you suggest, also sounds quite > expensive if there are lots of entities for which something's counted. What about a notion of "reduce", similar to the reduce operation in map-reduce? It seems for a lot of metrics/statistics/time-series there will be a natural way of combining parallelized computation of a given sort over the sequence of values. Vern From gregor at icir.org Wed May 18 02:27:06 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 18 May 2011 11:27:06 +0200 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <20110518035014.GC95709@icir.org> References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> <20110514004610.GN8623@icir.org> <4DCED01F.40205@icir.org> <20110516041717.GF1305@icir.org> <4DD1962C.3060406@icir.org> <20110518035014.GC95709@icir.org> Message-ID: <4DD390EA.9060708@icir.org> >> .... TOK_RECORD opt_ws { THE_OPAQUE_STRING } > > How about something like this: > > .... TOK_RECORD opt_ws { opaque_string } > > opaque_string : TOK_NO_BRACKET_STRING opaque_string > | '[' opaque_string ']' > | '{' opaque_string '}' > | '(' opaque_string ')' > | you probably only need to take care of '{' and '}'. Should be able to handle brackets and parentheses as opaque as well. > TOK_NO_BRACKET_STRING is something like '[^{}[]()]+'. For this to work yacc has to be able to switch the lexer mode. I don't know whether yacc can to do this (I think only the lexer can switch its mode/state). Finally, (although not a show-stopper because unlikely), the above has problems with quoted strings that have an unmatched number of braces in them. cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Wed May 18 09:09:59 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 18 May 2011 16:09:59 -0000 Subject: [Bro-Dev] #458: Debian Packaging via CPack In-Reply-To: <048.9b1834486ac88d67b57ae6dc4688573b@tracker.bro-ids.org> References: <048.9b1834486ac88d67b57ae6dc4688573b@tracker.bro-ids.org> Message-ID: <063.cf53a1638340cc9e1c2f838a4ad1da50@tracker.bro-ids.org> #458: Debian Packaging via CPack -----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed May 18 09:39:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 18 May 2011 16:39:50 -0000 Subject: [Bro-Dev] #298: Automatic build tests on NMI testbed. In-Reply-To: <046.52cb04f5dd135b7d304d18bd01deebaf@tracker.bro-ids.org> References: <046.52cb04f5dd135b7d304d18bd01deebaf@tracker.bro-ids.org> Message-ID: <061.b525e0cf1ee9f70f5e83c1ba267921ce@tracker.bro-ids.org> #298: Automatic build tests on NMI testbed. ---------------------+-------------------- Reporter: seth | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ---------------------+-------------------- Comment (by jsiwek): Just a reminder/note: we should make sure to build against optional dependencies, which are currently libgeoip, libmagic, libz (I think we can skip google-perftools). Current NMI builds don't look like they use libgeoip. The "final release" binary packages should also be built against these dependencies. -- Ticket URL: Bro Tracker Bro Issue Tracker From baxterw3232 at gmail.com Wed May 18 10:59:29 2011 From: baxterw3232 at gmail.com (Will) Date: Wed, 18 May 2011 13:59:29 -0400 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <20110518040100.GD95709@icir.org> References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> <20110514005125.GO8623@icir.org> <20110518040100.GD95709@icir.org> Message-ID: On Wed, May 18, 2011 at 12:01 AM, Robin Sommer wrote: > > On Tue, May 17, 2011 at 11:16 -0400, you wrote: > >> I wasn't aware of being able to specify and print a single variable >> from bro, as you did above, but ecstatic about how much easier that >> will make things when troubleshooting. > > Are you aware of broctl's "print" command? That shows you the value of > variable at runtime. Try running that with > "okay_to_lookup_sensitive_hosts" to see if the broctl configuration > gets it right. > No, I wasn't til now. Forgive my nubness, but what specifically are and ? i.e. process id of parent ps or variable name? If standalone, would node be just bro, localhost or something completely different? # broctl print "okay_to_lookup_sensitive_hosts" bro >> /usr/local/bro/share/bro/scan.bro, line 117: internal error: NB-DNS >> error in DNS_Mgr::WaitForReplies (ns_initparse(): Message too long) >> Abort trap: 6 (core dumped) > > These kind of errors usually indicate trouble with the system's DNS > setup. However, I don't think I've ever seen the "message too long" > message. > So everything has been running smoothly for the last 24 hours or so, then another crash. More details: Cannot access memory at address 0x5 ==== stderr.log pcap bufsize = 32768 listening on bge1 1305661424.619015 run-time error: string without NUL terminator: "\xff\xff*^Hbc0975.0\xc0^L\0^A\0^A\0\0^D\xb0\0^D^J^D^D.^Hbc097531\xc0^L\0^A\0^A\0\0^D\xb0\0^D\xac^Q(\xa7^Hbc097532.\xff\xff*^Hbc0975.0\xc0^L\0^A\0^A\0\0^D\xb0\0^D^J^D^D.^Hbc097531\xc0^L\0^A\0^A\0\0^D\xb0\0^D\xac^Q(\xa7^Hbc097532" 1305707636.214027 run-time error: string without NUL terminator: "hosta^Ecompany^Corg\0\xc0^L\0!\0^A\0\0^Bx\0\x1c\0\0\0d^A\x85^J04c2nvrs-a^Ecompa" 1305707636.259675 run-time error: string without NUL terminator: "hosta^Ecompany^Corg\0\xc0^L\0!\0^A\0\0^Bx\0\x1c\0\0\0d^A\x85^J04c2nvrs-a^Ecompa" 1305734703.016096 run-time error: string with embedded NUL: "oo^M\xc3\xca\0^A\0^A\0" 1305735623.373659 internal error: NB-DNS error in DNS_Mgr::Process (ns_initparse(): Message too long) /usr/local/bro/share/broctl/scripts/run-bro: line 73: 31891 Abort trap: 6 (core dumped) nohup $tmpbro $@ ==== stdout.log ==== .status TERMINATED [internal_error] ==== No prof.log. bro.core Core was generated by `bro'. Program terminated with signal 6, Aborted. #0 0x286e8a27 in kill () from /lib/libc.so.7 #0 0x286e8a27 in kill () from /lib/libc.so.7 #1 0x286e8986 in raise () from /lib/libc.so.7 #2 0x286e756a in abort () from /lib/libc.so.7 #3 0x080517a4 in internal_error () at SSLInterpreter.cc:30 #4 0x080a1691 in DNS_Mgr::Process (this=0xbfbfe554) at DNS_Mgr.cc:1069 #5 0x08147285 in net_run () at Net.cc:528 #6 0x0804fbff in main (argc=) at main.cc:999 Followed by this crash when broctl tried to restart 5 minutes later. So, do you still think this looks like a host configuration issue? This is on a freebsd 7.3 host, fyi. [bro] Variable "this" is not available. ==== stderr.log /usr/local/bro/share/bro/scan.bro, line 117: internal error: NB-DNS error in DNS_Mgr::WaitForReplies (ns_initparse(): Message too long) /usr/local/bro/share/broctl/scripts/run-bro: line 73: 55221 Abort trap: 6 (core dumped) nohup $tmpbro $@ ==== stdout.log ==== .status TERMINATED [internal_error] ==== No prof.log. bro.core Core was generated by `bro'. Program terminated with signal 6, Aborted. #0 0x286e8a27 in kill () from /lib/libc.so.7 #0 0x286e8a27 in kill () from /lib/libc.so.7 #1 0x286e8986 in raise () from /lib/libc.so.7 #2 0x286e756a in abort () from /lib/libc.so.7 #3 0x080517a4 in internal_error () at SSLInterpreter.cc:30 #4 0x080a199c in DNS_Mgr::Resolve (this=) at DNS_Mgr.cc:580 #5 0x080a1dbd in DNS_Mgr::LookupHost (this=0x82dc800, name=0x85a3939 "test-scooter.av.pa-x.dec.com") at DNS_Mgr.cc:468 #6 0x080682b7 in brolex () at scan.l:324 #7 0x08053bbc in yyparse () at p.c:2260 #8 0x0804ee16 in main (argc=17, argv=0xbfbfeb5c) at main.cc:749 Thanks, Will > Robin > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL ? ?* Fax ? +1 (510) 666-2956 * ? www.icir.org > From vern at icir.org Wed May 18 12:01:15 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 18 May 2011 12:01:15 -0700 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <4DD390EA.9060708@icir.org> (Wed, 18 May 2011 11:27:06 +0200). Message-ID: <20110518190115.D729C36A00C@taffy.ICSI.Berkeley.EDU> > For this to work yacc has to be able to switch the lexer mode. I don't > know whether yacc can to do this (I think only the lexer can switch its > mode/state). It can if you do it carefully. Basically, there's a "preproduction" (something reduced early on) that has an action to call into the scanner and flag for it to switch. Vern From baxterw3232 at gmail.com Wed May 18 12:56:58 2011 From: baxterw3232 at gmail.com (Will) Date: Wed, 18 May 2011 15:56:58 -0400 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> <20110514005125.GO8623@icir.org> <20110518040100.GD95709@icir.org> Message-ID: On Wed, May 18, 2011 at 1:59 PM, Will wrote: > On Wed, May 18, 2011 at 12:01 AM, Robin Sommer wrote: >> >> On Tue, May 17, 2011 at 11:16 -0400, you wrote: >> >>> I wasn't aware of being able to specify and print a single variable >>> from bro, as you did above, but ecstatic about how much easier that >>> will make things when troubleshooting. >> >> Are you aware of broctl's "print" command? That shows you the value of >> variable at runtime. Try running that with >> "okay_to_lookup_sensitive_hosts" to see if the broctl configuration >> gets it right. >> > > No, I wasn't til now. Forgive my nubness, but what specifically are > and ? i.e. process id of parent ps or variable name? If > standalone, would node be just bro, localhost or something completely > different? > > # broctl print "okay_to_lookup_sensitive_hosts" bro > >>> /usr/local/bro/share/bro/scan.bro, line 117: internal error: NB-DNS >>> error in DNS_Mgr::WaitForReplies (ns_initparse(): Message too long) >>> Abort trap: 6 (core dumped) >> >> These kind of errors usually indicate trouble with the system's DNS >> setup. However, I don't think I've ever seen the "message too long" >> message. >> Looks like a ticket is open for a similar issue: http://tracker.bro-ids.org/bro/ticket/255 The odd thing is, I don't know what changed that would cause these to start occurring recently. > > So everything has been running smoothly for the last 24 hours or so, > then another crash. More details: > > Cannot access memory at address 0x5 > ==== stderr.log > pcap bufsize = 32768 > listening on bge1 > 1305661424.619015 run-time error: string without NUL terminator: > "\xff\xff*^Hbc0975.0\xc0^L\0^A\0^A\0\0^D\xb0\0^D^J^D^D.^Hbc097531\xc0^L\0^A\0^A\0\0^D\xb0\0^D\xac^Q(\xa7^Hbc097532.\xff\xff*^Hbc0975.0\xc0^L\0^A\0^A\0\0^D\xb0\0^D^J^D^D.^Hbc097531\xc0^L\0^A\0^A\0\0^D\xb0\0^D\xac^Q(\xa7^Hbc097532" > 1305707636.214027 run-time error: string without NUL terminator: > "hosta^Ecompany^Corg\0\xc0^L\0!\0^A\0\0^Bx\0\x1c\0\0\0d^A\x85^J04c2nvrs-a^Ecompa" > 1305707636.259675 run-time error: string without NUL terminator: > "hosta^Ecompany^Corg\0\xc0^L\0!\0^A\0\0^Bx\0\x1c\0\0\0d^A\x85^J04c2nvrs-a^Ecompa" > 1305734703.016096 run-time error: string with embedded NUL: > "oo^M\xc3\xca\0^A\0^A\0" > 1305735623.373659 internal error: NB-DNS error in DNS_Mgr::Process > (ns_initparse(): Message too long) > /usr/local/bro/share/broctl/scripts/run-bro: line 73: 31891 Abort > trap: 6 ? ? ? ? ? (core dumped) nohup $tmpbro $@ > ==== stdout.log > ==== .status > TERMINATED [internal_error] > ==== No prof.log. > bro.core > Core was generated by `bro'. > Program terminated with signal 6, Aborted. > #0 ?0x286e8a27 in kill () from /lib/libc.so.7 > #0 ?0x286e8a27 in kill () from /lib/libc.so.7 > #1 ?0x286e8986 in raise () from /lib/libc.so.7 > #2 ?0x286e756a in abort () from /lib/libc.so.7 > #3 ?0x080517a4 in internal_error () at SSLInterpreter.cc:30 > #4 ?0x080a1691 in DNS_Mgr::Process (this=0xbfbfe554) at DNS_Mgr.cc:1069 > #5 0x08147285 in net_run () at Net.cc:528 > #6 ?0x0804fbff in main (argc=) at main.cc:999 > > Followed by this crash when broctl tried to restart 5 minutes later. > So, do you still think this looks like a host configuration issue? > This is on a freebsd 7.3 host, fyi. > > [bro] > Variable "this" is not available. > ==== stderr.log > /usr/local/bro/share/bro/scan.bro, line 117: internal error: NB-DNS > error in DNS_Mgr::WaitForReplies (ns_initparse(): Message too long) > /usr/local/bro/share/broctl/scripts/run-bro: line 73: 55221 Abort > trap: 6 ? ? ? ? ? (core dumped) nohup $tmpbro $@ > ==== stdout.log > ==== .status > TERMINATED [internal_error] > ==== No prof.log. > bro.core > Core was generated by `bro'. > Program terminated with signal 6, Aborted. > #0 ?0x286e8a27 in kill () from /lib/libc.so.7 > #0 ?0x286e8a27 in kill () from /lib/libc.so.7 > #1 ?0x286e8986 in raise () from /lib/libc.so.7 > #2 ?0x286e756a in abort () from /lib/libc.so.7 > #3 ?0x080517a4 in internal_error () at SSLInterpreter.cc:30 > #4 ?0x080a199c in DNS_Mgr::Resolve (this=) at DNS_Mgr.cc:580 > #5 ?0x080a1dbd in DNS_Mgr::LookupHost (this=0x82dc800, name=0x85a3939 > "test-scooter.av.pa-x.dec.com") at DNS_Mgr.cc:468 > #6 ?0x080682b7 in brolex () at scan.l:324 > #7 ?0x08053bbc in yyparse () at p.c:2260 > #8 ?0x0804ee16 in main (argc=17, argv=0xbfbfeb5c) at main.cc:749 > > Thanks, > > Will > >> Robin >> >> -- >> Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org >> ICSI/LBNL ? ?* Fax ? +1 (510) 666-2956 * ? www.icir.org >> > From bro at tracker.bro-ids.org Sun May 22 18:56:55 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 23 May 2011 01:56:55 -0000 Subject: [Bro-Dev] #457: Signatures only triggering once per connection In-Reply-To: <046.b6548e12e0a5f2730674675ef51e14c8@tracker.bro-ids.org> References: <046.b6548e12e0a5f2730674675ef51e14c8@tracker.bro-ids.org> Message-ID: <061.2c96c04b96edc11901bb3d9ee3b88ea9@tracker.bro-ids.org> #457: Signatures only triggering once per connection -----------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Rejected | Keywords: -----------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Rejected Comment: Replying to [comment:0 robin]: > And given that in the future the file > analyzer will take care of this anyway, I'd prefer to leave things as > they are. Sounds good.. closing. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun May 22 21:28:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 23 May 2011 04:28:31 -0000 Subject: [Bro-Dev] #200: broctl update resets/corrupts certain variables In-Reply-To: <048.70165624d985998c153024ccf3bf0cdb@tracker.bro-ids.org> References: <048.70165624d985998c153024ccf3bf0cdb@tracker.bro-ids.org> Message-ID: <063.894f415b5fd8701b8021dbaeedb4a10c@tracker.bro-ids.org> #200: broctl update resets/corrupts certain variables -----------------------------+--------------------------- Reporter: justin | Owner: seth Type: Fastpath | Status: closed Priority: Low | Milestone: Bro1.6 Component: BroControl | Version: 1.5.2 Resolution: Solved/Applied | Keywords: broctl update -----------------------------+--------------------------- Changes (by seth): * status: accepted => closed * type: defect => Fastpath * resolution: => Solved/Applied Comment: There is a fix for this problem in broctl's fastpath branch. I ended up changing the BiF that does the variable updating only work for consts with &redef. Previously it would work for any global (global and const) which caused problems with any script that maintains a state table... tables indexed by conn_id would be flattened if the table was initialized to an empty table when doing update. The new scripts are all being written in a way that they will cope correctly with this change and most of the old scripts would deal with this change just fine (I don't know which ones it breaks), but the current behavior breaks almost all scripts at the same time by reinitializing globals. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun May 22 21:28:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 23 May 2011 04:28:49 -0000 Subject: [Bro-Dev] #200: broctl update resets/corrupts certain variables In-Reply-To: <048.70165624d985998c153024ccf3bf0cdb@tracker.bro-ids.org> References: <048.70165624d985998c153024ccf3bf0cdb@tracker.bro-ids.org> Message-ID: <063.1d004315d88473b548d8c8b5a04f35d1@tracker.bro-ids.org> #200: broctl update resets/corrupts certain variables -------------------------+--------------------------- Reporter: justin | Owner: seth Type: Fastpath | Status: reopened Priority: Low | Milestone: Bro1.6 Component: BroControl | Version: 1.5.2 Resolution: | Keywords: broctl update -------------------------+--------------------------- Changes (by seth): * status: closed => reopened * resolution: Solved/Applied => -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun May 22 21:31:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 23 May 2011 04:31:47 -0000 Subject: [Bro-Dev] #23: DNS log format is hard to parse In-Reply-To: <046.1961881d7954cf5261c44aabb5c5338f@tracker.bro-ids.org> References: <046.1961881d7954cf5261c44aabb5c5338f@tracker.bro-ids.org> Message-ID: <061.f9be29ee5424128133a58bc2e9cd0bce@tracker.bro-ids.org> #23: DNS log format is hard to parse ------------------------------+-------------------- Reporter: vern | Owner: seth Type: Feature Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: ------------------------------+-------------------- Changes (by seth): * status: assigned => closed * resolution: => Solved/Applied Comment: Already rewritten for the next release. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun May 22 21:32:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 23 May 2011 04:32:49 -0000 Subject: [Bro-Dev] #299: Cleanup/update default policy scripts In-Reply-To: <046.5cf70f159db21509f90194e297addaa2@tracker.bro-ids.org> References: <046.5cf70f159db21509f90194e297addaa2@tracker.bro-ids.org> Message-ID: <061.b266b8eac3f04f06be3e72f694899299@tracker.bro-ids.org> #299: Cleanup/update default policy scripts -----------------------------+-------------------- Reporter: seth | Owner: seth Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: assigned => closed * resolution: => Solved/Applied Comment: This is seriously in progress (in the topic/policy-script-new branch) and not being tracked here as a single ticket so I'm closing this ticket. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun May 22 21:34:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 23 May 2011 04:34:53 -0000 Subject: [Bro-Dev] #207: Small bug in file rotate+setbuf In-Reply-To: <048.7c37b619e06710d4c868cb76da4ed734@tracker.bro-ids.org> References: <048.7c37b619e06710d4c868cb76da4ed734@tracker.bro-ids.org> Message-ID: <063.82993228d99cd8ebe5d995874d9aeb98@tracker.bro-ids.org> #207: Small bug in file rotate+setbuf -----------------------------+--------------------------- Reporter: justin | Owner: Type: Patch | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: Solved/Applied | Keywords: rotate buffer -----------------------------+--------------------------- Changes (by seth): * status: assigned => closed * resolution: => Solved/Applied Comment: This change seems to be in the master branch in git. I'm going to close this ticket. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun May 22 21:45:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 23 May 2011 04:45:47 -0000 Subject: [Bro-Dev] #411: Non-binpac analyzer generates incorrect weird In-Reply-To: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> References: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> Message-ID: <061.c120f8179c9f9f56bf1ce84b7bdc46a4@tracker.bro-ids.org> #411: Non-binpac analyzer generates incorrect weird ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): I just looked into this a bit more and I'm not sure how it should be handled. The 'if' that leads to this weird is also used for confirming the protocol but I don't know if this is the only case where a dual response from the server can be valid. I'm inclined to keep the protocol confirmation and just get rid of the weird. Thoughts? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun May 22 21:54:59 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 23 May 2011 04:54:59 -0000 Subject: [Bro-Dev] #425: Vector initializer is broken In-Reply-To: <046.23a48cfd7973b5a021a106c7de5a2c62@tracker.bro-ids.org> References: <046.23a48cfd7973b5a021a106c7de5a2c62@tracker.bro-ids.org> Message-ID: <061.b40b55b5a764c37eed0577c3f125867e@tracker.bro-ids.org> #425: Vector initializer is broken -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: And now this magically works... easiest. ticket. ever! and closed. (probably only easy because Robin addressed some vector stuff recently :) ) -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Sun May 22 23:55:24 2011 From: seth at icir.org (Seth Hall) Date: Mon, 23 May 2011 02:55:24 -0400 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> <20110514005125.GO8623@icir.org> <20110518040100.GD95709@icir.org> Message-ID: On May 18, 2011, at 1:59 PM, Will wrote: > No, I wasn't til now. Forgive my nubness, but what specifically are > and ? i.e. process id of parent ps or variable name? If > standalone, would node be just bro, localhost or something completely > different? ID is the name of the variable that you would like printed and node is the name of the Bro instance that you'd like to inspect the value on. If you don't give a node it will print the value from all nodes. >>> /usr/local/bro/share/bro/scan.bro, line 117: internal error: NB-DNS >>> error in DNS_Mgr::WaitForReplies (ns_initparse(): Message too long) >>> Abort trap: 6 (core dumped) >> >> These kind of errors usually indicate trouble with the system's DNS >> setup. However, I don't think I've ever seen the "message too long" >> message. >> > 1305735623.373659 internal error: NB-DNS error in DNS_Mgr::Process > (ns_initparse(): Message too long) > /usr/local/bro/share/broctl/scripts/run-bro: line 73: 31891 Abort > trap: 6 (core dumped) nohup $tmpbro $@ > ==== stdout.log > ==== .status > TERMINATED [internal_error] > ==== No prof.log. > bro.core > Core was generated by `bro'. > Program terminated with signal 6, Aborted. > #0 0x286e8a27 in kill () from /lib/libc.so.7 > #0 0x286e8a27 in kill () from /lib/libc.so.7 > #1 0x286e8986 in raise () from /lib/libc.so.7 > #2 0x286e756a in abort () from /lib/libc.so.7 > #3 0x080517a4 in internal_error () at SSLInterpreter.cc:30 Question for Robin/Vern/whomever, should this really be calling internal_error? It's definitely worthwhile to log that some sort of error was received from libbind, but I don't think Bro should be shutting down when it has a problem like this. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Sun May 22 23:58:07 2011 From: seth at icir.org (Seth Hall) Date: Mon, 23 May 2011 02:58:07 -0400 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> <20110514005125.GO8623@icir.org> <20110518040100.GD95709@icir.org> Message-ID: <484485AE-E6E2-4EEF-864A-733CA8E20BE2@icir.org> On May 18, 2011, at 3:56 PM, Will wrote: > The odd thing is, I don't know what changed that would cause these to > start occurring recently. It's usually that something has changed external to Bro. Perhaps a DNS server that hosts on your network tend to do traffic with occasionally started returning odd responses. This seems to be how most of the edge cases are found. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Mon May 23 00:09:10 2011 From: seth at icir.org (Seth Hall) Date: Mon, 23 May 2011 03:09:10 -0400 Subject: [Bro-Dev] metrics framework In-Reply-To: <20110518090433.D104F369FC8@taffy.ICSI.Berkeley.EDU> References: <20110518090433.D104F369FC8@taffy.ICSI.Berkeley.EDU> Message-ID: <043DCD15-8AB6-4C69-87E9-C35E126D743C@icir.org> On May 18, 2011, at 5:04 AM, Vern Paxson wrote: > What about a notion of "reduce", similar to the reduce operation in > map-reduce? It seems for a lot of metrics/statistics/time-series there > will be a natural way of combining parallelized computation of a given > sort over the sequence of values. I'm not completely sure that would apply because the only reduce operation that I'm currently envisioning is straight addition. It's basically taking the following example structure from all of the workers (with different counts on each worker obviously) and adding the values together on some break interval. { [1.2.3.0/24] = { ["GET"] = 20, ["POST"] = 1 }, [4.3.2.0/24] = { ["GET"] = 5304, ["POST"] = 45 }, .... and on, and on, and on.... } That would be an example of HTTP verbs used per /24 in requests. Each worker would have it's own table and on the break interval for the metric it would add together all of the values on the manager. It's certainly possible that I'm just plain missing your point too. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Mon May 23 05:34:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 23 May 2011 12:34:24 -0000 Subject: [Bro-Dev] #440: Add MD5 summing to existing file analyzer In-Reply-To: <046.52e936f836bfbc74d432dcf243af284c@tracker.bro-ids.org> References: <046.52e936f836bfbc74d432dcf243af284c@tracker.bro-ids.org> Message-ID: <061.e65311f9140411a7e48018f1201844c5@tracker.bro-ids.org> #440: Add MD5 summing to existing file analyzer ------------------------------+------------------------ Reporter: seth | Owner: Type: Feature Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Rejected | Keywords: ------------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Rejected Comment: All of the new scripts do this now where possible. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon May 23 05:35:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 23 May 2011 12:35:09 -0000 Subject: [Bro-Dev] #377: software.bro is non-functional In-Reply-To: <047.267509cc82949ffb5cf8883b11e832f2@tracker.bro-ids.org> References: <047.267509cc82949ffb5cf8883b11e832f2@tracker.bro-ids.org> Message-ID: <062.9824bb441e6458d6d030a07bd14fd65c@tracker.bro-ids.org> #377: software.bro is non-functional -----------------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Software.bro has already been overhauled for the next release. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon May 23 05:44:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 23 May 2011 12:44:11 -0000 Subject: [Bro-Dev] #372: bifcl cannot pass specific enum types In-Reply-To: <047.e8e6512c3965101e829a4a0884e3bb2b@tracker.bro-ids.org> References: <047.e8e6512c3965101e829a4a0884e3bb2b@tracker.bro-ids.org> Message-ID: <062.21ecebf3b0bf5d7e1dab924da5dd262e@tracker.bro-ids.org> #372: bifcl cannot pass specific enum types ----------------------+--------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ----------------------+--------------------- Comment (by seth): This stuff is integrated, right? Could someone close this ticket if it is. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon May 23 05:48:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 23 May 2011 12:48:09 -0000 Subject: [Bro-Dev] #394: all.bro loads non-existent policy and segfaults In-Reply-To: <048.54639988ae284713b2b5e1414267fed3@tracker.bro-ids.org> References: <048.54639988ae284713b2b5e1414267fed3@tracker.bro-ids.org> Message-ID: <063.d0ccd496b0a02eaf12517b92385c10d1@tracker.bro-ids.org> #394: all.bro loads non-existent policy and segfaults ----------------------+------------------------ Reporter: gregor | Owner: seth Type: Problem | Status: accepted Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: maketest ----------------------+------------------------ Changes (by seth): * keywords: => maketest * owner: => seth * status: new => accepted * milestone: Bro1.6 => Comment: I'm removing the milestone from this ticket since it's not directly relevant to the next release anymore, but I'm giving it the "maketest" tag because we should probably still figure out what's going on here and fix it. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon May 23 06:51:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 23 May 2011 13:51:05 -0000 Subject: [Bro-Dev] #349: Overly long timestamp segfault in cf In-Reply-To: <046.56407ac517a4a9a8207ec2f796d38db5@tracker.bro-ids.org> References: <046.56407ac517a4a9a8207ec2f796d38db5@tracker.bro-ids.org> Message-ID: <061.a8a5c04633fc392fb6ca9ccc569311bd@tracker.bro-ids.org> #349: Overly long timestamp segfault in cf ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: bro-aux | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): Just following up, any clue where you came across version 1.2.1? -- Ticket URL: Bro Tracker Bro Issue Tracker From baxterw3232 at gmail.com Mon May 23 07:20:45 2011 From: baxterw3232 at gmail.com (Will) Date: Mon, 23 May 2011 10:20:45 -0400 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <484485AE-E6E2-4EEF-864A-733CA8E20BE2@icir.org> References: <3015ebd5-91d6-4977-824c-f1fd5e4f2973@tangent.ncsa.illinois.edu> <20110512170933.GF47420@icir.org> <4DCC39D8.4040302@icir.org> <20110514005125.GO8623@icir.org> <20110518040100.GD95709@icir.org> <484485AE-E6E2-4EEF-864A-733CA8E20BE2@icir.org> Message-ID: On Mon, May 23, 2011 at 2:58 AM, Seth Hall wrote: > > On May 18, 2011, at 3:56 PM, Will wrote: > >> The odd thing is, I don't know what changed that would cause these to >> start occurring recently. > > > It's usually that something has changed external to Bro. ?Perhaps a DNS server that hosts on your network tend to do traffic with occasionally started returning odd responses. ?This seems to be how most of the edge cases are found. :) Yes, found this to be exactly the case. We built a new box and just transferred over the same bro scripts and haven't had any problems. Thanks! > > ?.Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > From vern at icir.org Mon May 23 11:01:34 2011 From: vern at icir.org (Vern Paxson) Date: Mon, 23 May 2011 11:01:34 -0700 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: (Mon, 23 May 2011 02:55:24 EDT). Message-ID: <20110523180134.573BC36A379@taffy.ICSI.Berkeley.EDU> > > 1305735623.373659 internal error: NB-DNS error in DNS_Mgr::Process > > (ns_initparse(): Message too long) > ... > Question for Robin/Vern/whomever, should this really be calling internal_error? No, it shouldn't. I suspect the problem is that nb_dns is returning an internal error and the caller lacks enough specifics to understand whether it represents something fatal or just means "forget about the current lookup". Vern From bro at tracker.bro-ids.org Tue May 24 07:18:39 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 14:18:39 -0000 Subject: [Bro-Dev] #356: Unidirectional MPLS tags, VLAN tags, and raw ethernet In-Reply-To: <046.980b718697d1c8726fdb19d9cc0ab5d4@tracker.bro-ids.org> References: <046.980b718697d1c8726fdb19d9cc0ab5d4@tracker.bro-ids.org> Message-ID: <061.aac4ae0f75a92800bbb5c8a69c304dfe@tracker.bro-ids.org> #356: Unidirectional MPLS tags, VLAN tags, and raw ethernet --------------------------------+------------------------ Reporter: seth | Owner: Type: Test Case Missing | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: --------------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: This has been added so closing this ticket. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue May 24 07:57:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 14:57:12 -0000 Subject: [Bro-Dev] #460: Problem with extended records and empty container types Message-ID: <046.e32eb4f1b2af66ecfcc23a676b3b6332@tracker.bro-ids.org> #460: Problem with extended records and empty container types ---------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ If you extend a record, the empty container types don't work anymore. The following code:: {{{ type Blah: record { whatever: string; }; redef record Blah += { test2: set[count] &default=set(); }; global foobar: Blah; }}} Results in this crash:: {{{ [build (topic/policy-scripts-new)]$ gdb --args ./src/bro test23.bro GNU gdb 6.3.50-20050815 (Apple version gdb-1469) (Wed May 5 04:36:56 UTC 2010) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared libraries ........ done (gdb) r Starting program: ./src/bro test23.bro Reading symbols for shared libraries .+++++++ done ./test23.bro, line 6 (set() and set[count]): error, &default value has inconsistent type Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000028 0x0000000100085138 in BroType::Tag (this=0x0) at Type.h:75 75 TypeTag Tag() const { return tag; } (gdb) bt #0 0x0000000100085138 in BroType::Tag (this=0x0) at Type.h:75 #1 0x0000000100224e60 in same_type (t1=0x100cb2be0, t2=0x0, is_init=0) at ./src/Type.cc:1645 #2 0x00000001000a5fe3 in Attributes::CheckAttr (this=0x100cb2d70, a=0x100cb2c90) at ./src/Attr.cc:268 #3 0x00000001000a6e2a in Attributes::AddAttr (this=0x100cb2d70, attr=0x100cb2c90) at ./src/Attr.cc:134 #4 0x00000001000a73b0 in Attributes::Attributes (this=0x100cb2d70, a=0x100cb2710, t=0x100cb27c0, arg_in_record=false) at ./src/Attr.cc:105 #5 0x0000000100220de3 in TypeDecl::TypeDecl (this=0x100cb2d50, t=0x100cb27c0, i=0x100b6e0c0 "test2", arg_attrs=0x100cb2710, in_record=false) at ./src/Type.cc:789 #6 0x0000000100080f22 in yyparse () at parse.y:954 #7 0x0000000100094a1d in main (argc=2, argv=0x7fff5fbfeba0) at ./src/main.cc:736 (gdb) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue May 24 08:14:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 15:14:06 -0000 Subject: [Bro-Dev] #410: Extension to init time pattern construction In-Reply-To: <046.38b30bae927e7dd85cebc62450515fe7@tracker.bro-ids.org> References: <046.38b30bae927e7dd85cebc62450515fe7@tracker.bro-ids.org> Message-ID: <061.d1fdbfc6edc310ea58da52f0c1f586e5@tracker.bro-ids.org> #410: Extension to init time pattern construction ------------------------------+---------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: language ------------------------------+---------------------- Changes (by seth): * keywords: => language * milestone: Bro1.6 => Bro1.7 Comment: Bumping this back to the next release as it's not critical for now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue May 24 08:14:39 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 15:14:39 -0000 Subject: [Bro-Dev] #371: New column-based logging interface. In-Reply-To: <047.efddc2565a07e6e2f01e0603fb54d6b5@tracker.bro-ids.org> References: <047.efddc2565a07e6e2f01e0603fb54d6b5@tracker.bro-ids.org> Message-ID: <062.77e50764bdd40adef5c34f95e12912e1@tracker.bro-ids.org> #371: New column-based logging interface. -----------------------------+----------------------- Reporter: robin | Owner: Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/topic Resolution: Solved/Applied | Keywords: -----------------------------+----------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: This is already merged into master. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue May 24 08:18:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 15:18:28 -0000 Subject: [Bro-Dev] #80: Checking for the existance keys in multi-dimensional tables gives error message In-Reply-To: <050.c01d8a094407695f72fd3d3d599c4f48@tracker.bro-ids.org> References: <050.c01d8a094407695f72fd3d3d599c4f48@tracker.bro-ids.org> Message-ID: <065.a7ab980f5abba919dfa62bff567f47a2@tracker.bro-ids.org> #80: Checking for the existance keys in multi-dimensional tables gives error message -----------------------+-------------------- Reporter: bernhard | Owner: Type: Problem | Status: seen Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.4 Resolution: | Keywords: -----------------------+-------------------- Comment (by seth): Is this going to be done in time for the next release? It would be really good because these multidimensional checks work in some cases and not in other which can be pretty confusing as it is. We could push it back if we need to. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue May 24 08:19:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 15:19:51 -0000 Subject: [Bro-Dev] #267: Patch to fix internal log_encryption_key handling In-Reply-To: <046.f3b2f414141b2c225cf35ac25c8fd93f@tracker.bro-ids.org> References: <046.f3b2f414141b2c225cf35ac25c8fd93f@tracker.bro-ids.org> Message-ID: <061.b5829351660791f3c4f3a836c8273744@tracker.bro-ids.org> #267: Patch to fix internal log_encryption_key handling ---------------------+-------------------- Reporter: seth | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: 1.5.2 Resolution: | Keywords: sprint ---------------------+-------------------- Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: I'm pushing this back to the next release since no one is seeing trouble with it anyway and we'll need to implement it for the logging framework too. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Tue May 24 08:28:20 2011 From: seth at icir.org (Seth Hall) Date: Tue, 24 May 2011 11:28:20 -0400 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <20110523180134.573BC36A379@taffy.ICSI.Berkeley.EDU> References: <20110523180134.573BC36A379@taffy.ICSI.Berkeley.EDU> Message-ID: On May 23, 2011, at 2:01 PM, Vern Paxson wrote: > No, it shouldn't. I suspect the problem is that nb_dns is returning an > internal error and the caller lacks enough specifics to understand whether > it represents something fatal or just means "forget about the current > lookup". I suppose the more direct question is, are there any times where values returned from nb_dns_activity being less than 0 would represent a fatal error? Offhand I can't think of any unless that same code path is used when doing remote Bro connections. Any other time (I think) it would represent some sort of analysis-based lookup in which I would rather Bro keeps running than completely failing. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Tue May 24 08:31:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 15:31:34 -0000 Subject: [Bro-Dev] #351: Incorrect bounds checking with truncated TCP options In-Reply-To: <048.13f9524c095200c055a4d9dc5ef882b7@tracker.bro-ids.org> References: <048.13f9524c095200c055a4d9dc5ef882b7@tracker.bro-ids.org> Message-ID: <063.248a96255d163dccb60448233d8f9d5e@tracker.bro-ids.org> #351: Incorrect bounds checking with truncated TCP options ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: Pushing this ticket back. -- Ticket URL: Bro Tracker Bro Issue Tracker From vern at icir.org Tue May 24 09:33:33 2011 From: vern at icir.org (Vern Paxson) Date: Tue, 24 May 2011 09:33:33 -0700 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: (Tue, 24 May 2011 11:28:20 EDT). Message-ID: <20110524163333.BE6FD36A37A@taffy.ICSI.Berkeley.EDU> > I suppose the more direct question is, are there any times where values returned from nb_dns_activity being less than 0 would represent a fatal error? IIRC, it's used by Bro at startup to resolve hostnames in the policy scripts. If those fail to resolve due to a serious problem (rather than just the name not existing), then arguably Bro is about to run with fundamentally incorrect/missing information, which is not very safe. That said, whether it should bomb out under such circumstances is still debatable. Vern From seth at icir.org Tue May 24 10:00:20 2011 From: seth at icir.org (Seth Hall) Date: Tue, 24 May 2011 13:00:20 -0400 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <20110524163333.BE6FD36A37A@taffy.ICSI.Berkeley.EDU> References: <20110524163333.BE6FD36A37A@taffy.ICSI.Berkeley.EDU> Message-ID: <7079C654-E5E9-4D08-8BB0-5CCB85C7924F@icir.org> On May 24, 2011, at 12:33 PM, Vern Paxson wrote: > IIRC, it's used by Bro at startup to resolve hostnames in the policy > scripts. If those fail to resolve due to a serious problem (rather than > just the name not existing), then arguably Bro is about to run with > fundamentally incorrect/missing information, which is not very safe. Agreed, but I would consider it a fairly minimal risk due to such extremely limited use of that feature anyway. In the scripts I've been working on, I haven't even used it at all so the risk is even lower. > That said, whether it should bomb out under such circumstances is > still debatable. I agree. If I get a chance soon I'll commit a change to fastpath changing those to runtime warnings instead of internal errors. Unless... Jon, would you like to do it? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From vern at icir.org Tue May 24 10:09:58 2011 From: vern at icir.org (Vern Paxson) Date: Tue, 24 May 2011 10:09:58 -0700 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <7079C654-E5E9-4D08-8BB0-5CCB85C7924F@icir.org> (Tue, 24 May 2011 13:00:20 EDT). Message-ID: <20110524170958.DEF7436A37C@taffy.ICSI.Berkeley.EDU> > > IIRC, it's used by Bro at startup to resolve hostnames in the policy > > scripts ... > > > Agreed, but I would consider it a fairly minimal risk due to such extremely > limited use of that feature anyway. In the scripts I've been working on, > I haven't even used it at all so the risk is even lower. ? You don't have things like tables of sensitive systems that are listed using hostnames rather than addresses? This strikes me as a pretty basic use case. Vern From jsiwek at ncsa.illinois.edu Tue May 24 10:27:47 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 24 May 2011 12:27:47 -0500 (CDT) Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <7079C654-E5E9-4D08-8BB0-5CCB85C7924F@icir.org> Message-ID: <1001133857.2343.1306258067104.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > I agree. If I get a chance soon I'll commit a change to fastpath > changing those to runtime warnings instead of internal errors. > Unless... Jon, would you like to do it? Maybe start a new ticket for this and continue any discussion there; the DNS-related issues in this thread are separate from what I was originally asking about. Then we'll just see who claims ownership of the ticket first. - Jon From bro at tracker.bro-ids.org Tue May 24 10:27:59 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 17:27:59 -0000 Subject: [Bro-Dev] #323: State of passive fingerprinting signatures? In-Reply-To: <047.9bd93768b3e416ec419d19ee657027c9@tracker.bro-ids.org> References: <047.9bd93768b3e416ec419d19ee657027c9@tracker.bro-ids.org> Message-ID: <062.d17f7a33d3c2a02debd7280ed9bdff20@tracker.bro-ids.org> #323: State of passive fingerprinting signatures? ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by seth): * milestone: Bro1.6 => Comment: It looks like p0f hasn't been updated since 2006 and the signatures are starting to get quite out of date. I'm thinking we should just remove any milestones from this ticket and come back to it later. It may be that ultimately the p0f code is just removed, it's really too unreliable to be of much use to anyone now in my opinion but it could be improved with SYN/ACK and RST modes but the database would still need to be updated and expanded by someone taking over the p0f project still. I'm going to remove the milestone from this ticket, but anyone should feel free to put it back if you plan to address p0f in Bro. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue May 24 10:49:10 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 17:49:10 -0000 Subject: [Bro-Dev] #461: internal_error called for async DNS failures Message-ID: <046.adc7e841ed311111a107267ffb4a852b@tracker.bro-ids.org> #461: internal_error called for async DNS failures ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- Bro shouldn't be shutting down when DNS lookups fail which is what internal_error leads to. It seems that the main area people are having problems is the handling of the return value of nb_dns_activity, but I think that all internal_error calls in DNS_Mgr.cc should be audited. Maybe for the next release we can change the places where internal_error is called based on the nb_dns_activity return value and audit the rest of the internal_error calls later. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue May 24 10:51:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 17:51:14 -0000 Subject: [Bro-Dev] #461: internal_error called for async DNS failures In-Reply-To: <046.adc7e841ed311111a107267ffb4a852b@tracker.bro-ids.org> References: <046.adc7e841ed311111a107267ffb4a852b@tracker.bro-ids.org> Message-ID: <061.e3bd3e3f5d0aca2c489ad3bf92b263f5@tracker.bro-ids.org> #461: internal_error called for async DNS failures ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): An email relevant to the discussion from Vern... {{{ IIRC, it's used by Bro at startup to resolve hostnames in the policy scripts. If those fail to resolve due to a serious problem (rather than just the name not existing), then arguably Bro is about to run with fundamentally incorrect/missing information, which is not very safe. That said, whether it should bomb out under such circumstances is still debatable. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Tue May 24 10:51:38 2011 From: seth at icir.org (Seth Hall) Date: Tue, 24 May 2011 13:51:38 -0400 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <1001133857.2343.1306258067104.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <1001133857.2343.1306258067104.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: On May 24, 2011, at 1:27 PM, Jonathan Siwek wrote: > Maybe start a new ticket for this and continue any discussion there; the DNS-related issues in this thread are separate from what I was originally asking about. Then we'll just see who claims ownership of the ticket first. Good point. Done. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jmellander at lbl.gov Tue May 24 12:11:32 2011 From: jmellander at lbl.gov (Jim Mellander) Date: Tue, 24 May 2011 11:11:32 -0800 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: <7079C654-E5E9-4D08-8BB0-5CCB85C7924F@icir.org> References: <20110524163333.BE6FD36A37A@taffy.ICSI.Berkeley.EDU> <7079C654-E5E9-4D08-8BB0-5CCB85C7924F@icir.org> Message-ID: On Tue, May 24, 2011 at 9:00 AM, Seth Hall wrote: > > On May 24, 2011, at 12:33 PM, Vern Paxson wrote: > >> IIRC, it's used by Bro at startup to resolve hostnames in the policy >> scripts. ?If those fail to resolve due to a serious problem (rather than >> just the name not existing), then arguably Bro is about to run with >> fundamentally incorrect/missing information, which is not very safe. > > Agreed, but I would consider it a fairly minimal risk due to such extremely limited use of that feature anyway. ?In the scripts I've been working on, I haven't even used it at all so the risk is even lower. How about triggering an event on such failure conditions (perhaps sending the hostname)? - each site could then determine globally or by hostname how to deal with this situation > >> That said, whether it should bomb out under such circumstances is >> still debatable. > > > I agree. ?If I get a chance soon I'll commit a change to fastpath changing those to runtime warnings instead of internal errors. ?Unless... Jon, would you like to do it? > > ?.Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From seth at icir.org Tue May 24 12:43:57 2011 From: seth at icir.org (Seth Hall) Date: Tue, 24 May 2011 15:43:57 -0400 Subject: [Bro-Dev] BiF parsing index types In-Reply-To: References: <20110524163333.BE6FD36A37A@taffy.ICSI.Berkeley.EDU> <7079C654-E5E9-4D08-8BB0-5CCB85C7924F@icir.org> Message-ID: <6D798A7A-2ADB-4352-82F7-9716860C1AAA@icir.org> On May 24, 2011, at 3:11 PM, Jim Mellander wrote: > How about triggering an event on such failure conditions (perhaps > sending the hostname)? - each site could then determine globally or by > hostname how to deal with this situation I think you may lose too much context about the initial lookup by that point. If it was a statically defined hostname in a script (that was really important) and it failed, what would you decide to do? You'd have to know the variable that it was supposed to be assigned to after being looked up if you wanted to try the lookup again. I don't think you'd ever want to kill Bro, but you'd probably want it logged which would already be taken care of by the runtime error. Perhaps you're right though in that if there is some *really* important domain name, you would have some recourse to make sure that name is looked up. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Tue May 24 13:33:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 20:33:53 -0000 Subject: [Bro-Dev] #462: topic/seth/ssl-binpac - core SSL/TLS analyzer work Message-ID: <046.33fc53d356c9729c83a221d2d154c213@tracker.bro-ids.org> #462: topic/seth/ssl-binpac - core SSL/TLS analyzer work ---------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------------+-------------------- This is the core analyzer work minus the scripts which will be developed in the policy-scripts-new branch. * The core events have been reworked. Less state is kept in the analyzer itself so that more can be done in policy scripts. * All certificates (including chained) are passed through to script-land as DER certificates and they are just strings in Bro. * A new BiF for doing certificate validation. * NotValidBefore and NotValidAfter dates extracted from certificates. * Lots of simplification to the core analyzer. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue May 24 13:34:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 20:34:40 -0000 Subject: [Bro-Dev] #329: Optimizing detect-protocols-http.bro (was: Optimizing detect-protocols.bro) In-Reply-To: <046.fc3e396e80d9cd99cc21e00aef84ff98@tracker.bro-ids.org> References: <046.fc3e396e80d9cd99cc21e00aef84ff98@tracker.bro-ids.org> Message-ID: <061.e96904e1afc4a725c72cd2d895ad14c4@tracker.bro-ids.org> #329: Optimizing detect-protocols-http.bro ---------------------+---------------------- Reporter: seth | Owner: Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: sprint ---------------------+---------------------- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue May 24 13:36:08 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 20:36:08 -0000 Subject: [Bro-Dev] #367: internal_error with &optional fields in records used as indexes In-Reply-To: <046.336d40831a3ce7c91fa8b8eed0b0363d@tracker.bro-ids.org> References: <046.336d40831a3ce7c91fa8b8eed0b0363d@tracker.bro-ids.org> Message-ID: <061.e028d3ea76978e8a74fd95c6a6cb59d4@tracker.bro-ids.org> #367: internal_error with &optional fields in records used as indexes -----------------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: logging -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: I'm going to close this ticket since the problem brought up initially is fixed. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue May 24 13:37:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 20:37:34 -0000 Subject: [Bro-Dev] #424: Loading sub-pathed bro script at command line In-Reply-To: <046.910986b9f9a32d99b3a7ca6cfac21f9a@tracker.bro-ids.org> References: <046.910986b9f9a32d99b3a7ca6cfac21f9a@tracker.bro-ids.org> Message-ID: <061.0a9a5561e62182a67a954d480587fad5@tracker.bro-ids.org> #424: Loading sub-pathed bro script at command line ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Invalid | Keywords: ----------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Invalid Comment: Never mind this does seem to work. I don't know what was going on before. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue May 24 13:38:41 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 24 May 2011 20:38:41 -0000 Subject: [Bro-Dev] #369: Expand and document local-*.bro scripts In-Reply-To: <046.2853dd4ad8ffa3316596d9d8109b14cb@tracker.bro-ids.org> References: <046.2853dd4ad8ffa3316596d9d8109b14cb@tracker.bro-ids.org> Message-ID: <061.06b4f94e606cefe5cb15109c9bed8eae@tracker.bro-ids.org> #369: Expand and document local-*.bro scripts -------------------------+-------------------- Reporter: seth | Owner: seth Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: Invalid | Keywords: -------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Invalid Comment: This ticket doesn't really serve and purpose at this point. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed May 25 04:42:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 25 May 2011 11:42:27 -0000 Subject: [Bro-Dev] #395: Cannot add global to module if it already exists in global namespace In-Reply-To: <048.ed7c07f7b808821a854fad6bf4ed9504@tracker.bro-ids.org> References: <048.ed7c07f7b808821a854fad6bf4ed9504@tracker.bro-ids.org> Message-ID: <063.44f9b927d8f34f11344a986cc03e13b8@tracker.bro-ids.org> #395: Cannot add global to module if it already exists in global namespace ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): This works for some reason now, but I don't seem to have access to the 'a' in the global namespace. GLOBAL::a doesn't even work. Gregor, could you close this ticket if it works now for what you needed it for? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed May 25 04:43:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 25 May 2011 11:43:56 -0000 Subject: [Bro-Dev] #427: Assertion failed: (!v), function Eval, file Trigger.cc In-Reply-To: <047.734f3fa34279853408b9e9abca37a2b4@tracker.bro-ids.org> References: <047.734f3fa34279853408b9e9abca37a2b4@tracker.bro-ids.org> Message-ID: <062.c8ca20c2bd67799c1074d0af342c6a21@tracker.bro-ids.org> #427: Assertion failed: (!v), function Eval, file Trigger.cc ----------------------+-------------------- Reporter: leres | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: 1.5.2 Resolution: | Keywords: ipv6 ----------------------+-------------------- Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: I'm pushing this ticket back to 1.7 so that we can include it in the more broad check over the IPv6 support. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed May 25 04:45:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 25 May 2011 11:45:24 -0000 Subject: [Bro-Dev] #263: --enable-int64 breaks bropipe In-Reply-To: <057.3ac8049e9b827637001887156ee2a0e6@tracker.bro-ids.org> References: <057.3ac8049e9b827637001887156ee2a0e6@tracker.bro-ids.org> Message-ID: <072.578acb36f363d3bf6e358b00c52e8cfc@tracker.bro-ids.org> #263: --enable-int64 breaks bropipe -----------------------------+----------------------- Reporter: ssakai@? | Owner: christian Type: defect | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: 1.5.1 Resolution: Solved/Applied | Keywords: inttypes -----------------------------+----------------------- Changes (by seth): * status: assigned => closed * resolution: => Solved/Applied Comment: I'm going to close this ticket since the problem it brings up is fixed. We should remember the pybroccoli problem since a test is failing. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed May 25 06:07:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 25 May 2011 13:07:36 -0000 Subject: [Bro-Dev] #378: Test for problem with optional record fields. In-Reply-To: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> References: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> Message-ID: <062.58703da08915ec0ac0ccdc98b5bd6286@tracker.bro-ids.org> #378: Test for problem with optional record fields. --------------------------------+-------------------- Reporter: robin | Owner: Type: Test Case Missing | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: --------------------------------+-------------------- Old description: > A test script for the problem in #367, to be added to the now test-suite > once we have it. New description: A test script for the problem in #367, to be added to the now test-suite once we have it. -- Comment (by seth): This test script is crashing Bro. {{{ [build (topic/policy-scripts-new)]$ gdb --args ./src/bro test28.bro GNU gdb 6.3.50-20050815 (Apple version gdb-1469) (Wed May 5 04:36:56 UTC 2010) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared libraries ........ done (gdb) r Starting program: ./src/bro test28.bro Reading symbols for shared libraries .+++++++ done { [a=3, b=], 1306328728.271705 internal error: under-ran key in CompositeHash::DescribeKey -8 [a=1, b=] Program received signal SIGABRT, Aborted. 0x00007fff889a85d6 in __kill () (gdb) bt #0 0x00007fff889a85d6 in __kill () #1 0x00007fff88a48cd6 in abort () #2 0x0000000100004703 in internal_error (fmt=Could not find the frame base for "internal_error(char const*, ...)". ) at ./src/util.cc:810 #3 0x00000001000bf799 in CompositeHash::RecoverVals (this=0x100cb9d20, k=0x100cb9bf0) at ./CompHash.cc:525 #4 0x00000001002309d6 in TableVal::Describe (this=0x100cb9e60, d=0x7fff5fbfe080) at ./src/Val.cc:2321 #5 0x000000010022edb1 in describe_vals (vals=0x100cb9900, d=0x7fff5fbfe080, offset=0) at ./src/Val.cc:3574 #6 0x000000010020b27d in ExprListStmt::PrintVals (this=0x100cb42c0, d=0x7fff5fbfe080, vals=0x100cb9900, offset=0) at /./src/Stmt.cc:226 #7 0x000000010020b48d in PrintStmt::DoExec (this=0x100cb42c0, vals=0x100cb9900) at ./src/Stmt.cc:328 #8 0x0000000100208c4c in ExprListStmt::Exec (this=0x100cb42c0, f=0x100cb9940, flow=@0x7fff5fbfe368) at ./src/Stmt.cc:197 #9 0x0000000100206d68 in StmtList::Exec (this=0x100cb0720, f=0x100cb9940, flow=@0x7fff5fbfe368) at ./src/Stmt.cc:1432 #10 0x0000000100206d68 in StmtList::Exec (this=0x100cb7c70, f=0x100cb9940, flow=@0x7fff5fbfe368) at ./src/Stmt.cc:1432 #11 0x00000001001585a2 in BroFunc::Call (this=0x100be4630, args=0x100cb9710, parent=0x0) at ./src/Func.cc:319 #12 0x00000001000fc6c4 in EventHandler::Call (this=0x100b1cfc0, vl=0x100cb9710, no_remote=false) at ./src/EventHandler.cc:72 #13 0x0000000100097455 in Event::Dispatch (this=0x100cb97c0, no_remote=false) at Event.h:43 #14 0x00000001000fbfe5 in EventMgr::Dispatch (this=0x1003ee1e0) at ./src/Event.cc:107 #15 0x00000001000fc055 in EventMgr::Drain (this=0x1003ee1e0) at ./src/Event.cc:119 #16 0x00000001000958ac in main (argc=2, argv=0x7fff5fbfeba0) at ./src/main.cc:974 (gdb) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed May 25 10:37:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 25 May 2011 17:37:43 -0000 Subject: [Bro-Dev] #447: Bro snaplen In-Reply-To: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> References: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> Message-ID: <061.60a3e41deae302be61e14e4c0d047052@tracker.bro-ids.org> #447: Bro snaplen ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): This is addressed in the fastpath branch. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed May 25 10:39:39 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 25 May 2011 17:39:39 -0000 Subject: [Bro-Dev] #463: Create a test for large packet support Message-ID: <046.16bb51ffe351d96ce1c263c8ade77e2b@tracker.bro-ids.org> #463: Create a test for large packet support -------------------------------+----------------- Reporter: seth | Owner: Type: Test Case Missing | Status: new Priority: Low | Milestone: Component: Bro | Version: Keywords: | -------------------------------+----------------- This requires that we find a tracefile with large packets to verify that we can successfully process them. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed May 25 13:06:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 25 May 2011 20:06:32 -0000 Subject: [Bro-Dev] #442: Hardcode Python path In-Reply-To: <047.c6380b050af26fb864dd8700d0e0521b@tracker.bro-ids.org> References: <047.c6380b050af26fb864dd8700d0e0521b@tracker.bro-ids.org> Message-ID: <062.0ca85ea19462ed96552879b570c41b25@tracker.bro-ids.org> #442: Hardcode Python path -------------------------+-------------------- Reporter: robin | Owner: jsiwek Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: -------------------------+-------------------- Comment (by jsiwek): Here's Craig's response that convinced me of taking the approach he proposed: {{{ >> BroControl currently uses `#! /usr/bin/env` to find the Python >> binary at runtime. Craig suggested this instead: > > Craig, can you explain more about what the problem was in this ticket? > I didn't understand how configuring the python shebang to a full path at build time was better than using the env shebang. If you use env, you'll pick up whatever version of python is first on the path instead of the version that was first on the path when broctl was built. This will be different for different users and can result in unexpected results and possible brokenness. The specific python binary path should be considered part of the broctl config so that the package builder is able to control it by setting the path when building broctl. Also, the end user only as to have broctl on his path to be able to run it. For example on FreeBSD, /usr/local/bin is not on the default path; the default path is used at bootup so if broctl doesn't have the path built in, it can't find python when it's run from a rc.d script. You could add /usr/local/bin to the path but that won't be right 100% of the time for all installations. > If an absolute path to a python interpreter is set in the shebang > at configure/build time, then to change the python interpreter that > is used, the user has to either (1) edit the script(s) or (2) > re-configure/build/install broctl. These seem like the more > "difficult" options to me. (How often does this actually happen?) Editing the scripts seems wrong; if you later rebuild (say to install a newer version) your changes will get over written. The binary executable is part of the broctl configuration so having to re-configure/build/install broctl to use a different python sounds completely reasonable to me. From my perspective, the inconvenience of rebuilding broctl seems minor compared to having different users picking up different versions of python. Finally, some folks consider it a security issue to use #!/usr/bin/env }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed May 25 13:15:39 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 25 May 2011 20:15:39 -0000 Subject: [Bro-Dev] #442: Hardcode Python path In-Reply-To: <047.c6380b050af26fb864dd8700d0e0521b@tracker.bro-ids.org> References: <047.c6380b050af26fb864dd8700d0e0521b@tracker.bro-ids.org> Message-ID: <062.ec6a4697a0b36814e6e47f8306a34ad4@tracker.bro-ids.org> #442: Hardcode Python path ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * version: => git/master * type: Problem => Merge Request Comment: I took this a step further than just doing broctl and tried to standardize all use of "#!/usr/bin/env " in installed shell scripts that may be executed by users. This is implemented in the `topic/jsiwek/abs-interp- path` branch in the broctl, trace-summary, and bro-aux repositories. A caveat is that the hashbangs are left alone when configuring in binary packaging mode (the `--binary-package` option to configure) for creating DEBs, RPMs, Mac packages. This is because we currently intend to build binary packages on NMI platforms which won't (can't) always have Python installed in a standard location for a given distro. -- Ticket URL: Bro Tracker Bro Issue Tracker From vern at icir.org Sun May 29 18:46:31 2011 From: vern at icir.org (Vern Paxson) Date: Sun, 29 May 2011 18:46:31 -0700 Subject: [Bro-Dev] metrics framework In-Reply-To: <043DCD15-8AB6-4C69-87E9-C35E126D743C@icir.org> (Mon, 23 May 2011 03:09:10 EDT). Message-ID: <20110530014631.B3D8536A035@taffy.ICSI.Berkeley.EDU> > I'm not completely sure that would apply because the only reduce operation > that I'm currently envisioning is straight addition. Right, straight addition is very likely a predominant case. However, I was considering that with the ability to apply a reduce operation, we can support other computations, too. A simple example would be computing maxima/minima. Vern From seth at icir.org Tue May 31 07:11:19 2011 From: seth at icir.org (Seth Hall) Date: Tue, 31 May 2011 10:11:19 -0400 Subject: [Bro-Dev] metrics framework In-Reply-To: <20110530014631.B3D8536A035@taffy.ICSI.Berkeley.EDU> References: <20110530014631.B3D8536A035@taffy.ICSI.Berkeley.EDU> Message-ID: On May 29, 2011, at 9:46 PM, Vern Paxson wrote: >> A simple example would be computing > maxima/minima. Ahhh, good point. I'll have to think about this a bit more. Related to this, do you have any thoughts on syntax for being able to dynamically call functions at runtime? If we had this capability a number of things would be easier. In this case for instance, we could store the reduce operations as anonymous functions call them dynamically when they're needed. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Tue May 31 11:50:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 31 May 2011 18:50:37 -0000 Subject: [Bro-Dev] #464: Bug with complex data structures Message-ID: <046.9f65dde0f285b762925aca06fe80721f@tracker.bro-ids.org> #464: Bug with complex data structures ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- This script: {{{ type MetaData: record { a: count; tags: set[string]; }; global ip_data: table[addr] of set[MetaData] = table(); local m: MetaData = [$a=4, $tags=set("blah")]; if ( 1.2.3.4 !in ip_data ) ip_data[1.2.3.4] = set(m); else add ip_data[1.2.3.4][m]; }}} Results in this error: {{{ 1306867820.149493 and ./test29.bro, line 10 ([a=4, tags={blah}] and list of record { a:count; tags:set[string]; }): error, index type doesn't match table }}} -- Ticket URL: Bro Tracker Bro Issue Tracker