From bro at tracker.bro-ids.org Wed Jun 1 11:36:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 01 Jun 2011 18:36:12 -0000 Subject: [Bro-Dev] #442: Hardcode Python path In-Reply-To: <047.c6380b050af26fb864dd8700d0e0521b@tracker.bro-ids.org> References: <047.c6380b050af26fb864dd8700d0e0521b@tracker.bro-ids.org> Message-ID: <062.1ef3174f59ea7db9867ac07d0e6e57ec@tracker.bro-ids.org> #442: Hardcode Python path -----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 1 11:38:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 01 Jun 2011 18:38:38 -0000 Subject: [Bro-Dev] #465: Fix up the MIME analyzer Message-ID: <046.cab68644f8f2eb3e5bb44bf373b6902d@tracker.bro-ids.org> #465: Fix up the MIME analyzer ---------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ The mime analyzer has a lot of inconsistency issues and is broken in a few places. * mime_all_headers loops and could potentially be a bad idea. More prone to DoS as well. Delete it? * mime_all_data is probably also a bad idea. Especially for large files. Delete it? * mime_entity_data seems very similar to mime_all_data and is not chunked as the similarity to the http_entity_data would imply. The current mime_entity_data should be removed and the current mime_all_data should be renamed to mime_entity_data. * mime_next_entity is never generated by the core or policy scripts and should either be fixed or deleted. * mime_one_header should probably be renamed to mime_header for consistency. * I have no clue what mime_event is for. Is it necessary? * mime_content_hash gives a non printable hash value and it could be removed since hash generation is done in the script now and eventually will be done in the file analyzer. * The wrong ifdef is used in the source: #ifdef DEBUG_BRO used instead of #ifdef DEBUG * mime_end_entity is generated generated multiple times in some cases when it shouldn't be. It's something to keep an eye out for, I never dug into it enough to find out what caused it. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Wed Jun 1 12:08:22 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 1 Jun 2011 12:08:22 -0700 Subject: [Bro-Dev] Loading directories Message-ID: <20110601190822.GB67978@icir.org> For policy files, we now have sub-directory structures like this: foo/base.bro foo/ext1.bro foo/ext2.bro foo.bro foo.bro contains @load foo/base.bro @load foo/ext1.bro @load foo/ext2.bro And the effect is that "@load foo" pulls in all of the scripts at once, which is neat. What I'm wondering hiwer is whether we should move "foo.bro" into the subdirectory under some magic name (like init.bro or __init__.bro or __package__.bro or ...). And then allow to write "@load foo" (with "foo" now being the directory name) with the effect of loading "foo/init.bro". In addition, inside foo/* we could allow "@load ext1.bro" instead of "@load foo/ext1.bro". (Coincidentally, this would be quite like Python does it ... :) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Wed Jun 1 12:22:33 2011 From: seth at icir.org (Seth Hall) Date: Wed, 1 Jun 2011 15:22:33 -0400 Subject: [Bro-Dev] Loading directories In-Reply-To: <20110601190822.GB67978@icir.org> References: <20110601190822.GB67978@icir.org> Message-ID: <3568225F-788B-4154-93DC-7C3C4CB1F6C5@icir.org> On Jun 1, 2011, at 3:08 PM, Robin Sommer wrote: > What I'm wondering hiwer is whether we should move "foo.bro" into the > subdirectory under some magic name (like init.bro or __init__.bro or > __package__.bro or ...) That makes a lot of sense to me. I've been thinking the same thing for a while, those wrapper scripts really litter things up for not much benefit. > In addition, inside foo/* we could allow "@load > ext1.bro" instead of "@load foo/ext1.bro". Hm, I'm not totally sure I like this. It could get confusing when two scripts have the same name. If you want to load the software framework and you are in http/init.bro how do you do it if there is a file named http/software.bro? > (Coincidentally, this would be quite like Python does it ... :) I can go for copying syntax and idioms to make the language feel a bit more familiar and comfortable. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Wed Jun 1 12:51:22 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 1 Jun 2011 12:51:22 -0700 Subject: [Bro-Dev] Loading directories In-Reply-To: <3568225F-788B-4154-93DC-7C3C4CB1F6C5@icir.org> References: <20110601190822.GB67978@icir.org> <3568225F-788B-4154-93DC-7C3C4CB1F6C5@icir.org> Message-ID: <20110601195122.GC67978@icir.org> On Wed, Jun 01, 2011 at 15:22 -0400, you wrote: > scripts have the same name. If you want to load the software > framework and you are in http/init.bro how do you do it if there is a > file named http/software.bro? Good question. :) Hmm... Then how about just using "@load ./ext1.bro" inside the directory. That should already work and also avoid the need to repeat the directory name. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Wed Jun 1 12:54:50 2011 From: seth at icir.org (Seth Hall) Date: Wed, 1 Jun 2011 15:54:50 -0400 Subject: [Bro-Dev] Loading directories In-Reply-To: <20110601195122.GC67978@icir.org> References: <20110601190822.GB67978@icir.org> <3568225F-788B-4154-93DC-7C3C4CB1F6C5@icir.org> <20110601195122.GC67978@icir.org> Message-ID: <72629922-F7F5-481C-83C2-C7A7C3B87A66@icir.org> On Jun 1, 2011, at 3:51 PM, Robin Sommer wrote: > Good question. :) Hmm... Then how about just using "@load ./ext1.bro" > inside the directory. That should already work and also avoid the need > to repeat the directory name. That sounds better to me. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Wed Jun 1 13:43:37 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 1 Jun 2011 15:43:37 -0500 (CDT) Subject: [Bro-Dev] Loading directories In-Reply-To: <20110601190822.GB67978@icir.org> Message-ID: > What I'm wondering hiwer is whether we should move "foo.bro" into the > subdirectory under some magic name (like init.bro or __init__.bro or > __package__.bro or ...). And then allow to write "@load foo" (with > "foo" now being the directory name) with the effect of loading > "foo/init.bro". I think that makes sense to me. A minor detail would be choosing which gets preference: a file or a directory. E.g. say you had a "foo.bro" file and a "foo/" directory that live on the same path, what does "@load foo" actually load? > In addition, inside foo/* we could allow "@load ext1.bro" instead of "@load foo/ext1.bro". > (Coincidentally, this would be quite like Python does it ... :) Initially only seemed to me like a minor convenience is gained from this, but I can't really argue against making things more analogous to an existing language. - Jon From vern at icir.org Wed Jun 1 14:26:52 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 01 Jun 2011 14:26:52 -0700 Subject: [Bro-Dev] Loading directories In-Reply-To: <20110601190822.GB67978@icir.org> (Wed, 01 Jun 2011 12:08:22 PDT). Message-ID: <20110601212652.743A136A38F@taffy.ICSI.Berkeley.EDU> > What I'm wondering hiwer is whether we should move "foo.bro" into the > subdirectory under some magic name (like init.bro or __init__.bro or > __package__.bro or ...) How about just foo/foo.bro? That would avoid problem of referring to the wrong instance of the script due to confusion regarding what directory one is working out of. BTW, if foo.bro itself pulls in ext1.bro etc., then what if the user doesn't *want* the ext1.bro functionality? Or is the model that they have to use it, and it's just split out into another script for keeping each script bite-sized? Vern From jsiwek at ncsa.illinois.edu Wed Jun 1 14:44:36 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 1 Jun 2011 16:44:36 -0500 (CDT) Subject: [Bro-Dev] Loading directories In-Reply-To: <20110601212652.743A136A38F@taffy.ICSI.Berkeley.EDU> Message-ID: <957b2fcf-e5e3-4148-b7b4-fc08b660c8ed@tangent.ncsa.illinois.edu> > BTW, if foo.bro itself pulls in ext1.bro etc., then what if the user > doesn't *want* the ext1.bro functionality? Or is the model that they have to > use it, and it's just split out into another script for keeping each > script bite-sized? I remember seeing that the @unload directive exists and looks like it's supposed to prevent future @load's of a given script. Don't know if it works, but that could be a way for a user to blacklist the loading of certain scripts in a package. - Jon From robin at icir.org Wed Jun 1 15:51:13 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 1 Jun 2011 15:51:13 -0700 Subject: [Bro-Dev] Loading directories In-Reply-To: <20110601212652.743A136A38F@taffy.ICSI.Berkeley.EDU> References: <20110601190822.GB67978@icir.org> <20110601212652.743A136A38F@taffy.ICSI.Berkeley.EDU> Message-ID: <20110601225113.GC30004@icir.org> On Wed, Jun 01, 2011 at 14:26 -0700, you wrote: > How about just foo/foo.bro? Hmm ... I think I'd prefer some visual indicator that the script is special, i.e., it gets loaded when the directory is specified. Having a consistent name for that would point that out. > BTW, if foo.bro itself pulls in ext1.bro etc., then what if the user doesn't > *want* the ext1.bro functionality? The model is that by "@load foo" he gets the full set. If he doesn't want that, he can still pick his pieces individually ("@load foo/base.bro; @load foo/ext2.bro"). The @unload may do it too, but I don't like that statement. :) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Wed Jun 1 16:51:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 01 Jun 2011 23:51:30 -0000 Subject: [Bro-Dev] #464: Bug with complex data structures In-Reply-To: <046.9f65dde0f285b762925aca06fe80721f@tracker.bro-ids.org> References: <046.9f65dde0f285b762925aca06fe80721f@tracker.bro-ids.org> Message-ID: <061.daa6674c97c092505f4b5f33d8a7c0e9@tracker.bro-ids.org> #464: Bug with complex data structures ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by robin): I'm actually getting a different error: `1306972010.461281 internal error: bad index type in CompositeHash::CompositeHash` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 1 16:59:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 01 Jun 2011 23:59:42 -0000 Subject: [Bro-Dev] #464: Bug with complex data structures In-Reply-To: <046.9f65dde0f285b762925aca06fe80721f@tracker.bro-ids.org> References: <046.9f65dde0f285b762925aca06fe80721f@tracker.bro-ids.org> Message-ID: <061.725499df366d56d0f4749d1d302a4c39@tracker.bro-ids.org> #464: Bug with complex data structures ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by robin): Turns out records that have more complex fields can't be used as indices for sets. I think we could add that, but it would be a bit of work. How important is it? Can you use a table[addr] of vector, where the vector then stores the MetaData? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 1 22:35:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 02 Jun 2011 05:35:40 -0000 Subject: [Bro-Dev] #450: Problem with newline character in logging framework text output plugin In-Reply-To: <046.2981651bfe949c46cd608c64b7fd7960@tracker.bro-ids.org> References: <046.2981651bfe949c46cd608c64b7fd7960@tracker.bro-ids.org> Message-ID: <061.03be2b6ab90d8687a6b94cd8cba7a47b@tracker.bro-ids.org> #450: Problem with newline character in logging framework text output plugin -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 1 22:37:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 02 Jun 2011 05:37:45 -0000 Subject: [Bro-Dev] #301: Switch to binary logging In-Reply-To: <046.26b2c9b3fb5b3c5022b521115d956856@tracker.bro-ids.org> References: <046.26b2c9b3fb5b3c5022b521115d956856@tracker.bro-ids.org> Message-ID: <061.f4403a982abf7a0358c6c56658bec395@tracker.bro-ids.org> #301: Switch to binary logging ------------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Changes (by robin): * milestone: Bro1.6 => Bro1.7 -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 1 22:39:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 02 Jun 2011 05:39:13 -0000 Subject: [Bro-Dev] #447: Bro snaplen In-Reply-To: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> References: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> Message-ID: <061.6428db79a49219be6a884796a75055f3@tracker.bro-ids.org> #447: Bro snaplen ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by robin): Reverted, as it seems that doesn't generally work. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jun 2 03:51:04 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 02 Jun 2011 10:51:04 -0000 Subject: [Bro-Dev] #464: Bug with complex data structures In-Reply-To: <046.9f65dde0f285b762925aca06fe80721f@tracker.bro-ids.org> References: <046.9f65dde0f285b762925aca06fe80721f@tracker.bro-ids.org> Message-ID: <061.14ac98b758a84fe37e7109b3f54a9fb4@tracker.bro-ids.org> #464: Bug with complex data structures ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): > I think we could add that, but it would be a bit of work. How important is > it? Can you use a table[addr] of vector, where the vector then stores the > MetaData? Ah, I see what you are saying. Yes, I should be able to do that. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jun 2 03:51:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 02 Jun 2011 10:51:50 -0000 Subject: [Bro-Dev] #464: Bug with complex data structures In-Reply-To: <046.9f65dde0f285b762925aca06fe80721f@tracker.bro-ids.org> References: <046.9f65dde0f285b762925aca06fe80721f@tracker.bro-ids.org> Message-ID: <061.50457266b49802a95408658756cd14b2@tracker.bro-ids.org> #464: Bug with complex data structures ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): > I'm actually getting a different error: `1306972010.461281 internal error: > bad index type in CompositeHash::CompositeHash` Yeah, I started making some changes in CompHash.cc and I forgot about it when I filed this ticket. The error you got is the one I meant to use for this ticket. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jun 2 07:33:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 02 Jun 2011 14:33:44 -0000 Subject: [Bro-Dev] #398: "make dist" requires cmake and swig In-Reply-To: <047.ea5da2efdb2169978f1d2dd3b5ae43f9@tracker.bro-ids.org> References: <047.ea5da2efdb2169978f1d2dd3b5ae43f9@tracker.bro-ids.org> Message-ID: <062.6585f36df9727e59dd7f77560f0b7c5b@tracker.bro-ids.org> #398: "make dist" requires cmake and swig ------------------------------+-------------------- Reporter: leres | Owner: jsiwek Type: Feature Request | Status: closed Priority: Normal | Milestone: Component: Bro | Version: Resolution: Solved/Applied | Keywords: ------------------------------+-------------------- Changes (by jsiwek): * status: assigned => closed * resolution: => Solved/Applied Comment: The top-level `make dist` target for creating source tarballs was rewritten as part of commit bf8480124980401052ee54f8121d6701ff043c3f, now on the master branch. It no longer requires CMake or the full dependency chain of Bro. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Thu Jun 2 07:45:46 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 2 Jun 2011 09:45:46 -0500 (CDT) Subject: [Bro-Dev] Loading directories In-Reply-To: <20110601225113.GC30004@icir.org> Message-ID: > > BTW, if foo.bro itself pulls in ext1.bro etc., then what if the > > user doesn't *want* the ext1.bro functionality? > > The model is that by "@load foo" he gets the full set. If he doesn't > want that, he can still pick his pieces individually ("@load > foo/base.bro; @load foo/ext2.bro"). > > The @unload may do it too, but I don't like that statement. :) If it's just the name of @unload (I'd agree the sound of it is misleading), it could be offered as a different command that does the same thing. I just think it's nice that a user can choose which approach is best for them: either a whitelist that picks individual pieces of a "package" or a blacklist that blocks only certain scripts of a "package". The blacklist approach allows a user to automatically start loading new scripts that get added to a package in future releases, but the whitelist must be manually updated in that situation. - Jon From robin at icir.org Thu Jun 2 09:22:32 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 2 Jun 2011 09:22:32 -0700 Subject: [Bro-Dev] Loading directories In-Reply-To: References: <20110601225113.GC30004@icir.org> Message-ID: <20110602162232.GH56087@icir.org> On Thu, Jun 02, 2011 at 09:45 -0500, you wrote: > If it's just the name of @unload (I'd agree the sound of it is > misleading), it could be offered as a different command that does the > same thing. The name is one piece, but it can also be confusing if one unloads a script that some other one depends on and then suddendly doesn't work as expected anymore (or not at all). And the order of @loads and @unloads matters, which isn't nice either. > I just think it's nice that a user can choose which approach is best > for them: either a whitelist that picks individual pieces of a > "package" or a blacklist that blocks only certain scripts of a > "package". That's right, I'm not saying nobody should use @unload, but I don't want it to be the only part of the story. Having both options is indeed helpful. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Thu Jun 2 10:14:23 2011 From: seth at icir.org (Seth Hall) Date: Thu, 2 Jun 2011 13:14:23 -0400 Subject: [Bro-Dev] Notices done as event instead of function Message-ID: <49DE6913-FDB5-4643-B802-7EB0426EBB96@icir.org> I was updating the new notice scripts based on feedback from Robin and I started to change the notice pathway to be event based instead of function based because it provided an easy way for people to extend the built in notice framework with their own functionality but then I noticed this comment... # This handler is useful for processing notices after the notice filters # have been applied and yielded an Notice::Action. # # It's tempting to make the default handler do the logging and # printing to notice_file, rather than NOTICE. I hesitate to do that, # though, because it perhaps could slow down notification, because # in the absence of event priorities, the event would have to wait # behind any other already-queued events. event notice_action(n: Notice::Info, action: Notice::Action) { } I think that doing the notification and printing through an event has a lot of benefits but I see the downside too. What does everyone else think? Especially whomever wrote that comment. :) An extra side thought too is that the current event priorities system is not the same as the priorities mentioned in the comment (i think). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Thu Jun 2 10:18:41 2011 From: seth at icir.org (Seth Hall) Date: Thu, 2 Jun 2011 13:18:41 -0400 Subject: [Bro-Dev] notice tags? Message-ID: Are notice tags used by anyone? They are the values that looks like this: "39-4b24-1" in the notice.log file. I'm not sure what the impetus for creating those was but they look to be a combination of timestamp derived value, process id, and incrementing counter. I've never had a use for them and wondered if it's worthwhile keeping them around. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From vern at icir.org Thu Jun 2 10:25:04 2011 From: vern at icir.org (Vern Paxson) Date: Thu, 02 Jun 2011 10:25:04 -0700 Subject: [Bro-Dev] notice tags? In-Reply-To: (Thu, 02 Jun 2011 13:18:41 EDT). Message-ID: <20110602172504.9CD3636A397@taffy.ICSI.Berkeley.EDU> > Are notice tags used by anyone? They are handy for grepping. The tag also appears in the conn file (as an $addl element), so you can link the two together. Vern From robin at icir.org Thu Jun 2 10:29:31 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 2 Jun 2011 10:29:31 -0700 Subject: [Bro-Dev] notice tags? In-Reply-To: <20110602172504.9CD3636A397@taffy.ICSI.Berkeley.EDU> References: <20110602172504.9CD3636A397@taffy.ICSI.Berkeley.EDU> Message-ID: <20110602172931.GK56087@icir.org> On Thu, Jun 02, 2011 at 10:25 -0700, you wrote: > They are handy for grepping. The tag also appears in the conn file > (as an $addl element), so you can link the two together. (... and the values don't have any further semantics, they are just supposed to be unique, pretty much like the new conn uids but per notice rather than per connection). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Thu Jun 2 10:33:33 2011 From: seth at icir.org (Seth Hall) Date: Thu, 2 Jun 2011 13:33:33 -0400 Subject: [Bro-Dev] notice tags? In-Reply-To: <20110602172504.9CD3636A397@taffy.ICSI.Berkeley.EDU> References: <20110602172504.9CD3636A397@taffy.ICSI.Berkeley.EDU> Message-ID: <6866A3DB-7A55-4E3F-A821-027A3C8D3D5C@icir.org> On Jun 2, 2011, at 1:25 PM, Vern Paxson wrote: >> Are notice tags used by anyone? > > They are handy for grepping. The tag also appears in the conn file > (as an $addl element), so you can link the two together. Heh, it actually doesn't right now. :) I'll add that back in (notice id in conn.log), I can see it being useful in some cases. With the record extension stuff, I think I'll be implementing it as a separate field just for notices and it will probably be implemented in the notice script itself. I never liked that $addl field, it was always hard to figure out what it was for. Should there be a separate and built in way to generate unique ids? What bothered me about the tag implementation in the notice script is that it's a bit difficult to figure what's going on due to dealing with non-determinism of the tags because of the data it bases the tags on. If we had a BiF that just generated unique IDs, we could build all of the determinism in there and wouldn't have to worry about it anymore where we need unique IDs. Thanks, .Seth From vern at icir.org Thu Jun 2 10:38:16 2011 From: vern at icir.org (Vern Paxson) Date: Thu, 02 Jun 2011 10:38:16 -0700 Subject: [Bro-Dev] notice tags? In-Reply-To: <6866A3DB-7A55-4E3F-A821-027A3C8D3D5C@icir.org> (Thu, 02 Jun 2011 13:33:33 EDT). Message-ID: <20110602173816.7084E36A398@taffy.ICSI.Berkeley.EDU> > Should there be a separate and built in way to generate unique ids? Nice thought. Vern From baxterw3232 at gmail.com Thu Jun 2 10:44:56 2011 From: baxterw3232 at gmail.com (Will) Date: Thu, 2 Jun 2011 13:44:56 -0400 Subject: [Bro-Dev] Notices done as event instead of function In-Reply-To: <49DE6913-FDB5-4643-B802-7EB0426EBB96@icir.org> References: <49DE6913-FDB5-4643-B802-7EB0426EBB96@icir.org> Message-ID: On Thu, Jun 2, 2011 at 1:14 PM, Seth Hall wrote: > I was updating the new notice scripts based on feedback from Robin and I started to change the notice pathway to be event based instead of function based because it provided an easy way for people to extend the built in notice framework with their own functionality but then I noticed this comment... Personally, I like where you are going with making it easier to extend the built in framework. There have been multiple occasions where I have wanted to generate multiple custom email notices from 1 event. For Example, if event occurs, log it and 'notice' as usual, but if it is a notice contains an old version of 'Java' also email the help desk at a different email address than the one that broctl knows to use. I would also want to create a custom message format. This was easily accomplished with a cron'd shell script that scrapes mail.log every ten minutes, checks a static file to see if the host was already alerted on, if not, add host to the file and send the email identifying the time, host and version of software that needs to be patched. I have figured there were ways to accomplish this in the current release of bro, but took the simplest route. > > # This handler is useful for processing notices after the notice filters > # have been applied and yielded an Notice::Action. > # > # It's tempting to make the default handler do the logging and > # printing to notice_file, rather than NOTICE. ?I hesitate to do that, > # though, because it perhaps could slow down notification, because > # in the absence of event priorities, the event would have to wait > # behind any other already-queued events. > event notice_action(n: Notice::Info, action: Notice::Action) > ? ? ? ?{ > ? ? ? ?} > > I think that doing the notification and printing through an event has a lot of benefits but I see the downside too. ?What does everyone else think? ?Especially whomever wrote that comment. :) What kind of delay or slow down are we talking here? Seconds or minutes? I can't imagine it being more than a minute, which would be the least of my worries as long as the time stamp in the notice was accurate. I didn't write the comment, but figured if you opened it up to "everyone"... thanks for letting me share. :) Will > > An extra side thought too is that the current event priorities system is not the same as the priorities mentioned in the comment (i think). > > ?.Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From seth at icir.org Thu Jun 2 10:45:17 2011 From: seth at icir.org (Seth Hall) Date: Thu, 2 Jun 2011 13:45:17 -0400 Subject: [Bro-Dev] notice tags? In-Reply-To: <20110602173816.7084E36A398@taffy.ICSI.Berkeley.EDU> References: <20110602173816.7084E36A398@taffy.ICSI.Berkeley.EDU> Message-ID: <2BB8FF05-C162-4C2B-B6DA-52D9AA1C94C2@icir.org> On Jun 2, 2011, at 1:38 PM, Vern Paxson wrote: >> Should there be a separate and built in way to generate unique ids? > > Nice thought. If we had a random() BiF I think we could do it pretty easily in scripting land because we could just fmt() the output from the random() function and trim it to a certain length. That gets us the on-demand determinism for testing and it's reusable for other things (if we ever need a random number generator again). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From vern at icir.org Thu Jun 2 10:47:44 2011 From: vern at icir.org (Vern Paxson) Date: Thu, 02 Jun 2011 10:47:44 -0700 Subject: [Bro-Dev] notice tags? In-Reply-To: <2BB8FF05-C162-4C2B-B6DA-52D9AA1C94C2@icir.org> (Thu, 02 Jun 2011 13:45:17 EDT). Message-ID: <20110602174744.A9F7936A398@taffy.ICSI.Berkeley.EDU> > If we had a random() BiF I think we could do it pretty easily in scripting land because we could just fmt() the output from the random() function and trim it to a certain length. I'm not sure that's as good. We either need long tags, or have to start worrying about collisions. (Though if we had formatting to something like radix-64, then we could probably get both short tags and almost no chance of collision.) > That gets us the on-demand determinism for testing and it's reusable for > other things (if we ever need a random number generator again). I think random() in general makes sense, just not sure this should be one of its applications. Vern From vern at icir.org Thu Jun 2 10:49:47 2011 From: vern at icir.org (Vern Paxson) Date: Thu, 02 Jun 2011 10:49:47 -0700 Subject: [Bro-Dev] Notices done as event instead of function In-Reply-To: (Thu, 02 Jun 2011 13:44:56 EDT). Message-ID: <20110602174947.4421D36A398@taffy.ICSI.Berkeley.EDU> > What kind of delay or slow down are we talking here? Seconds or > minutes? I can't imagine it being more than a minute, which would be > the least of my worries as long as the time stamp in the notice was > accurate. I'm pretty sure I'm the one who wrote the comment, and the concern is delays of 100s of msec, not even seconds. The reason is because one of the notice actions might be some form of "drop connectivity", and for automated malware the msec's matter regarding how quickly the drop goes in. That said, a better way of dealing with this concern would be to have a solid notion of event prioritization. Vern From seth at icir.org Thu Jun 2 10:54:27 2011 From: seth at icir.org (Seth Hall) Date: Thu, 2 Jun 2011 13:54:27 -0400 Subject: [Bro-Dev] Notices done as event instead of function In-Reply-To: References: <49DE6913-FDB5-4643-B802-7EB0426EBB96@icir.org> Message-ID: <54FBED85-A397-4A62-B448-4CFC5335D91B@icir.org> On Jun 2, 2011, at 1:44 PM, Will wrote: > Personally, I like where you are going with making it easier to extend > the built in framework. There have been multiple occasions where I > have wanted to generate multiple custom email notices from 1 event. I have frequently run into very similar trouble when working with the current notice framework. > What kind of delay or slow down are we talking here? Seconds or > minutes? I can't imagine it being more than a minute, which would be > the least of my worries as long as the time stamp in the notice was > accurate. Hopefully less than seconds even but there's no way to know what your event queue will look at any specific moment. > I didn't write the comment, but figured if you opened it up to > "everyone"... thanks for letting me share. :) I'll count that as one vote for flexibility over immediate immediacy (since in most cases it would still be very quick). :) Perhaps we could implement the notice pathway as events and then make a way to inject certain events higher in the event queue if it turns out to be problematic for anyone in the future. Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Thu Jun 2 10:58:29 2011 From: seth at icir.org (Seth Hall) Date: Thu, 2 Jun 2011 13:58:29 -0400 Subject: [Bro-Dev] notice tags? In-Reply-To: <20110602174744.A9F7936A398@taffy.ICSI.Berkeley.EDU> References: <20110602174744.A9F7936A398@taffy.ICSI.Berkeley.EDU> Message-ID: On Jun 2, 2011, at 1:47 PM, Vern Paxson wrote: >> If we had a random() BiF I think we could do it pretty easily in scripting land because we could just fmt() the output from the random() function and trim it to a certain length. > > I'm not sure that's as good. We either need long tags, or have to start > worrying about collisions. (Though if we had formatting to something like > radix-64, then we could probably get both short tags and almost no chance > of collision.) The random() data would only be used for a notice prefix. I think we'd only generate the random ID at init time and then use an incrementing counter as a postfix for the full ID of each notice after that which is how it's done now. The only chance we'd have for collisions would be in a cluster context between nodes generating notices. Each node would never generate a conflict unless it managed to overflow a 64-bit int. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Thu Jun 2 11:03:06 2011 From: seth at icir.org (Seth Hall) Date: Thu, 2 Jun 2011 14:03:06 -0400 Subject: [Bro-Dev] Notices done as event instead of function In-Reply-To: <20110602174947.4421D36A398@taffy.ICSI.Berkeley.EDU> References: <20110602174947.4421D36A398@taffy.ICSI.Berkeley.EDU> Message-ID: <3C0C48FB-D916-430B-ABA8-79B96C6964AA@icir.org> On Jun 2, 2011, at 1:49 PM, Vern Paxson wrote: > The reason is because one of the notice actions might be some form > of "drop connectivity", and for > automated malware the msec's matter regarding how quickly the drop goes in. Ohhh. That's a good point which I hadn't even considered. > That said, a better way of dealing with this concern would be to have > a solid notion of event prioritization. What about a new keyword to indicate that the event should be placed at the top of the event queue? immediate_event notice_action(n, ACTION_DROP); .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gc355804 at ohio.edu Thu Jun 2 11:55:27 2011 From: gc355804 at ohio.edu (Clark, Gilbert) Date: Thu, 2 Jun 2011 14:55:27 -0400 Subject: [Bro-Dev] notice tags? In-Reply-To: References: <20110602174744.A9F7936A398@taffy.ICSI.Berkeley.EDU>, Message-ID: What about synchronizing a seed across the bro cluster, and using that seed to generate random() concatenated with 48 bits obtained from the machine (adapter Ethernet address?) to ensure the starting value was unique across the whole cluster? --Gilbert ________________________________________ From: bro-dev-bounces at bro-ids.org [bro-dev-bounces at bro-ids.org] On Behalf Of Seth Hall [seth at icir.org] Sent: Thursday, June 02, 2011 1:58 PM To: Vern Paxson Cc: Bro Dev Subject: Re: [Bro-Dev] notice tags? On Jun 2, 2011, at 1:47 PM, Vern Paxson wrote: >> If we had a random() BiF I think we could do it pretty easily in scripting land because we could just fmt() the output from the random() function and trim it to a certain length. > > I'm not sure that's as good. We either need long tags, or have to start > worrying about collisions. (Though if we had formatting to something like > radix-64, then we could probably get both short tags and almost no chance > of collision.) The random() data would only be used for a notice prefix. I think we'd only generate the random ID at init time and then use an incrementing counter as a postfix for the full ID of each notice after that which is how it's done now. The only chance we'd have for collisions would be in a cluster context between nodes generating notices. Each node would never generate a conflict unless it managed to overflow a 64-bit int. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ _______________________________________________ bro-dev mailing list bro-dev at bro-ids.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From seth at icir.org Thu Jun 2 12:01:14 2011 From: seth at icir.org (Seth Hall) Date: Thu, 2 Jun 2011 15:01:14 -0400 Subject: [Bro-Dev] notice tags? In-Reply-To: References: <20110602174744.A9F7936A398@taffy.ICSI.Berkeley.EDU>, Message-ID: <00BFB88E-41ED-4D13-A235-9686B00E8EA9@icir.org> On Jun 2, 2011, at 2:55 PM, Clark, Gilbert wrote: > What about synchronizing a seed across the bro cluster, and using that seed to generate random() concatenated with 48 bits obtained from the machine (adapter Ethernet address?) to ensure the starting value was unique across the whole cluster? We could use the node name somewhere to help create uniqueness. MAC address wouldn't actually work because in many cases, people are only sniffing a single interface with a MAC address for their entire cluster. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Thu Jun 2 12:05:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 02 Jun 2011 19:05:43 -0000 Subject: [Bro-Dev] #466: Add += initializer support to vectors Message-ID: <046.29d551c288ce36f33bf0398b2a924cac@tracker.bro-ids.org> #466: Add += initializer support to vectors -----------------------------+----------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Component: Bro | Version: Keywords: language | -----------------------------+----------------- This code should work by effectively adding the element to the end of the vector.... {{{ const foobar: vector of string = vector() &redef; redef foobar += { "testing" }; }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Thu Jun 2 12:22:35 2011 From: seth at icir.org (Seth Hall) Date: Thu, 2 Jun 2011 15:22:35 -0400 Subject: [Bro-Dev] Notices done as event instead of function In-Reply-To: <20110602174947.4421D36A398@taffy.ICSI.Berkeley.EDU> References: <20110602174947.4421D36A398@taffy.ICSI.Berkeley.EDU> Message-ID: <90BA243C-B47D-4361-A0A5-092B1C063217@icir.org> On Jun 2, 2011, at 1:49 PM, Vern Paxson wrote: > That said, a better way of dealing with this concern would be to have > a solid notion of event prioritization. I just worked around this problem by allowing users to add anonymous functions to a set of functions that get called for each notice. It's sort of a parallel track to the event based pathway that is only used when things need to be run closer to real time. It will at least give us the ability to support the near realtime constraints of the drop script and the easy event based extension mechanism. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From vern at icir.org Thu Jun 2 12:46:00 2011 From: vern at icir.org (Vern Paxson) Date: Thu, 02 Jun 2011 12:46:00 -0700 Subject: [Bro-Dev] Notices done as event instead of function In-Reply-To: <90BA243C-B47D-4361-A0A5-092B1C063217@icir.org> (Thu, 02 Jun 2011 15:22:35 EDT). Message-ID: <20110602194600.3227536A398@taffy.ICSI.Berkeley.EDU> > I just worked around this problem by allowing users to add anonymous functions to a set of functions that get called for each notice. Hmmm, this suggests another easy fix, which is that NOTICE itself genereates an event, such as secondary_notice_event, and thus that want to hook non-time-critical stuff on can use that. Vern From seth at icir.org Thu Jun 2 12:50:04 2011 From: seth at icir.org (Seth Hall) Date: Thu, 2 Jun 2011 15:50:04 -0400 Subject: [Bro-Dev] Notices done as event instead of function In-Reply-To: <20110602194600.3227536A398@taffy.ICSI.Berkeley.EDU> References: <20110602194600.3227536A398@taffy.ICSI.Berkeley.EDU> Message-ID: <06A52452-AF20-41BB-8836-5A10C2B23ED4@icir.org> On Jun 2, 2011, at 3:46 PM, Vern Paxson wrote: >> I just worked around this problem by allowing users to add anonymous functions to a set of functions that get called for each notice. > > Hmmm, this suggests another easy fix, which is that NOTICE itself genereates > an event, such as secondary_notice_event, and thus that want to hook > non-time-critical stuff on can use that. Yep, that's essentially how it's done with emphasis on the event based notice extension and the function based notice code there in case someone needs it. I'm going to be committing a lot of updates to the notice code in just a minute if you'd like to take a look at it. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Thu Jun 2 13:26:18 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 02 Jun 2011 20:26:18 -0000 Subject: [Bro-Dev] #467: irc_dcc_message event is broken. Message-ID: <046.2cf508bb8daba167ac058271391d11c0@tracker.bro-ids.org> #467: irc_dcc_message event is broken. ---------------------+----------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: Keywords: | ---------------------+----------------- My tracefile I have used to test this doesn't seem to ever generate this event. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Jun 2 20:38:19 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 2 Jun 2011 20:38:19 -0700 Subject: [Bro-Dev] notice tags? In-Reply-To: <00BFB88E-41ED-4D13-A235-9686B00E8EA9@icir.org> References: <20110602174744.A9F7936A398@taffy.ICSI.Berkeley.EDU> <00BFB88E-41ED-4D13-A235-9686B00E8EA9@icir.org> Message-ID: <20110603033819.GA82503@icir.org> Didn't we just solve this problem for the conn uids? It even does radix-64. We could reuse some of that code to generate unique IDs for the script-land. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Thu Jun 2 22:04:16 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 2 Jun 2011 22:04:16 -0700 Subject: [Bro-Dev] Loading directories In-Reply-To: <20110601190822.GB67978@icir.org> References: <20110601190822.GB67978@icir.org> Message-ID: <20110603050416.GA85396@icir.org> On Wed, Jun 01, 2011 at 12:08 -0700, I wrote: > What I'm wondering hiwer is whether we should move "foo.bro" into the > subdirectory under some magic name I've just commited this to master. The magic file name is "__load__.bro". However, it turns out that "@load ./abc.bro" doesn't work because the cwd doesn't get adjusted when loading a file. That isn't easy to add, so I'm leaving that as it is. Seems fine for now. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Fri Jun 3 06:14:04 2011 From: seth at icir.org (Seth Hall) Date: Fri, 3 Jun 2011 09:14:04 -0400 Subject: [Bro-Dev] notice tags? In-Reply-To: <20110603033819.GA82503@icir.org> References: <20110602174744.A9F7936A398@taffy.ICSI.Berkeley.EDU> <00BFB88E-41ED-4D13-A235-9686B00E8EA9@icir.org> <20110603033819.GA82503@icir.org> Message-ID: On Jun 2, 2011, at 11:38 PM, Robin Sommer wrote: > Didn't we just solve this problem for the conn uids? It even does > radix-64. We could reuse some of that code to generate unique IDs for > the script-land. Sounds good to me. I'll update the notice script to use the new BiF when it exists. :P .Seth From bro at tracker.bro-ids.org Fri Jun 3 07:34:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 03 Jun 2011 14:34:00 -0000 Subject: [Bro-Dev] #468: topic/seth/syslog-analyzer - Merge syslog analyzer into master Message-ID: <046.0b0b77fccd28fbd608dcf846dae286ef@tracker.bro-ids.org> #468: topic/seth/syslog-analyzer - Merge syslog analyzer into master ---------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ New Syslog analyzer. This analyzer only support Syslog over UDP for now and doesn't support some of more recent options that have been added to Syslog. I don't think those options are commonly used anyway though so it shouldn't make much of a difference. The script that is implemented in this branch for now is essentially a throw away script and it will be updated in the policy- scripts-new branch. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Jun 3 08:28:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 03 Jun 2011 15:28:29 -0000 Subject: [Bro-Dev] #447: Bro snaplen In-Reply-To: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> References: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> Message-ID: <061.093560d94538901781abbeb3c48158f9@tracker.bro-ids.org> #447: Bro snaplen ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by gregor): If setting it to 0 doesn't work, does it make sense to at least set it to 65535 or at least 32KB or 16KB? -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Fri Jun 3 08:35:41 2011 From: gregor at icir.org (Gregor Maier) Date: Fri, 03 Jun 2011 08:35:41 -0700 Subject: [Bro-Dev] notice tags? In-Reply-To: <6866A3DB-7A55-4E3F-A821-027A3C8D3D5C@icir.org> References: <20110602172504.9CD3636A397@taffy.ICSI.Berkeley.EDU> <6866A3DB-7A55-4E3F-A821-027A3C8D3D5C@icir.org> Message-ID: <4DE8FF4D.2070108@icir.org> Do we still need the tags once we have unique connids for grepping? On 6/2/11 10:33 , Seth Hall wrote: > > On Jun 2, 2011, at 1:25 PM, Vern Paxson wrote: > >>> Are notice tags used by anyone? >> >> They are handy for grepping. The tag also appears in the conn file >> (as an $addl element), so you can link the two together. > > Heh, it actually doesn't right now. :) > > I'll add that back in (notice id in conn.log), I can see it being useful in some cases. With the record extension stuff, I think I'll be implementing it as a separate field just for notices and it will probably be implemented in the notice script itself. I never liked that $addl field, it was always hard to figure out what it was for. > > Should there be a separate and built in way to generate unique ids? What bothered me about the tag implementation in the notice script is that it's a bit difficult to figure what's going on due to dealing with non-determinism of the tags because of the data it bases the tags on. If we had a BiF that just generated unique IDs, we could build all of the determinism in there and wouldn't have to worry about it anymore where we need unique IDs. > > Thanks, > .Seth > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From seth at icir.org Fri Jun 3 09:02:33 2011 From: seth at icir.org (Seth Hall) Date: Fri, 3 Jun 2011 12:02:33 -0400 Subject: [Bro-Dev] notice tags? In-Reply-To: <4DE8FF4D.2070108@icir.org> References: <20110602172504.9CD3636A397@taffy.ICSI.Berkeley.EDU> <6866A3DB-7A55-4E3F-A821-027A3C8D3D5C@icir.org> <4DE8FF4D.2070108@icir.org> Message-ID: <2982C80F-F053-4400-B5DE-53582B072691@icir.org> On Jun 3, 2011, at 11:35 AM, Gregor Maier wrote: > Do we still need the tags once we have unique connids for grepping? It might be worthwhile still. It was interesting to me to see all of the notices attached to a connection. I'm a big proponent of browsing Bro logs and I like anything that makes my browsing experience better. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Fri Jun 3 09:06:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 03 Jun 2011 16:06:36 -0000 Subject: [Bro-Dev] #447: Bro snaplen In-Reply-To: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> References: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> Message-ID: <061.832640784fe6fd2623047811cbc7b9f7@tracker.bro-ids.org> #447: Bro snaplen ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): On Jun 3, 2011, at 11:28 AM, Bro Tracker wrote: > If setting it to 0 doesn't work, does it make sense to at least set it to > 65535 or at least 32KB or 16KB? Robin and I talked about making it a command line/script variable setting to make it similar to other tools. We could then set it to some reasonable value by default but ultimately leave it up to the user. -- Ticket URL: Bro Tracker Bro Issue Tracker From jones at tacc.utexas.edu Fri Jun 3 09:46:20 2011 From: jones at tacc.utexas.edu (William Jones) Date: Fri, 3 Jun 2011 11:46:20 -0500 Subject: [Bro-Dev] #447: Bro snaplen In-Reply-To: <061.832640784fe6fd2623047811cbc7b9f7@tracker.bro-ids.org> References: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> <061.832640784fe6fd2623047811cbc7b9f7@tracker.bro-ids.org> Message-ID: I would like to see a method to set the snaplen. I setting to 65534 to a ccommodate large segemtn offload in in the new 10 GigE nics. Bill Jones -----Original Message----- From: bro-dev-bounces at bro-ids.org [mailto:bro-dev-bounces at bro-ids.org] On Behalf Of Bro Tracker Sent: Friday, June 03, 2011 11:07 AM To: vern at icir.org Cc: bro-dev at bro-ids.org Subject: Re: [Bro-Dev] #447: Bro snaplen #447: Bro snaplen ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by seth): On Jun 3, 2011, at 11:28 AM, Bro Tracker wrote: > If setting it to 0 doesn't work, does it make sense to at least set it to > 65535 or at least 32KB or 16KB? Robin and I talked about making it a command line/script variable setting to make it similar to other tools. We could then set it to some reasonable value by default but ultimately leave it up to the user. -- Ticket URL: Bro Tracker Bro Issue Tracker _______________________________________________ bro-dev mailing list bro-dev at bro-ids.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From vern at icir.org Fri Jun 3 10:49:50 2011 From: vern at icir.org (Vern Paxson) Date: Fri, 03 Jun 2011 10:49:50 -0700 Subject: [Bro-Dev] notice tags? In-Reply-To: <2982C80F-F053-4400-B5DE-53582B072691@icir.org> (Fri, 03 Jun 2011 12:02:33 EDT). Message-ID: <20110603174950.B8FA736A39E@taffy.ICSI.Berkeley.EDU> > > Do we still need the tags once we have unique connids for grepping? > > It might be worthwhile still. It was interesting to me to see all of the notices attached to a connection. Seems you get that already if the notice just includes the connid. Where it would come up short is if you *don't* want all the notices (since there are a zillion boring ones), or if a given notice might have a tag associated with multiple connections (I don't think we do this presently, but in principle it would make sense). Vern From bro at tracker.bro-ids.org Fri Jun 3 10:52:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 03 Jun 2011 17:52:14 -0000 Subject: [Bro-Dev] #447: Bro snaplen In-Reply-To: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> References: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> Message-ID: <061.1c4e3f2fc8d601bac8ad71b8d22f66ce@tracker.bro-ids.org> #447: Bro snaplen ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by vern): To first order, I don't get why we'd want this to be user-settable. Bro doesn't work right unless it gets full packets. There's a *tuning* question about how to achieve full packets without wasting kernel memory, but what we'd really like is for the configure process to figure out how to set that, rather than expecting the user to know how. (Sure, gurus might want to set it. But they will be very rare.) -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Fri Jun 3 11:01:41 2011 From: seth at icir.org (Seth Hall) Date: Fri, 3 Jun 2011 14:01:41 -0400 Subject: [Bro-Dev] notice tags? In-Reply-To: <20110603174950.B8FA736A39E@taffy.ICSI.Berkeley.EDU> References: <20110603174950.B8FA736A39E@taffy.ICSI.Berkeley.EDU> Message-ID: <8E2AEB1B-8AEE-4AB0-966B-356130996CFC@icir.org> On Jun 3, 2011, at 1:49 PM, Vern Paxson wrote: > or if a given notice might have a tag associated > with multiple connections (I don't think we do this presently, but in > principle it would make sense). I did this in the new policy scripts yesterday. It makes it much easier to search for connections with Notices if the notice id is included in the conn log. We could implement this modularly though so that the notices (and the whole notice column itself) is only included if the script that implements that is loaded. It should get us the best of both depending on what you need. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Fri Jun 3 11:24:25 2011 From: seth at icir.org (Seth Hall) Date: Fri, 3 Jun 2011 14:24:25 -0400 Subject: [Bro-Dev] alarm function Message-ID: Is more of a purpose to the alarm function than just printing to the alarm.log file? .Seth From vern at icir.org Fri Jun 3 13:42:44 2011 From: vern at icir.org (Vern Paxson) Date: Fri, 03 Jun 2011 13:42:44 -0700 Subject: [Bro-Dev] alarm function In-Reply-To: (Fri, 03 Jun 2011 14:24:25 EDT). Message-ID: <20110603204244.DC93736A3A1@taffy.ICSI.Berkeley.EDU> > Is more of a purpose to the alarm function than just printing to the alarm.log file? Originally it was the interface to syslog. This has now been factored out into alarm_hook, which alarm will invoke if it's present. In principle we could get rid of it by replacing it with explicit calls to alarm_hook (if it's defined). I don't view this as a priority, though. Vern From seth at icir.org Fri Jun 3 14:01:40 2011 From: seth at icir.org (Seth Hall) Date: Fri, 3 Jun 2011 17:01:40 -0400 Subject: [Bro-Dev] alarm function In-Reply-To: <20110603204244.DC93736A3A1@taffy.ICSI.Berkeley.EDU> References: <20110603204244.DC93736A3A1@taffy.ICSI.Berkeley.EDU> Message-ID: On Jun 3, 2011, at 4:42 PM, Vern Paxson wrote: >> Is more of a purpose to the alarm function than just printing to the alarm.log file? > > Originally it was the interface to syslog. This has now been factored out > into alarm_hook, which alarm will invoke if it's present. In principle > we could get rid of it by replacing it with explicit calls to alarm_hook > (if it's defined). I don't view this as a priority, though. I may just remove the call to alarm then. The notice code has the notice_functions which are a set of synchronously called functions when notices are created. It's basically the same thing but completely implemented in a Bro script and you can have multiple functions instead of just one. It should open up the extension options a bit more and help prevent scripts that want to hook into the notice pipeline synchronously avoid stepping on each others toes. I can implement the alarm.log as a filter on the notice.log with the logging framework, but I'm not completely sure what benefits come from keeping a separate file since there is a field in the notice.log that indicates if it was alarmed on. Speaking of syslog, I just updated my syslog analyzer branch to be mergeable with master today. Bro can produce and consume (off the wire) syslog now. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From vern at icir.org Fri Jun 3 14:04:35 2011 From: vern at icir.org (Vern Paxson) Date: Fri, 03 Jun 2011 14:04:35 -0700 Subject: [Bro-Dev] alarm function In-Reply-To: (Fri, 03 Jun 2011 17:01:40 EDT). Message-ID: <20110603210435.53F0436A3A1@taffy.ICSI.Berkeley.EDU> > I can implement the alarm.log as a filter on the notice.log with the logging framework, but I'm not completely sure what benefits come from keeping a separate file since there is a field in the notice.log that indicates if it was alarmed on. One benefit is that alarm.log is often much smaller than notice.log (a factor of 10,000 smaller for my ICSI config). Sure, one can figure out how to grep the notice.log file for the particular needles in the haystack, but it can be nice to just have them sitting there directly. Vern From scampbell at lbl.gov Fri Jun 3 14:09:02 2011 From: scampbell at lbl.gov (Scott Campbell) Date: Fri, 03 Jun 2011 16:09:02 -0500 Subject: [Bro-Dev] alarm function In-Reply-To: References: <20110603204244.DC93736A3A1@taffy.ICSI.Berkeley.EDU> Message-ID: <4DE94D6E.4040005@lbl.gov> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - From an operational perspective, I find the alarm file quite helpful (in spite of it's redundancy). scott On 6/3/11 4:01 PM, Seth Hall wrote: > > On Jun 3, 2011, at 4:42 PM, Vern Paxson wrote: > >>> Is more of a purpose to the alarm function than just printing to the alarm.log file? >> >> Originally it was the interface to syslog. This has now been factored out >> into alarm_hook, which alarm will invoke if it's present. In principle >> we could get rid of it by replacing it with explicit calls to alarm_hook >> (if it's defined). I don't view this as a priority, though. > > > I may just remove the call to alarm then. The notice code has the notice_functions which are a set of synchronously called functions when notices are created. It's basically the same thing but completely implemented in a Bro script and you can have multiple functions instead of just one. It should open up the extension options a bit more and help prevent scripts that want to hook into the notice pipeline synchronously avoid stepping on each others toes. > > I can implement the alarm.log as a filter on the notice.log with the logging framework, but I'm not completely sure what benefits come from keeping a separate file since there is a field in the notice.log that indicates if it was alarmed on. > > Speaking of syslog, I just updated my syslog analyzer branch to be mergeable with master today. Bro can produce and consume (off the wire) syslog now. :) > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFN6U1uK2Plq8B7ZBwRAmT/AKCd/y5A0ZWkYbAi1uiHTAUtG6QNpQCeOI/A 4UQk3etT45KOYj8QrNE1Gj8= =E8Eo -----END PGP SIGNATURE----- From asharma at lbl.gov Fri Jun 3 14:16:55 2011 From: asharma at lbl.gov (Aashish Sharma) Date: Fri, 3 Jun 2011 14:16:55 -0700 Subject: [Bro-Dev] alarm function In-Reply-To: <20110603210435.53F0436A3A1@taffy.ICSI.Berkeley.EDU> References: <20110603210435.53F0436A3A1@taffy.ICSI.Berkeley.EDU> Message-ID: <20110603211654.GA7060@yaksha.lbl.gov> I would personally prefer having alarms go in a separate file (alarm.log): 1) Saves me to set up a post-process to extract NOTICE_ALARM from notice logs 2) Get alarm.log emailed periodically and saving NOTICE_EMAIL for really crucial notices. 3) Allows me to write a little generic rules with NOTICE_FILE since I don't much worry about bloated notice log Aashish On Fri, Jun 03, 2011 at 02:04:35PM -0700, Vern Paxson wrote: > > I can implement the alarm.log as a filter on the notice.log with the logging framework, but I'm not completely sure what benefits come from keeping a separate file since there is a field in the notice.log that indicates if it was alarmed on. > > One benefit is that alarm.log is often much smaller than notice.log > (a factor of 10,000 smaller for my ICSI config). Sure, one can figure > out how to grep the notice.log file for the particular needles in the > haystack, but it can be nice to just have them sitting there directly. > > Vern > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Aashish Sharma (asharma at lbl.gov) Cyber Security, Information Technology Division Lawrence Berkeley National Laboratory http://www.lbl.gov/cyber/pgp-aashish.txt Office: (510)-495-2680 Cell: (510)-457-1525 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110603/3113dfef/attachment.bin From seth at icir.org Fri Jun 3 19:39:57 2011 From: seth at icir.org (Seth Hall) Date: Fri, 3 Jun 2011 22:39:57 -0400 Subject: [Bro-Dev] alarm function In-Reply-To: <20110603210435.53F0436A3A1@taffy.ICSI.Berkeley.EDU> References: <20110603210435.53F0436A3A1@taffy.ICSI.Berkeley.EDU> Message-ID: <094666D2-C81C-40B6-8704-3173BCB6C5E2@icir.org> On Jun 3, 2011, at 5:04 PM, Vern Paxson wrote: > One benefit is that alarm.log is often much smaller than notice.log > (a factor of 10,000 smaller for my ICSI config). Sure, one can figure Sounds good. I'll put a filter in the notice script that filters alarms into a file named alarm.log. .Seth From robin at icir.org Mon Jun 6 08:12:46 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 6 Jun 2011 08:12:46 -0700 Subject: [Bro-Dev] http.bro In-Reply-To: <7c8529f4-c77a-4842-980b-ed28f99ddf97@tangent.ncsa.illinois.edu> References: <20110604211735.GB43759@icir.org> <7c8529f4-c77a-4842-980b-ed28f99ddf97@tangent.ncsa.illinois.edu> Message-ID: <20110606151246.GA42166@icir.org> On Mon, Jun 06, 2011 at 09:47 -0500, John wrote: > What's the reason for not providing a __load__.bro convenience for the optional packages? For the optional packages, there'll be some which make sense to load only in specific situations. My thinking is that by not providing a __load__, we make sure people actually take a look at what they are loading. I'm not too strong on this though and do see the convinience of having the __load__ there as well. > Another thing came to my mind: will "all.bro" be excluding optional > scripts/packages? That depends on what all.bro's purpose is. So far we have primarily used it for testing to make sure all scripts load correctly. For that, the optional pieces should be included. But perhaps we should provide two separate scripts: (1) the current all.bro for testing, perhaps renamed to "test-all.bro" or something like that; (2) an "all.bro" for users that pulls in all of the basic stuff that's reasonable to load by default, but not including the optional scripts. Perhaps we find a nicer name for that one too (it would be kind of a new "mt.bro"). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Mon Jun 6 10:15:26 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 06 Jun 2011 10:15:26 -0700 Subject: [Bro-Dev] http.bro In-Reply-To: <20110606151246.GA42166@icir.org> References: <20110604211735.GB43759@icir.org> <7c8529f4-c77a-4842-980b-ed28f99ddf97@tangent.ncsa.illinois.edu> <20110606151246.GA42166@icir.org> Message-ID: <4DED0B2E.1030808@icir.org> > But perhaps we should provide two separate scripts: (1) the current > all.bro for testing, perhaps renamed to "test-all.bro" or something > like that; (2) an "all.bro" for users that pulls in all of the basic > stuff that's reasonable to load by default, but not including the > optional scripts. Perhaps we find a nicer name for that one too (it > would be kind of a new "mt.bro"). how about * default.bro * basic.bro * standard.bro (just need to make sure that people don't get the impression that if they just load that without looking any further anything will be fine....) cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From robin at icir.org Mon Jun 6 10:20:22 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 6 Jun 2011 10:20:22 -0700 Subject: [Bro-Dev] http.bro In-Reply-To: <4DED0B2E.1030808@icir.org> References: <20110604211735.GB43759@icir.org> <7c8529f4-c77a-4842-980b-ed28f99ddf97@tangent.ncsa.illinois.edu> <20110606151246.GA42166@icir.org> <4DED0B2E.1030808@icir.org> Message-ID: <20110606172022.GF45185@icir.org> On Mon, Jun 06, 2011 at 10:15 -0700, you wrote: > * default.bro > * basic.bro > * standard.bro That sounds confusing. What would be the difference between default and standard? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Mon Jun 6 11:46:29 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 06 Jun 2011 11:46:29 -0700 Subject: [Bro-Dev] http.bro In-Reply-To: <20110606172022.GF45185@icir.org> References: <20110604211735.GB43759@icir.org> <7c8529f4-c77a-4842-980b-ed28f99ddf97@tangent.ncsa.illinois.edu> <20110606151246.GA42166@icir.org> <4DED0B2E.1030808@icir.org> <20110606172022.GF45185@icir.org> Message-ID: <4DED2085.5050904@icir.org> none. I though that either of those might be a nice new name for the all.bro that user's would load. I.e., I would just pick one of those (or maybe another name alltogether) cu gregor On 6/6/11 10:20 , Robin Sommer wrote: > > On Mon, Jun 06, 2011 at 10:15 -0700, you wrote: > >> * default.bro >> * basic.bro >> * standard.bro > > That sounds confusing. What would be the difference between default > and standard? > > Robin > -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Tue Jun 7 15:43:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 07 Jun 2011 22:43:42 -0000 Subject: [Bro-Dev] #462: topic/seth/ssl-binpac - core SSL/TLS analyzer work In-Reply-To: <046.33fc53d356c9729c83a221d2d154c213@tracker.bro-ids.org> References: <046.33fc53d356c9729c83a221d2d154c213@tracker.bro-ids.org> Message-ID: <061.f4ea9e5959972d2bb12c3efaa8cc7c5e@tracker.bro-ids.org> #462: topic/seth/ssl-binpac - core SSL/TLS analyzer work -----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Jun 7 15:44:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 07 Jun 2011 22:44:01 -0000 Subject: [Bro-Dev] #468: topic/seth/syslog-analyzer - Merge syslog analyzer into master In-Reply-To: <046.0b0b77fccd28fbd608dcf846dae286ef@tracker.bro-ids.org> References: <046.0b0b77fccd28fbd608dcf846dae286ef@tracker.bro-ids.org> Message-ID: <061.163283f4753f9e6bc9aec4ce5d68767a@tracker.bro-ids.org> #468: topic/seth/syslog-analyzer - Merge syslog analyzer into master -----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jun 9 10:54:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 09 Jun 2011 17:54:24 -0000 Subject: [Bro-Dev] #469: Core IRC analyzer needs to provide is_orig field. Message-ID: <046.ead7a0f871260990c028a605e157ef4b@tracker.bro-ids.org> #469: Core IRC analyzer needs to provide is_orig field. ---------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ None of the IRC related events indicate which side of the connection sent a message which causes problems for writing analysis scripts. Here's what I think we should do: - The irc_client and irc_server events should be removed. - The irc_message event should have an is_orig:bool field added as the last argument. - All other irc_* events should either be removed and implemented in a Bro script or have an is_orig argument added. Maybe add the is_orig argument for now with the thought of removing those events in the future once we implement everything in script-land with the irc_message event? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jun 9 21:10:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 10 Jun 2011 04:10:23 -0000 Subject: [Bro-Dev] #470: print helpful info when os.remove() fails Message-ID: <047.203d22df6ee6795bf7dd48222840bfe7@tracker.bro-ids.org> #470: print helpful info when os.remove() fails -----------------------------+------------------- Reporter: leres | Owner: robin Type: Feature Request | Status: new Priority: Normal | Milestone: Component: BroControl | Version: 1.5.3 Keywords: | -----------------------------+------------------- While installing bro/broctl to run as a non-root user, we found it very difficult to determine which directories needed to be writable. The attached patch solves this issue. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Jun 10 08:26:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 10 Jun 2011 15:26:38 -0000 Subject: [Bro-Dev] #470: print helpful info when os.remove() fails In-Reply-To: <047.203d22df6ee6795bf7dd48222840bfe7@tracker.bro-ids.org> References: <047.203d22df6ee6795bf7dd48222840bfe7@tracker.bro-ids.org> Message-ID: <062.c4cdd07c95e582e15cb5d6be9a70f5c7@tracker.bro-ids.org> #470: print helpful info when os.remove() fails -------------------------+---------------------- Reporter: leres | Owner: robin Type: Patch | Status: accepted Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: 1.5.3 Resolution: | Keywords: -------------------------+---------------------- Changes (by robin): * status: new => accepted * type: Feature Request => Patch * milestone: => Bro1.6 -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Fri Jun 10 13:18:56 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Fri, 10 Jun 2011 15:18:56 -0500 (CDT) Subject: [Bro-Dev] istate.events-ssl test In-Reply-To: Message-ID: <53db170d-708b-45e8-b5d6-e3bac6f52871@tangent.ncsa.illinois.edu> Robin, I think w/ the new commits I just did on fastpath all the current unit tests are passing except for istate.events-ssl. What's the status of that one? Looks like it's just missing a baseline? - Jon $ btest -D istate/events-ssl.bro istate.events-ssl ... failed % 'btest-diff sender/http.log' failed unexpectedly (exit code 100) % cat .diag == File =============================== 1307736535.043138 %events-send-1 start 141.42.64.125:56730 > 125.190.109.199:80 1307736535.043138 %events-send-1 > USER-AGENT: Wget/1.10 1307736535.043138 %events-send-1 > ACCEPT: */* 1307736535.043138 %events-send-1 > HOST: www.icir.org 1307736535.043138 %events-send-1 > CONNECTION: Keep-Alive 1307736535.227178 %events-send-1 < DATE: Fri, 07 Oct 2005 23:23:55 GMT 1307736535.227178 %events-send-1 < SERVER: Apache/1.3.33 (Unix) 1307736535.227178 %events-send-1 < LAST-MODIFIED: Fri, 07 Oct 2005 16:23:01 GMT 1307736535.227178 %events-send-1 < ETAG: "2c96c-23aa-4346a0e5" 1307736535.227178 %events-send-1 < ACCEPT-RANGES: bytes 1307736535.227178 %events-send-1 < CONTENT-LENGTH: 9130 1307736535.227178 %events-send-1 < KEEP-ALIVE: timeout=15, max=100 1307736535.227178 %events-send-1 < CONNECTION: Keep-Alive 1307736535.227178 %events-send-1 < CONTENT-TYPE: text/html 1307736535.411667 %events-send-1 <= 4096 bytes: "^J^J

^JPublications^J

^J
    ^J>> <<< [62543] bro ../receiver.bro 1307736550.854332 received termination signal >>> % 'btest-diff sender/http.log' failed unexpectedly (exit code 100) % 'btest-diff receiver/http.log' failed unexpectedly (exit code 100) % cat .diag == File =============================== == Error =============================== test-diff: no baseline found. ======================================= % cat .stderr <<< [62530] bro -C -r /Users/jsiwek/bro/testing/btest/Traces/web.trace --pseudo-realtime ../sender.bro >>> <<< [62543] bro ../receiver.bro 1307736550.854332 received termination signal >>> % 'btest-diff sender/http.log' failed unexpectedly (exit code 100) % 'btest-diff receiver/http.log' failed unexpectedly (exit code 100) % 'cmp http.rec.log http.snd.log' failed unexpectedly (exit code 1) % cat .diag == File =============================== == Error =============================== test-diff: no baseline found. ======================================= % cat .stderr <<< [62530] bro -C -r /Users/jsiwek/bro/testing/btest/Traces/web.trace --pseudo-realtime ../sender.bro >>> <<< [62543] bro ../receiver.bro 1307736550.854332 received termination signal >>> From seth at icir.org Fri Jun 10 13:22:53 2011 From: seth at icir.org (Seth Hall) Date: Fri, 10 Jun 2011 16:22:53 -0400 Subject: [Bro-Dev] istate.events-ssl test In-Reply-To: <53db170d-708b-45e8-b5d6-e3bac6f52871@tangent.ncsa.illinois.edu> References: <53db170d-708b-45e8-b5d6-e3bac6f52871@tangent.ncsa.illinois.edu> Message-ID: <08C5219D-B81D-449B-BAB0-849CD33A5476@icir.org> On Jun 10, 2011, at 4:18 PM, Jonathan Siwek wrote: > I think w/ the new commits I just did on fastpath all the current unit tests are passing except for istate.events-ssl. What's the status of that one? Looks like it's just missing a baseline? SSL is going to a bit messed up for now because the new analyzer has been merged in but there isn't a script to go with it. Partly because I'm running into a bug with it the core analyzer and I can't really write a comprehensive script yet. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Fri Jun 10 13:52:13 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 10 Jun 2011 13:52:13 -0700 Subject: [Bro-Dev] istate.events-ssl test In-Reply-To: <53db170d-708b-45e8-b5d6-e3bac6f52871@tangent.ncsa.illinois.edu> References: <53db170d-708b-45e8-b5d6-e3bac6f52871@tangent.ncsa.illinois.edu> Message-ID: <20110610205212.GG9550@icir.org> On Fri, Jun 10, 2011 at 15:18 -0500, you wrote: > I think w/ the new commits I just did on fastpath all the current unit > tests are passing except for istate.events-ssl. Great, thanks! I'll merge that soon. > What's the status of that one? Looks like it's just missing a > baseline? No, the SSL communication is actually not working for a reason I haven't figured out yet. The same kind of test worked fine with Bro 1.5 and the old istate test-suite but seems something broke when I rewrote the test for btest. Not sure yet what. > 1307736535.043138 %events-send-1 start 141.42.64.125:56730 > 125.190.109.199:80 Note that this is only the sender side; iirc, I didn't see anything received on at the other end, and the connection was just aborted. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Fri Jun 10 13:53:09 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 10 Jun 2011 13:53:09 -0700 Subject: [Bro-Dev] istate.events-ssl test In-Reply-To: <08C5219D-B81D-449B-BAB0-849CD33A5476@icir.org> References: <53db170d-708b-45e8-b5d6-e3bac6f52871@tangent.ncsa.illinois.edu> <08C5219D-B81D-449B-BAB0-849CD33A5476@icir.org> Message-ID: <20110610205309.GH9550@icir.org> On Fri, Jun 10, 2011 at 16:22 -0400, you wrote: > SSL is going to a bit messed up for now because the new analyzer has (This is about SSL-encrypted Bro-to-Bro communication, not related to the analyzer.) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Fri Jun 10 14:05:31 2011 From: seth at icir.org (Seth Hall) Date: Fri, 10 Jun 2011 17:05:31 -0400 Subject: [Bro-Dev] better name for "drop" Message-ID: <038921F9-89E2-4E83-BB62-C352037310D7@icir.org> Hi everyone, I'm looking for a better name for the drop.bro script. I think it would serve as it's own standalone framework that could be reused by other scripts, but "drop" is unfortunately not descriptive enough (in my opinion) for what it does. For those of your who don't know, drop.bro implements Bro's ability to call an external script and "drop" connectivity for an address. It also implements the notion of catch&release where a host would initially be blocked for a short period and released but if it continues doing things that would cause it to get blocked, it gets blocked for longer and longer periods each time. I am planning on merging that functionality with the functionality from the terminate-connection.bro script as well so the name of the framework would need to reflect this generic notion denying access to hosts or connections. Any thoughts? Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From vern at icir.org Fri Jun 10 15:17:40 2011 From: vern at icir.org (Vern Paxson) Date: Fri, 10 Jun 2011 15:17:40 -0700 Subject: [Bro-Dev] better name for "drop" In-Reply-To: <038921F9-89E2-4E83-BB62-C352037310D7@icir.org> (Fri, 10 Jun 2011 17:05:31 EDT). Message-ID: <20110610221740.144EA369FD0@taffy.ICSI.Berkeley.EDU> > I'm looking for a better name for the drop.bro script. What about something like react.bro? Vern From seth at icir.org Fri Jun 10 16:45:32 2011 From: seth at icir.org (Seth Hall) Date: Fri, 10 Jun 2011 19:45:32 -0400 Subject: [Bro-Dev] better name for "drop" In-Reply-To: <20110610221740.144EA369FD0@taffy.ICSI.Berkeley.EDU> References: <20110610221740.144EA369FD0@taffy.ICSI.Berkeley.EDU> Message-ID: On Jun 10, 2011, at 6:17 PM, Vern Paxson wrote: >> I'm looking for a better name for the drop.bro script. > > What about something like react.bro? Perfect, I'll use that. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Fri Jun 10 18:31:02 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 11 Jun 2011 01:31:02 -0000 Subject: [Bro-Dev] #471: topic/mozilla-ca-list - Ruby script to generate mozilla CA list. Message-ID: <046.a46ed554191672937e89d30805d56674@tracker.bro-ids.org> #471: topic/mozilla-ca-list - Ruby script to generate mozilla CA list. ---------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: bro-aux | Version: Keywords: | ---------------------------+-------------------- This is the script for the new SSL analyzer that generates the Bro script of CA's from the Mozilla foundation. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Mon Jun 13 10:31:42 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 13 Jun 2011 12:31:42 -0500 (CDT) Subject: [Bro-Dev] istate.events-ssl test In-Reply-To: <20110610205212.GG9550@icir.org> Message-ID: <0acc6fbf-e12c-46b9-9b2f-7e07c460f31b@tangent.ncsa.illinois.edu> > No, the SSL communication is actually not working for a reason I > haven't figured out yet. The same kind of test worked fine with Bro > 1.5 and the old istate test-suite but seems something broke when I > rewrote the test for btest. Not sure yet what. The CA cert that the test is using expired last March. (see output at bottom). > > 1307736535.043138 %events-send-1 start 141.42.64.125:56730 > > > 125.190.109.199:80 > > Note that this is only the sender side; iirc, I didn't see anything > received on at the other end, and the connection was just aborted. I tried generating my own keys and replacing what was in the test and the receiver looks like it saw everything (again output below). Let me know if it looks right to you and I can probably follow through and commit a working test unless you want to. - Jon $ openssl x509 -in ca_cert.pem -noout -enddate notAfter=Mar 10 04:13:23 2011 GMT $ ssldump -i lo0 New TCP connection #1: localhost(51344) <-> localhost(47756) 1 1 0.0011 (0.0011) C>S Handshake ClientHello Version 3.0 cipher suites Unknown value 0x3a Unknown value 0x39 Unknown value 0x38 Unknown value 0x35 Unknown value 0x34 Unknown value 0x33 Unknown value 0x32 Unknown value 0x2f SSL_DH_anon_WITH_3DES_EDE_CBC_SHA SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA SSL_RSA_WITH_3DES_EDE_CBC_SHA compression methods unknown value NULL 1 2 0.0012 (0.0001) S>C Handshake ServerHello Version 3.0 session_id[32]= d7 3d eb 9a c8 d9 f1 e7 de fb 3a 2a 2d c3 4f 64 cc 26 46 d1 e0 41 34 2c 95 a9 8c 37 f9 8c 51 43 cipherSuite Unknown value 0x35 compressionMethod unknown value 1 3 0.0012 (0.0000) S>C Handshake Certificate 1 4 0.0012 (0.0000) S>C Handshake CertificateRequest certificate_types rsa_sign certificate_types dss_sign ServerHelloDone 1 5 0.0023 (0.0010) C>S Alert level fatal value certificate_expired 1 6 0.0025 (0.0001) C>S Alert level fatal value certificate_expired 1 7 0.0027 (0.0001) C>S Alert level fatal value certificate_expired 1 0.0027 (0.0000) C>S TCP FIN 1 0.0029 (0.0001) S>C TCP FIN # after replacing keys $ btest -D istate/events-ssl.bro istate.events-ssl ... failed % 'btest-diff sender/http.log' failed unexpectedly (exit code 100) % cat .diag == File =============================== 1307985621.588992 %events-send-1 start 141.42.64.125:56730 > 125.190.109.199:80 1307985621.588992 %events-send-1 > USER-AGENT: Wget/1.10 1307985621.588992 %events-send-1 > ACCEPT: */* 1307985621.588992 %events-send-1 > HOST: www.icir.org 1307985621.588992 %events-send-1 > CONNECTION: Keep-Alive 1307985621.773032 %events-send-1 < DATE: Fri, 07 Oct 2005 23:23:55 GMT 1307985621.773032 %events-send-1 < SERVER: Apache/1.3.33 (Unix) 1307985621.773032 %events-send-1 < LAST-MODIFIED: Fri, 07 Oct 2005 16:23:01 GMT 1307985621.773032 %events-send-1 < ETAG: "2c96c-23aa-4346a0e5" 1307985621.773032 %events-send-1 < ACCEPT-RANGES: bytes 1307985621.773032 %events-send-1 < CONTENT-LENGTH: 9130 1307985621.773032 %events-send-1 < KEEP-ALIVE: timeout=15, max=100 1307985621.773032 %events-send-1 < CONNECTION: Keep-Alive 1307985621.773032 %events-send-1 < CONTENT-TYPE: text/html 1307985621.957521 %events-send-1 <= 4096 bytes: "^J^J

    ^JPublications^J

    ^J
      ^J>> <<< [49750] bro ../receiver.bro 1307985617.408817 processing suspended 1307985617.410215 processing continued 1307985623.207681 received termination signal >>> % 'btest-diff sender/http.log' failed unexpectedly (exit code 100) % 'btest-diff receiver/http.log' failed unexpectedly (exit code 100) % cat .diag == File =============================== 1307985621.655144 %events-rcv-1 start 141.42.64.125:56730 > 125.190.109.199:80 1307985621.655144 %events-rcv-1 > USER-AGENT: Wget/1.10 1307985621.655144 %events-rcv-1 > ACCEPT: */* 1307985621.655144 %events-rcv-1 > HOST: www.icir.org 1307985621.655144 %events-rcv-1 > CONNECTION: Keep-Alive 1307985621.835623 %events-rcv-1 < DATE: Fri, 07 Oct 2005 23:23:55 GMT 1307985621.835623 %events-rcv-1 < SERVER: Apache/1.3.33 (Unix) 1307985621.835623 %events-rcv-1 < LAST-MODIFIED: Fri, 07 Oct 2005 16:23:01 GMT 1307985621.835623 %events-rcv-1 < ETAG: "2c96c-23aa-4346a0e5" 1307985621.835623 %events-rcv-1 < ACCEPT-RANGES: bytes 1307985621.835623 %events-rcv-1 < CONTENT-LENGTH: 9130 1307985621.835623 %events-rcv-1 < KEEP-ALIVE: timeout=15, max=100 1307985621.835623 %events-rcv-1 < CONNECTION: Keep-Alive 1307985621.835623 %events-rcv-1 < CONTENT-TYPE: text/html 1307985622.037655 %events-rcv-1 <= 4096 bytes: "^J^J

      ^JPublications^J

      ^J
        ^J>> <<< [49750] bro ../receiver.bro 1307985617.408817 processing suspended 1307985617.410215 processing continued 1307985623.207681 received termination signal >>> 1 test failed From bro at tracker.bro-ids.org Mon Jun 13 16:03:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 13 Jun 2011 23:03:30 -0000 Subject: [Bro-Dev] #372: bifcl cannot pass specific enum types In-Reply-To: <047.e8e6512c3965101e829a4a0884e3bb2b@tracker.bro-ids.org> References: <047.e8e6512c3965101e829a4a0884e3bb2b@tracker.bro-ids.org> Message-ID: <062.ac34bc30381172640b5920ddd1ba5736@tracker.bro-ids.org> #372: bifcl cannot pass specific enum types -----------------------------+--------------------- Reporter: robin | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: logging -----------------------------+--------------------- Changes (by gregor): * status: new => closed * resolution: => Solved/Applied Comment: has been integrated for quite a while -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Jun 14 21:07:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 15 Jun 2011 04:07:48 -0000 Subject: [Bro-Dev] #456: topic/gregor/rpc -- Revamped RPC, portmap, and NFS analyzer In-Reply-To: <048.0b0d91bfb22f8b28349ee746687da4db@tracker.bro-ids.org> References: <048.0b0d91bfb22f8b28349ee746687da4db@tracker.bro-ids.org> Message-ID: <063.ffd5df32115d779fd531b53e571ded56@tracker.bro-ids.org> #456: topic/gregor/rpc -- Revamped RPC, portmap, and NFS analyzer -----------------------------+------------------------ Reporter: gregor | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Jun 14 21:11:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 15 Jun 2011 04:11:07 -0000 Subject: [Bro-Dev] #472: Cleanup scripts in bro-aux Message-ID: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> #472: Cleanup scripts in bro-aux --------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | --------------------+-------------------- There are some scripts in bro-aux which won't work anymore with the new policy scripts. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Jun 14 21:13:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 15 Jun 2011 04:13:11 -0000 Subject: [Bro-Dev] #471: topic/mozilla-ca-list - Ruby script to generate mozilla CA list. In-Reply-To: <046.a46ed554191672937e89d30805d56674@tracker.bro-ids.org> References: <046.a46ed554191672937e89d30805d56674@tracker.bro-ids.org> Message-ID: <061.5d6294f360c3589506fd5fce76f6bd61@tracker.bro-ids.org> #471: topic/mozilla-ca-list - Ruby script to generate mozilla CA list. -----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: bro-aux | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 15 05:06:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 15 Jun 2011 12:06:07 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: ScennaSpard Message-ID: <20110615120607.A00C936A37D@taffy.ICSI.Berkeley.EDU> New user registration for user ScennaSpard -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 15 06:33:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 15 Jun 2011 13:33:32 -0000 Subject: [Bro-Dev] #473: Remove or redirect "bro_logger" messages. Message-ID: <046.ac0045106dd553365c25462db1b26a3d@tracker.bro-ids.org> #473: Remove or redirect "bro_logger" messages. ---------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ Calls to the bro_logger->Log() method are being printed on stdout and not using the logging framework which is now a little out of place for output from Bro. I think those messages should probably be turned into some sort of event that can be handled in the scripting language. The most intrusive messages are probably the ones from the remote serializer ("processing suspended" and "processing continued") which really give very little information but it may be worthwhile to keep them around but they need to be moved into the communication framework log. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 15 08:16:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 15 Jun 2011 15:16:49 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: QCEdward Message-ID: <20110615151648.98B6736A37D@taffy.ICSI.Berkeley.EDU> New user registration for user QCEdward -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 15 09:23:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 15 Jun 2011 16:23:35 -0000 Subject: [Bro-Dev] #474: &raw_output turns null values into \0 Message-ID: <046.21443edb25709a7dff08ac2dadf6c6e0@tracker.bro-ids.org> #474: &raw_output turns null values into \0 ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- Files with the raw_output attribute shouldn't do any interpretation to the data. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 15 09:48:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 15 Jun 2011 16:48:58 -0000 Subject: [Bro-Dev] #475: Update Broccoli Test Scripts Message-ID: <048.a284abbfb80e3c325b0a34424a7f512f@tracker.bro-ids.org> #475: Update Broccoli Test Scripts ----------------------+---------------------- Reporter: jsiwek | Owner: kreibich Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Broccoli | Version: Keywords: | ----------------------+---------------------- The Bro scripts under broccoli's `test/` and `bindings/broccoli- python/tests/` need to be updated to work with the new policy scripts in 1.6. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 15 09:50:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 15 Jun 2011 16:50:05 -0000 Subject: [Bro-Dev] #475: Update Broccoli Test Scripts In-Reply-To: <048.a284abbfb80e3c325b0a34424a7f512f@tracker.bro-ids.org> References: <048.a284abbfb80e3c325b0a34424a7f512f@tracker.bro-ids.org> Message-ID: <063.87deff40cd76ebb956103acfd91456fe@tracker.bro-ids.org> #475: Update Broccoli Test Scripts -----------------------+---------------------- Reporter: jsiwek | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Broccoli | Version: Resolution: | Keywords: -----------------------+---------------------- Changes (by jsiwek): * owner: kreibich => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Jun 15 12:15:14 2011 From: seth at icir.org (Seth Hall) Date: Wed, 15 Jun 2011 15:15:14 -0400 Subject: [Bro-Dev] eval? Message-ID: <44615C2B-E6AB-4CFB-866A-E4473EF0C934@icir.org> I imagine that an eval BiF would be pretty easy to write and it would make it possible for me to do what I'm working on now. I'll write a bit about what I'm working on to justify an eval BiF. :) I'm reworking most of the notice framework to make it simpler and make it's features more well defined, mostly by removed the notice action filters. The notice action filters are confusing because they are applied after the notice policy but that's effectively creating two different ways for actions to be applied to notices but it just makes things confusing in the end. I think the notice action filters were originally created as a shorthand way to provide broad brush strokes to the notice policy by directing entire notice types toward certain actions. If we had an eval function, I could essentially replicate this functionality in the notice policy which would keep everything much clearer. As things stand now (before I do a commit), here's how it looks to filter out entire notice types: redef Notice::action_filters += { [[DPD::ProtocolViolation]] = Notice::ignore_action, }; I would like to do something like this... redef Notice::policy += { Notice::ignore_it(DPD::ProtocolViolation), }; The problem is that to implement the ignore_it function requires me to dynamically implement the predicate for the notice policy which Bro doesn't currently like. I tried implementing it similarly to this: function ignore_it(nt: Notice::Type): Notice::PolicyItem { return [$result=ACTION_IGNORE, $pred(n: Notice::Info) = { return n$note == nt; }, $priority=5]; } I think the problem comes up when Bro is interpreting "n$note == nt". It isn't pulling the values from the correct namespace. If I could call an eval function, I could implement it this way: function ignore_it(nt: Notice::Type): Notice::PolicyItem { eval(cat("return [$result=ACTION_IGNORE, $pred(n: Notice::Info) = { return n$note == ", nt, "; }, $priority=5]"); } I think it would be fairly reasonable for eval to only work at init time too. It scares me to think of eval being usable at runtime. Sorry for the long email, but any thoughts? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Wed Jun 15 12:54:04 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 15 Jun 2011 12:54:04 -0700 Subject: [Bro-Dev] istate.events-ssl test In-Reply-To: <0acc6fbf-e12c-46b9-9b2f-7e07c460f31b@tangent.ncsa.illinois.edu> References: <20110610205212.GG9550@icir.org> <0acc6fbf-e12c-46b9-9b2f-7e07c460f31b@tangent.ncsa.illinois.edu> Message-ID: <20110615195404.GD42975@icir.org> (This was supposed to have been sent already ...) On Mon, Jun 13, 2011 at 12:31 -0500, you wrote: > The CA cert that the test is using expired last March. (see output at bottom). D'oh! That's something Bro should report a bit more directly I guess ... > I tried generating my own keys and replacing what was in the test and > the receiver looks like it saw everything (again output below). Yes, looks good. It should match the output from the clear-text version of the test. If that's the case, feel free to commit. Thanks! Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From vern at icir.org Wed Jun 15 19:24:50 2011 From: vern at icir.org (Vern Paxson) Date: Wed, 15 Jun 2011 19:24:50 -0700 Subject: [Bro-Dev] eval? In-Reply-To: <44615C2B-E6AB-4CFB-866A-E4473EF0C934@icir.org> (Wed, 15 Jun 2011 15:15:14 EDT). Message-ID: <20110616022450.3624836A383@taffy.ICSI.Berkeley.EDU> Yuck. This strikes me as definitely going down the wrong path - making code less clear rather than more clear. > redef Notice::action_filters += { > [[DPD::ProtocolViolation]] = Notice::ignore_action, > }; > > I would like to do something like this... > > redef Notice::policy += { > Notice::ignore_it(DPD::ProtocolViolation), > }; I'm not understanding how the second is better than the first. Indeed, now I have to go look at how ignore_it works to know what's going on. > The problem is that to implement the ignore_it function requires me to dynamically implement the predicate for the notice policy which Bro doesn't currently like. I tried implementing it similarly to this: > > function ignore_it(nt: Notice::Type): Notice::PolicyItem > { > return [$result=ACTION_IGNORE, > $pred(n: Notice::Info) = { return n$note == nt; }, > $priority=5]; > } Here it seems if you must go this route, the right fix is either a notion of currying (or partial application) of functions, or to have a single PolicyItem for ignoring that looks up n$note in a table, and just add DPD::ProtocolViolation to that table. Vern From seth at icir.org Wed Jun 15 20:36:16 2011 From: seth at icir.org (Seth Hall) Date: Wed, 15 Jun 2011 23:36:16 -0400 Subject: [Bro-Dev] eval? In-Reply-To: <20110616022450.3624836A383@taffy.ICSI.Berkeley.EDU> References: <20110616022450.3624836A383@taffy.ICSI.Berkeley.EDU> Message-ID: <49E0F1B4-E06F-48F4-8B3A-D4F6B211B199@icir.org> On Jun 15, 2011, at 10:24 PM, Vern Paxson wrote: > This strikes me as definitely going down the wrong path - making code less > clear rather than more clear. Heh, exactly why I was looking for feedback. :) > I'm not understanding how the second is better than the first. Indeed, > now I have to go look at how ignore_it works to know what's going on. Syntactically it wasn't really any better. I guess I wasn't very clear in my email earlier. What it was doing was clearing up a somewhat ambiguously defined area internally in the notice code by getting rid of the action_filters since the notice policy does essentially the same thing but in a different way. Having both methods can bite you because you have to know the internal organization of the notice framework to effectively use it. > Here it seems if you must go this route, the right fix is either a notion > of currying (or partial application) of functions, or to have a single > PolicyItem for ignoring that looks up n$note in a table, and just add > DPD::ProtocolViolation to that table. That's exactly what I ended up doing right after I sent the email. :) I'm almost certain I've still missed some use case for the notice framework or made some odd design decisions (I think I'm well known for those!), but I'll fix everything once the script gets reviewed more closely. Thanks for trudging through that whole email. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Thu Jun 16 00:19:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 16 Jun 2011 07:19:33 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: ZegeReubs Message-ID: <20110616071930.D696136A38A@taffy.ICSI.Berkeley.EDU> New user registration for user ZegeReubs -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jun 16 10:49:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 16 Jun 2011 17:49:15 -0000 Subject: [Bro-Dev] #476: Problem with logging &optional subrecords Message-ID: <046.ce37cfe251f1c1334a8ac34e5dbf521f@tracker.bro-ids.org> #476: Problem with logging &optional subrecords ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Keywords: logging | ---------------------+-------------------- The logging framework has a problem when you try to log a subrecord that is &optional. No errors are generated, but the logging framework just drops the log line completely. In the code below, only the second Log::write call works. {{{ redef enum Log::ID += { TESTING }; type Foo: record { val1: count; val2: count; } &log; type Bar: record { a: Foo &log &optional; b: count &log; }; event bro_init() { Log::create_stream(TESTING, [$columns=Bar]); # This doesn't print anything to the log. Log::write(TESTING, [$b=6]); # This works. Log::write(TESTING, [$a=[$val1=1,$val2=2], $b=3]); } }}} This code prints out a file containing the following: {{{ # a.val1 a.val2 b 1 2 3 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Thu Jun 16 11:03:52 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 16 Jun 2011 13:03:52 -0500 (CDT) Subject: [Bro-Dev] script loading implementation In-Reply-To: <5e512a3f-4893-4e7e-9dab-ae659aa51979@tangent.ncsa.illinois.edu> Message-ID: <94c09c3b-a21c-4f0c-8561-2c725cb96f48@tangent.ncsa.illinois.edu> With the way the new policy scripts use (abuse?) loading scripts from subdirectories of BROPATH, it seems easier to confuse the mechanism by which the scanner knows whether a given script has already been loaded. Consider a policy tree like: . ??? A ??? ??? B ??? ??? script.bro ??? other scripts/directories (e.g. loader.bro) If BROPATH contains ".:A" (the policy root plus A/), then "script.bro" could potentially be loaded twice (thus causing errors) when both the "A/B/script" and "B/script" input forms are @load'd either from other scripts or from user input. The common pitfall would probably be when "loader.bro" is some script that always gets loaded by default and itself does a "@load B/script". A user attempting to run bro will see errors when they try `bro A/B/script` (or even `bro B/script.bro`), but `bro B/script` will work. I want to change it so that: 1) all paths branching off BROPATH that lead to the script-being-loaded should be considered as loaded 2) inputs to @loads (and user arguments) without a .bro extension should also consider the .bro input form as loaded (currently it only does the reverse) Does this sound good, or were there any other ideas to simplify the script loading process (the current scan.l already seems to be somewhat complicated in the way this happens) ? - Jon From seth at icir.org Thu Jun 16 12:05:45 2011 From: seth at icir.org (Seth Hall) Date: Thu, 16 Jun 2011 15:05:45 -0400 Subject: [Bro-Dev] script loading implementation In-Reply-To: <94c09c3b-a21c-4f0c-8561-2c725cb96f48@tangent.ncsa.illinois.edu> References: <94c09c3b-a21c-4f0c-8561-2c725cb96f48@tangent.ncsa.illinois.edu> Message-ID: On Jun 16, 2011, at 2:03 PM, Jonathan Siwek wrote: > 1) all paths branching off BROPATH that lead to the script-being-loaded should be considered as loaded I'm not sure I understand what you're saying here. > 2) inputs to @loads (and user arguments) without a .bro extension should also consider the .bro input form as loaded (currently it only does the reverse) Does that gain much benefit? Wouldn't the file with the same name as a directory already be inaccessible due to path ordering? > Does this sound good, or were there any other ideas to simplify the script loading process (the current scan.l already seems to be somewhat complicated in the way this happens) ? I've been thinking about this a bit some too and I've was considering removing the subpaths from the default BROPATH. It would make it so that if you wanted to load the core http analysis, you'd have to load protocols/http. I know that doesn't really address what you're talking about here, but it should help avoid any potential path problems in the default install. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Thu Jun 16 12:43:47 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 16 Jun 2011 14:43:47 -0500 (CDT) Subject: [Bro-Dev] script loading implementation In-Reply-To: Message-ID: <835712be-cc54-4810-9953-38c214faf2b0@tangent.ncsa.illinois.edu> > > 1) all paths branching off BROPATH that lead to the > > script-being-loaded should be considered as loaded > > I'm not sure I understand what you're saying here. If I have a script: policy/A/B/script.bro and a BROPATH containing: policy:policy/A then when a script is loaded via "@load B/script", the scanner needs to consider both "B/script" and "A/B/script" as already loaded (they're the same file, but we need to know all possible subpaths of paths in BROPATH that can lead to the file). Another way to do it could be to canonicalize inputs to the shortest subpath of a path in BROPATH that can load the script. If you want a real example to play around with, try figuring out the right way to load "policy/protocols/conn/base.bro" as a command line argument to bro in the policy-scripts-new branch. (pretend you don't know that the script actually gets loaded by default because of some crazy packet-filter -> notice -> conn dependencies) Here's the order of attempts I made fail: `./src/bro protocols/conn/base.bro` fail: `./src/bro protocols/conn/base` success: `./src/bro conn/base` > > 2) inputs to @loads (and user arguments) without a .bro extension > > should also consider the .bro input form as loaded (currently it > > only does the reverse) > > Does that gain much benefit? Wouldn't the file with the same name as > a directory already be inaccessible due to path ordering? This doesn't have to do with scripts and directories having the same name. This has to do with why this succeeds: `./src/bro conn/base` while this fails: `./src/bro conn/base.bro` > I've been thinking about this a bit some too and I've was considering > removing the subpaths from the default BROPATH. It would make it so > that if you wanted to load the core http analysis, you'd have to > load protocols/http. I know that doesn't really address what you're > talking about here, but it should help avoid any potential path > problems in the default install. Actually, I think that would address the first problem I'm talking about (not the second regarding .bro extension canonicalization). But seems like something someone might try to do again the future only to run into the same problems I did. - Jon From robin at icir.org Thu Jun 16 16:45:56 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 16 Jun 2011 16:45:56 -0700 Subject: [Bro-Dev] script loading implementation In-Reply-To: <835712be-cc54-4810-9953-38c214faf2b0@tangent.ncsa.illinois.edu> References: <835712be-cc54-4810-9953-38c214faf2b0@tangent.ncsa.illinois.edu> Message-ID: <20110616234556.GH35178@icir.org> Yeah, I think your examples show that there's a problem. I'm not sure I fully remember how the @load mechanism currently tracks what's loaded, but could it just generally canonicalize all scripts to their full *absolute* paths of the actual files being loaded (i.e., always including the .bro extensions) and then remember those? We do the same before loading a script and if the absolute path is already known, we don't load it again. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Jun 16 17:29:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 17 Jun 2011 00:29:13 -0000 Subject: [Bro-Dev] #476: Problem with logging &optional subrecords In-Reply-To: <046.ce37cfe251f1c1334a8ac34e5dbf521f@tracker.bro-ids.org> References: <046.ce37cfe251f1c1334a8ac34e5dbf521f@tracker.bro-ids.org> Message-ID: <061.2def802b01c33196fb73b5cae07836d6@tracker.bro-ids.org> #476: Problem with logging &optional subrecords -----------------------------+--------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: logging -----------------------------+--------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Jun 16 17:34:59 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 16 Jun 2011 17:34:59 -0700 Subject: [Bro-Dev] eval? In-Reply-To: <44615C2B-E6AB-4CFB-866A-E4473EF0C934@icir.org> References: <44615C2B-E6AB-4CFB-866A-E4473EF0C934@icir.org> Message-ID: <20110617003459.GL35178@icir.org> On Wed, Jun 15, 2011 at 15:15 -0400, you wrote: > I imagine that an eval BiF would be pretty easy to write For the record, I don't think this would be easy at all. :) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Fri Jun 17 06:23:12 2011 From: seth at icir.org (Seth Hall) Date: Fri, 17 Jun 2011 09:23:12 -0400 Subject: [Bro-Dev] eval? In-Reply-To: <20110617003459.GL35178@icir.org> References: <44615C2B-E6AB-4CFB-866A-E4473EF0C934@icir.org> <20110617003459.GL35178@icir.org> Message-ID: <30A75D42-D993-4F40-AA00-67D3D9731A3C@icir.org> On Jun 16, 2011, at 8:34 PM, Robin Sommer wrote: > For the record, I don't think this would be easy at all. :) Hah, ok. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Fri Jun 17 07:35:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 17 Jun 2011 14:35:14 -0000 Subject: [Bro-Dev] #477: Signing releases Message-ID: <047.31cdcff14734fc4bfcbdff3de17930a5@tracker.bro-ids.org> #477: Signing releases --------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | --------------------+-------------------- We should start GPG-signing Bro releases. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Jun 17 08:05:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 17 Jun 2011 15:05:42 -0000 Subject: [Bro-Dev] #477: Signing releases In-Reply-To: <047.31cdcff14734fc4bfcbdff3de17930a5@tracker.bro-ids.org> References: <047.31cdcff14734fc4bfcbdff3de17930a5@tracker.bro-ids.org> Message-ID: <062.23b553ecbf2d6699c1807d80304f80ea@tracker.bro-ids.org> #477: Signing releases ---------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ---------------------+-------------------- Comment (by jsiwek): Establishing trust in a PGP public key isn't always a lightweight operation for users, right? If (maybe additionally) the webserver hosting releases has an SSL cert. issued by a commonly trusted CA, users will find it easier to just download them or verify checksums over HTTPS. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Fri Jun 17 08:23:47 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Fri, 17 Jun 2011 10:23:47 -0500 (CDT) Subject: [Bro-Dev] script loading implementation In-Reply-To: <20110616234556.GH35178@icir.org> Message-ID: <32a4e58e-a482-4b70-b72f-6ba7fb8caba3@tangent.ncsa.illinois.edu> > I'm not sure I fully remember how the @load mechanism currently > tracks what's loaded, It looks like it tracks whatever string is given as an argument to @load (or as an arg to bro), and if it ends in .bro, it also tracks the version with that stripped. > but could it just generally canonicalize all scripts > to their full *absolute* paths of the actual files being loaded > (i.e., always including the .bro extensions) and then remember those? Yes, I think that works best/easiest. I can work on this. - Jon From bro at tracker.bro-ids.org Fri Jun 17 08:34:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 17 Jun 2011 15:34:01 -0000 Subject: [Bro-Dev] #477: Signing releases In-Reply-To: <047.31cdcff14734fc4bfcbdff3de17930a5@tracker.bro-ids.org> References: <047.31cdcff14734fc4bfcbdff3de17930a5@tracker.bro-ids.org> Message-ID: <062.dd2d34e2f121915674df33d61628ea4c@tracker.bro-ids.org> #477: Signing releases ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by robin): * version: => git/master * type: Task => Problem Comment: On Fri, Jun 17, 2011 at 15:05 -0000, you wrote: > Establishing trust in a PGP public key isn't always a lightweight > operation for users, right? Yeah, we should at least all sign they key with out private ones. > If (maybe additionally) the webserver hosting releases has an SSL > cert. issued by a commonly trusted CA, users will find it easier to > just download them or verify checksums over HTTPS. That's a good idea in any case, we should generally offer SSL access. Does anyone have experience with getting a trustworthy cert and can take the lead on that? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Jun 17 08:37:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 17 Jun 2011 15:37:30 -0000 Subject: [Bro-Dev] #477: Signing releases In-Reply-To: <047.31cdcff14734fc4bfcbdff3de17930a5@tracker.bro-ids.org> References: <047.31cdcff14734fc4bfcbdff3de17930a5@tracker.bro-ids.org> Message-ID: <062.cd080ce4a0aad6aa82f9cc1e63d3fd64@tracker.bro-ids.org> #477: Signing releases ---------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ---------------------+-------------------- Changes (by robin): * version: git/master => * type: Problem => Task -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Fri Jun 17 14:54:08 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Fri, 17 Jun 2011 16:54:08 -0500 (CDT) Subject: [Bro-Dev] script loading implementation In-Reply-To: <32a4e58e-a482-4b70-b72f-6ba7fb8caba3@tangent.ncsa.illinois.edu> Message-ID: > > but could it just generally canonicalize all scripts > > to their full *absolute* paths of the actual files being loaded > > (i.e., always including the .bro extensions) and then remember > > those? Err, I don't think it's trivial in C/C++ without extra libraries (e.g. Boost) to get absolute paths from relative ones (we can't rely just on BROPATH since that may also have relative paths in it) let alone compare that two lexical file representations actual resolve to the same file (duplicate '/', embedded './'s, etc.). I think it works to just track/remember inode numbers, though. - Jon From robin at icir.org Fri Jun 17 15:26:37 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 17 Jun 2011 15:26:37 -0700 Subject: [Bro-Dev] script loading implementation In-Reply-To: References: <32a4e58e-a482-4b70-b72f-6ba7fb8caba3@tangent.ncsa.illinois.edu> Message-ID: <20110617222637.GC75506@icir.org> On Fri, Jun 17, 2011 at 16:54 -0500, you wrote: > I think it works to just track/remember inode numbers, though. Yeah, nice thought, that should work as well (and even catches hard links :). For getting an absolute path, there's realpath(). However, the glibc man page says: "Avoid using this function. It is broken by design ..." Perhaps there are other C versions of that function out there (with a suitable licence) that are safer to use and we could borrrow? (In fact, the glibc version has a non-standard feature that avoids the problem it seems.) But either way seems fine. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Sat Jun 18 00:17:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 18 Jun 2011 07:17:48 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: vierniretot Message-ID: <20110618071746.4B3FB36A3AB@taffy.ICSI.Berkeley.EDU> New user registration for user vierniretot -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Jun 18 01:26:10 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 18 Jun 2011 08:26:10 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: tkwkmooesk Message-ID: <20110618082608.3215236A3AB@taffy.ICSI.Berkeley.EDU> New user registration for user tkwkmooesk -- Bro Tracker Bro Issue Tracker From gregor at icir.org Sat Jun 18 09:46:21 2011 From: gregor at icir.org (Gregor Maier) Date: Sat, 18 Jun 2011 09:46:21 -0700 Subject: [Bro-Dev] script loading implementation In-Reply-To: References: Message-ID: <4DFCD65D.2040008@icir.org> Instead of going for the path you can go for the inode of the file. See man fstat(2). That will even take care of links. cu Gregor On 6/17/11 14:54 , Jonathan Siwek wrote: >>> but could it just generally canonicalize all scripts >>> to their full *absolute* paths of the actual files being loaded >>> (i.e., always including the .bro extensions) and then remember >>> those? > > Err, I don't think it's trivial in C/C++ without extra libraries (e.g. Boost) to get absolute paths from relative ones (we can't rely just on BROPATH since that may also have relative paths in it) let alone compare that two lexical file representations actual resolve to the same file (duplicate '/', embedded './'s, etc.). > > I think it works to just track/remember inode numbers, though. > > - Jon > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Sat Jun 18 10:19:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 18 Jun 2011 17:19:28 -0000 Subject: [Bro-Dev] #478: Move BinPAC docs over to new server Message-ID: <047.826126acda25dd38cf616630391698f4@tracker.bro-ids.org> #478: Move BinPAC docs over to new server ----------------------------+-------------------- Reporter: robin | Owner: seth Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Website / Wiki | Version: Keywords: | ----------------------------+-------------------- There's some BinPAC documentation in the old Wiki that we should move over. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Mon Jun 20 07:15:09 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 20 Jun 2011 09:15:09 -0500 (CDT) Subject: [Bro-Dev] script loading implementation In-Reply-To: <4DFCD65D.2040008@icir.org> Message-ID: <2021302578.955.1308579309286.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > Instead of going for the path you can go for the inode of the file. > See man fstat(2). That will even take care of links. This is what I ended up doing on Friday, just didn't commit yet because I was trying to figure out why one of the unit tests failed w/ that change. Turns out the baseline was just wrong because the way the test file was named "conn-id.bro", it was loaded twice since the old path comparison method failed to detect a duplicate. - Jon From bro at tracker.bro-ids.org Mon Jun 20 10:32:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 20 Jun 2011 17:32:01 -0000 Subject: [Bro-Dev] #479: topic/jsiwek/pybroccoli-fixes Message-ID: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> #479: topic/jsiwek/pybroccoli-fixes -----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | -----------------------------+------------------------ This branch exists in both the `broccoli-python` and `bro` repositories and changes the following: * Record instances sent using the broccoli python bindings no longer have to initialize all fields defined in the record_type and also now send field names to which values correspond * Fix record instantiation in broccoli python bindings from taking on the type of the last-declared record_type instead of the correct type * Before calling local event handlers, Bro coerces any of their record arguments to the correct record type (so a peer may only send a subset of &optional fields) * Fix the pybroccoli unit test from being skipped on MacOS -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jun 20 10:32:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 20 Jun 2011 17:32:17 -0000 Subject: [Bro-Dev] #479: topic/jsiwek/pybroccoli-fixes In-Reply-To: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> References: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> Message-ID: <063.ca5dd7dec313001204ee17993e91da63@tracker.bro-ids.org> #479: topic/jsiwek/pybroccoli-fixes ----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Feature Request => Merge Request -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jun 20 10:39:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 20 Jun 2011 17:39:15 -0000 Subject: [Bro-Dev] #480: topic/jsiwek/script-load-cleanup Message-ID: <048.71712540bb3ceafffd47b237c36c1930@tracker.bro-ids.org> #480: topic/jsiwek/script-load-cleanup ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This branch changes the way bro detects whether a script has already been loaded from comparing path names to comparing inode numbers. Comparing path strings was too easy to break (unit tests have been created). The old `core.conn-id` unit test's baseline had to be updated since it erroneously loaded twice with the old path comparison method. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Tue Jun 21 06:18:57 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 21 Jun 2011 06:18:57 -0700 Subject: [Bro-Dev] IP6 addresses Message-ID: <20110621131857.GC35615@icir.org> Bro's code currently has a number of "#ifdef IPv6" blocks that I'd really like to get rid of. It's a pain to support two different compile-time modes, and setups where IPv6 isn't used at all are becoming more and more rare. I think the main internal changes for that would be (1) making addresses generally 128-bit wide, and (2) encapsulating the logic for manipulating addresses in its own class, something like IPAddr, or so. I'd like to hear if folks agree with that? (1) clearly has impact on memory usage, but I'd say that's ok these days. Alternatively, we could use (2) to do some clever scheme that stores less bytes for IPv4 addresses than for IPv6; but that would then quite likely have in turn an impact on CPU performance and I don't see that that'd be worth it. Thoughts? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Tue Jun 21 07:23:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 21 Jun 2011 14:23:49 -0000 Subject: [Bro-Dev] #481: topic/seth/net-stats-bif - New net_stats BiF Message-ID: <046.53c17b9462e3c4e7868a29afdc56ebec@tracker.bro-ids.org> #481: topic/seth/net-stats-bif - New net_stats BiF ---------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ New net_stats BiF to request network statistics on demand instead of using the old heartbeat_interval method. - Updated the scripts to use the new BiF. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Tue Jun 21 07:36:03 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 21 Jun 2011 09:36:03 -0500 (CDT) Subject: [Bro-Dev] IP6 addresses In-Reply-To: <20110621131857.GC35615@icir.org> Message-ID: <155b2533-60ee-43b8-82d4-b4d2a95e8607@tangent.ncsa.illinois.edu> > I'd like to hear if folks agree with that? (1) clearly has impact on > memory usage, but I'd say that's ok these days. Alternatively, we > could use (2) to do some clever scheme that stores less bytes for > IPv4 addresses than for IPv6; but that would then quite likely have in > turn an impact on CPU performance and I don't see that that'd be worth it. If there's any demand for this alternative, an idea I had was that there could probably be a compile-time flag to switch between the new IPAddr implementations. But generally I think I'm with you on just doing the consistent 128-bit width at cost of memory. - Jon From bro at tracker.bro-ids.org Tue Jun 21 10:02:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 21 Jun 2011 17:02:36 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: offesycle Message-ID: <20110621170235.D982A36A00A@taffy.ICSI.Berkeley.EDU> New user registration for user offesycle -- Bro Tracker Bro Issue Tracker From gregor at icir.org Tue Jun 21 12:54:18 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 21 Jun 2011 12:54:18 -0700 Subject: [Bro-Dev] IP6 addresses In-Reply-To: <20110621131857.GC35615@icir.org> References: <20110621131857.GC35615@icir.org> Message-ID: <4E00F6EA.7010808@icir.org> On 6/21/11 6:18 , Robin Sommer wrote: > Bro's code currently has a number of "#ifdef IPv6" blocks that I'd > really like to get rid of. It's a pain to support two different > compile-time modes, and setups where IPv6 isn't used at all are > becoming more and more rare. I agree. I think IPv6 should be enabled by default. > I think the main internal changes for that would be (1) making > addresses generally 128-bit wide, and (2) encapsulating the logic for > manipulating addresses in its own class, something like IPAddr, or so. I think that the memory impact for (1) is going to be a lot worse than you anticipate. A long time ago I did a benchmark and found that enabling IPv6 doubles the memory footprint. See: http://tracker.bro-ids.org/bro/ticket/68 Note, that the Analyzer tree already uses a scheme similar to (2) for parsing IP headers and making the fields available (see class IP_Hdr) > I'd like to hear if folks agree with that? (1) clearly has impact on > memory usage, but I'd say that's ok these days. Alternatively, we > could use (2) to do some clever scheme that stores less bytes for IPv4 > addresses than for IPv6; but that would then quite likely have in turn > an impact on CPU performance and I don't see that that'd be worth it. Therefore, I would opt for using some "clever scheme" that stores less bytes for IPv4. If we are going to use a class for manipulating IP addresses, then this alone is going to cause CPU overhead. I don't think that the additional overhead of implementing the scheme to save bytes for IPv4-only addresses is going to be significant. OTOH, using such an IPAddr class is already going to increase the memory footprint (now we have to allocate a class) significantly. Possibly more overhead then just enabling IPv6 (*) Also note, that most IP addresses are probably going to be used on the policy layer. Maybe we find a clever way of making those as efficient as possible (memory and space wise), while always using 128-bits in the C++ layer. cu Gregor (*) when IPv6 is enabled AddrVal stores a uint32 *. When it's instantiated a array with 4 uint32's is allocated. When IPv4-only is used, AddrVal stores the address as a uint32. -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From gregor at icir.org Tue Jun 21 12:55:22 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 21 Jun 2011 12:55:22 -0700 Subject: [Bro-Dev] IP6 addresses In-Reply-To: <155b2533-60ee-43b8-82d4-b4d2a95e8607@tangent.ncsa.illinois.edu> References: <155b2533-60ee-43b8-82d4-b4d2a95e8607@tangent.ncsa.illinois.edu> Message-ID: <4E00F72A.60805@icir.org> On 6/21/11 7:36 , Jonathan Siwek wrote: >> I'd like to hear if folks agree with that? (1) clearly has impact on >> memory usage, but I'd say that's ok these days. Alternatively, we >> could use (2) to do some clever scheme that stores less bytes for >> IPv4 addresses than for IPv6; but that would then quite likely have in >> turn an impact on CPU performance and I don't see that that'd be worth it. > > If there's any demand for this alternative, an idea I had was that there could probably be a compile-time flag to switch between the new IPAddr implementations. But generally I think I'm with you on just doing the consistent 128-bit width at cost of memory. I wouldn't do any compile time flags. That's what we have right now and it isn't pretty at all. cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Tue Jun 21 12:57:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 21 Jun 2011 19:57:03 -0000 Subject: [Bro-Dev] #68: BroV6 support -- memory usage In-Reply-To: <048.a7ea7982ce037ed963ed20f3871939b0@tracker.bro-ids.org> References: <048.a7ea7982ce037ed963ed20f3871939b0@tracker.bro-ids.org> Message-ID: <063.2a8f7f0bf3d8abf437c8956e01da9bee@tracker.bro-ids.org> #68: BroV6 support -- memory usage ---------------------+------------------------------------ Reporter: gregor | Owner: Type: Task | Status: seen Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: 1.5.2 Resolution: | Keywords: BroV6, binpac++, HILTI ---------------------+------------------------------------ Comment (by gregor): Replying to [comment:2 robin]: > Additional note: the connection compressor doesn't support v6 at the moment, and is thus skipped for all v6 connections. That certainly also contributes to the memory overhead. the trace I used at that time actually didn't contain any IPv6, so the connection compressor can't be the reason for the memory overhead. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Tue Jun 21 13:25:19 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 21 Jun 2011 15:25:19 -0500 (CDT) Subject: [Bro-Dev] IP6 addresses In-Reply-To: <4E00F72A.60805@icir.org> Message-ID: > > If there's any demand for this alternative, an idea I had was that > > there could probably be a compile-time flag to switch between the > > new IPAddr implementations. But generally I think I'm with you on > > just doing the consistent 128-bit width at cost of memory. > > I wouldn't do any compile time flags. That's what we have right now > and it isn't pretty at all. I probably wasn't clear, so restating in case: I meant that I agreed with getting rid of the compile-time flag to toggle IPv6 support and just supporting it by default. But a potential new compile-time flag could be offered to let users choose between the two different approaches Robin was talking about if we can't decide on one that's clearly better (CPU vs. memory tradeoff). i.e. maybe it has to be implemented and profiled both ways anyway before a decision is made. - Jon From gregor at icir.org Tue Jun 21 14:16:34 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 21 Jun 2011 14:16:34 -0700 Subject: [Bro-Dev] IP6 addresses In-Reply-To: References: Message-ID: <4E010A32.4020809@icir.org> >> I wouldn't do any compile time flags. That's what we have right now >> and it isn't pretty at all. > > I probably wasn't clear, so restating in case: I meant that I agreed with getting rid of the compile-time flag to toggle IPv6 support and just supporting it by default. But a potential new compile-time flag could be offered to let users choose between the two different approaches Robin was talking about if we can't decide on one that's clearly better (CPU vs. memory tradeoff). i.e. maybe it has to be implemented and profiled both ways anyway before a decision is made. Yeah. That's how I understood it. My concern is that we replace one compile time flag with another. While we would have IPv6 support for all builds then, we still have #ifdefs in the code and two different code paths that need to be tested and maintained. So I don't see a real improvement over the current situation there. OTOH, I agrees that we might want to / have to implement both approach and profile them to see what kind of overhead we'll actually have. cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Tue Jun 21 14:23:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 21 Jun 2011 21:23:50 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: Fonkilzaridos Message-ID: <20110621212349.B456F36A363@taffy.ICSI.Berkeley.EDU> New user registration for user Fonkilzaridos -- Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Tue Jun 21 14:29:45 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 21 Jun 2011 16:29:45 -0500 (CDT) Subject: [Bro-Dev] IP6 addresses In-Reply-To: <4E010A32.4020809@icir.org> Message-ID: > My concern is that we replace one > compile time flag with another. While we would have IPv6 support for > all builds then, we still have #ifdefs in the code and two different code > paths that need to be tested and maintained. Yeah, since you mention the burden of testing, I'm with you: we should arrive at a single implementation choice. - Jon From robin at icir.org Tue Jun 21 12:03:24 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 21 Jun 2011 12:03:24 -0700 Subject: [Bro-Dev] IP6 addresses In-Reply-To: <155b2533-60ee-43b8-82d4-b4d2a95e8607@tangent.ncsa.illinois.edu> References: <20110621131857.GC35615@icir.org> <155b2533-60ee-43b8-82d4-b4d2a95e8607@tangent.ncsa.illinois.edu> Message-ID: <20110621190324.GD6039@icir.org> On Tue, Jun 21, 2011 at 09:36 -0500, you wrote: > If there's any demand for this alternative, an idea I had was that > there could probably be a compile-time flag to switch between the new > IPAddr implementations. I'd like to avoid any compile-time option. That just makes it harder to maintain and test the code as there are always two versions to consider. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Tue Jun 21 18:07:39 2011 From: robin at icir.org (Robin Sommer) Date: Tue, 21 Jun 2011 18:07:39 -0700 Subject: [Bro-Dev] IP6 addresses In-Reply-To: <4E00F6EA.7010808@icir.org> References: <20110621131857.GC35615@icir.org> <4E00F6EA.7010808@icir.org> Message-ID: <20110622010739.GO71991@icir.org> On Tue, Jun 21, 2011 at 12:54 -0700, you wrote: > http://tracker.bro-ids.org/bro/ticket/68 Hmm ... Ok, I didn't remember it to be *that* bad. > Therefore, I would opt for using some "clever scheme" that stores less > bytes for IPv4. If we are going to use a class for manipulating IP > addresses, then this alone is going to cause CPU overhead. I don't think > that the additional overhead of implementing the scheme to save bytes > for IPv4-only addresses is going to be significant. > > OTOH, using such an IPAddr class is already going to increase the memory > footprint (now we have to allocate a class) significantly. Possibly more > overhead then just enabling IPv6 (*) This is all hard to predict and also depends on how we implement it (using a class by itself doesn't necessarily mean more CPU and/or memory; it may though). So seems we indeed need measurements with different appraoches here. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Tue Jun 21 21:37:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 04:37:58 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: lollugbub Message-ID: <20110622043756.E771436A3A3@taffy.ICSI.Berkeley.EDU> New user registration for user lollugbub -- Bro Tracker Bro Issue Tracker From seth at icir.org Wed Jun 22 04:06:34 2011 From: seth at icir.org (Seth Hall) Date: Wed, 22 Jun 2011 07:06:34 -0400 Subject: [Bro-Dev] IP6 addresses In-Reply-To: <20110622010739.GO71991@icir.org> References: <20110621131857.GC35615@icir.org> <4E00F6EA.7010808@icir.org> <20110622010739.GO71991@icir.org> Message-ID: On Jun 21, 2011, at 9:07 PM, Robin Sommer wrote: > > On Tue, Jun 21, 2011 at 12:54 -0700, you wrote: > >> http://tracker.bro-ids.org/bro/ticket/68 > > Hmm ... Ok, I didn't remember it to be *that* bad. That looks pretty similar to my experience recently running on live traffic with v6 enabled and then disabling it and running again. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Wed Jun 22 07:16:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 14:16:38 -0000 Subject: [Bro-Dev] #200: broctl update resets/corrupts certain variables In-Reply-To: <048.70165624d985998c153024ccf3bf0cdb@tracker.bro-ids.org> References: <048.70165624d985998c153024ccf3bf0cdb@tracker.bro-ids.org> Message-ID: <063.d8a31c0ad9e3686bbec084fff0c60803@tracker.bro-ids.org> #200: broctl update resets/corrupts certain variables -----------------------------+--------------------------- Reporter: justin | Owner: seth Type: Fastpath | Status: closed Priority: Low | Milestone: Bro1.6 Component: BroControl | Version: 1.5.2 Resolution: Solved/Applied | Keywords: broctl update -----------------------------+--------------------------- Changes (by seth): * status: reopened => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 07:19:18 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 14:19:18 -0000 Subject: [Bro-Dev] #80: Checking for the existance keys in multi-dimensional tables gives error message In-Reply-To: <050.c01d8a094407695f72fd3d3d599c4f48@tracker.bro-ids.org> References: <050.c01d8a094407695f72fd3d3d599c4f48@tracker.bro-ids.org> Message-ID: <065.02452af2b0d546ad409f0c588feaffdc@tracker.bro-ids.org> #80: Checking for the existance keys in multi-dimensional tables gives error message -----------------------+-------------------- Reporter: bernhard | Owner: Type: Problem | Status: seen Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: 1.4 Resolution: | Keywords: -----------------------+-------------------- Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: Pushing this back to the next release, we have too many other things that need to be finished already. It needs to be dealt with eventually, but it can be avoided by careful coding. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 07:22:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 14:22:29 -0000 Subject: [Bro-Dev] #452: Bug with event &priority In-Reply-To: <047.c21b4c0ff1b86ba91799b23686a432b6@tracker.bro-ids.org> References: <047.c21b4c0ff1b86ba91799b23686a432b6@tracker.bro-ids.org> Message-ID: <062.8171a7e11a5a97be1b539cd484fd86ae@tracker.bro-ids.org> #452: Bug with event &priority ----------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Invalid | Keywords: ----------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Invalid Comment: I can't replicate this now. I'm not sure what was going on that day. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 07:27:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 14:27:35 -0000 Subject: [Bro-Dev] #464: Bug with complex data structures In-Reply-To: <046.9f65dde0f285b762925aca06fe80721f@tracker.bro-ids.org> References: <046.9f65dde0f285b762925aca06fe80721f@tracker.bro-ids.org> Message-ID: <061.f1d6f566e4aa433cbec2fb49e9e8e7c9@tracker.bro-ids.org> #464: Bug with complex data structures ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: This needs fixed eventually, but it's not critical at the moment. Let's push it back to a later release. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 07:28:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 14:28:00 -0000 Subject: [Bro-Dev] #303: Design new website. In-Reply-To: <046.228451771d0155f114375b79382022ec@tracker.bro-ids.org> References: <046.228451771d0155f114375b79382022ec@tracker.bro-ids.org> Message-ID: <061.ff3fc7f23912cfcaa64737ae17e65f37@tracker.bro-ids.org> #303: Design new website. -----------------------------+-------------------- Reporter: seth | Owner: Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 07:28:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 14:28:38 -0000 Subject: [Bro-Dev] #302: New quick start guide In-Reply-To: <046.33ca184621c46a98331d944a318a3fb3@tracker.bro-ids.org> References: <046.33ca184621c46a98331d944a318a3fb3@tracker.bro-ids.org> Message-ID: <061.073fc5ee9332f032dafa581cabdd1515@tracker.bro-ids.org> #302: New quick start guide -----------------------------+-------------------- Reporter: seth | Owner: Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: In progress and being tweaked now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 08:50:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 15:50:14 -0000 Subject: [Bro-Dev] #368: topic/dnsttl - Patch for Reverse DNS Lookups and DNS TTL support (was: Patch for Reverse DNS Lookups and DNS TTL support) In-Reply-To: <054.ec7b9fe4a96a58e9acb9d7b759a8aea2@tracker.bro-ids.org> References: <054.ec7b9fe4a96a58e9acb9d7b759a8aea2@tracker.bro-ids.org> Message-ID: <069.6a6c3cdb30508ad558d41622d4ef1ffd@tracker.bro-ids.org> #368: topic/dnsttl - Patch for Reverse DNS Lookups and DNS TTL support ----------------------------+----------------------------- Reporter: thomas.other | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: DNS TTL Resolve ----------------------------+----------------------------- Changes (by seth): * type: Patch => Merge Request Comment: I added a branch in the git repository with the patch along with the small change of removing the ifdefs to make the BiF always abide by TTLs. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 08:52:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 15:52:36 -0000 Subject: [Bro-Dev] #349: Overly long timestamp segfault in cf In-Reply-To: <046.56407ac517a4a9a8207ec2f796d38db5@tracker.bro-ids.org> References: <046.56407ac517a4a9a8207ec2f796d38db5@tracker.bro-ids.org> Message-ID: <061.fa3601e34fde7a7f4c794854ed958a24@tracker.bro-ids.org> #349: Overly long timestamp segfault in cf -----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: bro-aux | Version: Resolution: Rejected | Keywords: -----------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Rejected Comment: This isn't our bug since we don't include cf in our distribution anymore. The current version that LBL distributes is 1.2.3 which should have this problem fixed anyway. Closing the ticket. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 09:03:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 16:03:13 -0000 Subject: [Bro-Dev] #395: Cannot add global to module if it already exists in global namespace In-Reply-To: <048.ed7c07f7b808821a854fad6bf4ed9504@tracker.bro-ids.org> References: <048.ed7c07f7b808821a854fad6bf4ed9504@tracker.bro-ids.org> Message-ID: <063.39984a2b794e2a2fe6d6051eef4b50c0@tracker.bro-ids.org> #395: Cannot add global to module if it already exists in global namespace ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: Since this generally works now (for variables and types) I'm going to push the ticket back. I'm leaving it open though because we should still address the fact that you can't access the same-named global variable or type from inside of a module that uses the same name. Here's my little test script for trying to access the global values and types: {{{ global a = 1; type Test: count; module FOO; export { type Test: string; global a = 2; global b: Test = "test"; global c: GLOBAL::Test = 5; } print a; print GLOBAL::a; print b; print c; }}} This should print out: {{{ 2 1 test c }}} But instead it currently fails with this error: {{{ ./test33.bro, line 9: error: identifier not defined: GLOBAL::Test error and ./test33.bro, line 9 (error and 5): error, arithmetic mixed with non-arithmetic ./test33.bro, line 13: error: unknown identifier GLOBAL::a, at or near "GLOBAL::a" }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 09:38:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 16:38:07 -0000 Subject: [Bro-Dev] #286: Script level varargs In-Reply-To: <046.77c367ba225ad9bc8f6ee0bce22fdf4a@tracker.bro-ids.org> References: <046.77c367ba225ad9bc8f6ee0bce22fdf4a@tracker.bro-ids.org> Message-ID: <061.5eaffa70a522ddacc46ee806bce20671@tracker.bro-ids.org> #286: Script level varargs ------------------------------+---------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Component: Bro | Version: 1.5.2 Resolution: | Keywords: language ------------------------------+---------------------- Changes (by seth): * keywords: => language -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 10:05:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 17:05:34 -0000 Subject: [Bro-Dev] #253: Can't bind to port 47760, Address already in use In-Reply-To: <056.21a044ffe4916939ea16cb28b2316505@tracker.bro-ids.org> References: <056.21a044ffe4916939ea16cb28b2316505@tracker.bro-ids.org> Message-ID: <071.4ba5134e76dff831985bc63268412cfe@tracker.bro-ids.org> #253: Can't bind to port 47760, Address already in use ------------------------------+-------------------- Reporter: Tyler.Schoenke | Owner: robin Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: 1.5.2 Resolution: | Keywords: ------------------------------+-------------------- Changes (by seth): * milestone: => Bro1.6 Comment: Let's try and get this fixed for the next release. I've run into it several time before myself. I think Tyler's right that broctl should probably print some error message at the console and maybe Bro should send an email and then shutdown if this error happens? Or at least do that if the manager gives this error. Workers shouldn't ever give this exact error since they don't open a port (they only initiate connections) and proxies are currently less critical to the entire cluster but it would be good to handle them having this error too. My main question is how do we deal with the unmanaged process that is causing the error? Maybe we just alert the admin and not try to fix it in any way? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 11:08:39 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 18:08:39 -0000 Subject: [Bro-Dev] #228: inacitvity.bro defaults should be changed In-Reply-To: <047.4cb06ff9ad63809e037a6b5543762605@tracker.bro-ids.org> References: <047.4cb06ff9ad63809e037a6b5543762605@tracker.bro-ids.org> Message-ID: <062.eae036532ed99c953e16573be594fa10@tracker.bro-ids.org> #228: inacitvity.bro defaults should be changed ---------------------+------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Component: Bro | Version: 1.5.1 Resolution: | Keywords: ---------------------+------------------- Comment (by seth): I'm porting the inactivity.bro script to the new scripts now, what do you think we should with http & smtp? I don't really understand the description. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 11:08:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 18:08:52 -0000 Subject: [Bro-Dev] #228: inacitvity.bro defaults should be changed In-Reply-To: <047.4cb06ff9ad63809e037a6b5543762605@tracker.bro-ids.org> References: <047.4cb06ff9ad63809e037a6b5543762605@tracker.bro-ids.org> Message-ID: <062.1962d1bc3d4d7b3f2a39687af2dabfce@tracker.bro-ids.org> #228: inacitvity.bro defaults should be changed ---------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.1 Resolution: | Keywords: ---------------------+-------------------- Changes (by seth): * milestone: => Bro1.6 -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 11:26:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 18:26:46 -0000 Subject: [Bro-Dev] #102: Excessive SSL_SessConIncon notices In-Reply-To: <059.889b13ae2b4069cf9cdbe5a98817ff3c@tracker.bro-ids.org> References: <059.889b13ae2b4069cf9cdbe5a98817ff3c@tracker.bro-ids.org> Message-ID: <074.4300fbee80335eb220e0b0d92f192d29@tracker.bro-ids.org> #102: Excessive SSL_SessConIncon notices -----------------------------+--------------------------------- Reporter: mccreary@? | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: branches-robin-work Resolution: Solved/Applied | Keywords: SSL_SessConIncon -----------------------------+--------------------------------- Changes (by seth): * status: seen => closed * resolution: => Solved/Applied Comment: This is no longer relevant with the new SSL analyzer. If the SSL_SessConVerMismatch notice is still desirable, we should be able to recreate this notice completely in the policy scripts but it likely won't be initially. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 11:30:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 18:30:22 -0000 Subject: [Bro-Dev] #10: Segmentation fault when running Bro with all.bro In-Reply-To: <050.cac94f4b2fdd078dea462af785a4fb9d@tracker.bro-ids.org> References: <050.cac94f4b2fdd078dea462af785a4fb9d@tracker.bro-ids.org> Message-ID: <065.dbd47e52a448cf7c3e1c20d98463a226@tracker.bro-ids.org> #10: Segmentation fault when running Bro with all.bro -----------------------------+-------------------- Reporter: vallenti | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: 1.5.2 Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: accepted => closed * resolution: => Solved/Applied Comment: With all of the core scripts being rewritten and the advent of the logging framework, I don't think this ticket is going to be particularly relevant to the 1.6 and futures releases so I'm closing it. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 11:32:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 18:32:53 -0000 Subject: [Bro-Dev] #21: $msg text in notices/alarms is ad hoc in structure In-Reply-To: <046.0428cfdf279b9e2e18f92a6351012766@tracker.bro-ids.org> References: <046.0428cfdf279b9e2e18f92a6351012766@tracker.bro-ids.org> Message-ID: <061.97b7d7f3eea863a1f3de94e94d970932@tracker.bro-ids.org> #21: $msg text in notices/alarms is ad hoc in structure ------------------------------+-------------------- Reporter: vern | Owner: Type: Feature Request | Status: closed Priority: Normal | Milestone: Component: Bro | Version: Resolution: Solved/Applied | Keywords: ------------------------------+-------------------- Changes (by seth): * status: seen => closed * resolution: => Solved/Applied Comment: Notice logging and output is being heavily reworked for the 1.6 release. I've been trying to take this ticket and discussion into consideration as I create notices and define how notices are rendered to email. I'm going to close the ticket though. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 11:38:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 18:38:47 -0000 Subject: [Bro-Dev] #20: notice handling needs to accommodate a set of actions, not just one action In-Reply-To: <046.3c03cca896ab132c2a98de8c91a0fff9@tracker.bro-ids.org> References: <046.3c03cca896ab132c2a98de8c91a0fff9@tracker.bro-ids.org> Message-ID: <061.d549bf54630cb4e2e9edbadebdd366e5@tracker.bro-ids.org> #20: notice handling needs to accommodate a set of actions, not just one action ------------------------------+------------------------------ Reporter: vern | Owner: Type: Feature Request | Status: closed Priority: Normal | Milestone: Component: Bro | Version: Resolution: Solved/Applied | Keywords: notice filtering ------------------------------+------------------------------ Changes (by seth): * status: seen => closed * resolution: => Solved/Applied Comment: This is in progress now. The notice framework should be much more straight forward this way. The only trouble I'm still running into is how to define "blockers" that block the progression of notice policy items (for instance, to completely ignore certain notice types). I'll be talking to Robin about this soon though to see if we can nail down a technique to approach this with. I'm closing this ticket now since it's another part of the script rewriting and is already underway. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 12:04:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 19:04:22 -0000 Subject: [Bro-Dev] #181: all.bro segfaults with capture-events In-Reply-To: <047.88dc58e66d267906ea55b01f8d6e480c@tracker.bro-ids.org> References: <047.88dc58e66d267906ea55b01f8d6e480c@tracker.bro-ids.org> Message-ID: <062.9528fe6f7c68c55d9ee2b4ac74742775@tracker.bro-ids.org> #181: all.bro segfaults with capture-events -----------------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: 1.5.2 Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by seth): * status: seen => closed * resolution: => Solved/Applied Comment: This problem does not present itself with the new scripts. I've cleaned up many of the circular dependencies between scripts which may be fixing the problem. Closing now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 12:10:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 19:10:49 -0000 Subject: [Bro-Dev] #104: Cluster shell does not copy certs/ subdir from spool on workers In-Reply-To: <059.2ef8849694b217d54216e822611ccdc1@tracker.bro-ids.org> References: <059.2ef8849694b217d54216e822611ccdc1@tracker.bro-ids.org> Message-ID: <074.da2bb990b76157d542078fb4bf785671@tracker.bro-ids.org> #104: Cluster shell does not copy certs/ subdir from spool on workers ------------------------------+--------------------------------- Reporter: mccreary@? | Owner: robin Type: Feature Request | Status: closed Priority: Low | Milestone: Component: BroControl | Version: branches-robin-work Resolution: Solved/Applied | Keywords: SSL certs ------------------------------+--------------------------------- Changes (by seth): * status: accepted => closed * resolution: => Solved/Applied Comment: The new SSL analyzer makes this easy. We're waiting on a fix to ticket #474 before it will fully work though. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 12:15:18 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 19:15:18 -0000 Subject: [Bro-Dev] #12: Capture from multiple interfaces In-Reply-To: <047.64011cc28bcfc8034dfa19cd40719ef4@tracker.bro-ids.org> References: <047.64011cc28bcfc8034dfa19cd40719ef4@tracker.bro-ids.org> Message-ID: <062.c0c735224935b20b8ee8083f1dce9d61@tracker.bro-ids.org> #12: Capture from multiple interfaces ------------------------------+---------------------- Reporter: robin | Owner: robin Type: Feature Request | Status: accepted Priority: Low | Milestone: Component: BroControl | Version: 1.5.2 Resolution: | Keywords: ------------------------------+---------------------- Comment (by seth): Any chance this could be a candidate for 1.6? I wouldn't place it at a very priority due to the fact that someone could just bridge the interfaces together if they need to sniff two interfaces. Honestly, that's probably what they should be doing anyway instead of doing it within Bro. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 12:19:08 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 19:19:08 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: Lauccidaorado Message-ID: <20110622191907.8D10F36A3C4@taffy.ICSI.Berkeley.EDU> New user registration for user Lauccidaorado -- Bro Tracker Bro Issue Tracker From jones at tacc.utexas.edu Wed Jun 22 13:00:50 2011 From: jones at tacc.utexas.edu (William Jones) Date: Wed, 22 Jun 2011 15:00:50 -0500 Subject: [Bro-Dev] #12: Capture from multiple interfaces In-Reply-To: <062.c0c735224935b20b8ee8083f1dce9d61@tracker.bro-ids.org> References: <047.64011cc28bcfc8034dfa19cd40719ef4@tracker.bro-ids.org> <062.c0c735224935b20b8ee8083f1dce9d61@tracker.bro-ids.org> Message-ID: >From my node.cfg file: [worker-1] type=worker host=mojo1.tacc.utexas.edu interface=eth1.600 -i eth0.600 -i eth1.521 -i eth0.521 -i eth1.3021 -i eth0.3021 -C aux_scripts=q0of8 -----Original Message----- From: bro-dev-bounces at bro-ids.org [mailto:bro-dev-bounces at bro-ids.org] On Behalf Of Bro Tracker Sent: Wednesday, June 22, 2011 2:15 PM To: robin at icir.org Cc: bro-dev at bro-ids.org Subject: Re: [Bro-Dev] #12: Capture from multiple interfaces #12: Capture from multiple interfaces ------------------------------+---------------------- Reporter: robin | Owner: robin Type: Feature Request | Status: accepted Priority: Low | Milestone: Component: BroControl | Version: 1.5.2 Resolution: | Keywords: ------------------------------+---------------------- Comment (by seth): Any chance this could be a candidate for 1.6? I wouldn't place it at a very priority due to the fact that someone could just bridge the interfaces together if they need to sniff two interfaces. Honestly, that's probably what they should be doing anyway instead of doing it within Bro. -- Ticket URL: Bro Tracker Bro Issue Tracker _______________________________________________ bro-dev mailing list bro-dev at bro-ids.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From jsiwek at ncsa.illinois.edu Wed Jun 22 13:36:21 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 22 Jun 2011 15:36:21 -0500 (CDT) Subject: [Bro-Dev] serialization problems In-Reply-To: <2d656229-f67e-4264-9613-2ba4621fe18f@tangent.ncsa.illinois.edu> Message-ID: <979cc5aa-bd57-4771-ba13-228af1984495@tangent.ncsa.illinois.edu> I've been having a heck of a time porting the istate.events unit test to the policy-scripts-new branch. I understand the script changes that need to be done and everything, but the way the test compares the serialized events from each bro instance has started to fail: they are reporting differing values of arguments in some places that I'm trying to understand. I think the difference isn't a result of the new policy scripts themselves, but probably just that they're exercising some part of the serialization code that wasn't before. Here's an simpler example script I wrote that (I think) shows the same kind of thing I was running into with the new http scripts: ---------------- event_serialize_test.bro ---------------- type I: record { method: string; cnt: string &default=""; }; type S: record { pending: table[count] of I; }; type C: record { somefield: string &default="blah"; state: S; }; global E: event(c: C); event E(c: C) { print c; c$state$pending[1]$method = "after event"; c$state$pending[1]$cnt += "*"; } event bro_init() { capture_events("events.bst"); local c: C; local i: I; c$state$pending[1] = i; c$state$pending[1]$method = "by init"; event E(c); event E(c); } ---------------- event_serialize_test.bro ---------------- Here's some output that looks ok: $ ./src/bro event_serialize_test.bro [somefield=blah, state=[pending={ [1] = [method=by init, cnt=] }]] [somefield=blah, state=[pending={ [1] = [method=after event, cnt=*] }]] But here's the serialized events: $ ./src/bro -x events.bst event_serialize_test.bro Date: Wed Jun 22 14:55:52 2011 Event [1308772552.798098] E([somefield="blah", state=[pending={[1] = [method="by init", cnt=""]}]]) Event [1308772552.798098] E([somefield="blah", state=[pending={[1] = [method="by init", cnt=""]}]]) So the value of the 'pending' table doesn't seem right to me for the second serialization of event E. After enabling the serialization debug logs, I think what I'm seeing is that the first event is serialized with the full table value, but the second event is serialized with just a reference to the first's even though that value has changed. Does this seem like a problem or am I not really on the right track? - Jon From bro at tracker.bro-ids.org Wed Jun 22 14:08:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 21:08:36 -0000 Subject: [Bro-Dev] #12: Capture from multiple interfaces In-Reply-To: <047.64011cc28bcfc8034dfa19cd40719ef4@tracker.bro-ids.org> References: <047.64011cc28bcfc8034dfa19cd40719ef4@tracker.bro-ids.org> Message-ID: <062.d827f76e10e0c0c77a97ca5846cd16d4@tracker.bro-ids.org> #12: Capture from multiple interfaces ------------------------------+---------------------- Reporter: robin | Owner: robin Type: Feature Request | Status: accepted Priority: Low | Milestone: Component: BroControl | Version: 1.5.2 Resolution: | Keywords: ------------------------------+---------------------- Comment (by vern): Is it clear that OS's of interest support bridging interfaces? At least in the past, we had to custom add this to FreeBSD, for example. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 15:59:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 22 Jun 2011 22:59:48 -0000 Subject: [Bro-Dev] #12: Capture from multiple interfaces In-Reply-To: <047.64011cc28bcfc8034dfa19cd40719ef4@tracker.bro-ids.org> References: <047.64011cc28bcfc8034dfa19cd40719ef4@tracker.bro-ids.org> Message-ID: <062.fe02aa0ed225a6c3e170507ede388b0c@tracker.bro-ids.org> #12: Capture from multiple interfaces ------------------------------+---------------------- Reporter: robin | Owner: robin Type: Feature Request | Status: accepted Priority: Low | Milestone: Component: BroControl | Version: 1.5.2 Resolution: | Keywords: ------------------------------+---------------------- Comment (by robin): On Wed, Jun 22, 2011 at 19:15 -0000, you wrote: > Any chance this could be a candidate for 1.6? Yeah, shouldn't be too difficult. The important part is finding/adapting all locations that use the interface information (capstats comes to mind ...) Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 17:44:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 23 Jun 2011 00:44:45 -0000 Subject: [Bro-Dev] #12: Capture from multiple interfaces In-Reply-To: <047.64011cc28bcfc8034dfa19cd40719ef4@tracker.bro-ids.org> References: <047.64011cc28bcfc8034dfa19cd40719ef4@tracker.bro-ids.org> Message-ID: <062.4ab3733b4d57cb30c7e3dca10ed1eb26@tracker.bro-ids.org> #12: Capture from multiple interfaces ------------------------------+---------------------- Reporter: robin | Owner: robin Type: Feature Request | Status: accepted Priority: Low | Milestone: Component: BroControl | Version: 1.5.2 Resolution: | Keywords: ------------------------------+---------------------- Comment (by seth): > Is it clear that OS's of interest support bridging interfaces? At least > in the past, we had to custom add this to FreeBSD, for example. FreeBSD does support it now and it's even very easy to setup (unlike the old netgraph way of interface bonding). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 22 21:04:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 23 Jun 2011 04:04:35 -0000 Subject: [Bro-Dev] #482: Bro command line filter doesn't work anymore Message-ID: <048.2587c36cc405d5974e0bb084ff8fa80a@tracker.bro-ids.org> #482: Bro command line filter doesn't work anymore ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ {{{ #!rst Setting a command-line filter with -f no longer works. The filter passed on the command line is used by build_default_pcap_filter(). Ultimately install_default_pcap_filter() is called is which then calls the BiF install_pcap_filter(). The problem is that install_pcap_filter() check's whether a filter was passed on the command line and if so, it refuses to set the filter. Thus the command line filter never gets set. I can fix this. However, I guess the question is: what is the best way to do this? In particular, the question is what kind of filters should be settable via the various interfaces? Right now bro supports: a) Command line. Override any other filter ever set (obviously this is currently broken) b) `all_packets=T` Bro sets a default filter that captures all packets c) `all_packets=F` Bro builds its default filter using capture_filter and restrict_filter (old behavior) d) a script can call the BiF install_pcap_filter so set any custom filter unless a command line filter is specified. * (a.), (b.), and (c.) are reflected by print-filter, but (d) isn't * "d" can override the default filter but not a command line specified filter. * "d" is used by load-level. * "a" is used by peer-status.bro, remote.bro, and remote-pcap.bro. I'm not sure what those scripts do and whether they might get confused if (d) is used. * When a PktSrc is instantiated the constructor receives a filter that the PktSrc compiles (at index 0) and then installs this filter. Currently, net_init() still sets this filter "tcp or icmp or udp" * When bro_init() is executed, pcap.bro will construct the default filter (a, b, or c) and then install it. Questions: * Should there be a function that returns the currently set filter, even if (d) is used? I think that would make sense. * I find the current way of setting filters confusing. Is there maybe a better way to set/use filter? * Then there's also the secondary path with its own filters. I don't know how they work exactly though. Suggestions: 1) bro.bif stores a table matching filter index to filter strings and it remembers which filter index is currently set. Then it's easy to write a get_current_filter() BiF. 2) Use filter index 0 for case (a). If there's such a filter, don't allow any other filter to be set. Use filter index 1 for case (b) Use filter index 2 for case (c) Use other filter indexes for other cases. Using install_pcap_filter() scripts could then even dynamically change between (b) and (c) I think (1) makes sense. Not sure whether (2) does. Thoughts? Comments? cu Gregor }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Wed Jun 22 22:06:16 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 22 Jun 2011 22:06:16 -0700 Subject: [Bro-Dev] serialization problems In-Reply-To: <979cc5aa-bd57-4771-ba13-228af1984495@tangent.ncsa.illinois.edu> References: <2d656229-f67e-4264-9613-2ba4621fe18f@tangent.ncsa.illinois.edu> <979cc5aa-bd57-4771-ba13-228af1984495@tangent.ncsa.illinois.edu> Message-ID: <20110623050616.GE7686@icir.org> > After enabling the serialization debug logs, I think what I'm seeing > is that the first event is serialized with the full table value, but > the second event is serialized with just a reference to the first's > even though that value has changed. Yes, that's exactly what's happening. Generally, the serialization framework sends an object only the first time, and from then on sends just references to it (i.e., unique IDs). While for your example this is not ideal, it's a trade-off with three other objectives: (1) keeping the volume manageable (always sending everything would quickly become a lot; and it's actually not easy to find what has changed since last time); (2) maintaining correct references where they are needed (think about two record types A and B both having a subfield of record type C; now separate instances of A and B can reference the *same* instance of C, and if we send A and B to a peer, that structure needs to be recreated over there); and (3) making this all work with remote peers that come and go during run-time as they please ... Also, the use-case you describe is rare in actual Bro scripts: it involves a single record instance that's passed into two events and modified in between. (Well, I *believe* it's still rare; with scripts getting more complex these days, that may be changing ...). And note that for Bro-to-Bro communication, record modifications are normally covered by using &synchronized. For that, state operations are send across the channel and then replayed on the other side to reflect the update. This whole sending-object-IDs around is, btw, also the reason why we can't do real broadcasts ... Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Jun 23 04:37:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 23 Jun 2011 11:37:45 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: GogCannar Message-ID: <20110623113745.0AF6236A3AA@taffy.ICSI.Berkeley.EDU> New user registration for user GogCannar -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jun 23 07:06:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 23 Jun 2011 14:06:51 -0000 Subject: [Bro-Dev] #482: Bro command line filter doesn't work anymore In-Reply-To: <048.2587c36cc405d5974e0bb084ff8fa80a@tracker.bro-ids.org> References: <048.2587c36cc405d5974e0bb084ff8fa80a@tracker.bro-ids.org> Message-ID: <063.9719c53fb49316583b7d54ef26163d96@tracker.bro-ids.org> #482: Bro command line filter doesn't work anymore ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): > Setting a command-line filter with -f no longer works. I see you marked that this was on master, I just wanted to confirm that you are in fact encountering this on master. I believe I fixed it several weeks ago on the topic/policy-scripts-new branch. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jun 23 07:19:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 23 Jun 2011 14:19:37 -0000 Subject: [Bro-Dev] #482: Bro command line filter doesn't work anymore In-Reply-To: <048.2587c36cc405d5974e0bb084ff8fa80a@tracker.bro-ids.org> References: <048.2587c36cc405d5974e0bb084ff8fa80a@tracker.bro-ids.org> Message-ID: <063.3ab742909f4ba892dd01b05344fb0b78@tracker.bro-ids.org> #482: Bro command line filter doesn't work anymore ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by gregor): Replying to [comment:0 seth]: > > Setting a command-line filter with -f no longer works. > > > I see you marked that this was on master, I just wanted to confirm that you are in fact encountering this on master. I believe I fixed it several weeks ago on the topic/policy-scripts-new branch. Yes. It's indeed on master. I didn't see that you fixed it on topic /policy-scripts-new. But FYI, it appears that on topic/policy-scripts-new one can now override a command line specified filter with the install_pcap_filter() BiF. (Which is not necessarily bad, we should just be aware of it). cu gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jun 23 07:26:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 23 Jun 2011 14:26:47 -0000 Subject: [Bro-Dev] #482: Bro command line filter doesn't work anymore In-Reply-To: <048.2587c36cc405d5974e0bb084ff8fa80a@tracker.bro-ids.org> References: <048.2587c36cc405d5974e0bb084ff8fa80a@tracker.bro-ids.org> Message-ID: <063.0862c200052fe04aa779141139018d67@tracker.bro-ids.org> #482: Bro command line filter doesn't work anymore ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): > But FYI, it appears that on topic/policy-scripts-new > one can now override a command line specified filter with the > install_pcap_filter() BiF. (Which is not necessarily bad, we should just > be aware of it). Yes, that's deliberate. I figure 99% of people will just be using the shipped scripts and the shipped scripts still follow that behavior. I see no reason stop people from shooting themselves in the foot if they really want to. :) -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Thu Jun 23 08:55:21 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 23 Jun 2011 10:55:21 -0500 (CDT) Subject: [Bro-Dev] serialization problems In-Reply-To: <20110623050616.GE7686@icir.org> Message-ID: <8e852b9b-f505-4f09-8750-693b4442541b@tangent.ncsa.illinois.edu> > > After enabling the serialization debug logs, I think what I'm > > seeing is that the first event is serialized with the full table value, > > but the second event is serialized with just a reference to the first's > > even though that value has changed. > > Yes, that's exactly what's happening. Generally, the serialization > framework sends an object only the first time, and from then on sends > just references to it (i.e., unique IDs). Ok. And I think it also makes more sense to me when I think about it in the context of how the serialized events will be replayed -- against the same given input with `bro -R`, the values should actually come out to be the same. But the simple example I gave doesn't really show what's happening with the istate.events test in policy-scripts-new like I thought it did... I think what's happening there is that the sender serializes events as expected into events.bst with references as appropriate, but for a given event the receiver serializes the full value into its events.bst. So when both sides read back their version of events.bst, some values actually differ. I could be wrong, though, the serial debugging log was pretty hard to read with that complex of a situation. I'll commit my current version of the test that's failing to policy-scripts-new and maybe you can take a look, else I'll try to come up with another simple example. - Jon From bro at tracker.bro-ids.org Thu Jun 23 09:02:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 23 Jun 2011 16:02:34 -0000 Subject: [Bro-Dev] #482: Bro command line filter doesn't work anymore In-Reply-To: <048.2587c36cc405d5974e0bb084ff8fa80a@tracker.bro-ids.org> References: <048.2587c36cc405d5974e0bb084ff8fa80a@tracker.bro-ids.org> Message-ID: <063.c6d8b4c728e44cd08543ce1e31fd0b38@tracker.bro-ids.org> #482: Bro command line filter doesn't work anymore -----------------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by gregor): * status: new => closed * resolution: => Solved/Applied Comment: This is fixed in topic/policy-scripts-new and will thus be included in master once this branch is merged. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jun 23 16:50:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 23 Jun 2011 23:50:28 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: geonerstiem Message-ID: <20110623235026.BCF5836A017@taffy.ICSI.Berkeley.EDU> New user registration for user geonerstiem -- Bro Tracker Bro Issue Tracker From robin at icir.org Thu Jun 23 20:58:20 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 23 Jun 2011 20:58:20 -0700 Subject: [Bro-Dev] serialization problems In-Reply-To: <8e852b9b-f505-4f09-8750-693b4442541b@tangent.ncsa.illinois.edu> References: <20110623050616.GE7686@icir.org> <8e852b9b-f505-4f09-8750-693b4442541b@tangent.ncsa.illinois.edu> Message-ID: <20110624035820.GC46096@icir.org> On Thu, Jun 23, 2011 at 10:55 -0500, you wrote: > I'll commit my current version of the test that's failing to > policy-scripts-new and maybe you can take a look, else I'll try to > come up with another simple example. Ok, I'll take a look (potentially not before we merge the branch but I hope that will be soon :) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Fri Jun 24 06:10:51 2011 From: seth at icir.org (Seth Hall) Date: Fri, 24 Jun 2011 09:10:51 -0400 Subject: [Bro-Dev] serialization problems In-Reply-To: <20110624035820.GC46096@icir.org> References: <20110623050616.GE7686@icir.org> <8e852b9b-f505-4f09-8750-693b4442541b@tangent.ncsa.illinois.edu> <20110624035820.GC46096@icir.org> Message-ID: <3F8BBE8E-83BB-44BF-B85D-142E59E5140E@icir.org> On Jun 23, 2011, at 11:58 PM, Robin Sommer wrote: > Ok, I'll take a look (potentially not before we merge the branch but I > hope that will be soon :) Give me a couple of days to do a bit more cleanup on stuff, SSL isn't done and I want to cleanup the Intelligence framework. I definitely agree we're very close to being able to merge it. The cluster framework can come after the merge, but I don't think I have too much longer on that either. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Fri Jun 24 10:09:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 24 Jun 2011 17:09:06 -0000 Subject: [Bro-Dev] #483: Logging framework using wrong name. Message-ID: <046.c5325e1914394fb73ca620cc764dd452@tracker.bro-ids.org> #483: Logging framework using wrong name. ---------------------+-------------------- Reporter: seth | Owner: robin Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- When the logging framework creates filters, it has a problem with enums in modules. If you do this: {{{ module Notice; export { redef enum Log::ID += { FOOBAR }; } event bro_init() { Log::create_stream(FOOBAR, [$columns=XYZ]); } }}} If you leave it alone and don't create an output stream, it should be pulling the default path (file) name from the stream ID but in the case above it would end up using the path name "Notice" because it's not dealing with modules correctly. The full value for the enum is Notice::FOOBAR. I think it should just take the actual enum name without the module because in most cases it would result in redundantly named log files. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Jun 25 16:50:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 25 Jun 2011 23:50:42 -0000 Subject: [Bro-Dev] #484: Capitalization standardization Message-ID: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> #484: Capitalization standardization ---------------------+------------------- Reporter: robin | Type: Task Status: new | Priority: Normal Component: Website | ---------------------+------------------- We need to document these on the development pages. ----- Forwarded message from Seth Hall ----- Date: Mon, 7 Mar 2011 16:02:32 -0500 From: Seth Hall Subject: [Bro-Dev] capitalization standardization Message-Id: <61A16165-5B99-4348-9604-ED432251E988 at icir.org> Now that I'm *finally* digging into these scripts, I've noticed that there isn't a lot of standardization across scripts with regard to capitalization. I want to propose the following style for scripts (keep in mind this won't be enforced by the language, just a common practice). Module names: Camel cased, no underscores (all uppercase for abbreviations) Examples: FTP, SSH, Notice, Remote, Signatures Types: Camel cased, no underscores Examples: Log, Connection, EntropyTestResult, GeoLocation, Packet Enum values: Upper-case with underscores Examples: KNOWN_HOSTS, SSH, NFS3_REG Variables: Lower-case with underscores Examples: example_variable, ftp_sessions This should follow the established conventions of most scripts (unfortunately not all). Did I leave anything out? The enum values doesn't quite work because Notice enum values have been defined as Camel-cased with underscores mostly. I'm sort of inclined to leave this as it is because it pinpoints when a notice type is being defined. I suppose that would be a subcategory of enum values that is just done specially. ----- End forwarded message ----- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Jun 25 16:51:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 25 Jun 2011 23:51:23 -0000 Subject: [Bro-Dev] #484: Capitalization standardization In-Reply-To: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> References: <047.b75bf15c0baa269c752add9d953ccd04@tracker.bro-ids.org> Message-ID: <062.92ebfd31a0aedfb1f02e7f804e359824@tracker.bro-ids.org> #484: Capitalization standardization ----------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Website | Version: Resolution: | Keywords: ----------------------+-------------------- Changes (by robin): * milestone: => Bro1.6 -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Jun 25 17:12:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 26 Jun 2011 00:12:16 -0000 Subject: [Bro-Dev] #480: topic/jsiwek/script-load-cleanup In-Reply-To: <048.71712540bb3ceafffd47b237c36c1930@tracker.bro-ids.org> References: <048.71712540bb3ceafffd47b237c36c1930@tracker.bro-ids.org> Message-ID: <063.68699280349e8edc50374289920fd99a@tracker.bro-ids.org> #480: topic/jsiwek/script-load-cleanup -----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Jun 25 17:12:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 26 Jun 2011 00:12:36 -0000 Subject: [Bro-Dev] #481: topic/seth/net-stats-bif - New net_stats BiF In-Reply-To: <046.53c17b9462e3c4e7868a29afdc56ebec@tracker.bro-ids.org> References: <046.53c17b9462e3c4e7868a29afdc56ebec@tracker.bro-ids.org> Message-ID: <061.b357dbb2d435a86c38da7c129da98162@tracker.bro-ids.org> #481: topic/seth/net-stats-bif - New net_stats BiF -----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Jun 25 17:46:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 26 Jun 2011 00:46:22 -0000 Subject: [Bro-Dev] #368: topic/dnsttl - Patch for Reverse DNS Lookups and DNS TTL support In-Reply-To: <054.ec7b9fe4a96a58e9acb9d7b759a8aea2@tracker.bro-ids.org> References: <054.ec7b9fe4a96a58e9acb9d7b759a8aea2@tracker.bro-ids.org> Message-ID: <069.b9d59034bdcc5bbb977c558d9081b653@tracker.bro-ids.org> #368: topic/dnsttl - Patch for Reverse DNS Lookups and DNS TTL support -----------------------------+----------------------------- Reporter: thomas.other | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: Solved/Applied | Keywords: DNS TTL Resolve -----------------------------+----------------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Jun 25 17:55:41 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 26 Jun 2011 00:55:41 -0000 Subject: [Bro-Dev] #479: topic/jsiwek/pybroccoli-fixes In-Reply-To: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> References: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> Message-ID: <063.9cbd79ec9b1721857a6ea9a9fa7322bf@tracker.bro-ids.org> #479: topic/jsiwek/pybroccoli-fixes ----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): I've applied the broccoli-python patch. But question regarding the Bro patch: do we really need to do the record coercion check for every event raised locally? Most of them will be generated locally, not received from remote. Can we move that to where we receive events in RemoteSerializer.cc? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Jun 25 18:00:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 26 Jun 2011 01:00:24 -0000 Subject: [Bro-Dev] #479: topic/jsiwek/pybroccoli-fixes In-Reply-To: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> References: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> Message-ID: <063.65b3b07af7125152742d971f30304959@tracker.bro-ids.org> #479: topic/jsiwek/pybroccoli-fixes ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => jsiwek * status: new => assigned -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Jun 26 16:49:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 26 Jun 2011 23:49:15 -0000 Subject: [Bro-Dev] #483: Logging framework using wrong name. In-Reply-To: <046.c5325e1914394fb73ca620cc764dd452@tracker.bro-ids.org> References: <046.c5325e1914394fb73ca620cc764dd452@tracker.bro-ids.org> Message-ID: <061.677f6106dc2ec98f5db902e52f25393e@tracker.bro-ids.org> #483: Logging framework using wrong name. -----------------------------+-------------------- Reporter: seth | Owner: robin Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Sun Jun 26 20:48:33 2011 From: gregor at icir.org (Gregor Maier) Date: Sun, 26 Jun 2011 20:48:33 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/reporting: Overhauling the internal reporting of messages to the user. (93894ee) In-Reply-To: <201106260123.p5Q1NIfc015267@bro-ids.icir.org> References: <201106260123.p5Q1NIfc015267@bro-ids.icir.org> Message-ID: <4E07FD91.2060009@icir.org> On 6/25/11 18:23 , Robin Sommer wrote: > - util.h: all the error()/warn()/message()/run_time()/pinpoint() > functions are gone. Use the bro_logger instead now. > > - Script errors are formatted a bit differently due to the > changes. What I've seen so far looks ok to me, but let me know > if there's something odd. How are errors in the script layer handled? If there's an error in a script you can't really call an event handler for to handle that error, since it might well be that the error is either in the event handler or that the error will prevent the event handler from working properly. cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From robin at icir.org Sun Jun 26 21:00:29 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 26 Jun 2011 21:00:29 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/reporting: Overhauling the internal reporting of messages to the user. (93894ee) In-Reply-To: <4E07FD91.2060009@icir.org> References: <201106260123.p5Q1NIfc015267@bro-ids.icir.org> <4E07FD91.2060009@icir.org> Message-ID: <20110627040029.GB19142@icir.org> On Sun, Jun 26, 2011 at 20:48 -0700, you wrote: > How are errors in the script layer handled? If there's an error in a > script you can't really call an event handler for to handle that > error, It's not really *handled*, it just raises a new log_error event that reports the error with a message. That event is then just queued normally, which shouldn't cause any further trouble. (Well, I guess the handler for log_error better doesn't cause any new errors; but even if, it will just flood the log file, Bro will keep processing.) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Mon Jun 27 04:10:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 27 Jun 2011 11:10:28 -0000 Subject: [Bro-Dev] #485: Error when defining vectors of records with optional fields Message-ID: <050.7eda304106534922559078736965b156@tracker.bro-ids.org> #485: Error when defining vectors of records with optional fields ----------------------+------------------------ Reporter: vallenti | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ----------------------+------------------------ When defining vectors of records that contain optional fields, Bro erroneously reports a type clash. Here is a minimal example that triggers the bug: {{{ type Foo: record { s: string; o: string &optional; }; const v: vector of Foo = { [$s="bar", $o="check"], [$s="baz"] } }}} The error is: {{{ ./vector.bro, line 8 ([$s=bar]): error, type mismatch at index 1 }}} That is, constructing the first records succeeds, but the second falsely fails. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Mon Jun 27 07:26:01 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 27 Jun 2011 07:26:01 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/reporting: Overhauling the internal reporting of messages to the user. (93894ee) In-Reply-To: <20110627040029.GB19142@icir.org> References: <201106260123.p5Q1NIfc015267@bro-ids.icir.org> <4E07FD91.2060009@icir.org> <20110627040029.GB19142@icir.org> Message-ID: <4E0892F9.2050800@icir.org> I really like the idea of having event handlers to handle error messages! A couple more comments: Can the log_* handlers also write to stderr instead of stdout? I think that would be better. It appears that there are several places, esp. in Expr.cc and Stmt.cc that still use the old Error(), Internal(), etc. functions (including the SetError). Is this on purpose? On 6/26/11 21:00 , Robin Sommer wrote: > > On Sun, Jun 26, 2011 at 20:48 -0700, you wrote: > >> How are errors in the script layer handled? If there's an error in a >> script you can't really call an event handler for to handle that >> error, > > It's not really *handled*, it just raises a new log_error event that > reports the error with a message. That event is then just queued > normally, which shouldn't cause any further trouble. I was more wondering what would happen if due to an error the handler for log_error cannot run properly. I.e., whether there can be a case were we don't see any error messages. > (Well, I guess the handler for log_error better doesn't cause any new > errors; but even if, it will just flood the log file, Bro will keep > processing.) Well, log_error will go in an endless loop and continue to queue events. So bro won't process any more packets because the event queue never gets empty, right? cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Mon Jun 27 07:53:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 27 Jun 2011 14:53:21 -0000 Subject: [Bro-Dev] #479: topic/jsiwek/pybroccoli-fixes In-Reply-To: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> References: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> Message-ID: <063.44394aabd7f519f15804690fa84bcc83@tracker.bro-ids.org> #479: topic/jsiwek/pybroccoli-fixes ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): > But question regarding the Bro patch: do we really need to do the record coercion check for every event raised locally? Most of them will be generated locally, not received from remote. Can we move that to where we receive events in RemoteSerializer.cc? Yeah, I think to fix the original problem Seth had, it suffices to just do it for received events. Though for consistency, does it make sense for someone to expect that locally raised events will coerce record types, too? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jun 27 08:21:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 27 Jun 2011 15:21:35 -0000 Subject: [Bro-Dev] #479: topic/jsiwek/pybroccoli-fixes In-Reply-To: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> References: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> Message-ID: <063.f2fc5c760bbffb61795de25496ae95f5@tracker.bro-ids.org> #479: topic/jsiwek/pybroccoli-fixes ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): On Mon, Jun 27, 2011 at 14:53 -0000, you wrote: > Yeah, I think to fix the original problem Seth had, it suffices to just do > it for received events. Though for consistency, does it make sense for > someone to expect that locally raised events will coerce record types, > too? For local events, Bro should already be coercing the records, otherwise there's a bug somewhere because the types don't match. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Jun 27 08:28:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 27 Jun 2011 15:28:40 -0000 Subject: [Bro-Dev] #479: topic/jsiwek/pybroccoli-fixes In-Reply-To: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> References: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> Message-ID: <063.2783eecd3fdd522b65944a0495785b39@tracker.bro-ids.org> #479: topic/jsiwek/pybroccoli-fixes ---------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Changes (by jsiwek): * type: Merge Request => Task Comment: > For local events, Bro should already be coercing the records, > otherwise there's a bug somewhere because the types don't match. Ah ok, let me see about moving that check to only happen for remote events then. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon Jun 27 08:32:40 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 27 Jun 2011 08:32:40 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/reporting: Overhauling the internal reporting of messages to the user. (93894ee) In-Reply-To: <4E0892F9.2050800@icir.org> References: <201106260123.p5Q1NIfc015267@bro-ids.icir.org> <4E07FD91.2060009@icir.org> <20110627040029.GB19142@icir.org> <4E0892F9.2050800@icir.org> Message-ID: <20110627153239.GB97459@icir.org> On Mon, Jun 27, 2011 at 07:26 -0700, you wrote: > Can the log_* handlers also write to stderr instead of stdout? The current implementation is just a placeholder. The output should probably go into its own log file but I let Seth figure that out. :-) > It appears that there are several places, esp. in Expr.cc and Stmt.cc > that still use the old Error(), Internal(), etc. functions (including > the SetError). Is this on purpose? Yes, these are BroObj's Error() etc. methods; I just removed the methods in the util.h. Generally, I left object-specific error methods in because (1) they sometimes do additional things, and (2) it would have required touching lots of places without making things much better. But: these methods used to call error() for doing their output and *that* has been replaced with Logger::Error(). Let me know if you still find a method logging errors directly to stderr/out (except for debugging stuff, I didn't touch that, like in Rule*.cc). > I was more wondering what would happen if due to an error the handler > for log_error cannot run properly. I.e., whether there can be a case > were we don't see any error messages. Can't think of anything. > Well, log_error will go in an endless loop and continue to queue events. Hmm, that's right. I forgot that we aren't limiting how many events we process each time we drain the manager (we do limit timers per packet). Seems that should be caught and errors reported in a log_error hanlder be written out to stderr. Another question: what do you guys think about merging Logger and LogMgr now? I.e., let LogMgr do both error reporting and normal log output. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at ncsa.illinois.edu Mon Jun 27 09:12:59 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 27 Jun 2011 11:12:59 -0500 (CDT) Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/reporting: Overhauling the internal reporting of messages to the user. (93894ee) In-Reply-To: <20110627153239.GB97459@icir.org> Message-ID: <8f157d11-ac90-4137-b52e-9492e0075cae@tangent.ncsa.illinois.edu> > Another question: what do you guys think about merging Logger and > LogMgr now? I.e., let LogMgr do both error reporting and normal log > output. Are you talking about just merging the Logger's currently implemented interface as-is under the LogMgr class, or was there some additional integration of the two you were thinking about? If the former, it seems like maybe just finding a better name for the Logger class would do (MsgReporter, maybe?). - Jon From robin at icir.org Mon Jun 27 09:15:06 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 27 Jun 2011 09:15:06 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/reporting: Overhauling the internal reporting of messages to the user. (93894ee) In-Reply-To: <8f157d11-ac90-4137-b52e-9492e0075cae@tangent.ncsa.illinois.edu> References: <20110627153239.GB97459@icir.org> <8f157d11-ac90-4137-b52e-9492e0075cae@tangent.ncsa.illinois.edu> Message-ID: <20110627161506.GA1115@icir.org> On Mon, Jun 27, 2011 at 11:12 -0500, you wrote: > Are you talking about just merging the Logger's currently implemented > interface as-is under the LogMgr class, Just as-is. I'm kind of torn whether the two belong together or not. > (MsgReporter, maybe?). Yeah, I can see that. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Mon Jun 27 15:18:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 27 Jun 2011 22:18:38 -0000 Subject: [Bro-Dev] #479: topic/jsiwek/pybroccoli-fixes In-Reply-To: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> References: <048.6459c0d511666c60843800ea9fb21da5@tracker.bro-ids.org> Message-ID: <063.5c9ae2e6988b0158e6ac79cd35dc09d7@tracker.bro-ids.org> #479: topic/jsiwek/pybroccoli-fixes ----------------------------+------------------------ Reporter: jsiwek | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Task => Merge Request Comment: I moved it, you can look it over again and merge if it seems ok. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Jun 28 09:45:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 28 Jun 2011 16:45:01 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: emommicky Message-ID: <20110628164500.AD6A436A3A6@taffy.ICSI.Berkeley.EDU> New user registration for user emommicky -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 29 18:40:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 30 Jun 2011 01:40:44 -0000 Subject: [Bro-Dev] [Bro Tracker] New user registration: MooggoveRat Message-ID: <20110630014042.EA17936A038@taffy.ICSI.Berkeley.EDU> New user registration for user MooggoveRat -- Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Jun 29 22:00:59 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 30 Jun 2011 05:00:59 -0000 Subject: [Bro-Dev] #486: Prefixes don't support directories Message-ID: <047.e78f97399b2dd07d57a5512021972454@tracker.bro-ids.org> #486: Prefixes don't support directories ------------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: Bro1.6 | Component: Bro Version: git/master | ------------------------+--------------------- The prefix loading mechanism can't deal with scripts in directories currently. Need to decide what to do. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jun 30 13:36:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 30 Jun 2011 20:36:56 -0000 Subject: [Bro-Dev] #487: Bug and missing feature in syslog analyzer Message-ID: <046.39317bf8c8ad376fb1f4afd399768ef7@tracker.bro-ids.org> #487: Bug and missing feature in syslog analyzer ---------------------+------------------------ Reporter: seth | Owner: seth Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ Some syslog packets are not analyzed correctly and the data include two extra bytes at the end of the message (\x0a\x00). The syslog analyzer also doesn't deal with TCP syslog and needs to. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Jun 30 22:42:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 01 Jul 2011 05:42:14 -0000 Subject: [Bro-Dev] #488: Bug in HTTP parser Message-ID: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> #488: Bug in HTTP parser ------------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: Bro1.6 | Component: Bro Version: git/master | ------------------------+--------------------- I'm going to commit a test protocols/http/http-header-crlf.bro that triggers a problem in the HTTP parser where for a multipart/byteranges message it gets confused whether its in a header or not. To work around the crash, I'm also going to commit a fix that turns the ASSERT noticing the confusion into a weird. That's actually something we should generally do: it's not a good idea to bail out with an assertion if a parser gets out of sync. So the other todo for this ticket: replace all the HTTP ASSERTs with weirds. -- Ticket URL: Bro Tracker Bro Issue Tracker