From juhoon at net.t-labs.tu-berlin.de Wed Dec 1 02:37:58 2010 From: juhoon at net.t-labs.tu-berlin.de (Juhoon Kim) Date: Wed, 01 Dec 2010 11:37:58 +0100 Subject: [Bro-Dev] TCP RTT estimation In-Reply-To: <20101130160140.33B8F36A4ED@taffy.ICSI.Berkeley.EDU> References: <20101130160140.33B8F36A4ED@taffy.ICSI.Berkeley.EDU> Message-ID: <1291199878.2705.52.camel@jkim> > > The idea what I have about RTT is measuring a gap between the time a > > segment is seen and the time a segment directly after the corresponding > > ack is seen in the monitoring point. > > Can you sketch in more detail (with an example) what you have in mind? > (I've been down this path in the past, and it gets pretty complex!) Sorry, my description was not clear enough. Following is an RTT what we want to measure. [Sender] [Monitor]-------------(a)------>[Receiver] [Sender]<-------[Monitor]<--------ack-of-(a)----[Receiver] [Sender]------->[Monitor] While Katrina's RTT estimation seems like following: [Sender] [Monitor]-------------(a)------>[Receiver] [Sender] [Monitor]<--------ack-of-(a)----[Receiver] > Also, what's your ultimate goal: path/performance characterization? > Katrina's is more focused on trouble-shooting, so finding exceptional > values rather than (for example) accurately tracking the RTO computation. Yes, our goal is to measure the performance of TCP flows. That's why we want to measure RTT as accurate as possible. Juhoon From bro at tracker.icir.org Wed Dec 1 08:13:20 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 01 Dec 2010 16:13:20 -0000 Subject: [Bro-Dev] #328: Programmatic pattern construction at init time. Message-ID: <043.f9c865007fdbbe40274ceab20806264f@tracker.icir.org> #328: Programmatic pattern construction at init time. -----------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | -----------------------------+-------------------- bro.bif contains functions for dynamically creating patterns but they are disabled if reading live traffic because the DFA and NFA data structures can't be fully cleaned up. It would be useful to be able to call these functions in bro_init handlers regardless of if live traffic is being analyzed which should be acceptable for memory concerns but allows patterns to be built from configuration variables. Here are the functions: {{{ function merge_pattern%(p1: pattern, p2: pattern%): pattern function string_to_pattern%(s: string, convert: bool%): pattern }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From vern at icir.org Wed Dec 1 08:21:30 2010 From: vern at icir.org (Vern Paxson) Date: Wed, 01 Dec 2010 08:21:30 -0800 Subject: [Bro-Dev] TCP RTT estimation In-Reply-To: <1291199878.2705.52.camel@jkim> (Wed, 01 Dec 2010 11:37:58 +0100). Message-ID: <20101201162130.813CF36A4F3@taffy.ICSI.Berkeley.EDU> > Sorry, my description was not clear enough. Following is an RTT what we > want to measure. > > [Sender] [Monitor]-------------(a)------>[Receiver] > [Sender]<-------[Monitor]<--------ack-of-(a)----[Receiver] > [Sender]------->[Monitor] How will you identify the correct segment for this last component? That's where it gets quite hard. You need to understand the precise congestion control algorithm that the Sender implements, along with dealing with headaches like the second component (ack-of-(a)) being lost between the Monitor and the Sender, or the Sender imposing a sender-side buffering limitation or inheriting an initial ssthresh. (A bunch of this is discussed in my tcpanaly paper from an eon ago.) It really is quite hard, though it's easier if you can confine yourself to simpler instances such as only measuring RTT prior to a loss occurring. Vern From juhoon at net.t-labs.tu-berlin.de Wed Dec 1 09:29:26 2010 From: juhoon at net.t-labs.tu-berlin.de (Juhoon Kim) Date: Wed, 01 Dec 2010 18:29:26 +0100 Subject: [Bro-Dev] TCP RTT estimation In-Reply-To: <20101201162130.813CF36A4F3@taffy.ICSI.Berkeley.EDU> References: <20101201162130.813CF36A4F3@taffy.ICSI.Berkeley.EDU> Message-ID: <1291224566.2705.131.camel@jkim> [Sender] [Monitor]-------------(a)------>[Receiver] [Sender]<-------[Monitor]<--------ack-of-(a)----[Receiver] [Sender]--(b)-->[Monitor] I thought that the acknowledgement number of the second component (ack-of-a) is always the same with the sequence number of the next segment (b). That is how I wanted to identify the segment (b). > You need to understand the precise congestion control algorithm that the Sender implements, However, it seems like it is not always the case. I will study more about this. > (A bunch of this is discussed > in my tcpanaly paper from an eon ago.) It really is quite hard, Maybe I underestimate this problem. I will restart from the paper. Thanks. From vern at icir.org Wed Dec 1 10:31:08 2010 From: vern at icir.org (Vern Paxson) Date: Wed, 01 Dec 2010 10:31:08 -0800 Subject: [Bro-Dev] TCP RTT estimation In-Reply-To: <1291224566.2705.131.camel@jkim> (Wed, 01 Dec 2010 18:29:26 +0100). Message-ID: <20101201183108.89E5636A031@taffy.ICSI.Berkeley.EDU> > [Sender] [Monitor]-------------(a)------>[Receiver] > [Sender]<-------[Monitor]<--------ack-of-(a)----[Receiver] > [Sender]--(b)-->[Monitor] > > I thought that the acknowledgement number of the second component > (ack-of-a) is always the same with the sequence number of the next > segment (b). That is how I wanted to identify the segment (b). No, instead seq-of-b will be seq-of-a's-ack *plus* the congestion window. This is where it gets hard. Vern From katrina at csail.mit.edu Wed Dec 1 12:15:14 2010 From: katrina at csail.mit.edu (Katrina LaCurts) Date: Wed, 1 Dec 2010 15:15:14 -0500 Subject: [Bro-Dev] TCP RTT estimation In-Reply-To: <20101201183108.89E5636A031@taffy.ICSI.Berkeley.EDU> References: <20101201183108.89E5636A031@taffy.ICSI.Berkeley.EDU> Message-ID: <352B1D6D-5152-46E1-842B-59E2544C2156@csail.mit.edu> Juhoon, this is one of the reasons why I recommended the handshake_sum_estimate event in my branch. It essentially does what you want to do, but just on TCP handshakes. That's because with the handshake we can easily figure out what segment (b) is (it's the ACK to the SYN-ACK). As Vern pointed out, determining (b) is tough in general. Katrina On Dec 1, 2010, at 1:31 PM, Vern Paxson wrote: >> [Sender] [Monitor]-------------(a)------>[Receiver] >> [Sender]<-------[Monitor]<--------ack-of-(a)----[Receiver] >> [Sender]--(b)-->[Monitor] >> >> I thought that the acknowledgement number of the second component >> (ack-of-a) is always the same with the sequence number of the next >> segment (b). That is how I wanted to identify the segment (b). > > No, instead seq-of-b will be seq-of-a's-ack *plus* the congestion window. > This is where it gets hard. > > Vern > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From juhoon at net.t-labs.tu-berlin.de Wed Dec 1 13:07:15 2010 From: juhoon at net.t-labs.tu-berlin.de (Juhoon Kim) Date: Wed, 1 Dec 2010 22:07:15 +0100 Subject: [Bro-Dev] TCP RTT estimation In-Reply-To: <352B1D6D-5152-46E1-842B-59E2544C2156@csail.mit.edu> References: <20101201183108.89E5636A031@taffy.ICSI.Berkeley.EDU> <352B1D6D-5152-46E1-842B-59E2544C2156@csail.mit.edu> Message-ID: <000001cb919b$bbbf5cb0$333e1610$@t-labs.tu-berlin.de> >> Juhoon, this is one of the reasons why I recommended the handshake_sum_estimate event in my branch. >> It essentially does what you want to do, but just on TCP handshakes. >> That's because with the handshake we can easily figure out what segment (b) is (it's the ACK to the SYN-ACK). Thanks Katrina, but I don't think that an RTT of a TCP handshake represents the average RTT of a whole flow. One of things we want to know is how it differs on flow sizes. >> As Vern pointed out, determining (b) is tough in general. Yes, I now realize that it is not that easy as I thought. I think I will come back to this discussion with a better idea. From robin at icir.org Wed Dec 1 18:57:01 2010 From: robin at icir.org (Robin Sommer) Date: Wed, 1 Dec 2010 18:57:01 -0800 Subject: [Bro-Dev] NFS and portmapper In-Reply-To: <4CF57BCA.6060608@icir.org> References: <4CF57BCA.6060608@icir.org> Message-ID: <20101202025701.GA88367@icir.org> On Tue, Nov 30, 2010 at 14:33 -0800, you wrote: > I was wondering whether the Portmapper code can/will activate the NFS > analyzer for dynamically negotiated NFS ports. Looking at portmap.bro > and nfs.bro I don't see anything along those lines, but I would have > expected it. Am I missing something or is this something that bro can't do? I don't think it does, but it should be possible to add that in a similar way as FTP data connections are handed on to the file analyzer. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From vern at icir.org Wed Dec 1 19:23:25 2010 From: vern at icir.org (Vern Paxson) Date: Wed, 01 Dec 2010 19:23:25 -0800 Subject: [Bro-Dev] Weird behavior in Katrina's code. In-Reply-To: <4CF5C40D.9020108@hiit.fi> (Tue, 30 Nov 2010 19:42:05 PST). Message-ID: <20101202032325.DCCE036A4EC@taffy.ICSI.Berkeley.EDU> > 1266506673.653157 ip1 port1 ip2 port2 60 0 888448966 0 S > 1266506673.653530 ip2 port2 ip1 port1 40 0 1921250427 888448967 RA > 1266506676.651348 ip1 port1 ip2 port2 60 0 888448966 0 S > 1266506676.651708 ip2 port2 ip1 port1 40 0 570721244 888448967 RA > 1266506682.651195 ip1 port1 ip2 port2 60 0 888448966 0 S > 1266506682.651622 ip2 port2 ip1 port1 40 0 1779909088 888448967 RA > 1266506694.651297 ip1 port1 ip2 port2 60 0 888448966 0 S > 1266506694.651669 ip2 port2 ip1 port1 40 0 2051408459 888448967 RA > 1266506718.651252 ip1 port1 ip2 port2 60 0 888448966 0 S > 1266506718.651676 ip2 port2 ip1 port1 60 0 3793171500 888448967 SA This is a pattern that Bro will interpret differently depending on the setting of various timeouts defined in bro.init. I'm not sure exactly which ones would apply here, but it could for example be # Upon seeing a RST, flush state after this much time. const tcp_reset_delay = 5 secs &redef; or # Upon seeing a normal connection close, flush state after this much time. const tcp_close_delay = 5 secs &redef; If those were different for the runs, that would explain why you're getting different results from the two Bro runs. Could that be what's going on? Vern From robin at icir.org Wed Dec 1 19:36:07 2010 From: robin at icir.org (Robin Sommer) Date: Wed, 1 Dec 2010 19:36:07 -0800 Subject: [Bro-Dev] Generated files and git Message-ID: <20101202033607.GK88367@icir.org> For the record, we discussed today (again :-) whether generated files should be in git and/or the distributed source tar balls. The consensus was that (at least for now) we won't be including them in either, as that doesn't fit well with CMake's separation of source and build directories. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From vern at icir.org Wed Dec 1 20:00:56 2010 From: vern at icir.org (Vern Paxson) Date: Wed, 01 Dec 2010 20:00:56 -0800 Subject: [Bro-Dev] NFS and portmapper In-Reply-To: <20101202025701.GA88367@icir.org> (Wed, 01 Dec 2010 18:57:01 PST). Message-ID: <20101202040056.8308236A4EC@taffy.ICSI.Berkeley.EDU> > I don't think it does, but it should be possible to add that in a > similar way as FTP data connections are handed on to the file > analyzer. I believe that's right (both points). Vern From bro at tracker.icir.org Thu Dec 2 06:20:08 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 02 Dec 2010 14:20:08 -0000 Subject: [Bro-Dev] #329: Remove detect-protocols-http.bro from Broctl's default local.bro Message-ID: <043.4171f047fca9bbca4e3e8c6e560433d7@tracker.icir.org> #329: Remove detect-protocols-http.bro from Broctl's default local.bro ------------------------+-------------------- Reporter: seth | Owner: robin Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Keywords: | ------------------------+-------------------- This script does a for loop over a 7 element table for every http_header and http_request event. In my opinion, I'd say that the benefit does not outweigh the cost and it should be removed from the default local.bro scripts. -- Ticket URL: Bro Tracker Bro Issue Tracker From boris.nechaev at hiit.fi Thu Dec 2 13:19:49 2010 From: boris.nechaev at hiit.fi (Boris Nechaev) Date: Thu, 02 Dec 2010 13:19:49 -0800 Subject: [Bro-Dev] Weird behavior in Katrina's code. In-Reply-To: <20101202032325.DCCE036A4EC@taffy.ICSI.Berkeley.EDU> References: <20101202032325.DCCE036A4EC@taffy.ICSI.Berkeley.EDU> Message-ID: <4CF80D75.1090007@hiit.fi> On 12/01/2010 07:23 PM, Vern Paxson wrote: >> 1266506673.653157 ip1 port1 ip2 port2 60 0 888448966 0 S >> 1266506673.653530 ip2 port2 ip1 port1 40 0 1921250427 888448967 RA >> 1266506676.651348 ip1 port1 ip2 port2 60 0 888448966 0 S >> 1266506676.651708 ip2 port2 ip1 port1 40 0 570721244 888448967 RA >> 1266506682.651195 ip1 port1 ip2 port2 60 0 888448966 0 S >> 1266506682.651622 ip2 port2 ip1 port1 40 0 1779909088 888448967 RA >> 1266506694.651297 ip1 port1 ip2 port2 60 0 888448966 0 S >> 1266506694.651669 ip2 port2 ip1 port1 40 0 2051408459 888448967 RA >> 1266506718.651252 ip1 port1 ip2 port2 60 0 888448966 0 S >> 1266506718.651676 ip2 port2 ip1 port1 60 0 3793171500 888448967 SA >> > This is a pattern that Bro will interpret differently depending on the > setting of various timeouts defined in bro.init. > ... > Could that be what's going on? > I've checked this, all the timeouts in bro.init are exactly the same, so this not what is going on. -- Best regards, Boris Nechaev. From gregor at icir.org Thu Dec 2 16:43:30 2010 From: gregor at icir.org (Gregor Maier) Date: Thu, 02 Dec 2010 16:43:30 -0800 Subject: [Bro-Dev] Tickets scheduled for inclusion in trunk Message-ID: <4CF83D32.2040605@icir.org> Hi, quick question: How do I mark a ticket for inclusion in trunk/master (for the 1.5.x release). IIRC, I assign it to Vern, set type to patch, and set version to something. Is it 1.5.2?? Thanks, Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From seth at icir.org Thu Dec 2 21:01:36 2010 From: seth at icir.org (Seth Hall) Date: Fri, 3 Dec 2010 00:01:36 -0500 Subject: [Bro-Dev] Tickets scheduled for inclusion in trunk In-Reply-To: <4CF83D32.2040605@icir.org> References: <4CF83D32.2040605@icir.org> Message-ID: <243279B2-2A39-4796-B0D8-E7AA4DACCB3F@icir.org> On Dec 2, 2010, at 7:43 PM, Gregor Maier wrote: > How do I mark a ticket for inclusion in trunk/master (for the 1.5.x > release). Oh, I guess there isn't really a way to do that at the moment and it seemed that everyone was ok with very few changes going into a final 1.5 release so it's probably not worthwhile creating a milestone for it. We'd probably be fine just doing it in an ad-hoc manner since I think that most of those tickets we were going to integrate have already been integrated. Which ticket is it? .Seth From bro at tracker.icir.org Thu Dec 2 22:24:22 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 03 Dec 2010 06:24:22 -0000 Subject: [Bro-Dev] #330: Nit: portmapper.bro needs to load scan.bro Message-ID: <045.6906275de61b10a445ed6ef3638fd852@tracker.icir.org> #330: Nit: portmapper.bro needs to load scan.bro ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ portmapper.bro requires scan.bro, so add a @load scan to portmapper.bro -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri Dec 3 06:38:53 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 03 Dec 2010 14:38:53 -0000 Subject: [Bro-Dev] #330: Nit: portmapper.bro needs to load scan.bro In-Reply-To: <045.6906275de61b10a445ed6ef3638fd852@tracker.icir.org> References: <045.6906275de61b10a445ed6ef3638fd852@tracker.icir.org> Message-ID: <060.2836b84764103f95c70bb44641135967@tracker.icir.org> #330: Nit: portmapper.bro needs to load scan.bro ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): I want to work towards scan.bro not being a dependency for anything for the 1.6 release. This can't currently be done, but I'd like to figure out a way to make it possible in the near future. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri Dec 3 07:17:30 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 03 Dec 2010 15:17:30 -0000 Subject: [Bro-Dev] #331: Remove snort2bro Message-ID: <043.60d18b4614b2db9168bd2be18da72288@tracker.icir.org> #331: Remove snort2bro ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- Parts needing removed.. * .sig files * conversion scripts (maybe already gone due to project split up?) * is there any core support for the snort rules? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri Dec 3 07:33:29 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 03 Dec 2010 15:33:29 -0000 Subject: [Bro-Dev] #332: Portmap analyzer segfaults when parsing portmap dump replies Message-ID: <045.fc87d35e27cf53e26f8771e6f7836df6@tracker.icir.org> #332: Portmap analyzer segfaults when parsing portmap dump replies --------------------+-------------------- Reporter: gregor | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Keywords: | --------------------+-------------------- {{{ #!rst There is a bug in the portmap analyzer that causes a segfault when parsing portmap dump reply message. The attached patch fixes the problem. Should this patch be included in the final 1.5.x release, since it fixes a segfault? *Background:* Note, part of the problem might actually be the way binpac handles array building, which can be confusing. I'll just put it here for the record in case somebody else stumbles across something similar. from portmap-protocol.pac:: type PortmapDumpEntry = record { cont: uint32; optmapping: case cont of { 0 -> none: empty; default -> mapping: PortmapMapping; }; }; type PortmapDumpResults = PortmapDumpEntry[] &until($element.cont != 1); Binpac will parse a PortmapDumpEntry out if the input data, add it to the PortmapDumpResults[] array, evaluate the until expression and quit if it is false. However, there is another form of the the until expression that performs test on the chunk of input, e.g.:: type HTTP_Headers = HTTP_Header[] &until($input.length() == 0); type MIME_Lines = MIME_Line[] &until($context.flow.is_end_of_multipart($input)); These until expression with $input are evaluated *before* the next element is parsed and if it is false, no more elements are parsed. The behavior makes sense, since it makes sense to perform test on the input before the input is parsed, and tests on the output after the output is parsed (However, one could ask, whether elements for which the until test were false should be added to the array). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri Dec 3 07:49:35 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 03 Dec 2010 15:49:35 -0000 Subject: [Bro-Dev] #332: Portmap analyzer segfaults when parsing portmap dump replies In-Reply-To: <045.fc87d35e27cf53e26f8771e6f7836df6@tracker.icir.org> References: <045.fc87d35e27cf53e26f8771e6f7836df6@tracker.icir.org> Message-ID: <060.acc2ec265d4e290643746f461f92893c@tracker.icir.org> #332: Portmap analyzer segfaults when parsing portmap dump replies ---------------------+-------------------- Reporter: gregor | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: ---------------------+-------------------- Comment (by seth): > Should this patch be included in the final 1.5.x release, since it > fixes a segfault? I'm inclined to delay on this until 1.6 since it has likely been a problem for a long time it has never been reported. I think the fewer changes we put into the final 1.5.x release, the better. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri Dec 3 08:33:39 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 03 Dec 2010 16:33:39 -0000 Subject: [Bro-Dev] #332: Portmap analyzer segfaults when parsing portmap dump replies In-Reply-To: <045.fc87d35e27cf53e26f8771e6f7836df6@tracker.icir.org> References: <045.fc87d35e27cf53e26f8771e6f7836df6@tracker.icir.org> Message-ID: <060.4f1e91c4d2c61afc78facd67060d5885@tracker.icir.org> #332: Portmap analyzer segfaults when parsing portmap dump replies ---------------------+-------------------- Reporter: gregor | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: ---------------------+-------------------- Comment (by robin): On Fri, Dec 03, 2010 at 15:49 -0000, you wrote: > I'm inclined to delay on this until 1.6 since it has likely been a problem > for a long time it has never been reported. I think the fewer changes we > put into the final 1.5.x release, the better. Same here. Let's do the 1.5.x soon, and then be done with it. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri Dec 3 08:42:07 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 03 Dec 2010 16:42:07 -0000 Subject: [Bro-Dev] #331: Remove snort2bro In-Reply-To: <043.60d18b4614b2db9168bd2be18da72288@tracker.icir.org> References: <043.60d18b4614b2db9168bd2be18da72288@tracker.icir.org> Message-ID: <058.170ee7611d7edc09746f535486e1324b@tracker.icir.org> #331: Remove snort2bro ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved | Keywords: ----------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved Comment: I think everything regarding snort2bro is already gone in current git master/devel. Closing the ticket, please reopen in case something's found left that still needs to be removed. There's no core support. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri Dec 3 08:44:40 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 03 Dec 2010 16:44:40 -0000 Subject: [Bro-Dev] #333: Add workflow transitions to Trac's email interface. Message-ID: <044.c6f90ad641521efe0cf5637cea94ede6@tracker.icir.org> #333: Add workflow transitions to Trac's email interface. ---------------------------+------------------ Reporter: robin | Owner: seth Type: Task | Status: new Priority: Normal | Milestone: Component: TicketTracker | Version: Keywords: | ---------------------------+------------------ Currently one can't simply close a ticket via mail while setting the resolution correctly. See https://subtrac.sara.nl/oss/email2trac/wiki/Email2tracConfiguration#Workflow Note that this should probably wait until we have updated the workflow we use in Trac (like with getting rid of the distinction between new/seen). -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Fri Dec 3 09:34:24 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 03 Dec 2010 09:34:24 -0800 Subject: [Bro-Dev] NFS Analyzer and UDP Message-ID: <4CF92A20.2090805@icir.org> Hi, it seems that although the NFS-Analyzer is started on UDP connections, it doesn't handle UDP connection, since it's missing a DeliverPacket() method. Is this intentional or something that got overlooked at some stage? (It does handle TCP connection and seems to be able to sync to midstream connections....) cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From robin at icir.org Fri Dec 3 09:40:29 2010 From: robin at icir.org (Robin Sommer) Date: Fri, 3 Dec 2010 09:40:29 -0800 Subject: [Bro-Dev] NFS Analyzer and UDP In-Reply-To: <4CF92A20.2090805@icir.org> References: <4CF92A20.2090805@icir.org> Message-ID: <20101203174029.GF52691@icir.org> On Fri, Dec 03, 2010 at 09:34 -0800, you wrote: > it seems that although the NFS-Analyzer is started on UDP connections, > it doesn't handle UDP connection, since it's missing a DeliverPacket() > method. Is this intentional or something that got overlooked at some stage? Probably an oversight, either in the original analyzer or while porting to the new analyzer framework. I think the problem with the NFS analyzer is that nobody is really using it a lot, and the test-suite probably doesn't catch any errors either. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Fri Dec 3 10:53:42 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 03 Dec 2010 10:53:42 -0800 Subject: [Bro-Dev] NFS Analyzer and UDP In-Reply-To: <4CF92A20.2090805@icir.org> References: <4CF92A20.2090805@icir.org> Message-ID: <4CF93CB6.70304@icir.org> actually, it does handle UDP connections. I just missed that part of the code. Sorry for the fuss... cu gregor On 12/3/10 9:34 , Gregor Maier wrote: > Hi, > > it seems that although the NFS-Analyzer is started on UDP connections, > it doesn't handle UDP connection, since it's missing a DeliverPacket() > method. Is this intentional or something that got overlooked at some stage? > (It does handle TCP connection and seems to be able to sync to midstream > connections....) > > cu > Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From seth at icir.org Fri Dec 3 11:50:12 2010 From: seth at icir.org (Seth Hall) Date: Fri, 3 Dec 2010 14:50:12 -0500 Subject: [Bro-Dev] Script comment periods Message-ID: <8E8B86BB-87CF-48AC-8C97-A007C10683BF@icir.org> After a conversation with Robin in the airport, I'm going to start regular script comment periods starting at the beginning of January. I'll send my comments on several (2-4?) Bro scripts to the bro-dev mailing list every Monday and Wednesday to allow for a couple of days of comments and discussion. Then, I'll coalesce those into something more succinct that will be posted somewhere. I'm bringing this up ahead of time because I wanted to see what people thought about announcing the bro-dev mailing list to the users list because I think it's a really good learning opportunity for people to watch Bro development in progress and I'd like as many people as are interested to join the discussion. Unless I hear complaints, I'm going to go ahead and announce the bro-dev list next week. .Seth From seth at icir.org Fri Dec 3 12:06:53 2010 From: seth at icir.org (Seth Hall) Date: Fri, 3 Dec 2010 15:06:53 -0500 Subject: [Bro-Dev] Thoughts on documenting scripts In-Reply-To: <20101123022439.GA63257@icir.org> References: <20101123022439.GA63257@icir.org> Message-ID: <43B070C6-99EE-49E5-9BE8-F6BBE77CA827@icir.org> On Nov 22, 2010, at 9:24 PM, Robin Sommer wrote: > I've added a page on how we might use Sphinx to auto-generate Bro > script documentation to the projects list, see One question I just came across was how to document the values for an enum. My example is below, I think it would be nice to either link to the potential values for that enum or include them inline (I think I prefer inline). # Enable and disable the log. If you only want logs for certain transactions, # that can be configured with this variable as well. const logging: Directions = LocalHosts &redef; In another file, the Directions enum might be: type Directions: enum{Inbound, Outbound, Enabled, Disabled}; It's a slightly convoluted example because the code doesn't exactly match the implementation that I currently have in my logging framework but it doesn't really matter since that will be changing anyway. .Seth From seth at icir.org Fri Dec 3 12:18:34 2010 From: seth at icir.org (Seth Hall) Date: Fri, 3 Dec 2010 15:18:34 -0500 Subject: [Bro-Dev] globals, consts, and &redef Message-ID: <08914E88-E7C9-47BD-ABD7-B289874E2310@icir.org> There was some discussion during the meeting at NCSA about the current applicability of the &redef attribute. Does it make sense to deprecate the attribute? It appears that the primary reason for it's existence is to create a public API to analysis scripts because you can withhold the &redef attribute from consts that you would like to remain private and apply it to consts you want to be public but that public/private distinction can now be implicitly defined with the export section in scripts. There is at least one problem that has come to light due to the &redef attribute being applied to globals[1] and it started making me wonder if it conceptually even makes sense to apply &redef to a global? It seems that the situations where &redef is applied to a global, the variable should actually either be a const or just be a global without the &redef attribute. Many of these questions are of course for Vern because I don't know if I'm misinterpreting the purpose and lineage of the &redef attribute. It does come across to me that during the script clean up, we could deprecate the &redef attribute and move "redef-able" variables into the export section of all of the scripts. I think that consts outside of modules will be inherently redef-able, we'd essentially be using modules as the way to "hide" variables. .Seth 1. http://tracker.icir.org/bro/ticket/200 From vern at icir.org Fri Dec 3 12:22:50 2010 From: vern at icir.org (Vern Paxson) Date: Fri, 03 Dec 2010 12:22:50 -0800 Subject: [Bro-Dev] Script comment periods In-Reply-To: <8E8B86BB-87CF-48AC-8C97-A007C10683BF@icir.org> (Fri, 03 Dec 2010 14:50:12 EST). Message-ID: <20101203202250.BABF736A41A@taffy.ICSI.Berkeley.EDU> > Unless I hear complaints, I'm going to go ahead and announce the bro-dev > list next week. I presume that this won't include the archives for previous bro-dev discussions? Because I imagine those contain some not-for-public-consumption postings. Vern From seth at icir.org Fri Dec 3 12:25:05 2010 From: seth at icir.org (Seth Hall) Date: Fri, 3 Dec 2010 15:25:05 -0500 Subject: [Bro-Dev] Script comment periods In-Reply-To: <20101203202250.BABF736A41A@taffy.ICSI.Berkeley.EDU> References: <20101203202250.BABF736A41A@taffy.ICSI.Berkeley.EDU> Message-ID: <5F93A729-8F5B-4399-B279-DAAD9CCD3A99@icir.org> On Dec 3, 2010, at 3:22 PM, Vern Paxson wrote: > I presume that this won't include the archives for previous bro-dev > discussions? Because I imagine those contain some not-for-public-consumption > postings. Nope, just announcing the fairly freshly created bro-dev at bro-ids.org. I think the old bro-dev at george.lbl.gov is gone where those conversations would have taken place. .Seth From seth at icir.org Fri Dec 3 12:25:55 2010 From: seth at icir.org (Seth Hall) Date: Fri, 3 Dec 2010 15:25:55 -0500 Subject: [Bro-Dev] Script comment periods In-Reply-To: <20101203202250.BABF736A41A@taffy.ICSI.Berkeley.EDU> References: <20101203202250.BABF736A41A@taffy.ICSI.Berkeley.EDU> Message-ID: On Dec 3, 2010, at 3:22 PM, Vern Paxson wrote: > I presume that this won't include the archives for previous bro-dev > discussions? Because I imagine those contain some not-for-public-consumption > postings. Yep, only going back to September... http://mailman.icsi.berkeley.edu/pipermail/bro-dev/ .Seth From slagell at ncsa.illinois.edu Fri Dec 3 12:26:27 2010 From: slagell at ncsa.illinois.edu (Adam J. Slagell) Date: Fri, 3 Dec 2010 14:26:27 -0600 Subject: [Bro-Dev] Script comment periods In-Reply-To: <20101203202250.BABF736A41A@taffy.ICSI.Berkeley.EDU> References: <20101203202250.BABF736A41A@taffy.ICSI.Berkeley.EDU> Message-ID: <0CAF9A3C-3F47-4F9F-801F-9DFF152E8F65@ncsa.illinois.edu> On Dec 3, 2010, at 2:22 PM, Vern Paxson wrote: >> Unless I hear complaints, I'm going to go ahead and announce the bro-dev >> list next week. > > I presume that this won't include the archives for previous bro-dev > discussions? Because I imagine those contain some not-for-public-consumption > postings. > > Vern Perfect Forward Security model for email lists. :-) From seth at icir.org Fri Dec 3 13:02:32 2010 From: seth at icir.org (Seth Hall) Date: Fri, 3 Dec 2010 16:02:32 -0500 Subject: [Bro-Dev] Key-value support for syslog Message-ID: <17DA5739-D010-47C8-8EA7-AB6A00684FF9@icir.org> I just sent an email off to one of the syslog-ng developers to see if they have support for this... http://tools.ietf.org/search/rfc5424#section-6.3 .Seth From robin at icir.org Fri Dec 3 15:23:03 2010 From: robin at icir.org (Robin Sommer) Date: Fri, 3 Dec 2010 15:23:03 -0800 Subject: [Bro-Dev] globals, consts, and &redef In-Reply-To: <08914E88-E7C9-47BD-ABD7-B289874E2310@icir.org> References: <08914E88-E7C9-47BD-ABD7-B289874E2310@icir.org> Message-ID: <20101203232303.GI63741@icir.org> On Fri, Dec 03, 2010 at 15:18 -0500, you wrote: > attribute. It does come across to me that during the script clean > up, we could deprecate the &redef attribute and move "redef-able" > variables into the export section of all of the scripts. I like this. It also matches with how we have already been moving away from non-const-but-redefable globals for a while. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Fri Dec 3 15:27:45 2010 From: robin at icir.org (Robin Sommer) Date: Fri, 3 Dec 2010 15:27:45 -0800 Subject: [Bro-Dev] Thoughts on documenting scripts In-Reply-To: <43B070C6-99EE-49E5-9BE8-F6BBE77CA827@icir.org> References: <20101123022439.GA63257@icir.org> <43B070C6-99EE-49E5-9BE8-F6BBE77CA827@icir.org> Message-ID: <20101203232745.GJ63741@icir.org> On Fri, Dec 03, 2010 at 15:06 -0500, you wrote: > One question I just came across was how to document the values for > an enum. My example is below, I think it would be nice to either > link to the potential values for that enum or include them inline (I > think I prefer inline). I'm not quite sure which alternatives you describe here, but my thiking would be to document (1) the enum tyoe itself, and then (2) the various values indvidually, e.g.: # Defines the alternatives for XYZ. type Directions: enum { ## Description for inbound. Inbound, ## Description for outbound. Outbound ... }; Is that what you mean by "inline"? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Fri Dec 3 17:22:39 2010 From: seth at icir.org (Seth Hall) Date: Fri, 3 Dec 2010 20:22:39 -0500 Subject: [Bro-Dev] Thoughts on documenting scripts In-Reply-To: <20101203232745.GJ63741@icir.org> References: <20101123022439.GA63257@icir.org> <43B070C6-99EE-49E5-9BE8-F6BBE77CA827@icir.org> <20101203232745.GJ63741@icir.org> Message-ID: <8D80DA7F-42E1-4E70-843B-B3022D695391@icir.org> On Dec 3, 2010, at 6:27 PM, Robin Sommer wrote: > Is that what you mean by "inline"? I meant inline in the output documentation as opposed to making you click a link to see the potential values for the enum. Does that make sense? Maybe in the output documentation, each enum value that is displayed could be a link to the documentation for the specific value, I hadn't even considered that yet. .Seth From seth at icir.org Fri Dec 3 17:30:48 2010 From: seth at icir.org (Seth Hall) Date: Fri, 3 Dec 2010 20:30:48 -0500 Subject: [Bro-Dev] Key-value support for syslog In-Reply-To: <17DA5739-D010-47C8-8EA7-AB6A00684FF9@icir.org> References: <17DA5739-D010-47C8-8EA7-AB6A00684FF9@icir.org> Message-ID: <99E18E82-D89F-4FB7-AD07-EB771D58470A@icir.org> On Dec 3, 2010, at 4:02 PM, Seth Hall wrote: > I just sent an email off to one of the syslog-ng developers to see if they have support for this... > http://tools.ietf.org/search/rfc5424#section-6.3 Apparently syslog-ng does support this. That will have to be another option for the syslog writer in the "to be written" logging framework. Syslog-ng can actually output to various databases too so it actually might make sense to implement a syslog writer with key-value support early on in the process so that people can start outputting to databases without us even needing to implement database writers. .Seth From robin at icir.org Fri Dec 3 17:33:12 2010 From: robin at icir.org (Robin Sommer) Date: Fri, 3 Dec 2010 17:33:12 -0800 Subject: [Bro-Dev] Thoughts on documenting scripts In-Reply-To: <8D80DA7F-42E1-4E70-843B-B3022D695391@icir.org> References: <20101123022439.GA63257@icir.org> <43B070C6-99EE-49E5-9BE8-F6BBE77CA827@icir.org> <20101203232745.GJ63741@icir.org> <8D80DA7F-42E1-4E70-843B-B3022D695391@icir.org> Message-ID: <20101204013312.GD70534@icir.org> On Fri, Dec 03, 2010 at 20:22 -0500, you wrote: > I meant inline in the output documentation as opposed to making you click a link to see the potential values for the enum. Does that make sense? Ah, I misunderstood, sorry. Yes, inline then sounds fine to me. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Fri Dec 3 17:35:35 2010 From: robin at icir.org (Robin Sommer) Date: Fri, 3 Dec 2010 17:35:35 -0800 Subject: [Bro-Dev] Masters updated In-Reply-To: <1371758317.6239.1291153651929.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <20101128044744.GB75232@icir.org> <1371758317.6239.1291153651929.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20101204013535.GE70534@icir.org> On Tue, Nov 30, 2010 at 15:47 -0600, you wrote: > If you want to merge topic/cmake-port across the board into master Done (and into devel). The merging is really easy. Updating the submodules is somewhat cumbersome, but perhaps I haven't found the right recipe for doing all of them at once yet (I'm already using "git submodule foreach" but still). Robin PS: I've installed a new commit notification script which for now is sending just to me. If it works fine, I'll remove the old one and switch over. -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Fri Dec 3 18:35:34 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 03 Dec 2010 18:35:34 -0800 Subject: [Bro-Dev] Unique connection ID for bro <-> logging framework Message-ID: <4CF9A8F6.1020205@icir.org> Hi, I was wondering whether it would make sense to assign each connection an ID that's unique for this bro run. This ID can just be a 64-bit counter that gets incremented on every new connection. Why: If we add this ID to log outputs, it would be much easier to correlate activity across logs (e.g., find the connection in http.log, alarm.log, and conn.log, without having to match 5-tuples and timestamps) I think this would be a rather nice (and very easy to implement) feature. Cluster considerations: maybe add a nodeID or something to the connection ID. E.g., in the high-order 8 or 16 bits. Thoughts? Comments? cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From seth at icir.org Fri Dec 3 18:42:32 2010 From: seth at icir.org (Seth Hall) Date: Fri, 3 Dec 2010 21:42:32 -0500 Subject: [Bro-Dev] Unique connection ID for bro <-> logging framework In-Reply-To: <4CF9A8F6.1020205@icir.org> References: <4CF9A8F6.1020205@icir.org> Message-ID: On Dec 3, 2010, at 9:35 PM, Gregor Maier wrote: > I was wondering whether it would make sense to assign each connection an > ID that's unique for this bro run. This ID can just be a 64-bit counter > that gets incremented on every new connection. That's an interesting idea. > Why: If we add this ID to log outputs, it would be much easier to > correlate activity across logs (e.g., find the connection in http.log, > alarm.log, and conn.log, without having to match 5-tuples and timestamps) My only question is under what circumstance you do that activity correlation activity within a single connection? I'm unable to think of a single time when I've needed to do something like that where I wasn't able to just search for the single IP address that I was interested in because I was interested in anything that IP address was referenced in and not just that single connection. .Seth From gregor at icir.org Fri Dec 3 19:04:32 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 03 Dec 2010 19:04:32 -0800 Subject: [Bro-Dev] Unique connection ID for bro <-> logging framework In-Reply-To: References: <4CF9A8F6.1020205@icir.org> Message-ID: <4CF9AFC0.1090203@icir.org> >> Why: If we add this ID to log outputs, it would be much easier to >> correlate activity across logs (e.g., find the connection in http.log, >> alarm.log, and conn.log, without having to match 5-tuples and timestamps) > > My only question is under what circumstance you do that activity correlation activity within a single connection? I'm unable to think of a single time when I've needed to do something like that where I wasn't able to just search for the single IP address that I was interested in because I was interested in anything that IP address was referenced in and not just that single connection. Some examples: * I want to count the number of HTTP request per connection * I do per connection stats (e.g., number of packets, number of bytes, retransmissions, RTTs), store them in their own log files and then want to correlate with the conn.log or the http.log * Easier debugging / analysis: I can just grep for the connectionID, instead of having to map between different connection formattings (e.g., notices have origIP:origPort -> respIP:respPort but when I want to grep for them in conn.log, I have to do some awk to get there) * ... I guess I have more a measurement point of view here.... cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From seth at icir.org Fri Dec 3 20:07:57 2010 From: seth at icir.org (Seth Hall) Date: Fri, 3 Dec 2010 23:07:57 -0500 Subject: [Bro-Dev] Unique connection ID for bro <-> logging framework In-Reply-To: <4CF9AFC0.1090203@icir.org> References: <4CF9A8F6.1020205@icir.org> <4CF9AFC0.1090203@icir.org> Message-ID: On Dec 3, 2010, at 10:04 PM, Gregor Maier wrote: > * I want to count the number of HTTP request per connection Ah, ok. Now that you mention it, I have done searches for that before too. :) > (e.g., notices > have origIP:origPort -> respIP:respPort but when I want to grep for > them in conn.log, I have to do some awk to get there) If the logging framework proceeds in the direction that Robin and I have been outlining, most of this trouble will go away. > I guess I have more a measurement point of view here.... Yeah, makes sense. I just wasn't understanding that before. :) Reading the things you need to do does remind me that Justin Azoff and I need to get back to the metrics framework we've been talking about. It could help you output logs with a lot of the measurement type data you are looking to get instead of having to do post-processing on the existing logs. Getting back to your question though, it's an interesting idea but I wonder if it will still be necessary once the "normal" logging output changes. At the very least, if you output tab separated value data, you should be able to do something like this.... cat whatever.log | grep "1.2.3.4352314.3.2.180" The binary log output may make that even easier too. .Seth From gregor at icir.org Fri Dec 3 22:53:48 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 03 Dec 2010 22:53:48 -0800 Subject: [Bro-Dev] Unique connection ID for bro <-> logging framework In-Reply-To: References: <4CF9A8F6.1020205@icir.org> <4CF9AFC0.1090203@icir.org> Message-ID: <4CF9E57C.8000107@icir.org> > Getting back to your question though, it's an interesting idea but I wonder if it will still be necessary once the "normal" logging output changes. At the very least, if you output tab separated value data, you should be able to do something like this.... > > cat whatever.log | grep "1.2.3.4352314.3.2.180" In general yes, as long as the 5-tuple isn't reused. (I can basically do this right now, if I use awk to reorder the connection-tuple so that I can grep for it. Might thought was that having a single numeric ID might make life easier.) > The binary log output may make that even easier too. Being able to use grep, sed, awk, and co. is still very nice, so I'll probably end up using a binary to ascii converter quite frequently. cu gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From bro at tracker.icir.org Sun Dec 5 11:27:53 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Sun, 05 Dec 2010 19:27:53 -0000 Subject: [Bro-Dev] #329: Remove detect-protocols-http.bro from Broctl's default local.bro In-Reply-To: <043.4171f047fca9bbca4e3e8c6e560433d7@tracker.icir.org> References: <043.4171f047fca9bbca4e3e8c6e560433d7@tracker.icir.org> Message-ID: <058.83d12e7a0f949bad4e281c0f9697b416@tracker.icir.org> #329: Remove detect-protocols-http.bro from Broctl's default local.bro -------------------------+-------------------- Reporter: seth | Owner: robin Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: -------------------------+-------------------- Comment (by vern): Two thoughts on keeping it by making it more efficient: (1) Rewrite the for loop to instead use the '''match''' primitive. (2) Provide language support for constructing a merged version of the regular expressions listed in the table, and then first test whether there's a match to the disjunction of them before bothering to loop over to find which particular one matched. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Sun Dec 5 11:31:32 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Sun, 05 Dec 2010 19:31:32 -0000 Subject: [Bro-Dev] #257: Commandline flag to enable BRO_FAKE_DNS In-Reply-To: <049.463a3aa2505ed0b2731a01396c4cbef8@tracker.icir.org> References: <049.463a3aa2505ed0b2731a01396c4cbef8@tracker.icir.org> Message-ID: <064.805b17c959b379c438bacdae81615cb0@tracker.icir.org> #257: Commandline flag to enable BRO_FAKE_DNS -------------------------+---------------------- Reporter: brosenberg | Owner: vern Type: Patch | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: -------------------------+---------------------- Comment (by vern): Hmmm, I perhaps. Personally, for the environments where I frequently run off-line (like my laptop or my desktop), I just set BRO_DNS_FAKE in my main environment and this never bothers me. If the only real benefit of adding a flag is it showing up in the command-line help, why not just add BRO_DNS_FAKE there? It already lists BROPATH and BRO_PREFIXES. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Mon Dec 6 07:09:00 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 06 Dec 2010 15:09:00 -0000 Subject: [Bro-Dev] #329: Remove detect-protocols-http.bro from Broctl's default local.bro In-Reply-To: <043.4171f047fca9bbca4e3e8c6e560433d7@tracker.icir.org> References: <043.4171f047fca9bbca4e3e8c6e560433d7@tracker.icir.org> Message-ID: <058.bad6c5073babdd00e94dafd36500cc57@tracker.icir.org> #329: Remove detect-protocols-http.bro from Broctl's default local.bro -------------------------+-------------------- Reporter: seth | Owner: robin Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: -------------------------+-------------------- Comment (by seth): > (1) Rewrite the for loop to instead use the '''match''' primitive. > > (2) Provide language support for constructing a merged version of the > regular expressions listed in the table, and then first test whether > there's a match to the disjunction of them before bothering to loop over > to find which particular one matched. Great ideas! I was actually planning on implementing pattern construction at init time this week anyway. This is generally good though, it will be heading down the path of how signatures are implemented internally but in scripting land. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Mon Dec 6 09:29:24 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 06 Dec 2010 09:29:24 -0800 Subject: [Bro-Dev] Unique connection ID for bro <-> logging framework In-Reply-To: <4CF9A8F6.1020205@icir.org> References: <4CF9A8F6.1020205@icir.org> Message-ID: <4CFD1D74.4040600@icir.org> BTW, the addl field in conn.log is sometimes used for something similar. E.g., http.bro will create a unique ID for each HTTP-session and put this session ID into the connections addl.... cu gregor On 12/3/10 18:35 , Gregor Maier wrote: > Hi, > > I was wondering whether it would make sense to assign each connection an > ID that's unique for this bro run. This ID can just be a 64-bit counter > that gets incremented on every new connection. > > Why: If we add this ID to log outputs, it would be much easier to > correlate activity across logs (e.g., find the connection in http.log, > alarm.log, and conn.log, without having to match 5-tuples and timestamps) > > I think this would be a rather nice (and very easy to implement) feature. > > Cluster considerations: maybe add a nodeID or something to the > connection ID. E.g., in the high-order 8 or 16 bits. > > > Thoughts? > Comments? > cu > Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Mon Dec 6 09:30:40 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 06 Dec 2010 09:30:40 -0800 Subject: [Bro-Dev] Portmapper logging Message-ID: <4CFD1DC0.5050207@icir.org> Hi, I dug around and checked how portmapper.bro does logging, notices, and interaction with other policy scripts. It seems that it * "logs" activity by setting the "addl" field for conn.log * creates NOTICEs for some activity * modifies the services field for connections that have an RPC service (e.g., NFS, ypserv, etc.) See the attached file for details. * The notices generation is very convoluted. I was wondering whether it makes sense to clean that up. (the attached file at least documents the behavior). * I want to add an actual portmapper.log file to log portmapper activity. If we have that, we wouldn't need the "addl" anymore. Is it worth removing it? (Esp. wrt the new logging framework) cu gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: portmapper-logging.txt Url: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20101206/4faec88d/attachment.txt From gregor at icir.org Mon Dec 6 09:33:49 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 06 Dec 2010 09:33:49 -0800 Subject: [Bro-Dev] RPC, portmapper, and NFS oddities Message-ID: <4CFD1E7D.4060104@icir.org> Hi, I've come across some "oddities" and some things that maybe "could be done nice (TM)" in the RPC, portmapper, and NFS code. I don't know whether it's worth tackling any/some of them. Presumably only when we move the analyzer to binpacc++ at some stage. OTOH, some of the points might be worth addressing.... cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rpc-notes.txt Url: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20101206/58dec11c/attachment.txt From vern at icir.org Mon Dec 6 09:57:00 2010 From: vern at icir.org (Vern Paxson) Date: Mon, 06 Dec 2010 09:57:00 -0800 Subject: [Bro-Dev] Portmapper logging In-Reply-To: <4CFD1DC0.5050207@icir.org> (Mon, 06 Dec 2010 09:30:40 PST). Message-ID: <20101206175700.6B47C36A416@taffy.ICSI.Berkeley.EDU> > I dug around and checked how portmapper.bro does logging, notices, and > interaction with other policy scripts. It seems that it Yeah, portmapper.bro is in fact one of the earliest script components that tackled complex protocols interactions at the scripting level. (The other is FTP.) Because it predates the whole notice framework, it has a bunch of cruft that approximates the sort of policy filtering that later gave birth to notices. > * The notices generation is very convoluted. I was wondering whether it > makes sense to clean that up. (the attached file at least documents > the behavior). I am a *strong* believer in don't fix things like this unless there's a pressing need. I'm assuming your interest here is driven by the NFS analysis for the enterprise traces. If so, it would make sense to fix (solely) those things necessary to expedite that analysis. But don't go further; it's too much of a potential time sink (cleaning up code can be beguilingly fun :-) for insufficient benefit. Vern From jsiwek at ncsa.illinois.edu Mon Dec 6 10:26:23 2010 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 6 Dec 2010 12:26:23 -0600 (CST) Subject: [Bro-Dev] Masters updated In-Reply-To: <20101204013535.GE70534@icir.org> Message-ID: <4558123.4.1291659980448.JavaMail.jsiwek@tangent.ncsa.illinois.edu> > The merging is really easy. Updating the submodules is somewhat > cumbersome, but perhaps I haven't found the right recipe for doing > all of them at once yet (I'm already using "git submodule foreach" > but still). That's the best way I found; I don't think it will be that bad after things settle down and we don't need to make these updates that span all repositories. Can you re-check that the broccoli/master repository has an updated submodule for broccoli-python; it didn't look like it was using the master version of broccoli-python. - Jon From bro at tracker.icir.org Mon Dec 6 11:08:00 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 06 Dec 2010 19:08:00 -0000 Subject: [Bro-Dev] #334: Portmapper.bro documentation and script interaction Message-ID: <045.22091bfa7a1b9ed9a5da7d14447b0c6f@tracker.icir.org> #334: Portmapper.bro documentation and script interaction ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ Hi, just adding this ticket so the the information doesn't get lost. Notes on how portmapper.bro does its logging and interaction with other scripts. Hopefully helpful for the script documentation / cleanup push. See also: http://bro.icir.org/devel/rpc-portmap-nfs-notes.html -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Mon Dec 6 11:37:30 2010 From: seth at icir.org (Seth Hall) Date: Mon, 6 Dec 2010 14:37:30 -0500 Subject: [Bro-Dev] Portmapper logging In-Reply-To: <4CFD1DC0.5050207@icir.org> References: <4CFD1DC0.5050207@icir.org> Message-ID: On Dec 6, 2010, at 12:30 PM, Gregor Maier wrote: > * I want to add an actual portmapper.log file to log portmapper > activity. If we have that, we wouldn't need the "addl" anymore. Is it > worth removing it? (Esp. wrt the new logging framework) I would really like to see any activity logs for policy scripts moved out into their own logs. At OSU for instance, we didn't even keep the conn.log (we closed the log file) because it was mostly repetitive data that we didn't get much benefit from keeping. It's obviously much faster to grep smaller logs too. :)  .Seth From gregor at icir.org Mon Dec 6 11:41:57 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 06 Dec 2010 11:41:57 -0800 Subject: [Bro-Dev] Portmapper logging In-Reply-To: References: <4CFD1DC0.5050207@icir.org> Message-ID: <4CFD3C85.7000702@icir.org> On 12/6/10 11:37 , Seth Hall wrote: > > On Dec 6, 2010, at 12:30 PM, Gregor Maier wrote: > >> * I want to add an actual portmapper.log file to log portmapper >> activity. If we have that, we wouldn't need the "addl" anymore. Is it >> worth removing it? (Esp. wrt the new logging framework) > > I would really like to see any activity logs for policy scripts moved out into their own logs. At OSU for instance, we didn't even keep the conn.log (we closed the log file) because it was mostly repetitive data that we didn't get much benefit from keeping. It's obviously much faster to grep smaller logs too. :) That's the one change I actually need/want to do for my NFS/portmapper analysis. I want to have this information in its own portmapper.log file. Removing the code that adds stuff to "addl" in portmapper.bro is straight-forward if desired. cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From vern at icir.org Mon Dec 6 12:44:53 2010 From: vern at icir.org (Vern Paxson) Date: Mon, 06 Dec 2010 12:44:53 -0800 Subject: [Bro-Dev] Portmapper logging In-Reply-To: (Mon, 06 Dec 2010 14:37:30 EST). Message-ID: <20101206204453.22A0836A031@taffy.ICSI.Berkeley.EDU> > I would really like to see any activity logs for policy scripts moved > out into their own logs. I like that notion too. (But still with the don't-fix-what's-not-broke model of prioritization.) > At OSU for instance, we didn't even keep the conn.log (we closed the log > file) because it was mostly repetitive data that we didn't get much benefit > from keeping. I find that very surprising. At LBL, the conn logs are often of crucial forensic significance. One form of this comes up when an attacker sets up a backdoor on port XYZ, which will only appear in the conn logs. Another is when the activity involves a service for which there's no Bro analyzer. Vern From vern at icir.org Mon Dec 6 12:45:43 2010 From: vern at icir.org (Vern Paxson) Date: Mon, 06 Dec 2010 12:45:43 -0800 Subject: [Bro-Dev] Portmapper logging In-Reply-To: <4CFD3C85.7000702@icir.org> (Mon, 06 Dec 2010 11:41:57 PST). Message-ID: <20101206204543.45B5E36A031@taffy.ICSI.Berkeley.EDU> > That's the one change I actually need/want to do for my NFS/portmapper > analysis. I want to have this information in its own portmapper.log (Fair 'nuf, then. I withdraw my don't-fix-what's-not-broke concern.) From seth at icir.org Mon Dec 6 12:47:50 2010 From: seth at icir.org (Seth Hall) Date: Mon, 6 Dec 2010 15:47:50 -0500 Subject: [Bro-Dev] Portmapper logging In-Reply-To: <20101206204453.22A0836A031@taffy.ICSI.Berkeley.EDU> References: <20101206204453.22A0836A031@taffy.ICSI.Berkeley.EDU> Message-ID: On Dec 6, 2010, at 3:44 PM, Vern Paxson wrote: > I find that very surprising. At LBL, the conn logs are often of crucial > forensic significance. One form of this comes up when an attacker sets > up a backdoor on port XYZ, which will only appear in the conn logs. Another > is when the activity involves a service for which there's no Bro analyzer. Netflow logs. :) OSU being OSU, we tried to carefully maintain our archive of netflow. .Seth From bro at tracker.icir.org Mon Dec 6 12:55:46 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 06 Dec 2010 20:55:46 -0000 Subject: [Bro-Dev] #335: Small warning about "main" Message-ID: <043.c670c730a2bb23ea92564c42e5dd23a9@tracker.icir.org> #335: Small warning about "main" ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- Small warning during compile about the main function:: {{{ main.cc: In function ?int main(int, char**)?: main.cc:415: warning: deprecated conversion from string constant to ?char*? }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Mon Dec 6 13:33:03 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 06 Dec 2010 13:33:03 -0800 Subject: [Bro-Dev] Portmapper logging In-Reply-To: <20101206204543.45B5E36A031@taffy.ICSI.Berkeley.EDU> References: <20101206204543.45B5E36A031@taffy.ICSI.Berkeley.EDU> Message-ID: <4CFD568F.1080702@icir.org> On 12/6/10 12:45 , Vern Paxson wrote: >> That's the one change I actually need/want to do for my NFS/portmapper >> analysis. I want to have this information in its own portmapper.log > > (Fair 'nuf, then. I withdraw my don't-fix-what's-not-broke concern.) No worries. I think I'm in general a bit more "trigger-happy" with fixing/changing stuff..... ;-) cu gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From seth at icir.org Mon Dec 6 14:09:39 2010 From: seth at icir.org (Seth Hall) Date: Mon, 6 Dec 2010 17:09:39 -0500 Subject: [Bro-Dev] Portmapper logging In-Reply-To: <4CFD568F.1080702@icir.org> References: <20101206204543.45B5E36A031@taffy.ICSI.Berkeley.EDU> <4CFD568F.1080702@icir.org> Message-ID: <10347496-4B69-4BAA-8CCE-799E900A1ED6@icir.org> On Dec 6, 2010, at 4:33 PM, Gregor Maier wrote: > No worries. I think I'm in general a bit more "trigger-happy" with > fixing/changing stuff..... ;-) I can understand that! I say go ahead and make changes you want and store them somewhere, then when the review time for the portmapper script comes around, we'll use your version rather than the existing version. .Seth From gregor at icir.org Mon Dec 6 14:31:36 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 06 Dec 2010 14:31:36 -0800 Subject: [Bro-Dev] Per connection byte and packet counting Message-ID: <4CFD6448.2020306@icir.org> Hi, I've played around with adding support to count packets and (IP-)bytes per connection and direction. (Sort of like in addition to conn.log's osize and rsize, which are based on sequence numbers). This adds 4 counters per connection. Basically, there are two ways to implement this: a) hard coded into the transport analyzers that keep track of the connection record (which includes osize and rsize). I've done that and added the counters to the "endpoint" record val. (A connection record val has an orig and resp "endpoint" record val). Advantage: every connection event, i.e., the majority of events, now has access to these counters whenever the event is generated. Disadvantage: one cannot turn it off and it will result in a memory overhead per connection (see below). b) implement it as a child-analyzer to the transport analyzers and add this child-analyzer to the initial analyzer tree. One can configure whether to enable/disable this size-analyzer via policy scripts. I've done that too. Advantage: can be switched on and off. No overhead when off. Disadvantage / Questions: * How to pass the information to policy scripts? (see below) * Slightly slower due to calls to an additional analyzer (and virtualized method lookups) I've implemented both variants to check their performance. Here are the results using a quite large Time Machine trace and only loading conn and weird. The trace has some 14.8 M connections total. * trunk: 26.5min--27min runtime, 900MB virtual mem, 890MB residual. * a: 27min--28min runtime, 960MB virtual mem, 948MB residual. * b: 29.25min runtime, 952MB virtual mem, 940MB residual. WRT variant b) (separate analyzer): So far I've implemented an event when the analyzer is Done, i.e., when the connection ends. So, the policy script can collect the counter values only after the connection ends. However, I think it might be helpful, if one could access these counters at any time. * Is there any way to do so? E.g., by having a table that's indexed by connid that can be read from the policy layer but is updated by the event engine? Can this be done from an analyzer, or are there some hacks required? * Other ideas to make the counters available to the policy script. * Do you guys think it's important to have access to the counters at any time, or is it sufficient to just get them when the connection ends? Any thoughts or comments? Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Mon Dec 6 14:41:42 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 06 Dec 2010 14:41:42 -0800 Subject: [Bro-Dev] #335: Small warning about "main" In-Reply-To: <043.c670c730a2bb23ea92564c42e5dd23a9@tracker.icir.org> References: <043.c670c730a2bb23ea92564c42e5dd23a9@tracker.icir.org> Message-ID: <4CFD66A6.1000604@icir.org> On 12/6/10 12:55 , Bro Tracker wrote: > #335: Small warning about "main" > ---------------------+-------------------- > Reporter: seth | Owner: > Type: Problem | Status: new > Priority: Normal | Milestone: Bro1.6 > Component: Bro | Version: > Keywords: | > ---------------------+-------------------- > Small warning during compile about the main function:: > > {{{ > main.cc: In function ?int main(int, char**)?: > main.cc:415: warning: deprecated conversion from string constant to > ?char*? > }}} There are a ton of compiler warnings with current gcc. So it might make sense to make a generic: "track down and remove all compiler warnings" ticket..... cu gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From bro at tracker.icir.org Mon Dec 6 17:40:52 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 07 Dec 2010 01:40:52 -0000 Subject: [Bro-Dev] #335: Fix compiler warnings (was: Small warning about "main") In-Reply-To: <043.c670c730a2bb23ea92564c42e5dd23a9@tracker.icir.org> References: <043.c670c730a2bb23ea92564c42e5dd23a9@tracker.icir.org> Message-ID: <058.1efea5d8c1102d3d59b37071997213fd@tracker.icir.org> #335: Fix compiler warnings ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Description changed by seth: Old description: > Small warning during compile about the main function:: > > {{{ > main.cc: In function ?int main(int, char**)?: > main.cc:415: warning: deprecated conversion from string constant to > ?char*? > }}} New description: {{{ main.cc: In function ?int main(int, char**)?: main.cc:415: warning: deprecated conversion from string constant to ?char*? }}} {{{ DCE_RPC.cc: In constructor ?UUID::UUID(const char*)?: DCE_RPC.cc:85: warning: format not a string literal and no format arguments }}} {{{ PrefixTable.cc: In member function ?void* PrefixTable::Lookup(const Val*, bool) const?: PrefixTable.cc:103: warning: format not a string literal and no format arguments }}} {{{ RemoteSerializer.cc: In member function ?bool RemoteSerializer::DoMessage()?: RemoteSerializer.cc:1508: warning: format ?%d? expects type ?int?, but argument 3 has type ?long long unsigned int? RemoteSerializer.cc:1514: warning: format ?%d? expects type ?int?, but argument 3 has type ?long long unsigned int? RemoteSerializer.cc: In member function ?bool RemoteSerializer::SendToChild(char, RemoteSerializer::Peer*, char*, int)?: RemoteSerializer.cc:2601: warning: format ?%d? expects type ?int?, but argument 3 has type ?SourceID? RemoteSerializer.cc: In member function ?bool RemoteSerializer::SendToChild(char, RemoteSerializer::Peer*, int, ...)?: RemoteSerializer.cc:2625: warning: format ?%d? expects type ?int?, but argument 3 has type ?SourceID? RemoteSerializer.cc: In member function ?void RemoteSerializer::InternalCommError(const char*)?: RemoteSerializer.cc:2706: warning: format not a string literal and no format arguments RemoteSerializer.cc: In member function ?bool SocketComm::ProcessParentMessage()?: RemoteSerializer.cc:3056: warning: format not a string literal and no format arguments RemoteSerializer.cc: In member function ?bool SocketComm::ForwardChunkToPeer()?: RemoteSerializer.cc:3226: warning: format ?%d? expects type ?int?, but argument 2 has type ?SourceID? RemoteSerializer.cc: In member function ?bool SocketComm::ProcessRemoteMessage(SocketComm::Peer*)?: RemoteSerializer.cc:3309: warning: format ?%d? expects type ?int?, but argument 3 has type ?SourceID? RemoteSerializer.cc: In member function ?bool SocketComm::SendToParent(char, SocketComm::Peer*, const char*, int)?: RemoteSerializer.cc:3786: warning: format ?%d? expects type ?int?, but argument 3 has type ?SourceID? RemoteSerializer.cc: In member function ?bool SocketComm::SendToParent(char, SocketComm::Peer*, int, ...)?: RemoteSerializer.cc:3805: warning: format ?%d? expects type ?int?, but argument 3 has type ?SourceID? RemoteSerializer.cc: In member function ?bool SocketComm::SendToPeer(SocketComm::Peer*, char, const char*, int)?: RemoteSerializer.cc:3841: warning: format ?%d? expects type ?int?, but argument 3 has type ?SourceID? RemoteSerializer.cc: In member function ?bool SocketComm::SendToPeer(SocketComm::Peer*, char, int, ...)?: RemoteSerializer.cc:3860: warning: format ?%d? expects type ?int?, but argument 3 has type ?SourceID? RemoteSerializer.cc: In member function ?bool SocketComm::SendToPeer(SocketComm::Peer*, ChunkedIO::Chunk*)?: RemoteSerializer.cc:3881: warning: format ?%d? expects type ?int?, but argument 3 has type ?SourceID? }}} {{{ SMB.cc: In member function ?void SMB_Session::Deliver(int, int, const u_char*)?: SMB.cc:169: warning: format ?%d? expects type ?int?, but argument 3 has type ?long int? }}} {{{ Sessions.cc: In member function ?void NetSessions::Internal(const char*, const pcap_pkthdr*, const u_char*)?: Sessions.cc:1357: warning: format not a string literal and no format arguments }}} {{{ cq.c: In function ?cq_debugbucket?: cq.c:574: warning: format ?%d? expects type ?int?, but argument 4 has type ?long int? }}} {{{ patricia.c: In function ?lookup_then_remove?: patricia.c:1030: warning: suggest parentheses around assignment used as truth value }}} -- -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Mon Dec 6 17:43:35 2010 From: seth at icir.org (Seth Hall) Date: Mon, 6 Dec 2010 20:43:35 -0500 Subject: [Bro-Dev] #335: Small warning about "main" In-Reply-To: <4CFD66A6.1000604@icir.org> References: <043.c670c730a2bb23ea92564c42e5dd23a9@tracker.icir.org> <4CFD66A6.1000604@icir.org> Message-ID: On Dec 6, 2010, at 5:41 PM, Gregor Maier wrote: > There are a ton of compiler warnings with current gcc. So it might make > sense to make a generic: "track down and remove all compiler warnings" > ticket..... Yep. Fixed. :) .Seth From bro at tracker.icir.org Mon Dec 6 19:41:02 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 07 Dec 2010 03:41:02 -0000 Subject: [Bro-Dev] #194: RFB (vnc) analyzer In-Reply-To: <045.2c1d50501927948db25aef301893eb73@tracker.icir.org> References: <045.2c1d50501927948db25aef301893eb73@tracker.icir.org> Message-ID: <060.704ecd8ebf071124a632841549e254eb@tracker.icir.org> #194: RFB (vnc) analyzer ---------------------+--------------------- Reporter: Justin | Owner: Type: Patch | Status: seen Priority: Low | Milestone: Component: Bro | Version: Resolution: | Keywords: RFB VNC ---------------------+--------------------- Comment (by seth): Any chance you'd be interested in working on this analyzer more? The events that it currently generates could be implemented at the scripting/signature layer but it would be awesome if it did more! -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Tue Dec 7 05:47:59 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 07 Dec 2010 13:47:59 -0000 Subject: [Bro-Dev] #257: Commandline flag to enable BRO_FAKE_DNS In-Reply-To: <049.463a3aa2505ed0b2731a01396c4cbef8@tracker.icir.org> References: <049.463a3aa2505ed0b2731a01396c4cbef8@tracker.icir.org> Message-ID: <064.5fd4b244f4349326552e4d72bad0c95a@tracker.icir.org> #257: Commandline flag to enable BRO_FAKE_DNS -------------------------+---------------------- Reporter: brosenberg | Owner: vern Type: Patch | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: -------------------------+---------------------- Comment (by robin): On Sun, Dec 05, 2010 at 19:31 -0000, you wrote: > main environment and this never bothers me. If the only real benefit of > adding a flag is it showing up in the command-line help, why not just add > BRO_DNS_FAKE there? It already lists BROPATH and BRO_PREFIXES. Ok, that works for me. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Tue Dec 7 07:31:37 2010 From: seth at icir.org (Seth Hall) Date: Tue, 7 Dec 2010 10:31:37 -0500 Subject: [Bro-Dev] test message Message-ID: <34F27DE7-E884-440F-B9D3-BFBA5C2F264D@icir.org> Testing message archiving, please ignore. .Seth From bro at tracker.icir.org Tue Dec 7 12:15:31 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 07 Dec 2010 20:15:31 -0000 Subject: [Bro-Dev] #328: Patch to add programmatic pattern construction at init time. (was: Programmatic pattern construction at init time.) In-Reply-To: <043.f9c865007fdbbe40274ceab20806264f@tracker.icir.org> References: <043.f9c865007fdbbe40274ceab20806264f@tracker.icir.org> Message-ID: <058.71102b1a850dcf6ffe4b841dfe2687ef@tracker.icir.org> #328: Patch to add programmatic pattern construction at init time. ---------------------+---------------------- Reporter: seth | Owner: robin Type: Patch | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ---------------------+---------------------- Changes (by seth): * owner: => robin * status: new => assigned * type: Feature Request => Patch Old description: > bro.bif contains functions for dynamically creating patterns but they are > disabled if reading live traffic because the DFA and NFA data structures > can't be fully cleaned up. It would be useful to be able to call these > functions in bro_init handlers regardless of if live traffic is being > analyzed which should be acceptable for memory concerns but allows > patterns to be built from configuration variables. > > Here are the functions: > > {{{ > function merge_pattern%(p1: pattern, p2: pattern%): pattern > function string_to_pattern%(s: string, convert: bool%): pattern > }}} New description: bro.bif contains functions for dynamically creating patterns but they are disabled if reading live traffic because the DFA and NFA data structures can't be fully cleaned up. It would be useful to be able to call these functions in bro_init handlers regardless of if live traffic is being analyzed which should be acceptable for memory concerns but allows patterns to be built from configuration variables. Here are the functions: {{{ function merge_pattern%(p1: pattern, p2: pattern%): pattern function string_to_pattern%(s: string, convert: bool%): pattern }}} ** Updated ** Now it's possible to use these functions in bro_init handlers. Previously it was possible to use them anywhere if not running on live traffic, but that capability has been removed to avoid potential confusion when writing scripts. This code is definitely going to need reviewed, I made changes to the Event class that I don't feel completely confident about. -- Comment: Patch attached -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Tue Dec 7 14:33:12 2010 From: gregor at icir.org (Gregor Maier) Date: Tue, 07 Dec 2010 14:33:12 -0800 Subject: [Bro-Dev] Null / None / undef in Bro policy scripts Message-ID: <4CFEB628.2030803@icir.org> Hi, is there a NULL (or nil, None, undef) for the policy layer for record types? I would like to have an event, with a record type in its parameter list. But the record can be NULL.... How can I do this? Thx, Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From bro at tracker.icir.org Tue Dec 7 15:25:28 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 07 Dec 2010 23:25:28 -0000 Subject: [Bro-Dev] #336: Global config variable to disable packet filtering (i.e., accept all packets) Message-ID: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> #336: Global config variable to disable packet filtering (i.e., accept all packets) -----------------------------+----------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Component: Bro | Version: Keywords: | -----------------------------+----------------- Hi, I think might be nice to have a policy layer variable (e.g., pcap_accept_all) that can be set to True to disable filtering and accept all packets. I know that there is the unrestricted_filter global and the -f command line option, but I still think that a global variable to enable/disable filtering would still be very handy. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Tue Dec 7 17:09:17 2010 From: seth at icir.org (Seth Hall) Date: Tue, 7 Dec 2010 20:09:17 -0500 Subject: [Bro-Dev] Null / None / undef in Bro policy scripts In-Reply-To: <4CFEB628.2030803@icir.org> References: <4CFEB628.2030803@icir.org> Message-ID: <796280FE-7D16-45AD-BDFE-8E6955A6C7AD@icir.org> On Dec 7, 2010, at 5:33 PM, Gregor Maier wrote: > is there a NULL (or nil, None, undef) for the policy layer for record > types? I would like to have an event, with a record type in its > parameter list. But the record can be NULL.... > How can I do this? There isn't a way to specify a missing record, but you could do a record with all optional and default fields. Or maybe even something like this... type whatever: record { empty: bool &default=T; ....rest of data... }; You could test for the empty key being set to T and when you fill out the record in your analyzer you would make sure to set the empty value to F. From gregor at icir.org Tue Dec 7 18:31:19 2010 From: gregor at icir.org (Gregor Maier) Date: Tue, 07 Dec 2010 18:31:19 -0800 Subject: [Bro-Dev] Policy script enums (and event engine interaction) Message-ID: <4CFEEDF7.4080207@icir.org> Hi, another question: it seems that I can't specify the value that my enums should have, i.e., I can't do a enum { foo = 2, bar = 3 } So, it seems that I'll have to use global const to simulate the behavior. But the nice feature of enums is that if I print them with %s, I get their name instead of their value. Is there any way to simulate this (without using a table to map values->names?) Looking a bit more at the event engine, it seems that there's no case in which an enum is passed from the event engine to the policy layer as part of an event. It also seems that there's no way to coerce an integer into an enum, right (neither policy layer nor event engine). I'm wondering whether there was a reason for not implementing that (e.g., would break something, enums should be policy layer only). If not, I guess my question is whether it would make sense to add it (at some stage) as I think it would make handling of binary protocols easier. (There seems to be one case (transport_proto) were there's interaction in bro.bif, but it requires a lot of glue functions). cu gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Tue Dec 7 22:26:29 2010 From: gregor at icir.org (Gregor Maier) Date: Tue, 07 Dec 2010 22:26:29 -0800 Subject: [Bro-Dev] Policy script enums (and event engine interaction) In-Reply-To: <4CFEEDF7.4080207@icir.org> References: <4CFEEDF7.4080207@icir.org> Message-ID: <4CFF2515.8090806@icir.org> Hmm, I partially found the solution. Adding an enum in const.bif seem to do the trick for using one enum definition in both the policy layer and the event engine. Also found how to coerce integer into EnumVal in the event engine (but still don't know how to achieve this in the policy layer). Enums with pre-assigned values still don't work though :-( cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Tue Dec 7 23:54:53 2010 From: gregor at icir.org (Gregor Maier) Date: Tue, 07 Dec 2010 23:54:53 -0800 Subject: [Bro-Dev] Policy script enums (and event engine interaction) In-Reply-To: <4CFF2515.8090806@icir.org> References: <4CFEEDF7.4080207@icir.org> <4CFF2515.8090806@icir.org> Message-ID: <4CFF39CD.9020306@icir.org> On 12/7/10 22:26 , Gregor Maier wrote: > Enums with pre-assigned values still don't work though :-( I think I'll actually add this to the bif-language and the bro-policy-lanague, so that one can indeed specify enum { ASDF=4, FOO=7, BAR=0, } in .bro and .bif files. .... unless somebody thinks this is a bad idea. cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From bro at tracker.icir.org Wed Dec 8 02:51:03 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 08 Dec 2010 10:51:03 -0000 Subject: [Bro-Dev] #337: BroCtl's top has trouble with large values Message-ID: <044.194bdf0be7a34bfafe07f70c1ae7871f@tracker.icir.org> #337: BroCtl's top has trouble with large values ------------------------+------------------------ Reporter: robin | Owner: robin Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master ------------------------+------------------------ From Craig: File "/home/users/bro/bro-151/lib/broctl/BroControl/control.py", line 588, in getTopOutput d["vsize"] = int(p[1]) ValueError: invalid literal for int(): 2.17684e+09 Here's a possible fix for this: d["vsize"] = int(float(p[1])) Craig fun 2 % !! ipython Python 2.6.6 (r266:84292, Sep 29 2010, 08:18:31) Type "copyright", "credits" or "license" for more information. IPython 0.10.1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object'. ?object also works, ?? prints more. In [1]: p = [ 'foo', '2.17684e+09' ] In [2]: print type(p[1]) In [3]: print p[1] 2.17684e+09 In [4]: d = {} In [5]: d["vsize"] = int(float(p[1])) In [6]: print d["vsize"] 2176840000 In [7]: print type(d["vsize"]) -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Wed Dec 8 05:04:11 2010 From: robin at icir.org (Robin Sommer) Date: Wed, 8 Dec 2010 05:04:11 -0800 Subject: [Bro-Dev] Masters updated In-Reply-To: <4558123.4.1291659980448.JavaMail.jsiwek@tangent.ncsa.illinois.edu> References: <20101204013535.GE70534@icir.org> <4558123.4.1291659980448.JavaMail.jsiwek@tangent.ncsa.illinois.edu> Message-ID: <20101208130411.GR93196@icir.org> On Mon, Dec 06, 2010 at 12:26 -0600, you wrote: > Can you re-check that the broccoli/master repository has an updated > submodule for broccoli-python; it didn't look like it was using the > master version of broccoli-python. Updated. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Wed Dec 8 05:10:15 2010 From: robin at icir.org (Robin Sommer) Date: Wed, 8 Dec 2010 05:10:15 -0800 Subject: [Bro-Dev] New commit notifications Message-ID: <20101208131015.GS93196@icir.org> As you guys might have seen, I've now activated my new commit notification script, it has been working fine for me so far. Let's see how it goes. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Wed Dec 8 12:46:26 2010 From: seth at icir.org (Seth Hall) Date: Wed, 8 Dec 2010 15:46:26 -0500 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: <4CFD6448.2020306@icir.org> References: <4CFD6448.2020306@icir.org> Message-ID: On Dec 6, 2010, at 5:31 PM, Gregor Maier wrote: > I've played around with adding support to count packets and (IP-)bytes > per connection and direction. Awesome! > b) implement it as a child-analyzer to the transport analyzers and add > this child-analyzer to the initial analyzer tree. One can configure > whether to enable/disable this size-analyzer via policy scripts. I've > done that too. > > Advantage: can be switched on and off. No overhead when off. I like this (option b). > * Slightly slower due to calls to an additional analyzer (and > virtualized method lookups) I think the fact that it's slightly slower is mitigated by it being optional. > * Other ideas to make the counters available to the policy script. Couldn't it just be in the connection record? Maybe you could add the extra data to the endpoint record? Making it look something like this... type endpoint: record { size: count; state: count; counted_size: count &optional; counted_packets: count &optional; }; That should be accessible from the core (since size and state are currently being filled in from the core) and it makes it available at arbitrary times since it's in the connection record. > * Do you guys think it's important to have access to the counters at > any time, or is it sufficient to just get them when the connection > ends? Yes, I would really like to access these counters at any time. I would even say it's a requirement for this feature (for me at least). .Seth From bro at tracker.icir.org Wed Dec 8 12:47:55 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 08 Dec 2010 20:47:55 -0000 Subject: [Bro-Dev] #336: Global config variable to disable packet filtering (i.e., accept all packets) In-Reply-To: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> References: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> Message-ID: <060.ad5c606aa2f0edfb7d87e4402f30399a@tracker.icir.org> #336: Global config variable to disable packet filtering (i.e., accept all packets) ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Changes (by seth): * milestone: => Bro1.6 Comment: I've been wanting this feature too. Let's tentatively set it for the 1.6 release as I believe it should be fairly easy to implement. We can always bump it back if it's more work than expected. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Dec 8 12:56:12 2010 From: seth at icir.org (Seth Hall) Date: Wed, 8 Dec 2010 15:56:12 -0500 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: References: <4CFD6448.2020306@icir.org> Message-ID: <1A5B5CFE-099F-4D2E-A47E-502864D4AAEF@icir.org> On Dec 8, 2010, at 3:46 PM, Seth Hall wrote: > That should be accessible from the core (since size and state are currently being filled in from the core) and it makes it available at arbitrary times since it's in the connection record. Oh, and it would be awesome if it was possible to use these values in "when" statements. Currently, these endpoint record values can't be used in when statements, but it would make for really simple code in some cases. For example, in my ssh-ext script I have to set timers to regularly check back into connections to see if a byte threshold has been crossed like this: event ssh_watcher(c: connection) { local id = c$id; # don't go any further if this connection is gone already! if ( !connection_exists(id) ) { delete active_ssh_conns[id]; return; } # run the code to check the size of the connection event check_ssh_connection(c, F); if ( c$id in active_ssh_conns ) schedule +15secs { ssh_watcher(c) }; } event protocol_confirmation(c: connection, atype: count, aid: count) { if ( atype == ANALYZER_SSH ) { local tmp: ssh_ext_session_info; active_ssh_conns[c$id]=tmp; schedule +15secs { ssh_watcher(c) }; } } It would much easier to do... event protocol_confirmation(c: connection, atype: count, aid: count) { if ( atype == ANALYZER_SSH ) { when ( c$resp$counted_size > 5120 ) { # This is a heuristically derived "successful" SSH connection. } } } Of course, I don't know why this doesn't currently work or if it's something that could even be reasonably implemented but it would be *really* nice. :) If it did work, it would be one less "gotcha" in the scripting language. .Seth From vern at icir.org Wed Dec 8 13:37:03 2010 From: vern at icir.org (Vern Paxson) Date: Wed, 08 Dec 2010 13:37:03 -0800 Subject: [Bro-Dev] Policy script enums (and event engine interaction) In-Reply-To: <4CFF39CD.9020306@icir.org> (Tue, 07 Dec 2010 23:54:53 PST). Message-ID: <20101208213703.9515136A016@taffy.ICSI.Berkeley.EDU> > .... unless somebody thinks this is a bad idea. For completeness, Gregor & I just discussed this and the plan is to go ahead with adding this feature at some point. (Note, the support won't include enum<->int conversion.) Vern From gregor at icir.org Wed Dec 8 13:44:14 2010 From: gregor at icir.org (Gregor Maier) Date: Wed, 08 Dec 2010 13:44:14 -0800 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: References: <4CFD6448.2020306@icir.org> Message-ID: <4CFFFC2E.4090306@icir.org> > Couldn't it just be in the connection record? Maybe you could add the extra data to the endpoint record? Making it look something like this... > > type endpoint: record { > size: count; > state: count; > counted_size: count &optional; > counted_packets: count &optional; > }; If I do this, then I basically have variant (a). (with the addition that it's slower than (a)). The advantage of (b) is that it has no memory overhead if the counters are not used, but if I add them to the connection record, this advantage is gone. The only way for (b) would be to have a global table, indexed by connid, that yields the counters. But I think this could be painful, because I would have to update this table from the event engine for each received packet. cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From seth at icir.org Wed Dec 8 13:45:28 2010 From: seth at icir.org (Seth Hall) Date: Wed, 8 Dec 2010 16:45:28 -0500 Subject: [Bro-Dev] List publicly announced Message-ID: Be aware that this list is going to be getting a lot of new subscribers and is *definitely* no longer a private list. .Seth From seth at icir.org Wed Dec 8 13:49:23 2010 From: seth at icir.org (Seth Hall) Date: Wed, 8 Dec 2010 16:49:23 -0500 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: <4CFFFC2E.4090306@icir.org> References: <4CFD6448.2020306@icir.org> <4CFFFC2E.4090306@icir.org> Message-ID: On Dec 8, 2010, at 4:44 PM, Gregor Maier wrote: > If I do this, then I basically have variant (a). (with the addition that > it's slower than (a)). Heh, the worst of both worlds. :) > The advantage of (b) is that it has no memory overhead if the counters > are not used, but if I add them to the connection record, this advantage > is gone. Bro allocates memory for unused optional elements? Oh, I guess I hadn't thought about this enough anyway. My thought was that you'd be able to give some signal back to the core with a BiF call or something which would then enable the counting for the connection but that doesn't work because by the time you get your first event (like connection_established) you've already missed several packets. Hm. > The only way for (b) would be to have a global table, indexed by connid, > that yields the counters. But I think this could be painful, because I > would have to update this table from the event engine for each received > packet. That seems sort of hacky too. .Seth From gregor at icir.org Wed Dec 8 14:34:01 2010 From: gregor at icir.org (Gregor Maier) Date: Wed, 08 Dec 2010 14:34:01 -0800 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: <1A5B5CFE-099F-4D2E-A47E-502864D4AAEF@icir.org> References: <4CFD6448.2020306@icir.org> <1A5B5CFE-099F-4D2E-A47E-502864D4AAEF@icir.org> Message-ID: <4D0007D9.8010606@icir.org> On 12/8/10 12:56 , Seth Hall wrote: > > On Dec 8, 2010, at 3:46 PM, Seth Hall wrote: > >> That should be accessible from the core (since size and state are currently being filled in from the core) and it makes it available at arbitrary times since it's in the connection record. > > Oh, and it would be awesome if it was possible to use these values in "when" statements. Currently, these endpoint record values can't be used in when statements, but it would make for really simple code in some cases. For example, in my ssh-ext script I have to set timers to regularly check back into connections to see if a byte threshold has been crossed like this: > > [...] > Hmm. Actually this currently only works for connection records due to the way the connection record is allocated in the event engine. Each connection instance (C++ class instance) has a conn_val Val* that get's allocated once is then updated and passed to each event. If you save this record in the policy layer, it will get updated "in the background" by the event engine. However, it will only be updated if the event engine calls BuildConnVal(), which is in general only done if a event for this connection is generated. So, if the protocol_confirmation() event is the only event for this connection that's generated, then the connection value would not be updated. Note, that somebody could also generate a connection event and just pass a newly allocated connection record to it. I this case, the above approach would also break (e.g., if you use this connection record further down the line, it would never changes its values). > It would much easier to do... > [...] > > Of course, I don't know why this doesn't currently work or if it's something that could even be reasonably implemented but it would be *really* nice. :) If it did work, it would be one less "gotcha" in the scripting language. > (partially guessing, since I haven't dug too deep into how "when" statements work). I think the problem is that the connection record (and thus the conn_id) is only valid in the scope of the function. The when statement clones the local stack frame. So for everything that's in the local scope, you only get a static copy. I think this would work, if you keep a global table with "conn_ids_to_watch", add the conn_id to this table and then use the table for the "when" statement. Maybe this code works: table conns_to_watch[conn_id] of connection; event protocol_confirmation(c: connection, atype: count, aid: count) { if ( atype == ANALYZER_SSH ) { conns_to_watch[c$id] = c when ( conns_to_watch[c$id]$resp$counted_size > 5120 ) { # .... } } } You probably don't need to save to full connection record in the table. I'm not sure what happens when the connection get's removed before then when triggers..... cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From katrina at csail.mit.edu Wed Dec 8 14:36:46 2010 From: katrina at csail.mit.edu (Katrina LaCurts) Date: Wed, 8 Dec 2010 17:36:46 -0500 Subject: [Bro-Dev] Weird behavior in Katrina's code. In-Reply-To: <4CF80D75.1090007@hiit.fi> References: <20101202032325.DCCE036A4EC@taffy.ICSI.Berkeley.EDU> <4CF80D75.1090007@hiit.fi> Message-ID: <6F542B53-87AE-42C1-9178-FF2165355B43@csail.mit.edu> This was tricky to track down. Basically, I had screwed up the connection state when doing a check for un-acked data after a reset. It's now correct, and conn.log should have the expected output. On Dec 2, 2010, at 4:19 PM, Boris Nechaev wrote: > On 12/01/2010 07:23 PM, Vern Paxson wrote: >>> 1266506673.653157 ip1 port1 ip2 port2 60 0 888448966 0 S >>> 1266506673.653530 ip2 port2 ip1 port1 40 0 1921250427 888448967 RA >>> 1266506676.651348 ip1 port1 ip2 port2 60 0 888448966 0 S >>> 1266506676.651708 ip2 port2 ip1 port1 40 0 570721244 888448967 RA >>> 1266506682.651195 ip1 port1 ip2 port2 60 0 888448966 0 S >>> 1266506682.651622 ip2 port2 ip1 port1 40 0 1779909088 888448967 RA >>> 1266506694.651297 ip1 port1 ip2 port2 60 0 888448966 0 S >>> 1266506694.651669 ip2 port2 ip1 port1 40 0 2051408459 888448967 RA >>> 1266506718.651252 ip1 port1 ip2 port2 60 0 888448966 0 S >>> 1266506718.651676 ip2 port2 ip1 port1 60 0 3793171500 888448967 SA >>> >> This is a pattern that Bro will interpret differently depending on the >> setting of various timeouts defined in bro.init. >> ... >> Could that be what's going on? >> > > I've checked this, all the timeouts in bro.init are exactly the same, so > this not what is going on. > > -- > Best regards, Boris Nechaev. > > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From bro at tracker.icir.org Wed Dec 8 15:25:01 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Wed, 08 Dec 2010 23:25:01 -0000 Subject: [Bro-Dev] #338: Start time semantics in ConnCompressor when seeing multiple SYNs Message-ID: <045.d47f21f1434fb366c8024e70fefc463d@tracker.icir.org> #338: Start time semantics in ConnCompressor when seeing multiple SYNs ---------------------+-------------------- Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Keywords: | ---------------------+-------------------- Bro used to set the start time of a TCP connection to the timestamp of *last* SYN-packet. This was fixed/changed with #77 (and svn r6724). However, Robin mentioned that the semantics of the ConnectionCompressor have to be changed as well, but it seems this never happened. I.e., the ConnCompressor still sets the start time to the last SYN received. The attached patch should fix this. **However, somebody who knows more about the ConnCompressor should verify that this patch is indeed ok** -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed Dec 8 16:21:04 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 00:21:04 -0000 Subject: [Bro-Dev] #339: Tweaking HTTP Analyzer to be able to parse some weird HTTP connections Message-ID: <045.99e25b1ba25b7cba448edd24b88a0f44@tracker.icir.org> #339: Tweaking HTTP Analyzer to be able to parse some weird HTTP connections --------------------+------------------------ Reporter: gregor | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Keywords: | --------------------+------------------------ {{{ #!rst * Fixing a couple of minor issues in the HTTP analyzer, that made the analyzer raise a ProtocolViolation() on strange but seemingly legal HTTP traffic. Well, the traffic might not necessarily be adhering the RFC, but the server has understood it. * Also stops parsing when the message is interrupted to prevent further parsing and ProtocolViolation() generation. * skip_http_entity_data: check return value of FindAnalyzer, since it can be NULL }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed Dec 8 16:23:03 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 00:23:03 -0000 Subject: [Bro-Dev] #339: Tweaking HTTP Analyzer to be able to parse some weird HTTP connections In-Reply-To: <045.99e25b1ba25b7cba448edd24b88a0f44@tracker.icir.org> References: <045.99e25b1ba25b7cba448edd24b88a0f44@tracker.icir.org> Message-ID: <060.b4c1f240fe6c524b8cfb445ee727eda8@tracker.icir.org> #339: Tweaking HTTP Analyzer to be able to parse some weird HTTP connections ---------------------+------------------------ Reporter: gregor | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by gregor): (note, these changes have been sitting in my work branch for quite a while, but I never got around to file them as a ticket.) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed Dec 8 17:36:49 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 01:36:49 -0000 Subject: [Bro-Dev] #336: Global config variable to disable packet filtering (i.e., accept all packets) In-Reply-To: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> References: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> Message-ID: <060.9941a80d9b6f21b330173b8cfec984ef@tracker.icir.org> #336: Global config variable to disable packet filtering (i.e., accept all packets) ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by gregor): {{{ #!rst That should be fairly easy to achieve. However, I guess the question is: * should this global flag disable all filters for good (sort of like using -f), * or should it just disable the default filter (which is constructed from restrict_filters and capture_filters), but still enable users to use precompile_pcap_filter() and install_pcap_filter() bifs. (See, e.g., load-level.bro, remote-pcap.bro, peer-status.bro, remote.bro) print-filter.bro should probably be adjusted as well. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Wed Dec 8 19:02:31 2010 From: gregor at icir.org (Gregor Maier) Date: Wed, 08 Dec 2010 19:02:31 -0800 Subject: [Bro-Dev] Bro script parser: shift/reduce conflicts. Message-ID: <4D0046C7.8080803@icir.org> Hi, The bro policy script parser, parse.y has 71 shift/reduce conflicts. (Which are resolved by bison by always choosing shift). This results in bison issuing a warning. Presumably these 71 conflicts are all fine and expected. So I would add a %expect 71 clause the parse.y. This will make bison expect 71 shift/reduce conflicts. If the number differs, then bison will exit with an error (and make exit with an error). If the number matches, bison will be silent. Question: are there any reason to not use this? cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From vern at icir.org Wed Dec 8 20:02:05 2010 From: vern at icir.org (Vern Paxson) Date: Wed, 08 Dec 2010 20:02:05 -0800 Subject: [Bro-Dev] Bro script parser: shift/reduce conflicts. In-Reply-To: <4D0046C7.8080803@icir.org> (Wed, 08 Dec 2010 19:02:31 PST). Message-ID: <20101209040205.8760736A016@taffy.ICSI.Berkeley.EDU> > Presumably these 71 conflicts are all fine and expected. So I would add > a %expect 71 clause the parse.y. Will byacc just ignore (or honor) such a clause, or will it complain? Vern From vern at icir.org Wed Dec 8 20:18:16 2010 From: vern at icir.org (Vern Paxson) Date: Wed, 08 Dec 2010 20:18:16 -0800 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: <4CFFFC2E.4090306@icir.org> (Wed, 08 Dec 2010 13:44:14 PST). Message-ID: <20101209041816.8664236A017@taffy.ICSI.Berkeley.EDU> > > counted_size: count &optional; > > counted_packets: count &optional; > > }; > > If I do this, then I basically have variant (a). (with the addition that > it's slower than (a)). I don't understand this. &optional fields will be null pointers if the field isn't present. So it's just 4 bytes each when not in use, rather than a full Val record (plus 4 bytes to point to it). I also don't see why it would be slower, other than the smidgen of cycles it takes to initial the null pointers. > The only way for (b) would be to have a global table, indexed by connid, > that yields the counters. Yuck. But first let's understand the issue above. Vern From vern at icir.org Wed Dec 8 20:21:16 2010 From: vern at icir.org (Vern Paxson) Date: Wed, 08 Dec 2010 20:21:16 -0800 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: <4D0007D9.8010606@icir.org> (Wed, 08 Dec 2010 14:34:01 PST). Message-ID: <20101209042116.245FE36A017@taffy.ICSI.Berkeley.EDU> > However, it will only be updated if the event > engine calls BuildConnVal(), which is in general only done if a event > for this connection is generated. Yep. Though with the optional approach, the event engine could know to build the connection value for every packet if there's a pending "when" that cares (at least, I think it can tell when this is the case - Robin is the definitive viewpoint here). > Note, that somebody could also generate a connection event and just pass I don't see any need to worry about that. No one is supposed to be generating connection events. Connection records are linked with the event engine and thus shouldn't be created separately. > I think the problem is that the connection record (and thus the conn_id) > is only valid in the scope of the function. This is also a question for Robin, as it involves the particulars of just how "when" is implemented uner the hood. Vern From bro at tracker.icir.org Wed Dec 8 20:23:19 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 04:23:19 -0000 Subject: [Bro-Dev] #336: Global config variable to disable packet filtering (i.e., accept all packets) In-Reply-To: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> References: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> Message-ID: <060.fb829c5d2078a58cab90ad9dba386167@tracker.icir.org> #336: Global config variable to disable packet filtering (i.e., accept all packets) ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by vern): Can you guys explain why this is worth adding a feature, given there's existing mechanism to do it already (though somewhat clunky)? I worry about creeping featurism. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed Dec 8 20:32:24 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 04:32:24 -0000 Subject: [Bro-Dev] #336: Global config variable to disable packet filtering (i.e., accept all packets) In-Reply-To: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> References: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> Message-ID: <060.2ed379fd8a186a306fd317ac46cdfd85@tracker.icir.org> #336: Global config variable to disable packet filtering (i.e., accept all packets) ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by seth): > Can you guys explain why this is worth adding a feature, given there's > existing mechanism to do it already (though somewhat clunky)? What feature are you referring to? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed Dec 8 20:37:05 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 04:37:05 -0000 Subject: [Bro-Dev] #336: Global config variable to disable packet filtering (i.e., accept all packets) In-Reply-To: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> References: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> Message-ID: <060.db8498d956f035426c02f34794741e66@tracker.icir.org> #336: Global config variable to disable packet filtering (i.e., accept all packets) ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by vern): The one that Seth refers to above as "I've been wanting this feature too" :-). I.e., ''Global config variable to disable packet filtering'' -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed Dec 8 20:47:03 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 04:47:03 -0000 Subject: [Bro-Dev] #336: Global config variable to disable packet filtering (i.e., accept all packets) In-Reply-To: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> References: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> Message-ID: <060.23466c309decf0d65bbbf623f6fca0d9@tracker.icir.org> #336: Global config variable to disable packet filtering (i.e., accept all packets) ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by seth): Oops, I meant what existing mechanism? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Wed Dec 8 21:02:30 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 05:02:30 -0000 Subject: [Bro-Dev] #336: Global config variable to disable packet filtering (i.e., accept all packets) In-Reply-To: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> References: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> Message-ID: <060.71c913c62e373a68087b68c66b558bd1@tracker.icir.org> #336: Global config variable to disable packet filtering (i.e., accept all packets) ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by seth): Nevermind, after chatting with Vern I'm convinced this feature isn't worth it. Maybe we should make some default way to capture all traffic with pcap.bro though? Vern pointed out the filter "ip or not ip" captures everything and it even works when mpls or vlan tags are present in the traffic. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Wed Dec 8 22:30:24 2010 From: gregor at icir.org (Gregor Maier) Date: Wed, 08 Dec 2010 22:30:24 -0800 Subject: [Bro-Dev] Bro script parser: shift/reduce conflicts. In-Reply-To: <20101209040205.8760736A016@taffy.ICSI.Berkeley.EDU> References: <20101209040205.8760736A016@taffy.ICSI.Berkeley.EDU> Message-ID: <4D007780.70406@icir.org> On 12/8/10 20:02 , Vern Paxson wrote: >> Presumably these 71 conflicts are all fine and expected. So I would add >> a %expect 71 clause the parse.y. > > Will byacc just ignore (or honor) such a clause, or will it complain? Depends. They either support and honor it, or I guess they complain if they don't have support for it. Some byacc's support it (e.g., the top hit on Google when searching for byacc), but some don't. That being said, as part of the move-to-CMake discussion the consensus was to only support bison (and remove make-parser.pl). (IIRC) cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Wed Dec 8 22:39:00 2010 From: gregor at icir.org (Gregor Maier) Date: Wed, 08 Dec 2010 22:39:00 -0800 Subject: [Bro-Dev] #336: Global config variable to disable packet filtering (i.e., accept all packets) In-Reply-To: <060.71c913c62e373a68087b68c66b558bd1@tracker.icir.org> References: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> <060.71c913c62e373a68087b68c66b558bd1@tracker.icir.org> Message-ID: <4D007984.8070302@icir.org> (only sending to bro-dev, not the tracker). > Nevermind, after chatting with Vern I'm convinced this feature isn't worth > it. Maybe we should make some default way to capture all traffic with > pcap.bro though? sounds good to me. > Vern pointed out the filter "ip or not ip" captures > everything and it even works when mpls or vlan tags are present in the > traffic. How would one set that? Using -f command line argument? Or using unrestricted_filter? Or is there another way that I'm missing. cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From robin at icir.org Thu Dec 9 01:45:55 2010 From: robin at icir.org (Robin Sommer) Date: Thu, 9 Dec 2010 01:45:55 -0800 Subject: [Bro-Dev] #339: Tweaking HTTP Analyzer to be able to parse some weird HTTP connections In-Reply-To: <045.99e25b1ba25b7cba448edd24b88a0f44@tracker.icir.org> References: <045.99e25b1ba25b7cba448edd24b88a0f44@tracker.icir.org> Message-ID: <20101209094555.GA78587@icir.org> Gregor, let's use git for future patches. You can commit such little tweaks into fastpath and include the description into the commit message. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.icir.org Thu Dec 9 02:38:11 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 10:38:11 -0000 Subject: [Bro-Dev] #336: Global config variable to disable packet filtering (i.e., accept all packets) In-Reply-To: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> References: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> Message-ID: <060.721fee5bf203ffc9f4140a1f19803cca@tracker.icir.org> #336: Global config variable to disable packet filtering (i.e., accept all packets) ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by robin): > Nevermind, after chatting with Vern I'm convinced this feature isn't worth > it. Maybe we should make some default way to capture all traffic with > pcap.bro though? What exactly is "not worth it"? I would have suggested to implement this in pcap.bro anyway, where it would be easy to do and not really require any further changes to anything else. We could a flag "no_default_filter" there and it would just install the "ip or not ip" instead of the standard one if set. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Thu Dec 9 02:45:56 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 10:45:56 -0000 Subject: [Bro-Dev] #338: Start time semantics in ConnCompressor when seeing multiple SYNs In-Reply-To: <045.d47f21f1434fb366c8024e70fefc463d@tracker.icir.org> References: <045.d47f21f1434fb366c8024e70fefc463d@tracker.icir.org> Message-ID: <060.90c7af9f093d0b5d915c49fc15f3205f@tracker.icir.org> #338: Start time semantics in ConnCompressor when seeing multiple SYNs ----------------------+-------------------- Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: ----------------------+-------------------- Comment (by robin): On Wed, Dec 08, 2010 at 23:25 -0000, you wrote: > The attached patch should fix this. I like the patch becaue it removes a special case, and it looks right. Have you tried the test-suite to see what kind differences it introduces? Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Dec 9 02:58:36 2010 From: robin at icir.org (Robin Sommer) Date: Thu, 9 Dec 2010 02:58:36 -0800 Subject: [Bro-Dev] Policy script enums (and event engine interaction) In-Reply-To: <20101208213703.9515136A016@taffy.ICSI.Berkeley.EDU> References: <4CFF39CD.9020306@icir.org> <20101208213703.9515136A016@taffy.ICSI.Berkeley.EDU> Message-ID: <20101209105836.GO78587@icir.org> On Wed, Dec 08, 2010 at 13:37 -0800, you wrote: > (Note, the support won't > include enum<->int conversion.) Can you guys elaborate on this? I would think we dont' want implicit conversions, but some form of explicit cast would be nice to have both ways. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.icir.org Thu Dec 9 06:27:35 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 14:27:35 -0000 Subject: [Bro-Dev] #336: Global config variable to disable packet filtering (i.e., accept all packets) In-Reply-To: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> References: <045.52e9f4c6d1c96231af2e6b5c4bcf4038@tracker.icir.org> Message-ID: <060.4e64784efcc76913a97825c82e174f7f@tracker.icir.org> #336: Global config variable to disable packet filtering (i.e., accept all packets) ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by seth): > What exactly is "not worth it"? I would have suggested to implement > this in pcap.bro anyway, where it would be easy to do and not really > require any further changes to anything else. We could a flag > "no_default_filter" there and it would just install the "ip or not > ip" instead of the standard one if set. I was referring to it not being worthwhile to implement code in the core that actually removes all bpf filters, or doesn't apply them in the first place by not calling the pcap_setfilter function. I think your suggestion is the correct way. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Thu Dec 9 08:27:26 2010 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 9 Dec 2010 10:27:26 -0600 (CST) Subject: [Bro-Dev] Bro script parser: shift/reduce conflicts. In-Reply-To: <4D007780.70406@icir.org> Message-ID: <13394802.38.1291912043137.JavaMail.jsiwek@tangent.ncsa.illinois.edu> > That being said, as part of the move-to-CMake discussion the consensus > was to only support bison (and remove make-parser.pl). (IIRC) Recollection confirmed. - Jon From vern at icir.org Thu Dec 9 09:25:17 2010 From: vern at icir.org (Vern Paxson) Date: Thu, 09 Dec 2010 09:25:17 -0800 Subject: [Bro-Dev] Policy script enums (and event engine interaction) In-Reply-To: <20101209105836.GO78587@icir.org> (Thu, 09 Dec 2010 02:58:36 PST). Message-ID: <20101209172517.CA9E836A034@taffy.ICSI.Berkeley.EDU> > > (Note, the support won't > > include enum<->int conversion.) > > Can you guys elaborate on this? I would think we dont' want implicit > conversions, but some form of explicit cast would be nice to have > both ways. I resist adding language features/flexibility without clear use cases. Casting enum<->int weakens type safety, so there should be a clear need to have it anyway. So far, we don't have that. If we do, then we can revisit it to determine whether general explicit coercion is indeed the best way to support the need. Vern From gregor at icir.org Thu Dec 9 09:39:04 2010 From: gregor at icir.org (Gregor Maier) Date: Thu, 09 Dec 2010 09:39:04 -0800 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: <4CFFFC2E.4090306@icir.org> References: <4CFD6448.2020306@icir.org> <4CFFFC2E.4090306@icir.org> Message-ID: <4D011438.9070902@icir.org> Actually, I was wrong with the &optional fields. I thought they would always need to allocate memory in the background. So, I did some more test and here are the results: * 901 MB ... baseline trunk For these, I added two optional counts to the endpoint record, and set them to NULL: * 903 MB ... baseline w/ optionals. No counting what-so-ever In addition, I now added counting (so far still directly in the TCP_Analyzer and UDP_Analyzer). * 908 MB ... Counters as uint64_t directly in UDP_Analyzer and TCP_Endpoint, but keep optional fields in endpoint record as NULL. * 905 MB ... Counters in their own struct, pointers to this struct in UDP_Analyzer and TCP_Endpoint. But pointers are not allocated. Optionals are NULL as well. * 917 MB ... Counters as structs as above, but this time allocated. Optionals are NULL (this case is rather pointless, as we do count here, but we don't report it) * 970 MB ... Counters as structs, allocated. Report values in endpoint record. I haven't yet tested the "Counting in separate Child Analyzer, reporting via connection record's endpoint record". I have to figure out an elegant way to do this. (See other mail). BTW, a question, it seems that for optionals I can either assign a NULL pointer: rv->Assign(n, NULL) or not assign anything at all. The policy-layer seems to handle both variants in the same way. cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From vern at icir.org Thu Dec 9 09:57:06 2010 From: vern at icir.org (Vern Paxson) Date: Thu, 09 Dec 2010 09:57:06 -0800 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: <4D011438.9070902@icir.org> (Thu, 09 Dec 2010 09:39:04 PST). Message-ID: <20101209175706.C113436A034@taffy.ICSI.Berkeley.EDU> > BTW, a question, it seems that for optionals I can either assign a NULL > pointer: > rv->Assign(n, NULL) > or not assign anything at all. The policy-layer seems to handle both > variants in the same way. I think because they're equivalent. All the above does is assign the nth field to a NULL pointer, and presumably it's initialized to that anyway. Vern From gregor at icir.org Thu Dec 9 10:33:15 2010 From: gregor at icir.org (Gregor Maier) Date: Thu, 09 Dec 2010 10:33:15 -0800 Subject: [Bro-Dev] Updating / Accessing ConnVal from child analyzers Message-ID: <4D0120EB.8020506@icir.org> Hi, has anybody (Robin?) a good idea on how I could update / access a connections's ConnVal from Child analyzers? In my case that would be to facilitate counting and reporting of number of packets (and bytes) seen on the wire for this connection. What Bro currently does: * Conn keeps a ConnVal pointer. A call to BuildConnVal() will update that pointer. * BuildConnVal() calls root_anlyzer->UpdateEndpointVal() to get the current size and state. (root_analyzer is either UDP_Analyzer or TCP_Analyzer). * UpdateEndpointVal() is abstract in TransportAnalyzer, which both UDP_Analyzer and TCP_Analyzer inherit. My counting analyzer (ConnSize) are children (in the DPD sense, not the class hierarchy sense) of UDP_Analyzer and TCP_Analyzer. Two ways come to mind: a) My ConnSize analyzer could just update the ConnVal on every packet. Every analyzer has a pointer to its Conn class. But the pointer is private in Analyzer.h, so derived classes don't have access to it. However, the Conn instance is passed to the ConnSize constructer, so I could just keep a copy of this conn instance. Then I have access to the conn instance. In order to access to the ConnVal, I'd need to add a method to conn to update the ConnVal, or make my analyzer a friend of Conn (since the ConnVal is protected). I would also have to update ConnVal on every packet (instead of only when requested by BuildConnVal). b) I make my ConnSize analyzer inherit from TransportAnalyzer and implement UpdateEndpointVal. TCP_Analyzer::UpdateEndpointVal and UDP_Analyzer::UpdateEndpointVal could then iterate through their children, check whether they are derived from TransportAnalyzer (I think I can do this check with a dynamic_cast, but I don't know how standard this is). If the child is a TransportAnalyzer, the parent can call the child's UpdateEndpointVal. I guess one consideration here would be whether we see other use-cases in which child analyzers update the connection record (actually it's the endpoint record) or whether that would only ever be used for my counting. I think a) is really ugly. What are your thoughts on b)? Does anybody have better/other ideas? cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From bro at tracker.icir.org Thu Dec 9 11:43:46 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 19:43:46 -0000 Subject: [Bro-Dev] #14: Fix "make install" In-Reply-To: <044.21d03859c3a52e6c97fbd49239e7aea2@tracker.icir.org> References: <044.21d03859c3a52e6c97fbd49239e7aea2@tracker.icir.org> Message-ID: <059.7ab124d6fa6de37d81ca522347081a27@tracker.icir.org> #14: Fix "make install" -----------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: 1.4 Resolution: Rejected | Keywords: -----------------------+-------------------- Changes (by seth): * status: seen => closed * resolution: => Rejected -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Thu Dec 9 11:45:51 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 19:45:51 -0000 Subject: [Bro-Dev] #32: Bug in SteppingStone analyzer causes segmentation faults In-Reply-To: <047.2c2b92cbc7ae028cc70759ce3b8c03dc@tracker.icir.org> References: <047.2c2b92cbc7ae028cc70759ce3b8c03dc@tracker.icir.org> Message-ID: <062.7c9684253eec93a334591e0969f84ed5@tracker.icir.org> #32: Bug in SteppingStone analyzer causes segmentation faults -------------------------------+--------------------------------- Reporter: vallenti | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: branches-robin-work Resolution: Feedback Missing | Keywords: -------------------------------+--------------------------------- Changes (by seth): * status: seen => closed * resolution: => Feedback Missing Comment: I think 2 years is long enough to let this ticket stand in it's current condition. Closing now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Thu Dec 9 12:29:23 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 20:29:23 -0000 Subject: [Bro-Dev] #272: Patch to fix do_split function In-Reply-To: <043.1e378c2cf2d2f272b054d3bd5b2f44d9@tracker.icir.org> References: <043.1e378c2cf2d2f272b054d3bd5b2f44d9@tracker.icir.org> Message-ID: <058.d94b417570214e32eb61bcc794ce5a2e@tracker.icir.org> #272: Patch to fix do_split function ---------------------+---------------------- Reporter: seth | Owner: seth Type: Patch | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: ---------------------+---------------------- Comment (by seth): Replying to [comment:4 robin]: > With this patch, I'm seeing a number of test-suite problem in devel, all in the SMTP analyzer and of the following kind: > > error: '<' not found in argument to RCPT: TO: This problem has been fixed here: http://tracker.icir.org/bro/changeset/a598bdb5556a9f2038a52a473c53a7429d8b2491/bro/ I will re-merge it into devel, I think it's ready for more testing. The patch attached to this ticket is now deprecated, I don't know if I can delete it though. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Thu Dec 9 12:41:02 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Thu, 09 Dec 2010 20:41:02 -0000 Subject: [Bro-Dev] #272: Patch to fix do_split function In-Reply-To: <043.1e378c2cf2d2f272b054d3bd5b2f44d9@tracker.icir.org> References: <043.1e378c2cf2d2f272b054d3bd5b2f44d9@tracker.icir.org> Message-ID: <058.1aa870524e9438f272582eb0e9046c47@tracker.icir.org> #272: Patch to fix do_split function ---------------------+---------------------- Reporter: seth | Owner: seth Type: Patch | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: ---------------------+---------------------- Comment (by seth): Oh, wait. The other changes I made to string.bif functions are missing. That must have happened because I merged master with my branch. I'll add them back to my branch so that they can be tested. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Dec 9 16:59:30 2010 From: gregor at icir.org (Gregor Maier) Date: Thu, 09 Dec 2010 16:59:30 -0800 Subject: [Bro-Dev] Bifcl and CMake Message-ID: <4D017B72.50408@icir.org> Hi, it seems that Bifcl doesn't clean up its output files when it fails. This means that if one types make # bifcl has error, make aborts but bifcl has been created make # make since that bifcl was successful because it finds the # bifcl output. Error messages happen were bifcl-outputfiles # are included in C-files. Can you tell CMake to remove the generated output files (if they exist) when an add_custom_command failed? Thanks, Gregor PS: I'll change bifcl, so that it cleans up after itself. However, my current error is a segfault in bifcl (which I'll try to fix as well). But that makes me think that it would be indeed good if make can clean up when a command fails. -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Thu Dec 9 17:31:57 2010 From: gregor at icir.org (Gregor Maier) Date: Thu, 09 Dec 2010 17:31:57 -0800 Subject: [Bro-Dev] Bifcl and CMake In-Reply-To: <4D017B72.50408@icir.org> References: <4D017B72.50408@icir.org> Message-ID: <4D01830D.3060306@icir.org> On 12/9/10 16:59 , Gregor Maier wrote: > it seems that Bifcl doesn't clean up its output files when it fails. > This means that if one types > make > # bifcl has error, make aborts but bifcl has been created > make > # make since that bifcl was successful because it finds the ^^^^^^ should read "thinks" > # bifcl output. Error messages happen were bifcl-outputfiles > # are included in C-files. -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From jsiwek at ncsa.illinois.edu Thu Dec 9 17:50:42 2010 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 9 Dec 2010 19:50:42 -0600 (CST) Subject: [Bro-Dev] Bifcl and CMake In-Reply-To: <4D017B72.50408@icir.org> Message-ID: <2059468232.6164.1291945842660.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > Can you tell CMake to remove the generated output files (if they > exist) when an add_custom_command failed? I'm not sure there's a special CMake way to do it, but you can try appending " || (rm -f ${bifOutputs} && exit 1)" to the ARGS parameter of the add_custom_command. Then when `bifcl` exists with non-zero exit status, the `rm` would get invoked and then the `exit 1` would cause the CMake build to abort. Want to try it out and put it on the fastpath if it works for you? - Jon From gregor at icir.org Thu Dec 9 18:09:51 2010 From: gregor at icir.org (Gregor Maier) Date: Thu, 09 Dec 2010 18:09:51 -0800 Subject: [Bro-Dev] Bifcl and CMake In-Reply-To: <2059468232.6164.1291945842660.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <2059468232.6164.1291945842660.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <4D018BEF.5000408@icir.org> On 12/9/10 17:50 , Jonathan Siwek wrote: >> Can you tell CMake to remove the generated output files (if they >> exist) when an add_custom_command failed? > > I'm not sure there's a special CMake way to do it, but you can try appending " || (rm -f ${bifOutputs} && exit 1)" to the ARGS parameter of the add_custom_command. > > Then when `bifcl` exists with non-zero exit status, the `rm` would get invoked and then the `exit 1` would cause the CMake build to abort. Cool thanks. That worked. However, I'm just wondering which shell CMake / make use and whether that syntax will always work ... > Want to try it out and put it on the fastpath if it works for you? I don't think it's urgent (the autoconf based build seems to have the same issue). I have it in my current branch with some other tweaks to bifcl. I think it's fine if they all get merged together. cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From jsiwek at ncsa.illinois.edu Thu Dec 9 18:44:26 2010 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 9 Dec 2010 20:44:26 -0600 (CST) Subject: [Bro-Dev] Bifcl and CMake In-Reply-To: <4D018BEF.5000408@icir.org> Message-ID: <382778271.6198.1291949066661.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > However, I'm just wondering which shell CMake / make use and whether > that syntax will always work ... Good point. It's not a truly "portable" solution, but [csh, tsch, sh, bash, zsh, ksh] are all cool w/ it so far. - Jon From gregor at icir.org Thu Dec 9 18:47:12 2010 From: gregor at icir.org (Gregor Maier) Date: Thu, 09 Dec 2010 18:47:12 -0800 Subject: [Bro-Dev] Bifcl and CMake In-Reply-To: <382778271.6198.1291949066661.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <382778271.6198.1291949066661.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <4D0194B0.3030705@icir.org> On 12/9/10 18:44 , Jonathan Siwek wrote: >> However, I'm just wondering which shell CMake / make use and whether >> that syntax will always work ... > > Good point. It's not a truly "portable" solution, but [csh, tsch, sh, bash, zsh, ksh] are all cool w/ it so far. cool. That's definitely fine than. As a pure bash user myself I was more wondering about csh, tcsh, .... thanks gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From robin at icir.org Fri Dec 10 02:32:31 2010 From: robin at icir.org (Robin Sommer) Date: Fri, 10 Dec 2010 02:32:31 -0800 Subject: [Bro-Dev] Unique connection ID for bro <-> logging framework In-Reply-To: <4CFD1D74.4040600@icir.org> References: <4CF9A8F6.1020205@icir.org> <4CFD1D74.4040600@icir.org> Message-ID: <20101210103231.GK15294@icir.org> Two more thoughts on htis: - generally, I see that such an ID could be quite helpful. While with the new logging, filtering for 4-tuples should be quite easy, having a per-conn ID gets us more crisp semantics and generally simplifies things. - however, I think there'd need to be one more piece to that story: the IDs should be unique across Bro runs. Otherwise crunching information from a big log archive woulnd't be much better than it is today. But that would probably mean we'd need to go beyond 64-bit integers, perhaps to a string prefixed with something likely to be unique. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Fri Dec 10 02:51:03 2010 From: robin at icir.org (Robin Sommer) Date: Fri, 10 Dec 2010 02:51:03 -0800 Subject: [Bro-Dev] Updating / Accessing ConnVal from child analyzers In-Reply-To: <4D0120EB.8020506@icir.org> References: <4D0120EB.8020506@icir.org> Message-ID: <20101210105103.GL15294@icir.org> On Thu, Dec 09, 2010 at 10:33 -0800, you wrote: > a) My ConnSize analyzer could just update the ConnVal on every packet. > Every analyzer has a pointer to its Conn class. But the pointer is > private in Analyzer.h, so derived classes don't have access to it. (It's private but there is an accessor method, Conn(), so you would not have to store a copy.) > I would also have to update ConnVal on every packet (instead of only > when requested by BuildConnVal). Yeah, that's the part I don't like here. > I think a) is really ugly. > > What are your thoughts on b)? b) is really ugly as well. :-) (Even more so than (a) IMHO ...) Here's another idea: - move UpdateEndpointVal() from TransportAnalyzer to Analyzer - have BuildConnVal() iterate over the analyzer tree and call *every* analyzer's UpdateEndpointVal. Doing so gives all analyzers the chance to update the value as appropiate, with most often just doing nothing. In principle, there could be conflicts with multiple analyzers updating the same fields, but in practice that shouldn't happen and we can just declare that situation as "undefined". This adds a tiny bit of overhead but my guess is that it would be quite a bit less noticable (if at all) than updating the counters with every packet. What do you think? Robn -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Fri Dec 10 03:01:29 2010 From: robin at icir.org (Robin Sommer) Date: Fri, 10 Dec 2010 03:01:29 -0800 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: <4D011438.9070902@icir.org> References: <4CFD6448.2020306@icir.org> <4CFFFC2E.4090306@icir.org> <4D011438.9070902@icir.org> Message-ID: <20101210110128.GM15294@icir.org> On Thu, Dec 09, 2010 at 09:39 -0800, you wrote: > * 901 MB ... baseline trunk > * 903 MB ... baseline w/ optionals. No counting what-so-ever This looks good enough to me, seems fine adding the fields. > I haven't yet tested the "Counting in separate Child Analyzer, reporting > via connection record's endpoint record". Conceptually this would be my preferred approach. See my other mail for thoughts on how to implement it. But it would be interesting to see memory usage and potential run-time ffects for this case before deciding. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Fri Dec 10 06:48:22 2010 From: seth at icir.org (Seth Hall) Date: Fri, 10 Dec 2010 09:48:22 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] rpc: Add support for enum with explicit enumerator values. (0d37c42) In-Reply-To: <201012100230.oBA2Uw76030942@envoy.icir.org> References: <201012100230.oBA2Uw76030942@envoy.icir.org> Message-ID: On Dec 9, 2010, at 9:30 PM, Gregor Maier wrote: > redef enum foo += { > BAR_E, # value will be 12 > BAR_F = 5, # value will be 5 > BAR_G, # value will be 6 > }; Quick question, what if this situation is encountered but the enums being autoincremented go up to 11? Will it skip and make the next one 13? .Seth From vern at icir.org Fri Dec 10 08:31:33 2010 From: vern at icir.org (Vern Paxson) Date: Fri, 10 Dec 2010 08:31:33 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/bro] rpc: Add support for enum with explicit enumerator values. (0d37c42) In-Reply-To: (Fri, 10 Dec 2010 09:48:22 EST). Message-ID: <20101210163133.DAC9936A414@taffy.ICSI.Berkeley.EDU> > > redef enum foo += { > > BAR_E, # value will be 12 > > BAR_F = 5, # value will be 5 > > BAR_G, # value will be 6 > > }; > > > Quick question, what if this situation is encountered but the enums being autoincremented go up to 11? Will it skip and make the next one 13? I'm having a hard time seeing why one would want autoincrement once an explicit value is used, and it strikes me as buggy. My view is that either *all* enums are defined with = X (and it's an error to define two with the same value of X), or they're all implicit/autoincrement. Vern From gregor at icir.org Fri Dec 10 08:39:14 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 10 Dec 2010 08:39:14 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/bro] rpc: Add support for enum with explicit enumerator values. (0d37c42) In-Reply-To: References: <201012100230.oBA2Uw76030942@envoy.icir.org> Message-ID: <4D0257B2.9080709@icir.org> On 12/10/10 6:48 , Seth Hall wrote: > > On Dec 9, 2010, at 9:30 PM, Gregor Maier wrote: > >> redef enum foo += { >> BAR_E, # value will be 12 >> BAR_F = 5, # value will be 5 >> BAR_G, # value will be 6 >> }; > > > Quick question, what if this situation is encountered but the enums being autoincremented go up to 11? Will it skip and make the next one 13? I think I don't understand the question. BAR_E is autoincremented to 12, because BAR_D was 11. If the autoincrement hits a number that has been defined previously, it will generate a (parse) error and let you know. Example: { A = 10, B = 8, C, # implicit 9 D, # would be implicit 10. ERROR. 10 already exists } A this version would work. { B=8, A=10, C, D } cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Fri Dec 10 08:51:42 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 10 Dec 2010 08:51:42 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/bro] rpc: Add support for enum with explicit enumerator values. (0d37c42) In-Reply-To: <20101210163133.DAC9936A414@taffy.ICSI.Berkeley.EDU> References: <20101210163133.DAC9936A414@taffy.ICSI.Berkeley.EDU> Message-ID: <4D025A9E.8050302@icir.org> On 12/10/10 8:31 , Vern Paxson wrote: >>> redef enum foo += { >>> BAR_E, # value will be 12 >>> BAR_F = 5, # value will be 5 >>> BAR_G, # value will be 6 >>> }; >> >> >> Quick question, what if this situation is encountered but the enums being autoincremented go up to 11? Will it skip and make the next one 13? > > I'm having a hard time seeing why one would want autoincrement once an > explicit value is used, and it strikes me as buggy. My view is that either > *all* enums are defined with = X (and it's an error to define two with the > same value of X), or they're all implicit/autoincrement. I guess the only legitimate reason is to provide a seed value for the following auto-increment values. But I agree, that mixing them freely is a bad idea. E.g., { A = 100, B, C, D, ...... } Should I change it so that you can't mix auto-increment and explicit assignments? cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From vern at icir.org Fri Dec 10 08:55:48 2010 From: vern at icir.org (Vern Paxson) Date: Fri, 10 Dec 2010 08:55:48 -0800 Subject: [Bro-Dev] [Bro-Commits] [git/bro] rpc: Add support for enum with explicit enumerator values. (0d37c42) In-Reply-To: <4D025A9E.8050302@icir.org> (Fri, 10 Dec 2010 08:51:42 PST). Message-ID: <20101210165548.77AD036A414@taffy.ICSI.Berkeley.EDU> > I guess the only legitimate reason is to provide a seed value for the > following auto-increment values. Yeah, though for our purposes that still strikes me as buggy. Seems we'll only want an enum to have a particular value if there's a network protocol that associates names with values, and we want to mirror it. For those, it seems safer to explicitly mirror its values rather than implicitly mirroring them. Perhaps we can later add an exception (a single initial seed is okay), but for now I think we should go with no mixing of auto-increment & explicit. Vern From gregor at icir.org Fri Dec 10 10:58:00 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 10 Dec 2010 10:58:00 -0800 Subject: [Bro-Dev] Updating / Accessing ConnVal from child analyzers In-Reply-To: <20101210105103.GL15294@icir.org> References: <4D0120EB.8020506@icir.org> <20101210105103.GL15294@icir.org> Message-ID: <4D027838.3090109@icir.org> > Here's another idea: > > - move UpdateEndpointVal() from TransportAnalyzer to Analyzer > > - have BuildConnVal() iterate over the analyzer tree and call > *every* analyzer's UpdateEndpointVal. Doing so gives all > analyzers the chance to update the value as appropiate, with > most often just doing nothing. In principle, there could be > conflicts with multiple analyzers updating the same fields, > but in practice that shouldn't happen and we can just declare > that situation as "undefined". > > This adds a tiny bit of overhead but my guess is that it would be > quite a bit less noticable (if at all) than updating the counters > with every packet. I really think that the overhead should be small. After all, you have to call DeliverPacket/DeliverStream on every *packet*, but UpdateEndpointVal() only whenever an event is generated. > What do you think? If we do that, we should probably generalize this, so that Analyzers can update the full ConnVal record and not just the endpoints. Afterall, it might well be the case that analyzers want to store some information that's for the connection as a whole and not just one side of the connection. So instead of of UpdateEndpointVal, we could have an: UpdateConnVal (RecordVal *connval, RecordVal *orig_endp, RecordVal *resp_endp) (note that the endpoint val would also be accessible by extracting them from the connval, but I think this is easier). One disadvantage is now, that analyzers can change the actual ConnVal, including starttime. (OTOH, they can also change the services set). Plus, analyzers should not change the pointers to the endpoints that are stored in the connval. What we could do though is to add an additional RecordVal to ConnVal, that analyzers can overwrite: type connection: record { id: conn_id; orig: endpoint; resp: endpoint; start_time: time; duration: interval; service: string_set; # if empty, service hasn't been determined addl: string; hot: count; # how hot; 0 = don't know or not hot history: string; foo: conn_addl &optional; }; type foo_addl: record { bar: string &optional; } Then we could have an UpdateConnVal (RecordVal *foo_addl, RecordVal *orig_endp, RecordVal *resp_endp) (We could then even move the "history" field into the optional foo_addl, since it's often not set. This might save some memory.) (I actually wanted to name the additional foo field in connection addl, but unfortunately this name is already taken :-( ) What do you think about that? cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Fri Dec 10 11:06:53 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 10 Dec 2010 11:06:53 -0800 Subject: [Bro-Dev] Unique connection ID for bro <-> logging framework In-Reply-To: <20101210103231.GK15294@icir.org> References: <4CF9A8F6.1020205@icir.org> <4CFD1D74.4040600@icir.org> <20101210103231.GK15294@icir.org> Message-ID: <4D027A4D.7050406@icir.org> > - however, I think there'd need to be one more piece to that story: > the IDs should be unique across Bro runs. Otherwise crunching > information from a big log archive woulnd't be much better than it > is today. But that would probably mean we'd need to go beyond > 64-bit integers, perhaps to a string prefixed with something likely > to be unique. We can probably keep a 64 bit counter internally and also add a bro_instance_ID, that's globally unique across Bro runs. For logging, we can then log the 64 bit counter and the instance_ID, or concatenate the two (I would guess that the instance_ID will be handy in other situations too). Doesn't the cluster already have/need something like that? In order to generate such an instance_ID, we could: a) make sure it's truly globally unique, e.g., by using a cryptographically secure, long (128 bit, maybe even 160 or more) random number. Possibly from an entropy pool (can we use OpenSSL for that?) b) the user supplies a "hostID", we can then add time and PID and hash all that together to get the instance ID, e.g., md5(hostID + PID + gettimeofday()) (this should probably be fairly tolerant even if the hostID gets reused across machines). cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Fri Dec 10 13:10:19 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 10 Dec 2010 13:10:19 -0800 Subject: [Bro-Dev] Error handling for policy parser Message-ID: <4D02973B.70200@icir.org> Hi, I was wondering on how errors should be handled when parsing policy script, esp. when parsing types (in my case enums). Between Type.cc and parse.y it seem that three different strategies are used: a) The method / function from Type.cc returns a error indicator (int or such) and parse.y calls error(). b) The method / function from Type.cc calls Error(), which it inherited from BroObj() c) The method / function from Type.cc calls error(). Which is the preferred way of reporting errors and/or are there reason to use or not use a particular variant? (BTW, this is for the enum type stuff. The method in question is only called from parse.y to add element to the enum type. I kinda want to avoid a), because it involves code replication) cu gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Fri Dec 10 13:17:38 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 10 Dec 2010 13:17:38 -0800 Subject: [Bro-Dev] Error handling for policy parser In-Reply-To: <4D02973B.70200@icir.org> References: <4D02973B.70200@icir.org> Message-ID: <4D0298F2.4000207@icir.org> Also, should SetError() be used? (Which sets the type's tag to TYPE_ERROR)?? On 12/10/10 13:10 , Gregor Maier wrote: > Hi, > > I was wondering on how errors should be handled when parsing policy > script, esp. when parsing types (in my case enums). > > Between Type.cc and parse.y it seem that three different strategies are > used: > > a) The method / function from Type.cc returns a error indicator (int or > such) and parse.y calls error(). > b) The method / function from Type.cc calls Error(), which it > inherited from BroObj() > c) The method / function from Type.cc calls error(). > > Which is the preferred way of reporting errors and/or are there reason > to use or not use a particular variant? > > (BTW, this is for the enum type stuff. The method in question is only > called from parse.y to add element to the enum type. I kinda want to > avoid a), because it involves code replication) > > cu > gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Fri Dec 10 14:20:56 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 10 Dec 2010 14:20:56 -0800 Subject: [Bro-Dev] Error handling for policy parser In-Reply-To: <4D02973B.70200@icir.org> References: <4D02973B.70200@icir.org> Message-ID: <4D02A7C8.6080804@icir.org> hmmm, one difference between error() and BroObj::Error() is the location that's reported in the error message. error() uses the current parser location, BroObj::Error() uses the location were the object was created. Normally BroObj::Error() would probably be fine to use, however, for enum we can't. The location where the EnumType object is created is the place of the type foo: enum {...} statement, which is not necessarily were the error occurred, if the enum if later "redef'ed". (I think EnumType is the only *type* that can be redefined). More generally, I think that the "location" of an EnumType is ill-defined if it is redef'ed. So, I can't use b) (Sorry for the many mails, just trying to understand how the policy layer is implemented ....) cu Gregor On 12/10/10 13:10 , Gregor Maier wrote: > Hi, > > I was wondering on how errors should be handled when parsing policy > script, esp. when parsing types (in my case enums). > > Between Type.cc and parse.y it seem that three different strategies are > used: > > a) The method / function from Type.cc returns a error indicator (int or > such) and parse.y calls error(). > b) The method / function from Type.cc calls Error(), which it > inherited from BroObj() > c) The method / function from Type.cc calls error(). > > Which is the preferred way of reporting errors and/or are there reason > to use or not use a particular variant? > > (BTW, this is for the enum type stuff. The method in question is only > called from parse.y to add element to the enum type. I kinda want to > avoid a), because it involves code replication) > > cu > gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Fri Dec 10 14:43:53 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 10 Dec 2010 14:43:53 -0800 Subject: [Bro-Dev] Were to define global const (const.bif VS. bro.init + NetVar.{cc|h} Message-ID: <4D02AD29.5090905@icir.org> Hi, I just noticed that multiple ways are used to define global const that is accessed / redefined from both event engine and policy layer: * in const.bif OR in bro.init and NetVar.{h|cc} * in const.bif OR in a specific .bro script and NetVar.{h|cc} it seems that all three options are used. I was wondering whether it makes sense to unify them wrt the general Bro documentation / cleanup efforts. At least the ones that use const.bif and bro.init could be unified. Then there's only a single place (bro.init or const.bif) were users have to look to find out about global configuration settings. For analyzer specific configuration options (i.e., ones that are defined in a particular policy script) be more logical to keep them in the policy script. (Or they could still be defined in const.bif and then just redef'ed in the policy script, so that users who look at the policy script are aware that the const exists) But, YMMV. cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From vern at icir.org Fri Dec 10 17:59:15 2010 From: vern at icir.org (Vern Paxson) Date: Fri, 10 Dec 2010 17:59:15 -0800 Subject: [Bro-Dev] Error handling for policy parser In-Reply-To: <4D02A7C8.6080804@icir.org> (Fri, 10 Dec 2010 14:20:56 PST). Message-ID: <20101211015915.0FA0036A418@taffy.ICSI.Berkeley.EDU> > hmmm, one difference between error() and BroObj::Error() is the location > that's reported in the error message. error() uses the current parser > location, BroObj::Error() uses the location were the object was created. IIRC, the convention is that you use BroObj::Error() if there's a clear Type object with which to associate the error, and otherwise error(). So it sounds like error() is the right approach for you. Vern From vern at icir.org Fri Dec 10 18:00:39 2010 From: vern at icir.org (Vern Paxson) Date: Fri, 10 Dec 2010 18:00:39 -0800 Subject: [Bro-Dev] Error handling for policy parser In-Reply-To: <4D0298F2.4000207@icir.org> (Fri, 10 Dec 2010 13:17:38 PST). Message-ID: <20101211020039.3990236A418@taffy.ICSI.Berkeley.EDU> > Also, should SetError() be used? (Which sets the type's tag to > TYPE_ERROR)?? Generally, yes (again if you have a Type object to work with). This helps diminish cascading error reports. Vern From vern at icir.org Fri Dec 10 18:12:34 2010 From: vern at icir.org (Vern Paxson) Date: Fri, 10 Dec 2010 18:12:34 -0800 Subject: [Bro-Dev] Were to define global const (const.bif VS. bro.init + NetVar.{cc|h} In-Reply-To: <4D02AD29.5090905@icir.org> (Fri, 10 Dec 2010 14:43:53 PST). Message-ID: <20101211021234.555C636A418@taffy.ICSI.Berkeley.EDU> > * in const.bif OR in bro.init and NetVar.{h|cc} > * in const.bif OR in a specific .bro script and NetVar.{h|cc} > > it seems that all three options are used. I was wondering whether it > makes sense to unify them wrt the general Bro documentation / cleanup > efforts. I believe const.bif is the right way to go. It's cleaner than the others, but we never managed to consistently only use it (much less migrate the older stuff over to it). > For analyzer specific configuration options (i.e., ones that are defined > in a particular policy script) be more logical to keep them in the > policy script. (Or they could still be defined in const.bif and then > just redef'ed in the policy script, so that users who look at the policy > script are aware that the const exists) That latter makes sense to me. Vern From vern at icir.org Fri Dec 10 18:16:55 2010 From: vern at icir.org (Vern Paxson) Date: Fri, 10 Dec 2010 18:16:55 -0800 Subject: [Bro-Dev] Updating / Accessing ConnVal from child analyzers In-Reply-To: <4D027838.3090109@icir.org> (Fri, 10 Dec 2010 10:58:00 PST). Message-ID: <20101211021656.0B5CB36A418@taffy.ICSI.Berkeley.EDU> > > Here's another idea: > > > > - move UpdateEndpointVal() from TransportAnalyzer to Analyzer > > > > - have BuildConnVal() iterate over the analyzer tree and call This sounds good to me too. > One disadvantage is now, that analyzers can change the actual ConnVal, > including starttime. This doesn't strike me as a significant problem in practice. It's not as though we're dealing with adversarial Analyzer's :-). > What we could do though is to add an additional RecordVal to ConnVal, > that analyzers can overwrite: That strikes me as more complexity than is merited if the only concern is isolating what Analyzers can do. (However, maybe this would be reasonable as a generalization of $history.) Vern From gregor at icir.org Fri Dec 10 18:31:45 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 10 Dec 2010 18:31:45 -0800 Subject: [Bro-Dev] Alignment check in RPC code Message-ID: <4D02E291.10907@icir.org> Hi, it seems that the XDR extractor, which is used by the RPC code checks the alignment of a memory buffer before extracting data. uint32 extract_XDR_uint32(const u_char*& buf, int& len) { /* ... check that buf has enough len */ uint32 bits32 = XDR_aligned(buf) ? *(uint32*) buf : ((buf[0] << 24) | (buf[1] << 16) | (buf[2] << 8) | buf[3]); buf += 4; len -= 4; return ntohl(bits32); } inline int XDR_aligned(const u_char* buf) { return (((unsigned long) buf) & 0x3) == 0; } However, on my Mac OS, buf it not 4-byte aligned. If this is the case than the 32-bit value is byte-swapped (which is a bug, which makes the testsuite fail on MAC OS X, which is why I discovered this code). Mac OS X is fine with doing un-aligned reads. More general, with current architectures and OSes, does one still need to check for 4- or 8-byte alignments? If so, should this be done higher up in the Analyzer tree? But my thought is that we can just remove the alignment check here. cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From vern at icir.org Fri Dec 10 19:28:15 2010 From: vern at icir.org (Vern Paxson) Date: Fri, 10 Dec 2010 19:28:15 -0800 Subject: [Bro-Dev] Alignment check in RPC code In-Reply-To: <4D02E291.10907@icir.org> (Fri, 10 Dec 2010 18:31:45 PST). Message-ID: <20101211032815.229A736A418@taffy.ICSI.Berkeley.EDU> > However, on my Mac OS, buf it not 4-byte aligned. If this is the case > than the 32-bit value is byte-swapped (which is a bug, which makes the > testsuite fail on MAC OS X, which is why I discovered this code). Weird, as I run the testsuite pretty much exclusively under MacOS X. Perhaps there's something different in our environments such that it's aligned in mine but not in yours. > Mac OS X is fine with doing un-aligned reads. More general, with current > architectures and OSes, does one still need to check for 4- or 8-byte > alignments? I don't think it's prudent to throw out this check. There's a difference between "predominant" architectures and OSes vs. "current". Your statement is really about the former, and will potentially limit portability (in a nasty way, i.e., a crash on a maybe-not-commonly-executed code path). > If so, should this be done higher up in the Analyzer tree? > But my thought is that we can just remove the alignment check here. I think significantly better is to use: uint32 bits32 = XDR_aligned(buf) ? ntohl(*(uint32*) buf) : ((buf[0] << 24) | (buf[1] << 16) | (buf[2] << 8) | buf[3]); and remove the later call to ntohl(). Vern From robin at icir.org Sat Dec 11 08:59:22 2010 From: robin at icir.org (Robin Sommer) Date: Sat, 11 Dec 2010 08:59:22 -0800 Subject: [Bro-Dev] Alignment check in RPC code In-Reply-To: <20101211032815.229A736A418@taffy.ICSI.Berkeley.EDU> References: <4D02E291.10907@icir.org> <20101211032815.229A736A418@taffy.ICSI.Berkeley.EDU> Message-ID: <20101211165922.GF70457@icir.org> Gregor wrote: > Mac OS X is fine with doing un-aligned reads. This surprises me. Isn't that a CPU property rather than a OS property? Vern wrote: > I don't think it's prudent to throw out this check. Ack. > uint32 bits32 = XDR_aligned(buf) ? ntohl(*(uint32*) buf) : > ((buf[0] << 24) | (buf[1] << 16) | (buf[2] << 8) | buf[3]); I'm wondering how much benefit we get by doing this if-else clause vs. just always doing the safe version that doesn't care about alignment? Is it worth it? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From vern at icir.org Sat Dec 11 09:09:11 2010 From: vern at icir.org (Vern Paxson) Date: Sat, 11 Dec 2010 09:09:11 -0800 Subject: [Bro-Dev] Alignment check in RPC code In-Reply-To: <20101211165922.GF70457@icir.org> (Sat, 11 Dec 2010 08:59:22 PST). Message-ID: <20101211170912.091F736A42C@taffy.ICSI.Berkeley.EDU> > > uint32 bits32 = XDR_aligned(buf) ? ntohl(*(uint32*) buf) : > > ((buf[0] << 24) | (buf[1] << 16) | (buf[2] << 8) | buf[3]); > > I'm wondering how much benefit we get by doing this if-else clause > vs. just always doing the safe version that doesn't care about > alignment? Is it worth it? Good point. Chucking the optimized version and just going with the second line is the way to go. Vern From gregor at icir.org Sat Dec 11 10:50:26 2010 From: gregor at icir.org (Gregor Maier) Date: Sat, 11 Dec 2010 10:50:26 -0800 Subject: [Bro-Dev] Alignment check in RPC code In-Reply-To: <20101211165922.GF70457@icir.org> References: <4D02E291.10907@icir.org> <20101211032815.229A736A418@taffy.ICSI.Berkeley.EDU> <20101211165922.GF70457@icir.org> Message-ID: <4D03C7F2.9040606@icir.org> On 12/11/10 8:59 , Robin Sommer wrote: >> Mac OS X is fine with doing un-aligned reads. > > This surprises me. Isn't that a CPU property rather than a OS property? IIRC that depends. The CPU could generate an alignment fault on unaligned load/store and the OS can then recover. But x86 have always been able to handle unaligned memory access. (I was actually just saying that on my system unaligned reads worked, not that Mac OS X did the recovery). >> uint32 bits32 = XDR_aligned(buf) ? ntohl(*(uint32*) buf) : >> ((buf[0] << 24) | (buf[1] << 16) | (buf[2] << 8) | buf[3]); > > I'm wondering how much benefit we get by doing this if-else clause > vs. just always doing the safe version that doesn't care about > alignment? Is it worth it? I'm also wondering alignment checks should be done higher up in the analyzer tree? E.g., to guarantee to protocol analyzers that the buffer in DeliverStream() and DeliverPacket() calls are 4- or 8-byte aligned (e.g., by copying the data into an aligned buffer). OTOH, if we use binpac++ in the future, than binpacc++ can do alignment checks. BTW, It seems that binpac doesn't do alignment-safe data extraction. Is it worth filing a ticket for that? cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From vern at icir.org Sat Dec 11 10:55:19 2010 From: vern at icir.org (Vern Paxson) Date: Sat, 11 Dec 2010 10:55:19 -0800 Subject: [Bro-Dev] Alignment check in RPC code In-Reply-To: <4D03C7F2.9040606@icir.org> (Sat, 11 Dec 2010 10:50:26 PST). Message-ID: <20101211185519.1338B36A42C@taffy.ICSI.Berkeley.EDU> > I'm also wondering alignment checks should be done higher up in the > analyzer tree? E.g., to guarantee to protocol analyzers that the buffer > in DeliverStream() and DeliverPacket() calls are 4- or 8-byte aligned I don't see how this can work in general, because higher up the layout of data in the buffer isn't known. Plus this is a potentially expensive cost (copying the payload in bulk) that might not be needed at all, depending on what the analyzer does with the payload. Vern From bro at tracker.icir.org Sat Dec 11 12:29:44 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Sat, 11 Dec 2010 20:29:44 -0000 Subject: [Bro-Dev] #340: Cleanup: unify where global consts are defined (access from policy layer and event engine) Message-ID: <045.e26a2c4cd7e7f98be9266f24419513fb@tracker.icir.org> #340: Cleanup: unify where global consts are defined (access from policy layer and event engine) -----------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Bro1.6 Component: Bro | Version: Keywords: | -----------------------------+-------------------- {{{ #!rst Global ``const``'s that are accessible from the policy layer and event engine (e.g., to configure features) are currently defined in different ways: 1. in ``bro.init`` and ``NetVar.{cc|h}`` 2. in a specific .bro policy script and ``NetVar.{cc|h}`` 3. in ``const.bif`` According to our discussion on bro-dev, we should change it to only use ``const.bif``. For case 2. we should add a ``redef`` in the .bro policy scripts, so that users looking at the script see that the const exists (TODO: how to best auto-doucment these). Setting milestone to 1.6 as it seems this can be done together with the general policy script overhaul, but can also be pushed backed. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Sat Dec 11 21:22:32 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Sun, 12 Dec 2010 05:22:32 -0000 Subject: [Bro-Dev] #86: Bro-Bro event communication tricky to test In-Reply-To: <047.506ab4129936e190cc70d752016f502b@tracker.icir.org> References: <047.506ab4129936e190cc70d752016f502b@tracker.icir.org> Message-ID: <062.1238d246ff2e5fc2fa6e56ddc8a86982@tracker.icir.org> #86: Bro-Bro event communication tricky to test -----------------------+------------------------------------------------- Reporter: kreibich | Owner: robin Type: Problem | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: communication, bro, schedule, event -----------------------+------------------------------------------------- Changes (by seth): * owner: kreibich => robin * status: seen => assigned * milestone: => Bro1.6 Comment: This looks like a fairly small bug in the RemoteSerializer code. I think the following patch should fix the problem (in addition to cleaning up the use of the current_peer variable in a few spots). It looks like the if-statement I removed in the first chunk of the patch is where the problem was and the erased comment does not seem to be true after testing. Events will not be sent back to the event originator even without that line due to the code path taken by remotely received events. I pasted the patch here though because it really needs checked out by Robin before going anywhere. {{{ diff --git a/src/RemoteSerializer.cc b/src/RemoteSerializer.cc index a9329cc..51add7c 100644 --- a/src/RemoteSerializer.cc +++ b/src/RemoteSerializer.cc @@ -823,14 +823,9 @@ bool RemoteSerializer::SendCall(SerialInfo* info, PeerID id, if ( ! peer ) return false; - // Do not send events back to originating peer. - if ( current_peer == peer ) - return true; - return SendCall(info, peer, name, vl); } - bool RemoteSerializer::SendCall(SerialInfo* info, Peer* peer, const char* name, val_list* vl) { @@ -1841,10 +1836,9 @@ bool RemoteSerializer::EnterPhaseRunning(Peer* peer) if ( in_sync == peer ) in_sync = 0; - current_peer->phase = Peer::RUNNING; + peer->phase = Peer::RUNNING; Log(LogInfo, "phase: running", peer); - - RaiseEvent(remote_connection_handshake_done, current_peer); + RaiseEvent(remote_connection_handshake_done, peer); if ( remote_trace_sync_interval ) { @@ -2008,12 +2002,11 @@ bool RemoteSerializer::HandshakeDone(Peer* peer) return false; #endif - if ( ! (current_peer->caps & Peer::PID_64BIT) ) - Log(LogInfo, "peer does not support 64bit PIDs; using compatibility mode", current_peer); + if ( ! (peer->caps & Peer::PID_64BIT) ) + Log(LogInfo, "peer does not support 64bit PIDs; using compatibility mode", peer); - if ( (current_peer->caps & Peer::NEW_CACHE_STRATEGY) ) - Log(LogInfo, "peer supports keep-in-cache; using that", - current_peer); + if ( (peer->caps & Peer::NEW_CACHE_STRATEGY) ) + Log(LogInfo, "peer supports keep-in-cache; using that", peer); if ( peer->sync_requested != Peer::NONE ) { @@ -2030,7 +2023,7 @@ bool RemoteSerializer::HandshakeDone(Peer* peer) { Log(LogError, "misconfiguration: authoritative state on both sides", current_peer); - CloseConnection(current_peer); + CloseConnection(peer); return false; } }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Sun Dec 12 20:13:22 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 13 Dec 2010 04:13:22 -0000 Subject: [Bro-Dev] #220: Segmentation fault in BroFile::Rotate In-Reply-To: <053.82e3d763bdde275e75ae061af1ab13d2@tracker.icir.org> References: <053.82e3d763bdde275e75ae061af1ab13d2@tracker.icir.org> Message-ID: <068.7743c0a49d610a58ebc89b8fb0b601b9@tracker.icir.org> #220: Segmentation fault in BroFile::Rotate -----------------------------+-------------------- Reporter: Tyler.Schoenke | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: 1.5.2 Resolution: Solved | Keywords: -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved Comment: These issues are tracked in other tickets, particularly #245. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Sun Dec 12 21:47:06 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 13 Dec 2010 05:47:06 -0000 Subject: [Bro-Dev] #261: Created policy/dpd.ssh.bro In-Reply-To: <049.620b178ae31eec20ca8e08e0cb046232@tracker.icir.org> References: <049.620b178ae31eec20ca8e08e0cb046232@tracker.icir.org> Message-ID: <064.82951324f6541e9770d3168fde217121@tracker.icir.org> #261: Created policy/dpd.ssh.bro -------------------------+-------------------- Reporter: brosenberg | Owner: Type: Patch | Status: closed Priority: Normal | Milestone: Component: Bro | Version: 1.5.2 Resolution: Rejected | Keywords: -------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Rejected Comment: I don't think there is anything else to get from this. I'm going to close it. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Mon Dec 13 00:28:51 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Mon, 13 Dec 2010 08:28:51 -0000 Subject: [Bro-Dev] #341: Verify MACs seen Message-ID: <044.7eaaf2204fb336403834d3f5d42b5e69@tracker.icir.org> #341: Verify MACs seen -----------------------+------------------------ Reporter: robin | Owner: robin Type: Task | Status: new Priority: Normal | Component: BroControl Version: git/master | -----------------------+------------------------ BroControl's "cron" should check the number of MAC addresses seen by the worker nodes. If it changes, that may be an indicator of a problem (like the switch flooding packets to all nodes). -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon Dec 13 00:37:01 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 13 Dec 2010 00:37:01 -0800 Subject: [Bro-Dev] Updating / Accessing ConnVal from child analyzers In-Reply-To: <4D027838.3090109@icir.org> References: <4D0120EB.8020506@icir.org> <20101210105103.GL15294@icir.org> <4D027838.3090109@icir.org> Message-ID: <20101213083701.GD45029@icir.org> On Fri, Dec 10, 2010 at 10:58 -0800, you wrote: > UpdateConnVal (RecordVal *connval, > RecordVal *orig_endp, RecordVal *resp_endp) I can see passing the fill conn_val in, that would give more flexibility at not much cost. But: > (note that the endpoint val would also be accessible by extracting them > from the connval, but I think this is easier). ... not sure about this: passing them in seems redundant, and it's not much work to get the fields, so I'd just do the connval. > foo: conn_addl &optional; > type foo_addl: record { > bar: string &optional; > } I don't see much benefit with this, given that one will still have to adapt foo_addl when an analyzer needs to add something. It also creates another structure that mixes information from different analyzers (which "connection" already does). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon Dec 13 00:50:54 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 13 Dec 2010 00:50:54 -0800 Subject: [Bro-Dev] Unique connection ID for bro <-> logging framework In-Reply-To: <4D027A4D.7050406@icir.org> References: <4CF9A8F6.1020205@icir.org> <4CFD1D74.4040600@icir.org> <20101210103231.GK15294@icir.org> <4D027A4D.7050406@icir.org> Message-ID: <20101213085054.GE45029@icir.org> On Fri, Dec 10, 2010 at 11:06 -0800, you wrote: > We can probably keep a 64 bit counter internally and also add a > bro_instance_ID, that's globally unique across Bro runs. For logging, we > can then log the 64 bit counter and the instance_ID, or concatenate the > two (I would guess that the instance_ID will be handy in other > situations too). Doesn't the cluster already have/need something like that? There's a global peer_description (string) that if set will be used as prefix for IDs in logs; see prefixed_id() in bro.init. The cluster sets that differently for each a node. However, the cluster currently also doesn't give unique IDs across runs, just unique across nodes within a single run). > In order to generate such an instance_ID, we could: My main concern is not wasting too many bytes for these IDs, as I imagine they would be included in pretty much every log entry. On the other hand, I don't think we need to be 100% sure that the IDs are unique as long as the probability of a collision is small. Seems that a single 64-bit int should be able to achieve that already if we hash all information in. > b) the user supplies a "hostID", we can then add time and PID > and hash all that together to get the instance ID, e.g., > md5(hostID + PID + gettimeofday()) I generally like this, and the hostID can be the peer_description. But I think we can hash into 64-bit instead and probably take a simpler hash function as well. And then we can just add the 64-bit counter to that value. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Mon Dec 13 07:58:09 2010 From: seth at icir.org (Seth Hall) Date: Mon, 13 Dec 2010 10:58:09 -0500 Subject: [Bro-Dev] bro and code coverage Message-ID: <977FFBFC-B282-4414-AF97-FA4B36F17774@icir.org> Here's an interesting view of Bro. I compiled it with the gcc flags to enable gcov output. After I ran Bro on a short tracefile with a few analyzers enabled, I processed the gcov output files with lcov and created this view of the executed code in that run. http://www.icir.org/seth/bro-lcov/src/index.html I think the gcov output could be a good starting point for writing tests since it's pretty easy to find parts of code that aren't executed commonly. At least we could use it to develop the test suite to make sure that we are actually testing as many lines of code as possible. Right now there seems to be lots of code that isn't executed (or at least wasn't with my tiny test). It should actually work for verifying that we are testing the script execution code too. Check the Expr.cc file and you can see certain script level functionality (driven at the script level at least) that was never executed, like RecordMatchExpr::ExprDescribe[1] (who would have ever thought to test that?). The gcc/g++ flags to enable gcov output are "-fprofile-arcs -ftest-coverage". Are those something that should be added to the build system? I don't think they need added to the configure script, but maybe to the CMakeLists.txt file? Here are the docs I followed for getting gcov output and processing with lcov: http://gcc.gnu.org/onlinedocs/gcc/Invoking-Gcov.html http://ltp.sourceforge.net/coverage/lcov/readme.php gcov is probably installed everywhere already, but lcov is a non-gcc package and probably need installed. lcov was in macports though. .Seth 1. http://www.icir.org/seth/bro-lcov/src/Expr.cc.gcov.html#3744 From jsiwek at ncsa.illinois.edu Mon Dec 13 08:31:48 2010 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 13 Dec 2010 10:31:48 -0600 (CST) Subject: [Bro-Dev] bro and code coverage In-Reply-To: <977FFBFC-B282-4414-AF97-FA4B36F17774@icir.org> Message-ID: <7731384.100.1292257906405.JavaMail.jsiwek@tangent.ncsa.illinois.edu> > The gcc/g++ flags to enable gcov output are "-fprofile-arcs > -ftest-coverage". Are those something that should be added to the > build system? I don't think they need added to the configure script, > but maybe to the CMakeLists.txt file? If that's really all that's needed I think you could get away with just passing the flags through environment variables in the standard way to the configure script, e.g.: CFLAGS="-fprofile-arcs -ftest-coverage" CXXFLAGS="-fprofile-arcs -ftest-coverage" ./configure Passing the flags only works for the first-time configure of a given build directory. The lcov doc you referenced also mentioned linking to -lgcov. If something like that's necessary, it probably makes sense to add a --enable-code-coverage option to the configure wrapper that triggers CMake to search for the required stuff and, if it's all there, add the necessary compile and link flags, else fail. - Jon From seth at icir.org Mon Dec 13 10:21:31 2010 From: seth at icir.org (Seth Hall) Date: Mon, 13 Dec 2010 13:21:31 -0500 Subject: [Bro-Dev] bro and code coverage In-Reply-To: <7731384.100.1292257906405.JavaMail.jsiwek@tangent.ncsa.illinois.edu> References: <7731384.100.1292257906405.JavaMail.jsiwek@tangent.ncsa.illinois.edu> Message-ID: On Dec 13, 2010, at 11:31 AM, Jonathan Siwek wrote: > CFLAGS="-fprofile-arcs -ftest-coverage" CXXFLAGS="-fprofile-arcs -ftest-coverage" ./configure > > Passing the flags only works for the first-time configure of a given build directory. Ah, that would actually work fine then. It seems like it would mostly be used for the automated tests (after we make sure we have tests that hit the code paths in the first place) to make sure that we continue to hit all code paths possible with our tests and we could use it provide assurance that we continue to hit new code paths in our tests. > The lcov doc you referenced also mentioned linking to -lgcov. If something like that's necessary, it probably makes sense to add a --enable-code-coverage option to the configure wrapper that triggers CMake to search for the required stuff and, if it's all there, add the necessary compile and link flags, else fail. lcov isn't required for gcov, it just generates the nice html output from the raw gcov files but for the automated tests I imagine that we would just use the gcov generated files. It's probably good to avoid stuffing too many options in the configure script anyway. .Seth From gregor at icir.org Mon Dec 13 11:29:41 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 13 Dec 2010 11:29:41 -0800 Subject: [Bro-Dev] Unique connection ID for bro <-> logging framework In-Reply-To: <20101213085054.GE45029@icir.org> References: <4CF9A8F6.1020205@icir.org> <4CFD1D74.4040600@icir.org> <20101210103231.GK15294@icir.org> <4D027A4D.7050406@icir.org> <20101213085054.GE45029@icir.org> Message-ID: <4D067425.70304@icir.org> >> b) the user supplies a "hostID", we can then add time and PID >> and hash all that together to get the instance ID, e.g., >> md5(hostID + PID + gettimeofday()) > > I generally like this, and the hostID can be the peer_description. > But I think we can hash into 64-bit instead and probably take a > simpler hash function as well. And then we can just add the 64-bit > counter to that value. I'd prefer to keep the counter and the runID separate. E.g., by making the runID n-bits and the counter 64-n bits and the OR-ing them together. OTOH, I don't think we need to worry too much about wasting bytes by using a say, 32 bit runID + 64 bit unique-ID. We now use ACSII for logging, if we move to binary (and possibly compression), then this will save *way* more memory than our ID will ever add. (we could also just add the run-id once per log-file or log-group inside a file, but might be too cumbersome) cu gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Mon Dec 13 16:06:26 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 13 Dec 2010 16:06:26 -0800 Subject: [Bro-Dev] Incorrect bounds checking with truncated TCP options Message-ID: <4D06B502.6040306@icir.org> Hi, there is a potential problem in Bro when it receives a packet with truncated TCP options (i.e., the packet isn't long enough to accommodate all options). This can happen: a) in the ConnCompressor: it calls ParseTCPOptions without checking whether the packet is long enough to contain all options. ConnCompressor needs to parse the TCP options to know the window scaling factor. b) in TCP.cc, when caplen < len and len is long enough for the TCP options but caplen is not. ExtractTCPHeader() ensures that the len is long enough to contain the options and that caplen is long enough to contain a struct tcphdr (but doesn't check for options). Presumably this is done to enable parsing of header only traces that truncate options. Nevertheless, the TCP Analyzer correctly checks caplen before ParseTCPOptions(). But there are also situations when options are parsed without checking for caplen: * BuildSYNVal(), which is called on every SYN to get the window scaling options. * BuildOSVal(), which is only called when the OS_version_found event has a handler * TCP TraceRewriter (presumably we can ignore this, as we were going to remove it anyway) So, question is: what's the best way to tackle this? One option is to not parse packets that are truncated. But that's probably not a good idea wrt header traces. The other option is to check for the caplen whenever we parse options. That might be cumbersome as this information needs to be passed to many functions, e.g. in TCP_Analyzer: ProcessFlags -> ProcessSYN -> BuildSYNPacketVal. (In any case, truncated packets mean that we can't learn the window scaling....) cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From bro at tracker.icir.org Mon Dec 13 19:33:52 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 14 Dec 2010 03:33:52 -0000 Subject: [Bro-Dev] #339: Tweaking HTTP Analyzer to be able to parse some weird HTTP connections In-Reply-To: <045.99e25b1ba25b7cba448edd24b88a0f44@tracker.icir.org> References: <045.99e25b1ba25b7cba448edd24b88a0f44@tracker.icir.org> Message-ID: <060.da3dcadf7d1e10fe34b8dcb0425e89b3@tracker.icir.org> #339: Tweaking HTTP Analyzer to be able to parse some weird HTTP connections ---------------------+------------------------ Reporter: gregor | Owner: Type: Patch | Status: closed Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: Solved | Keywords: ---------------------+------------------------ Changes (by gregor): * status: new => closed * resolution: => Solved Comment: Checked in to fastpath commit 763a446182934679f5e56146bee26b2e7cad1f3b -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Mon Dec 13 19:36:16 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 14 Dec 2010 03:36:16 -0000 Subject: [Bro-Dev] #321: Load http.bro script from http-header In-Reply-To: <045.cdd32311ecb41b074829e56fad7216c2@tracker.icir.org> References: <045.cdd32311ecb41b074829e56fad7216c2@tracker.icir.org> Message-ID: <060.90259176028feafbbafc41460f5be10b@tracker.icir.org> #321: Load http.bro script from http-header ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: closed Priority: Low | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved | Keywords: ----------------------+------------------------ Changes (by gregor): * status: new => closed * resolution: => Solved Comment: Checked in to fastpath commit 763a446182934679f5e56146bee26b2e7cad1f3b -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Tue Dec 14 07:42:33 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 14 Dec 2010 15:42:33 -0000 Subject: [Bro-Dev] #342: Add payload to ICMP analyzer Message-ID: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> #342: Add payload to ICMP analyzer --------------------+-------------------- Reporter: seth | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Keywords: | --------------------+-------------------- This is a patch from Julien Sentier on the mailing list that makes ICMP payloads available at the scripting layer. Is there a reason this isn't already available? I would have committed it to fastpath except I don't know if it's not already doing this due to the potential overhead of creating a lot of strings in ICMP floods. At the very least, I suppose it could be optional (which the patch doesn't currently do). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Tue Dec 14 08:02:09 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 14 Dec 2010 16:02:09 -0000 Subject: [Bro-Dev] #342: Add payload to ICMP analyzer In-Reply-To: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> References: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> Message-ID: <058.2737368ff276bd702c71080f311c9d69@tracker.icir.org> #342: Add payload to ICMP analyzer ---------------------+-------------------- Reporter: seth | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: ---------------------+-------------------- Comment (by gregor): {{{ #!rst note that one has to check for min(len, caplen) (due to the possibility of layer 2 padding, so the change at line 91 should be):: BroString* payload = new BroString(data, min(len,caplen), 0); }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Tue Dec 14 08:05:19 2010 From: seth at icir.org (Seth Hall) Date: Tue, 14 Dec 2010 11:05:19 -0500 Subject: [Bro-Dev] connection_established behavior Message-ID: <1FECDEA4-0DE9-4036-93AC-687BF390F768@icir.org> This is related to this ticket: http://tracker.icir.org/bro/ticket/285 I am fairly lost on this, hopefully someone will have insight into the behavior. In the following snippet of code, there's a strange behavior (maybe intentional?) where a packet with SYN/ACK flags set and no other packets related to the connection are seen, the connection_established event will be generated. ====CODE==== if ( peer->state == TCP_ENDPOINT_SYN_SENT ) peer->SetState(TCP_ENDPOINT_ESTABLISHED); else if ( peer->state == TCP_ENDPOINT_INACTIVE ) { // If we were to ignore SYNs and // only instantiate state on SYN // acks, then we'd do: // peer->SetState(TCP_ENDPOINT_ESTABLISHED); // here. Weird("unsolicited_SYN_response"); } endpoint->SetState(TCP_ENDPOINT_ESTABLISHED); ====END CODE==== This only seems to be true when the connection compressor is disabled though. The connection compressor *seems* to prevent this effect. I'll include steps to reproduce the problem here and I'll attach the example tracefile to this email... ===POLICY SCRIPT (test.bro)=== @load conn event connection_established(c: connection) { print fmt("gah! there shouldn't be a connection established (%s)", id_string(c$id)); } ===END POLICY SCRIPT=== Here's the output with a connection established that shouldn't happen: [seth at Blake build (master)]$ ./src/bro -C -f "ip" -r connection_established-problem.trace test use_connection_compressor=F 1285716061.336160 weird: unsolicited_SYN_response gah! there shouldn't be a connection established (128.146.242.61/3072 > 121.254.178.6/http) Here's the (lack of) output with the connection compressor enabled: [seth at Blake build (master)]$ ./src/bro -C -f "ip" -r connection_established-problem.trace test use_connection_compressor=T [seth at Blake build (master)]$ I also disabled the checksum validation because the TCP header has an invalid checksum in the packet for some reason. Sorry for the long email, but any thoughts on the behavior of the sans connection compressor behavior? .Seth -------------- next part -------------- A non-text attachment was scrubbed... Name: connection_established-problem.trace Type: application/octet-stream Size: 100 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20101214/84bf1bbc/attachment.obj From gregor at icir.org Tue Dec 14 08:35:26 2010 From: gregor at icir.org (Gregor Maier) Date: Tue, 14 Dec 2010 08:35:26 -0800 Subject: [Bro-Dev] connection_established behavior In-Reply-To: <1FECDEA4-0DE9-4036-93AC-687BF390F768@icir.org> References: <1FECDEA4-0DE9-4036-93AC-687BF390F768@icir.org> Message-ID: <4D079CCE.7090406@icir.org> The reason is why the behavior differs is, that the "connection" has only one packet and thus the ConnectionCompressor will not instantiate the full Connection. If there were a packet from the other endpoint after the SYN/ACK, then ConnectionCompressor would instantiate a connection and you should see the same behavior when the packets are passed to the TCP Analyzer (i.e., you would the Weird and the connection_established event) cu Gregor On 12/14/10 8:05 , Seth Hall wrote: > This is related to this ticket: http://tracker.icir.org/bro/ticket/285 > > I am fairly lost on this, hopefully someone will have insight into the behavior. In the following snippet of code, there's a strange behavior (maybe intentional?) where a packet with SYN/ACK flags set and no other packets related to the connection are seen, the connection_established event will be generated. > > ====CODE==== > if ( peer->state == TCP_ENDPOINT_SYN_SENT ) > peer->SetState(TCP_ENDPOINT_ESTABLISHED); > else if ( peer->state == TCP_ENDPOINT_INACTIVE ) > { > // If we were to ignore SYNs and > // only instantiate state on SYN > // acks, then we'd do: > // peer->SetState(TCP_ENDPOINT_ESTABLISHED); > // here. > Weird("unsolicited_SYN_response"); > } > > endpoint->SetState(TCP_ENDPOINT_ESTABLISHED); > ====END CODE==== > > This only seems to be true when the connection compressor is disabled though. The connection compressor *seems* to prevent this effect. I'll include steps to reproduce the problem here and I'll attach the example tracefile to this email... > > ===POLICY SCRIPT (test.bro)=== > @load conn > event connection_established(c: connection) > { > print fmt("gah! there shouldn't be a connection established (%s)", id_string(c$id)); > } > ===END POLICY SCRIPT=== > > Here's the output with a connection established that shouldn't happen: > [seth at Blake build (master)]$ ./src/bro -C -f "ip" -r connection_established-problem.trace test use_connection_compressor=F > 1285716061.336160 weird: unsolicited_SYN_response > gah! there shouldn't be a connection established (128.146.242.61/3072 > 121.254.178.6/http) > > Here's the (lack of) output with the connection compressor enabled: > [seth at Blake build (master)]$ ./src/bro -C -f "ip" -r connection_established-problem.trace test use_connection_compressor=T > [seth at Blake build (master)]$ > > I also disabled the checksum validation because the TCP header has an invalid checksum in the packet for some reason. > > Sorry for the long email, but any thoughts on the behavior of the sans connection compressor behavior? > > .Seth > > > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From seth at icir.org Tue Dec 14 09:38:35 2010 From: seth at icir.org (Seth Hall) Date: Tue, 14 Dec 2010 12:38:35 -0500 Subject: [Bro-Dev] connection_established behavior In-Reply-To: <4D079CCE.7090406@icir.org> References: <1FECDEA4-0DE9-4036-93AC-687BF390F768@icir.org> <4D079CCE.7090406@icir.org> Message-ID: <4F01E937-2272-43E0-AAD9-38FB2B3ED5CA@icir.org> On Dec 14, 2010, at 11:35 AM, Gregor Maier wrote: > The reason is why the behavior differs is, that the "connection" has > only one packet and thus the ConnectionCompressor will not instantiate > the full Connection. If there were a packet from the other endpoint > after the SYN/ACK, then ConnectionCompressor would instantiate a > connection and you should see the same behavior when the packets are > passed to the TCP Analyzer (i.e., you would the Weird and the > connection_established event) I guess then I'm interested to hear everyone's take on this due to spoofed syn packets (remote machines at other sites responding to spoofed traffic appearing to come from your address space). If there is nothing listening at the site's address that a spurious syn-ack is directed, then you could see a lot of connection_established events generated if you aren't running the connection compressor due to no rst packet being sent back. I would expect most sites would be running the connection compressor when running on live traffic, but they seem to be having this issue at OSU and as best I can tell, they're running the connection compressor. I'm still trying to replicate the problem exactly as they're seeing it though. Hopefully I have some more information on this soon. The problem with this for the guys at OSU is if the connection_established event ends up being generated and the responder is on a list of known-bad sites then they get an alert about a connection to a botnet or something when there wasn't really a connection at all, just a spurious syn-ack. .Seth From vern at icir.org Tue Dec 14 09:42:43 2010 From: vern at icir.org (Vern Paxson) Date: Tue, 14 Dec 2010 09:42:43 -0800 Subject: [Bro-Dev] connection_established behavior In-Reply-To: <4F01E937-2272-43E0-AAD9-38FB2B3ED5CA@icir.org> (Tue, 14 Dec 2010 12:38:35 EST). Message-ID: <20101214174243.E8CB036A414@taffy.ICSI.Berkeley.EDU> I haven't yet read this thread closely, so this may be off the mark, but a couple of quick comments (before I disappear for a while): - Instantiating on SYN ACK came about due to coping with Bro deployments with split routing, such that they never saw initial SYNs for some connections. - One way to deal with backscatter would be to inspect $history to see that no initial SYN was ever seen. - A more principled way to deal with it would be to fix Bro's basic notion of connection states. When I designed it, I didn't realize that all sorts of non-compliant states could arise (e.g., connections for which only one side is seen). In principle, Bro should stop trying to follow the RFC 793 notion of TCP states, and instead go with an empirical set. $history allows this but in an implicit fashion, rather than with explicit states. The latter would be better, though it's not clear to me that it's really worth the work. - Vern From gregor at icir.org Tue Dec 14 10:55:45 2010 From: gregor at icir.org (Gregor Maier) Date: Tue, 14 Dec 2010 10:55:45 -0800 Subject: [Bro-Dev] TCP SYN timeouts Message-ID: <4D07BDB1.7050402@icir.org> Hi, while working on http://tracker.icir.org/bro/ticket/338 I found some oddities. But see also http://tracker.icir.org/bro/ticket/77 Bro has: * tcp_SYN_timeout * tcp_attempt_delay both are set to 5sec. Both can time out a TCP connection for which only SYNs are seen. However, tcp_attempt_delay is used only when the the connection_attempt event has a script level handler. tcp_attempt_delay is installed on the *first* SYN packet. Thus tcp_attempt_delay always triggers relative to the first SYN packet (and did this even before #77 was applied. I.e., #77 did not change the timer behavior) Note that conn.bro has a connection_attempt handler. tcp_SYN_timeout is used when TCP_Analyzer is instantiated to set the ExpireTimer() (which can also happen on a pure data packet, BTW). The ExpireTimer only does any "expiring" if ( Conn()->LastTime() + tcp_connection_linger < t ) (so, no check for tcp_SYN_timeout). However, ExpireTimer() checks whether there is an connection_attempt handler and if there's one it will not generate connection_timeout() events for connections that only had SYNs and no other packets. (ExpireTimer otherwise only handles connections for which at least one side is already closed) Then there's also the tcp_inactivity_timeout, which is set to 5min and managed in Conn.cc. There's a dated documentation (pre-DPD) in the Wiki concerning these timers: http://www.bro-ids.org/wiki/index.php/Connection_Timers It seems that originally tcp_connection_linger < tcp_SYN_timeout < tcp_attempt_delay which makes sense given the way these timers interact, but since we now have tcp_SYN_timeout == tcp_attempt_delay < tcp_connection_linger the overall timer behavior is weird. Note, the connection compressor currently *only* uses tcp_attempt_delay, and bases it on the time of the last SYN packet. So, the question is: what should we do? cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From bro at tracker.icir.org Tue Dec 14 11:27:03 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Tue, 14 Dec 2010 19:27:03 -0000 Subject: [Bro-Dev] #340: Cleanup: unify where global consts are defined (access from policy layer and event engine) In-Reply-To: <045.e26a2c4cd7e7f98be9266f24419513fb@tracker.icir.org> References: <045.e26a2c4cd7e7f98be9266f24419513fb@tracker.icir.org> Message-ID: <060.ff3e22956db3512ddae1aa8cab0f6926@tracker.icir.org> #340: Cleanup: unify where global consts are defined (access from policy layer and event engine) ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by gregor): {{{ #!rst In addition, the bif-language could be extended to allow declaration of new types, so that types don't have to be added to bro.init and NetVar "by hand". In particular, NetVar.h currently manually adds: * RecordTypes * TableTypes I think it might make sense to automate this. A nice, not too important side-effect: If we automate this, then it would also prevent users from modifying type definitions in bro.init that the event engine expects to be in a well defined form. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Tue Dec 14 13:09:46 2010 From: seth at icir.org (Seth Hall) Date: Tue, 14 Dec 2010 16:09:46 -0500 Subject: [Bro-Dev] connection_established behavior In-Reply-To: <20101214174243.E8CB036A414@taffy.ICSI.Berkeley.EDU> References: <20101214174243.E8CB036A414@taffy.ICSI.Berkeley.EDU> Message-ID: <678CA492-3C34-4F35-911C-D3BB3ADCE13E@icir.org> On Dec 14, 2010, at 12:42 PM, Vern Paxson wrote: > - Instantiating on SYN ACK came about due to coping with Bro > deployments with split routing, such that they never saw > initial SYNs for some connections. Ah, ok. > In principle, > Bro should stop trying to follow the RFC 793 notion of TCP states, > and instead go with an empirical set. Thanks for the backstory on this, it makes this much clearer for me. Perhaps I'll file a ticket for someone to look into doing this for 1.7. The only thing that is still nagging me is that the behavior is different with the connection compressor than it is without it. Does it make sense to do anything to make the connection_compressor and non connection_compressor scenarios end up with the same result? .Seth From seth at icir.org Tue Dec 14 13:42:14 2010 From: seth at icir.org (Seth Hall) Date: Tue, 14 Dec 2010 16:42:14 -0500 Subject: [Bro-Dev] Problem installing Message-ID: <949DDAC7-A967-4A7F-8589-B5D862C37343@icir.org> When I do "sudo make install" I'm getting these errors (from the master branch in git with --enable-cluster)... CMake Warning at aux/broccoli/cmake_install.cmake:33 (message): Skipping install of /bro/etc/broccoli.conf to avoid overwrite of existing config file Call Stack (most recent call first): cmake_install.cmake:35 (INCLUDE) -- Up-to-date: /bro/bin/broccoli-config -- Installing: /bro/lib/libbroccoli.3.0.0.dylib -- Up-to-date: /bro/lib/libbroccoli.3.dylib -- Up-to-date: /bro/lib/libbroccoli.dylib -- Up-to-date: /bro/include/broccoli.h -- Up-to-date: /bro/lib/broctl/broccoli.py -- Installing: /bro/lib/broctl/broccoli.pyc -- Installing: /bro/lib/broctl/_broccoli_intern.so Listing BroControl ... Can't list BroControl -- Up-to-date: /bro/bin/broctl CMake Error at aux/broctl/cmake_install.cmake:47 (FILE): file INSTALL cannot find "/Users/seth/bro/bro.git/build/aux/broctl/BroControl". Call Stack (most recent call first): cmake_install.cmake:36 (INCLUDE) make: *** [install] Error 1 [seth at Blake build (master)]$ .Seth From jsiwek at ncsa.illinois.edu Tue Dec 14 14:34:25 2010 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 14 Dec 2010 16:34:25 -0600 (CST) Subject: [Bro-Dev] Problem installing In-Reply-To: <949DDAC7-A967-4A7F-8589-B5D862C37343@icir.org> Message-ID: <24759267.206.1292366062390.JavaMail.jsiwek@tangent.ncsa.illinois.edu> > file INSTALL cannot find > "/Users/seth/bro/bro.git/build/aux/broctl/BroControl". At "make install" time, this is basically what should happen: 1) cp BroControl ${CURRENT_BINARY_DIR}/BroControl 2) compile Python sources in ${CURRENT_BINARY_DIR}/BroControl 3) install ${CURRENT_BINARY_DIR}/BroControl But it looks like it skipped straight to #3 and I don't understand why. Does that directory really not exist? I wasn't able to reproduce this error; can you give explicit steps/commands w/ CMake version and operating system? - Jon From gregor at icir.org Tue Dec 14 16:03:34 2010 From: gregor at icir.org (Gregor Maier) Date: Tue, 14 Dec 2010 16:03:34 -0800 Subject: [Bro-Dev] Per connection byte and packet counting Message-ID: <4D0805D6.7030206@icir.org> Hi, I've played around with packet and byte counting for connections some more. I think I've got a pretty neat version: * Counting is done in a separate analyzer that can be added to the initial analyzer tree based on a global boolean const. * Reporting is done via the endpoint records in the connection record ( c$orig, c$resp). The endpoint records have to new &optional fields for number of packets and number of bytes. * Updating the counters in the endpoint is done by: + Removing UpdateEndpointVal() from TransportAnalyzer + Adding a UpdateConnVal(conn_val) to Analyzer.cc. When a Conn object needs to update the conn_val (e.g., to generate an event) it calls the root analyzer's UpdateConnVal(). This call is then "forwarded" to every analyzer in the tree. I.e., every analyzer could update the conn_val, e.g., by adding additional, optional fields to the connection record. In terms of speed and memory consumption this is surprisingly lightweight. Runtime as reported by time, memory consumption from top. I've only loaded conn and weird: * Baseline. Current git master: (3 runs) 2m42s 900MB 2m37s 900MB 2m37s 900MB * My version, with counting disabled (i.e., counting analyzer not instantiated, no the optional fields in endpoint are NULL): 2m34s 898MB 2m36s 898MB 2m36s 898MB It seems to be faster and using less memory(!). There is an explanation why it needs slightly less memory: the git-master Conn.cc had two RecordVal* for orig_endp, and resp_endp that were passed to UpdateEndpointVal. Since I now only pass the conn_val to UpdateConnVal, I don't need these pointers in Conn.cc anymore. I've checked the testsuite it my version passes it. * My version with counting enabled: 2m40s 1000MB 2m41s 1000MB 2m41s 1000MB 2m41s 1000MB So: no speed penalty and about 100MB for the analyzer instance and the additional fields in endpoint (these are responsible for most of the memory. Vals are expensive memory-wise). * As an additional experiment, I made the history field in conn_val &optional, so that it's only allocated, if record_state_history=T. Results: with counting: 2m39s 862MB wihtout counting: 2m33s 962MB So I think it might make sense to also integrate this change. (Currently the history is always tracked and always part of the connection record, but it is only logged in conn.log if record_state_history=T). Finally, a longer trace with way more data and more policy scripts (conn weird dpd http-request htt-reply dns): master / baseline 88m38 2906MB no counting 88m37 2907MB with counting 87m47 2963MB So, all in all I would say: we should go with this version, as this seems to be the ideal solution: overhead when disabled, only memory overhead when enabled (and this overhead is inevitable!). In addition, I would suggest to make c$history &optional and only allocate and assign a Val* to it if record_state_history=T, since this seems to safe quite a bit of memory! cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From seth at icir.org Tue Dec 14 19:11:28 2010 From: seth at icir.org (Seth Hall) Date: Tue, 14 Dec 2010 22:11:28 -0500 Subject: [Bro-Dev] Problem installing In-Reply-To: <24759267.206.1292366062390.JavaMail.jsiwek@tangent.ncsa.illinois.edu> References: <24759267.206.1292366062390.JavaMail.jsiwek@tangent.ncsa.illinois.edu> Message-ID: <81FDCA3A-6FF3-438D-A147-0C593CC14A67@icir.org> On Dec 14, 2010, at 5:34 PM, Jonathan Siwek wrote: > But it looks like it skipped straight to #3 and I don't understand why. Does that directory really not exist? As I was going through documenting everything you asked for, I found the problem. I installed the binary copy of cmake for MacOS X and it's installed in the Applications directory with a symlink to /usr/bin/. /Applications/CMake 2.8-2.app/Contents/bin/cmake The space in the path makes things not work so well. :) .Seth From seth at icir.org Tue Dec 14 21:06:17 2010 From: seth at icir.org (Seth Hall) Date: Wed, 15 Dec 2010 00:06:17 -0500 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: <4D0805D6.7030206@icir.org> References: <4D0805D6.7030206@icir.org> Message-ID: <6F9A107A-07B2-4A0D-BB33-7B6B6119159B@icir.org> On Dec 14, 2010, at 7:03 PM, Gregor Maier wrote: > I've played around with packet and byte counting for connections some > more. I think I've got a pretty neat version: This sounds awesome! The only question I have is what is the $history field used for? This is second time I've seen it referred to today, but I have no idea what it's for. :) .Seth From gregor at icir.org Wed Dec 15 07:43:28 2010 From: gregor at icir.org (Gregor Maier) Date: Wed, 15 Dec 2010 07:43:28 -0800 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: <6F9A107A-07B2-4A0D-BB33-7B6B6119159B@icir.org> References: <4D0805D6.7030206@icir.org> <6F9A107A-07B2-4A0D-BB33-7B6B6119159B@icir.org> Message-ID: <4D08E220.9050701@icir.org> On 12/14/10 21:06 , Seth Hall wrote: > The only question I have is what is the $history field used for? This is second time I've seen it referred to today, but I have no idea what it's for. :) It records the state history of (TCP) connections. When record_state_history=T it will add an additional column to conn.log. It's probably mostly used for measurements rather than live security deployment. The "history" is a string of letters: -8<------ s == a SYN w/o the ACK bit set h == a SYN+ACK ("handshake") a == a pure ACK d == packet with payload ("data") f == packet with FIN bit set r == packet with RST bit set I also think there is a "c" which means we saw a packet with a bad checksum and "i" for "inconsistent" packets---basically just weird crap like both SYN and RST bits lit up. If the code is in upper case it means the event comes from the originator and lower case then means the responder. Also, there is compression. We only record one "d" in each direction, for instance. I.e., we just record that data went in that direction. This history is not meant to encode how much data that happened to be. ->8------ cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From jsiwek at ncsa.illinois.edu Wed Dec 15 09:00:20 2010 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Wed, 15 Dec 2010 11:00:20 -0600 (CST) Subject: [Bro-Dev] Problem installing In-Reply-To: <81FDCA3A-6FF3-438D-A147-0C593CC14A67@icir.org> Message-ID: <21617372.230.1292432415544.JavaMail.jsiwek@tangent.ncsa.illinois.edu> > I found the problem. I installed the binary copy of cmake for MacOS X and it's > installed in the Applications directory with a symlink to /usr/bin/. > > /Applications/CMake 2.8-2.app/Contents/bin/cmake > > The space in the path makes things not work so well. :) Ah ok, that makes sense to me now. At "make install" time I made the copy of the BroControl directory with `${CMAKE_COMMAND} -E copy_directory` instead of `cp` because I thought that would be the more 'portable' way to do things, but it seems I need to add some quotes around ${CMAKE_COMMAND} so that spaces in the path get escaped. I'll fix it on fastpath. - Jon From seth at icir.org Wed Dec 15 13:12:21 2010 From: seth at icir.org (Seth Hall) Date: Wed, 15 Dec 2010 16:12:21 -0500 Subject: [Bro-Dev] Problem installing In-Reply-To: <21617372.230.1292432415544.JavaMail.jsiwek@tangent.ncsa.illinois.edu> References: <21617372.230.1292432415544.JavaMail.jsiwek@tangent.ncsa.illinois.edu> Message-ID: <54D799D1-AFF3-47F8-BB82-AFE01B36D1B8@icir.org> On Dec 15, 2010, at 12:00 PM, Jonathan Siwek wrote: > add some quotes around ${CMAKE_COMMAND} so that spaces in the path get escaped. I never would have expected to encounter spaces there either. :P > I'll fix it on fastpath. Thanks! .Seth From bro at tracker.icir.org Thu Dec 16 20:53:55 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 17 Dec 2010 04:53:55 -0000 Subject: [Bro-Dev] #342: Add payload to ICMP analyzer In-Reply-To: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> References: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> Message-ID: <058.83ff2df2fa72fe7350c0f23f75fefb3c@tracker.icir.org> #342: Add payload to ICMP analyzer ---------------------+-------------------- Reporter: seth | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: ---------------------+-------------------- Comment (by vern): I don't believe there's any particular reason why this functionality wasn't previously available. That said, the point about potential overhead is something to consider. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Thu Dec 16 21:12:23 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 17 Dec 2010 05:12:23 -0000 Subject: [Bro-Dev] #342: Add payload to ICMP analyzer In-Reply-To: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> References: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> Message-ID: <058.571baf427561ad4bb4ef785cc900c0fe@tracker.icir.org> #342: Add payload to ICMP analyzer ---------------------+-------------------- Reporter: seth | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: ---------------------+-------------------- Comment (by matti): I suppose that the payload would not be usefull for all of the ICMP messages sent, so I would probably just include that for the types where it is of use. I think it would make sense to rather have a whole lot of different ICMP message events and then have the payload and other data only available where it is relevant. This would at least avoid some of the overhead. I'll add the payload for the new ICMP events in my branch where it is usefull. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Thu Dec 16 21:26:16 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 17 Dec 2010 05:26:16 -0000 Subject: [Bro-Dev] #342: Add payload to ICMP analyzer In-Reply-To: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> References: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> Message-ID: <058.abf55de8a4478d802f65b680c52339b1@tracker.icir.org> #342: Add payload to ICMP analyzer ---------------------+-------------------- Reporter: seth | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: ---------------------+-------------------- Comment (by vern): Yeah, good point. I was thinking ICMP-echo-request/reply in particular. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Dec 16 23:10:48 2010 From: robin at icir.org (Robin Sommer) Date: Thu, 16 Dec 2010 23:10:48 -0800 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: <4D0805D6.7030206@icir.org> References: <4D0805D6.7030206@icir.org> Message-ID: <20101217071048.GL94616@icir.org> On Tue, Dec 14, 2010 at 16:03 -0800, you wrote: > So, all in all I would say: we should go with this version Yes, this all sounds really good. One more thought: does expect_connection() work with this (see, e.g., ftp.bro for how it's used). Then one could enable it for individual future sessions (like file transfers). In principle, it seems an enable_analyzer() function (which we still don't have; there's just disable_analyzer()) could be nice for this too. However, it would only get a chance to add the counting once a few packets have already passed so not sure how useful it would really be. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Thu Dec 16 23:20:08 2010 From: robin at icir.org (Robin Sommer) Date: Thu, 16 Dec 2010 23:20:08 -0800 Subject: [Bro-Dev] Problem installing In-Reply-To: <21617372.230.1292432415544.JavaMail.jsiwek@tangent.ncsa.illinois.edu> References: <81FDCA3A-6FF3-438D-A147-0C593CC14A67@icir.org> <21617372.230.1292432415544.JavaMail.jsiwek@tangent.ncsa.illinois.edu> Message-ID: <20101217072008.GB96720@icir.org> On Wed, Dec 15, 2010 at 11:00 -0600, you wrote: > I'll fix it on fastpath. I'll merge in it, but for the future, please create new branches when doing such changes across all (or many) of the repositories. That makes merging easier as then I don't need to pay attention what else is already in fastpath but not yet merged. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.icir.org Thu Dec 16 23:27:07 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 17 Dec 2010 07:27:07 -0000 Subject: [Bro-Dev] #342: Add payload to ICMP analyzer In-Reply-To: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> References: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> Message-ID: <058.a6377d94145e46e9722eeebffe7efa12@tracker.icir.org> #342: Add payload to ICMP analyzer ---------------------+-------------------- Reporter: seth | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: ---------------------+-------------------- Comment (by matti): Actually the echo message events already have the payload available to them on the scripting level. This is before any of my changes, but they are currently the only ones that get the payload passed to them. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri Dec 17 00:49:03 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 17 Dec 2010 08:49:03 -0000 Subject: [Bro-Dev] #340: Cleanup: unify where global consts are defined (access from policy layer and event engine) In-Reply-To: <045.e26a2c4cd7e7f98be9266f24419513fb@tracker.icir.org> References: <045.e26a2c4cd7e7f98be9266f24419513fb@tracker.icir.org> Message-ID: <060.249f50afc8609a4e9106972522b41ed7@tracker.icir.org> #340: Cleanup: unify where global consts are defined (access from policy layer and event engine) ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Changes (by robin): * milestone: Bro1.6 => Bro1.7 Comment: Changing milestone to 1.7, as this is mostly about internals, and less about user-visiable aspects of the scripts. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Fri Dec 17 01:07:12 2010 From: robin at icir.org (Robin Sommer) Date: Fri, 17 Dec 2010 01:07:12 -0800 Subject: [Bro-Dev] connection_established behavior In-Reply-To: <20101214174243.E8CB036A414@taffy.ICSI.Berkeley.EDU> References: <4F01E937-2272-43E0-AAD9-38FB2B3ED5CA@icir.org> <20101214174243.E8CB036A414@taffy.ICSI.Berkeley.EDU> Message-ID: <20101217090712.GH96720@icir.org> On Tue, Dec 14, 2010 at 09:42 -0800, you wrote: > - Instantiating on SYN ACK came about due to coping with Bro > deployments with split routing, such that they never saw > initial SYNs for some connections. Isn't this controlled by this option: # If true, instantiate connection state when a SYN ack is seen # but not the initial SYN (even if partial_connection_ok is false). const tcp_SYN_ack_ok = T &redef; Perhaps we should just change the default? > and instead go with an empirical set. $history allows this but > in an implicit fashion, rather than with explicit states. The > latter would be better, though it's not clear to me that it's > really worth the work. Not for the time being I would say, as this would be quite a majro change. However, at some point, we should definitlu look more closely at the TCP code. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Fri Dec 17 01:31:28 2010 From: robin at icir.org (Robin Sommer) Date: Fri, 17 Dec 2010 01:31:28 -0800 Subject: [Bro-Dev] Unique connection ID for bro <-> logging framework In-Reply-To: <4D067425.70304@icir.org> References: <4CF9A8F6.1020205@icir.org> <4CFD1D74.4040600@icir.org> <20101210103231.GK15294@icir.org> <4D027A4D.7050406@icir.org> <20101213085054.GE45029@icir.org> <4D067425.70304@icir.org> Message-ID: <20101217093128.GK96720@icir.org> On Mon, Dec 13, 2010 at 11:29 -0800, you wrote: > I'd prefer to keep the counter and the runID separate. E.g., by making > the runID n-bits and the counter 64-n bits and the OR-ing them together. What's the advatnage of keeping them separate? > using a say, 32 bit runID + 64 bit unique-ID. We now use ACSII for > logging, if we move to binary (and possibly compression), then this will > save *way* more memory than our ID will ever add. Well, that's right, but still no excuse for waste if we don't need it. :) (I'm also thingking about potentially piping logs into a DB and perhaps indexing on such an ID as a key.) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.icir.org Fri Dec 17 06:10:47 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 17 Dec 2010 14:10:47 -0000 Subject: [Bro-Dev] #342: Add payload to ICMP analyzer In-Reply-To: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> References: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> Message-ID: <058.f534293462986567a8143e8c1deeb06f@tracker.icir.org> #342: Add payload to ICMP analyzer ---------------------+-------------------- Reporter: seth | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: ---------------------+-------------------- Comment (by seth): > Actually the echo message events already have the payload available to > them on the scripting level. Oh, what event does this patch add the payload body to? -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Fri Dec 17 06:13:16 2010 From: seth at icir.org (Seth Hall) Date: Fri, 17 Dec 2010 09:13:16 -0500 Subject: [Bro-Dev] connection_established behavior In-Reply-To: <20101217090712.GH96720@icir.org> References: <4F01E937-2272-43E0-AAD9-38FB2B3ED5CA@icir.org> <20101214174243.E8CB036A414@taffy.ICSI.Berkeley.EDU> <20101217090712.GH96720@icir.org> Message-ID: <20C6CC11-84FA-4A87-9BA1-E4BD5D95439B@icir.org> On Dec 17, 2010, at 4:07 AM, Robin Sommer wrote: > # If true, instantiate connection state when a SYN ack is seen > # but not the initial SYN (even if partial_connection_ok is false). > const tcp_SYN_ack_ok = T &redef; > > Perhaps we should just change the default? Ah! I should have looked. I didn't even know that about that option. It would say that it's worthwhile to change the default because it would at least make the output the same with and without connection compressor (and not changing any other settings). .Seth From bro at tracker.icir.org Fri Dec 17 07:08:40 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 17 Dec 2010 15:08:40 -0000 Subject: [Bro-Dev] #340: Cleanup: unify where global consts are defined (access from policy layer and event engine) In-Reply-To: <045.e26a2c4cd7e7f98be9266f24419513fb@tracker.icir.org> References: <045.e26a2c4cd7e7f98be9266f24419513fb@tracker.icir.org> Message-ID: <060.7e4c3f4a939f1a7eafd57d6d61706610@tracker.icir.org> #340: Cleanup: unify where global consts are defined (access from policy layer and event engine) ------------------------------+-------------------- Reporter: gregor | Owner: Type: Feature Request | Status: new Priority: Low | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by gregor): My thought was that this is easy and quick to do when we go through all the scripts anyway. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.icir.org Fri Dec 17 07:23:12 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Fri, 17 Dec 2010 15:23:12 -0000 Subject: [Bro-Dev] #342: Add payload to ICMP analyzer In-Reply-To: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> References: <043.9f47668a205ed0345f897209d20f2055@tracker.icir.org> Message-ID: <058.abaebc547a9d95d902fc86f51d776cbe@tracker.icir.org> #342: Add payload to ICMP analyzer ---------------------+-------------------- Reporter: seth | Owner: Type: Patch | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: 1.5.2 Resolution: | Keywords: ---------------------+-------------------- Comment (by matti): Replying to [comment:7 seth]: > > Actually the echo message events already have the payload available to > > them on the scripting level. > > Oh, what event does this patch add the payload body to? That seems to be the generic icmp_sent message which is raised when there isn't any of the more specific events raised. -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Fri Dec 17 07:30:15 2010 From: gregor at icir.org (Gregor Maier) Date: Fri, 17 Dec 2010 07:30:15 -0800 Subject: [Bro-Dev] Per connection byte and packet counting In-Reply-To: <20101217071048.GL94616@icir.org> References: <4D0805D6.7030206@icir.org> <20101217071048.GL94616@icir.org> Message-ID: <4D0B8207.6030705@icir.org> > One more thought: does expect_connection() work with this (see, > e.g., ftp.bro for how it's used). Then one could enable it for > individual future sessions (like file transfers). Don't think so, since it needs to be a child *packet* analyzer of TCP_Analyzer. > In principle, it seems an enable_analyzer() function (which we still > don't have; there's just disable_analyzer()) could be nice for this > too. However, it would only get a chance to add the counting once a > few packets have already passed so not sure how useful it would > really be. In addition to enable_analyzer() and add_to_initial_tree() function would also be nice, so that users can select analyzers that they want to run on all connections. -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From vern at icir.org Fri Dec 17 07:33:39 2010 From: vern at icir.org (Vern Paxson) Date: Fri, 17 Dec 2010 07:33:39 -0800 Subject: [Bro-Dev] connection_established behavior In-Reply-To: <20C6CC11-84FA-4A87-9BA1-E4BD5D95439B@icir.org> (Fri, 17 Dec 2010 09:13:16 EST). Message-ID: <20101217153339.DAC47369FC7@taffy.ICSI.Berkeley.EDU> > > const tcp_SYN_ack_ok = T &redef; > > > > Perhaps we should just change the default? > > Ah! I should have looked. I didn't even know that about that option. Yeah, I'd forgotten about it too. FWIW, I believe that was added for Bro deployments with high packet loss rates, since they will sometimes miss the initial SYN but not the SYN-ACK. Vern From bro at tracker.icir.org Fri Dec 17 22:17:34 2010 From: bro at tracker.icir.org (Bro Tracker) Date: Sat, 18 Dec 2010 06:17:34 -0000 Subject: [Bro-Dev] #343: broctl's top helper and cpu % on Mac OS X Message-ID: <043.dc057d41edc3cf424e4384682f6df6fd@tracker.icir.org> #343: broctl's top helper and cpu % on Mac OS X ------------------------+------------------------ Reporter: seth | Owner: robin Type: Problem | Status: new Priority: Low | Milestone: Component: BroControl | Version: git/master Keywords: | ------------------------+------------------------ BroControl's "top" helper doesn't give a value for the CPU percent because Mac OS X's top doesn't give a value for that column on the first sample but the helper script only gets one sample at a time. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Fri Dec 17 22:34:38 2010 From: seth at icir.org (Seth Hall) Date: Sat, 18 Dec 2010 01:34:38 -0500 Subject: [Bro-Dev] macports and bro Message-ID: For fun, and to see how well the new build system could work with a packaging system, I built some port files for macports tonight and had a few comments. *It seems that when "make install" is called in broctl, the broctl-config.sh script is not generated. Maybe "broctl install" just needs to be run as part of the "make install" process? * Perhaps the "spool" directory for broctl should be moved to "/var/spool/bro"? MacPorts gives the following warning messages: Warning: violation by /opt/local/spool Warning: broctl violates the layout of the ports-filesystems! Warning: Please fix or indicate this misbehavior (if it is intended), it will be an error in future releases! * I wonder if broctl's configuration files should be put into a subdirectory in etc, like /etc/broctl/node.cfg, etc.? Right now, if Bro is installed alongside other software you end up with /etc/node.cfg which is a little confusing. .Seth From robin at icir.org Sat Dec 18 00:06:50 2010 From: robin at icir.org (Robin Sommer) Date: Sat, 18 Dec 2010 00:06:50 -0800 Subject: [Bro-Dev] macports and bro In-Reply-To: References: Message-ID: <20101218080650.GA36413@icir.org> On Sat, Dec 18, 2010 at 01:34 -0500, you wrote: > *It seems that when "make install" is called in broctl, the > broctl-config.sh script is not generated. Maybe "broctl install" > just needs to be run as part of the "make install" process? (Jon knows best where/when this is generated.) > * Perhaps the "spool" directory for broctl should be moved to > "/var/spool/bro"? > * I wonder if broctl's configuration files should be put into a > subdirectory in etc, like /etc/broctl/node.cfg, etc.? Two issues different issues here: one is what a particular distribution/port-system/etc. wants, and the other is what we want to be our default. For the latter, I'd be happy to change the defaults if we find something more appropiate. For the former, however, we can't make everybody happy at the same time and thus should have a way to easily adapt when building packages/ports. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at ncsa.illinois.edu Sat Dec 18 07:15:54 2010 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Sat, 18 Dec 2010 09:15:54 -0600 (CST) Subject: [Bro-Dev] macports and bro In-Reply-To: Message-ID: <1578085329.5937.1292685354987.JavaMail.root@zimbra-1.ncsa.uiuc.edu> > *It seems that when "make install" is called in broctl, the > broctl-config.sh script is not generated. Maybe "broctl install" just > needs to be run as part of the "make install" process? Yeah, broctl-config.sh should be generated at "broctl install" time. So I think it makes sense for the packager to run that once in order to get a broctl-config.sh that they can include. - Jon From gregor at icir.org Sun Dec 19 09:21:04 2010 From: gregor at icir.org (Gregor Maier) Date: Sun, 19 Dec 2010 09:21:04 -0800 Subject: [Bro-Dev] TCP Reassembler question Message-ID: <4D0E3F00.3080802@icir.org> Hi, I have some questions regarding the TCP resassembler. I have a midstream NFS con nection (i.e., no handshake) with tons of data. The NFS analyzer can handle gaps and partial connections, however it seems that there are some content gaps and that the TCP Reassembler doesn't recover from them. * When I look at the packet level, I see data packets all the time, but the analyzer's DeliverStreams stops being called somewhere half through the trace. * I don't get any calls to Undelivered() either (actually I get some, at the very end of the trace, but the delivery stops way way earlier. * I *don't* get content_gap and ack above hole message, because the connections doesn't have a handshake. Can I force that somehow? (So that I can debug where the gaps happen). * What's the Reassemblers default / indented bevhavior wrt gaps in partial connections? Are there any policy-level settings I can tweak? cu gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From gregor at icir.org Sun Dec 19 09:32:02 2010 From: gregor at icir.org (Gregor Maier) Date: Sun, 19 Dec 2010 09:32:02 -0800 Subject: [Bro-Dev] connection_established behavior In-Reply-To: <20101217090712.GH96720@icir.org> References: <4F01E937-2272-43E0-AAD9-38FB2B3ED5CA@icir.org> <20101214174243.E8CB036A414@taffy.ICSI.Berkeley.EDU> <20101217090712.GH96720@icir.org> Message-ID: <4D0E4192.1060000@icir.org> On 12/17/10 1:07 , Robin Sommer wrote: >> - Instantiating on SYN ACK came about due to coping with Bro >> deployments with split routing, such that they never saw >> initial SYNs for some connections. > > Isn't this controlled by this option: > > # If true, instantiate connection state when a SYN ack is seen > # but not the initial SYN (even if partial_connection_ok is false). > const tcp_SYN_ack_ok = T &redef; > > > Perhaps we should just change the default? Then you would only instantiate on connections with a full handshake. So, I would keep it as it is! (It only takes effect if partial_connection_ok=F) Changing this option wouldn't necessarily change event generation either, I think. We just have to unify the way the connection_established event is generated in TCP.cc and Connection Compressor. >> and instead go with an empirical set. $history allows this but >> in an implicit fashion, rather than with explicit states. The >> latter would be better, though it's not clear to me that it's >> really worth the work. > > Not for the time being I would say, as this would be quite a majro > change. However, at some point, we should definitlu look more > closely at the TCP code. Yeah. That's basically a rewrite of most of the TCP Analyzer. Probably something to consider at some point though. cu gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From robin at icir.org Sun Dec 19 12:59:14 2010 From: robin at icir.org (Robin Sommer) Date: Sun, 19 Dec 2010 12:59:14 -0800 Subject: [Bro-Dev] macports and bro In-Reply-To: <1578085329.5937.1292685354987.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <1578085329.5937.1292685354987.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <20101219205914.GC94184@icir.org> On Sat, Dec 18, 2010 at 09:15 -0600, you wrote: > Yeah, broctl-config.sh should be generated at "broctl install" time. > So I think it makes sense for the packager to run that once in order > to get a broctl-config.sh that they can include. So, do we need to change something here? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Sun Dec 19 14:00:23 2010 From: seth at icir.org (Seth Hall) Date: Sun, 19 Dec 2010 17:00:23 -0500 Subject: [Bro-Dev] macports and bro In-Reply-To: <20101219205914.GC94184@icir.org> References: <1578085329.5937.1292685354987.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20101219205914.GC94184@icir.org> Message-ID: <7A6FDEF5-0259-4A5C-A423-16C21BBC2C9B@icir.org> On Dec 19, 2010, at 3:59 PM, Robin Sommer wrote: > > On Sat, Dec 18, 2010 at 09:15 -0600, you wrote: > >> Yeah, broctl-config.sh should be generated at "broctl install" time. >> So I think it makes sense for the packager to run that once in order >> to get a broctl-config.sh that they can include. > > So, do we need to change something here? I think Jon's right. Without applying any configuration to broctl, it won't work anyway so the "install" command would need to be run after the user does some amount of configuration. What could be worthwhile is to give the user some warning when they go to start Bro if they a. don't have the generated scripts (broctl-config.sh and the local-nets, etc). or b. have changes to their scripts in their site directory that haven't been installed yet. .Seth From robin at icir.org Mon Dec 20 06:19:31 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 20 Dec 2010 06:19:31 -0800 Subject: [Bro-Dev] macports and bro In-Reply-To: <7A6FDEF5-0259-4A5C-A423-16C21BBC2C9B@icir.org> References: <1578085329.5937.1292685354987.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20101219205914.GC94184@icir.org> <7A6FDEF5-0259-4A5C-A423-16C21BBC2C9B@icir.org> Message-ID: <20101220141931.GD32362@icir.org> On Sun, Dec 19, 2010 at 17:00 -0500, you wrote: > a. don't have the generated scripts (broctl-config.sh and the local-nets, etc). Yeah, that makes sense. > b. have changes to their scripts in their site directory that haven't been installed yet. This likewise but it's quite hard to do for everything that could potentially change (it's not only the scripts, but also broctl.cfg, which in turn may set things differnetly so that now new pieces somewhere else become relevant). I think we can skip this at least for now. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon Dec 20 10:12:21 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 20 Dec 2010 10:12:21 -0800 Subject: [Bro-Dev] connection_established behavior In-Reply-To: <4D0E4192.1060000@icir.org> References: <4F01E937-2272-43E0-AAD9-38FB2B3ED5CA@icir.org> <20101214174243.E8CB036A414@taffy.ICSI.Berkeley.EDU> <20101217090712.GH96720@icir.org> <4D0E4192.1060000@icir.org> Message-ID: <20101220181221.GD36236@icir.org> On Sun, Dec 19, 2010 at 09:32 -0800, you wrote: > > Perhaps we should just change the default? > > Then you would only instantiate on connections with a full handshake. Right, that was the intent of the original question, wasn't it? > Changing this option wouldn't necessarily change event generation > either, I think. I think it would. If there's no connection state, there's no connection_established. > We just have to unify the way the connection_established event is > generated in TCP.cc and Connection Compressor. Note that the CC doesn't generate connection_established itself. It acts as a filter: once it lets packets through, TCP.cc will generate the event. Also, I'm thinking there might actually be a bug in the CC here (or at least semantics not originally intended): it has an option cc_handle_only_syns that, if on, tells it to take care only of initial SYN packets and forward everything else (e.g., a stray data packet) to normal processing. The motivation for this was to have less difference to normal TCP processing when the cc is on, and it's per default enabled (bro.init calls this mode "connection compressor light"). However, it turns out that a SYN/ACK counts here and *is* processed by the CC, rather than being passed on. If we would change that so only pure SYNs were handled, we shouldn't see a difference between cc on vs. off anymore in this case. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Mon Dec 20 14:16:34 2010 From: gregor at icir.org (Gregor Maier) Date: Mon, 20 Dec 2010 17:16:34 -0500 Subject: [Bro-Dev] connection_established behavior In-Reply-To: <20101220181221.GD36236@icir.org> References: <4F01E937-2272-43E0-AAD9-38FB2B3ED5CA@icir.org> <20101214174243.E8CB036A414@taffy.ICSI.Berkeley.EDU> <20101217090712.GH96720@icir.org> <4D0E4192.1060000@icir.org> <20101220181221.GD36236@icir.org> Message-ID: <4D0FD5C2.5000202@icir.org> >>> Perhaps we should just change the default? >> >> Then you would only instantiate on connections with a full handshake. > > Right, that was the intent of the original question, wasn't it? No, it was about the connection established event, IIRC. In case the conn compressor is disabled, TCP.cc would generate the connection_established event on a pure SYN/ACK while the conn compressor would not do this. (If the "connection" has other packets in it, the conn compressor would instantiate a connection, pass it to TCP and thus the connection_established event would be generated.) The tcp_SYN_ack_ok and partial_connection_ok will prevent any connection without full handshake to be processed by bro. I don't think that we want this as default. I think that the current default, namely that bro tries to parse partial connections is better. >> Changing this option wouldn't necessarily change event generation >> either, I think. > > I think it would. If there's no connection state, there's no > connection_established. Actually that's true, if there's no Connection instance, there's no TCP Analyzer so no event can be genreated. >> We just have to unify the way the connection_established event is >> generated in TCP.cc and Connection Compressor. > > Note that the CC doesn't generate connection_established itself. It > acts as a filter: once it lets packets through, TCP.cc will generate > the event. > > Also, I'm thinking there might actually be a bug in the CC here (or > at least semantics not originally intended): it has an option > cc_handle_only_syns that, if on, tells it to take care only of > initial SYN packets and forward everything else (e.g., a stray data > packet) to normal processing. The motivation for this was to have > less difference to normal TCP processing when the cc is on, and it's > per default enabled (bro.init calls this mode "connection compressor > light"). Hmm. My intuition would have been that SYN/ACK also count here... > However, it turns out that a SYN/ACK counts here and *is* processed > by the CC, rather than being passed on. If we would change that so > only pure SYNs were handled, we shouldn't see a difference between > cc on vs. off anymore in this case. Correct. However, one can also argue that the "instantiate if we see a packet from each endpoint" makes sense in order to not instantiate on scans or floods, but instantiate if we "just" miss part of the handshake. I guess the questions are: * what is the semantic of the connection_established even? If we've defined that, we should unify CC and non-CC behavior. * what to do with partial connections or connections that start with a SYN/ACK (and that have further data packets). * what to do with connections that only have a SYN/ACK. cu Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA From seth at icir.org Mon Dec 20 19:42:02 2010 From: seth at icir.org (Seth Hall) Date: Mon, 20 Dec 2010 22:42:02 -0500 Subject: [Bro-Dev] macports and bro In-Reply-To: <20101220141931.GD32362@icir.org> References: <1578085329.5937.1292685354987.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20101219205914.GC94184@icir.org> <7A6FDEF5-0259-4A5C-A423-16C21BBC2C9B@icir.org> <20101220141931.GD32362@icir.org> Message-ID: <7577918F-088F-4EC9-8F46-DB90E51CBDAA@icir.org> On Dec 20, 2010, at 9:19 AM, Robin Sommer wrote: > This likewise but it's quite hard to do for everything that could > potentially change (it's not only the scripts, but also broctl.cfg, > which in turn may set things differnetly so that now new pieces > somewhere else become relevant). I think we can skip this at least > for now. Maybe for now we could do the minimal check for changes in the site directory or changes to the etc files and giving a warning? Even if that check isn't 100% complete, it will solve a lot of trouble for people. We could file a ticket to remind ourselves to go back later and see if there are any other situations where we should warn. I just know that myself and the other guys at OSU have run into the "forgot to install" problem multiple times when updating scripts in the site directory. .Seth From robin at icir.org Mon Dec 20 21:15:01 2010 From: robin at icir.org (Robin Sommer) Date: Mon, 20 Dec 2010 21:15:01 -0800 Subject: [Bro-Dev] macports and bro In-Reply-To: <7577918F-088F-4EC9-8F46-DB90E51CBDAA@icir.org> References: <1578085329.5937.1292685354987.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <20101219205914.GC94184@icir.org> <7A6FDEF5-0259-4A5C-A423-16C21BBC2C9B@icir.org> <20101220141931.GD32362@icir.org> <7577918F-088F-4EC9-8F46-DB90E51CBDAA@icir.org> Message-ID: <20101221051501.GB22771@icir.org> On Mon, Dec 20, 2010 at 22:42 -0500, you wrote: > Maybe for now we could do the minimal check for changes in the site > directory or changes to the etc files and giving a warning? Ok, makes sense. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From vern at icir.org Wed Dec 22 18:20:55 2010 From: vern at icir.org (Vern Paxson) Date: Wed, 22 Dec 2010 18:20:55 -0800 Subject: [Bro-Dev] TCP Reassembler question In-Reply-To: <4D0E3F00.3080802@icir.org> (Sun, 19 Dec 2010 09:21:04 PST). Message-ID: <20101223022055.226F6369FC7@taffy.ICSI.Berkeley.EDU> > I have some questions regarding the TCP resassembler. I have a midstream > NFS connection (i.e., no handshake) with tons of data. The NFS analyzer can > handle gaps and partial connections Are you sure it does okay with those? It's layered on top of the RPC handler, and I would think that that will make it brittle to missing byte-stream elements. > * When I look at the packet level, I see data packets all the time, but > the analyzer's DeliverStreams stops being called somewhere half > through the trace. If it's a huge trace, then I'm guessing you may be encountering wrap-round with: if ( seq_delta(start_block->seq, last_reassem_seq) > 0 || seq_delta(start_block->upper, last_reassem_seq) <= 0 ) return; Or have you already converted all of those variables to 64-bit? Alternatively, you may be running into the behavior I sketch in response to your next question: > * I don't get any calls to Undelivered() either (actually I get some, > at the very end of the trace, but the delivery stops way way earlier. This sounds like skip_deliveries is getting set. This can happen due to the following logic: if ( Endpoint()->NoDataAcked() && tcp_max_above_hole_without_any_acks && NumUndeliveredBytes() > tcp_max_above_hole_without_any_acks ) { tcp_analyzer->Weird("above_hole_data_without_any_acks"); ClearBlocks(); skip_deliveries = 1; } if ( tcp_excessive_data_without_further_acks && NumUndeliveredBytes() > tcp_excessive_data_without_further_acks ) { tcp_analyzer->Weird("excessive_data_without_further_acks"); ClearBlocks(); skip_deliveries = 1; } So it could be that you have enough loss at some point that you're tripping over one of these thresholds. > * I *don't* get content_gap and ack above hole message, > because the connections doesn't have a handshake. Can I force that > somehow? (So that I can debug where the gaps happen). Changing this would require source-code edits, because it's wired in as: if ( content_gap && endpoint->state == TCP_ENDPOINT_ESTABLISHED && peer->state == TCP_ENDPOINT_ESTABLISHED ) (there's a comment before this explaining false positives that can result for non-established connections, though I think it's reasonable giving users an option to abide these possibilities) Vern From gregor at icir.org Thu Dec 23 07:14:08 2010 From: gregor at icir.org (Gregor Maier) Date: Thu, 23 Dec 2010 10:14:08 -0500 Subject: [Bro-Dev] TCP Reassembler question In-Reply-To: <20101223022055.226F6369FC7@taffy.ICSI.Berkeley.EDU> References: <20101223022055.226F6369FC7@taffy.ICSI.Berkeley.EDU> Message-ID: <4D136740.3060303@icir.org> On 12/22/10 21:20 , Vern Paxson wrote: >> [Re-sync] > > Are you sure it does okay with those? It's layered on top of the RPC > handler, and I would think that that will make it brittle to missing > byte-stream elements. Yes. The RPC-analyzer has some heuristics to re-sync to a TCP-stream with gaps. It definitely works at the start of the trace. (My problem is also that DeliverStream doesn't get called at some stage) >> * When I look at the packet level, I see data packets all the time, but >> the analyzer's DeliverStreams stops being called somewhere half >> through the trace. > > If it's a huge trace, then I'm guessing you may be encountering wrap-round with: > > [snip] > Or have you already converted all of those variables to 64-bit? I haven't converted them to 64 bit and the trace is indeed huge. However, I thought that seq_delta() handles wrap-around correctly. (Also, I think that the problem occurs after the first 4GB of data). Or can there be a problem if a hole spans over the seq wrap-around? I'll have a look at the reassembler code,. >> * I don't get any calls to Undelivered() either (actually I get some, >> at the very end of the trace, but the delivery stops way way earlier. > > This sounds like skip_deliveries is getting set. This can happen due > to the following logic: > > [snip] > > So it could be that you have enough loss at some point that you're tripping > over one of these thresholds. I was thinking that too, however, I don't see any Weird message from the Reassembler that would hint at any of those (but I think that there are other places where skip_deliveries) gets sets where there isn't a Weird. >> * I *don't* get content_gap and ack above hole message, >> because the connections doesn't have a handshake. Can I force that >> somehow? (So that I can debug where the gaps happen). > > Changing this would require source-code edits, because it's wired in as: > > if ( content_gap && > endpoint->state == TCP_ENDPOINT_ESTABLISHED && > peer->state == TCP_ENDPOINT_ESTABLISHED ) > > (there's a comment before this explaining false positives that can result > for non-established connections, though I think it's reasonable giving > users an option to abide these possibilities) I'll make this policy-configurable then. I'll investigate this some more. See how many gaps I have, if skip_deliveries gets set, etc. thanks, Gregor -- Gregor Maier gregor at icir.org Int. Computer Science Institute (ICSI) gregor at icsi.berkeley.edu 1947 Center St., Ste. 600 http://www.icir.org/gregor/ Berkeley, CA 94704 USA