From mike.muratet at torchtechnologies.com Wed Aug 3 08:06:35 2005 From: mike.muratet at torchtechnologies.com (Mike Muratet) Date: Wed, 3 Aug 2005 10:06:35 -0500 Subject: [Bro] conn.bro Message-ID: <004f01c5983c$f2104760$5501a8c0@muratet> Greetings I have been doing some experiments with broccoli (thanks again, Christian) and I'm trying to understand the differences between what I see in real time from broccoli vis a vis some earlier work I did using bro -r a_tcpdump_file conn.bro. Looking at the conn.bro policy, it looks like it records a connection record for the events connection_attempt, _partial_close, _finished, _half_finished, _rejected, and _reset, using record_connection(connection_record, string). The string parameter denotes the type of event, but this does not get printed. It does record a state as part of the connection record which gives you an idea of what the event was all about. Is my interpretation correct? Thanks Mike From christian at whoop.org Wed Aug 3 10:01:15 2005 From: christian at whoop.org (Christian Kreibich) Date: Wed, 03 Aug 2005 18:01:15 +0100 Subject: [Bro] Emacs mode Message-ID: <1123088476.7525.138.camel@localhost> Hi, I've been meaning to do this since forever: here's a very humble Emacs mode for Bro policy files. Note: I have no clue of how to program Emacs, so beware ... http://www.icir.org/twiki/bin/view/Bro/EmacsBroMode Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From mike.muratet at torchtechnologies.com Wed Aug 3 12:34:30 2005 From: mike.muratet at torchtechnologies.com (Mike Muratet) Date: Wed, 3 Aug 2005 14:34:30 -0500 Subject: [Bro] is_tcp_port Message-ID: <007a01c59862$5ef3cc60$5501a8c0@muratet> Hello Again I was trying to figure out the tcp/udp flag in conn.bro:record_connection from is_tcp_port and I got as far as bro.bif before I got lost. I was looking at the iana.org site and it looks like it's a pretty degenerate test, i.e., all the ports can be either. How does bro determine tcp vs udp? thanks Mike From christian at whoop.org Wed Aug 3 13:30:21 2005 From: christian at whoop.org (Christian Kreibich) Date: Wed, 03 Aug 2005 21:30:21 +0100 Subject: [Bro] conn.bro In-Reply-To: <004f01c5983c$f2104760$5501a8c0@muratet> References: <004f01c5983c$f2104760$5501a8c0@muratet> Message-ID: <1123101021.7525.209.camel@localhost> Hi Mike, On Wed, 2005-08-03 at 10:06 -0500, Mike Muratet wrote: > Greetings > > I have been doing some experiments with broccoli (thanks again, Christian) > and I'm trying to understand the differences between what I see in real time > from broccoli vis a vis some earlier work I did using bro -r a_tcpdump_file > conn.bro. Looking at the conn.bro policy, it looks like it records a > connection record for the events connection_attempt, _partial_close, > _finished, _half_finished, _rejected, and _reset, using > record_connection(connection_record, string). uhmmm is this using a recent Bro? On 0.9x record_connection() is only triggered when a connection is expired, that is, from event connection_state_remove(). Also, its signature is function record_connection(f: file, c: connection) > The string parameter denotes > the type of event, but this does not get printed. It does record a state as > part of the connection record which gives you an idea of what the event was > all about. Is my interpretation correct? Yes, record_connection() prints a summary of the connection's lifetime, including a short summary of how Bro thinks it got created, torn down, etc. See http://www.bro-ids.org/Bro-reference-manual/Connection-summaries.html Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From christian at whoop.org Wed Aug 3 13:41:26 2005 From: christian at whoop.org (Christian Kreibich) Date: Wed, 03 Aug 2005 21:41:26 +0100 Subject: [Bro] is_tcp_port In-Reply-To: <007a01c59862$5ef3cc60$5501a8c0@muratet> References: <007a01c59862$5ef3cc60$5501a8c0@muratet> Message-ID: <1123101687.7525.219.camel@localhost> On Wed, 2005-08-03 at 14:34 -0500, Mike Muratet wrote: > Hello Again > > I was trying to figure out the tcp/udp flag in conn.bro:record_connection > from is_tcp_port and I got as far as bro.bif before I got lost. The functions listed in .bif files are Built-In Functions, i.e., functions implemented in the core. To find such a function's implementation, go to the src directory, and find the corresponding .bif file, in this case bro.bif, which shows: function is_tcp_port%(p: portval%): bool %{ return new Val(p->IsTCP(), TYPE_BOOL); %} Next you need to know that a portval in the policy is mapped to a PortVal object in the core, so the implementation of PortVal::IsTCP() is found in Val.cc. > I was > looking at the iana.org site and it looks like it's a pretty degenerate > test, i.e., all the ports can be either. Yeah sure, both UDP and TCP can have ports 0-65535. > How does bro determine tcp vs udp? >From looking at the IP header of the sniffed packets, which tell Bro what protocol is at the transport layer. Check NetSessions::DoNextPacket around line 436 and Stevens' TCP/IP Illustrated Vol 1 for details. Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From christian at whoop.org Wed Aug 3 13:49:11 2005 From: christian at whoop.org (Christian Kreibich) Date: Wed, 03 Aug 2005 21:49:11 +0100 Subject: [Bro] conn.bro In-Reply-To: <008b01c5986b$27659ae0$5501a8c0@muratet> References: <004f01c5983c$f2104760$5501a8c0@muratet> <1123101021.7525.209.camel@localhost> <008b01c5986b$27659ae0$5501a8c0@muratet> Message-ID: <1123102151.7525.224.camel@localhost> On Wed, 2005-08-03 at 15:37 -0500, Mike Muratet wrote: > > > uhmmm is this using a recent Bro? On 0.9x record_connection() is only > > triggered when a connection is expired, that is, from event > > connection_state_remove(). Also, its signature is > > > > No, it was 0.8. That was the verison I used to process all the data I had > collected. I am using 0.9a as the server for broccoli, though. I am trying > to reproduce exactly the fields from the bro processing of the tcpdump file > in the broccoli interface. I think I've got it for the most part, although I > am still chasing the basis for the is_tcp call. Ah I see -- you want the equivalent of record_connection() in your Broccoli client. I'd recommend modeling your code directly after 0.8's record_connection()'s implementation. On the Broccoli end, once you obtain a port as a pointer to a BroPort structure, you can tell the transport layer type by looking at the port_proto member. That should be all you need? Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From casado at cs.stanford.edu Wed Aug 3 13:52:42 2005 From: casado at cs.stanford.edu (Martin Casado) Date: Wed, 03 Aug 2005 13:52:42 -0700 Subject: [Bro] .bro VIM syntax file Message-ID: <42F12E9A.7050403@cs.stanford.edu> Not to be outdone .. I'll match Christian's Emacs Mode with a bro-script VIM syntax file. And, similar Christian .. no hablo 'Vim Script' so expect idiosyncrasies. http://www.icir.org/twiki/bin/view/Bro/BroVimSyntaxFile Cheers, Martin From vern at icir.org Wed Aug 3 21:15:34 2005 From: vern at icir.org (Vern Paxson) Date: Wed, 03 Aug 2005 21:15:34 -0700 Subject: [Bro] is_tcp_port In-Reply-To: Your message of Wed, 03 Aug 2005 14:34:30 CDT. Message-ID: <200508040415.j744FYsw006162@jaguar.icir.org> > I got lost. I was > looking at the iana.org site and it looks like it's a pretty degenerate > test, i.e., all the ports can be either. How does bro determine tcp vs udp? Internally, it sets bit 17 for UDP ports and bit 18 for ICMP "ports". See UDP_PORT_MASK and ICMP_PORT_MASK in Val.h. Vern From koukis at ics.forth.gr Mon Aug 8 09:29:55 2005 From: koukis at ics.forth.gr (Dimitris Koukis) Date: Mon, 08 Aug 2005 19:29:55 +0300 Subject: [Bro] Anonymization Message-ID: <42F78883.5090406@ics.forth.gr> Is there anyway to anonymize only IP addresses using Bro? Using just the policy described anon.bro, no output is produced. This is due to the fact that no payload is kept so no packets are produced? If i include anon.bro in http-rewriter everything is ok, IP addresses get anonymized. Thanx -- Dimitris Koukis From casado at cs.stanford.edu Mon Aug 8 14:55:55 2005 From: casado at cs.stanford.edu (Martin Casado) Date: Mon, 08 Aug 2005 14:55:55 -0700 Subject: [Bro] String whitespace strip method Message-ID: <42F7D4EB.90909@cs.stanford.edu> I've written a simple method for stripping whitespace from the begining and end of strings (same thing as python's strip() or java's trim()). It has come in pretty useful for application level string handling. Would it be possible to add it to the code base? Also, is there any plan to modulerize the builtins? (e.g. at load builtins.string), as the project grows, bro.bif is going to get (further) out of control. thanks :) .martin function strip%(str: string%): string %{ const char* s = str->CheckString(); int n = strlen(s) + 1; char* strip_s = new char[n]; while(isspace(*s)) { s++; } strncpy(strip_s, s, n); s = strip_s; char* e = (char*)(s + (strlen(s) - 1)); while(isspace(*e)) { e--; } *(e+1) = 0; return new StringVal(new BroString(1, byte_vec(s), (e-s)+1)); %} From christian at whoop.org Mon Aug 8 17:54:48 2005 From: christian at whoop.org (Christian Kreibich) Date: Tue, 09 Aug 2005 01:54:48 +0100 Subject: [Bro] String whitespace strip method In-Reply-To: <42F7D4EB.90909@cs.stanford.edu> References: <42F7D4EB.90909@cs.stanford.edu> Message-ID: <1123548888.7378.56.camel@localhost> Hi Martin, On Mon, 2005-08-08 at 14:55 -0700, Martin Casado wrote: > > Also, is there any plan to modulerize the builtins? (e.g. at load > builtins.string), as the project > grows, bro.bif is going to get (further) out of control. fyi I've started to do this in my tree. There are no separate namespaces so far, but at least things have moved from one humongous file into multiple, more manageable ones. I've also started to make the function naming more uniform. I should probably push a bunch of changes to Vern sometime soon. Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From adayadil.thomas at gmail.com Tue Aug 9 13:24:13 2005 From: adayadil.thomas at gmail.com (Adayadil Thomas) Date: Tue, 9 Aug 2005 16:24:13 -0400 Subject: [Bro] TCP Sack processing Message-ID: Greetings. Does TCP stream reassembly algorithm need TCP SACK processing for completeness ? Does bro handle this ? Any pointers is appreciated. Thanks Thomas From vern at icir.org Tue Aug 9 13:32:49 2005 From: vern at icir.org (Vern Paxson) Date: Tue, 09 Aug 2005 13:32:49 -0700 Subject: [Bro] TCP Sack processing In-Reply-To: Your message of Tue, 09 Aug 2005 16:24:13 EDT. Message-ID: <200508092032.j79KWnlQ068258@jaguar.icir.org> > Does TCP stream reassembly algorithm need TCP SACK processing for completeness ? No. SACKs are advisory, not guarantees, and can be ignored for purposes of stream reassembly. Vern From adayadil.thomas at gmail.com Tue Aug 9 13:53:23 2005 From: adayadil.thomas at gmail.com (Adayadil Thomas) Date: Tue, 9 Aug 2005 16:53:23 -0400 Subject: [Bro] TCP Sack processing In-Reply-To: <200508092032.j79KWnlQ068258@jaguar.icir.org> References: <200508092032.j79KWnlQ068258@jaguar.icir.org> Message-ID: Thanks for the reply. >>SACKs are advisory, not guarantees Could you be a little more elaborate on that. or point me to some rfc or other doc ? Thanks a lot On 8/9/05, Vern Paxson wrote: > > Does TCP stream reassembly algorithm need TCP SACK processing for completeness ? > > No. SACKs are advisory, not guarantees, and can be ignored for purposes > of stream reassembly. > > Vern > From vern at icir.org Tue Aug 9 14:00:14 2005 From: vern at icir.org (Vern Paxson) Date: Tue, 09 Aug 2005 14:00:14 -0700 Subject: [Bro] TCP Sack processing In-Reply-To: Your message of Tue, 09 Aug 2005 16:53:23 EDT. Message-ID: <200508092100.j79L0E3Y068969@jaguar.icir.org> > >>SACKs are advisory, not guarantees > Could you be a little more elaborate on that. or point me to some rfc > or other doc ? From vern at icir.org Fri Aug 12 00:57:16 2005 From: vern at icir.org (Vern Paxson) Date: Fri, 12 Aug 2005 00:57:16 -0700 Subject: [Bro] Anonymization In-Reply-To: Your message of Mon, 08 Aug 2005 19:29:55 +0300. Message-ID: <200508120757.j7C7vG9x045947@jaguar.icir.org> > Is there anyway to anonymize only IP addresses using Bro? Basically, no. (Ruoming, please correct me if I'm wrong.) Address anonymization only happens as a part of trace transformation, So you could write transformation scripts that don't change anything other than addresses, but this will only work for those protocols for which the event engine supports transformation. That said, Ruoming is close to releasing a separate trace anonymization tool (tcpmkpub) that will support this. Vern From adayadil.thomas at gmail.com Thu Aug 18 14:36:08 2005 From: adayadil.thomas at gmail.com (Adayadil Thomas) Date: Thu, 18 Aug 2005 17:36:08 -0400 Subject: [Bro] Intrusion prevention Message-ID: Greetings. Is there any plan to make Bro run in 'inline' mode as an IPS ? Thanks From chema at cs.berkeley.edu Thu Aug 18 16:07:59 2005 From: chema at cs.berkeley.edu (Jose M. Gonzalez) Date: Thu, 18 Aug 2005 16:07:59 -0700 Subject: [Bro] Intrusion prevention In-Reply-To: References: Message-ID: <20050818230759.GA17410@cs.berkeley.edu> As part of my thesis work, I've instrumented Bro to work as IPS. (It just requires injecting back traffic after having analyzed it.) -Chema Adayadil Thomas wrote: > Greetings. > > Is there any plan to make Bro run in 'inline' mode as an IPS ? > > Thanks > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From hhoffman at ip-solutions.net Thu Aug 18 23:30:25 2005 From: hhoffman at ip-solutions.net (Harry Hoffman) Date: Fri, 19 Aug 2005 02:30:25 -0400 Subject: [Bro] Intrusion prevention In-Reply-To: References: Message-ID: <43057C81.3000706@ip-solutions.net> I don't really know what the plans are but I find that bro runs better as a application logger rather then something like snort. This is just my $0.02 so plz take it as such :-) Cheers, Harry Adayadil Thomas wrote: > Greetings. > > Is there any plan to make Bro run in 'inline' mode as an IPS ? > > Thanks > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From dgtized at gmail.com Fri Aug 19 11:34:11 2005 From: dgtized at gmail.com (Charles Comstock) Date: Fri, 19 Aug 2005 13:34:11 -0500 Subject: [Bro] Bro install problems on linux In-Reply-To: <3582ae12050818123045ed6bc4@mail.gmail.com> References: <3582ae12050818123045ed6bc4@mail.gmail.com> Message-ID: <3582ae12050819113410476b0e@mail.gmail.com> I recently installed bro on a 64bit Fedora Core 3 box. I'm using bro-0.9a9. Unfortuneatly this is my first time trying to setup bro, and compiling and installing is about as far as I have been able to get. The bro_config file doesn't seem to autodetect the networks I'm listening to. If I use the default configuration it segfaults, the only way to get anything out of it is to comment out the @load tcp from the default host configuration. Trying @load udp instead just to get something also causes a segfault. Anyone have any ideas on this? Charles Comstock From manuel.crotti at ing.unibs.it Mon Aug 22 00:52:20 2005 From: manuel.crotti at ing.unibs.it (Manuel Crotti) Date: Mon, 22 Aug 2005 09:52:20 +0200 Subject: [Bro] protocol recognition Message-ID: Hi all, is bro capable to make automatic layer-7 protocol recognition from a dump? (e.g. I have a dump of a router with several protocols running on non standard port and I want to know which protocol runs on which port) TnX in advance Manuel. From manuel.crotti at ing.unibs.it Tue Aug 23 03:47:14 2005 From: manuel.crotti at ing.unibs.it (Manuel Crotti) Date: Tue, 23 Aug 2005 12:47:14 +0200 Subject: [Bro] reliable off-line protocol detection Message-ID: <2962C5F8-A353-45FA-90CE-C9B31E5FA153@ing.unibs.it> Ciao, I'm trying to make protocol detection on non-standart ports. I set-up a apaci server on 777 and a mail server (postfix) on 778. I captured packets with tcpdump. I parsed dumps with a "bro -C -r protocolName.dump backdoor" those are the "backdoor.log" results: http.dump > 1124720330.438091 10.20.188.212/32770 > 10.20.10.34/777 http-sig > 1124720338.627503 10.20.188.212/32773 > 10.20.10.34/777 http-sig > 1124720425.113738 10.20.188.212/32784 > 10.20.10.34/777 http-sig smtp.dump > 1124785239.632272 127.0.0.1/56034 > 127.0.0.1/778 ftp-sig > 1124785306.080354 127.0.0.1/56037 > 127.0.0.1/778 ftp-sig > 1124785591.602025 127.0.0.1/56048 > 127.0.0.1/778 ftp-sig > 1124785606.143460 127.0.0.1/56050 > 127.0.0.1/778 ftp-sig WHY? ( ?1992 Annie Lennox) Best regards, Manuel. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20050823/c2ce7f68/attachment.html From christian at whoop.org Tue Aug 23 09:42:50 2005 From: christian at whoop.org (Christian Kreibich) Date: Tue, 23 Aug 2005 17:42:50 +0100 Subject: [Bro] Bro install problems on linux In-Reply-To: <3582ae12050819113410476b0e@mail.gmail.com> References: <3582ae12050818123045ed6bc4@mail.gmail.com> <3582ae12050819113410476b0e@mail.gmail.com> Message-ID: <1124815371.13083.16.camel@localhost> Hi Charles, On Fri, 2005-08-19 at 13:34 -0500, Charles Comstock wrote: > I recently installed bro on a 64bit Fedora Core 3 box. I'm using > bro-0.9a9. Unfortuneatly this is my first time trying to setup bro, > and compiling and installing is about as far as I have been able to > get. The bro_config file doesn't seem to autodetect the networks I'm > listening to. If I use the default configuration it segfaults, the > only way to get anything out of it is to comment out the @load tcp > from the default host configuration. Trying @load udp instead just to > get something also causes a segfault. Anyone have any ideas on this? do the segfaults also occur if you start Bro manually, say with bro -i eth0 conn If so, could you post a gdb backtrace to the list please? Thanks, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From vern at icir.org Tue Aug 23 11:01:18 2005 From: vern at icir.org (Vern Paxson) Date: Tue, 23 Aug 2005 11:01:18 -0700 Subject: [Bro] Small bug in TCP_Rewriter In-Reply-To: Your message of Fri, 15 Jul 2005 14:01:50 PDT. Message-ID: <200508231801.j7NI1InL011574@jaguar.icir.org> [catching up on some old threads - apologies for the delay] > perhaps .. > > if(!next_packet->AppendData(data, left)) > { ASSERT(0); } Thanks, that looks like the correct fix. I've added this. By the way, does anyone know whether ASSERT() is guaranteed to be equivalent to a statement, so that the {}'s above can be safely omitted? > Plz. let me know if this isn't the appropriate forum for submitting bugs. This is a fine forum for doing so (with again my apologies for the delay in following up). Vern From vern at icir.org Tue Aug 23 11:01:31 2005 From: vern at icir.org (Vern Paxson) Date: Tue, 23 Aug 2005 11:01:31 -0700 Subject: [Bro] re: Alternative from addresses in emails (diff output, and fixed) In-Reply-To: Your message of Wed, 27 Jul 2005 19:18:23 PDT. Message-ID: <200508231801.j7NI1Veh011585@jaguar.icir.org> Thanks for the patches, I've integrated these for the next release. Vern From vern at icir.org Tue Aug 23 11:01:35 2005 From: vern at icir.org (Vern Paxson) Date: Tue, 23 Aug 2005 11:01:35 -0700 Subject: [Bro] String whitespace strip method In-Reply-To: Your message of Mon, 08 Aug 2005 14:55:55 PDT. Message-ID: <200508231801.j7NI1ZWt011593@jaguar.icir.org> > I've written a simple method for stripping whitespace from the begining > and end of strings Thanks for the contribution. I'll let Christian to incorporate this in the set of string updates he'll be sending me. > Also, is there any plan to modulerize the builtins? (e.g. at load > builtins.string), as the project > grows, bro.bif is going to get (further) out of control. Do you mean modularize in terms of the source files (in which case this shouldn't be visible to the user, i.e., they wouldn't specify @load builtins.string), or at the script level? The latter requires some careful thought regarding partitioning of the built-ins and possibly syntax for accessing them in an easy/consistent/scoped fashion. (Which is a long-winded way of saying, no, there's no current plans, other than Christian's regularizing the naming of string functions and adding some string operators.) Vern From vern at icir.org Tue Aug 23 11:01:39 2005 From: vern at icir.org (Vern Paxson) Date: Tue, 23 Aug 2005 11:01:39 -0700 Subject: [Bro] Intrusion prevention In-Reply-To: Your message of Thu, 18 Aug 2005 17:36:08 EDT. Message-ID: <200508231801.j7NI1dIO011601@jaguar.icir.org> > Is there any plan to make Bro run in 'inline' mode as an IPS ? We're doing this as part of a DOE-sponsored project for coupling Bro with custom hardware (per Chema's comments about his thesis work). The resulting design will support Bro running as a stand-alone IPS (i.e., without needing the custom hardware), though that's not our near-term focus. Vern From vern at icir.org Tue Aug 23 11:01:43 2005 From: vern at icir.org (Vern Paxson) Date: Tue, 23 Aug 2005 11:01:43 -0700 Subject: [Bro] Bro install problems on linux In-Reply-To: Your message of Fri, 19 Aug 2005 13:34:11 CDT. Message-ID: <200508231801.j7NI1hsE011609@jaguar.icir.org> > If I use the default configuration it segfaults As is always the case when reporting crashes, it is highly helpful if you can please include a debugger traceback showing the problem. Otherwise we're pretty much stuck with trying to fix it if it doesn't happen to reproduce in our development environments. Vern From vern at icir.org Tue Aug 23 11:01:48 2005 From: vern at icir.org (Vern Paxson) Date: Tue, 23 Aug 2005 11:01:48 -0700 Subject: [Bro] protocol recognition In-Reply-To: Your message of Mon, 22 Aug 2005 09:52:20 +0200. Message-ID: <200508231801.j7NI1m6j011617@jaguar.icir.org> > is bro capable to make automatic layer-7 protocol recognition from a > dump? It's backdoor detectors do this (with varying degrees of accuracy), but it's not yet able to then do the next step and switch the processing of the connection over to an analyzer for the discovered protocol. > (e.g. I have a dump of a router with several protocols running on non > standard port and I want to know which protocol runs on which port) It currently detects SSH, Gnutella, HTTP and HTTP proxies quite accurately, SMTP, KaZaA, Napster and FTP/IMAP/POP (it lumps these together, but unfortunately just terms "FTP"; and sometimes the FTP detector triggers on SMTP connections) fairly well, and Telnet/Rlogin with limited success. There are also new IRC and Gaobot detectors, which are still being tuned. Vern From christian at whoop.org Tue Aug 23 11:20:41 2005 From: christian at whoop.org (Christian Kreibich) Date: Tue, 23 Aug 2005 19:20:41 +0100 Subject: [Bro] Small bug in TCP_Rewriter In-Reply-To: <200508231801.j7NI1InL011574@jaguar.icir.org> References: <200508231801.j7NI1InL011574@jaguar.icir.org> Message-ID: <1124821242.13083.53.camel@localhost> On Tue, 2005-08-23 at 11:01 -0700, Vern Paxson wrote: > [catching up on some old threads - apologies for the delay] > > > perhaps .. > > > > if(!next_packet->AppendData(data, left)) > > { ASSERT(0); } > > Thanks, that looks like the correct fix. I've added this. > > By the way, does anyone know whether ASSERT() is guaranteed to be equivalent > to a statement, so that the {}'s above can be safely omitted? Mhmm avoid the problem by saying int success = next_packet->AppendData(data, left); ASSERT(success); ? Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From vern at icir.org Tue Aug 23 11:21:27 2005 From: vern at icir.org (Vern Paxson) Date: Tue, 23 Aug 2005 11:21:27 -0700 Subject: [Bro] Small bug in TCP_Rewriter In-Reply-To: Your message of Tue, 23 Aug 2005 19:20:41 BST. Message-ID: <200508231821.j7NILRSO012577@jaguar.icir.org> > Mhmm avoid the problem by saying > > int success = next_packet->AppendData(data, left); > ASSERT(success); Sure. However, there's a chance that the standard (if indeed ASSERT is part of the standard) already addresses this issue, in which case I'd like to go with the slightly more streamlined syntax of omitting the {}'s. (I've already done it without {}'s for integrating the change and g++ is happy, but that of course doesn't mean it's standard-compliant. I [idly] wonder how many folks build Bro with anything other than g++, anyway.) Vern From casado at cs.stanford.edu Tue Aug 23 11:40:55 2005 From: casado at cs.stanford.edu (Martin Casado) Date: Tue, 23 Aug 2005 11:40:55 -0700 Subject: [Bro] Small bug in TCP_Rewriter In-Reply-To: <1124821242.13083.53.camel@localhost> References: <200508231801.j7NI1InL011574@jaguar.icir.org> <1124821242.13083.53.camel@localhost> Message-ID: <430B6DB7.2050305@cs.stanford.edu> >>[catching up on some old threads - apologies for the delay] >> >> >> >>>perhaps .. >>> >>> if(!next_packet->AppendData(data, left)) >>> { ASSERT(0); } >>> >>> >>Thanks, that looks like the correct fix. I've added this. >> >>By the way, does anyone know whether ASSERT() is guaranteed to be equivalent >>to a statement, so that the {}'s above can be safely omitted? >> >> > > > if(statement) ; Is valid C so the braces shouldn't be necessary. I don't see anywhere in the standard that assert is guaranteed to be a statement ... .m >Mhmm avoid the problem by saying > > int success = next_packet->AppendData(data, left); > ASSERT(success); >? > >Cheers, >Christian. > > From lamont at gurulabs.com Tue Aug 23 12:38:45 2005 From: lamont at gurulabs.com (Lamont R. Peterson) Date: Tue, 23 Aug 2005 13:38:45 -0600 Subject: [Bro] Small bug in TCP_Rewriter In-Reply-To: <200508231821.j7NILRSO012577@jaguar.icir.org> References: <200508231821.j7NILRSO012577@jaguar.icir.org> Message-ID: <200508231338.50547.lamont@gurulabs.com> On Tuesday 23 August 2005 12:21pm, Vern Paxson wrote: > > Mhmm avoid the problem by saying > > > > int success = next_packet->AppendData(data, left); > > ASSERT(success); It is always frustrating to see good programmers & software engineers using bad-lazy practices (there are good-lazy practices too). IMNHO (I'll be honest about that), omitting structure cues, whether or not they are strictly necessary, is wrong. Readability. Consistency. Folks, these are important. Many, many bugs I have had to deal with in other people's code, if not most, have boiled down to just such poor, inconsistent, "bad-lazy" practices. For example, omitting the braces and then coming back and adding more code but forgetting to add them. That one happens a lot, even with the brightest among us. If I have to stop scanning through code and read back to figure out for sure that there is a "block" there, then that wastes time. Sure, if I do it enough I get pretty good at it, but it still takes additional time. Contrast that with braces everywhere any type of block exists (whether empty, with just one statement or many)...the code is *significantly* easier to read, not to mention that the intention of the coder who wrote it is much more clear. Please, don't be part of the bad-lazy crown. Put the braces in. It does not negatively impact compile time, so there should be no excuses. BTW: Easier to read, structured, formatted code is significantly easier to audit. > Sure. However, there's a chance that the standard (if indeed ASSERT is > part of the standard) already addresses this issue, in which case I'd like > to go with the slightly more streamlined syntax of omitting the {}'s. I hope that you will consider (and then do it ;) changing those back to include braces { } wherever a block structure exists and they have been omitted. > (I've already done it without {}'s for integrating the change and g++ > is happy, but that of course doesn't mean it's standard-compliant. > I [idly] wonder how many folks build Bro with anything other than g++, > anyway.) Not me :). -- Lamont R. Peterson Senior Instructor Guru Labs, L.C. [ http://www.GuruLabs.com/ ] -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20050823/6dfdd515/attachment.bin From vern at icir.org Tue Aug 23 12:43:39 2005 From: vern at icir.org (Vern Paxson) Date: Tue, 23 Aug 2005 12:43:39 -0700 Subject: coding style (Re: [Bro] Small bug in TCP_Rewriter) In-Reply-To: Your message of Tue, 23 Aug 2005 13:38:45 MDT. Message-ID: <200508231943.j7NJhdUW017092@jaguar.icir.org> I agree over the importance of *consistency* of coding style. Let's agree to disagree over which in particular is the One True Style and please leave it at that. Vern From lamont at gurulabs.com Tue Aug 23 13:09:02 2005 From: lamont at gurulabs.com (Lamont R. Peterson) Date: Tue, 23 Aug 2005 14:09:02 -0600 Subject: coding style (Re: [Bro] Small bug in TCP_Rewriter) In-Reply-To: <200508231943.j7NJhdUW017092@jaguar.icir.org> References: <200508231943.j7NJhdUW017092@jaguar.icir.org> Message-ID: <200508231409.07166.lamont@gurulabs.com> On Tuesday 23 August 2005 01:43pm, Vern Paxson wrote: > I agree over the importance of *consistency* of coding style. > > Let's agree to disagree over which in particular is the One True Style > and please leave it at that. I have no problem with that and I apologize if I came across as trying to "suggest policy" to the Bro development team. I merely wanted to point out that my reasons come from long, hard-won experience. As long as a (security tool, in particular) project has a style and enforces it, that's what matters. Consistency makes everyone's life much easier when they go through the code. -- Lamont R. Peterson Senior Instructor Guru Labs, L.C. [ http://www.GuruLabs.com/ ] -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20050823/1d2a2d7e/attachment.bin From adayadil.thomas at gmail.com Tue Aug 23 13:38:07 2005 From: adayadil.thomas at gmail.com (Adayadil Thomas) Date: Tue, 23 Aug 2005 16:38:07 -0400 Subject: [Bro] Intrusion prevention In-Reply-To: <200508231801.j7NI1dIO011601@jaguar.icir.org> References: <200508231801.j7NI1dIO011601@jaguar.icir.org> Message-ID: Will it be interfacing with a firewall like iptables (like snort-inline does) ? Thanks On 8/23/05, Vern Paxson wrote: > > Is there any plan to make Bro run in 'inline' mode as an IPS ? > > We're doing this as part of a DOE-sponsored project for coupling Bro with > custom hardware (per Chema's comments about his thesis work). The resulting > design will support Bro running as a stand-alone IPS (i.e., without needing > the custom hardware), though that's not our near-term focus. > > Vern > From dgtized at gmail.com Tue Aug 23 13:52:13 2005 From: dgtized at gmail.com (Charles Comstock) Date: Tue, 23 Aug 2005 15:52:13 -0500 Subject: [Bro] Bro install problems on linux In-Reply-To: <200508231801.j7NI1hsE011609@jaguar.icir.org> References: <200508231801.j7NI1hsE011609@jaguar.icir.org> Message-ID: <3582ae12050823135269189023@mail.gmail.com> I tried compiling with CFLAGS='-g' and CXXFLAGS='-g' and while it included the -g in all the compile lines when it went by I still don't seem to get additional info in the gdb backtrace. However during compile I get about 30 or 40 warnings per file about mismatched sizes for ints. So I'm kinda guessing it might be related to 64bit compatability if it's not just something i am doing wrong. Whatever the case gdb seems unable to give me any more info then that one line of output. If anyone has a better suggestion on how to get debugger traceback with more info I will be happy to post that output. Thanks, Charles comstock Linux bb4.arl.wustl.edu 2.6.9-1.667smp #1 SMP Tue Nov 2 15:09:11 EST 2004 x86_64 x86_64 x86_64 GNU/Linux root at bb4:~/bro-0.9a9 [15:42:59]#] src/bro -i eth3 bb4.arl.wustl.edu.bro Segmentation fault root at bb4:~/bro-0.9a9 [15:43:14]#] gdb src/bro GNU gdb Red Hat Linux (6.1post-1.20040607.43rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1". (gdb) run -i eth3 bb4.arl.wustl.edu.bro Starting program: /root/bro-0.9a9/src/bro -i eth3 bb4.arl.wustl.edu.bro Program received signal SIGSEGV, Segmentation fault. 0x0000000000528da1 in __ns_initparse () On 8/23/05, Vern Paxson wrote: > > If I use the default configuration it segfaults > > As is always the case when reporting crashes, it is highly helpful if you > can please include a debugger traceback showing the problem. Otherwise > we're pretty much stuck with trying to fix it if it doesn't happen to > reproduce in our development environments. > > Vern > From chema at cs.berkeley.edu Tue Aug 23 15:03:35 2005 From: chema at cs.berkeley.edu (Jose M. Gonzalez) Date: Tue, 23 Aug 2005 15:03:35 -0700 Subject: [Bro] Intrusion prevention In-Reply-To: References: <200508231801.j7NI1dIO011601@jaguar.icir.org> Message-ID: <20050823220335.GA6129@cs.berkeley.edu> Adayadil Thomas wrote: > Will it be interfacing with a firewall like iptables (like snort-inline > does) ? Sort of. I'm writing the details for my thesis. Will let you know once it's finished. -Chema From christian at whoop.org Thu Aug 25 08:06:38 2005 From: christian at whoop.org (Christian Kreibich) Date: Thu, 25 Aug 2005 16:06:38 +0100 Subject: [Bro] Bro install problems on linux In-Reply-To: <3582ae12050823135269189023@mail.gmail.com> References: <200508231801.j7NI1hsE011609@jaguar.icir.org> <3582ae12050823135269189023@mail.gmail.com> Message-ID: <1124982399.16503.99.camel@localhost> On Tue, 2005-08-23 at 15:52 -0500, Charles Comstock wrote: > I tried compiling with CFLAGS='-g' and CXXFLAGS='-g' and while it > included the -g in all the compile lines when it went by I still don't > seem to get additional info in the gdb backtrace. However during > compile I get about 30 or 40 warnings per file about mismatched sizes > for ints. Could you post a few examples of those (or alternatively, send me the full output off-line)? > So I'm kinda guessing it might be related to 64bit > compatability if it's not just something i am doing wrong. Whatever > the case gdb seems unable to give me any more info then that one line > of output. If anyone has a better suggestion on how to get debugger > traceback with more info I will be happy to post that output. What happens when you type "bt" at the gdb prompt, when gdb shows you the message you're posting below? > Thanks, > Charles comstock > > Linux bb4.arl.wustl.edu 2.6.9-1.667smp #1 SMP Tue Nov 2 15:09:11 EST > 2004 x86_64 x86_64 x86_64 GNU/Linux > root at bb4:~/bro-0.9a9 [15:42:59]#] src/bro -i eth3 bb4.arl.wustl.edu.bro > Segmentation fault > root at bb4:~/bro-0.9a9 [15:43:14]#] gdb src/bro > GNU gdb Red Hat Linux (6.1post-1.20040607.43rh) > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu"...Using host > libthread_db library "/lib64/tls/libthread_db.so.1". > > (gdb) run -i eth3 bb4.arl.wustl.edu.bro > Starting program: /root/bro-0.9a9/src/bro -i eth3 bb4.arl.wustl.edu.bro > > Program received signal SIGSEGV, Segmentation fault. > 0x0000000000528da1 in __ns_initparse () Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From dgtized at gmail.com Thu Aug 25 15:04:49 2005 From: dgtized at gmail.com (Charles Comstock) Date: Thu, 25 Aug 2005 17:04:49 -0500 Subject: [Bro] Bro install problems on linux In-Reply-To: <1124982399.16503.99.camel@localhost> References: <200508231801.j7NI1hsE011609@jaguar.icir.org> <3582ae12050823135269189023@mail.gmail.com> <1124982399.16503.99.camel@localhost> Message-ID: <3582ae1205082515047c624e4@mail.gmail.com> On 8/25/05, Christian Kreibich wrote: > On Tue, 2005-08-23 at 15:52 -0500, Charles Comstock wrote: > > I tried compiling with CFLAGS='-g' and CXXFLAGS='-g' and while it > > included the -g in all the compile lines when it went by I still don't > > seem to get additional info in the gdb backtrace. However during > > compile I get about 30 or 40 warnings per file about mismatched sizes > > for ints. > > Could you post a few examples of those (or alternatively, send me the > full output off-line)? > Here is some example output, it's like this for alot of files though. g++ -DHAVE_CONFIG_H -I. -I. -I.. -I. -I../src -I. -I.. -Ilibedit -I../linux-include -I../aux/libpcap-0.7.2 -O -g -g -c -o main.o `test -f main.cc || echo './'`main.cc In file included from DNS_Mgr.h:28, from main.cc:42: EventHandler.h: In member function `void EventHandler::SourceIDList::insert(SourceID)': EventHandler.h:53: warning: cast to pointer from integer of different size EventHandler.h: In member function `void EventHandler::SourceIDList::sortedinsert(SourceID, int (*)(const void*, const void*))': EventHandler.h:53: warning: cast to pointer from integer of different size EventHandler.h: In member function `void EventHandler::SourceIDList::append(SourceID)': EventHandler.h:53: warning: cast to pointer from integer of different size EventHandler.h: In member function `SourceID EventHandler::SourceIDList::remove(SourceID)': EventHandler.h:53: warning: cast to pointer from integer of different size EventHandler.h:53: warning: cast from pointer to integer of different size EventHandler.h: In member function `SourceID EventHandler::SourceIDList::remove_nth(int)': EventHandler.h:53: warning: cast from pointer to integer of different size EventHandler.h: In member function `SourceID EventHandler::SourceIDList::get()': EventHandler.h:53: warning: cast from pointer to integer of different size EventHandler.h: In member function `SourceID EventHandler::SourceIDList::last()': EventHandler.h:53: warning: cast from pointer to integer of different size EventHandler.h: In member function `SourceID EventHandler::SourceIDList::replace(int, SourceID)': EventHandler.h:53: warning: cast to pointer from integer of different size EventHandler.h:53: warning: cast from pointer to integer of different size EventHandler.h: In member function `SourceID EventHandler::SourceIDList::is_member(SourceID) const': EventHandler.h:53: warning: cast to pointer from integer of different size EventHandler.h:53: warning: cast from pointer to integer of different size EventHandler.h: In member function `int EventHandler::SourceIDList::member_pos(SourceID) const': EventHandler.h:53: warning: cast to pointer from integer of different size EventHandler.h: In member function `SourceID EventHandler::SourceIDList::operator[](int) const': EventHandler.h:53: warning: cast from pointer to integer of different size > > So I'm kinda guessing it might be related to 64bit > > compatability if it's not just something i am doing wrong. Whatever > > the case gdb seems unable to give me any more info then that one line > > of output. If anyone has a better suggestion on how to get debugger > > traceback with more info I will be happy to post that output. > > What happens when you type "bt" at the gdb prompt, when gdb shows you > the message you're posting below? > When I print a bt I get the same line: 0x0000000000528da1 in __ns_initparse () I think by default gdb displays bt when the program exits abnormally. Is there a different flag then -g I should be feeding the compiler in order to get more backtrace info? Thanks, Charles Comstock From casado at cs.stanford.edu Fri Aug 26 10:01:40 2005 From: casado at cs.stanford.edu (Martin Casado) Date: Fri, 26 Aug 2005 10:01:40 -0700 Subject: [Bro] Bro install problems on linux In-Reply-To: <3582ae1205082515047c624e4@mail.gmail.com> References: <200508231801.j7NI1hsE011609@jaguar.icir.org> <3582ae12050823135269189023@mail.gmail.com> <1124982399.16503.99.camel@localhost> <3582ae1205082515047c624e4@mail.gmail.com> Message-ID: <430F4AF4.1060901@cs.stanford.edu> > > > >Here is some example output, it's like this for alot of files though. > >g++ -DHAVE_CONFIG_H -I. -I. -I.. -I. -I../src -I. -I.. -Ilibedit >-I../linux-include -I../aux/libpcap-0.7.2 -O -g -g -c -o main.o >`test -f main.cc || echo './'`main.cc >In file included from DNS_Mgr.h:28, > from main.cc:42: >EventHandler.h: In member function `void >EventHandler::SourceIDList::insert(SourceID)': >EventHandler.h:53: warning: cast to pointer from integer of different size >EventHandler.h: In member function `void >EventHandler::SourceIDList::sortedinsert(SourceID, int (*)(const >void*, const void*))': >EventHandler.h:53: warning: cast to pointer from integer of different size >EventHandler.h: In member function `void >EventHandler::SourceIDList::append(SourceID)': >EventHandler.h:53: warning: cast to pointer from integer of different size >EventHandler.h: In member function `SourceID > > > Unless I'm missing something, this does look broken. Here are the snippets from the post-processed translation unit of EventHandler.h: typedef void* ent; typedef uint32 SourceID; struct SourceIDList : BaseList { void insert(SourceID a) { BaseList::insert(ent(a)); } Needless to say, on a 64 bit architecture casting a 32bit into to a 64bit value is a mistake. I'm assuming this code was written before templates became practical in a performance sensitive environment. Perhaps its time for an upgrade? .m From vern at icir.org Fri Aug 26 15:21:24 2005 From: vern at icir.org (Vern Paxson) Date: Fri, 26 Aug 2005 15:21:24 -0700 Subject: [Bro] reliable off-line protocol detection In-Reply-To: Your message of Tue, 23 Aug 2005 12:47:14 +0200. Message-ID: <200508262221.j7QMLOaK066851@jaguar.icir.org> > smtp.dump > > > 1124785239.632272 127.0.0.1/56034 > 127.0.0.1/778 ftp-sig > > 1124785306.080354 127.0.0.1/56037 > 127.0.0.1/778 ftp-sig > > 1124785591.602025 127.0.0.1/56048 > 127.0.0.1/778 ftp-sig > > 1124785606.143460 127.0.0.1/56050 > 127.0.0.1/778 ftp-sig > > WHY? ( =A91992 Annie Lennox) The FTP backdoor detector isn't precise - it looks for initial 220 or 426 replies, which SMTP servers can generate too. Ideally, the SMTP detector would trigger first (based on seeing EHLO or HELO). If you have a simple trace that shows it's failing to do so, go ahead and send it to me and I'll see what's up. Vern From vern at icir.org Fri Aug 26 15:23:05 2005 From: vern at icir.org (Vern Paxson) Date: Fri, 26 Aug 2005 15:23:05 -0700 Subject: [Bro] Intrusion prevention In-Reply-To: Your message of Tue, 23 Aug 2005 16:38:07 EDT. Message-ID: <200508262223.j7QMN5cT066893@jaguar.icir.org> > Will it be interfacing with a firewall like iptables (like snort-inline does) ? It's a different API, and not directly suitable for use with something like iptables (our approach is quite fine-grained). Note, we already (and for a long time) run Bro in a reactive fashion, for which the policy script can drop hostile traffic. But this isn't the full power of an IPS since there's latency between discovering a problem and blocking a host, so damage can still occur. Vern From vern at icir.org Fri Aug 26 15:23:56 2005 From: vern at icir.org (Vern Paxson) Date: Fri, 26 Aug 2005 15:23:56 -0700 Subject: coding style (Re: [Bro] Small bug in TCP_Rewriter) In-Reply-To: Your message of Tue, 23 Aug 2005 14:09:02 MDT. Message-ID: <200508262223.j7QMNusO066947@jaguar.icir.org> > As long as a (security tool, in particular) project has a style and enforces > it, that's what matters. Consistency makes everyone's life much easier when > they go through the code. Agreed. I'm also these days not a big fan of Bro's current style, but like to keep it uniform. The students I work with like it even less :-), and in fact an item on our wish-list is to find a good pretty-printer so we can automate things to ensure consistency. (This will also be an opportunity to change the style somewhat.) Vern From vern at icir.org Fri Aug 26 15:32:18 2005 From: vern at icir.org (Vern Paxson) Date: Fri, 26 Aug 2005 15:32:18 -0700 Subject: [Bro] Bro install problems on linux In-Reply-To: Your message of Tue, 23 Aug 2005 15:52:13 CDT. Message-ID: <200508262232.j7QMWIGA067424@jaguar.icir.org> > So I'm kinda guessing it might be related to 64bit > compatability if it's not just something i am doing wrong. Hmmm, that could be. You might try contacting Martin Arlitt , whom I believe has Bro working on 64-bit Linux, though I think he may have had to make some hacks to do so. Vern From bchen at cs.ucf.edu Mon Aug 29 09:06:06 2005 From: bchen at cs.ucf.edu (bchen at cs.ucf.edu) Date: Mon, 29 Aug 2005 12:06:06 -0400 Subject: [Bro] successful_RPC_reply_to_invalid_request and ContentGap In-Reply-To: <200508271901.j7RJ1Fw3027217@fruitcake.ICSI.Berkeley.EDU> References: <200508271901.j7RJ1Fw3027217@fruitcake.ICSI.Berkeley.EDU> Message-ID: <20050829120606.g17s6if64go48wk4@mail.cs.ucf.edu> Hi everyone, I used Bro to read a trace file that was captured from a local network. In Bro's log files, the majority of log records are about successful_RPC_reply_to_invalid_request and ContentGap. What situations can cause these two alerts? thanks Bing From rpang at cs.princeton.edu Mon Aug 29 11:47:42 2005 From: rpang at cs.princeton.edu (Ruoming Pang) Date: Mon, 29 Aug 2005 14:47:42 -0400 Subject: [Bro] successful_RPC_reply_to_invalid_request and ContentGap In-Reply-To: <20050829120606.g17s6if64go48wk4@mail.cs.ucf.edu> References: <200508271901.j7RJ1Fw3027217@fruitcake.ICSI.Berkeley.EDU> <20050829120606.g17s6if64go48wk4@mail.cs.ucf.edu> Message-ID: <4313584E.20202@cs.princeton.edu> > Hi everyone, > I used Bro to read a trace file that was captured from a local > network. In > Bro's log files, the majority of log records are about > successful_RPC_reply_to_invalid_request and ContentGap. What > situations can > cause these two alerts? "successful_RPC_reply_to_invalid_request" is a deficiency to be fixed. Bro marks RPC requests that it does not understand as "invalid", and it gets bewildered when it sees the RPC server understands the requests. Please ignore this for now. "ContentGap" is reported when a TCP segment is not seen but ACK'ed. There can be two cases: (1) most likely, some packets are not captured in the trace file; (2) packets are captured on two interfaces and occasionally get out of order such that some ACKs appear before the corresponding segments. There's little Bro can do with (1), but for (2), Bro can reorder the packets in a small window by TCP sequence number instead of timestamps (redef packet_sort_window = 1 sec). I hope it helps. Ruoming From christian at whoop.org Mon Aug 29 16:38:32 2005 From: christian at whoop.org (Christian Kreibich) Date: Mon, 29 Aug 2005 16:38:32 -0700 Subject: [Bro] Bro install problems on linux In-Reply-To: <430F4AF4.1060901@cs.stanford.edu> References: <200508231801.j7NI1hsE011609@jaguar.icir.org> <3582ae12050823135269189023@mail.gmail.com> <1124982399.16503.99.camel@localhost> <3582ae1205082515047c624e4@mail.gmail.com> <430F4AF4.1060901@cs.stanford.edu> Message-ID: <1125358712.3203.89.camel@localhost> On Fri, 2005-08-26 at 10:01 -0700, Martin Casado wrote: > > > > > > > >Here is some example output, it's like this for alot of files though. > > > >g++ -DHAVE_CONFIG_H -I. -I. -I.. -I. -I../src -I. -I.. -Ilibedit > >-I../linux-include -I../aux/libpcap-0.7.2 -O -g -g -c -o main.o > >`test -f main.cc || echo './'`main.cc > >In file included from DNS_Mgr.h:28, > > from main.cc:42: > >EventHandler.h: In member function `void > >EventHandler::SourceIDList::insert(SourceID)': > >EventHandler.h:53: warning: cast to pointer from integer of different size > >EventHandler.h: In member function `void > >EventHandler::SourceIDList::sortedinsert(SourceID, int (*)(const > >void*, const void*))': > >EventHandler.h:53: warning: cast to pointer from integer of different size > >EventHandler.h: In member function `void > >EventHandler::SourceIDList::append(SourceID)': > >EventHandler.h:53: warning: cast to pointer from integer of different size > >EventHandler.h: In member function `SourceID Thanks for the output, Charles (and Martin Arlitt, who sent me output off-line). > Unless I'm missing something, this does look broken. Here are the > snippets from the post-processed translation unit > of EventHandler.h: > > typedef void* ent; > typedef uint32 SourceID; > struct SourceIDList : BaseList { void insert(SourceID a) { > BaseList::insert(ent(a)); } > > Needless to say, on a 64 bit architecture casting a 32bit into to a > 64bit value is a mistake. Yeah. The problem is the hard-coded difference between lists of instances of types (cf. List macros) and lists of pointers to instances (cf. PList macros). Here the former is used, which breaks on 64-bit architectures. Luckily List is used very rarely, I can only find declare(List,int) in CCL.h declare(List, SourceID) in EventHandler.h I presume it's somewhat tedious to change the code from List to PList since it'll involve rethinking allocation/deallocation of container elements, but it's the only solution I can think of. Mhm. Unless we throw out the macro containers and switch to STL ones. What's your take on this Vern? > I'm assuming this code was written before templates became practical in a > performance sensitive environment. Perhaps its time for an upgrade? Fwiw, I'd like to see the self-implemented containers go as well. Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From christian at whoop.org Mon Aug 29 16:50:30 2005 From: christian at whoop.org (Christian Kreibich) Date: Mon, 29 Aug 2005 16:50:30 -0700 Subject: [Bro] Bro install problems on linux In-Reply-To: <1125358712.3203.89.camel@localhost> References: <200508231801.j7NI1hsE011609@jaguar.icir.org> <3582ae12050823135269189023@mail.gmail.com> <1124982399.16503.99.camel@localhost> <3582ae1205082515047c624e4@mail.gmail.com> <430F4AF4.1060901@cs.stanford.edu> <1125358712.3203.89.camel@localhost> Message-ID: <1125359430.3203.96.camel@localhost> On Mon, 2005-08-29 at 16:38 -0700, Christian Kreibich wrote: > > I presume it's somewhat tedious to change the code from List to PList > since it'll involve rethinking allocation/deallocation of container > elements, but it's the only solution I can think of. Mhm. Actually, wouldn't it work if we changed typedef uint32 SourceID to typedef uint64 SourceID in util.h on architectures where pointers aren't 32bit? I'd still prefer to get rid of the macro containers but this might be a quicker fix. Well, except for any spots in the code that rely on SourceIDs being 32- bit (serialization maybe?), of course, and for the configure kung-fu needed. Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From sommer at in.tum.de Mon Aug 29 23:56:03 2005 From: sommer at in.tum.de (Robin Sommer) Date: Tue, 30 Aug 2005 08:56:03 +0200 Subject: [Bro] Bro install problems on linux In-Reply-To: <1125359430.3203.96.camel@localhost> <1125358712.3203.89.camel@localhost> References: <3582ae1205082515047c624e4@mail.gmail.com> <430F4AF4.1060901@cs.stanford.edu> <1125358712.3203.89.camel@localhost> <1125359430.3203.96.camel@localhost> <200508231801.j7NI1hsE011609@jaguar.icir.org> <3582ae12050823135269189023@mail.gmail.com> <1124982399.16503.99.camel@localhost> <3582ae1205082515047c624e4@mail.gmail.com> <430F4AF4.1060901@cs.stanford.edu> <1125358712.3203.89.camel@localhost> Message-ID: <20050830065603.GA11613@net.informatik.tu-muenchen.de> On Mon, Aug 29, 2005 at 16:38 -0700, Christian Kreibich wrote: > elements, but it's the only solution I can think of. Mhm. Unless we > throw out the macro containers and switch to STL ones. For the few occurenes of List this should be fine. Not sure if it makes sense to replace all macro containers. While I'd like to see that, too, it's appears to be a lot of work and may influence performance significantly (either positively or negatively, not sure). On Mon, Aug 29, 2005 at 16:50 -0700, Christian Kreibich wrote: > typedef uint64 SourceID Should be fine. SourceIDs are not serialized and they are rarely stored somewhere, i.e. they are not consuming a significant amount of memory. > configure kung-fu TransientID::ID already depends on being 64-bit and uses a long-long for its tyoe (plus a compile-time check whether long-long are indeed 64-bit). I remember that there was a problem with a nicer "kung-fu" but I don't recall what is was... Robin -- Robin Sommer * Room 01.08.055 * www.net.in.tum.de TU Muenchen * Phone (089) 289-18006 * sommer at in.tum.de From christian at whoop.org Tue Aug 30 17:17:45 2005 From: christian at whoop.org (Christian Kreibich) Date: Tue, 30 Aug 2005 17:17:45 -0700 Subject: [Bro] Bro install problems on linux In-Reply-To: <20050830065603.GA11613@net.informatik.tu-muenchen.de> References: <3582ae1205082515047c624e4@mail.gmail.com> <430F4AF4.1060901@cs.stanford.edu> <1125358712.3203.89.camel@localhost> <1125359430.3203.96.camel@localhost> <200508231801.j7NI1hsE011609@jaguar.icir.org> <3582ae12050823135269189023@mail.gmail.com> <1124982399.16503.99.camel@localhost> <3582ae1205082515047c624e4@mail.gmail.com> <430F4AF4.1060901@cs.stanford.edu> <1125358712.3203.89.camel@localhost> <20050830065603.GA11613@net.informatik.tu-muenchen.de> Message-ID: <1125447465.5492.9.camel@localhost> On Tue, 2005-08-30 at 08:56 +0200, Robin Sommer wrote: > > > configure kung-fu > > TransientID::ID already depends on being 64-bit and uses a long-long > for its tyoe (plus a compile-time check whether long-long are indeed > 64-bit). I remember that there was a problem with a nicer "kung-fu" > but I don't recall what is was... Nice, that'll help. We'll also need a check for the size of pointers on the system we're building for, but that shouldn't be too hard. Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From christian at whoop.org Tue Aug 30 22:52:06 2005 From: christian at whoop.org (Christian Kreibich) Date: Tue, 30 Aug 2005 22:52:06 -0700 Subject: [Bro] Bro install problems on linux In-Reply-To: <1125447465.5492.9.camel@localhost> References: <3582ae1205082515047c624e4@mail.gmail.com> <430F4AF4.1060901@cs.stanford.edu> <1125358712.3203.89.camel@localhost> <1125359430.3203.96.camel@localhost> <200508231801.j7NI1hsE011609@jaguar.icir.org> <3582ae12050823135269189023@mail.gmail.com> <1124982399.16503.99.camel@localhost> <3582ae1205082515047c624e4@mail.gmail.com> <430F4AF4.1060901@cs.stanford.edu> <1125358712.3203.89.camel@localhost> <20050830065603.GA11613@net.informatik.tu-muenchen.de> <1125447465.5492.9.camel@localhost> Message-ID: <1125467526.5492.47.camel@localhost> Hi, I'm attaching a patch for the 64-bit people out there to try. Note that I could only do build tests (using FC3 on an Opteron on SF's compile farm, *so* slow!) but not do any operational tests. Let us know if it works -- feedback appreciated! Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org -------------- next part -------------- A non-text attachment was scrubbed... Name: bro-64bit.diff Type: text/x-patch Size: 6323 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20050830/9894f7e1/attachment.bin From christian at whoop.org Wed Aug 31 04:58:15 2005 From: christian at whoop.org (Christian Kreibich) Date: Wed, 31 Aug 2005 04:58:15 -0700 Subject: [Bro] Side effects on record fields when triggering events Message-ID: <1125489495.5492.100.camel@localhost> Hi, is the following correct: when I trigger an event in the policy layer via the event() statement and pass a record value as an event argument, then at the time the triggered event is processed, the value is taken from the same location in memory that was passed to the event() statement. For example, this ... type foo: record { a: count &default = 0; }; event e1(f: foo) { print fmt("e1: %d", f$a); } event bro_init() { local f: foo; print fmt("bro_init: %d", f$a); event e1(f); ++f$a; } ... yields: bro_init: 0 e1: 1 So it seems that what is passed to the event handler is the value in its state before it goes out of scope (end of bro_init()). Any modifications of the record value that happen between the time of triggering the event and its processing will be visible to the event handler. Isn't that rather dangerous? My intuition was that an event() statement passes copies of all parameters to the event handlers to guarantee they all get the same values. I now do this manually where it is a problem in my policies. It gets more interesting -- why does the following work? type foo: record { a: count &default = 0; }; global bar: table[count] of foo; event e1(f: foo) { print fmt("e1: %d", f$a); } function f1(): foo { local f: foo; return f; } event bro_init() { bar[1] = f1(); print fmt("bro_init: %d", bar[1]$a); event e1(bar[1]); delete bar[1]; } It yields: bro_init: 0 e1: 0 After event e1 is triggered, the value passed to the e1 handler is erased from the global table. So why does this not give a run-time error? I'm confused at this point about the assumptions I can make about record values received by event handlers. If I missed the explanations of the call semantics in the manual then please point me to them, otherwise I think they should really be added... Thanks in advance! Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From christian at whoop.org Wed Aug 31 05:44:21 2005 From: christian at whoop.org (Christian Kreibich) Date: Wed, 31 Aug 2005 05:44:21 -0700 Subject: [Bro] Side effects on record fields when triggering events In-Reply-To: <1125489495.5492.100.camel@localhost> References: <1125489495.5492.100.camel@localhost> Message-ID: <1125492261.5492.106.camel@localhost> Okay Chema and I tracked it down to the reference counting that is going on in the core -- upon triggering, the reference count is bumped up by 1 per matching event handler and the value eventually dies after those handlers have been execuded. Until then any changes to the value are visible wherever a reference to the value still exists. I'm amazed this hasn't caused me problems earlier. Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From vern at icir.org Tue Aug 30 21:47:26 2005 From: vern at icir.org (Vern Paxson) Date: Tue, 30 Aug 2005 21:47:26 -0700 Subject: [Bro] Side effects on record fields when triggering events In-Reply-To: Your message of Wed, 31 Aug 2005 05:44:21 PDT. Message-ID: <200508310447.j7V4lQEd052838@jaguar.icir.org> > Okay Chema and I tracked it down to the reference counting that is going > on in the core Exactly. Aggregates in Bro are passed as shallow copiess, not deep copies. There are times when it would be handy to have an operator to specify deep-copy semantics. I share your concern at the counter-intuitive behavior that shallow-copy semantics can provide, but blanket deep-copy can be very expensive, and needlessly so in many cases. Vern From christian at whoop.org Wed Aug 31 18:39:47 2005 From: christian at whoop.org (Christian Kreibich) Date: Wed, 31 Aug 2005 18:39:47 -0700 Subject: [Bro] Side effects on record fields when triggering events In-Reply-To: <200508310447.j7V4lQEd052838@jaguar.icir.org> References: <200508310447.j7V4lQEd052838@jaguar.icir.org> Message-ID: <1125538788.5492.112.camel@localhost> On Tue, 2005-08-30 at 21:47 -0700, Vern Paxson wrote: > > Okay Chema and I tracked it down to the reference counting that is going > > on in the core > > Exactly. Aggregates in Bro are passed as shallow copiess, not deep copies. > There are times when it would be handy to have an operator to specify > deep-copy semantics. I share your concern at the counter-intuitive behavior > that shallow-copy semantics can provide, but blanket deep-copy can be very > expensive, and needlessly so in many cases. Yeah I agree. It's also not too big a big deal if you have the way it works in mind while writing policy code. I think I'll grow your patch queue with some documentation. :) Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From vern at icir.org Wed Aug 31 23:36:29 2005 From: vern at icir.org (Vern Paxson) Date: Wed, 31 Aug 2005 23:36:29 -0700 Subject: [Bro] Bro install problems on linux In-Reply-To: Your message of Mon, 29 Aug 2005 16:38:32 PDT. Message-ID: <200509010636.j816aTNN082592@jaguar.icir.org> > Unless we > throw out the macro containers and switch to STL ones. What's your take > on this Vern? That's fine with me, though it'll be a fair amount of work due to the widespread use of the List/PList classes. > > I'm assuming this code was written before templates became practical in a Ohhhhh yes - that code is the oldest in the entire distribution, dating to 1985! It's amazed it still compiles :-). Vern From vern at icir.org Wed Aug 31 23:38:24 2005 From: vern at icir.org (Vern Paxson) Date: Wed, 31 Aug 2005 23:38:24 -0700 Subject: [Bro] Bro install problems on linux In-Reply-To: Your message of Tue, 30 Aug 2005 22:52:06 PDT. Message-ID: <200509010638.j816cOKp082671@jaguar.icir.org> > I'm attaching a patch for the 64-bit people out there to try. Thanks! I plan to integrate this (with a name change for intp), unless folks find it causes problems. Vern