From vern at icir.org Thu Jan 3 00:12:55 2002 From: vern at icir.org (Vern Paxson) Date: Thu, 03 Jan 2002 00:12:55 -0800 Subject: =?iso-8859-1?Q?Bro_and_port_scan?= In-Reply-To: Your message of Wed, 12 Dec 2001 16:05:48 +0100. Message-ID: <200201030812.g038CtH99662@yak.icir.org> Sorry it took a long while for me to reply to your note :-( - it arrived in the midst of travel, followed by the holidays. > I have another questions regarding bro(version 07a90).I'am testing the > capability > of bro to detect port scanner.After , i have found others problem: > > 1) It don't detect Fin, Xmas and Null scans. Correct (though actually I'd have thought it would detect Xmas scans). Bro's scan detection is implemented in terms of connection_attempt, connection_rejected, and connection_established events, and Xmas and Null packets don't lead to these, since they're illegal-from-the-get-go. You could quite easily extend the event engine to aid with detecting these, though. What you'd do is modify TCP_Connection::NextPacket to generate events when it sees those types of packets, and then in your policy script hand the corresponding connections to check_scan(), to process like any other possible connection scan. Regarding detecting FIN scans, this is a bit tricky. The reason is because when you see a FIN go by, you don't know if it might be legitimate (for a connection that was established before Bro started running) or not. The way to tell is to see what sort of reply it elicits; basically, you want to generate an event if you see a FIN elicit a RST reply, and if there was no connection state in existence prior to the transmission of the FIN. This could likewise be added to TCP.cc, but requires more care, because it's a new connection state (FIN-transmitted-but-no-state-in-effect-when-it-was). > 2) Bro don't detect two time the seme scan betweent the same hosts on > the same ports. That's a policy decision. You could if you wanted modify check_scan() to clear the corresponding entries in the connection state tables in order to pick this up. > 3) It consume a great quantity of memory. Yes, and this needs work for high-volume environments. The first thing you can do is "@load reduce-memory", which trims a lot of timers and the like in order to decrease memory consumption. It helps considerably. A better solution is in the pipeline - Robin Sommer of the University of Saarbrucken has developed a general mechanism for timing out Bro state, which I'm working on integrating into the next release. > To resolve the first i have added a script to detect Fin, Xmas and Null > scans usuing the weird event their produce.And all works good. That certainly will work, though it's not the cleanest match to the problem (which would entail new events, per the above). > To resolve the 2? and the 3?, i have added to scan.bro a recursive > function (that use bro delete function and a table of support) to delete > the record of the table scan_triples(is deleted also the table of support). Yes, that should help with #2. > While the first problem seems to be resolved, the second is gotten worse. I'm not sure why it makes it worse, but it also likely wouldn't help, since for big scans, the state will just get created again anyway. > There is a way to easy delete a subset of a table? Only if you know which elements you want to get rid of. > or another way to > reduce the consume of memory of the scan analyzer? If reduce-memory doesn't do the trick for you, let me know and I'll try to put together a distribution snapshot with Robin's state expiration hooks so you can try using that. Vern From vern at icir.org Thu Jan 3 00:19:35 2002 From: vern at icir.org (Vern Paxson) Date: Thu, 03 Jan 2002 00:19:35 -0800 Subject: regarding mechanism , policy. In-Reply-To: Your message of Wed, 26 Dec 2001 16:56:24 EST. Message-ID: <200201030819.g038JZH99761@yak.icir.org> > It is very advantageous to have mechanism and policy separated > (as mentioned in Bro paper) but does it incur any overhead ? > > The paper says: for each event passed to the interpreter, > it retrieves the (semi-) compiled code for the corresponding handler, > binds the values of the events to the arguments of the handler, and > interprets the code. > > Would'nt this take some extra time compared to if it was already compiled? > (although the flexibility is lost in the latter case) There are two separate issues. One is whether interpreting the policy script incurs a performance penalty over compiling it. The second is whether decoupling mechanism from policy (so the fact that the event engine doesn't know what the handler will do) incurs a performance penalty. Regarding the first, profiling shows that Bro would certainly run faster if the script were compiled into a faster-to-execute form. Developing a compiler is on the Wish List, and I may have a student working on it at some point in the future; but it's quite involved, and before embarking on it in earnest, I want to develop better mechanisms for profiling just where Bro scripts spend most of their time. The student working on all of this is Umesh Shankar; he's developed an interactive Bro policy script debugger, which will be included in the next release. Once that's done, I'm hoping we can extend the framework with profiling hooks, so we can find out where do various scripts spend their time. It may turn out that the bottlenecks can be optimized without moving towards full-blown compilation of policy scripts. Regarding the second, I don't think the mechanism/policy separation is all that expensive per se. It entails one layer of binding between the event engine and the policy scripts, but I don't think that's inherently expensive, due to the event engine's ability to skip the work associated with events which the policy script doesn't handle - that's the most important feedback it needs from the policy script. Vern From vern at icir.org Thu Jan 3 00:20:53 2002 From: vern at icir.org (Vern Paxson) Date: Thu, 03 Jan 2002 00:20:53 -0800 Subject: internal error: bad reference count In-Reply-To: Your message of Sat, 29 Dec 2001 01:54:20 EST. Message-ID: <200201030820.g038KrH99841@yak.icir.org> > I am using bro 0.7a48 version on openBSD 2.9. First thing to try is the latest alpha snapshot: ftp://ftp.ee.lbl.gov/.vp-bro-pub-0.7a90.tar.gz If it still occurs with that, then see if you can get a version of the problem to reproduce (by capturing the related traffic in a tcpdump save file). Vern From athomas at unity.ncsu.edu Tue Jan 29 13:42:23 2002 From: athomas at unity.ncsu.edu (Ashley Thomas) Date: Tue, 29 Jan 2002 16:42:23 -0500 (EST) Subject: printing on stdout Message-ID: hi all, i am doing print fmt("String !"); but it does not come on the stdout It comes only when i terminate bro. ----------------------------------------------- # ./bro -i fxp1 mt listening on fxp1 ^C1012340454.288326 received termination signal 5 packets received on interface fxp1, 0 dropped String ! ----------------------------------------------- How can i make it to print out immediately not waiting for bro to terminate ? thanks ashley From vern at icir.org Tue Jan 29 14:01:29 2002 From: vern at icir.org (Vern Paxson) Date: Tue, 29 Jan 2002 14:01:29 -0800 Subject: printing on stdout In-Reply-To: Your message of Tue, 29 Jan 2002 16:42:23 EST. Message-ID: <200201292201.g0TM1TH97691@yak.icir.org> > i am doing print fmt("String !"); > but it does not come on the stdout > > It comes only when i terminate bro. > ... > How can i make it to print out immediately not waiting for bro to > terminate ? Call flush_all(). Vern From athomas at unity.ncsu.edu Tue Jan 29 14:23:23 2002 From: athomas at unity.ncsu.edu (Ashley Thomas) Date: Tue, 29 Jan 2002 17:23:23 -0500 (EST) Subject: printing on stdout In-Reply-To: <200201292201.g0TM1TH97691@yak.icir.org> Message-ID: i added flush_all() after the print fmt() but still does'nt come immediatley. Where as the print fmt in record_connection() comes immediately. i do'nt know if i'm missing something else like some setting ? thanks a lot ashley On Tue, 29 Jan 2002, Vern Paxson wrote: > > i am doing print fmt("String !"); > > but it does not come on the stdout > > > > It comes only when i terminate bro. > > ... > > How can i make it to print out immediately not waiting for bro to > > terminate ? > > Call flush_all(). > > Vern > From vern at icir.org Tue Jan 29 14:24:36 2002 From: vern at icir.org (Vern Paxson) Date: Tue, 29 Jan 2002 14:24:36 -0800 Subject: printing on stdout In-Reply-To: Your message of Tue, 29 Jan 2002 17:23:23 EST. Message-ID: <200201292224.g0TMOaH98481@yak.icir.org> > i added flush_all() after the print fmt() but still does'nt come > immediatley. > Where as the print fmt in record_connection() comes immediately. > > i do'nt know if i'm missing something else like some setting ? As usual, please send a working example that demonstrates the problem. (And probably we should take this off-line at this point.) Vern From jtyme at kuhalabs.com Wed Jan 30 08:03:58 2002 From: jtyme at kuhalabs.com (justin) Date: Wed, 30 Jan 2002 11:03:58 -0500 (EST) Subject: printing on stdout In-Reply-To: Message-ID: Hi Ashley, Try ending that string with a newline (\n) -justin On Tue, 29 Jan 2002, Ashley Thomas wrote: > hi all, > > i am doing print fmt("String !"); > but it does not come on the stdout > > It comes only when i terminate bro. > > ----------------------------------------------- > # ./bro -i fxp1 mt > listening on fxp1 > ^C1012340454.288326 received termination signal > 5 packets received on interface fxp1, 0 dropped > String ! > ----------------------------------------------- > > How can i make it to print out immediately not waiting for bro to > terminate ? > > thanks > ashley > > > >