From thodime_raghu at yahoo.com Tue Feb 1 17:15:24 2005 From: thodime_raghu at yahoo.com (Raghavendra Thodime) Date: Tue, 1 Feb 2005 17:15:24 -0800 (PST) Subject: [Bro] Help about Code Message-ID: <20050202011524.13770.qmail@web50109.mail.yahoo.com> Hi, I want to start looking at BRO open source code. Can anybody help how and where to start to make it faster? Does anybody has control flow description spec. (may be design spec)? Thanks --Raghu __________________________________ Do you Yahoo!? Meet the all-new My Yahoo! - Try it today! http://my.yahoo.com From christian.kreibich at cl.cam.ac.uk Wed Feb 2 02:57:49 2005 From: christian.kreibich at cl.cam.ac.uk (Christian Kreibich) Date: Wed, 02 Feb 2005 10:57:49 +0000 Subject: [Bro] Help about Code In-Reply-To: <20050202011524.13770.qmail@web50109.mail.yahoo.com> References: <20050202011524.13770.qmail@web50109.mail.yahoo.com> Message-ID: <1107341869.20546.242.camel@localhost> Hi Raghu, On Tue, 2005-02-01 at 17:15 -0800, Raghavendra Thodime wrote: > Hi, > I want to start looking at BRO open source code. > Can anybody help how and where to start to make it > faster? there's no obvious way to make Bro faster because there are so many ways in which it can be configured to be slow :) So first you should try to understand why you believe it to be slow, and how you can adjust your policy to speed things up. > Does anybody has control flow description > spec. (may be design spec)? The following might help you: http://www.icir.org/twiki/bin/view/Bro/BroInternalsAug2004 However, for the latest you'll always have to look at the source code yourself... Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From sommer at in.tum.de Wed Feb 2 04:55:04 2005 From: sommer at in.tum.de (Robin Sommer) Date: Wed, 2 Feb 2005 13:55:04 +0100 Subject: [Bro] Help about Code In-Reply-To: <20050202011524.13770.qmail@web50109.mail.yahoo.com> References: <20050202011524.13770.qmail@web50109.mail.yahoo.com> Message-ID: <20050202125504.GA11794@net.informatik.tu-muenchen.de> On Tue, Feb 01, 2005 at 17:15 -0800, Raghavendra Thodime wrote: > faster? Does anybody has control flow description > spec. (may be design spec)? Running doxygen on the code produces some nice graphs of Bro's class hierarchy. Robin -- Robin Sommer * Room 01.08.055 * www.net.in.tum.de TU Muenchen * Phone (089) 289-18006 * sommer at in.tum.de From rpang at CS.Princeton.EDU Wed Feb 2 10:31:10 2005 From: rpang at CS.Princeton.EDU (Ruoming Pang) Date: Wed, 2 Feb 2005 13:31:10 -0500 Subject: [Bro] Help about Code In-Reply-To: <1107341869.20546.242.camel@localhost> References: <20050202011524.13770.qmail@web50109.mail.yahoo.com> <1107341869.20546.242.camel@localhost> Message-ID: <9C59FC16-7548-11D9-8E05-000D9335A7D8@cs.princeton.edu> Raghu, Bro usually spends most of its time in executing policy scripts (this, of course, depends on the configuration). And when it's too slow, my experience has been that it's often because some event is invoked too many times. Thus the first step I would take is to find out which events are invoked most frequently. There are a class of "high-risk" events, for example, tcp_packet, http_header, etc., that can easily be invoked too often and should be avoided when dealing with high volume live traffic. > The following might help you: > > http://www.icir.org/twiki/bin/view/Bro/BroInternalsAug2004 > And Vern's original paper: ftp://ftp.ee.lbl.gov/papers/bro-CN99.ps.gz > However, for the latest you'll always have to look at the source code > yourself... Exactly. Ruoming From goeldich at ee.ethz.ch Thu Feb 3 02:24:19 2005 From: goeldich at ee.ethz.ch (Christoph Goeldi) Date: Thu, 03 Feb 2005 11:24:19 +0100 Subject: [Bro] true global functions/tables/variables Message-ID: <4201FBD3.3030007@ee.ethz.ch> hi everybody today i noticed that i can't define functions, tables or variables and use it different policy files. it only worked, after i placed these things in the bro.init file. i think it is a drawback if it's impossible to implement a certain functionality with a bunch of files and use the same functions, tables etc. in them. if an algorithm is a bit complex you have no chance to implement it with a nice structure... thanx for any help or comments christoph From goeldich at ee.ethz.ch Thu Feb 3 05:20:00 2005 From: goeldich at ee.ethz.ch (Christoph Goeldi) Date: Thu, 03 Feb 2005 14:20:00 +0100 Subject: [Bro] using &*_expire with interval values defined before Message-ID: <42022500.3030801@ee.ethz.ch> hi everybody does anyone know why this doesn't work and results in an error? policy-script: > const tcp_scan_period = 2 min &redef; > global tcp_scan: table[addr] of count &write_expire=tcp_scan_period; error: > line xx (tcp_scan_period): bad tag in ExprVal::Val thanx for any help christoph From christian at whoop.org Thu Feb 3 07:57:41 2005 From: christian at whoop.org (Christian Kreibich) Date: Thu, 03 Feb 2005 15:57:41 +0000 Subject: [Bro] true global functions/tables/variables In-Reply-To: <4201FBD3.3030007@ee.ethz.ch> References: <4201FBD3.3030007@ee.ethz.ch> Message-ID: <1107446262.31848.34.camel@localhost> Hi Christoph, I'm far from a Bro language guru but I don't think that's true. Just as an example, function service_name() is defined in conn.bro but used in conn.bro and login.bro. Are you @load'ing things correctly? I hope I'm not just missing your point ... On Thu, 2005-02-03 at 11:24 +0100, Christoph Goeldi wrote: > hi everybody > > today i noticed that i can't define functions, tables or variables and > use it different policy files. it only worked, after i placed these > things in the bro.init file. > i think it is a drawback if it's impossible to implement a certain > functionality with a bunch of files and use the same functions, tables > etc. in them. > if an algorithm is a bit complex you have no chance to implement it with > a nice structure... > > thanx for any help or comments > christoph Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From goeldich at ee.ethz.ch Thu Feb 3 09:22:17 2005 From: goeldich at ee.ethz.ch (=?ISO-8859-1?Q?Christoph_G=F6ldi?=) Date: Thu, 03 Feb 2005 18:22:17 +0100 Subject: [Bro] true global functions/tables/variables In-Reply-To: <1107446262.31848.34.camel@localhost> References: <4201FBD3.3030007@ee.ethz.ch> <1107446262.31848.34.camel@localhost> Message-ID: <972133CDE66F89B642CFDF8D@[192.168.0.2]> Hi sorry, your right. i messed up with the @load commands. i didn't realize that the order of loading files and defining functions is important. thanx christoph --On Donnerstag, 3. Februar 2005 15:57 +0000 Christian Kreibich wrote: > Hi Christoph, > > I'm far from a Bro language guru but I don't think that's true. Just as > an example, function service_name() is defined in conn.bro but used in > conn.bro and login.bro. Are you @load'ing things correctly? I hope I'm > not just missing your point ... > > On Thu, 2005-02-03 at 11:24 +0100, Christoph Goeldi wrote: >> hi everybody >> >> today i noticed that i can't define functions, tables or variables and >> use it different policy files. it only worked, after i placed these >> things in the bro.init file. >> i think it is a drawback if it's impossible to implement a certain >> functionality with a bunch of files and use the same functions, tables >> etc. in them. >> if an algorithm is a bit complex you have no chance to implement it with >> a nice structure... >> >> thanx for any help or comments >> christoph > > Cheers, > Christian. > -- > ________________________________________________________________________ > http://www.cl.cam.ac.uk/~cpk25 > http://www.whoop.org > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From thodime_raghu at yahoo.com Fri Feb 4 16:03:03 2005 From: thodime_raghu at yahoo.com (Raghavendra Thodime) Date: Fri, 4 Feb 2005 16:03:03 -0800 (PST) Subject: [Bro] Problem sniffing RTP in Linux Message-ID: <20050205000303.45564.qmail@web50102.mail.yahoo.com> In Linux if I give ./tethereal -R "sip or rtp" I only see sip messages. I don't see rtp messages. Is this because tethereal I have is older one. I know that thereal sniffs rtp packets based on SDP info it sees in SIP packet. Thanks Raghu __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From vern at icir.org Sat Feb 5 16:45:54 2005 From: vern at icir.org (Vern Paxson) Date: Sat, 05 Feb 2005 16:45:54 -0800 Subject: [Bro] using &*_expire with interval values defined before In-Reply-To: Your message of Thu, 03 Feb 2005 14:20:00 +0100. Message-ID: <200502060046.j160jsIV087373@jaguar.icir.org> > does anyone know why this doesn't work and results in an error? > > policy-script: > > const tcp_scan_period = 2 min &redef; > > global tcp_scan: table[addr] of count &write_expire=tcp_scan_period; > > error: > > line xx (tcp_scan_period): bad tag in ExprVal::Val Definitely a bug. I'm able to reproduce it and will include a fix with the next release (which has been delayed due to some other late-breaking bugs, sigh). By the way, any time you see "bad tag", it's definitely a bug, that's an internal error message. Vern From goeldich at ee.ethz.ch Mon Feb 7 07:28:10 2005 From: goeldich at ee.ethz.ch (Christoph Goeldi) Date: Mon, 07 Feb 2005 16:28:10 +0100 Subject: [Bro] how to get the oldest entry out of table Message-ID: <4207890A.2070003@ee.ethz.ch> hi everybody is it possible to get the oldest entry out of a table? for example, i want to get the earliest occurence of an event from a certain source ip. i have saved all occurences in a table like this: > global test: table[addr, time] of count &write_expire=10min; how can i get the oldest entry? does a for-loop go through the table in a predictable way? thanx for any help christoph From sommer at in.tum.de Mon Feb 7 07:57:12 2005 From: sommer at in.tum.de (Robin Sommer) Date: Mon, 7 Feb 2005 16:57:12 +0100 Subject: [Bro] how to get the oldest entry out of table In-Reply-To: <4207890A.2070003@ee.ethz.ch> References: <4207890A.2070003@ee.ethz.ch> Message-ID: <20050207155712.GB27537@net.informatik.tu-muenchen.de> On Mon, Feb 07, 2005 at 16:28 +0100, Christoph Goeldi wrote: > is it possible to get the oldest entry out of a table? For tables, I do not see a way other than iterating over all entries. But perhaps you can use a combination of tables and vectors; something like type entry: record { t: time; c: count; }; global test: table[addr] of vector of entry; Then you can keep the vector entries sorted by time. > does a for-loop go through the table in a predictable way? No, for tables the order is derived from the internal hashing, i.e. basically random. Robin -- Robin Sommer * Room 01.08.055 * www.net.in.tum.de TU Muenchen * Phone (089) 289-18006 * sommer at in.tum.de From goeldich at ee.ethz.ch Mon Feb 7 08:23:41 2005 From: goeldich at ee.ethz.ch (Christoph =?iso-8859-1?b?R/ZsZGk=?=) Date: Mon, 7 Feb 2005 16:23:41 +0000 Subject: [Bro] how to get the oldest entry out of table In-Reply-To: <20050207155712.GB27537@net.informatik.tu-muenchen.de> References: <4207890A.2070003@ee.ethz.ch> <20050207155712.GB27537@net.informatik.tu-muenchen.de> Message-ID: <1107793421.4207960d38e55@email.ee.ethz.ch> hi > > is it possible to get the oldest entry out of a table? > > For tables, I do not see a way other than iterating over all > entries. But perhaps you can use a combination of tables and > vectors; something like > > type entry: record { > t: time; > c: count; > }; > > global test: table[addr] of vector of entry; > > Then you can keep the vector entries sorted by time. is vector a new type which is not documented in the bro ref manual? however, what you suggest doesn't solve my problem, because i have no chance to expire and delete each entry in the table individually. i can only delete all entries of one addr at the same time with the expire attribut for tables. thanx christoph From christian.kreibich at cl.cam.ac.uk Mon Feb 7 15:20:02 2005 From: christian.kreibich at cl.cam.ac.uk (Christian Kreibich) Date: Mon, 07 Feb 2005 23:20:02 +0000 Subject: [Bro] how to get the oldest entry out of table In-Reply-To: <1107793421.4207960d38e55@email.ee.ethz.ch> References: <4207890A.2070003@ee.ethz.ch> <20050207155712.GB27537@net.informatik.tu-muenchen.de> <1107793421.4207960d38e55@email.ee.ethz.ch> Message-ID: <1107818402.6653.101.camel@localhost> Hi Christoph, On Mon, 2005-02-07 at 16:23 +0000, Christoph G?ldi wrote: > > is vector a new type which is not documented in the bro ref manual? not really new, but not well documented either. Check the CHANGES file. > however, what you suggest doesn't solve my problem, because i have no > chance to expire and delete each entry in the table individually. i can only > delete all entries of one addr at the same time with the expire > attribut for tables. Maybe we could help you better if you explained your problem setting a bit. It sounds like what you'd like to have is a priority queue instead of a basic table with expiration timers? Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From vern at icir.org Mon Feb 7 19:00:35 2005 From: vern at icir.org (Vern Paxson) Date: Mon, 07 Feb 2005 19:00:35 -0800 Subject: [Bro] using &*_expire with interval values defined before In-Reply-To: Your message of Thu, 03 Feb 2005 14:20:00 +0100. Message-ID: <200502080300.j1830Zs6055666@jaguar.icir.org> > policy-script: > > const tcp_scan_period = 2 min &redef; > > global tcp_scan: table[addr] of count &write_expire=tcp_scan_period; > > error: > > line xx (tcp_scan_period): bad tag in ExprVal::Val Here's a patch to fix the bug. (Also, a work-around would be to instead stick "2 min" directly in rather than using a variable.) This will be included in the upcoming release. Vern --- Val.cc.ORIG 2005/02/07 23:46:08 1.1 +++ Val.cc 2005/02/08 00:49:59 @@ -1490,7 +1490,7 @@ if ( a ) { - expire_time = a->AttrExpr()->ExprVal()->AsInterval(); + expire_time = a->AttrExpr()->Eval(0)->AsInterval(); // As network_time is not necessarily initialized yet, // we set a timer which fires immediately. From christian at whoop.org Tue Feb 8 06:56:14 2005 From: christian at whoop.org (Christian Kreibich) Date: Tue, 08 Feb 2005 14:56:14 +0000 Subject: [Bro] Broccoli 0.6 released. Message-ID: <1107874575.14086.30.camel@localhost> Hi all, I've just released version 0.6 of the Bro client communications library. For further information, downloads, and the manual please visit http://www.cl.cam.ac.uk/~cpk25/broccoli/index.html Feedback is always welcome. Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From goeldich at ee.ethz.ch Tue Feb 8 11:56:54 2005 From: goeldich at ee.ethz.ch (=?ISO-8859-1?Q?Christoph_G=F6ldi?=) Date: Tue, 08 Feb 2005 20:56:54 +0100 Subject: [Bro] how to get the oldest entry out of table In-Reply-To: <1107818402.6653.101.camel@localhost> References: <4207890A.2070003@ee.ethz.ch> <20050207155712.GB27537@net.informatik.tu-muenchen.de> <1107793421.4207960d38e55@email.ee.ethz.ch> <1107818402.6653.101.camel@localhost> Message-ID: <28E00D7321396A5A698329E2@[192.168.0.2]> hi > Maybe we could help you better if you explained your problem setting a > bit. It sounds like what you'd like to have is a priority queue instead > of a basic table with expiration timers? okay. let's say i want to measure the failed connection attempts rate of a host. therefore, i maintain a table which contains the occurences of these events and a counter which holds the number of entries in the table per source host: > global failed_attempts: table[addr, addr, time] of bool > &expire_create=1min &expire_func=decrease_counter; > global failed_attempt_counter: table[addr] of count &default=0; > > event connection_attempt(c: connection){ > local srcIP = c$id$orig_h; > local destIP = c$id$resp_h; > > failed_attempts[srcIP, destIP, network_time()]=T; > ++failed_attempt_counter[srcIP]; > if (failed_attempt_counter[srcIP] > 100) { > alarm fmt("more than 100 failed attempts of host %s in 1 minute", > srcIP); > } > } > > function decrease_counter(t: table[addr, addr, time], idx: any){ > local srcIP: addr; > local destIP: addr; > local stamp: time; > [srcIP,destIP,time]=idx; > --failed_attempt_counter[srcIP]; > } when a worm becomes active the infected host will possibly have 500 failed attempts per second! the above solution can't give me this information. therefore, it would be very nice to have the time of occurence of the oldest entry in the table "failed_attempts" conserning a certain source IP. with this information it would be possible to calculate how long it took until the 100 failed attempts had happened. sorry, my case is a bit complicated. i hope you understand what i mean. thanx christoph From vern at icir.org Tue Feb 8 22:37:12 2005 From: vern at icir.org (Vern Paxson) Date: Tue, 08 Feb 2005 22:37:12 -0800 Subject: [Bro] how to get the oldest entry out of table In-Reply-To: Your message of Tue, 08 Feb 2005 20:56:54 +0100. Message-ID: <200502090637.j196bClI096085@jaguar.icir.org> You can implement a FIFO of type "foo" using "table[count] of foo" and explicitly tracking its first and last element in order to pull from the front and push onto the back. With this idiom, you can track connection failures like you sketch using something like the appended. What I've implemented counts multiple failures to the same host; I think you instead want multiple failures regardless of whether they're to the same host or different, but I wasn't quite sure since you traced destination IP in your failed_attempts variable. Hopefully it's clear how to modify this for the different functionality - let me know if not. Also, while I implemented a FIFO that tracks , for your example use all that's actually needed is a count. That would be much simpler, just have a table[addr] of count indexed per source, bump it up on each connection_attempt event and set a timer to decrement it a minute later. Vern type pair: record { src: addr; dst: addr; }; type failure: record { p: pair; t: time; }; global conn_failures: table[count, pair] of failure; global first_conn_failure: table[pair] of count; global last_conn_failure: table[pair] of count; global failure_thresh = 100 &redef; global failure_interval = 1 min &redef; event remove_conn_failure(p: pair) { if ( pair !in first_conn_failure ) # Shouldn't happen - could print an error and exit. return; # Pop from front. delete conn_failures[first_conn_failure[p], p]; ++first_conn_failure[p]; # move past now vacated element } event connection_attempt(c: connection) { local p = [$src = c$id$orig_h, $dst = c$id$resp_h]; if ( p !in first_conn_failure ) { first_conn_failure[p] = 1; last_conn_failure[p] = 0; # about to increment } local f = [$p = p, $t = c$start_time]; # Append to end. conn_failures[++last_conn_failure[p], p] = f; if ( last_conn_failure[p] - first_conn_failure[p] > failure_thresh ) alarm fmt("more than %d failed attempts of host %s in 1 minute", failure_thresh, f$p$src); schedule failure_interval { remove_conn_failure(p) }; } From goeldich at ee.ethz.ch Thu Feb 10 02:55:01 2005 From: goeldich at ee.ethz.ch (Christoph Goeldi) Date: Thu, 10 Feb 2005 11:55:01 +0100 Subject: [Bro] problem with &expire_func Message-ID: <420B3D85.6090504@ee.ethz.ch> hi there i found another problem with bro. this time it is the expire_func which makes trouble. when i have a policy file like this: > global mytable: table[addr] of bool &write_expire=10sec &expire_func=myfunc; > > function myfunc(t: table[addr] of bool , idx: any): interval { > local srcIP: addr; > > print network_time(),"help!!!"; > [srcIP] = idx; > > return 0secs; > } it ends in an error like this: > 1108032092.49407, help!!! > 1108032092.494068 (192.168.0.75): bad tag in Val::CONVERTER i tested several constellations and noticed, that it is impossible to use idx when a table has only one index. any help welcome. thanx christoph From vern at icir.org Thu Feb 10 12:49:11 2005 From: vern at icir.org (Vern Paxson) Date: Thu, 10 Feb 2005 12:49:11 -0800 Subject: [Bro] problem with &expire_func In-Reply-To: Your message of Thu, 10 Feb 2005 11:55:01 +0100. Message-ID: <200502102049.j1AKnBH8090340@jaguar.icir.org> > > global mytable: table[addr] of bool &write_expire=10sec &expire_func=myfunc; > > > > function myfunc(t: table[addr] of bool , idx: any): interval { > > local srcIP: addr; > > > > print network_time(),"help!!!"; > > [srcIP] = idx; > > > > return 0secs; > > } The problem here is that if there's just a single index, it's transmitted in the call to the expire_func as its own type rather than as a combo "any" type. So you need to instead use: function myfunc(t: table[addr] of bool , idx: addr): interval { local srcIP: addr; srcIP = idx; return 0secs; } (Or "local srcIP = idx", or just use idx directly in the function.) Bro shouldn't of course crash for the example above as you gave it. I've entered fixing this into the bug-tracker, but using the type directly as I show is the proper way to do this (i.e., not merely a workaround). Vern From goeldich at ee.ethz.ch Thu Feb 10 13:20:36 2005 From: goeldich at ee.ethz.ch (=?ISO-8859-1?Q?Christoph_G=F6ldi?=) Date: Thu, 10 Feb 2005 22:20:36 +0100 Subject: [Bro] problem with &expire_func In-Reply-To: <200502102049.j1AKnBH8090340@jaguar.icir.org> References: <200502102049.j1AKnBH8090340@jaguar.icir.org> Message-ID: <9D0B8156C505D6626403688B@[192.168.0.2]> hi vern > The problem here is that if there's just a single index, it's transmitted > in the call to the expire_func as its own type rather than as a combo > "any" type. So you need to instead use: > > function myfunc(t: table[addr] of bool , idx: addr): interval { > local srcIP: addr; > srcIP = idx; > return 0secs; > } > > (Or "local srcIP = idx", or just use idx directly in the function.) > > Bro shouldn't of course crash for the example above as you gave it. > I've entered fixing this into the bug-tracker, but using the type directly > as I show is the proper way to do this (i.e., not merely a workaround). i already tried these cases: > function myfunc(t: table[addr] of bool , idx: addr): interval { > local srcIP: addr; > srcIP = idx; > return 0secs; > } results in this error when i want to start bro: > policy/test.bro, line xx (srcIP = idx): error, type clash in assignment and: > function myfunc(t: table[addr] of count, idx: any): interval { > local srcIP = idx; > print t[srcIP]; > return 0secs; > } results in: > addr, line 0 and policy/test.bro, line xx > (addr and srcIP): error, type clash > policy/test.bro, line xx and addr, line 0 > (srcIP and addr): error, type mismatch > policy/test.bro, line xx (t[srcIP]): error, not an index type any further ideas? cheers christoph From vern at icir.org Thu Feb 10 13:29:22 2005 From: vern at icir.org (Vern Paxson) Date: Thu, 10 Feb 2005 13:29:22 -0800 Subject: [Bro] problem with &expire_func In-Reply-To: Your message of Thu, 10 Feb 2005 22:20:36 +0100. Message-ID: <200502102129.j1ALTMgD092776@jaguar.icir.org> > > function myfunc(t: table[addr] of bool , idx: addr): interval { > > local srcIP: addr; > > srcIP = idx; > > return 0secs; > > } > > results in this error when i want to start bro: > > policy/test.bro, line xx (srcIP = idx): error, type clash in assignment Please send the full script exhibiting this problem. I actually *did* test my suggestion and it works for me, and I just fed the above into Bro and it didn't generate any error message. Vern From goeldich at ee.ethz.ch Thu Feb 10 13:57:23 2005 From: goeldich at ee.ethz.ch (=?ISO-8859-1?Q?Christoph_G=F6ldi?=) Date: Thu, 10 Feb 2005 22:57:23 +0100 Subject: [Bro] problem with &expire_func In-Reply-To: <200502102129.j1ALTMgD092776@jaguar.icir.org> References: <200502102129.j1ALTMgD092776@jaguar.icir.org> Message-ID: <70BB22B66E07FB53CF00861B@[192.168.0.2]> --On Donnerstag, 10. Februar 2005 13:29 -0800 Vern Paxson wrote: >> > function myfunc(t: table[addr] of bool , idx: addr): interval { >> > local srcIP: addr; >> > srcIP = idx; >> > return 0secs; >> > } >> >> results in this error when i want to start bro: >> > policy/test.bro, line xx (srcIP = idx): error, type clash in assignment > > Please send the full script exhibiting this problem. I actually *did* > test my suggestion and it works for me, and I just fed the above into Bro > and it didn't generate any error message. here: ######### test.bro > global myfunc: function(t: table[addr] of count, idx: any): interval; > > global tcp_host_state: table[addr] of count &default=0 &write_expire=1day > &expire_func=myfunc; > > function myfunc(t: table[addr] of count, idx: any): interval { > local srcIP: addr; > srcIP = idx; > print t[srcIP]; > return 0secs; > } result: > christoph:/usr/local/bro # ./bin/bro -i eth0 test > policy/test.bro, line 7 (srcIP = idx): error, type clash in assignment thanx christoph From vern at icir.org Thu Feb 10 14:11:36 2005 From: vern at icir.org (Vern Paxson) Date: Thu, 10 Feb 2005 14:11:36 -0800 Subject: [Bro] problem with &expire_func In-Reply-To: Your message of Thu, 10 Feb 2005 22:57:23 +0100. Message-ID: <200502102211.j1AMBaxl094216@jaguar.icir.org> > > global myfunc: function(t: table[addr] of count, idx: any): interval; ^^^ Change "any" here to "addr", per my previous note. Vern From goeldich at ee.ethz.ch Thu Feb 10 14:14:19 2005 From: goeldich at ee.ethz.ch (=?ISO-8859-1?Q?Christoph_G=F6ldi?=) Date: Thu, 10 Feb 2005 23:14:19 +0100 Subject: [Bro] problem with &expire_func In-Reply-To: <200502102211.j1AMBaxl094216@jaguar.icir.org> References: <200502102211.j1AMBaxl094216@jaguar.icir.org> Message-ID: <9D60FBC0C650ACB02786D4AA@[192.168.0.2]> ou, sorry, i misunderstood you... thanx christoph --On Donnerstag, 10. Februar 2005 14:11 -0800 Vern Paxson wrote: >> > global myfunc: function(t: table[addr] of count, idx: any): interval; > ^^^ > Change "any" here to "addr", per my previous note. > > Vern From goeldich at ee.ethz.ch Fri Feb 11 05:34:03 2005 From: goeldich at ee.ethz.ch (Christoph Goeldi) Date: Fri, 11 Feb 2005 14:34:03 +0100 Subject: [Bro] udp_reply event instead of supposed udp_request event Message-ID: <420CB44B.9090702@ee.ethz.ch> hi i have another riddle for you today ;-) the following policy script shows not the behaviour i suppose: > # test.bro > > event udp_request(u: connection) { > local srcIP = u$id$orig_h; > local destIP = u$id$resp_h; > print "udp_request", u$id; > } > > event udp_reply(u: connection) { > local srcIP = u$id$orig_h; > local destIP = u$id$resp_h; > print "udp_reply", u$id; > } i produced traffic from the source host x.x.x.75 to randomly chosen hosts. source port was 53 (dns). the sent packets didn't contain any dns replies. > [root at det:/usr/local/bro]# ./bin/bro -i eth3 test > listening on eth3 > Reading .state/state.bst ... > udp_reply, [orig_h=28.239.208.235, orig_p=51647/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=128.212.59.231, orig_p=27613/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=82.146.148.227, orig_p=53106/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=58.2.1.68, orig_p=61607/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=83.88.197.211, orig_p=10120/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=51.85.5.250, orig_p=54565/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=109.89.191.91, orig_p=25624/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=17.93.105.34, orig_p=59958/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=105.107.199.36, orig_p=56071/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=113.201.33.243, orig_p=6563/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=32.154.25.148, orig_p=19877/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=84.150.247.84, orig_p=44926/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=114.90.239.185, orig_p=6913/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=43.22.154.204, orig_p=16223/udp, resp_h=x.x.x.75, resp_p=53/udp] > udp_reply, [orig_h=50.56.252.49, orig_p=18381/udp, resp_h=x.x.x.75, resp_p=53/udp] > 1108127885.586962 received termination signal > 401 packets received on interface eth3, 0 dropped > 1108127885.586962 Saving state... this is the corresponding tcpdump output: > [root at det:~]# tcpdump -i eth3 -n -q host x.x.x.75 and udp > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on eth3, link-type EN10MB (Ethernet), capture size 68 bytes > 14:18:02.481453 IP x.x.x.75.53 > 28.239.208.235.51647: UDP, length: 33 > 14:18:02.581319 IP x.x.x.75.53 > 128.212.59.231.27613: UDP, length: 33 > 14:18:02.681322 IP x.x.x.75.53 > 82.146.148.227.53106: UDP, length: 33 > 14:18:02.781343 IP x.x.x.75.53 > 58.2.1.68.61607: UDP, length: 33 > 14:18:02.881326 IP x.x.x.75.53 > 83.88.197.211.10120: UDP, length: 33 > 14:18:02.981365 IP x.x.x.75.53 > 51.85.5.250.54565: UDP, length: 33 > 14:18:03.081301 IP x.x.x.75.53 > 109.89.191.91.25624: UDP, length: 33 > 14:18:03.181319 IP x.x.x.75.53 > 17.93.105.34.59958: UDP, length: 33 > 14:18:03.281291 IP x.x.x.75.53 > 105.107.199.36.56071: UDP, length: 33 > 14:18:03.381325 IP x.x.x.75.53 > 113.201.33.243.6563: UDP, length: 33 > 14:18:03.481325 IP x.x.x.75.53 > 32.154.25.148.19877: UDP, length: 33 > 14:18:03.581255 IP x.x.x.75.53 > 84.150.247.84.44926: UDP, length: 33 > 14:18:03.681327 IP x.x.x.75.53 > 114.90.239.185.6913: UDP, length: 33 > 14:18:03.781367 IP x.x.x.75.53 > 43.22.154.204.16223: UDP, length: 33 > 14:18:03.881327 IP x.x.x.75.53 > 50.56.252.49.18381: UDP, length: 33 > > 15 packets captured > 15 packets received by filter > 0 packets dropped by kernel my question is: why does bro recognizes udp_reply events and not udp_request events? the packets were only sent from one host to another and there were no packets in the opposite direction. i know that udp packets from port 53 are often dns replies but an assumption which is made because of the application layer protocol shouldn't have any impact on events on the transport protocol layer... thanx christoph From goeldich at ee.ethz.ch Fri Feb 11 06:35:37 2005 From: goeldich at ee.ethz.ch (Christoph =?iso-8859-1?b?R/ZsZGk=?=) Date: Fri, 11 Feb 2005 14:35:37 +0000 Subject: [Bro] udp_reply event instead of supposed udp_request event In-Reply-To: <420CB44B.9090702@ee.ethz.ch> References: <420CB44B.9090702@ee.ethz.ch> Message-ID: <1108132537.420cc2b9b4b1e@email.ee.ethz.ch> the same behavior with sending port 111 and port 123 instead of port 53. Zitat von Christoph Goeldi : > hi > > > i have another riddle for you today ;-) > the following policy script shows not the behaviour i suppose: > > > # test.bro > > > > event udp_request(u: connection) { > > local srcIP = u$id$orig_h; > > local destIP = u$id$resp_h; > > print "udp_request", u$id; > > } > > > > event udp_reply(u: connection) { > > local srcIP = u$id$orig_h; > > local destIP = u$id$resp_h; > > print "udp_reply", u$id; > > } > > > i produced traffic from the source host x.x.x.75 to randomly chosen > hosts. source port was 53 (dns). the sent packets didn't > contain any dns replies. > > > [root at det:/usr/local/bro]# ./bin/bro -i eth3 test > > listening on eth3 > > Reading .state/state.bst ... > > udp_reply, [orig_h=28.239.208.235, orig_p=51647/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=128.212.59.231, orig_p=27613/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=82.146.148.227, orig_p=53106/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=58.2.1.68, orig_p=61607/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=83.88.197.211, orig_p=10120/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=51.85.5.250, orig_p=54565/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=109.89.191.91, orig_p=25624/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=17.93.105.34, orig_p=59958/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=105.107.199.36, orig_p=56071/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=113.201.33.243, orig_p=6563/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=32.154.25.148, orig_p=19877/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=84.150.247.84, orig_p=44926/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=114.90.239.185, orig_p=6913/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=43.22.154.204, orig_p=16223/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > udp_reply, [orig_h=50.56.252.49, orig_p=18381/udp, resp_h=x.x.x.75, > resp_p=53/udp] > > 1108127885.586962 received termination signal > > 401 packets received on interface eth3, 0 dropped > > 1108127885.586962 Saving state... > > > this is the corresponding tcpdump output: > > > [root at det:~]# tcpdump -i eth3 -n -q host x.x.x.75 and udp > > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > > listening on eth3, link-type EN10MB (Ethernet), capture size 68 bytes > > 14:18:02.481453 IP x.x.x.75.53 > 28.239.208.235.51647: UDP, length: 33 > > 14:18:02.581319 IP x.x.x.75.53 > 128.212.59.231.27613: UDP, length: 33 > > 14:18:02.681322 IP x.x.x.75.53 > 82.146.148.227.53106: UDP, length: 33 > > 14:18:02.781343 IP x.x.x.75.53 > 58.2.1.68.61607: UDP, length: 33 > > 14:18:02.881326 IP x.x.x.75.53 > 83.88.197.211.10120: UDP, length: 33 > > 14:18:02.981365 IP x.x.x.75.53 > 51.85.5.250.54565: UDP, length: 33 > > 14:18:03.081301 IP x.x.x.75.53 > 109.89.191.91.25624: UDP, length: 33 > > 14:18:03.181319 IP x.x.x.75.53 > 17.93.105.34.59958: UDP, length: 33 > > 14:18:03.281291 IP x.x.x.75.53 > 105.107.199.36.56071: UDP, length: 33 > > 14:18:03.381325 IP x.x.x.75.53 > 113.201.33.243.6563: UDP, length: 33 > > 14:18:03.481325 IP x.x.x.75.53 > 32.154.25.148.19877: UDP, length: 33 > > 14:18:03.581255 IP x.x.x.75.53 > 84.150.247.84.44926: UDP, length: 33 > > 14:18:03.681327 IP x.x.x.75.53 > 114.90.239.185.6913: UDP, length: 33 > > 14:18:03.781367 IP x.x.x.75.53 > 43.22.154.204.16223: UDP, length: 33 > > 14:18:03.881327 IP x.x.x.75.53 > 50.56.252.49.18381: UDP, length: 33 > > > > 15 packets captured > > 15 packets received by filter > > 0 packets dropped by kernel > > my question is: why does bro recognizes udp_reply events and not udp_request > events? the packets were only sent from one host to another and there > were no packets in the opposite direction. > > i know that udp packets from port 53 are often dns replies but an > assumption which is made because of the application layer protocol > shouldn't have any impact on events on the transport protocol layer... > > > thanx > christoph > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From christian at whoop.org Fri Feb 11 07:02:21 2005 From: christian at whoop.org (Christian Kreibich) Date: Fri, 11 Feb 2005 15:02:21 +0000 Subject: [Bro] udp_reply event instead of supposed udp_request event In-Reply-To: <420CB44B.9090702@ee.ethz.ch> References: <420CB44B.9090702@ee.ethz.ch> Message-ID: <1108134142.16852.34.camel@localhost> On Fri, 2005-02-11 at 14:34 +0100, Christoph Goeldi wrote: > > my question is: why does bro recognizes udp_reply events and not udp_request > events? the packets were only sent from one host to another and there > were no packets in the opposite direction. It's hardcoded. Sessions.cc, around 1247. > i know that udp packets from port 53 are often dns replies but an > assumption which is made because of the application layer protocol > shouldn't have any impact on events on the transport protocol layer... Bro assumes that UDP port 53 is DNS anyway, so assuming that traffic sourced from port 53 is a DNS reply doesn't make much of a difference. You could make that flip in Sessions.cc policy-controlled if it really gets in your way ... Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From christian at whoop.org Fri Feb 11 07:41:00 2005 From: christian at whoop.org (Christian Kreibich) Date: Fri, 11 Feb 2005 15:41:00 +0000 Subject: [Bro] udp_reply event instead of supposed udp_request event In-Reply-To: <1108132537.420cc2b9b4b1e@email.ee.ethz.ch> References: <420CB44B.9090702@ee.ethz.ch> <1108132537.420cc2b9b4b1e@email.ee.ethz.ch> Message-ID: <1108136461.16852.37.camel@localhost> On Fri, 2005-02-11 at 14:35 +0000, Christoph G?ldi wrote: > the same behavior with sending port 111 and port 123 instead of port 53. Indeed, as you'll find if you look at the code I pointed out: those are assumed to be the Portmapper and NTP. Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From vern at icir.org Fri Feb 11 08:41:19 2005 From: vern at icir.org (Vern Paxson) Date: Fri, 11 Feb 2005 08:41:19 -0800 Subject: [Bro] udp_reply event instead of supposed udp_request event In-Reply-To: Your message of Fri, 11 Feb 2005 14:34:03 +0100. Message-ID: <200502111641.j1BGfJp9034580@jaguar.icir.org> > my question is: why does bro recognizes udp_reply events and not udp_request > events? the packets were only sent from one host to another and there > were no packets in the opposite direction. > > i know that udp packets from port 53 are often dns replies but an > assumption which is made because of the application layer protocol > shouldn't have any impact on events on the transport protocol layer... Shouldn't - yes, that would be ideal. But in complex environments where you don't necessarily see both sides of a request/response (due to reordering caused by dual NICs, or multipathing, or drops, or "cold start" where the request happened before Bro began running), it's proven beneficial to infer directionality based on well-known ports. It would be reasonable to add a script variable that turns this off, if you want to contribute a patch. Vern From rpang at CS.Princeton.EDU Fri Feb 11 11:46:16 2005 From: rpang at CS.Princeton.EDU (Ruoming Pang) Date: Fri, 11 Feb 2005 14:46:16 -0500 Subject: [Bro] udp_reply event instead of supposed udp_request event In-Reply-To: <1108134142.16852.34.camel@localhost> References: <420CB44B.9090702@ee.ethz.ch> <1108134142.16852.34.camel@localhost> Message-ID: <97A04B1C-7C65-11D9-8E05-000D9335A7D8@cs.princeton.edu> >> my question is: why does bro recognizes udp_reply events and not >> udp_request >> events? the packets were only sent from one host to another and there >> were no packets in the opposite direction. > > It's hardcoded. Sessions.cc, around 1247. Yes, Bro currently tries to guess which port is the service port, because it may not see the complete connection, for example, it may miss the initial DNS request. What Bro really should do is to look at the packet contents in addition to port numbers in its guessing. We have been puzzled, too, by non-DNS packets with source port 53 (the source port was probably selected to fool firewalls). I don't know if anyone is working on this kind of content-based port selection, but for now, the problem can be circumvented by tweaking NetSessions::IsLikelyServerPort(). Ruoming From vern at icir.org Fri Feb 11 11:56:14 2005 From: vern at icir.org (Vern Paxson) Date: Fri, 11 Feb 2005 11:56:14 -0800 Subject: [Bro] udp_reply event instead of supposed udp_request event In-Reply-To: Your message of Fri, 11 Feb 2005 14:46:16 EST. Message-ID: <200502111956.j1BJuECN041973@jaguar.icir.org> > What Bro really should do is to look at > the packet contents in addition to port numbers in its guessing. Yes, exactly. > I don't know if > anyone is working on this kind of content-based port selection I believe some folks at TU Munich are starting to work on this - Robin? Vern From sommer at in.tum.de Fri Feb 11 12:25:49 2005 From: sommer at in.tum.de (Robin Sommer) Date: Fri, 11 Feb 2005 21:25:49 +0100 Subject: [Bro] udp_reply event instead of supposed udp_request event In-Reply-To: <200502111956.j1BJuECN041973@jaguar.icir.org> References: <200502111956.j1BJuECN041973@jaguar.icir.org> Message-ID: <20050211202549.GB3152@net.informatik.tu-muenchen.de> On Fri, Feb 11, 2005 at 11:56 -0800, Vern Paxson wrote: > I believe some folks at TU Munich are starting to work on this - Robin? Right, a student here is going to tackle this. Our goal is to provide Bro with the ability to decide dynamically which protocol analyzer is appropiate (and, if required, to take the decision back) I believe that this will become very powerful. Robin -- Robin Sommer * Room 01.08.055 * www.net.in.tum.de TU Muenchen * Phone (089) 289-18006 * sommer at in.tum.de From christian at whoop.org Sun Feb 13 08:25:47 2005 From: christian at whoop.org (Christian Kreibich) Date: Sun, 13 Feb 2005 16:25:47 +0000 Subject: [Bro] udp_reply event instead of supposed udp_request event In-Reply-To: <20050211202549.GB3152@net.informatik.tu-muenchen.de> References: <200502111956.j1BJuECN041973@jaguar.icir.org> <20050211202549.GB3152@net.informatik.tu-muenchen.de> Message-ID: <1108311947.22577.68.camel@localhost> On Fri, 2005-02-11 at 21:25 +0100, Robin Sommer wrote: > On Fri, Feb 11, 2005 at 11:56 -0800, Vern Paxson wrote: > > > I believe some folks at TU Munich are starting to work on this - Robin? > > Right, a student here is going to tackle this. Our goal is to > provide Bro with the ability to decide dynamically which protocol > analyzer is appropiate (and, if required, to take the decision back) > I believe that this will become very powerful. A few folks in our group + Intel have recently done work on traffic classifiers along those lines, comparing content-based vs. header-only learners, bayesian nets etc. Fun stuff: http://www.cl.cam.ac.uk/users/awm22/publication/ Cheers, Christian. -- ________________________________________________________________________ http://www.cl.cam.ac.uk/~cpk25 http://www.whoop.org From copaseticsituation at gmail.com Mon Feb 14 19:51:45 2005 From: copaseticsituation at gmail.com (Jeff) Date: Mon, 14 Feb 2005 22:51:45 -0500 Subject: [Bro] New install and looking for help. Message-ID: Hi, I'm trying to install bro on OpenBSD 3.6 and receiving the following error during the make. Also I've trie to install on FreeBSD on 5.3. No Luck with either. OpenBSD error: bash-2.05b# make g++ -I. -Ilibedit -O -c PktSrc.cc PktSrc.cc: In method `void PktSrc::SetHdrSize()': PktSrc.cc:239: `DLT_LINUX_SLL' undeclared (first use this function) PktSrc.cc:239: (Each undeclared identifier is reported only once PktSrc.cc:239: for each function it appears in.) *** Error code 1 Stop in /home/jsa/bro/bro-pub-0.8a88 (line 350 of Makefile). FreeBSD error: Nesium# make g++ -I. -Ilibedit -O -c main.cc In file included from PacketFilter.h:9, from Sessions.h:29, from RuleMatcher.h:12, from main.cc:54: PrefixTable.h:48: error: struct PrefixTable::iterator redeclared with different access main.cc: In function `int main(int, char**)': main.cc:317: error: array bound forbidden after parenthesized type-id main.cc:317: note: try removing the parentheses around the type-id *** Error code 1 Stop in /usr/home/jsa/Bro/bro-pub-0.8a88. I have not been able to find very much information on troubleshooting the install. Appreciate the help! Jeff From vern at icir.org Mon Feb 14 20:03:48 2005 From: vern at icir.org (Vern Paxson) Date: Mon, 14 Feb 2005 20:03:48 -0800 Subject: [Bro] New install and looking for help. In-Reply-To: Your message of Mon, 14 Feb 2005 22:51:45 EST. Message-ID: <200502150403.j1F43mZu022112@jaguar.icir.org> > I'm trying to install bro on OpenBSD 3.6 and receiving the following > error during the make. Also I've trie to install on FreeBSD on 5.3. These are both fixed with the upcoming release - please hang on for a little bit until I can get it out the door. I'm still fixing some bugs recently introduced into it :-(. Vern From goeldich at ee.ethz.ch Tue Feb 15 01:32:21 2005 From: goeldich at ee.ethz.ch (Christoph Goeldi) Date: Tue, 15 Feb 2005 10:32:21 +0100 Subject: [Bro] question: memory requirements of tables Message-ID: <4211C1A5.3060902@ee.ethz.ch> hi sorry, but i have already another question. i noticed that my bro (version 0.9a8.17) eats up my memory. even with the reduce-memory.bro script it needed up to 2GB or more after a while and swapped all the time. when we reduced the traffic the memory usage decreased only a little bit, but remained very high. because we use &*_expire tags and scheduler to keep tables small, we tried to reproduce our problem with a small script: > ### memory_test.bro > > # check all 10 secs if table entries are expired > redef table_expire_interval = 10 secs; > > # When expiring, don't check more than this many table entries at a time. > redef table_expire_size = 10000; > > global test_table: table[addr] of count &create_expire=30secs; > > > event print_table () { > print "size of test_table in bytes", val_size(test_table); > print "length of test_table", length(test_table); > print "test_table", test_table; > schedule 1 min { print_table() }; > } > > event bro_init () { > schedule 1 min { print_table() }; > } > > event udp_request(u: connection) { > test_table[u$id$resp_h] = 0; > } the following is the output of our script when doing some udp scanning. this logs shows that all entries in the table have been deleted, but the table uses still a lot of memory! what happened. we tried different constellations and noticed that it doesn't matter if the table entries have been deleted with the "delete" function or the expire tag. we also tested it over a longer period (let it running through the night) but the memory will not be deallocated. > size of test_table in bytes, 304 > length of test_table, 0 > test_table, { > > } > size of test_table in bytes, 760 > length of test_table, 3 > test_table, { > [x.x.228.20] = 0, > [x.x.229.229] = 0, > [x.x.229.5] = 0 > } > size of test_table in bytes, 768 > length of test_table, 1 > test_table, { > [x.x.229.229] = 0 > } > size of test_table in bytes, 840 > length of test_table, 1 > test_table, { > [x.x.229.229] = 0 > } > size of test_table in bytes, 1675832 > length of test_table, 12545 > test_table, { > [93.172.138.23] = 0, > [60.114.97.75] = 0, > [4.103.6.3] = 0, > [51.164.85.16] = 0, > [85.83.252.215] = 0, > ... > ... > } > size of test_table in bytes, 4285296 > length of test_table, 26532 > test_table, { > [114.68.42.217] = 0, > [4.48.74.147] = 0, > [58.127.37.141] = 0, > ... > ... > } > size of test_table in bytes, 1526280 > length of test_table, 3 > test_table, { > [x.x.229.229] = 0, > [x.x.228.20] = 0, > [x.x.229.5] = 0 > } > size of test_table in bytes, 1526384 > length of test_table, 4 > test_table, { > [x.x.201.77] = 0, > [x.x.201.145] = 0, > [x.x.201.49] = 0, > [x.x.201.52] = 0 > } > size of test_table in bytes, 1526072 > length of test_table, 1 > test_table, { > [x.x.201.70] = 0 > } > size of test_table in bytes, 1525968 > length of test_table, 0 > test_table, { > > } > size of test_table in bytes, 1526072 > length of test_table, 1 > test_table, { > [x.x.229.229] = 0 > } > size of test_table in bytes, 1526072 > length of test_table, 1 > test_table, { > [x.x.229.229] = 0 > } cheers christoph From hdreger at net.in.tum.de Tue Feb 15 02:36:42 2005 From: hdreger at net.in.tum.de (Holger Dreger) Date: Tue, 15 Feb 2005 11:36:42 +0100 Subject: [Bro] question: memory requirements of tables In-Reply-To: <4211C1A5.3060902@ee.ethz.ch> References: <4211C1A5.3060902@ee.ethz.ch> Message-ID: <20050215103642.GG31328@net.informatik.tu-muenchen.de> Hi Christoph, On Tue, Feb 15, 2005 at 10:32:21AM +0100, Christoph Goeldi wrote: > i noticed that my bro (version 0.9a8.17) eats up my memory. > even with the reduce-memory.bro script it needed up to 2GB > or more after a while and swapped all the time. > when we reduced the traffic the memory usage decreased only > a little bit, but remained very high. Have you verified that the memory is indeed consumed by your scripts? Just load the profiling policy script. It outputs some very useful resource consumption logs. The only problem: you do not want to run it online "in operative mode" since it causes bro to use lots of CPU every time it generates its statistics. But anyway, it works for debugging purposes. > because we use &*_expire tags and scheduler to keep tables > small, we tried to reproduce our problem with a small script: > ... > the following is the output of our script when doing some udp scanning. > this logs shows that all entries in the table have been deleted, but the > table uses still a lot of memory! what happened. mmmh, I'm really not sure whether what you see in your script is really your problem of running out of memory. The table first grows but then again shrinks (0->4M->1.5M). I think that suggests that there is not really a larger problem with table expiration. Hope this helps, Holger From goeldich at ee.ethz.ch Tue Feb 15 06:03:29 2005 From: goeldich at ee.ethz.ch (Christoph Goeldi) Date: Tue, 15 Feb 2005 15:03:29 +0100 Subject: [Bro] icmp_time_exceeded Message-ID: <42120131.6040706@ee.ethz.ch> hi there what does the icmp_time_exceeded event mean? an event which is called when an icmp_request (ping) times out? it doesn't work for me then. or how long does it take until an icmp timeout is exceeded? i didn't find a variable for this. thanx christoph From hdreger at net.in.tum.de Tue Feb 15 10:17:39 2005 From: hdreger at net.in.tum.de (Holger Dreger) Date: Tue, 15 Feb 2005 19:17:39 +0100 Subject: [Bro] question: memory requirements of tables In-Reply-To: <1108475816.4211ffa878b29@email.ee.ethz.ch> References: <4211C1A5.3060902@ee.ethz.ch> <20050215103642.GG31328@net.informatik.tu-muenchen.de> <1108475816.4211ffa878b29@email.ee.ethz.ch> Message-ID: <20050215181739.GH31328@net.informatik.tu-muenchen.de> I put this back onto the mailing list, since I think we found again an interesting memory issue... On Tue, Feb 15, 2005 at 01:56:56PM +0000, Christoph Goeldi wrote: > i have no clue what most of the output of the profiling script means, > but this is interesting: > > the test_table really seems to decrease its memory usage massively. > but why not proportionally to the containing entries? > when i have a lot of tables it is really a problem, that each table > consumes 3 MB even if there are (almost) no entries in it anymore... we just quickly checked the code: tables are only resized when they grow beyond a certain threshold. They are not resized when enough elements are deleted. So far this was not a problem since there are usually a few tables that grow large (and usually remain large). You run into problems as soon as you have tables that grow and shrink only once in their lifetime. > your right, there is a second problem: > even if the tables decrease its memory consumption massively bro does not > seem to deallocate this memory space. > > have a look at this log file from the profiling script: > > ..... > > 1108468702.534722 Memory: total=28449K total_adj=27700K malloced: 4842K > ..... This line from the profiling.log file indicates that the system accounts 28M memory allocated by the bro process (field total). On the other hand the malloc library claims bro actually uses only 5M (field malloced). The difference is memory that is not (yet) returned to the system (by the malloc lib!) but will be reused as soon as you reallocate memory. On some (all?) systems you can somehow configure the malloc lib to return freed memory to the system. Holger From vern at icir.org Tue Feb 15 23:48:45 2005 From: vern at icir.org (Vern Paxson) Date: Tue, 15 Feb 2005 23:48:45 -0800 Subject: [Bro] icmp_time_exceeded In-Reply-To: Your message of Tue, 15 Feb 2005 15:03:29 +0100. Message-ID: <200502160748.j1G7mjEF062736@jaguar.icir.org> > what does the icmp_time_exceeded event mean? It's its own ICMP message (it indicated a datagram whose TTL expired, so for example traceroute uses these) - it does not have any relationship to other ICMP's timing out. Vern From goeldich at ee.ethz.ch Wed Feb 16 01:27:19 2005 From: goeldich at ee.ethz.ch (Christoph Goeldi) Date: Wed, 16 Feb 2005 09:27:19 +0000 Subject: [Bro] icmp_time_exceeded In-Reply-To: <200502160748.j1G7mjEF062736@jaguar.icir.org> References: <200502160748.j1G7mjEF062736@jaguar.icir.org> Message-ID: <1108546039.421311f7ce69c@email.ee.ethz.ch> hi vern isn't there a possibility (an event) to recognize icmp requests dropped by the firewall. like the event connection_attempt in case of tcp. for example this would be useful to detect the welchia worm which scans for victims via icmp. thanx christoph Zitat von Vern Paxson : > > what does the icmp_time_exceeded event mean? > > It's its own ICMP message (it indicated a datagram whose TTL expired, so > for example traceroute uses these) - it does not have any relationship to > other ICMP's timing out. > > Vern > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From jean-philippe.luiggi at sagem.com Wed Feb 16 07:20:03 2005 From: jean-philippe.luiggi at sagem.com (Jean-Philippe LUIGGI) Date: Wed, 16 Feb 2005 16:20:03 +0100 Subject: [Bro] =?iso-8859-1?Q?R=E9f=2E_=3A_[Bro]_New_install_and_looking_for_help=2E?= Message-ID: Hello, As Vern said in a mail, Bro is now working on OpenBSD (both 3.6 and current snapshots releases). I think Vern will release the working code with a new version. Jeff cc : (ccc : Jean-Philippe LUIGGI/DADM/SAGEM) Objet : [Bro] New install and looking for help. Envoy? par : bro-admin at ICSI.Berk eley.EDU 15/02/2005 04:51 Veuillez r?pondre ? Jeff Remis le : 15/02/2005 04:52 Hi, I'm trying to install bro on OpenBSD 3.6 and receiving the following error during the make. Also I've trie to install on FreeBSD on 5.3. No Luck with either. OpenBSD error: bash-2.05b# make g++ -I. -Ilibedit -O -c PktSrc.cc PktSrc.cc: In method `void PktSrc::SetHdrSize()': PktSrc.cc:239: `DLT_LINUX_SLL' undeclared (first use this function) PktSrc.cc:239: (Each undeclared identifier is reported only once PktSrc.cc:239: for each function it appears in.) *** Error code 1 Stop in /home/jsa/bro/bro-pub-0.8a88 (line 350 of Makefile). FreeBSD error: Nesium# make g++ -I. -Ilibedit -O -c main.cc In file included from PacketFilter.h:9, from Sessions.h:29, from RuleMatcher.h:12, from main.cc:54: PrefixTable.h:48: error: struct PrefixTable::iterator redeclared with different access main.cc: In function `int main(int, char**)': main.cc:317: error: array bound forbidden after parenthesized type-id main.cc:317: note: try removing the parentheses around the type-id *** Error code 1 Stop in /usr/home/jsa/Bro/bro-pub-0.8a88. I have not been able to find very much information on troubleshooting the install. Appreciate the help! Jeff _______________________________________________ Bro mailing list bro at bro-ids.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From vern at icir.org Wed Feb 16 14:26:22 2005 From: vern at icir.org (Vern Paxson) Date: Wed, 16 Feb 2005 14:26:22 -0800 Subject: [Bro] icmp_time_exceeded In-Reply-To: Your message of Wed, 16 Feb 2005 09:27:19 GMT. Message-ID: <200502162226.j1GMQMIl014443@jaguar.icir.org> > isn't there a possibility (an event) to recognize icmp requests dropped > by the firewall. Do you mean ICMP unreachables with "administratively prohibited" as the subcode? Those should generate icmp_unreachable events *if* the firewall is configured to send the ICMPs (it might instead just silently drop). Vern From vern at icir.org Wed Feb 16 17:09:34 2005 From: vern at icir.org (Vern Paxson) Date: Wed, 16 Feb 2005 17:09:34 -0800 Subject: [Bro] new Bro CURRENT release (0.9a8) Message-ID: <200502170109.j1H19Yei086377@jaguar.icir.org> A new CURRENT release, 0.9a8, is now available from: ftp://ftp.ee.lbl.gov/bro-0.9-current.tar.gz This release includes a large number of changes and bug fixes (appended). Vern -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 0.9a8 - aux/rst/ contains the source for the "rst" tool used by Bro (via the policy script function terminate_connection() in conn.bro) to tear down established connections by forging RST packets. - Bro's main event loop has been reworked (Robin Sommer). This should (1) not cause any visible differences in most cases, (2) improve performance in some cases, (3) fixed problems running Bro without a network input (but still receiving asynchronous input from remote event sources). There are some more changes coming to this soon. - Passive OS fingerprinting has been added, based on Michal Zalewski's "p0f" tool (Holger Dreger). Currently, it's limited to fingerprinting clients based on the initial SYNs they send. To use it, define an event handler: OS_version_found(c: connection, host: addr, OS: OS_version) OS_version is a record containing a string $genre (e.g., "Solaris"), a string $detail (e.g., "2.0.27"), a count $dist (hop-count distance from monitor to host), and $match_type, which specifies via an enumerated type whether the match was direct from a signature, generic to the genre, or "fuzzy". The match is done against a passive fingerprinting signature file, which is specified by the variable passive_fingerprint_file. It defaults to "sigs/p0fsyn", which is found using $BROPATH and has an "osf" suffix added. You can restrict the matching to only be performed for hosts from particular subnets by adding those subnets to the variable generate_OS_version_event. If it's empty (default), then all subnets are analyzed. Note, the passive fingerprinting should be integrated with the version-tracking in software.bro, but this hasn't been done yet. - Support for IPv6 has been repaired and brought up to date. Note, however, that inter-Bro communication currently only works over IPv4. - Signature-matching is now off by default in brolite.bro. If you want to use it, define use_signatures = T prior to @load'ing it. - Notices are now tied to their corresponding connections (Scott Campbell). - New backdoor detectors for IRC, SMTP, Gaobot (Scott Campbell). - Signature matches now have a connection associated with them (Scott Campbell). - Bro scripts that set initial timers (via "schedule" statements in a bro_init handler) but don't have any source of network input (trace files or live interfaces) now execute in real-time, with network_time set to the current time, rather than having their timers expire immediately. - Default timeouts have been added to tables in trw.bro and http.bro, which have been found operationally to potentially grow very large (Scott Campbell). - The new policy script large-conns.bro can be included in order to track the size of TCP connections (each direction is referred to as a "flow") using a secondary packet filter (Chema Gonzalez). This method is completely separate from Bro's usual size accounting, and offers the advantages that it tracks sizes even for connections that don't terminate (or for which Bro misses their establishment) and for connections with sizes > 4 GB. The interface is via the function: function estimate_flow_size_and_remove(cid: conn_id, orig: bool) If $orig=T, then an estimate of the size of the forward (originator) direction is returned. If $orig=F, then the reverse (responder) direction is returned. In both cases, what's returned is a "flow_size_est" record, which includes a flag, $have_est, indicating whether there was any estimate formed. If $have_est is T, then the record also includes an estimate in bounded by $lower (lower bound) and $upper (upper bound). The estimate also includes $num_inconsistent, which, if > 0, means that the estimates came from sequence numbers that were inconsistent, and thus something is wrong - perhaps packet drops by the secondary filter). Finally, calling the function causes the flow's record to be deleted. - An RSH analyzer has been contributed by Manu (ManuX at rstack.org). It generates rsh_request and rsh_reply events, and the following notices: DifferentRSH_Usernames Client and server username differ. FailedRSH_Authentication Attempt to authenticate via RSH failed. InteractiveRSH The RSH session appears to be interactive (multiple lines of user commands). SensitiveRSH_Input SensitiveRSH_Output RSH client input or server output match input_trouble/ full_input_trouble or output_trouble/full_output_trouble. - The new notice action NOTICE_EMAIL indicates that in addition to logging an alarm, it should also be sent out as email (Scott Campbell). By default, email is only sent if Bro is running on live traffic; you can override this via redef'ing the script variable mail_notification. Mail is sent using the script specified by the mail_script variable (default: "mail_script.sh", which is now included in the distribution, but at present is not installed), which must be in $PATH. The mail is sent to the username specified in mail_dest (default: the local "bro" user, though you can change this to name at domain). Note that specifying email as a separate notice action may change in the future, to instead be an attribute that's associated with other notice actions. For example, it may make sense to want to specify both NOTICE_ALARM_PER_CONN and NOTICE_EMAIL; currently, however, you can't do this. - A similar notice action NOTICE_PAGE does the same thing as NOTICE_EMAIL except it send the mail to mail_page_dest (Scott Campbell). - You can now use the attribute &rotate_size for file objects to specify the maximum file size in bytes (Robin Sommer). If the limit is reached, the file is rotated similiarly as is already done with &rotate (which, for consistency, has been renamed to &rotate_interval). For both &rotate_size and &rotate_interval, when they trigger they now generate events (rotate_size and rotate_interval, respectively; each takes the file as the sole argument) rather than invoking &postprocessor, which has been removed. There's also a new variable log_rotate_size to set a global size maximum. Related to log rotation are the following new built-in functions: rotate_file(f: file) closes the file, moves it to a temporary name, and opens a new one. It returns the new "rotate_info" record, which gives the temporary name and the open/close times. rotate_file_by_name(s: string): similar, but call by the name of the file rather than a Bro script value. This is needed because some files are not represented by file objects but need to be rotated nevertheless (most importantly, the tcpdump save file and the dump files for dump_current_packet()). This function rotates the file with the given name. Finally, you can load the new policy script rotate-logs.bro to get default behavior of rotating all log files every hour. - The new "@unload