From rmkml at wanadoo.fr  Thu Apr  1 01:16:09 2004
From: rmkml at wanadoo.fr (rmkml)
Date: Thu, 1 Apr 2004 11:16:09 +0200 (CEST)
Subject: bro 0.8a79 - pb with fbsd49
Message-ID: <Pine.LNX.4.58.0404011112520.9843@hp>

Hi,

I have pb starting bro 08a79

with fbsd 49 :

bash-2.05b$ /usr/local/bin/bro08a79 -i fxp0 mt
/xxx/confL/policy/bro.init, line 748: internal error: built-in
function join_string_array missing

but I don't have pb with 08a70 ...

bro 79 compile syntax :
$ ./configure
$ make

Regards

Rmkml at wanadoo.fr



From rmkml at wanadoo.fr  Thu Apr  1 01:38:51 2004
From: rmkml at wanadoo.fr (rmkml)
Date: Thu, 1 Apr 2004 11:38:51 +0200 (CEST)
Subject: bro 0.8a79 - pb with fbsd49
In-Reply-To: <20040401093911.GA25853@net.informatik.tu-muenchen.de>
References: <Pine.LNX.4.58.0404011112520.9843@hp>
 <20040401093911.GA25853@net.informatik.tu-muenchen.de>
Message-ID: <Pine.LNX.4.58.0404011138100.9843@hp>

Yes solved
Thanks
Rmkml at Wanadoo.fr


On Thu, 1 Apr 2004, Robin Sommer wrote:

> Date: Thu, 1 Apr 2004 11:39:11 +0200
> From: Robin Sommer <sommer at in.tum.de>
> To: rmkml <rmkml at wanadoo.fr>
> Cc: bro at lbl.gov
> Subject: Re: bro 0.8a79 - pb with fbsd49
>
>
> On Thu, Apr 01, 2004 at 11:16 +0200, rmkml wrote:
>
> > bash-2.05b$ /usr/local/bin/bro08a79 -i fxp0 mt
> > /xxx/confL/policy/bro.init, line 748: internal error: built-in
> > function join_string_array missing
> >
> > but I don't have pb with 08a70 ...
>
> Sounds like you're using old policy files. Is your BROPATH set
> correctly to point to the new version?
>
> Robin
>
> --
> Robin Sommer * Room        01.08.055 * www.net.in.tum.de
> TU Munich    * Phone (089) 289-18006 *  sommer at in.tum.de
>


From bpatters at fit.edu  Thu Apr  1 03:55:01 2004
From: bpatters at fit.edu (Bryan)
Date: Thu, 1 Apr 2004 06:55:01 -0500
Subject: reading live traffic
Message-ID: <000001c417e0$2a0f85b0$0501a8c0@BRYANSDESK>

Hello,

I can get bro to read dump captures without any problems...
>bro -r <capture.file> mt ...

but I'm not quite sure on the live traffic

Do I type ">bro -i eth0" ?

Thanks,
Bryan
Florida Tech




From cjwa at umich.edu  Thu Apr  1 04:03:54 2004
From: cjwa at umich.edu (Charles John Wa)
Date: Thu, 1 Apr 2004 07:03:54 -0500 (EST)
Subject: Passing a Table to the policy engine
Message-ID: <Pine.SOL.4.58.0404010703370.17883@millipede.gpcc.itd.umich.edu>


Hi,

I am new to using Bro, and was wondering how i can pass a table from my
event generator (where i have my .cc code) to a policy script.
Specifically, have an array of integers in my C code, and i want to use it
as a table in my Policy script.

I assume that a table of integers should be a
	table[count] of count;
in my policy script.

I tried doing this in my myfile.cc file:

	int pathArr[100];
	val_list* vl = new val_list;
	...
	TableVal* myPath = new TableVal(internal_type("count")->AsTableType());
	for ( int i=0; i < 100; i++ ) {
		myPath->Assign(new Val(i, TYPE_INT), new Val(pathArr[i], TYPE_INT));
	}
	updaterec->Assign(1, myPath);
	vl->append(updaterec);
	mgr.QueueEvent(update, vl, Event::LOCAL, this);

but I beleve that my TableVal* myPath assigmnet is incorrect, as it
does not recognize "count" as a type when i run it.

Any help is appreicated

Charles Wa


From bpatters at fit.edu  Thu Apr  1 04:04:06 2004
From: bpatters at fit.edu (Bryan)
Date: Thu, 1 Apr 2004 07:04:06 -0500
Subject: piping to a text file
Message-ID: <000101c417e1$6eeb95b0$0501a8c0@BRYANSDESK>

Hello again,

I know that you can pipe results to a binary file with the -w flag and
then you can read it with the -r flag using bro again. While running
bro, can you pipe the output to a text file?
i.e. >bro -r <filename> mt >> readResults //where 'readResults' is my
new file

I can't get it to work.

Thanks,
Bryan
Florida Tech




From bpatters at fit.edu  Thu Apr  1 07:26:34 2004
From: bpatters at fit.edu (Bryan Patterson)
Date: Thu, 01 Apr 2004 10:26:34 -0500
Subject: piping to a text file
In-Reply-To: <Pine.LNX.4.58.0404010841110.18538@willy.CS.Princeton.EDU>
References: <000101c417e1$6eeb95b0$0501a8c0@BRYANSDESK>
	 <Pine.LNX.4.58.0404010841110.18538@willy.CS.Princeton.EDU>
Message-ID: <1080833194.6482.7.camel@localhost.localdomain>

Hello,

I am reading tcpdump trace files that my professor gave me. 

[root at localhost bro_files]# bro -r tcpdump/attack_file_8.tcpdump mt -w
scan_results/attack8.scan

I am writing them to 'attack8.scan', but that is a binary file which I
need to read with bro again. On that note, what is the proper
syntax/flag to use when reading bro output "as-is" with the bro
application?

#bro -r attack8.scan <????>

THANKS!
Bryan


On Thu, 2004-04-01 at 08:44, Ruoming Pang wrote:
> > While running bro, can you pipe the output to a text file?
> 
> Hi, Bryan,
> 
> What kind of output do you like to pipe to a text file? Is it tcpdump
> trace or Bro alerts or something else?
> 
> Ruoming
> 
> > i.e. >bro -r <filename> mt >> readResults //where 'readResults' is my
> > new file
> >
> > I can't get it to work.
> >
> > Thanks,
> > Bryan
> > Florida Tech
> >
> >



From bpatters at fit.edu  Thu Apr  1 07:34:32 2004
From: bpatters at fit.edu (Bryan Patterson)
Date: Thu, 01 Apr 2004 10:34:32 -0500
Subject: reading live traffic
In-Reply-To: <406C30BF.1050308@cc.gatech.edu>
References: <000001c417e0$2a0f85b0$0501a8c0@BRYANSDESK>
	 <406C30BF.1050308@cc.gatech.edu>
Message-ID: <1080833672.6482.9.camel@localhost.localdomain>

Thanks!

On Thu, 2004-04-01 at 10:09, Ashley Thomas wrote:
> Yeah.
> 
> bro -i eth0 mt
> 
> Bryan wrote:
> 
> >Hello,
> >
> >I can get bro to read dump captures without any problems...
> >  
> >
> >>bro -r <capture.file> mt ...
> >>    
> >>
> >
> >but I'm not quite sure on the live traffic
> >
> >Do I type ">bro -i eth0" ?
> >
> >Thanks,
> >Bryan
> >Florida Tech
> >
> >  
> >
> 
> 
> 



From bpatters at fit.edu  Thu Apr  1 08:13:08 2004
From: bpatters at fit.edu (Bryan Patterson)
Date: Thu, 01 Apr 2004 11:13:08 -0500
Subject: piping to a text file
In-Reply-To: <Pine.LNX.4.58.0404011029070.18538@willy.CS.Princeton.EDU>
References: <000101c417e1$6eeb95b0$0501a8c0@BRYANSDESK>
	 <Pine.LNX.4.58.0404010841110.18538@willy.CS.Princeton.EDU>
	 <1080833194.6482.7.camel@localhost.localdomain>
	 <Pine.LNX.4.58.0404011029070.18538@willy.CS.Princeton.EDU>
Message-ID: <1080835988.6780.29.camel@localhost.localdomain>

Hello,

> The binary file is in tcpdump format, so you can use tcpdump -r <file> to
> see the trace in text format (try flag -nX). 

Where does the '-nX' flag go? tcpdump or bro? Does it go in place of the
'-r' or appended to it?

> Also if you say 'bro ... -w
> -', the output trace will be dumped to stdout and you can pipe it with
> tcpdump as well.

>From the following line...
	>#bro -r <trace file> mt -w <output file>
How do I do what you are saying above so that I can read the data in a
text reader (vi,gedit,emacs...)?

I am running a bash shell on Fedora/Redhat. 
Sorry, I have only been "officially" running Linux for a few months.

> 
> Does this answer your question? (I don't know what you meant by "as-is"
> though.)
> 
> 
When I said "as-is" I just meant that I didn't want bro to analyze the
data, just repeat it back.

THANKS!
Bryan




From bpatters at fit.edu  Thu Apr  1 08:50:43 2004
From: bpatters at fit.edu (Bryan Patterson)
Date: Thu, 01 Apr 2004 11:50:43 -0500
Subject: piping to a text file
In-Reply-To: <20040401163720.GB9193@sfu.ca>
References: <000101c417e1$6eeb95b0$0501a8c0@BRYANSDESK>
	 <Pine.LNX.4.58.0404010841110.18538@willy.CS.Princeton.EDU>
	 <1080833194.6482.7.camel@localhost.localdomain>
	 <Pine.LNX.4.58.0404011029070.18538@willy.CS.Princeton.EDU>
	 <1080835988.6780.29.camel@localhost.localdomain>
	 <20040401163720.GB9193@sfu.ca>
Message-ID: <1080838243.7134.1.camel@localhost.localdomain>

Thanks for the help!

Bryan

On Thu, 2004-04-01 at 11:37, Peter Van Epp wrote:
> 	tcpdump -s 1510 -nXr <file> > text_file
> 
> will dump up to 1510 bytes (the -s flag and if your capture contains that 
> much, 128 bytes is the default if the capture didn't have a -s flag usually) 
> from "file" (the tcpdump input file with or without bro, if all you want is a 
> printout of the raw packets you don't need bro in the loop). The -n suppresses 
> DNS conversion, the X prints the entire packet in readable form and -r tells 
> it what tcpdump file to read. The "> text_file" redirects the output (which is 
> by default to the screen) in to file "text_file" which you can edit. From the 
> command prompt "man tcpdump" will give you the man page for tcpdump which 
> explains all the various options.
> 
> Peter Van Epp / Operations and Technical Support 
> Simon Fraser University, Burnaby, B.C. Canada
> 
> On Thu, Apr 01, 2004 at 11:13:08AM -0500, Bryan Patterson wrote:
> > Hello,
> > 
> > > The binary file is in tcpdump format, so you can use tcpdump -r <file> to
> > > see the trace in text format (try flag -nX). 
> > 
> > Where does the '-nX' flag go? tcpdump or bro? Does it go in place of the
> > '-r' or appended to it?
> > 
> > > Also if you say 'bro ... -w
> > > -', the output trace will be dumped to stdout and you can pipe it with
> > > tcpdump as well.
> > 
> > >From the following line...
> > 	>#bro -r <trace file> mt -w <output file>
> > How do I do what you are saying above so that I can read the data in a
> > text reader (vi,gedit,emacs...)?
> > 
> > I am running a bash shell on Fedora/Redhat. 
> > Sorry, I have only been "officially" running Linux for a few months.
> > 
> > > 
> > > Does this answer your question? (I don't know what you meant by "as-is"
> > > though.)
> > > 
> > > 
> > When I said "as-is" I just meant that I didn't want bro to analyze the
> > data, just repeat it back.
> > 
> > THANKS!
> > Bryan
> > 



From vern at icir.org  Thu Apr  1 12:17:57 2004
From: vern at icir.org (Vern Paxson)
Date: Thu, 01 Apr 2004 12:17:57 -0800
Subject: Passing a Table to the policy engine
In-Reply-To: Your message of Thu, 01 Apr 2004 07:03:54 EST.
Message-ID: <200404012017.i31KHvCq016556@jaguar.icir.org>

> I tried doing this in my myfile.cc file:
> 
> 	int pathArr[100];
> 	val_list* vl = new val_list;
> 	...
> 	TableVal* myPath = new TableVal(internal_type("count")->AsTableType());
> 	for ( int i=0; i < 100; i++ ) {
> 		myPath->Assign(new Val(i, TYPE_INT), new Val(pathArr[i], TYPE_INT));
> 	}
> 	updaterec->Assign(1, myPath);
> 	vl->append(updaterec);
> 	mgr.QueueEvent(update, vl, Event::LOCAL, this);
> 
> but I beleve that my TableVal* myPath assigmnet is incorrect, as it
> does not recognize "count" as a type when i run it.

You have the right idea.  Two problems I see are: (1) you should use
TYPE_COUNT rather than TYPE_INT (count is unsigned, int is signed), though
that shouldn't matter in practice; (2) I don't understand the line:

	updaterec->Assign(1, myPath);

What is updaterec and why are you assigning element 1 rather than element 0?

		Vern


From cjwa at umich.edu  Thu Apr  1 13:16:44 2004
From: cjwa at umich.edu (Charles John Wa)
Date: Thu, 1 Apr 2004 16:16:44 -0500 (EST)
Subject: Passing a Table to the policy engine
In-Reply-To: <200404012017.i31KHvCq016556@jaguar.icir.org>
References: <200404012017.i31KHvCq016556@jaguar.icir.org>
Message-ID: <Pine.SOL.4.58.0404011600050.6088@timepilot.gpcc.itd.umich.edu>

Hi,


> > 	int pathArr[100];
> > 	val_list* vl = new val_list;
> > 	...
> > 	TableVal* myPath = new TableVal(internal_type("count")->AsTableType());
> > 	for ( int i=0; i < 100; i++ ) {
> > 		myPath->Assign(new Val(i, TYPE_INT), new Val(pathArr[i], TYPE_INT));
> > 	}
> > 	updaterec->Assign(1, myPath);
> > 	vl->append(updaterec);
> > 	mgr.QueueEvent(update, vl, Event::LOCAL, this);
> >
> > but I beleve that my TableVal* myPath assigmnet is incorrect, as it
> > does not recognize "count" as a type when i run it.
>
> You have the right idea.  Two problems I see are: (1) you should use
> TYPE_COUNT rather than TYPE_INT (count is unsigned, int is signed), though
> that shouldn't matter in practice;

Okay, let me explain.  In my bro.init, i have a record:

	type some_rec: record {
		path_table: table[count] of count;
	}


in my myfile.cc code, i tried doing this:

	myFuncInC ()
	{
		printf ("I am in this function\n");
    		int pathArr[100];
    		val_list* vl = new val_list;
     		RecordVal* myrec = new RecordVal(some_rec);
     		...
     		TableVal* myPath = new TableVal(internal_type("count")->AsTableType());
     		for ( int i=0; i < 100; i++ ) {
     		        myPath->Assign(new Val(i, TYPE_COUNT), new Val(pathArr[i], TYPE_COUNT));
     		}
     		myrec->Assign(0, myPath);
     		vl->append(myrec);
     		mgr.QueueEvent(some_event, vl, Event::LOCAL, this);
	}

But i get a runtime error:

%I am in this function
%1078809250.202295 internal error: internal type count missing

I dont think its because of my TYPC_COUNT as opposed to TYPE_INT.  it does
not like me using a count for my table, maybe because i have not defined
it?  It could be my line:

	TableVal* myPath = new TableVal(internal_type("count")->AsTableType());

where I am using internal_type("count")->AsTableType(), which might not
work.  However, i dont know how else to create a table[count] of count
otherwise.

Thanks

Charles




From sommer at in.tum.de  Thu Apr  1 23:58:27 2004
From: sommer at in.tum.de (Robin Sommer)
Date: Fri, 2 Apr 2004 09:58:27 +0200
Subject: Passing a Table to the policy engine
In-Reply-To: <Pine.SOL.4.58.0404011600050.6088@timepilot.gpcc.itd.umich.edu>
References: <200404012017.i31KHvCq016556@jaguar.icir.org> <Pine.SOL.4.58.0404011600050.6088@timepilot.gpcc.itd.umich.edu>
Message-ID: <20040402075827.GA28934@net.informatik.tu-muenchen.de>


On Thu, Apr 01, 2004 at 16:16 -0500, Charles John Wa wrote:

> in my myfile.cc code, i tried doing this:
> 
> 	myFuncInC ()
> 	{
> 		printf ("I am in this function\n");
>     		int pathArr[100];
>     		val_list* vl = new val_list;
>      		RecordVal* myrec = new RecordVal(some_rec);
>      		...
>      		TableVal* myPath = new TableVal(internal_type("count")->AsTableType());
>      		for ( int i=0; i < 100; i++ ) {
>      		        myPath->Assign(new Val(i, TYPE_COUNT), new Val(pathArr[i], TYPE_COUNT));
>      		}
>      		myrec->Assign(0, myPath);
>      		vl->append(myrec);
>      		mgr.QueueEvent(some_event, vl, Event::LOCAL, this);
> 	}
> 
> But i get a runtime error:
> 
> %I am in this function
> %1078809250.202295 internal error: internal type count missing
> 
> I dont think its because of my TYPC_COUNT as opposed to TYPE_INT.  it does
> not like me using a count for my table, maybe because i have not defined
> it?  It could be my line:
> 
> 	TableVal* myPath = new TableVal(internal_type("count")->AsTableType());
> 
> where I am using internal_type("count")->AsTableType(), which might not
> work.  However, i dont know how else to create a table[count] of count
> otherwise.

There are two possibilities:

- you can define a new type in bro.init and then access it:

      bro.init:
            type my_table_count_of_count: table[count] of count;      
      *.cc:             
         	TableVal* myPath = new TableVal(internal_type("my_table_count_of_count")->AsTableType());

- you instantiate a new BroType yourself:

      TypeList *myIndices = new TypeList(base_type(TYPE_COUNT));
      myIndices->Append(base_type(TYPE_COUNT));
      TableType* myType = new TableType(myIndices, base_type(TYPE_COUNT));
      
      TableVal* myPath = new TableVal(myType);
            
(Both are completely untested which means that they probably won't
work out-of-the-box :-)

Robin

-- 
Robin Sommer * Room        01.08.055 * www.net.in.tum.de
TU Munich    * Phone (089) 289-18006 *  sommer at in.tum.de 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20040402/afe83e85/attachment.bin 

From sommer at in.tum.de  Fri Apr  2 00:15:47 2004
From: sommer at in.tum.de (Robin Sommer)
Date: Fri, 2 Apr 2004 10:15:47 +0200
Subject: new bro "CURRENT" release - 0.8a79
In-Reply-To: <Pine.LNX.4.58.0403251232130.24690@anton.netforensics.com>
References: <200403251725.i2PHPdCq024443@jaguar.icir.org> <Pine.LNX.4.58.0403251232130.24690@anton.netforensics.com>
Message-ID: <20040402081546.GB28934@net.informatik.tu-muenchen.de>


On Thu, Mar 25, 2004 at 12:33 -0500, Anton Chuvakin, Ph.D. wrote:

> Just curious, what is the motivation for IDMEF support? Just to be
> consistent with industry "standard" or something else? Basically, I am
> asking how users are supposed to use IDMEF in production enviorment.

From my point of view, there are two main motivations: 

First, it's indeed simply a standardized way to talk to other
systems. If you're using different kinds of NIDSs (either at
different locations or even at the same place), they may share their
results with IDMEF. 

Second, it's interesting to see how Bro's semantics map to IDMEF and
vice versa. Most parts of Bro work on a lower-level than IDMEF. So,
a large fraction of Bro's state is not (reasonably) convertible to
IDMEF. On the other hand, Bro's alert framework looks quite similar
to IDMEF's model. By adding IDMEF support we should be able to
better understand what kind of information can actually be
represented in this format (and if it's sufficient for the task its
supposed to do).

Regarding the question how to use it: if you want to connect
multiple Bros, IDMEF is probably not the best way; there are other
mechanisms now (which are still experimental though). But if you
want to share alerts with other systems, IDMEF could be an option.

Robin

-- 
Robin Sommer * Room        01.08.055 * www.net.in.tum.de
TU Munich    * Phone (089) 289-18006 *  sommer at in.tum.de 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20040402/b802007b/attachment.bin 

From cjwa at umich.edu  Fri Apr  2 10:43:05 2004
From: cjwa at umich.edu (Charles John Wa)
Date: Fri, 2 Apr 2004 13:43:05 -0500 (EST)
Subject: Passing a Table to the policy engine
In-Reply-To: <20040402075827.GA28934@net.informatik.tu-muenchen.de>
References: <200404012017.i31KHvCq016556@jaguar.icir.org>
 <Pine.SOL.4.58.0404011600050.6088@timepilot.gpcc.itd.umich.edu>
 <20040402075827.GA28934@net.informatik.tu-muenchen.de>
Message-ID: <Pine.SOL.4.58.0404021342090.20270@timepilot.gpcc.itd.umich.edu>


>
> There are two possibilities:
>
> - you can define a new type in bro.init and then access it:
>
>       bro.init:
>             type my_table_count_of_count: table[count] of count;
>       *.cc:
>          	TableVal* myPath = new TableVal(internal_type("my_table_count_of_count")->AsTableType());

This works right out of the box!  thanks!

charles

>
> - you instantiate a new BroType yourself:
>
>       TypeList *myIndices = new TypeList(base_type(TYPE_COUNT));
>       myIndices->Append(base_type(TYPE_COUNT));
>       TableType* myType = new TableType(myIndices, base_type(TYPE_COUNT));
>
>       TableVal* myPath = new TableVal(myType);
>
> (Both are completely untested which means that they probably won't
> work out-of-the-box :-)
>
> Robin
>
> --
> Robin Sommer * Room        01.08.055 * www.net.in.tum.de
> TU Munich    * Phone (089) 289-18006 *  sommer at in.tum.de
>


From yohann.thomas at rd.francetelecom.com  Tue Apr  6 07:38:33 2004
From: yohann.thomas at rd.francetelecom.com (Yohann Thomas)
Date: Tue, 06 Apr 2004 16:38:33 +0200
Subject: Getting matched substrings ???
Message-ID: <4072C0E9.4080308@rd.francetelecom.com>

Hi !

I'm having a look at Bro and I'd like first to congratulate people 
involved in the project for this great work !!!

The concept of contextual signature language seems very interesting, but 
I'm having a little problem...In fact, I read in the paper "Bro: A 
System for Detecting Network Intruders in Real-Time" this phrase about 
REGEX implementation : "Second, we anticipate matching sets of patterns 
and wanting to know which subset were matched by a given set of 
text...". I thought I could get the matched substring by the signatures, 
but unfortunately I can't get out of it...

Is it possible to get these substrings in a policy script when a 
signature matches, or am I misunderstanding the quoted phrase ???

Thanks.

Yohann.





From sommer at in.tum.de  Tue Apr  6 09:02:50 2004
From: sommer at in.tum.de (Robin Sommer)
Date: Tue, 6 Apr 2004 18:02:50 +0200
Subject: Getting matched substrings ???
In-Reply-To: <4072C0E9.4080308@rd.francetelecom.com>
References: <4072C0E9.4080308@rd.francetelecom.com>
Message-ID: <20040406160250.GA11229@net.informatik.tu-muenchen.de>


On Tue, Apr 06, 2004 at 16:38 +0200, Yohann Thomas wrote:

> text...". I thought I could get the matched substring by the signatures, 
> but unfortunately I can't get out of it...

event signature_match(state: signature_state, msg: string, data: string)

The 'data' parameter of the signature_match event contains the
payload that lead to the match. (More precisely, it contains the
last chunk of payload that eventually triggered the match; for TCP,
it depends on the reassembly what exactly is passed).

Is this what you're looking for?

Robin

-- 
Robin Sommer * Room        01.08.055 * www.net.in.tum.de
TU Munich    * Phone (089) 289-18006 *  sommer at in.tum.de 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20040406/57b5d200/attachment.bin 

From vern at icir.org  Tue Apr  6 09:52:53 2004
From: vern at icir.org (Vern Paxson)
Date: Tue, 06 Apr 2004 09:52:53 -0700
Subject: Getting matched substrings ???
In-Reply-To: Your message of Tue, 06 Apr 2004 16:38:33 +0200.
Message-ID: <200404061652.i36GqrCq074686@jaguar.icir.org>

> I read in the paper "Bro: A 
> System for Detecting Network Intruders in Real-Time" this phrase about 
> REGEX implementation : "Second, we anticipate matching sets of patterns 
> and wanting to know which subset were matched by a given set of 
> text...". I thought I could get the matched substring by the signatures, 
> but unfortunately I can't get out of it...

(That text refers to regular-expression matching on general strings, rather
 than the context-based signature analyzer that Robin added to Bro, by the
 way.)

Since writing that, Bro's style has moved more towards pushing extraction
of elements into either the event engine itself, or into built-in functions,
rather than trying to do it using regular expressions over strings.  If it
were easy to add subexpressions to Bro's RE matcher, I'd be happy to do so,
but it's quite a bit of work.

If you give an example of where you want to do this, perhaps we can suggest
alternate ways of structuring your analyzer.

		Vern


From yohann.thomas at rd.francetelecom.com  Wed Apr  7 00:17:50 2004
From: yohann.thomas at rd.francetelecom.com (Yohann Thomas)
Date: Wed, 07 Apr 2004 09:17:50 +0200
Subject: Getting matched substrings ???
In-Reply-To: <200404061652.i36GqrCq074686@jaguar.icir.org>
References: <200404061652.i36GqrCq074686@jaguar.icir.org>
Message-ID: <4073AB1E.2090805@rd.francetelecom.com>

Robin Sommer wrote:

>On Tue, Apr 06, 2004 at 16:38 +0200, Yohann Thomas wrote:
>
>  
>
>>text...". I thought I could get the matched substring by the signatures, 
>>but unfortunately I can't get out of it...
>>    
>>
>
>event signature_match(state: signature_state, msg: string, data: string)
>
>The 'data' parameter of the signature_match event contains the
>payload that lead to the match. (More precisely, it contains the
>last chunk of payload that eventually triggered the match; for TCP,
>it depends on the reassembly what exactly is passed).
>
>Is this what you're looking for?
>
>Robin
>
>  
>

Vern Paxson wrote:

> > I read in the paper "Bro: A
> > System for Detecting Network Intruders in Real-Time" this phrase about
> > REGEX implementation : "Second, we anticipate matching sets of patterns
> > and wanting to know which subset were matched by a given set of
> > text...". I thought I could get the matched substring by the 
> signatures,
> > but unfortunately I can't get out of it...
>
> (That text refers to regular-expression matching on general strings, 
> rather
>  than the context-based signature analyzer that Robin added to Bro, by 
> the
>  way.)
>
> Since writing that, Bro's style has moved more towards pushing extraction
> of elements into either the event engine itself, or into built-in 
> functions,
> rather than trying to do it using regular expressions over strings.  
> If it
> were easy to add subexpressions to Bro's RE matcher, I'd be happy to 
> do so,
> but it's quite a bit of work.
>
> If you give an example of where you want to do this, perhaps we can 
> suggest
> alternate ways of structuring your analyzer.
>
>                 Vern
>

In fact, I use the "data" parameter at the moment to get the whole 
payload, but the real idea was to get only the part that matched.
Here is a simple example of what I'd like to do :

*signature apache-server {
    ip-proto == tcp
    src-port == 80
    payload /Server: [aA][pP][aA][cC][hH][eE].***/
    event "Apache"
    tcp-state responder
}

Then, in a policy script, I thought I could get "Apache//version/", 
using the function sub_bytes(), associated to the IP@ of the host 
(contained in the signature_state). It was an easy way to know that the 
information I needed was 8 characters ("Server: ") after the beginning 
of the matched substring.

*To sum up, I'd like to get some hosts characteristics like : *this host 
(IP@ W.X.Y.Z) is now running Apache 1.3.29*.
*
*But, first, I noticed that it's not possible to match this REGEX. In 
fact, I have to add "*.**" at the beginning of the pattern (which is not 
necessary with PCRE). Then, since you're explaining me it's not possible 
to get the matched substring, there's no more reason for me to pursue 
this way. So, I have to find something else...

Any ideas are welcome !!!

Yohann.






From sommer at in.tum.de  Wed Apr  7 00:38:57 2004
From: sommer at in.tum.de (Robin Sommer)
Date: Wed, 7 Apr 2004 09:38:57 +0200
Subject: Getting matched substrings ???
In-Reply-To: <4073AB1E.2090805@rd.francetelecom.com>
References: <200404061652.i36GqrCq074686@jaguar.icir.org> <4073AB1E.2090805@rd.francetelecom.com>
Message-ID: <20040407073857.GA13198@net.informatik.tu-muenchen.de>


On Wed, Apr 07, 2004 at 09:17 +0200, Yohann Thomas wrote:

> *To sum up, I'd like to get some hosts characteristics like : *this host 
> (IP@ W.X.Y.Z) is now running Apache 1.3.29*.

This sounds exactly like what software.bro is doing. Have you tried
that? (You also need to load http-reply.bro as it doesn't use the
signature engine but the HTTP decoder).

Robin

-- 
Robin Sommer * Room        01.08.055 * www.net.in.tum.de
TU Munich    * Phone (089) 289-18006 *  sommer at in.tum.de 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20040407/e74b66f4/attachment.bin 

From yohann.thomas at rd.francetelecom.com  Wed Apr  7 02:10:48 2004
From: yohann.thomas at rd.francetelecom.com (Yohann Thomas)
Date: Wed, 07 Apr 2004 11:10:48 +0200
Subject: Getting matched substrings ???
In-Reply-To: <20040407073857.GA13198@net.informatik.tu-muenchen.de>
References: <200404061652.i36GqrCq074686@jaguar.icir.org> <4073AB1E.2090805@rd.francetelecom.com> <20040407073857.GA13198@net.informatik.tu-muenchen.de>
Message-ID: <4073C598.6050108@rd.francetelecom.com>

Robin Sommer wrote:

>On Wed, Apr 07, 2004 at 09:17 +0200, Yohann Thomas wrote:
>
>  
>
>>*To sum up, I'd like to get some hosts characteristics like : *this host 
>>(IP@ W.X.Y.Z) is now running Apache 1.3.29*.
>>    
>>
>
>This sounds exactly like what software.bro is doing. Have you tried
>that? (You also need to load http-reply.bro as it doesn't use the
>signature engine but the HTTP decoder).
>
>Robin
>
>  
>
Sounds great in fact !!! I've just tested it, and it will help me for 
sure. Thanks !!!

Now, next stage will be to get other information contained in URLs, like 
sensitive CGIs.
I've just seen http-request.bro implements such features, so I'm having 
a look at it...

Yohann.



From dul at netpower.com.cn  Thu Apr  8 17:51:11 2004
From: dul at netpower.com.cn (dul)
Date: Fri, 9 Apr 2004 08:51:11 +0800
Subject: record
Message-ID: <OAENIHPPKANAHAJGFEDDAEEBCBAA.dul@netpower.com.cn>



>>Vern,
     I find that in parse.y,you write like this
"decl:
		??
	|	TOK_TYPE global_id ':' refined_type opt_attr opt_attr_attr';'
"
"refined_type:
		type_list '{' type_decl_list '}' 
"

Why is there a 'type_list' here but a type???

Because in the manual,we define a record like this
"type conn_id : record{
				orig_h : addr ; # Address of originating host.
				Orig_p: port; # Port used by originator.
				Resp_h: addr; # Address of responding host.
				Resp_p : port; # Port used by responder.
		};
"
There  is only one record here ???


Please help me!
               

Tad



From vern at icir.org  Thu Apr  8 19:54:21 2004
From: vern at icir.org (Vern Paxson)
Date: Thu, 08 Apr 2004 19:54:21 -0700
Subject: record
In-Reply-To: Your message of Fri, 09 Apr 2004 08:51:11 +0800.
Message-ID: <200404090254.i392sLCq070859@jaguar.icir.org>

> "refined_type:
> 		type_list '{' type_decl_list '}' 
> "
> 
> Why is there a 'type_list' here but a type???

Good eye!

That's a holdover from when I was experimenting with adding a form of
inheritence to Bro records.  You should ignore it for now.

		Vern


From liaozj at netpower.com.cn  Fri Apr  9 02:59:05 2004
From: liaozj at netpower.com.cn (=?gb2312?B?wc7Vwr78?=)
Date: Fri, 9 Apr 2004 17:59:05 +0800
Subject: BroString
Message-ID: <200404091749125.SM00876@RavProxy>

broVern,
    I find that in the scan.l there is a regular expression
              \"([^\\\n\"]|{ESCSEQ})*\"
    I know that this expression is used for matching a BroString.But what does it mean?Does it mean there should not be a '\n' or '\' in the string?I try "liao\nliao" and "lsjf\ljf",but they run just ok.
    By the way,what is Serializer.h/.cc used for? Only for efficiency?
   
    Thanks!

Tad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: face-1.gif
Type: image/gif
Size: 922 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20040409/1906901e/attachment.gif 

From liaozj at netpower.com.cn  Mon Apr 12 23:45:49 2004
From: liaozj at netpower.com.cn (Tad)
Date: Tue, 13 Apr 2004 14:45:49 +0800
Subject: execuse me
Message-ID: <20040413143693.SM00860@RavProxy>

Vern,
   Sorry to ask you such a broad question .Would you help me another one?

   I find that in the scan.l there is a regular expression
              \"([^\\\n\"]|{ESCSEQ})*\"
I know that this expression is used for matching a BroString.But what does it mean?Does it mean there should not be a '\n' or '\' in the string?I try "liao\nliao" and "lsjf\ljf",but they run just ok.
   Thank you!
Tad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: face-1.gif
Type: image/gif
Size: 922 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20040413/0409031c/attachment.gif 

From vern at icir.org  Mon Apr 12 23:59:34 2004
From: vern at icir.org (Vern Paxson)
Date: Mon, 12 Apr 2004 23:59:34 -0700
Subject: execuse me
In-Reply-To: Your message of Tue, 13 Apr 2004 14:45:49 +0800.
Message-ID: <200404130659.i3D6xYCq007113@jaguar.icir.org>

>               \"([^\\\n\"]|{ESCSEQ})*\"
> ... Does it mean there should not be a '\n' or '\' in the string?

Please read the flex manual to resolve questions like this one.  For scan.l,
ESCSEQ allows '\n' and '\' to appear in strings.

		Vern


From srg018 at latech.edu  Wed Apr 21 18:11:58 2004
From: srg018 at latech.edu (Shekhar Reddy Gaddam)
Date: Wed, 21 Apr 2004 20:11:58 -0500
Subject: loading analyzer
Message-ID: <1082596318.40871bde5b1f6@webmail.LaTech.edu>



Dear All,

How I can load analyzers using bro @load directive. I'd like analyse hot attacks
in tcpdump file.

Regards,
Shekhar

-------------------------------------------------
This mail sent through IMP: http://horde.org/imp/



From zangds at netpower.com.cn  Thu Apr 22 23:44:49 2004
From: zangds at netpower.com.cn (=?gb2312?B?6rC2rMvJ?=)
Date: Fri, 23 Apr 2004 14:44:49 +0800
Subject: Bro's problems:Out of memory !
Message-ID: <007d01c428fe$798dd900$7400a8c0@netpowerdonal>

Hi,Vern,
    We are intresting in Bro.But when we test it under a simulative network,
it was killed a few minutes by the linux systerm,giving the message of"
Out of memory",while Bro 's message is "internal error:double signal".
    Besides considering the reasons of the traffic speed(about 300 http 
connections per sec) and the hardware we used,I wonder wherether there's 
some ways that Bro used to control it's memory allocation?
    I notice that in util.cc there's something like:

#ifdef malloc
...
extern "C" {
void* malloc(size_t);
void* realloc(void*, size_t);
void free(void*);
}
void* operator new(size_t t);
.
.
.
    Is it used to redefine the "new" and "delete" operator?Has that been used in Bro now?
Please give me some suggestions to resolve this.Thanks alot!

Donal.



From hdreger at net.in.tum.de  Fri Apr 23 01:46:06 2004
From: hdreger at net.in.tum.de (Holger Dreger)
Date: Fri, 23 Apr 2004 10:46:06 +0200
Subject: Bro's problems:Out of memory
Message-ID: <20040423084606.GB13169@net.informatik.tu-muenchen.de>

On Fri, Apr 23, 2004 at 02:44:49PM +0800, ?????? wrote:
>     We are intresting in Bro.But when we test it under a simulative network,
> it was killed a few minutes by the linux systerm,giving the message of"
> Out of memory",while Bro 's message is "internal error:double signal".

The amount of memory bro uses heavily depends on the policy scripts
that you are running. If you additionally load the script
statistics.bro you'll get a statistics.log file which should provide
you information where the all the memory has gone. One common trick is
to tune the various timeouts like the script reduce-memory.bro does.
(Note: reduce memory sets timeouts that are perhaps not suitable to
your needs)

Holger

--
Holger Dreger * http://www.net.in.tum.de * Phone: +49 (0)89 289-18006
   Computer Science Department * Technische Universitaet Muenchen


From yohann.thomas at rd.francetelecom.com  Fri Apr 23 06:54:49 2004
From: yohann.thomas at rd.francetelecom.com (Yohann Thomas)
Date: Fri, 23 Apr 2004 15:54:49 +0200
Subject: Log entire payloads
Message-ID: <40892029.2000304@rd.francetelecom.com>

Hi everybody !!!

I'd like to log http payloads for each connection seen on my network.

In fact, I'd like to get something like :
Src_IP;Dst_IP;Request_Payload;Reply_Payload

but with entire payloads (not only URIs, but also banners...)

At the moment, the only way I found to manage that is to load the 
signature module and write a signature file using payload /.*/, in order 
to get the payloads on signature_match events with the data string.

It works, but unfortunately, this solution is a bit heavy in term of CPU 
usage. There's probably a way to get these payloads using a built-in 
function (and avoiding signatures module) ??? In fact, having a look at 
the http-related modules, I can't find how I can handle this problem...

Anybody to help me ??? (or just to tell me it's not possible with 
built-in functionnalities... ;-( )

Yohann.



From rpang at CS.Princeton.EDU  Fri Apr 23 07:41:16 2004
From: rpang at CS.Princeton.EDU (Ruoming Pang)
Date: Fri, 23 Apr 2004 10:41:16 -0400 (EDT)
Subject: Log entire payloads
In-Reply-To: <40892029.2000304@rd.francetelecom.com>
References: <40892029.2000304@rd.francetelecom.com>
Message-ID: <Pine.LNX.4.58.0404231038030.661@willy.CS.Princeton.EDU>

> I'd like to log http payloads for each connection seen on my network.
>
> In fact, I'd like to get something like :
> Src_IP;Dst_IP;Request_Payload;Reply_Payload
>
> but with entire payloads (not only URIs, but also banners...)

Hi, Yohann,

One possibility is to load the contents.bro script. It will write the
contents of every connection to two files (contents-*), one for each
direction. Note that it does writing for every connection, not just HTTP
ones. If you want the latter, you might want to adapt the script
accordingly.

Ruoming


From vern at icir.org  Sat Apr 24 12:02:40 2004
From: vern at icir.org (Vern Paxson)
Date: Sat, 24 Apr 2004 12:02:40 -0700
Subject: Log entire payloads 
In-Reply-To: Your message of Fri, 23 Apr 2004 10:41:16 EDT.
Message-ID: <200404241902.i3OJ2eCq029110@jaguar.icir.org>

> One possibility is to load the contents.bro script. It will write the
> contents of every connection to two files (contents-*), one for each
> direction. Note that it does writing for every connection, not just HTTP
> ones. If you want the latter, you might want to adapt the script
> accordingly.

Note, depending on your broader use, you can avoid adapting the script
by using capture_filters to only capture tcp port 80.

		Vern


From vern at icir.org  Sat Apr 24 12:02:32 2004
From: vern at icir.org (Vern Paxson)
Date: Sat, 24 Apr 2004 12:02:32 -0700
Subject: Log entire payloads 
In-Reply-To: Your message of Fri, 23 Apr 2004 15:54:49 +0200.
Message-ID: <200404241902.i3OJ2WCq029102@jaguar.icir.org>

> I'd like to log http payloads for each connection seen on my network.
> 
> In fact, I'd like to get something like :
> Src_IP;Dst_IP;Request_Payload;Reply_Payload
> 
> but with entire payloads (not only URIs, but also banners...)

You can get all of this via

	bro -r http.trace http-reply http-header http-body

The only problem is that http-body has hardwired into it that contents
(not headers) larger than 40 bytes are truncated to 40 bytes.  I will
fix that.  In the interim, if you need entire contents you can make a
copy of the script with the limitation removed and use that instead.

		Vern


From christian at whoop.org  Sun Apr 25 19:22:30 2004
From: christian at whoop.org (Christian Kreibich)
Date: Sun, 25 Apr 2004 19:22:30 -0700
Subject: Need help understanding serializer
Message-ID: <1082946150.20788.64.camel@localhost.localdomain>

Hi Robin,


I've been looking at the serialization code in Bro for a while now and
I've hit a dead end. It'd be really cool if you could help me out
because I think I simply don't get it :)

I think I've basically understood the concepts of SerialObj, Serializer
and SerializationFormat -- I think I would have structured things the
same way. I'm getting lost in the details though:

- When exactly and why does a class have to implement Unserialize() and
Serialize()? What's their relationship to DoSerialize() and
DoUnserialize()? The comments in SerialObj.h are a bit vague in that
regard.

- It seems the idea of sending ahead an identifier to the receiving end
that tells it what to do with the following data exists three times:

      *  in the SER_xxx constants and the factory approach in
        IMPLEMENT_SERIAL
      *  in the character constants 'i', 'e', 's' etc in Serializer.cc
      *  in the MSG_xxx constants in RemoteSerializer.cc.

I think the latter are partially internal to the remote<->local
communication and can hence mostly be ignored for understanding the
serialization code, right? If you could quickly explain the difference
between the first two that'd be great.

The next thing I noticed are the hardcoded (un)seralization methods in
Serializer: 

        bool Serialize(const ID& id, const SerialInfo& info);
        bool Serialize(const char* func, val_list* args);
        bool Serialize(const StateAccess& s);
        bool Serialize(const Connection& c);
        bool Serialize(const Timer& t);

        virtual void GotID(ID* id, Val* val) = 0;
        virtual void GotEvent(const char* name, double time,
                                EventHandlerPtr event, val_list* a) = 0;
        virtual void GotFunctionCall(const char* name, double time,
                                Func* func, val_list* args) = 0;
        virtual void GotStateAccess(StateAccess* s) = 0;
        virtual void GotTimer(Timer* t) = 0;
        virtual void GotConnection(Connection* c) = 0;

        bool UnserializeID();
        bool UnserializeCall();
        bool UnserializeStateAccess();
        bool UnserializeTimer();
        bool UnserializeConnection();

Are these special in a way to have them implemented this way? Couldn't
there be a "received" callback per SER_xxx constant that resides as
a static method in the serializable classes themselves? So we can avoid
hardcoding anything?

- Following the comments in SerialObj.h, I see what I need to do to make
a class's objects serializable. I presume that the correct way to ship
an object to a serializer is by calling SerialObj::Serialize() with the
appropriate serializer. What are my options for picking them up at the
receiving end?

Oh and RemoteSerializer::ProcessSerialization() calls Unserialize()
passing a SerialInfo, but Serializer::Unserialize() expects a bool -- is
that intended?

The reason why I'm looking at this is that I'm trying to find the right
knobs to tweak to allow arbitrary *local* client applications to feed
information into Bro (like a tuned sshd that can feed its events and
traffic to a local Bro) without reinventing the wheel ...

Thanks so much!

Best,
Christian.
-- 
________________________________________________________________________
                                          http://www.cl.cam.ac.uk/~cpk25
                                                    http://www.whoop.org




From sommer at in.tum.de  Mon Apr 26 06:46:38 2004
From: sommer at in.tum.de (Robin Sommer)
Date: Mon, 26 Apr 2004 15:46:38 +0200
Subject: Need help understanding serializer
In-Reply-To: <1082946150.20788.64.camel@localhost.localdomain>
References: <1082946150.20788.64.camel@localhost.localdomain>
Message-ID: <20040426134638.GA20334@net.informatik.tu-muenchen.de>


On Sun, Apr 25, 2004 at 19:22 -0700, Christian Kreibich wrote:

> I think I've basically understood the concepts of SerialObj, Serializer
> and SerializationFormat -- I think I would have structured things the
> same way. I'm getting lost in the details though:

First of all, there are still a few loose ends which need some
clean-up. I just haven't found time to do that yet. :-(

Most importantly, some classes are serialized slightly different
than others, because there are two types of interfaces, an old one
and a new one (the new one correctly handles shallow-copied objects,
which the old doesn't). For example, Val uses the old
interface while Conn uses the new one. You can differentiate the two
by looking for a DECLARE_SERIAL in the class definition; if it's
there it's the new interface. If the semantincs are different, the
explanations below refer to the new one. I am going to adapt the
other classes asap (and I intend to write a short doc about this
stuff).

> - When exactly and why does a class have to implement Unserialize() and
> Serialize()? What's their relationship to DoSerialize() and
> DoUnserialize()? The comments in SerialObj.h are a bit vague in that
> regard.

Unserialize()/Serialize() are (non-virtual) methods which are to be
called when you want to actually serialize/unserialize an object,
i.e. that's the "user-interface". They are only defined inside the
base class of an hierarchy[1] (e.g. Conn has as Serialize(), but
TCP_Connection doesn't).

Unserialize()/Serialize(), in turn, call the (virtual)
DoSerialize()/DoUnserialize() which need to be implemented in every
class derived from such a base class. DoSerialize()/DoUnserialize()
are supposed to call their parent's implementation first, and then
read/write their own attributes.

[1] I did not use BroObj as the base but the classes on the next
layer. So, for this discussion Val, Stmt, Expr, Conn, etc. all start
their own hierarchy.
 
>       *  in the SER_xxx constants and the factory approach in
>         IMPLEMENT_SERIAL
>       *  in the character constants 'i', 'e', 's' etc in Serializer.cc
>       *  in the MSG_xxx constants in RemoteSerializer.cc.
> 
> I think the latter are partially internal to the remote<->local
> communication and can hence mostly be ignored for understanding the
> serialization code, right?

Right.

> If you could quickly explain the difference
> between the first two that'd be great.

The types of objects that a Serializer handles are different from
the classes containing Serialize/Unserialize methods; e.g.
Serializer serializes function calls for which there is no
corresponding class. The characters indicate which kind of
"top-level" serialization follows, while the SER_* constants specify
a concrete class.

Perhaps this could be avoided somehow, but it would introduce more
dependencies between the Serializer and Serialize/Unserialize
methods. And the additional overhead is only small.

> Are these special in a way to have them implemented this way? Couldn't
> there be a "received" callback per SER_xxx constant that resides as
> a static method in the serializable classes themselves? So we can avoid
> hardcoding anything?

Putting them into the serializable classes themselves doesn't work
as it depends on the serializer what we need to do (e.g. the
RemoteSerializer treats a received ID different than the
PersistenceSerializer). 

It could be an alternative to use only one of the
Serialize()/Unserialize()/Got() methods which would handle all
cases. But I don't think that would be much nicer: first, each
serializer would use some switch-construct anyway, and second, we
would lose the static type checking.

> - Following the comments in SerialObj.h, I see what I need to do to make
> a class's objects serializable. I presume that the correct way to ship
> an object to a serializer is by calling SerialObj::Serialize() with the
> appropriate serializer.

Correct.

> What are my options for picking them up at the
> receiving end?

You need a Serializer-derived class at the receiving end that
implements the Serializer::Got*() methods, and calls
Serializer::Unserialize() to actually do the work.
Serializer::Unserialize() gets it data from its member "io" which is
an instance of ChunkedIO and has to be initialized before (e.g. from
a fd by using a ChunkedIOFd).

Try taking a look at the implementation of FileSerializer to see an
example of how this work to read data back from a file.

> Oh and RemoteSerializer::ProcessSerialization() calls Unserialize()
> passing a SerialInfo, but Serializer::Unserialize() expects a bool -- is
> that intended?

Your eyes are quite good. :-)

No, that's indeed wrong (and actually I'm quite surprised that the
compiler implicitly converts the pointer to a bool here; but I guess
it does this at all places and not just inside an "if (...)" :-).

Unserialize(false) should be better.

> The reason why I'm looking at this is that I'm trying to find the right
> knobs to tweak to allow arbitrary *local* client applications to feed
> information into Bro (like a tuned sshd that can feed its events and
> traffic to a local Bro) without reinventing the wheel ...

If you'd like to pass in data from other applications than Bro, it
could perhaps make sense to think about a more well-defined data
format. The serializations are a representation of internal Bro
structures which could be quite hard to generate externally.

Robin

-- 
Robin Sommer * Room        01.08.055 * www.net.in.tum.de
TU Munich    * Phone (089) 289-18006 *  sommer at in.tum.de 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20040426/e5131c87/attachment.bin 

From srg018 at latech.edu  Mon Apr 26 08:34:46 2004
From: srg018 at latech.edu (Shekhar Reddy Gaddam)
Date: Mon, 26 Apr 2004 10:34:46 -0500
Subject: Extraction of features from DARPA dataset tcpdump files
In-Reply-To: <200404241902.i3OJ2eCq029110@jaguar.icir.org>
References: <200404241902.i3OJ2eCq029110@jaguar.icir.org>
Message-ID: <1082993686.408d2c1664435@webmail.LaTech.edu>


Dear Dr. Vern,

I'd like to extract 41 features and their corresponding attack classes based on
the DARPA 1999 dataset and 2000 dataset
(http://www.ll.mit.edu/IST/ideval/data/data_index.html) like the KDD Cup 99
dataset ( http://kdd.ics.uci.edu/databases/kddcup99/kddcup.names ). Is it
possible  to extract all the 41 features from tcpdump files. I was able to
extract 12 features only. I'm newbie to the bro tool, and i'd like to know if i
can extract all the 41 features from tcpdump files.

         

Regards,

Shekhar




-------------------------------------------------
This mail sent through IMP: http://horde.org/imp/



From christian at whoop.org  Mon Apr 26 12:27:13 2004
From: christian at whoop.org (Christian Kreibich)
Date: Mon, 26 Apr 2004 12:27:13 -0700
Subject: Need help understanding serializer
In-Reply-To: <20040426134638.GA20334@net.informatik.tu-muenchen.de>
References: <1082946150.20788.64.camel@localhost.localdomain>
	 <20040426134638.GA20334@net.informatik.tu-muenchen.de>
Message-ID: <1083007591.2793.94.camel@localhost.localdomain>

On Mon, 2004-04-26 at 06:46, Robin Sommer wrote:
>
> First of all, there are still a few loose ends which need some
> clean-up. I just haven't found time to do that yet. :-(

Okay sure -- this is neat stuff, and if I'm a pita then just tell me to
bugger off and I'll come back once you're happy with it :)

> Most importantly, some classes are serialized slightly different
> than others, because there are two types of interfaces, an old one
> and a new one (the new one correctly handles shallow-copied objects,
> which the old doesn't). For example, Val uses the old
> interface while Conn uses the new one. You can differentiate the two
> by looking for a DECLARE_SERIAL in the class definition; if it's
> there it's the new interface. If the semantincs are different, the
> explanations below refer to the new one. I am going to adapt the
> other classes asap

I see, thanks!

> (and I intend to write a short doc about this
> stuff).

That'd be cool. Actually I think a Hacker's Guide to Bro would be really
useful. The thing is really quite big now ...

> Unserialize()/Serialize() are (non-virtual) methods which are to be
> called when you want to actually serialize/unserialize an object,
> i.e. that's the "user-interface". They are only defined inside the
> base class of an hierarchy[1] (e.g. Conn has as Serialize(), but
> TCP_Connection doesn't).

Okay. I'm still a bit confused because Connections declare

  bool Serialize(Serializer* s) const;
  static Connection* Unserialize(Serializer* ser);

but SerialObj's also have those, but different signatures:

  bool Serialize(Serializer* s, SerialInfo* i, bool cache = true) const;
  static SerialObj* Unserialize(Serializer* s, SerialType type,
                                bool cache = true);

Is that related to the different APIs?

> Unserialize()/Serialize(), in turn, call the (virtual)
> DoSerialize()/DoUnserialize() which need to be implemented in every
> class derived from such a base class. DoSerialize()/DoUnserialize()
> are supposed to call their parent's implementation first, and then
> read/write their own attributes.

Yeah I saw that -- nice!

> The types of objects that a Serializer handles are different from
> the classes containing Serialize/Unserialize methods; e.g.
> Serializer serializes function calls for which there is no
> corresponding class.

<enlightenment> Aaaaaaah! </enlightenment>

> The characters indicate which kind of
> "top-level" serialization follows, while the SER_* constants specify
> a concrete class.
> 
> Perhaps this could be avoided somehow, but it would introduce more
> dependencies between the Serializer and Serialize/Unserialize
> methods. And the additional overhead is only small.

Okay sure. I just want to understand it! :)

> > Are these special in a way to have them implemented this way? Couldn't
> > there be a "received" callback per SER_xxx constant that resides as
> > a static method in the serializable classes themselves? So we can avoid
> > hardcoding anything?
> 
> Putting them into the serializable classes themselves doesn't work
> as it depends on the serializer what we need to do (e.g. the
> RemoteSerializer treats a received ID different than the
> PersistenceSerializer). 

Oh, I see. Thanks.

> It could be an alternative to use only one of the
> Serialize()/Unserialize()/Got() methods which would handle all
> cases. But I don't think that would be much nicer: first, each
> serializer would use some switch-construct anyway, and second, we
> would lose the static type checking.

True. And if I understand you correctly, I normally won't have to deal
with a serializer's internals anyway because I only need the
Serialize()/Unserialize() stuff at some point in the hierarchies you're
mentioning above. I think I'm starting to get it.

> > What are my options for picking them up at the
> > receiving end?
> 
> You need a Serializer-derived class at the receiving end that
> implements the Serializer::Got*() methods, and calls
> Serializer::Unserialize() to actually do the work.

Mhmm that still confuses me. You're saying above that the types of
objects that Serializers handle are different from the
Serialize()/Unserialize() hierarchies. And the Got* methods aren't for
arbitrary serializable objects but just for the specific types that
their names indicate, right?

So say I have a class Foo that implements DoSerialize() and
DoUnserialize() following the comments in SerialObj.h, and higher up
Foo's hierarchy is Bar that implements Unserialize() as you're
describing above. Now I ship a Foo instance using Bar::Serialize(s,
...). How do I get from the Serializer at the receiving end noticing
that something arrives, to the Bar::Unserialize() call at the far end?

> Serializer::Unserialize() gets it data from its member "io" which is
> an instance of ChunkedIO and has to be initialized before (e.g. from
> a fd by using a ChunkedIOFd).

Yep thanks for writing the chunk stuff, that's really useful.

> No, that's indeed wrong (and actually I'm quite surprised that the
> compiler implicitly converts the pointer to a bool here; but I guess

So was I!

> If you'd like to pass in data from other applications than Bro, it
> could perhaps make sense to think about a more well-defined data
> format. The serializations are a representation of internal Bro
> structures which could be quite hard to generate externally.

Well for now I was really just looking for a way to pump back and forth
simple structs and maybe an occasional dynamic length byte string. What
I had in mind is roughly this:

- Bro subsytem X can handle input from client application X.
- Some LocalSerializer handles local comms through domain sockets
- Client app X registers with Bro at startup
- Client app X sends some data 
- Bro side recognizes that data for subsystem X are coming up and
notifies subsystem X
- Subsystem X extracts next data item from the link and processes it

And vice versa. I'd leave it entirely up to the subsystems how to define
the data layout. As long as I could use the various Read/Write methods
of the Serializer I'd be happy .. The main hurdle I didn't manage in the
current code was how to get from the Serializer noticing that data
arrives to an arbitrary subsytem. 

I even have a name for the client library already: libbroccoli. BRO
Client COmmunications LIbrary ;)

Thanks Robin!

Best,
Christian.
-- 
________________________________________________________________________
                                          http://www.cl.cam.ac.uk/~cpk25
                                                    http://www.whoop.org




From christian at whoop.org  Mon Apr 26 22:48:21 2004
From: christian at whoop.org (Christian Kreibich)
Date: Mon, 26 Apr 2004 22:48:21 -0700
Subject: Need help understanding serializer
In-Reply-To: <1083007591.2793.94.camel@localhost.localdomain>
References: <1082946150.20788.64.camel@localhost.localdomain>
	 <20040426134638.GA20334@net.informatik.tu-muenchen.de>
	 <1083007591.2793.94.camel@localhost.localdomain>
Message-ID: <1083044900.9096.6.camel@localhost.localdomain>

Robin,

On Mon, 2004-04-26 at 12:27, Christian Kreibich wrote:
> And vice versa. I'd leave it entirely up to the subsystems how to define
> the data layout. As long as I could use the various Read/Write methods
> of the Serializer I'd be happy .. The main hurdle I didn't manage in the
> current code was how to get from the Serializer noticing that data
> arrives to an arbitrary subsytem. 

I think I just thought of a reasonable way to do this ... would it make
sense to you to add one extra step in the hierarchy below Serializer,
say BroSerializer, and keep the toplevel Serializer class free of any
Bro-Bro specific stuff? And let the toplevel Serializer be a very
bare-bones interface that only lets you read/write primitives (ie the
various Read/Write methods)?

I think I could derive what's best for my needs from such a base class
(for example code to dispatch to subsystems etc), while you could keep
working on the inter-Bro stuff from your BroSerializer class that would
contain the various Serialize() methods, plus the GotXXX callbacks...
does that make any sense?

Regards,
Christian.
-- 
________________________________________________________________________
                                          http://www.cl.cam.ac.uk/~cpk25
                                                    http://www.whoop.org





From vern at icir.org  Mon Apr 26 23:32:55 2004
From: vern at icir.org (Vern Paxson)
Date: Mon, 26 Apr 2004 23:32:55 -0700
Subject: new bro "CURRENT" release - 0.8a81
Message-ID: <200404270632.i3R6WtCq053601@jaguar.icir.org>

An updated "CURRENT" version of Bro is now available from the usual location:

	ftp://ftp.ee.lbl.gov/bro-pub-0.8-current.tar.gz

This version has several minor-but-handy changes to the last "CURRENT"
version (0.8a79).

                Vern


-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

0.8a81 Mon Apr 26 22:46:37 PDT 2004

- A bunch of memory leaks fixed (Chema Gonzalez).

- A new HTTP analyzer variable, content_truncation_limit, controls
  how much of an HTTP request/reply contents element (i.e., what's
  passed to http_entity_data - this is *not* the entire content, but
  the next chunk's worth) is logged in the log file.  It defaults
  to 40 bytes.  Setting it to 0 means "log all of it".

- Fix to avoid crashing for malformed RPC requests.

- Improved OpenSSL auto-configuration (Robin Sommer).

- Fix for compiling without OpenSSL.

- A new built-in, double_to_count(), converts a value of type "double"
  to the corresponding "count" (Chema Gonzalez).  We should probably
  add floor(), ceil(), etc.

- Parameterization of trw.bro tweaked (Jaeyeon Jung).


From zangds at netpower.com.cn  Tue Apr 27 01:48:21 2004
From: zangds at netpower.com.cn (=?utf-8?B?6Ien5Yas5p2+?=)
Date: Tue, 27 Apr 2004 16:48:21 +0800
Subject: Bro's problems:Out of memory
References: <20040423084606.GB13169@net.informatik.tu-muenchen.de>
Message-ID: <005d01c42c34$6aee4c20$7400a8c0@netpowerdonal>


----- Original Message ----- 
From: "Holger Dreger" <hdreger at net.in.tum.de>
To: <bro at lbl.gov>
Sent: Friday, April 23, 2004 4:46 PM
Subject: Re: Bro's problems:Out of memory


> On Fri, Apr 23, 2004 at 02:44:49PM +0800, ?????? wrote:
> >     We are intresting in Bro.But when we test it under a simulative network,
> > it was killed a few minutes by the linux systerm,giving the message of"
> > Out of memory",while Bro 's message is "internal error:double signal".
> 
> The amount of memory bro uses heavily depends on the policy scripts
> that you are running. If you additionally load the script
> statistics.bro you'll get a statistics.log file which should provide
> you information where the all the memory has gone. One common trick is
> to tune the various timeouts like the script reduce-memory.bro does.
> (Note: reduce memory sets timeouts that are perhaps not suitable to
> your needs)
> 
> Holger
> 
> --
> Holger Dreger * http://www.net.in.tum.de * Phone: +49 (0)89 289-18006
>    Computer Science Department * Technische Universitaet Muenchen

Thanks alot for your help!
But what I am thinking is whether there is a way to control Bro,so it will not run "out of memory"
any way.
For example,Bro will malloc a upper bound of 10M memory,and if there are more packets,we just 
drop them until there's free memory(other packet has gone,and released the memory).Thus,Bro will 
never go out of memory no matter how busy the network is!
As your suggest,I test Bro use:./bro -i eth1 finger.bro&.Here I modifies finger.bro:add 
"redef capture_filters += { ["finger"] = "tcp dst port 80 or tcp src port 80" };"in the begigning
of the script.Since finger.bro do a little things,then we can say the memory Bro use now is less depends
on the policy scripts.Then I simulate a stream of 5000/sec http connections,and after a few minutes,the same 
thing happed(out of memory):(.
Any suggestion?
Besides,if you can tell me where Bro malloc the memory space,and where and when to free them?
BEST WISHES!

Donal



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20040427/659d0379/attachment.html 

From rmkml at wanadoo.fr  Tue Apr 27 05:19:14 2004
From: rmkml at wanadoo.fr (rmkml)
Date: Tue, 27 Apr 2004 14:19:14 +0200 (CEST)
Subject: how simply disable dns in last bro ?
Message-ID: <Pine.LNX.4.58.0404271413200.237@hp>

Hi,

Possible disable dns request

simply on last bro (a81) ?

because I don't have resolver on my fbsd host.

$ /usr/local/bin/bro08a81 -r
/home/f/200401121453-iptcpudpfrag-telefonicaESP.tcpdump mt
/L/policy/scan.bro, line 91: internal error: NB-DNS error in
DNS_Mgr::WaitForReplies (recvfrom(): Connection refused)

Thanks

Rmkml at Wanadoo.fr


From vern at icir.org  Tue Apr 27 10:19:41 2004
From: vern at icir.org (Vern Paxson)
Date: Tue, 27 Apr 2004 10:19:41 -0700
Subject: how simply disable dns in last bro ?
In-Reply-To: Your message of Tue, 27 Apr 2004 14:19:14 +0200.
Message-ID: <200404271719.i3RHJfCq075658@jaguar.icir.org>

> $ /usr/local/bin/bro08a81 -r
> /home/f/200401121453-iptcpudpfrag-telefonicaESP.tcpdump mt
> /L/policy/scan.bro, line 91: internal error: NB-DNS error in
> DNS_Mgr::WaitForReplies (recvfrom(): Connection refused)

You can do this by editing Makefile and removing "nb_dns.o" from
the "OBJ =" target (it's at the very end), and editing config.h to
change 

	#define HAVE_NB_DNS

to

	#undef HAVE_NB_DNS

Finally, set the environment variable $BRO_DNS_FAKE.

If someone would contribute the necessary autoconf magic to do this
automatically, that would be great - it's beyond my meager autoconf
skills.

		Vern


From sommer at in.tum.de  Tue Apr 27 10:16:48 2004
From: sommer at in.tum.de (Robin Sommer)
Date: Tue, 27 Apr 2004 19:16:48 +0200
Subject: Need help understanding serializer
In-Reply-To: <1083044900.9096.6.camel@localhost.localdomain> <1083007591.2793.94.camel@localhost.localdomain>
References: <1082946150.20788.64.camel@localhost.localdomain> <20040426134638.GA20334@net.informatik.tu-muenchen.de> <1083007591.2793.94.camel@localhost.localdomain> <1083044900.9096.6.camel@localhost.localdomain> <1082946150.20788.64.camel@localhost.localdomain> <20040426134638.GA20334@net.informatik.tu-muenchen.de> <1083007591.2793.94.camel@localhost.localdomain>
Message-ID: <20040427171648.GB25544@net.informatik.tu-muenchen.de>


On Mon, Apr 26, 2004 at 12:27 -0700, Christian Kreibich wrote:

>   bool Serialize(Serializer* s) const;
>   bool Serialize(Serializer* s, SerialInfo* i, bool cache = true) const;

Good point. The difference is that you don't call
SerialObj::Serialize() directly but Conn::Serialize() which, in
turn, uses SerialObj::Serialize() to do its work; i.e.
SerialObj::Serialize() does not really belong to the user-interface.

I guess SerialObj::Serialize() should better be protected
(and perhaps even renamed). I'll change this. Thanks!

> arbitrary serializable objects but just for the specific types that
> their names indicate, right?

Right.

> ...). How do I get from the Serializer at the receiving end noticing
> that something arrives, to the Bar::Unserialize() call at the far end?

If I understand your question correctly, you're expecting some
call-back mechanism, right? That's not the case here. The receiving
Serializer sees that one of the basic entities (i.e. those with
character constants) has arrived. Then it will call the
corresponding Unserialize() method. If that is, e.g., a Connection,
then Connection::Unserialize() is called. If the connection contains
a Foo object, Connection::DoUnserialize() will call
Bar::Unserialize().

> I even have a name for the client library already: libbroccoli. BRO
> Client COmmunications LIbrary ;)

Cool! :-)

On Mon, Apr 26, 2004 at 22:48 -0700, Christian Kreibich wrote:

> Bro-Bro specific stuff? And let the toplevel Serializer be a very
> bare-bones interface that only lets you read/write primitives (ie the
> various Read/Write methods)?

In general, adding an extra step would be fine with me. But I am not
sure if I got your idea right. Do you indeed want to access only the
Read/Write methods (not any of the Serialize/Unserialize)? If yes,
perhaps the SerializationFormat itself would suffice?

Robin

-- 
Robin Sommer * Room        01.08.055 * www.net.in.tum.de
TU Munich    * Phone (089) 289-18006 *  sommer at in.tum.de 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20040427/dcd0dba8/attachment.bin 

From vern at icir.org  Tue Apr 27 10:23:17 2004
From: vern at icir.org (Vern Paxson)
Date: Tue, 27 Apr 2004 10:23:17 -0700
Subject: Extraction of features from DARPA dataset tcpdump files
In-Reply-To: Your message of Mon, 26 Apr 2004 10:34:46 CDT.
Message-ID: <200404271723.i3RHNHCq076025@jaguar.icir.org>

> I'd like to extract 41 features and their corresponding attack classes based on
> the DARPA 1999 dataset and 2000 dataset

Bro doesn't have these features directly coded into its analyzers.
You could probably add them with not that much work, but I'd advise you
to first consider whether you really want to do so: those datasets, while
invaluable for the evaluations for which they were originally developed,
are notorious for how they are misapplied for subsequent intrusion detection
research.  The main problem is that they have artifacts due to their synthetic
nature.  In particular, the feature set from the KDD Cup is known to be
seriously flawed.  See McHugh's critique of the original datasets and
Mahoney/Chen's RAID 2003 paper on the problems with the KDD Cup feature set.

		Vern


From christian at whoop.org  Tue Apr 27 10:50:13 2004
From: christian at whoop.org (Christian Kreibich)
Date: Tue, 27 Apr 2004 10:50:13 -0700
Subject: Need help understanding serializer
In-Reply-To: <20040427171648.GB25544@net.informatik.tu-muenchen.de>
References: <1082946150.20788.64.camel@localhost.localdomain>
	 <20040426134638.GA20334@net.informatik.tu-muenchen.de>
	 <1083007591.2793.94.camel@localhost.localdomain>
	 <1083044900.9096.6.camel@localhost.localdomain>
	 <1082946150.20788.64.camel@localhost.localdomain>
	 <20040426134638.GA20334@net.informatik.tu-muenchen.de>
	 <1083007591.2793.94.camel@localhost.localdomain>
	 <20040427171648.GB25544@net.informatik.tu-muenchen.de>
Message-ID: <1083088212.9095.79.camel@localhost.localdomain>

Hi!

On Tue, 2004-04-27 at 10:16, Robin Sommer wrote:
> On Mon, Apr 26, 2004 at 12:27 -0700, Christian Kreibich wrote:
> 
> >   bool Serialize(Serializer* s) const;
> >   bool Serialize(Serializer* s, SerialInfo* i, bool cache = true) const;
> 
> Good point. The difference is that you don't call
> SerialObj::Serialize() directly but Conn::Serialize() which, in
> turn, uses SerialObj::Serialize() to do its work; i.e.
> SerialObj::Serialize() does not really belong to the user-interface.

All is clear :)

> > ...). How do I get from the Serializer at the receiving end noticing
> > that something arrives, to the Bar::Unserialize() call at the far end?
> 
> If I understand your question correctly, you're expecting some
> call-back mechanism, right?

Yup sort of. I guess my problem was that I approached the thing with a
different idea in mind how it would work. 

> That's not the case here. The receiving
> Serializer sees that one of the basic entities (i.e. those with
> character constants) has arrived. Then it will call the
> corresponding Unserialize() method. If that is, e.g., a Connection,
> then Connection::Unserialize() is called. If the connection contains
> a Foo object, Connection::DoUnserialize() will call
> Bar::Unserialize().

Ah I think I'm getting there. You're using a callback mechanism for the
basic entities only, and then the DoSerialize/DoUnserialize API to
handle the objects referenced by the basic entities.

> > Bro-Bro specific stuff? And let the toplevel Serializer be a very
> > bare-bones interface that only lets you read/write primitives (ie the
> > various Read/Write methods)?
> 
> In general, adding an extra step would be fine with me. But I am not
> sure if I got your idea right. Do you indeed want to access only the
> Read/Write methods (not any of the Serialize/Unserialize)? If yes,
> perhaps the SerializationFormat itself would suffice?

Well see I need a mechanism to communicate with local sensors (in
intrusion detection parlance), and I was thinking of communicating
through unix domain sockets (largely because I have code that I could
reuse for that). So at the Bro end, another Serializer sounds like a
good idea.

The problem is that the existing Serializer class already implies that I
want to send connections, IDs, etc (ie things that are largely
interesting for Bro-Bro communication only) which is not necessarily the
case for what I need. The SerializationFormat gives me what I need to do
the marshalling, but it doesn't handle I/O. And a format plus I/O are
pretty much the core ingredients of a Serializer, right? Also I could
build the callback API that I was looking for in my own Serializer...

Thanks,
Christian.
-- 
________________________________________________________________________
                                          http://www.cl.cam.ac.uk/~cpk25
                                                    http://www.whoop.org




From vern at icir.org  Tue Apr 27 12:05:17 2004
From: vern at icir.org (Vern Paxson)
Date: Tue, 27 Apr 2004 12:05:17 -0700
Subject: new bro "CURRENT" release - 0.8a82
Message-ID: <200404271905.i3RJ5HCq086562@jaguar.icir.org>

Sorry to immediately bug folks with another release, but this one, available
from the usual location:

	ftp://ftp.ee.lbl.gov/bro-pub-0.8-current.tar.gz

fixes a rarely-tickled-but-fatal bug in timer processing.  CHANGES
and the one-character fix appended.

                Vern


-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

0.8a82 Tue Apr 27 11:53:24 PDT 2004

- Fixed inactivity timer loop when a packet arrives exactly when
  the timer is set to expire.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

--- Conn.cc	2004/03/15 20:14:19	1.49
+++ Conn.cc	2004/04/27 19:00:13
@@ -1,4 +1,4 @@
-// $Id: Conn.cc,v 1.49 2004/03/15 20:14:19 vern Exp $
+// $Id: Conn.cc,v 1.50 2004/04/27 19:00:03 vern Exp $
 //
 // Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002
 //      The Regents of the University of California.  All rights reserved.
@@ -294,7 +294,7 @@
 	// timeout once, but it's disabled now. We do nothing then.
 	if ( inactivity_timeout )
 		{
-		if ( last_time + inactivity_timeout < t )
+		if ( last_time + inactivity_timeout <= t )
 			{
 			Event(connection_timeout);
 			sessions->Remove(this, 1);


From hdreger at net.in.tum.de  Tue Apr 27 12:08:03 2004
From: hdreger at net.in.tum.de (Holger Dreger)
Date: Tue, 27 Apr 2004 21:08:03 +0200
Subject: Bro's problems:Out of memory
In-Reply-To: <005d01c42c34$6aee4c20$7400a8c0@netpowerdonal>
References: <20040423084606.GB13169@net.informatik.tu-muenchen.de> <005d01c42c34$6aee4c20$7400a8c0@netpowerdonal>
Message-ID: <20040427190803.GE26027@net.informatik.tu-muenchen.de>

On Tue, Apr 27, 2004 at 04:48:21PM +0800, ????????? wrote:
> But what I am thinking is whether there is a way to control Bro,so it will not run "out of memory"
> any way.
> For example,Bro will malloc a upper bound of 10M memory,and if there are more packets,we just 
> drop them until there's free memory(other packet has gone,and released the memory).Thus,Bro will 
> never go out of memory no matter how busy the network is!

That's not implemented in Bro. One of Bro's design goal is *not* to
lose packets (independent of the cost in resources). In practice
that's not possible and that's where the differnet tweaks like
timeouts play an important role. But there is no explicit mechanism to
hard-limit the memory usage.

> As your suggest,I test Bro use:./bro -i eth1 finger.bro&.Here I modifies finger.bro:add 
> "redef capture_filters += { ["finger"] = "tcp dst port 80 or tcp src port 80" };"in the begigning
> of the script.Since finger.bro do a little things,then we can say the memory Bro use now is less depends
> on the policy scripts.

Yes and no: By defining the filter (on script level) you told Bro to
do Transport-Layer analysis on all port 80 packets (i.e. that means
TCP-analysis via a state machine)

>Then I simulate a stream of 5000/sec http connections,and after a few minutes,the same 
> thing happed(out of memory):(.
> Any suggestion?

Sounds like the TCP statemachine of Bro needed more and more memory to
keep track of the connections it analyzes. Note that with default
config state is not discarded once Bro sees the connection
established, except by regular TCP-teardown. It may happen, that it
misses some of the connection teardowns and cumulates the state for
those connections forever (until it crashes;-))

> Besides,if you can tell me where Bro malloc the memory space,and where and when to free them?

If it's really the connection-state memory that causes that problem,
than the memory is probably allocated via the construction of new
connection objects (see Sessions.cc)

Holger

--
Holger Dreger * http://www.net.in.tum.de * Phone: +49 (0)89 289-18006
   Computer Science Department * Technische Universitaet Muenchen


From christian at whoop.org  Tue Apr 27 20:09:57 2004
From: christian at whoop.org (Christian Kreibich)
Date: Tue, 27 Apr 2004 20:09:57 -0700
Subject: Need help understanding serializer
In-Reply-To: <1083088212.9095.79.camel@localhost.localdomain>
References: <1082946150.20788.64.camel@localhost.localdomain>
	 <20040426134638.GA20334@net.informatik.tu-muenchen.de>
	 <1083007591.2793.94.camel@localhost.localdomain>
	 <1083044900.9096.6.camel@localhost.localdomain>
	 <1082946150.20788.64.camel@localhost.localdomain>
	 <20040426134638.GA20334@net.informatik.tu-muenchen.de>
	 <1083007591.2793.94.camel@localhost.localdomain>
	 <20040427171648.GB25544@net.informatik.tu-muenchen.de>
	 <1083088212.9095.79.camel@localhost.localdomain>
Message-ID: <1083121773.9090.191.camel@localhost.localdomain>

On Tue, 2004-04-27 at 10:50, Christian Kreibich wrote:
>
> Well see I need a mechanism to communicate with local sensors (in
> intrusion detection parlance), and I was thinking of communicating
> through unix domain sockets (largely because I have code that I could
> reuse for that). So at the Bro end, another Serializer sounds like a
> good idea.
> 
> The problem is that the existing Serializer class already implies that I
> want to send connections, IDs, etc (ie things that are largely
> interesting for Bro-Bro communication only) which is not necessarily the
> case for what I need. The SerializationFormat gives me what I need to do
> the marshalling, but it doesn't handle I/O. And a format plus I/O are
> pretty much the core ingredients of a Serializer, right? Also I could
> build the callback API that I was looking for in my own Serializer...

To clarify what I mean I've put up a patch at

http://www.cl.cam.ac.uk/~cpk25/BroSerializer.diff.gz 

that moves all the callback and Serialize()/Unserialize() stuff down to
a new child class that I called BroSerializer (as it's geared towards
Bro-Bro communication). The new Serializer class only has only the basic
read/write methods, the cache subsystem, and the error stuff left.
RemoteSerializer is now derived from BroSerializer.

I could now derive my own serializer from this Serializer class without
much trouble. The patch should apply cleanly against 0.8a82.

The separation was pretty easy -- maybe that's an indication that this
would be a good move ... does this look reasonable?

Thanks,
Christian.
-- 
________________________________________________________________________
                                          http://www.cl.cam.ac.uk/~cpk25
                                                    http://www.whoop.org




From sommer at in.tum.de  Wed Apr 28 11:48:07 2004
From: sommer at in.tum.de (Robin Sommer)
Date: Wed, 28 Apr 2004 20:48:07 +0200
Subject: Need help understanding serializer
In-Reply-To: <1083121773.9090.191.camel@localhost.localdomain>
References: <1082946150.20788.64.camel@localhost.localdomain> <20040426134638.GA20334@net.informatik.tu-muenchen.de> <1083007591.2793.94.camel@localhost.localdomain> <1083044900.9096.6.camel@localhost.localdomain> <1082946150.20788.64.camel@localhost.localdomain> <20040426134638.GA20334@net.informatik.tu-muenchen.de> <1083007591.2793.94.camel@localhost.localdomain> <20040427171648.GB25544@net.informatik.tu-muenchen.de> <1083088212.9095.79.camel@localhost.localdomain> <1083121773.9090.191.camel@localhost.localdomain>
Message-ID: <20040428184807.GE21394@net.informatik.tu-muenchen.de>


On Tue, Apr 27, 2004 at 20:09 -0700, Christian Kreibich wrote:

> > case for what I need. The SerializationFormat gives me what I need to do
> > the marshalling, but it doesn't handle I/O. And a format plus I/O are
> > pretty much the core ingredients of a Serializer, right? Also I could
> > build the callback API that I was looking for in my own Serializer...

Ok, I see.

> http://www.cl.cam.ac.uk/~cpk25/BroSerializer.diff.gz 

Thanks! I skimmed over it and it seems to be fine. So, feel free to
go ahead.

Robin

-- 
Robin Sommer * Room        01.08.055 * www.net.in.tum.de
TU Munich    * Phone (089) 289-18006 *  sommer at in.tum.de 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20040428/21ea32f2/attachment.bin