From jags0nhak3r at engineer.com  Sun Nov  1 19:44:03 2009
From: jags0nhak3r at engineer.com (jags0nhak3r at engineer.com)
Date: Sun, 01 Nov 2009 22:44:03 -0500
Subject: [Bro] failed to start BRO
In-Reply-To: <20091030115554.GC11601@192.168.2.103>
References: <8CC26816F6E0240-D08-24C1@web-mmc-d05.sysops.aol.com>
	<20091030005838.GB11601@192.168.2.103>
	<8CC271AA43AEB01-B0C-30B3@web-mmc-d19.sysops.aol.com>
	<20091030115554.GC11601@192.168.2.103>
Message-ID: <8CC2985DB0D9966-95C-5766@web-mmc-m10.sysops.aol.com>


 

 Thanks jean

I have done it..




 

-----Original Message-----
From: jean-philippe luiggi <jean-philippe.luiggi at didconcept.com>
To: jags0nhak3r at engineer.com
Cc: bro at ICSI.Berkeley.EDU
Sent: Fri, Oct 30, 2009 7:55 pm
Subject: Re: [Bro] failed to start BRO










* jags0nhak3r at engineer.com <jags0nhak3r at engineer.com> [2009-10-29 21:51:33 
-0400]:

> 
>  
> 
>  
> Hi, 
> 
> Thanks for your Re
> 
> I figured out that localhost.localdomain.bro is file and BRO needs to open it 
when it starts. that file should be located at {BROPATH}, that is right.
> 
> here is my BROPATH
> 
> # Bro policy paths
> BROPATH="/usr/local/bro/share/bro/site:/usr/local/bro/share/bro:/usr/local/bro/share/bro/sigs:/usr/local/bro/share/bro/time-machine"
> export BROPATH
> 
> # Filename of the Bro start policy.  Must be located in one of the directories 
in $BROPATH
> BRO_START_POLICY="localhost.localdomain.bro"
> 
> I wonder why the so called file localhost.localdomain.bro is not created in 
BROPATH by default. Thus, I created it in this PATH
> /usr/local/bro/share/bro manually and BRO successfully started. 
> 
> I also would like to know what is the purpose of that file what should be in 
it?
> 
> btw, 
> 1- what and how should I start to capture packets, analyze them? 
> 2-  what commands shall I run where the analysis files are stored?
> 
> I read in the BRO user manual, it mentions that to run BRO type the following 
comman 
> 
> bro  -[options]
> but when I run bro, which is a binary file, I get    bash: bro: command not 
found
> 
> what is wrong with my configuration...
> 
> 
> Please I need assistance, 
> 
> Regards

  Hello,
  
  Bro is very good in various things and one of them is the customisation of
  it on behalf of your specific environment.
  You may need to tweak the NIDS in order to make it in accordance with your
  network, etc. so the reason of such a file (localhost.localdomain.bro).
  
  In order to capture data, you need to specify what do you want to catch so 
here
  are the various *.bro.
  
  my config file (bro.cfg) have :
  
  BRO_START_POLICY="mygw"
  
  and this file contains only one line :
  
  @load brolite 
  
  ("brolite.bro" is found in one of the directory specified by BROPATH).
  
  Considering the analyze, apart from the BROPATH we talk about before, there's 
  too a BROLOGS's environment variable.
  
  Here is mine :
  
  # Directory containing Bro logs
  BROLOGS="/opt/share/bro-1.4/logs"
  export BROLOGS
  
  So as soon as bro begins, it'll report various things in this directory.
  
  I've some (perhaps) stupid questions :
  
  Did you setup a "bro.cfg" (you can do it using "bro_config").
  
  And do you run bro using "bro.rc" ?
  
  With regards,
  
  Jean-Philippe.
  
  



 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20091101/92b0ca22/attachment.html 

From fh at open.ch  Mon Nov  2 01:53:50 2009
From: fh at open.ch (Fabian Hugelshofer)
Date: Mon, 02 Nov 2009 10:53:50 +0100
Subject: [Bro] BroPing Connection Failure
In-Reply-To: <20091030171341.GF13504@icir.org>
References: <4AEAFF34.5000304@open.ch> <20091030171341.GF13504@icir.org>
Message-ID: <4AEEAC2E.3020206@open.ch>

Thanks for the fast response, Robin.

Robin Sommer wrote:
> Sounds like a nasty race condition of some sort. The remote.log only
> shows that something weird is going on but isn't detailed enough to
> understand what's causing it. Please enable debugging output on both
> sides. For Broccoli, see here:
> http://www.icir.org/christian/broccoli/manual/c84.html#AEN814 
> 
> For Bro, configure with --enable-debug and then run with "-B comm".
> That should produce a debug.log with lots of information.

I enabled debugging on the client with "-d -d" (bropingc.txt) and on the 
server with "-B comm" (bropings.txt). I also attached a tcpdump trace of 
the communication.

Even if the server seems to finish the handshake, it doesn't send 
anything back to the client (see tcpdump trace). As a result of that the 
client times out.

I found out that compiling without "--disable-select-loop" doesn't show 
these connection problems. I'm not sure whether this is related or a 
coincidence. I can use this as a workaround, but would prefer another 
solution.

> It would also be good if you could try it with the current
> development version from SVN to see if the problem still occurs with
> that one.

The current development version of Bro 1.5 does still show the same 
behaviour.

Fabian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bropingc.txt.gz
Type: application/x-gzip
Size: 3037 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20091102/329d671e/attachment.gz 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bropings.txt.gz
Type: application/x-gzip
Size: 871 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20091102/329d671e/attachment-0001.gz 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: broping.pcap.gz
Type: application/x-gzip
Size: 338 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20091102/329d671e/attachment-0002.gz 

From JAzoff at uamail.albany.edu  Mon Nov  2 05:14:38 2009
From: JAzoff at uamail.albany.edu (Justin Azoff)
Date: Mon, 2 Nov 2009 08:14:38 -0500
Subject: [Bro] Small bug in file rotate+setbuf
Message-ID: <20091102131437.GA12593@dell.datacomm.albany.edu>

I tried to submit a ticket for this but it kept triggering the spam detecting...

When files are rotated they lose their buffered flag, this is because
File::Open only does a SetBuf when it opens the file itself, but Rotate calls
rotate_file to open the file.

It's a one line fix, I'm not sure if it would be better to call SetBuf at the
end of Rotate or if the call to SetBuf in Open should be moved outside of the
if( ! f ) block.

-- 
-- Justin Azoff
-- Network Performance Analyst


From antonionestola_ at libero.it  Mon Nov  2 05:33:28 2009
From: antonionestola_ at libero.it (antonionestola_ at libero.it)
Date: Mon,  2 Nov 2009 14:33:28 +0100
Subject: [Bro] read trace  offline
Message-ID: <KSHHNS$4A7B18C82A760ABE3116A55ECCD107AC@libero.it>

Hi,I have a stupid question:Can I do an offline-analysis with Bro of a trace file in pcap form?thank you..





From hall.692 at osu.edu  Mon Nov  2 06:19:25 2009
From: hall.692 at osu.edu (Seth Hall)
Date: Mon, 2 Nov 2009 09:19:25 -0500
Subject: [Bro] Small bug in file rotate+setbuf
In-Reply-To: <20091102131437.GA12593@dell.datacomm.albany.edu>
References: <20091102131437.GA12593@dell.datacomm.albany.edu>
Message-ID: <6488D8AD-793A-45D4-948A-569D0DF6D445@osu.edu>


On Nov 2, 2009, at 8:14 AM, Justin Azoff wrote:

> I tried to submit a ticket for this but it kept triggering the spam  
> detecting...

Sorry about that, I'll add an account for you now.

If anyone else runs into issues with this, send me or Robin an email  
and we'll get you setup with an account on tracker.  We've had to  
continually make the ticket tracker more and more restricted due to  
trouble with spammers.  At this point, you really need to have an  
account to file or comment on a ticket.

> When files are rotated they lose their buffered flag, this is because
> File::Open only does a SetBuf when it opens the file itself, but  
> Rotate calls
> rotate_file to open the file.

This seems to be related the issue that seems to be coming up more and  
more frequently where any attribute applied to a file is lost if a new  
file is assigned to the variable.  I'm wondering if when a file value  
is assigned to a variable, we should be copying the attributes from  
the existing variable to the value that's going to be assigned.  It  
would mean that the attributes are more directly attached to the  
variable than the value for files, but I'm starting the think that's  
the correct behavior.  Robin, Vern?

   .Seth

---
Seth Hall
Network Security - Office of the CIO
The Ohio State University
Phone: 614-292-9721



From hall.692 at osu.edu  Mon Nov  2 06:20:03 2009
From: hall.692 at osu.edu (Seth Hall)
Date: Mon, 2 Nov 2009 09:20:03 -0500
Subject: [Bro] read trace  offline
In-Reply-To: <KSHHNS$4A7B18C82A760ABE3116A55ECCD107AC@libero.it>
References: <KSHHNS$4A7B18C82A760ABE3116A55ECCD107AC@libero.it>
Message-ID: <28C13455-B5B1-40B6-843B-15E9EEF47C70@osu.edu>


On Nov 2, 2009, at 8:33 AM, antonionestola_ at libero.it wrote:

> Hi,I have a stupid question:Can I do an offline-analysis with Bro of  
> a trace file in pcap form?thank you..


The Bro binary has the "-r" option similar to tcpdump for reading in  
pcap formatted tracefiles.

   .Seth

---
Seth Hall
Network Security - Office of the CIO
The Ohio State University
Phone: 614-292-9721



From grutz at jingojango.net  Mon Nov  2 20:45:36 2009
From: grutz at jingojango.net (Kurt Grutzmacher)
Date: Mon, 2 Nov 2009 20:45:36 -0800
Subject: [Bro] brolval undefined?
Message-ID: <cf4f8b670911022045t375b7324i91c9edc1def42d36@mail.gmail.com>

Odd error cropped up while trying to build latest svn trunk release on
ubuntu 9.04 64-bit:

~/bro-svn/bro-102809/aux/broccoli/test$ make
/bin/bash ../libtool --tag=CC   --mode=link gcc  -g -O2   -o broping
broping.o ../src/.libs/libbroccoli.a  -lssl -lcrypto
libtool: link: gcc -g -O2 -o broping broping.o  ../src/.libs/libbroccoli.a
-lssl -lcrypto
../src/.libs/libbroccoli.a(bro_lexer.o): In function `brolex':
/home/grutz/bro-svn/bro-102809/aux/broccoli/src/bro_lexer.l:25: undefined
reference to `brolval'
/home/grutz/bro-svn/bro-102809/aux/broccoli/src/bro_lexer.l:26: undefined
reference to `brolval'
/home/grutz/bro-svn/bro-102809/aux/broccoli/src/bro_lexer.l:28: undefined
reference to `brolval'
/home/grutz/bro-svn/bro-102809/aux/broccoli/src/bro_lexer.l:29: undefined
reference to `brolval'
/home/grutz/bro-svn/bro-102809/aux/broccoli/src/bro_lexer.l:30: undefined
reference to `brolval'
../src/.libs/libbroccoli.a(bro_lexer.o):/home/grutz/bro-svn/bro-102809/aux/broccoli/src/bro_lexer.l:31:
more undefined references to `brolval' follow
collect2: ld returned 1 exit status
make: *** [broping] Error 1

GCC Version:

~/bro-svn/bro-102809/aux/broccoli/src$ gcc -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu
4.3.3-5ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.3/README.Bugs
--enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared
--with-system-zlib --libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix --enable-nls
--with-gxx-include-dir=/usr/include/c++/4.3 --program-suffix=-4.3
--enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr
--with-tune=generic --enable-checking=release --build=x86_64-linux-gnu
--host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4)

Of course brolval IS being set .. contents of bro_parser.h:

#define BROINT 257
#define BROWORD 258
#define BROSTRING 259
#define BRODOUBLE 260
typedef union {
  int   i;
  char *s;
  double d;
} YYSTYPE;
extern YYSTYPE brolval;

Doesn't seem to make much sense to me. Any thoughts?

--
Kurt Grutzmacher -=- grutz at jingojango.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20091102/4839eb07/attachment.html 

From JAzoff at uamail.albany.edu  Tue Nov  3 13:52:20 2009
From: JAzoff at uamail.albany.edu (Justin Azoff)
Date: Tue, 3 Nov 2009 16:52:20 -0500
Subject: [Bro] strange dropped packets issue
Message-ID: <20091103215220.GA23722@dell.datacomm.albany.edu>

I've been trying to enable drop-adapt, but I've run into a really odd issue
with dropped packets.

Things start out working fine, but then as soon as any packets are dropped it
goes all the way back up to Level 10 and stays there:

Nov  3 15:49:55 switched to LoadLevel9
Nov  3 15:52:25 switched to LoadLevel8
Nov  3 15:54:55 switched to LoadLevel7
Nov  3 15:57:25 switched to LoadLevel6
Nov  3 15:59:55 switched to LoadLevel5
Nov  3 16:01:55 switched to LoadLevel6
Nov  3 16:02:45 switched to LoadLevel7
Nov  3 16:03:35 switched to LoadLevel8
Nov  3 16:04:25 switched to LoadLevel9
Nov  3 16:05:15 switched to LoadLevel10

netstats will then show dropped increasing at about 80% the rate of recvd.

The odd part is if I run capstats with the -f option corresponding to the Level
10 filter and run netstats in 10 seconds intervals, the pkts= matches up almost
exactly.

So it seems that Bro isn't actually dropping any packets, but it thinks it is.
If I restart bro, it goes right back to 0 dropped packets.

I think I'm running into some sort of libpcap issue on Linux, but I'm not sure.
It seems everything goes wrong as soon as it starts changing the capture filter
once packets are dropped.  Though it might just be that things go wrong once
packets are dropped in general, but I don't really know how to test that.

-- 
-- Justin Azoff
-- Network Performance Analyst


From christian at whoop.org  Tue Nov  3 14:31:09 2009
From: christian at whoop.org (Christian Kreibich)
Date: Tue, 03 Nov 2009 14:31:09 -0800
Subject: [Bro] brolval undefined?
In-Reply-To: <cf4f8b670911022045t375b7324i91c9edc1def42d36@mail.gmail.com>
References: <cf4f8b670911022045t375b7324i91c9edc1def42d36@mail.gmail.com>
Message-ID: <1257287469.21659.34.camel@localhost.localdomain>

Hi Kurt,

On Mon, 2009-11-02 at 20:45 -0800, Kurt Grutzmacher wrote:
> ../src/.libs/libbroccoli.a(bro_lexer.o): In function `brolex':
> /home/grutz/bro-svn/bro-102809/aux/broccoli/src/bro_lexer.l:25:
> undefined reference to `brolval'
[...]

> Doesn't seem to make much sense to me. Any thoughts?

Mhmm ... I suspect what's going on here is that there is confusion in
the configure setup when it tries to detect your lex/yacc availability.
(We try to use lex/yacc when available to generate up-to-date parser
code, but fall back to pregenerated C code when they're not available.)

So as a workaround, I'd suggest making sure that you have some version
of lex and yacc installed (likely flex and bison). The output of the
configure script will tell you whether they were detected successfully.

In any case, could you please send me the full config.log from
aux/broccoli?

-- 
Cheers,
Christian



From jones at tacc.utexas.edu  Tue Nov  3 15:03:28 2009
From: jones at tacc.utexas.edu (William L. Jones)
Date: Tue, 3 Nov 2009 17:03:28 -0600
Subject: [Bro] strange dropped packets issue
In-Reply-To: <20091103215220.GA23722@dell.datacomm.albany.edu>
References: <20091103215220.GA23722@dell.datacomm.albany.edu>
Message-ID: <0E07074B82CE4B4A9982802A8484B696861FB4B6D6@EXCHANGE2K7.tacc.utexas.edu>

Is your bro running on a linux platform? 

Bill Jones

-----Original Message-----
From: bro-bounces at ICSI.Berkeley.EDU [mailto:bro-bounces at ICSI.Berkeley.EDU] On Behalf Of Justin Azoff
Sent: Tuesday, November 03, 2009 3:52 PM
To: bro at ICSI.Berkeley.EDU
Subject: [Bro] strange dropped packets issue

I've been trying to enable drop-adapt, but I've run into a really odd issue
with dropped packets.

Things start out working fine, but then as soon as any packets are dropped it
goes all the way back up to Level 10 and stays there:

Nov  3 15:49:55 switched to LoadLevel9
Nov  3 15:52:25 switched to LoadLevel8
Nov  3 15:54:55 switched to LoadLevel7
Nov  3 15:57:25 switched to LoadLevel6
Nov  3 15:59:55 switched to LoadLevel5
Nov  3 16:01:55 switched to LoadLevel6
Nov  3 16:02:45 switched to LoadLevel7
Nov  3 16:03:35 switched to LoadLevel8
Nov  3 16:04:25 switched to LoadLevel9
Nov  3 16:05:15 switched to LoadLevel10

netstats will then show dropped increasing at about 80% the rate of recvd.

The odd part is if I run capstats with the -f option corresponding to the Level
10 filter and run netstats in 10 seconds intervals, the pkts= matches up almost
exactly.

So it seems that Bro isn't actually dropping any packets, but it thinks it is.
If I restart bro, it goes right back to 0 dropped packets.

I think I'm running into some sort of libpcap issue on Linux, but I'm not sure.
It seems everything goes wrong as soon as it starts changing the capture filter
once packets are dropped.  Though it might just be that things go wrong once
packets are dropped in general, but I don't really know how to test that.

-- 
-- Justin Azoff
-- Network Performance Analyst
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From jones at tacc.utexas.edu  Tue Nov  3 15:12:49 2009
From: jones at tacc.utexas.edu (William L. Jones)
Date: Tue, 3 Nov 2009 17:12:49 -0600
Subject: [Bro] strange dropped packets issue
In-Reply-To: <20091103215220.GA23722@dell.datacomm.albany.edu>
References: <20091103215220.GA23722@dell.datacomm.albany.edu>
Message-ID: <0E07074B82CE4B4A9982802A8484B696861FB4B6DB@EXCHANGE2K7.tacc.utexas.edu>

Should have read all you message.  At some point libpcap was change to have the same semantics as libpcap on bsd, the error counter return the number of errors seen on the interfaces was opened instead of the old linux behavior of return the number of errors sense the previous status. 

  
In PktSrc.cc change:

#ifdef HAVE_LINUX
        // Linux clears its counters each time.
        s->received = stats.received;
        s->dropped = stats.dropped + pstat.ps_drop;
        s->link = 0; // not available
#else
        // Default assumes FreeBSD's semantics.
        s->received = stats.received;
        s->dropped = pstat.ps_drop;
        s->link = pstat.ps_recv;
#endif

To


        // Default assumes FreeBSD's semantics.
        s->received = stats.received;
        s->dropped = pstat.ps_drop;
        s->link = pstat.ps_recv;


Bill Jones


-----Original Message-----
From: bro-bounces at ICSI.Berkeley.EDU [mailto:bro-bounces at ICSI.Berkeley.EDU] On Behalf Of Justin Azoff
Sent: Tuesday, November 03, 2009 3:52 PM
To: bro at ICSI.Berkeley.EDU
Subject: [Bro] strange dropped packets issue

I've been trying to enable drop-adapt, but I've run into a really odd issue
with dropped packets.

Things start out working fine, but then as soon as any packets are dropped it
goes all the way back up to Level 10 and stays there:

Nov  3 15:49:55 switched to LoadLevel9
Nov  3 15:52:25 switched to LoadLevel8
Nov  3 15:54:55 switched to LoadLevel7
Nov  3 15:57:25 switched to LoadLevel6
Nov  3 15:59:55 switched to LoadLevel5
Nov  3 16:01:55 switched to LoadLevel6
Nov  3 16:02:45 switched to LoadLevel7
Nov  3 16:03:35 switched to LoadLevel8
Nov  3 16:04:25 switched to LoadLevel9
Nov  3 16:05:15 switched to LoadLevel10

netstats will then show dropped increasing at about 80% the rate of recvd.

The odd part is if I run capstats with the -f option corresponding to the Level
10 filter and run netstats in 10 seconds intervals, the pkts= matches up almost
exactly.

So it seems that Bro isn't actually dropping any packets, but it thinks it is.
If I restart bro, it goes right back to 0 dropped packets.

I think I'm running into some sort of libpcap issue on Linux, but I'm not sure.
It seems everything goes wrong as soon as it starts changing the capture filter
once packets are dropped.  Though it might just be that things go wrong once
packets are dropped in general, but I don't really know how to test that.

-- 
-- Justin Azoff
-- Network Performance Analyst
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From JAzoff at uamail.albany.edu  Tue Nov  3 15:32:06 2009
From: JAzoff at uamail.albany.edu (Justin Azoff)
Date: Tue, 3 Nov 2009 18:32:06 -0500
Subject: [Bro] strange dropped packets issue
In-Reply-To: <0E07074B82CE4B4A9982802A8484B696861FB4B6DB@EXCHANGE2K7.tacc.utexas.edu>
References: <20091103215220.GA23722@dell.datacomm.albany.edu>
	<0E07074B82CE4B4A9982802A8484B696861FB4B6DB@EXCHANGE2K7.tacc.utexas.edu>
Message-ID: <20091103233206.GB23722@dell.datacomm.albany.edu>

On Tue, Nov 03, 2009 at 05:12:49PM -0600, William L. Jones wrote:
> Should have read all you message.  At some point libpcap was change to have the same semantics as libpcap on bsd, the error counter return the number of errors seen on the interfaces was opened instead of the old linux behavior of return the number of errors sense the previous status. 
> 
>   
> In PktSrc.cc change:
> 
> #ifdef HAVE_LINUX
>         // Linux clears its counters each time.
>         s->received = stats.received;
>         s->dropped = stats.dropped + pstat.ps_drop;
>         s->link = 0; // not available
> #else
>         // Default assumes FreeBSD's semantics.
>         s->received = stats.received;
>         s->dropped = pstat.ps_drop;
>         s->link = pstat.ps_recv;
> #endif
> 
> To
> 
> 
>         // Default assumes FreeBSD's semantics.
>         s->received = stats.received;
>         s->dropped = pstat.ps_drop;
>         s->link = pstat.ps_recv;
> 
> 
> Bill Jones

Ah, I think you nailed it.  I took a look at some values of dropped, and not
only were they going up, they were going up by multiples of the same exact
value.  This exaplains why I was seeing a huge number of dropped packets but no
cpu load.

With that fixed, I have found another bug, I notice when bro starts it prints to the load log:

"0.000000 switched to LoadLevel9"

(I set the default to 9, but it's the same with 10)

But it does not actually change the pcap filter.

I's only when the load level changes:
"1257290873.632846 switched to LoadLevel8"

does the packet filter change.  I bet this is a race condition somewhere, the
fact that the time is 0 probably means something isn't fully setup yet.

-- 
-- Justin Azoff
-- Network Performance Analyst


From robin at icir.org  Tue Nov  3 15:40:13 2009
From: robin at icir.org (Robin Sommer)
Date: Tue, 3 Nov 2009 15:40:13 -0800
Subject: [Bro] strange dropped packets issue
In-Reply-To: <20091103215220.GA23722@dell.datacomm.albany.edu>
References: <20091103215220.GA23722@dell.datacomm.albany.edu>
Message-ID: <20091103234013.GG83401@icir.org>


On Tue, Nov 03, 2009 at 16:52 -0500, Justin Azoff wrote:

> I think I'm running into some sort of libpcap issue on Linux, but I'm not sure.

Any chance it's this?

    http://tracker.icir.org/bro/ticket/18

Robin

-- 
Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org 
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From hall.692 at osu.edu  Wed Nov  4 06:45:00 2009
From: hall.692 at osu.edu (Seth Hall)
Date: Wed, 4 Nov 2009 09:45:00 -0500
Subject: [Bro] strange dropped packets issue
In-Reply-To: <20091103234013.GG83401@icir.org>
References: <20091103215220.GA23722@dell.datacomm.albany.edu>
	<20091103234013.GG83401@icir.org>
Message-ID: <0BFB5177-0FE1-450D-B978-9E170C16A3F7@osu.edu>


On Nov 3, 2009, at 6:40 PM, Robin Sommer wrote:
>
> On Tue, Nov 03, 2009 at 16:52 -0500, Justin Azoff wrote:
>
>> I think I'm running into some sort of libpcap issue on Linux, but  
>> I'm not sure.
>
> Any chance it's this?
>
>    http://tracker.icir.org/bro/ticket/18


Would it make sense to just remove the linux specific code?  Is anyone  
running older Linux distros for their analyzers?

Based on one of the links in the ticket, it looks like Debian updated  
their libpcap and got rid of this issue in 2005.

   .Seth

---
Seth Hall
Network Security - Office of the CIO
The Ohio State University
Phone: 614-292-9721



From jones at tacc.utexas.edu  Wed Nov  4 07:39:25 2009
From: jones at tacc.utexas.edu (William L. Jones)
Date: Wed, 4 Nov 2009 09:39:25 -0600
Subject: [Bro] strange dropped packets issue
In-Reply-To: <0BFB5177-0FE1-450D-B978-9E170C16A3F7@osu.edu>
References: <20091103215220.GA23722@dell.datacomm.albany.edu>
	<20091103234013.GG83401@icir.org>
	<0BFB5177-0FE1-450D-B978-9E170C16A3F7@osu.edu>
Message-ID: <0E07074B82CE4B4A9982802A8484B696861FB4B733@EXCHANGE2K7.tacc.utexas.edu>

I would remove the code.  

The change to libpcap was made years ago.   It won't break the really old system and would make it easier for bro install an new linux systems - you won't have to change the code by hand!

-----Original Message-----
From: bro-bounces at ICSI.Berkeley.EDU [mailto:bro-bounces at ICSI.Berkeley.EDU] On Behalf Of Seth Hall
Sent: Wednesday, November 04, 2009 8:45 AM
To: Robin Sommer
Cc: Justin Azoff; bro at ICSI.Berkeley.EDU
Subject: Re: [Bro] strange dropped packets issue


On Nov 3, 2009, at 6:40 PM, Robin Sommer wrote:
>
> On Tue, Nov 03, 2009 at 16:52 -0500, Justin Azoff wrote:
>
>> I think I'm running into some sort of libpcap issue on Linux, but  
>> I'm not sure.
>
> Any chance it's this?
>
>    http://tracker.icir.org/bro/ticket/18


Would it make sense to just remove the linux specific code?  Is anyone  
running older Linux distros for their analyzers?

Based on one of the links in the ticket, it looks like Debian updated  
their libpcap and got rid of this issue in 2005.

   .Seth

---
Seth Hall
Network Security - Office of the CIO
The Ohio State University
Phone: 614-292-9721

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From robin at icir.org  Wed Nov  4 16:21:03 2009
From: robin at icir.org (Robin Sommer)
Date: Wed, 4 Nov 2009 16:21:03 -0800
Subject: [Bro] Bro tutorial at ACSAC
Message-ID: <20091105002103.GA78715@icir.org>

A reminder for folks interested in more Bro training: Vern and I
will be giving a one-day Bro tutorial next month at this year's
"Annual Computer Security Applications Conference" (ACSAC). See here
for more information:

   http://www.acsac.org/2009/program/tutorials/view.php?t=3

Registration for the tutorial is open. Feel free to forward this to
other people you think might be interested. 

Robin      

-- 
Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org 
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From kosinovsky1 at llnl.gov  Wed Nov  4 16:34:19 2009
From: kosinovsky1 at llnl.gov (Greg Kosinovsky)
Date: Wed, 04 Nov 2009 16:34:19 -0800
Subject: [Bro] looking for bro script
In-Reply-To: <7.1.0.9.2.20091008121646.025c5ae0@llnl.gov>
References: <7.1.0.9.2.20090728110016.02550740@llnl.gov>
	<7.1.0.9.2.20091008121646.025c5ae0@llnl.gov>
Message-ID: <80955b$29qfa7@smtp.llnl.gov>

Is there an existing bro script that sessionizes the incoming packet 
flow but keeps all of the original packets associated with each 
recognized session?

This seems like something fairly "standard", so I probably don't need 
to reinvent the wheel on this one.

Thank you,

Greg



From vern at icir.org  Thu Nov  5 06:22:15 2009
From: vern at icir.org (Vern Paxson)
Date: Thu, 05 Nov 2009 06:22:15 -0800
Subject: [Bro] looking for bro script
In-Reply-To: <80955b$29qfa7@smtp.llnl.gov> (Wed, 04 Nov 2009 16:34:19 PST).
Message-ID: <200911051422.nA5EMKIS015506@pork.ICSI.Berkeley.EDU>

> Is there an existing bro script that sessionizes the incoming packet 
> flow but keeps all of the original packets associated with each 
> recognized session?

I'm not following - what do you mean by "sessionizes"?

		Vern


From kosinovsky1 at llnl.gov  Thu Nov  5 15:21:28 2009
From: kosinovsky1 at llnl.gov (Greg Kosinovsky)
Date: Thu, 05 Nov 2009 15:21:28 -0800
Subject: [Bro] looking for bro script
In-Reply-To: <200911051422.nA5EMKIS015506@pork.ICSI.Berkeley.EDU>
References: <80955b$29qfa7@smtp.llnl.gov>
	<200911051422.nA5EMKIS015506@pork.ICSI.Berkeley.EDU>
Message-ID: <806j7o$1oo8i4@nspiron-2.llnl.gov>

Basically I mean collects and orders the packets from the same flow 
(such as an http session, but not limited to any particular protocol).

Thank you.

Greg

At 06:22 AM 11/5/2009, Vern Paxson wrote:
> > Is there an existing bro script that sessionizes the incoming packet
> > flow but keeps all of the original packets associated with each
> > recognized session?
>
>I'm not following - what do you mean by "sessionizes"?
>
>                 Vern




From rob.shanley at gmail.com  Mon Nov  9 11:03:19 2009
From: rob.shanley at gmail.com (Rob Shanley)
Date: Mon, 9 Nov 2009 14:03:19 -0500
Subject: [Bro] Automated Identification of Notice/Alarm Generating Packets
Message-ID: <d3663e260911091103u436a3cfelaf1766b45e5cb49a@mail.gmail.com>

Hi Bro List,

I am new to bro and I'm trying to find out if there is an easy / automated
way to identify the packets that triggered a notice/alarm. I am focused on
offline-analysis. For example can you use a technique to read/write, ex. -r
a.pcap -w b.pcap, where a.pcap contains all traffic and b.pcap contains all
traffic that triggered an alarm or conversely all traffic that did not
trigger an alarm? This would be similar to how snort writes a pcap log of
suspect traffic. If this is not possible is there sufficient information in
the logs to identify individual packets that triggered an alarm/notice? And
are there any bundled tools for faciliating this process? I am focused on
http traffic.

Essentialy my goal is to pass large sets of data through bro and segregate
the traffic into either clean or suspect subsets. I have done some searching
on the wiki but it seems that most of my leads bring me to the "Reference
Manual: Missing Documentation" page.

I greatly appreciate any guidance you can provide!

Thanks,
-Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20091109/31ddd368/attachment.html 

From rmkml at free.fr  Mon Nov  9 01:49:52 2009
From: rmkml at free.fr (rmkml)
Date: Mon, 9 Nov 2009 10:49:52 +0100 (CET)
Subject: [Bro] Automated Identification of Notice/Alarm Generating
 Packets
In-Reply-To: <d3663e260911091103u436a3cfelaf1766b45e5cb49a@mail.gmail.com>
References: <d3663e260911091103u436a3cfelaf1766b45e5cb49a@mail.gmail.com>
Message-ID: <alpine.DEB.1.00.0911091049400.17355@macmini>

Hi Rob,
I don't known if bro write (-w) exactly then snort,
but bro have another good options:
-all (http) log contains unix timestamp and http session number
-http log contains uri request and http code reply
-http mime extract/decoding/parsing (zip)
-dont miss binpac and dpd option for recognize http trafic on all port (require bpf filters)
-bro http/http-request/http-request-body/http-reply-body/http-body/http-request-header/http-reply-header option
-powerfull bro script language
Regards
Rmkml
Crusoe-Researches.com


On Mon, 9 Nov 2009, Rob Shanley wrote:

> Hi Bro List,
>
> I am new to bro and I'm trying to find out if there is an easy / automated
> way to identify the packets that triggered a notice/alarm. I am focused on
> offline-analysis. For example can you use a technique to read/write, ex. -r
> a.pcap -w b.pcap, where a.pcap contains all traffic and b.pcap contains all
> traffic that triggered an alarm or conversely all traffic that did not
> trigger an alarm? This would be similar to how snort writes a pcap log of
> suspect traffic. If this is not possible is there sufficient information in
> the logs to identify individual packets that triggered an alarm/notice? And
> are there any bundled tools for faciliating this process? I am focused on
> http traffic.
>
> Essentialy my goal is to pass large sets of data through bro and segregate
> the traffic into either clean or suspect subsets. I have done some searching
> on the wiki but it seems that most of my leads bring me to the "Reference
> Manual: Missing Documentation" page.
>
> I greatly appreciate any guidance you can provide!
>
> Thanks,
> -Rob
>


From robin at icir.org  Mon Nov  9 21:20:46 2009
From: robin at icir.org (Robin Sommer)
Date: Mon, 9 Nov 2009 21:20:46 -0800
Subject: [Bro] Automated Identification of Notice/Alarm
	Generating	Packets
In-Reply-To: <d3663e260911091103u436a3cfelaf1766b45e5cb49a@mail.gmail.com>
References: <d3663e260911091103u436a3cfelaf1766b45e5cb49a@mail.gmail.com>
Message-ID: <20091110052046.GI83713@icir.org>


On Mon, Nov 09, 2009 at 14:03 -0500, you wrote:

> I am new to bro and I'm trying to find out if there is an easy / automated
> way to identify the packets that triggered a notice/alarm. 

No, not really. The main reason is that at the point when the
decision is taken Bro doesn't really have the notion of packets
anymore, it's working at a higher semantic level and it's in general
not possible to go back and pinpoint individual packets which led to
the decision. 

What often works well however is doing this at the connection/flow
level. Most alarms are associated with a particular connection and
once one has the 4-tuple of host & ports, one can extract the
connection's packet from the input. With an offline analysis, that
should be pretty straight-forward to do. For alarms not associated
with a connection, there's usually still at least a certain IP
involved and one could filter for that (depends on your application
whether that makes sense or not I guess). 

There's also the "Time Machine"[1], which can buffer large amounts of
packets and provides an interface to, e.g., extract individual
packets from its buffers. The TM can also work offline from traces. 

Robin

[1] http://www.net.t-labs.tu-berlin.de/research/tm/

-- 
Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org 
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From rob.shanley at gmail.com  Tue Nov 10 05:30:14 2009
From: rob.shanley at gmail.com (Rob Shanley)
Date: Tue, 10 Nov 2009 08:30:14 -0500
Subject: [Bro] Automated Identification of Notice/Alarm Generating
	Packets
In-Reply-To: <20091110052046.GI83713@icir.org>
References: <d3663e260911091103u436a3cfelaf1766b45e5cb49a@mail.gmail.com>
	<20091110052046.GI83713@icir.org>
Message-ID: <d3663e260911100530l5b001107jc2d911356c1a0311@mail.gmail.com>

Thank you for the prompt response! I will look into the connections/flows.

On Tue, Nov 10, 2009 at 12:20 AM, Robin Sommer <robin at icir.org> wrote:

>
> On Mon, Nov 09, 2009 at 14:03 -0500, you wrote:
>
> > I am new to bro and I'm trying to find out if there is an easy /
> automated
> > way to identify the packets that triggered a notice/alarm.
>
> No, not really. The main reason is that at the point when the
> decision is taken Bro doesn't really have the notion of packets
> anymore, it's working at a higher semantic level and it's in general
> not possible to go back and pinpoint individual packets which led to
> the decision.
>
> What often works well however is doing this at the connection/flow
> level. Most alarms are associated with a particular connection and
> once one has the 4-tuple of host & ports, one can extract the
> connection's packet from the input. With an offline analysis, that
> should be pretty straight-forward to do. For alarms not associated
> with a connection, there's usually still at least a certain IP
> involved and one could filter for that (depends on your application
> whether that makes sense or not I guess).
>
> There's also the "Time Machine"[1], which can buffer large amounts of
> packets and provides an interface to, e.g., extract individual
> packets from its buffers. The TM can also work offline from traces.
>
> Robin
>
> [1] http://www.net.t-labs.tu-berlin.de/research/tm/
>
> --
> Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org
> ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20091110/e34aa0d1/attachment.html 

From robin at icir.org  Tue Nov 10 09:45:11 2009
From: robin at icir.org (Robin Sommer)
Date: Tue, 10 Nov 2009 09:45:11 -0800
Subject: [Bro] Automated Identification of Notice/Alarm
	Generating	Packets
In-Reply-To: <d3663e260911100530l5b001107jc2d911356c1a0311@mail.gmail.com>
References: <d3663e260911091103u436a3cfelaf1766b45e5cb49a@mail.gmail.com>
	<20091110052046.GI83713@icir.org>
	<d3663e260911100530l5b001107jc2d911356c1a0311@mail.gmail.com>
Message-ID: <20091110174511.GF31226@icir.org>


On Tue, Nov 10, 2009 at 08:30 -0500, I wrote:

> > packets and provides an interface to, e.g., extract individual
> > packets from its buffers.

s/individual packets/individual connections/ 

Robin

-- 
Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org 
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From fh at open.ch  Wed Nov 11 05:34:25 2009
From: fh at open.ch (Fabian Hugelshofer)
Date: Wed, 11 Nov 2009 14:34:25 +0100
Subject: [Bro] BroPing Connection Failure
In-Reply-To: <4AEEAC2E.3020206@open.ch>
References: <4AEAFF34.5000304@open.ch> <20091030171341.GF13504@icir.org>
	<4AEEAC2E.3020206@open.ch>
Message-ID: <4AFABD61.7040100@open.ch>

Fabian Hugelshofer wrote:
> Robin Sommer wrote:
>> Sounds like a nasty race condition of some sort. The remote.log only
>> shows that something weird is going on but isn't detailed enough to
>> understand what's causing it. Please enable debugging output on both
>> sides. For Broccoli, see here:
> 
> I enabled debugging on the client with "-d -d" (bropingc.txt) and on the 
> server with "-B comm" (bropings.txt). I also attached a tcpdump trace of 
> the communication.

Are you still looking at this? Do you have an idea about the cause of
this behaviour? Or a hint on where to start looking?

Fabian


From Tyler.Schoenke at colorado.edu  Wed Nov 11 14:40:32 2009
From: Tyler.Schoenke at colorado.edu (Tyler Schoenke)
Date: Wed, 11 Nov 2009 15:40:32 -0700
Subject: [Bro] DNS logging
Message-ID: <4AFB3D60.1090208@colorado.edu>

I noticed that the DNS.log is no longer being created when running the 
cluster.  I don't think I changed anything to disable it.  It is still 
being loaded by local.bro.  I am running the latest trunk.

$ broctl print DNS::logging

    manager   DNS::logging = F
    proxy-1   DNS::logging = F
   worker-1   DNS::logging = F

The dns.bro script shows logging defaulted to true.  Where is it getting 
set to false when the cluster runs?


Tyler

--
Tyler Schoenke
Network Security Analyst
IT Security Office
University of Colorado - Boulder


From lruppert at syr.edu  Thu Nov 12 04:46:05 2009
From: lruppert at syr.edu (Louis F Ruppert)
Date: Thu, 12 Nov 2009 07:46:05 -0500
Subject: [Bro] DNS logging
In-Reply-To: <4AFB3D60.1090208@colorado.edu>
References: <4AFB3D60.1090208@colorado.edu>
Message-ID: <32E412533D42EF41ACBCFA96B6C230F4E202A078B9@SUEX07-MBX-07.ad.syr.edu>

Mine is set to false here:

$BROHOME/share/bro/broctl/cluster.dns.bro


-Lou
(who also spent some time trying to figure this out)

________________________________________
From: bro-bounces at ICSI.Berkeley.EDU [bro-bounces at ICSI.Berkeley.EDU] On Behalf Of Tyler Schoenke [Tyler.Schoenke at colorado.edu]
Sent: Wednesday, November 11, 2009 5:40 PM
To: bro at ICSI.Berkeley.EDU
Subject: [Bro] DNS logging

I noticed that the DNS.log is no longer being created when running the
cluster.  I don't think I changed anything to disable it.  It is still
being loaded by local.bro.  I am running the latest trunk.

$ broctl print DNS::logging

    manager   DNS::logging = F
    proxy-1   DNS::logging = F
   worker-1   DNS::logging = F

The dns.bro script shows logging defaulted to true.  Where is it getting
set to false when the cluster runs?


Tyler

--
Tyler Schoenke
Network Security Analyst
IT Security Office
University of Colorado - Boulder
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From Tyler.Schoenke at colorado.edu  Thu Nov 12 08:03:51 2009
From: Tyler.Schoenke at colorado.edu (Tyler Schoenke)
Date: Thu, 12 Nov 2009 09:03:51 -0700
Subject: [Bro] DNS logging
In-Reply-To: <32E412533D42EF41ACBCFA96B6C230F4E202A078B9@SUEX07-MBX-07.ad.syr.edu>
References: <4AFB3D60.1090208@colorado.edu>
	<32E412533D42EF41ACBCFA96B6C230F4E202A078B9@SUEX07-MBX-07.ad.syr.edu>
Message-ID: <4AFC31E7.3030201@colorado.edu>

Thanks, that did the trick. -Tyler

On 11/12/2009 05:46 AM, Louis F Ruppert wrote:
> Mine is set to false here:
>
> $BROHOME/share/bro/broctl/cluster.dns.bro
>
>
> -Lou
> (who also spent some time trying to figure this out)
>


From robin at icir.org  Thu Nov 12 17:40:53 2009
From: robin at icir.org (Robin Sommer)
Date: Thu, 12 Nov 2009 17:40:53 -0800
Subject: [Bro] DNS logging
In-Reply-To: <32E412533D42EF41ACBCFA96B6C230F4E202A078B9@SUEX07-MBX-07.ad.syr.edu>
References: <4AFB3D60.1090208@colorado.edu>
	<32E412533D42EF41ACBCFA96B6C230F4E202A078B9@SUEX07-MBX-07.ad.syr.edu>
Message-ID: <20091113014053.GR37620@icir.org>


On Thu, Nov 12, 2009 at 07:46 -0500, Louis F Ruppert wrote:

> $BROHOME/share/bro/broctl/cluster.dns.bro

Yes, indeed. The cluster config is changing some defaults to values
which seem to be more reasonable in a large setting. It's of course
debatable what the definition of "reasonable" here is :-) With DNS
one gets these huge logs which often aren't very helpful. 

So, the general guideline is when you're looking for a specific
setting, also grep through the cluster's *.bro scripts.

> (who also spent some time trying to figure this out)

Sorry. :)

Robin

-- 
Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org 
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From robin at icir.org  Fri Nov 13 14:57:59 2009
From: robin at icir.org (Robin Sommer)
Date: Fri, 13 Nov 2009 14:57:59 -0800
Subject: [Bro] BroPing Connection Failure
In-Reply-To: <4AEEAC2E.3020206@open.ch>
References: <4AEAFF34.5000304@open.ch> <20091030171341.GF13504@icir.org>
	<4AEEAC2E.3020206@open.ch>
Message-ID: <20091113225759.GD11370@icir.org>


On Mon, Nov 02, 2009 at 10:53 +0100, Fabian Hugelshofer wrote:

> I found out that compiling without "--disable-select-loop" doesn't show  

Before digging into this, what's the reason for compiling with
--disable-select-loop? Seeing that the problem disappears without
that switch, the problem likley lies somewhere in the alternative
code path it enables. That code is rather old and rarely used these
days, and I'm thinking it can actually be removed completely in
future versions. So, what's your use case here?

Robin

-- 
Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org 
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From fh at open.ch  Sun Nov 15 23:29:26 2009
From: fh at open.ch (Fabian Hugelshofer)
Date: Mon, 16 Nov 2009 08:29:26 +0100
Subject: [Bro] BroPing Connection Failure
In-Reply-To: <20091113225759.GD11370@icir.org>
References: <4AEAFF34.5000304@open.ch> <20091030171341.GF13504@icir.org>
	<4AEEAC2E.3020206@open.ch> <20091113225759.GD11370@icir.org>
Message-ID: <4B00FF56.7010309@open.ch>

Robin Sommer wrote:
> Before digging into this, what's the reason for compiling with
> --disable-select-loop? Seeing that the problem disappears without
> that switch, the problem likley lies somewhere in the alternative
> code path it enables. That code is rather old and rarely used these
> days, and I'm thinking it can actually be removed completely in
> future versions. So, what's your use case here?

I use it because without, Bro causes a high CPU usage, even if there 
isn't any traffic to analyse. That's about 30% CPU usage on a Pentium 4 
with 2.4 Ghz.

I read about using --disable-select-loop on 
http://bro-ids.org/wiki/index.php/User_Manual:_Performance_Tuning. There 
it sais that Phil Wood's libpcap is buggy in non-blocking mode. I'm 
using Phil's libpcap (0.9.8.20081128).

Fabian


From hall.692 at osu.edu  Mon Nov 16 04:39:04 2009
From: hall.692 at osu.edu (Seth Hall)
Date: Mon, 16 Nov 2009 07:39:04 -0500
Subject: [Bro] BroPing Connection Failure
In-Reply-To: <4B00FF56.7010309@open.ch>
References: <4AEAFF34.5000304@open.ch> <20091030171341.GF13504@icir.org>
	<4AEEAC2E.3020206@open.ch> <20091113225759.GD11370@icir.org>
	<4B00FF56.7010309@open.ch>
Message-ID: <0102901F-DF69-4ADB-A942-1A2F501C800F@osu.edu>


On Nov 16, 2009, at 2:29 AM, Fabian Hugelshofer wrote:

> I use it because without, Bro causes a high CPU usage, even if there
> isn't any traffic to analyse. That's about 30% CPU usage on a  
> Pentium 4
> with 2.4 Ghz.


If you are running Bro 1.4 and loading listen-clear.bro, you could be  
encountering this issue:
   http://tracker.icir.org/bro/ticket/31

If that's where your problem is, it's fixed in trunk.

  .Seth

---
Seth Hall
Network Security - Office of the CIO
The Ohio State University
Phone: 614-292-9721



From robin at icir.org  Mon Nov 16 09:40:39 2009
From: robin at icir.org (Robin Sommer)
Date: Mon, 16 Nov 2009 09:40:39 -0800
Subject: [Bro] BroPing Connection Failure
In-Reply-To: <4B00FF56.7010309@open.ch>
References: <4AEAFF34.5000304@open.ch> <20091030171341.GF13504@icir.org>
	<4AEEAC2E.3020206@open.ch> <20091113225759.GD11370@icir.org>
	<4B00FF56.7010309@open.ch>
Message-ID: <20091116174039.GB9525@icir.org>


On Mon, Nov 16, 2009 at 08:29 +0100, Fabian Hugelshofer wrote:

> I read about using --disable-select-loop on  
> http://bro-ids.org/wiki/index.php/User_Manual:_Performance_Tuning. There  
> it sais that Phil Wood's libpcap is buggy in non-blocking mode. 

I see. That's actually pretty old and I don't remember what
specifically the problem was; it might be fixed with newer versions
of Phil's libpcap, don't know. Do you see the high CPU when using a
standard pcap?

The select-based loop was introduced specifically to combine pcap
input with simultaneous remote communication (like via Broccoli);
which is what you're doing it seems. The suggestion in the Wiki is
meant for configurations not involving any communication (even
though it doesn't say so ...), I wouldn't recommend using
--disable-select-loop when doing both as that won't work reliable.

Robin

P.S.: And yes, please try Seth's fix to see that already helps. 

-- 
Robin Sommer * Phone +1 (510) 666-2886 * robin at icir.org 
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From fh at open.ch  Fri Nov 20 04:29:09 2009
From: fh at open.ch (Fabian Hugelshofer)
Date: Fri, 20 Nov 2009 13:29:09 +0100
Subject: [Bro] BroPing Connection Failure
In-Reply-To: <20091116174039.GB9525@icir.org>
References: <4AEAFF34.5000304@open.ch> <20091030171341.GF13504@icir.org>
	<4AEEAC2E.3020206@open.ch> <20091113225759.GD11370@icir.org>
	<4B00FF56.7010309@open.ch> <20091116174039.GB9525@icir.org>
Message-ID: <4B068B95.4090605@open.ch>

Hi all,

Robin Sommer wrote:
> I see. That's actually pretty old and I don't remember what
> specifically the problem was; it might be fixed with newer versions
> of Phil's libpcap, don't know. Do you see the high CPU when using a
> standard pcap?
[...]
> P.S.: And yes, please try Seth's fix to see that already helps. 

I did some more experiments regarding this performance issue. Without
debug mode enabled, the CPU usage is not as high as I described
earlier. I am using our own policy scripts that include listen-clear.

 From a shell script, I ran Bro with different binaries for 10 minutes 
and then read the CPU time it consumed with top. There wasn't a lot of 
traffic in the network.

Bro 1.4 noloop 13.90s (parent 13.76s), child 0.14s)
Bro 1.4 default 51.06s (parent 26.56s, child 24.50s)
Bro 1.4 Patch 115.41s (parent 115.28s, child 0.13s)

The best result over all is with "--disable-select-loop". With the 
default options, the child process uses quite a lot of CPU time. Seth's 
patch fixes that for the child, but the CPU usage of the parent increases.

Bro 1.4 PCAP+default 39.21s (parent 21.05s, child 18.16s)
Bro 1.4 PCAP+Patch 82.70s (parent 81.96s, child 0.74s)

As I mentioned, I am using Phil Wood's libpcap (the current one). With 
the default version of libpcap, the performance is a bit better than 
with Phil Wood's. Still it's not great and Seth's patch again increased 
the CPU usage of the parent process.

Bro 1.5 84.66s (parent 84.41s, child 0.25s)

For the current version from trunk, it is a bit better than with 
1.4+patch. While being mainly idle, using blocking sockets is still the 
only acceptable solution for me. Bro can't use more than 10% CPU time 
for basically nothing.

Is there a way of reducing the negative effects from Seth's patch? It's 
a bit irritating that the CPU usage for the parent increases. I wouldn't 
have expected that.

Fabian


From hall.692 at osu.edu  Fri Nov 20 04:42:27 2009
From: hall.692 at osu.edu (Seth Hall)
Date: Fri, 20 Nov 2009 07:42:27 -0500
Subject: [Bro] BroPing Connection Failure
In-Reply-To: <4B068B95.4090605@open.ch>
References: <4AEAFF34.5000304@open.ch> <20091030171341.GF13504@icir.org>
	<4AEEAC2E.3020206@open.ch> <20091113225759.GD11370@icir.org>
	<4B00FF56.7010309@open.ch> <20091116174039.GB9525@icir.org>
	<4B068B95.4090605@open.ch>
Message-ID: <D9D6D941-3A16-4F97-96A6-01D9B8940F9C@osu.edu>


On Nov 20, 2009, at 7:29 AM, Fabian Hugelshofer wrote:

> Is there a way of reducing the negative effects from Seth's patch?  
> It's
> a bit irritating that the CPU usage for the parent increases. I  
> wouldn't
> have expected that.


What scripts are you loading?  Any custom ones?  I'd like to attempt  
to replicate the effects you're seeing.

Thanks,
   .Seth

---
Seth Hall
Network Security - Office of the CIO
The Ohio State University
Phone: 614-292-9721



From fh at open.ch  Fri Nov 20 06:07:41 2009
From: fh at open.ch (Fabian Hugelshofer)
Date: Fri, 20 Nov 2009 15:07:41 +0100
Subject: [Bro] BroPing Connection Failure
In-Reply-To: <D9D6D941-3A16-4F97-96A6-01D9B8940F9C@osu.edu>
References: <4AEAFF34.5000304@open.ch> <20091030171341.GF13504@icir.org>
	<4AEEAC2E.3020206@open.ch> <20091113225759.GD11370@icir.org>
	<4B00FF56.7010309@open.ch> <20091116174039.GB9525@icir.org>
	<4B068B95.4090605@open.ch>
	<D9D6D941-3A16-4F97-96A6-01D9B8940F9C@osu.edu>
Message-ID: <4B06A2AD.1010807@open.ch>

Seth Hall wrote:
> On Nov 20, 2009, at 7:29 AM, Fabian Hugelshofer wrote:
> 
>> Is there a way of reducing the negative effects from Seth's patch? It's
>> a bit irritating that the CPU usage for the parent increases. I wouldn't
>> have expected that.
> 
> 
> What scripts are you loading?  Any custom ones?  I'd like to attempt to 
> replicate the effects you're seeing.

Before I showed you the results from my custom scripts. To exclude any 
unnecessary influence, I let it run again: on the loopback device and 
just with broping.bro:

src/bro -i lo -C aux/broccoli/test/broping.bro

The results:
noloop     13.26
default    89.68
patch      77.14
pcap       32.7
pcap+patch 50.2
1.5        87.45

The regression of the patch is visible with the default version of 
libpcap, but not with Phil Wood's. Not sure wether I had a calculation 
error there before (dnt have the resutls from the first run anymore).

Fabian