From redlamb19 at gmail.com  Wed Feb  1 12:39:46 2012
From: redlamb19 at gmail.com (Peter Erickson)
Date: Wed, 1 Feb 2012 14:39:46 -0600
Subject: [Bro] Obtain src/dst mac addrs from connection record
In-Reply-To: <4684BC2D-2AE5-487A-8A92-919B703E2706@icir.org>
References: <20120131224659.GA32513@does.not.exist>
	<4684BC2D-2AE5-487A-8A92-919B703E2706@icir.org>
Message-ID: <20120201203946.GB18012@does.not.exist>

** Seth Hall <seth at icir.org> [2012-01-31 21:54:45 -0500] **
> 
> On Jan 31, 2012, at 5:47 PM, Peter Erickson wrote:
> 
> > Is there a way to obtain the source and/or destination mac address from
> > a connection record? I've been looking through the scripts and BIFs, but
> > am not seeing anything. I'm wondering it I missed something.
> 
> You didn't miss anything.  There is actually a very good reason that
> the MAC addresses aren't available.  Ethernet has no notion of a
> connection so a single connection could involve any number of IP
> addresses.  The connection you are looking into may not even be over
> ethernet so no MAC addresses would be available.  In most "normal"
> cases of border sniffing you will only see the MAC addresses of two
> routers anyway.

I figured that was the reason, but never hurts to ask.

> That said... you could probably make it work by writing a script that
> uses the ARP analyzer to create MAC->IP address mappings and then
> looking up the MAC address that is using a particular IP address.  You
> could even extend the conn.log file with orig_mac and resp_mac fields
> so that the MAC addresses would be located there.  I don't think
> that's something we would ship with Bro directly due to how deployment
> specific it would be (would work great on LAN span ports, but for
> border sniffing it would be useless).  It would be nice to have a
> script like that for our contributed scripts repository though!

For the reasons that you just said, I'm hesitant to add mac addresses to
the conn.log because it won't make much sense for 90% of all the traffic.
However, as mentioned offline, I'm re-writing the dhcp script from
pre-2.0 and identifying possible relays will be much easier with the ip
to mac addr mappings. So, with that said, I'll just use the arp script
that I previously provided.

Thanks for the feedback.

-- 
Peter Erickson
redlamb19 at gmail.com


From daveangelo831 at gmail.com  Thu Feb  2 03:54:35 2012
From: daveangelo831 at gmail.com (Dave Angelo)
Date: Thu, 2 Feb 2012 13:54:35 +0200
Subject: [Bro] BRO and SQL
Message-ID: <CAA=Y8dj9U7dG1QquiPhDxhbmkvmPtzAV2JQ0FY-LMAbByyzwmA@mail.gmail.com>

Hello,

I am trying to add BRO the ability to ignore traffic from certain IP ranges
dynamically.

I have a DB with IP addresses (that chances once in a while) and I would
like to write a BRO script that will query the DB once in a while, grab
those IP addresses and drop new connections with these IP's.

Question:
1. Is it possible to query a DB from BRO scripts? is there any examples?
2. Assuming yes, should i implement this logic at the 'new_connection'
event? (I would like to drop connections from these IP's as soon as
possible).

Thank You
Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120202/bbf408f7/attachment.html 

From seth at icir.org  Thu Feb  2 05:44:11 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 2 Feb 2012 08:44:11 -0500
Subject: [Bro] BRO and SQL
In-Reply-To: <CAA=Y8dj9U7dG1QquiPhDxhbmkvmPtzAV2JQ0FY-LMAbByyzwmA@mail.gmail.com>
References: <CAA=Y8dj9U7dG1QquiPhDxhbmkvmPtzAV2JQ0FY-LMAbByyzwmA@mail.gmail.com>
Message-ID: <054662CF-F77F-4F9E-A2BE-B7AF5AFF7459@icir.org>


On Feb 2, 2012, at 6:54 AM, Dave Angelo wrote:

> I am trying to add BRO the ability to ignore traffic from certain IP ranges dynamically.

We're hoping to do this more generally with the 2.1 release.  2.0 has an Intelligence (Intel) framework which is meant for being updated at runtime but we didn't get it's API in shape to really be used.  For 2.1 we're also hoping to have the Reaction framework completed which will provide you a way to hook Bro together with external devices which can block traffic or take other actions.  These two frameworks should cover this use case really well.  Unfortunately Bro's not quite there yet.

> 1. Is it possible to query a DB from BRO scripts? is there any examples?

Not yet, but we have another option that will almost certainly work well for your scenario.  BroControl (broctl) has an "update" command which can update variables defined as const at runtime.  You can use the update command if you maintain your list of subnets in a variable like this:

const block_these_networks = { 
	1.2.3.0/24,
	6.5.4.0/24,
} &redef;

You would be able to change the values in that script then go into BroControl and run the "check" command to make sure that your script doesn't have any syntax errors.  Then run "install", then "update".  That will update all const values in all Bro instances (in the event that you are running a cluster).

Feel free to ask again if you need more help.  We are working on making many of these jobs easier with each release.

> 2. Assuming yes, should i implement this logic at the 'new_connection' event? (I would like to drop connections from these IP's as soon as possible).


Yes, I think new_connection would work.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From mvv at sentia.nl  Fri Feb  3 05:38:09 2012
From: mvv at sentia.nl (Machiel van Veen)
Date: Fri, 3 Feb 2012 14:38:09 +0100
Subject: [Bro] Bro 2.0 packets dropped
Message-ID: <201202031438.09356.mvv@sentia.nl>

Hello,

I am getting a lot of messages reporting dropped packages. This is on CentOS 
6.2 running 3 workers on a quad core machine with 8 gb of ram, sas disks. 

The link I monitor has low traffic, 0.2 k packets p/s avg, max 2.0 k peaks, 
the server load is very low. I would expect no packets to be dropped. 

Besides tuning the receive buffer and queue length is there anything else I 
can do about this?

worker-1: 1328274953.996680 recvd=129059158 dropped=114860 link=129174018
worker-2: 1328274954.197859 recvd=129059218 dropped=115120 link=129174338
worker-3: 1328274954.397642 recvd=129052866 dropped=122170 link=129175036

Thank you, Machiel.


From seth at icir.org  Fri Feb  3 05:52:11 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 3 Feb 2012 08:52:11 -0500
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <201202031438.09356.mvv@sentia.nl>
References: <201202031438.09356.mvv@sentia.nl>
Message-ID: <2033B469-93B3-4F23-B428-49DB256D0E25@icir.org>


On Feb 3, 2012, at 8:38 AM, Machiel van Veen wrote:

> Besides tuning the receive buffer and queue length is there anything else I 
> can do about this?
> 
> worker-1: 1328274953.996680 recvd=129059158 dropped=114860 link=129174018
> worker-2: 1328274954.197859 recvd=129059218 dropped=115120 link=129174338
> worker-3: 1328274954.397642 recvd=129052866 dropped=122170 link=129175036


Are you monitoring 3 separate links on three interfaces?  I'm a little suspicious that you may be monitoring the same traffic three separate times.  You will need to load balance the traffic across those three workers if it's a single interface (I'm working on automating this now).

Could you add a line to load the misc/capture-loss script to your local.bro?
@load misc/capture-loss

After you do that, make sure you do "check", "install", "restart" in broctl.  The capture-loss script will give you another measure of packet loss that is not based on information being received from the NIC.  Oh, that brings up another question.  What NICs are you using?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Fri Feb  3 06:03:53 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 3 Feb 2012 09:03:53 -0500
Subject: [Bro] Errors in  Connection summary
In-Reply-To: <CAPtFaQtwDDPG17gJOZxYp1SgEPS17GM+-O+kSdFCFJkVUr0keA@mail.gmail.com>
References: <CAPtFaQtwDDPG17gJOZxYp1SgEPS17GM+-O+kSdFCFJkVUr0keA@mail.gmail.com>
Message-ID: <78616118-612E-47BC-A2FB-920EF01B674B@icir.org>


On Jan 28, 2012, at 2:00 PM, Larry Gadallah wrote:

> (I apologize for incorrectly posting this earlier on the bro-dev list
> which seems to be intended only for the Bro-Tracker bot to post to.)

Heh, it's actually not.  That list is for development discussion but we use the tracker heavily enough that it can certainly get overwhelmed with tracker messages if you aren't filtering those to a separate mail box. :)

> Incidentally, I get the same kind of linker errors at bro startup that
> were seen on OpenBSD with bro a couple of years ago:

We're going to begin doing continuous integration (automated building and testing after every commit) of the Bro source code soon and the plan currently is to also do these build/test cycles on a range of operating systems.  I think that OpenBSD is going to be one of the ones we test so hopefully our OpenBSD support will be regularly tested (which it currently isn't) so we can prevent regressions from popping up.

> Is this a case of some kind of incompatibility between the libc and
> libbind resolver functions?


I have no clue, Jon Siwek may know the answer though.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From mvv at sentia.nl  Fri Feb  3 07:18:29 2012
From: mvv at sentia.nl (Machiel van Veen)
Date: Fri, 3 Feb 2012 16:18:29 +0100
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <2033B469-93B3-4F23-B428-49DB256D0E25@icir.org>
References: <201202031438.09356.mvv@sentia.nl>
	<2033B469-93B3-4F23-B428-49DB256D0E25@icir.org>
Message-ID: <201202031618.29452.mvv@sentia.nl>

On Friday 03 February 2012 14:52:11 Seth Hall wrote:
> On Feb 3, 2012, at 8:38 AM, Machiel van Veen wrote:
> > Besides tuning the receive buffer and queue length is there anything else
> > I can do about this?
> >
> > worker-1: 1328274953.996680 recvd=129059158 dropped=114860 link=129174018
> > worker-2: 1328274954.197859 recvd=129059218 dropped=115120 link=129174338
> > worker-3: 1328274954.397642 recvd=129052866 dropped=122170 link=129175036
> 
> Are you monitoring 3 separate links on three interfaces?  I'm a little
>  suspicious that you may be monitoring the same traffic three separate
>  times.  You will need to load balance the traffic across those three
>  workers if it's a single interface (I'm working on automating this now).

It is one interface, there might be a problem load balancing. I've switched to 
a standalone setup for now.

"listening on eth1, capture length 8192 bytes"

"bro: 1328281729.277621 recvd=3553337 dropped=4503 link=3557842"

The packetloss is still there though.

> 
> Could you add a line to load the misc/capture-loss script to your
>  local.bro? @load misc/capture-loss
> 
> After you do that, make sure you do "check", "install", "restart" in
>  broctl.  The capture-loss script will give you another measure of packet
>  loss that is not based on information being received from the NIC.  

>From the alarm summary:

"2012-02-03-15:39:46 CaptureLoss::Too_Much_Loss
The capture loss script detected an estimated loss rate above 27.282%"

> Oh, that brings up another question.  What NICs are you using?

Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz
driver: bnx2
version: 2.1.11
firmware-version: bc 4.6.0 ipms 1.6.0

> 
>   .Seth
> 

Thanks again, Machiel.



From jsiwek at illinois.edu  Fri Feb  3 08:53:48 2012
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Fri, 3 Feb 2012 16:53:48 +0000
Subject: [Bro] Errors in  Connection summary
In-Reply-To: <78616118-612E-47BC-A2FB-920EF01B674B@icir.org>
References: <CAPtFaQtwDDPG17gJOZxYp1SgEPS17GM+-O+kSdFCFJkVUr0keA@mail.gmail.com>
	<78616118-612E-47BC-A2FB-920EF01B674B@icir.org>
Message-ID: <01475743-13C2-4EBD-950C-F9FFCED78992@illinois.edu>


>> Incidentally, I get the same kind of linker errors at bro startup that
>> were seen on OpenBSD with bro a couple of years ago:
> 
>  hopefully our OpenBSD support will be regularly tested (which it currently isn't) so we can prevent regressions from popping up.

I saw some discussion in the mailing list archive about those linker warnings, too, and I think it maybe not be a regression, but the original warnings were never resolved.

>> Is this a case of some kind of incompatibility between the libc and
>> libbind resolver functions?

That's my guess is that something is out of sync between those two -- compiling and running a simple test program that links against libbind will yield the same warnings, so it might be an issue that the OpenBSD ports mailing list or libbind package maintainer would better help.

+Jon


From jones at tacc.utexas.edu  Sun Feb  5 10:53:49 2012
From: jones at tacc.utexas.edu (William Jones)
Date: Sun, 5 Feb 2012 18:53:49 +0000
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <201202031438.09356.mvv@sentia.nl>
References: <201202031438.09356.mvv@sentia.nl>
Message-ID: <D0BD4A5C03717E40AC104854AF37B8C2107EDB34@EXMBX01.austin.utexas.edu>

Are you using pf_ring?

-----Original Message-----
From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf Of Machiel van Veen
Sent: Friday, February 03, 2012 7:38 AM
To: bro at bro-ids.org
Subject: [Bro] Bro 2.0 packets dropped

Hello,

I am getting a lot of messages reporting dropped packages. This is on CentOS 
6.2 running 3 workers on a quad core machine with 8 gb of ram, sas disks. 

The link I monitor has low traffic, 0.2 k packets p/s avg, max 2.0 k peaks, 
the server load is very low. I would expect no packets to be dropped. 

Besides tuning the receive buffer and queue length is there anything else I 
can do about this?

worker-1: 1328274953.996680 recvd=129059158 dropped=114860 link=129174018
worker-2: 1328274954.197859 recvd=129059218 dropped=115120 link=129174338
worker-3: 1328274954.397642 recvd=129052866 dropped=122170 link=129175036

Thank you, Machiel.
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From jones at tacc.utexas.edu  Sun Feb  5 12:03:20 2012
From: jones at tacc.utexas.edu (William Jones)
Date: Sun, 5 Feb 2012 20:03:20 +0000
Subject: [Bro] What is the offical number of workess per node thet bro
	supports
In-Reply-To: <D0BD4A5C03717E40AC104854AF37B8C2107EDB34@EXMBX01.austin.utexas.edu>
References: <201202031438.09356.mvv@sentia.nl>
	<D0BD4A5C03717E40AC104854AF37B8C2107EDB34@EXMBX01.austin.utexas.edu>
Message-ID: <D0BD4A5C03717E40AC104854AF37B8C2107EDB85@EXMBX01.austin.utexas.edu>

It looks like its 10.   





From jones at tacc.utexas.edu  Sun Feb  5 13:16:00 2012
From: jones at tacc.utexas.edu (William Jones)
Date: Sun, 5 Feb 2012 21:16:00 +0000
Subject: [Bro] Hass any one noticed a increase in scan from chine
Message-ID: <D0BD4A5C03717E40AC104854AF37B8C2107EDC1B@EXMBX01.austin.utexas.edu>

There scanners seem tobe  getting more aggressive scanning 1000's of host on a network in a few seconds.   This was causing my bro senders to drop packets.    I "fixed" the problem by pouting  filter in that ignored packet to a select few ports that I block at the barder router,  likt 1433, 445 ...

Bill Jones


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120205/d7ef67e6/attachment.html 

From seth at icir.org  Mon Feb  6 10:55:26 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 6 Feb 2012 13:55:26 -0500
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <201202031618.29452.mvv@sentia.nl>
References: <201202031438.09356.mvv@sentia.nl>
	<2033B469-93B3-4F23-B428-49DB256D0E25@icir.org>
	<201202031618.29452.mvv@sentia.nl>
Message-ID: <F1AB8BEC-7AD9-4D64-A652-B72CA897332C@icir.org>

On Feb 3, 2012, at 10:18 AM, Machiel van Veen wrote:

> It is one interface, there might be a problem load balancing. I've switched to 
> a standalone setup for now.

If you aren't taking any steps to load balance the traffic then it definitely isn't working.  We don't have automated load balanced configuration available in BroControl yet. :)

Today, I did just write a script that automates a BPF based load balancing technique on clusters which will be getting merged in along with the rest of the automated load balancing code soon.

> "bro: 1328281729.277621 recvd=3553337 dropped=4503 link=3557842"
> "2012-02-03-15:39:46 CaptureLoss::Too_Much_Loss

> The capture loss script detected an estimated loss rate above 27.282%"

Are sniffing from a tap or a SPAN port?  I'm a little suspicious because the first line indicates that the NIC was showing 0.1% packet loss, but the second line indicates much more loss.  The misc/capture-loss.bro script can detect loss due to reasons beyond the monitoring host (like an overloaded SPAN port) so I'm just trying to figure out where there is a such a huge disparity between the two measurements.

Oh, one other thought.  Are you disabling all of the offload features of your NIC?  Here's an article about it:
	1. http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html

Is the MTU on your NIC larger than 8192 (Bro 2.0's default snaplen).  If there are packets larger than that they won't be seen by default.

>> Oh, that brings up another question.  What NICs are you using?
> 
> Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz
> driver: bnx2
> version: 2.1.11
> firmware-version: bc 4.6.0 ipms 1.6.0


I usually recommend not using Broadcom nics for monitoring.  At times with various broadcom nics I've run into weird problems so I tend to avoid them.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Mon Feb  6 10:56:44 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 6 Feb 2012 13:56:44 -0500
Subject: [Bro] What is the offical number of workess per node thet bro
	supports
In-Reply-To: <D0BD4A5C03717E40AC104854AF37B8C2107EDB85@EXMBX01.austin.utexas.edu>
References: <201202031438.09356.mvv@sentia.nl>
	<D0BD4A5C03717E40AC104854AF37B8C2107EDB34@EXMBX01.austin.utexas.edu>
	<D0BD4A5C03717E40AC104854AF37B8C2107EDB85@EXMBX01.austin.utexas.edu>
Message-ID: <C9161D77-D22D-4938-B66E-7FA6582BF8F8@icir.org>


On Feb 5, 2012, at 3:03 PM, William Jones wrote:

> It looks like its 10.   


I'm not sure what you're asking.  There is no hard coded number of workers that can be defined for a cluster.  We have deployments of anywhere from 2 workers up to over a hundred.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jones at tacc.utexas.edu  Mon Feb  6 11:30:22 2012
From: jones at tacc.utexas.edu (William Jones)
Date: Mon, 6 Feb 2012 19:30:22 +0000
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <F1AB8BEC-7AD9-4D64-A652-B72CA897332C@icir.org>
References: <201202031438.09356.mvv@sentia.nl>
	<2033B469-93B3-4F23-B428-49DB256D0E25@icir.org>
	<201202031618.29452.mvv@sentia.nl>
	<F1AB8BEC-7AD9-4D64-A652-B72CA897332C@icir.org>
Message-ID: <D0BD4A5C03717E40AC104854AF37B8C2107EF252@EXMBX01.austin.utexas.edu>

Here my problem.   I have a single server and a defined 10 works on it to divided up the load.  Here the output of "broctl status"

root at homey manager]# broctl status
Name       Type       Host       Status        Pid    Peers  Started              
manager    manager    homey.tacc.utexas.edu running       6202   11     05 Feb 15:22:15  
proxy-1    proxy      homey.tacc.utexas.edu running       6237   11     05 Feb 15:22:17  
worker-1   worker     mojo1.tacc.utexas.edu running       18356  2      05 Feb 15:26:41  
worker-10  worker     mojo1.tacc.utexas.edu running       18350  2      05 Feb 15:26:41  
worker-2   worker     mojo1.tacc.utexas.edu running       18348  2      05 Feb 15:26:41  
worker-3   worker     mojo1.tacc.utexas.edu running       18349  2      05 Feb 15:26:41  
worker-4   worker     mojo1.tacc.utexas.edu running       18357  2      05 Feb 15:26:41  
worker-5   worker     mojo1.tacc.utexas.edu running       18352  2      05 Feb 15:26:41  
worker-6   worker     mojo1.tacc.utexas.edu running       18353  2      05 Feb 15:26:41  
worker-7   worker     mojo1.tacc.utexas.edu running       18354  2      05 Feb 15:26:41  
worker-8   worker     mojo1.tacc.utexas.edu running       18355  2      05 Feb 15:26:41  
worker-9   worker     mojo1.tacc.utexas.edu running       18351  2      05 Feb 15:26:41  


I now add woker-11 to to the configuration and "bro status" returns:


BroControl] > status
Name       Type       Host       Status        Pid    Peers  Started              
manager    manager    homey.tacc.utexas.edu running       29316  12     06 Feb 13:16:59  
proxy-1    proxy      homey.tacc.utexas.edu running       29351  12     06 Feb 13:17:01  
worker-1   worker     mojo1.tacc.utexas.edu running       25026  2      06 Feb 13:17:06  
worker-10  worker     mojo1.tacc.utexas.edu running       25028  2      06 Feb 13:17:06  
worker-11  worker     mojo1.tacc.utexas.edu running       25033  2      06 Feb 13:17:06  
worker-2   worker     mojo1.tacc.utexas.edu running       25032  2      06 Feb 13:17:06  
worker-3   worker     mojo1.tacc.utexas.edu running       25025  2      06 Feb 13:17:06  
worker-4   worker     mojo1.tacc.utexas.edu running       25031  2      06 Feb 13:17:06  
worker-5   worker     mojo1.tacc.utexas.edu running       25029  2      06 Feb 13:17:06  
worker-6   worker     mojo1.tacc.utexas.edu running       25027  2      06 Feb 13:17:06  
worker-7   worker     mojo1.tacc.utexas.edu running       25034  2      06 Feb 13:17:06  
worker-8   worker     mojo1.tacc.utexas.edu running       25030  2      06 Feb 13:17:06  
worker-9   worker     mojo1.tacc.utexas.edu running       25036  ???    06 Feb 13:17:06  


Notice the ???.   It an indication that something is not working  correct;y the bro communication library. 





-----Original Message-----
From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf Of Seth Hall
Sent: Monday, February 06, 2012 12:55 PM
To: Machiel van Veen
Cc: bro at bro-ids.org
Subject: Re: [Bro] Bro 2.0 packets dropped

On Feb 3, 2012, at 10:18 AM, Machiel van Veen wrote:

> It is one interface, there might be a problem load balancing. I've switched to 
> a standalone setup for now.

If you aren't taking any steps to load balance the traffic then it definitely isn't working.  We don't have automated load balanced configuration available in BroControl yet. :)

Today, I did just write a script that automates a BPF based load balancing technique on clusters which will be getting merged in along with the rest of the automated load balancing code soon.

> "bro: 1328281729.277621 recvd=3553337 dropped=4503 link=3557842"
> "2012-02-03-15:39:46 CaptureLoss::Too_Much_Loss

> The capture loss script detected an estimated loss rate above 27.282%"

Are sniffing from a tap or a SPAN port?  I'm a little suspicious because the first line indicates that the NIC was showing 0.1% packet loss, but the second line indicates much more loss.  The misc/capture-loss.bro script can detect loss due to reasons beyond the monitoring host (like an overloaded SPAN port) so I'm just trying to figure out where there is a such a huge disparity between the two measurements.

Oh, one other thought.  Are you disabling all of the offload features of your NIC?  Here's an article about it:
	1. http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html

Is the MTU on your NIC larger than 8192 (Bro 2.0's default snaplen).  If there are packets larger than that they won't be seen by default.

>> Oh, that brings up another question.  What NICs are you using?
> 
> Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz
> driver: bnx2
> version: 2.1.11
> firmware-version: bc 4.6.0 ipms 1.6.0


I usually recommend not using Broadcom nics for monitoring.  At times with various broadcom nics I've run into weird problems so I tend to avoid them.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/


_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at icir.org  Mon Feb  6 11:46:02 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 6 Feb 2012 14:46:02 -0500
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <D0BD4A5C03717E40AC104854AF37B8C2107EF252@EXMBX01.austin.utexas.edu>
References: <201202031438.09356.mvv@sentia.nl>
	<2033B469-93B3-4F23-B428-49DB256D0E25@icir.org>
	<201202031618.29452.mvv@sentia.nl>
	<F1AB8BEC-7AD9-4D64-A652-B72CA897332C@icir.org>
	<D0BD4A5C03717E40AC104854AF37B8C2107EF252@EXMBX01.austin.utexas.edu>
Message-ID: <C8D102D3-DE45-4E96-ACB7-27234B0C7DF2@icir.org>


On Feb 6, 2012, at 2:30 PM, William Jones wrote:

> Notice the ???.   It an indication that something is not working  correct;y the bro communication library. 


If you run "ps.bro" in broctl, what do you get?  I'm suspicious that the old #9 process didn't die and it's still holding the communication port open which would result in that error.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From robin at icir.org  Tue Feb  7 01:13:35 2012
From: robin at icir.org (Robin Sommer)
Date: Tue, 7 Feb 2012 01:13:35 -0800
Subject: [Bro] Internship Opening
Message-ID: <20120207091335.GM89621@icir.org>

The Bro Project has an opening for a three month summer internship. If
you are interesteded in helping us improve Bro and develop new
functionality, please apply!

    http://www.bro-ids.org/intern.html

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From daveangelo831 at gmail.com  Tue Feb  7 08:25:51 2012
From: daveangelo831 at gmail.com (Dave Angelo)
Date: Tue, 7 Feb 2012 18:25:51 +0200
Subject: [Bro] Connection UID repeat itself
Message-ID: <CAA=Y8dggwKSzGWyH0gDLdSyib6gWLbND_0fzqCtJC8NHDE1gMg@mail.gmail.com>

Hey,

I registered on the SMTP::mime_end_entity event to inspect some SMTP
traffic.

I am printing the '*c$smtp$uid*' to a log file and I noticed that once in a
while the UID repeats itself.

I started debugging this and I noticed that a certain email X that was sent
5 times to 5 different recipient had the *same* UID.

I was looking at *calculate_unique_id() *that generates this UID and I
cannot understand why this happens.

Why all those specific emails have the same UID?

Thanks,
Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120207/52150bc3/attachment.html 

From seth at icir.org  Tue Feb  7 11:45:36 2012
From: seth at icir.org (Seth Hall)
Date: Tue, 7 Feb 2012 14:45:36 -0500
Subject: [Bro] Connection UID repeat itself
In-Reply-To: <CAA=Y8dggwKSzGWyH0gDLdSyib6gWLbND_0fzqCtJC8NHDE1gMg@mail.gmail.com>
References: <CAA=Y8dggwKSzGWyH0gDLdSyib6gWLbND_0fzqCtJC8NHDE1gMg@mail.gmail.com>
Message-ID: <C0B5C424-B780-4365-83C5-1E7A11D7949B@icir.org>


On Feb 7, 2012, at 11:25 AM, Dave Angelo wrote:

> I registered on the SMTP::mime_end_entity event to inspect some SMTP traffic.
> 
> I am printing the 'c$smtp$uid' to a log file and I noticed that once in a while the UID repeats itself.

Hopefully you're logging more than that since the uid value by itself it pretty useless. :)

> Why all those specific emails have the same UID?

That would mean that all of those emails were sent over the same TCP connection.  UID values are a fairly arbitrary representation for a single connection.  For instance, if two hosts establish a connection then shut it down and then establish a new connection using the exact same ports there would be a new UID value.  It's primarily used as a very easy method of joining various log files together without having to match on the 2 IP addresses and 2 ports with the added benefit of distinguishing between connections reusing ports.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From stephane.chazelas at gmail.com  Wed Feb  8 09:54:21 2012
From: stephane.chazelas at gmail.com (Stephane Chazelas)
Date: Wed, 8 Feb 2012 17:54:21 +0000
Subject: [Bro] Adding SSL certs to Bro 2.0
In-Reply-To: <3BCA38E4-AA36-4D3D-8DE0-E826D44C2B64@icir.org>
References: <4EC3F2C7.50206@vanderbilt.edu>
	<3BCA38E4-AA36-4D3D-8DE0-E826D44C2B64@icir.org>
Message-ID: <20120208175412.GD12752@yahoo.fr>

2011-11-16 13:16:40 -0500, Seth Hall:
> On Nov 16, 2011, at 12:28 PM, Mathew Binkley wrote:
[...]
> > I see share/bro/base/protocols/ssl/mozilla-ca-list has a bundle of root
> > CA certs.   Is there a way to add our own to that or to a separate file?
> >  How is that file generated?   Thanks.
> 
> 
> We have a exercise from the workshop that specifically addresses this situation.  We will be posting the workshop material really soon too.
> 
> Ultimately, you need to take a DER formatted version of your root public key and convert it to Bro's hex string representation and add it to the SSL::root_certs table.  Like this....
> 
> redef SSL::root_certs += {
> 	["your root certificates subject"] = "\x30\x82\x03\x75\x30\x82<snip a lot more of this>";
> };
[...]

In case it may be of some help to anyone, here is a script to
convert a PEM CA cert bundle such as
/etc/ssl/certs/ca-certificates.crt as found on debian based
system to bro's format:

<BEGIN>
#! /usr/bin/perl
use Encode;
use Crypt::OpenSSL::X509;

while (<>) {
  if (/BEGIN /) {
    $p="";
    $inside=1;
  }
  $p .= $_ if $inside;
  if (/END /) {
    $c = Crypt::OpenSSL::X509->new_from_string($p);
    @s = map { $_->as_string } reverse @{$c->subject_name->entries};
    s/[\\,]/\\$&/g for @s;
    $s = join ",", @s;
    $s = encode("UTF-8", $s);
    $s =~ s/[\200-\377]/sprintf("\\%X",ord$&)/ge;
    $x = join "", map { "\\x" . uc$_ } unpack("(H2)*", $c->as_string(Crypt::OpenSSL::X509::FORMAT_ASN1));
    print "\t[\"$s\"] = \"$x\",\n"; $inside = 0;
  }
}
<END>

(this gives the same output as found in the mozilla-ca.bro file)

Then, I have a /etc/ca-certificates/update.d/bro-cacerts to
update Bro's root_certs everytime the system CA certs are
updated:

<BEGIN>
#! /bin/sh -
BRO_CERTSTORE=/usr/local/share/bro/site/certs.bro

if [ -f /etc/default/cacerts ]; then
    . /etc/default/cacerts
fi

echo
if [ "$cacerts_updates" != yes ] || [ "$CACERT_UPDATES" = disabled ]; then
    echo "updates of cacerts keystore disabled."
    exit 0
fi

printf '%s\n' "Updating Bro IDS CA Cert store $BRO_CERTSTORE"

set -C
{
  echo "redef SSL::root_certs += {" &&
    /usr/local/bin/crt-to-bro < /etc/ssl/certs/ca-certificates.crt &&
    echo "};"
} > "$BRO_CERTSTORE.new" || exit
mv -f "$BRO_CERTSTORE.new" "$BRO_CERTSTORE"  || exit

echo "Restarting bro"
broctl check &&
  broctl install &&
  broctl restart
<END>

This way, Bro uses the same rootca as the system's to verify
certificates.

HTH
Stephane



From seth at icir.org  Wed Feb  8 11:57:39 2012
From: seth at icir.org (Seth Hall)
Date: Wed, 8 Feb 2012 14:57:39 -0500
Subject: [Bro] Adding SSL certs to Bro 2.0
In-Reply-To: <20120208175412.GD12752@yahoo.fr>
References: <4EC3F2C7.50206@vanderbilt.edu>
	<3BCA38E4-AA36-4D3D-8DE0-E826D44C2B64@icir.org>
	<20120208175412.GD12752@yahoo.fr>
Message-ID: <E38FB79A-9F2C-4DA1-998B-0E5222FBF0B7@icir.org>


On Feb 8, 2012, at 12:54 PM, Stephane Chazelas wrote:

> In case it may be of some help to anyone, here is a script to
> convert a PEM CA cert bundle such as
> /etc/ssl/certs/ca-certificates.crt as found on debian based
> system to bro's format:


Cool, thanks.  We have a script in bro-aux that generates the CA list Bro script directly from the Mozilla repository too.  If you have a copy of our source tree, the script is here:
	aux/bro-aux/devel-tools/gen-mozilla-ca-list.rb

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From mvv at sentia.nl  Wed Feb  8 23:41:25 2012
From: mvv at sentia.nl (Machiel van Veen)
Date: Thu, 9 Feb 2012 08:41:25 +0100
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <F1AB8BEC-7AD9-4D64-A652-B72CA897332C@icir.org>
References: <201202031438.09356.mvv@sentia.nl>
	<201202031618.29452.mvv@sentia.nl>
	<F1AB8BEC-7AD9-4D64-A652-B72CA897332C@icir.org>
Message-ID: <201202090841.25202.mvv@sentia.nl>

On Monday 06 February 2012 19:55:26 Seth Hall wrote:
> >> Oh, that brings up another question.  What NICs are you using?
> >
> > Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz
> > driver: bnx2
> > version: 2.1.11
> > firmware-version: bc 4.6.0 ipms 1.6.0
> 
> I usually recommend not using Broadcom nics for monitoring.  At times with
>  various broadcom nics I've run into weird problems so I tend to avoid
>  them.
> 
>   .Seth

Sorry for replying so late, I was unexpectedly off-line for some time... I 
tried all the suggestions yesterday and found the issue to be the Broadcom 
NIC. The tap is on a Intel NIC now and dropped packages are at an acceptable 
level.

bro: 1328772549.657147 recvd=22987443 dropped=9811 link=22997255

There have been no alerts regarding packet loss either. Thanks for you support 
on this issue, back to clustering for me ;) 

-- 

Machiel.


From seth at icir.org  Thu Feb  9 05:55:07 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 9 Feb 2012 08:55:07 -0500
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <201202090841.25202.mvv@sentia.nl>
References: <201202031438.09356.mvv@sentia.nl>
	<201202031618.29452.mvv@sentia.nl>
	<F1AB8BEC-7AD9-4D64-A652-B72CA897332C@icir.org>
	<201202090841.25202.mvv@sentia.nl>
Message-ID: <6117B7B8-69CD-4BA5-8BAD-6EF9F2D3FF88@icir.org>


On Feb 9, 2012, at 2:41 AM, Machiel van Veen wrote:

> There have been no alerts regarding packet loss either. Thanks for you support 
> on this issue, back to clustering for me ;) 


Great to hear!  Broadcom nics have the weirdest problems.

Are you doing network based clustering (multiple physical hosts)?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From sri at basam.org  Thu Feb  9 07:23:02 2012
From: sri at basam.org (sridhar basam)
Date: Thu, 9 Feb 2012 10:23:02 -0500
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <6117B7B8-69CD-4BA5-8BAD-6EF9F2D3FF88@icir.org>
References: <201202031438.09356.mvv@sentia.nl>
	<201202031618.29452.mvv@sentia.nl>
	<F1AB8BEC-7AD9-4D64-A652-B72CA897332C@icir.org>
	<201202090841.25202.mvv@sentia.nl>
	<6117B7B8-69CD-4BA5-8BAD-6EF9F2D3FF88@icir.org>
Message-ID: <CAGnVnBkEtcPjftS-cf_eQHQmHK-OLJ++NRVjH2k_fdTruMqK6g@mail.gmail.com>

On Thu, Feb 9, 2012 at 8:55 AM, Seth Hall <seth at icir.org> wrote:
>
> On Feb 9, 2012, at 2:41 AM, Machiel van Veen wrote:
>
>> There have been no alerts regarding packet loss either. Thanks for you support
>> on this issue, back to clustering for me ;)
>
>
> Great to hear! ?Broadcom nics have the weirdest problems.
>
> Are you doing network based clustering (multiple physical hosts)?

Not exactly related to Machiel problem but if people are stuck with
Broadcom adapters. You can try updating the firmware on the cards. The
version above 4.x is extremely old.
http://www.broadcom.com/support/ethernet_nic/netxtremeii.php for
getting software for your netextreme-ii adapters.

You can also increase the ring buffer from the default 255 to 1024 or
more so that your receive ring isn't overflowing. "/sbin/ethtool -G
eth0 rx 1024"
The same would apply for intel adapters too, if you notice your
interface card dropping packets.

      sridhar



From stephane.chazelas at gmail.com  Thu Feb  9 06:43:37 2012
From: stephane.chazelas at gmail.com (Stephane Chazelas)
Date: Thu, 9 Feb 2012 14:43:37 +0000
Subject: [Bro] high cpu usage and strange select(2) behavior
Message-ID: <20120209144337.GB6571@yahoo.fr>

Hiya,

here with a standalone bro-2.0, I'm seeing 2 bro processes
using about 30% of the time of a cpu each regarless of whether
there's traffic or not to watch.

Doing a strace on them I see, for one process only select system
calls in a loop with null or ridiculously low timeouts:

select(12, [0 11], [], [], {0, 10})     = 0 (Timeout)
select(12, [11], NULL, NULL, {0, 0})    = 0 (Timeout)
select(12, [11], NULL, NULL, {0, 0})    = 0 (Timeout)
select(12, [0 11], [], [], {0, 10})     = 0 (Timeout)
select(12, [11], NULL, NULL, {0, 0})    = 0 (Timeout)
select(12, [11], NULL, NULL, {0, 0})    = 0 (Timeout)
select(12, [0 11], [], [], {0, 10})     = 0 (Timeout)
select(12, [11], NULL, NULL, {0, 0})    = 0 (Timeout)

0 being tcp *:47760 (LISTEN), and 11 a unix domain socket, probably
a socket pair to communicate with the other bro process. Why
those 3 selects in an infinite loop, why a timeout? Can't the
select just sit on [0 11] if it's the only inputs it gets? Here
it seems it is just wasting 30% of the time of the CPU.

The other process behavior is even weirder:

select(0, NULL, NULL, NULL, {0, 20})    = 0 (Timeout)
select(11, [4 7 10], [0], [0], {0, 0})  = 1 (out [0], left {0, 0})
kill(15287, SIG_0)                      = 0
read(10, 0xb6ebc008, 1048576)           = -1 EAGAIN (Resource temporarily unavailable)
gettimeofday({1328798101, 327305}, NULL) = 0
select(0, NULL, NULL, NULL, {0, 20})    = 0 (Timeout)
select(11, [4 7 10], [0], [0], {0, 0})  = 1 (out [0], left {0, 0})
kill(15287, SIG_0)                      = 0
read(10, 0xb6ebc008, 1048576)           = -1 EAGAIN (Resource temporarily unavailable)
gettimeofday({1328798101, 328513}, NULL) = 0
select(0, NULL, NULL, NULL, {0, 20})    = 0 (Timeout)
select(11, [4 7 10], [0], [0], {0, 0})  = 1 (out [0], left {0, 0})
kill(15287, SIG_0)                      = 0
read(10, 0xb6ebc008, 1048576)           = -1 EAGAIN (Resource temporarily unavailable)


A select with no fd to watch (why not use nanosleep?) a read()
on a non-blocking fd following a select that returns on timeout
(null timeout with nothing for that fd). And again, why not have
the select() sit waiting for input?

That doesn't seem quite right.

Is that the expected behavior, or is there any way to configure
it so that it behaves itself?

-- 
Stephane



From sri at basam.org  Thu Feb  9 09:31:29 2012
From: sri at basam.org (sridhar basam)
Date: Thu, 9 Feb 2012 12:31:29 -0500
Subject: [Bro] high cpu usage and strange select(2) behavior
In-Reply-To: <20120209144337.GB6571@yahoo.fr>
References: <20120209144337.GB6571@yahoo.fr>
Message-ID: <CAGnVnBn+e95JYkaeG2f9zBOhT6wT3p8tEmaZvwcA6mASwgGGDw@mail.gmail.com>

On Thu, Feb 9, 2012 at 9:43 AM, Stephane Chazelas
<stephane.chazelas at gmail.com> wrote:
> Hiya,
>
> here with a standalone bro-2.0, I'm seeing 2 bro processes
> using about 30% of the time of a cpu each regarless of whether
> there's traffic or not to watch.
>
> Doing a strace on them I see, for one process only select system
> calls in a loop with null or ridiculously low timeouts:
>
> select(12, [0 11], [], [], {0, 10}) ? ? = 0 (Timeout)
> select(12, [11], NULL, NULL, {0, 0}) ? ?= 0 (Timeout)
> select(12, [11], NULL, NULL, {0, 0}) ? ?= 0 (Timeout)
> select(12, [0 11], [], [], {0, 10}) ? ? = 0 (Timeout)
> select(12, [11], NULL, NULL, {0, 0}) ? ?= 0 (Timeout)
> select(12, [11], NULL, NULL, {0, 0}) ? ?= 0 (Timeout)
> select(12, [0 11], [], [], {0, 10}) ? ? = 0 (Timeout)
> select(12, [11], NULL, NULL, {0, 0}) ? ?= 0 (Timeout)
>
> 0 being tcp *:47760 (LISTEN), and 11 a unix domain socket, probably
> a socket pair to communicate with the other bro process. Why
> those 3 selects in an infinite loop, why a timeout? Can't the
> select just sit on [0 11] if it's the only inputs it gets? Here
> it seems it is just wasting 30% of the time of the CPU.
>
> The other process behavior is even weirder:
>
> select(0, NULL, NULL, NULL, {0, 20}) ? ?= 0 (Timeout)
> select(11, [4 7 10], [0], [0], {0, 0}) ?= 1 (out [0], left {0, 0})
> kill(15287, SIG_0) ? ? ? ? ? ? ? ? ? ? ?= 0
> read(10, 0xb6ebc008, 1048576) ? ? ? ? ? = -1 EAGAIN (Resource temporarily unavailable)
> gettimeofday({1328798101, 327305}, NULL) = 0
> select(0, NULL, NULL, NULL, {0, 20}) ? ?= 0 (Timeout)
> select(11, [4 7 10], [0], [0], {0, 0}) ?= 1 (out [0], left {0, 0})
> kill(15287, SIG_0) ? ? ? ? ? ? ? ? ? ? ?= 0
> read(10, 0xb6ebc008, 1048576) ? ? ? ? ? = -1 EAGAIN (Resource temporarily unavailable)
> gettimeofday({1328798101, 328513}, NULL) = 0
> select(0, NULL, NULL, NULL, {0, 20}) ? ?= 0 (Timeout)
> select(11, [4 7 10], [0], [0], {0, 0}) ?= 1 (out [0], left {0, 0})
> kill(15287, SIG_0) ? ? ? ? ? ? ? ? ? ? ?= 0
> read(10, 0xb6ebc008, 1048576) ? ? ? ? ? = -1 EAGAIN (Resource temporarily unavailable)
>
>
> A select with no fd to watch (why not use nanosleep?) a read()
> on a non-blocking fd following a select that returns on timeout
> (null timeout with nothing for that fd). And again, why not have
> the select() sit waiting for input?
>
> That doesn't seem quite right.
>
> Is that the expected behavior, or is there any way to configure
> it so that it behaves itself?

IMO looping through select isn't an issue. On an idle/lightly loaded
system, select is going to be the dominate usage of cpu but when you
start seeing increased load, the rest of the system begins to pick up
in usage with select taking a smaller and smaller portion of the CPU.
Is it really an issue using 30% CPU on an otherwise idle system?

About using nanosleep vs select,  i don't really see any difference in
terms of what you want to achieve. In either case the process doing
the select or nanosleep is going to be put to sleep till the timer
expires so they both end up achieving the same thing. It isn't
uncommon for one to use select/poll in that way to sleep.

 Sridhar



From mvv at sentia.nl  Fri Feb 10 00:12:07 2012
From: mvv at sentia.nl (Machiel van Veen)
Date: Fri, 10 Feb 2012 09:12:07 +0100
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <6117B7B8-69CD-4BA5-8BAD-6EF9F2D3FF88@icir.org>
References: <201202031438.09356.mvv@sentia.nl>
	<201202090841.25202.mvv@sentia.nl>
	<6117B7B8-69CD-4BA5-8BAD-6EF9F2D3FF88@icir.org>
Message-ID: <201202100912.07901.mvv@sentia.nl>

On Thursday 09 February 2012 14:55:07 Seth Hall wrote:
> On Feb 9, 2012, at 2:41 AM, Machiel van Veen wrote:
> > There have been no alerts regarding packet loss either. Thanks for you
> > support on this issue, back to clustering for me ;)
> 
> Great to hear!  Broadcom nics have the weirdest problems.
> 
> Are you doing network based clustering (multiple physical hosts)?
> 

I'm looking to run one physical host for now and add a second with several 
workers later. However load balancing has not worked yet, the module and 
driver are loaded and pfcount shows traffic but Bro reports everything triple, 
no load balancing.

Is there any configuration apart from configuring the manager, proxy and 
workers in node.cfg done in Bro to get this working? 

As I understood there should not be, as long as you link to the pf_ring 
enabled libpcap it should work when pf_ring is active.

Machiel.


From stephane.chazelas at gmail.com  Fri Feb 10 03:24:02 2012
From: stephane.chazelas at gmail.com (Stephane Chazelas)
Date: Fri, 10 Feb 2012 11:24:02 +0000
Subject: [Bro] high cpu usage and strange select(2) behavior
In-Reply-To: <CAGnVnBn+e95JYkaeG2f9zBOhT6wT3p8tEmaZvwcA6mASwgGGDw@mail.gmail.com>
References: <20120209144337.GB6571@yahoo.fr>
	<CAGnVnBn+e95JYkaeG2f9zBOhT6wT3p8tEmaZvwcA6mASwgGGDw@mail.gmail.com>
Message-ID: <20120210112402.GB4052@yahoo.fr>

2012-02-09 12:31:29 -0500, sridhar basam:
[...]
> IMO looping through select isn't an issue. On an idle/lightly loaded
> system, select is going to be the dominate usage of cpu but when you
> start seeing increased load, the rest of the system begins to pick up
> in usage with select taking a smaller and smaller portion of the CPU.
> Is it really an issue using 30% CPU on an otherwise idle system?
[...]

Thanks Sridhar for the feedback.

That's assuming a system where only bro is running.

Here, bro is running in a VM, and I've got other VMs running
there (with CPU overcommit) and other software running on that
VM (other IDS which also do offline deep inspection in the
background), so those CPU cycles wasted by bro are not available
to those other processes and to the other VMs. (and it's 60%
CPU, not 30).

But the reason I asked was because I thought it was a
configuration problem of mine, because I found it abnormal for
bro to use that much CPU when idle, and thought that could
explain the alerts about dropped packets where the other IDSes
are fine.

> About using nanosleep vs select,  i don't really see any difference in
> terms of what you want to achieve. In either case the process doing
> the select or nanosleep is going to be put to sleep till the timer
> expires so they both end up achieving the same thing. It isn't
> uncommon for one to use select/poll in that way to sleep.
[...]

OK, I agree that's a minor point, but why not use the right tool
for the job? it makes the code easier to read and saves a few
CPU cycles.

Cheers,
Stephane


From seth at icir.org  Fri Feb 10 06:16:52 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 10 Feb 2012 09:16:52 -0500
Subject: [Bro] high cpu usage and strange select(2) behavior
In-Reply-To: <20120210112402.GB4052@yahoo.fr>
References: <20120209144337.GB6571@yahoo.fr>
	<CAGnVnBn+e95JYkaeG2f9zBOhT6wT3p8tEmaZvwcA6mASwgGGDw@mail.gmail.com>
	<20120210112402.GB4052@yahoo.fr>
Message-ID: <DAE533D0-8CD5-42AB-8EBB-26903BA33360@icir.org>


On Feb 10, 2012, at 6:24 AM, Stephane Chazelas wrote:

> But the reason I asked was because I thought it was a
> configuration problem of mine, because I found it abnormal for
> bro to use that much CPU when idle, and thought that could
> explain the alerts about dropped packets where the other IDSes
> are fine.


The high overhead is due to the Bro communication loop.  If you run in standalone mode (or just manually run a Bro process without loading frameworks/communication/listen.bro) you won't see the high cpu load.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Fri Feb 10 06:25:04 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 10 Feb 2012 09:25:04 -0500
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <201202100912.07901.mvv@sentia.nl>
References: <201202031438.09356.mvv@sentia.nl>
	<201202090841.25202.mvv@sentia.nl>
	<6117B7B8-69CD-4BA5-8BAD-6EF9F2D3FF88@icir.org>
	<201202100912.07901.mvv@sentia.nl>
Message-ID: <FBC9EFED-FE22-42E4-B0BC-FBD626394A7C@icir.org>


On Feb 10, 2012, at 3:12 AM, Machiel van Veen wrote:

> Is there any configuration apart from configuring the manager, proxy and 
> workers in node.cfg done in Bro to get this working? 


Could you send me the content of your node.cfg and broctl.cfg files?  This is fortunate timing, I've been preparing a blog post about using PF_Ring load balancing with Bro and it would be good to find out if there are any problems with it.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From mvv at sentia.nl  Fri Feb 10 06:49:47 2012
From: mvv at sentia.nl (Machiel van Veen)
Date: Fri, 10 Feb 2012 15:49:47 +0100
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <FBC9EFED-FE22-42E4-B0BC-FBD626394A7C@icir.org>
References: <201202031438.09356.mvv@sentia.nl>
	<201202100912.07901.mvv@sentia.nl>
	<FBC9EFED-FE22-42E4-B0BC-FBD626394A7C@icir.org>
Message-ID: <201202101549.47457.mvv@sentia.nl>

On Friday 10 February 2012 15:25:04 Seth Hall wrote:
> On Feb 10, 2012, at 3:12 AM, Machiel van Veen wrote:
> > Is there any configuration apart from configuring the manager, proxy and
> > workers in node.cfg done in Bro to get this working?
> 
> Could you send me the content of your node.cfg and broctl.cfg files?  This
>  is fortunate timing, I've been preparing a blog post about using PF_Ring
>  load balancing with Bro and it would be good to find out if there are any
>  problems with it.
> 
>   .Seth

I'm using the following in node.cfg:

[manager]
type=manager
host=10.0.0.42

[proxy-1]
type=proxy
host=10.0.0.42

[worker-1]
type=worker
host=10.0.0.42
interface=p1p1

[worker-2]
type=worker
host=10.0.0.42
interface=p1p1

[worker-3]
type=worker
host=10.0.0.42
interface=p1p1

And these settings in broctl.cfg:

MailTo = root at localhost
SitePolicyStandalone = local.bro
SpoolDir = /var/opt/bro/spool
LogDir = /var/opt/bro/logs
LogRotationInterval = 3600
MinDiskSpace = 5
Debug = 1

Regards, Machiel.


From seth at icir.org  Fri Feb 10 07:26:48 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 10 Feb 2012 10:26:48 -0500
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <201202101549.47457.mvv@sentia.nl>
References: <201202031438.09356.mvv@sentia.nl>
	<201202100912.07901.mvv@sentia.nl>
	<FBC9EFED-FE22-42E4-B0BC-FBD626394A7C@icir.org>
	<201202101549.47457.mvv@sentia.nl>
Message-ID: <ADEC3C4D-A815-411E-911A-44C685643214@icir.org>


On Feb 10, 2012, at 9:49 AM, Machiel van Veen wrote:

> On Friday 10 February 2012 15:25:04 Seth Hall wrote:
>> On Feb 10, 2012, at 3:12 AM, Machiel van Veen wrote:
>>> Is there any configuration apart from configuring the manager, proxy and
>>> workers in node.cfg done in Bro to get this working?
>> 
>> Could you send me the content of your node.cfg and broctl.cfg files?  This
>> is fortunate timing, I've been preparing a blog post about using PF_Ring
>> load balancing with Bro and it would be good to find out if there are any
>> problems with it.
>> 
>>  .Seth
> 
> I'm using the following in node.cfg:

That seems fine.

> And these settings in broctl.cfg:
> 
> MailTo = root at localhost
> SitePolicyStandalone = local.bro
> SpoolDir = /var/opt/bro/spool
> LogDir = /var/opt/bro/logs
> LogRotationInterval = 3600
> MinDiskSpace = 5
> Debug = 1


It looks like you are missing the setting that turns on the pf_ring clustering support.  If you built against the pf_ring libpcap wrapper it should have been put in there automatically (unless you installed over top of a previous installation?).

Add this to your broctl.cfg and do "check", "install", "restart" in broctl.
PFRingClusterId = 21

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From mvv at sentia.nl  Fri Feb 10 07:46:40 2012
From: mvv at sentia.nl (Machiel van Veen)
Date: Fri, 10 Feb 2012 16:46:40 +0100
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <ADEC3C4D-A815-411E-911A-44C685643214@icir.org>
References: <201202031438.09356.mvv@sentia.nl>
	<201202101549.47457.mvv@sentia.nl>
	<ADEC3C4D-A815-411E-911A-44C685643214@icir.org>
Message-ID: <201202101646.40289.mvv@sentia.nl>

On Friday 10 February 2012 16:26:48 Seth Hall wrote:
> It looks like you are missing the setting that turns on the pf_ring
>  clustering support.  If you built against the pf_ring libpcap wrapper it
>  should have been put in there automatically (unless you installed over top
>  of a previous installation?).
> 
> Add this to your broctl.cfg and do "check", "install", "restart" in broctl.
> PFRingClusterId = 21
> 
>   .Seth

I've added the option, there is no difference. I did notice in the debug logs 
before that this option has been set by default. At startup i see the 
following for all workers, proxy and manager:

"PCAP_PF_RING_USE_CLUSTER_PER_FLOW=1 PCAP_PF_RING_CLUSTER_ID=21"

The bro binary does seem to use the correct lib:

$ ldd /opt/bro/bin/bro | grep pcap
libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x00007fae5cad2000)

I'll go ahead and do this again on monday, perhaps I did make a mistake during 
the build process.

Thanks, Machiel.


From stephane.chazelas at gmail.com  Fri Feb 10 08:05:30 2012
From: stephane.chazelas at gmail.com (Stephane Chazelas)
Date: Fri, 10 Feb 2012 16:05:30 +0000
Subject: [Bro] high cpu usage and strange select(2) behavior
In-Reply-To: <DAE533D0-8CD5-42AB-8EBB-26903BA33360@icir.org>
References: <20120209144337.GB6571@yahoo.fr>
	<CAGnVnBn+e95JYkaeG2f9zBOhT6wT3p8tEmaZvwcA6mASwgGGDw@mail.gmail.com>
	<20120210112402.GB4052@yahoo.fr>
	<DAE533D0-8CD5-42AB-8EBB-26903BA33360@icir.org>
Message-ID: <20120210160530.GG4052@yahoo.fr>

2012-02-10 09:16:52 -0500, Seth Hall:
> 
> On Feb 10, 2012, at 6:24 AM, Stephane Chazelas wrote:
> 
> > But the reason I asked was because I thought it was a
> > configuration problem of mine, because I found it abnormal for
> > bro to use that much CPU when idle, and thought that could
> > explain the alerts about dropped packets where the other IDSes
> > are fine.
> 
> 
> The high overhead is due to the Bro communication loop.  If
> you run in standalone mode (or just manually run a Bro process
> without loading frameworks/communication/listen.bro) you won't
> see the high cpu load.
[...]

Thanks Seth,

I think we're onto something here. node.cfg has "standalone" and
yet the cpu load is high. Where should I be looking? (I have to
admit I'm quite new to bro).

# broctl status
waiting for lock ...... ok
Name       Type       Host       Status        Pid    Peers  Started
bro        standalone localhost  running       4508   0      10 Feb 14:16:37

policy/auto/standalone-layou.bro has

# Automatically generated. Do not edit.
redef Communication::listen_port = 47760/tcp;
redef Communication::nodes += {
        ["control"] = [$host=127.0.0.1, $class="control", $events=Control::controller_events],
};

# lsof -i tcp -ac bro
COMMAND  PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
bro     4526 root    0u  IPv4 3729419      0t0  TCP *:47760 (LISTEN)

# broctl config | grep -e polic -e standalone
defsitepolicypath = /usr/local/share/bro/site
policydir = /usr/local/share/bro
policydirbroctl = /nsm/bro/spool/policy/broctl
policydirsiteinstall = /nsm/bro/spool/policy/site
policydirsiteinstallauto = /nsm/bro/spool/policy/auto
sitepolicymanager = local-manager.bro
sitepolicypath = /usr/local/share/bro/site
sitepolicystandalone = local.bro
sitepolicyworker = local-worker.bro
standalone = 1

Thanks,
Stephane


From seth at icir.org  Fri Feb 10 08:40:17 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 10 Feb 2012 11:40:17 -0500
Subject: [Bro] high cpu usage and strange select(2) behavior
In-Reply-To: <20120210160530.GG4052@yahoo.fr>
References: <20120209144337.GB6571@yahoo.fr>
	<CAGnVnBn+e95JYkaeG2f9zBOhT6wT3p8tEmaZvwcA6mASwgGGDw@mail.gmail.com>
	<20120210112402.GB4052@yahoo.fr>
	<DAE533D0-8CD5-42AB-8EBB-26903BA33360@icir.org>
	<20120210160530.GG4052@yahoo.fr>
Message-ID: <6DDF961E-940E-44D2-945C-DC962D668174@icir.org>


On Feb 10, 2012, at 11:05 AM, Stephane Chazelas wrote:

> I think we're onto something here. node.cfg has "standalone" and
> yet the cpu load is high. Where should I be looking? (I have to
> admit I'm quite new to bro).
> # Automatically generated. Do not edit.
> redef Communication::listen_port = 47760/tcp;
> redef Communication::nodes += {
>        ["control"] = [$host=127.0.0.1, $class="control", $events=Control::controller_events],
> };


Oops!  I completely forgot that the communications framework is always enabled when you use BroControl so that the control framework is available.  It makes it possible to communicate with and control Bro at runtime through broctl.

There aren't any knobs to disable that at the moment since BroControl's functionality would be sporadically broken without it.  I've been bugging Robin about the high cpu load with the communications framework enabled and low traffic for quite a few years but you can actually compensate for it quite a bit now by increasing the timeout values that you were writing about before.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From mcholste at gmail.com  Fri Feb 10 13:42:43 2012
From: mcholste at gmail.com (Martin Holste)
Date: Fri, 10 Feb 2012 15:42:43 -0600
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <201202101646.40289.mvv@sentia.nl>
References: <201202031438.09356.mvv@sentia.nl>
	<201202101549.47457.mvv@sentia.nl>
	<ADEC3C4D-A815-411E-911A-44C685643214@icir.org>
	<201202101646.40289.mvv@sentia.nl>
Message-ID: <CANpnLHiL3pgnvaiCwTaDTfM9pSu5E9DFH5ptOo9appAZMMH6Nw@mail.gmail.com>

What do you see in /proc/net/pf_ring/ ?  If you cat a file matching
the PID of one of the Bro processes, it should say what the cluster_id
is.  If they are all 21, then it is working.

On Fri, Feb 10, 2012 at 9:46 AM, Machiel van Veen <mvv at sentia.nl> wrote:
> On Friday 10 February 2012 16:26:48 Seth Hall wrote:
>> It looks like you are missing the setting that turns on the pf_ring
>> ?clustering support. ?If you built against the pf_ring libpcap wrapper it
>> ?should have been put in there automatically (unless you installed over top
>> ?of a previous installation?).
>>
>> Add this to your broctl.cfg and do "check", "install", "restart" in broctl.
>> PFRingClusterId = 21
>>
>> ? .Seth
>
> I've added the option, there is no difference. I did notice in the debug logs
> before that this option has been set by default. At startup i see the
> following for all workers, proxy and manager:
>
> "PCAP_PF_RING_USE_CLUSTER_PER_FLOW=1 PCAP_PF_RING_CLUSTER_ID=21"
>
> The bro binary does seem to use the correct lib:
>
> $ ldd /opt/bro/bin/bro | grep pcap
> libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x00007fae5cad2000)
>
> I'll go ahead and do this again on monday, perhaps I did make a mistake during
> the build process.
>
> Thanks, Machiel.
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From jones at tacc.utexas.edu  Sun Feb 12 07:38:45 2012
From: jones at tacc.utexas.edu (William Jones)
Date: Sun, 12 Feb 2012 15:38:45 +0000
Subject: [Bro]  Bro 2.0 packets dropped
In-Reply-To: <6117B7B8-69CD-4BA5-8BAD-6EF9F2D3FF88@icir.org>
References: <201202031438.09356.mvv@sentia.nl>
	<201202031618.29452.mvv@sentia.nl>
	<F1AB8BEC-7AD9-4D64-A652-B72CA897332C@icir.org>
	<201202090841.25202.mvv@sentia.nl>
	<6117B7B8-69CD-4BA5-8BAD-6EF9F2D3FF88@icir.org>
Message-ID: <D0BD4A5C03717E40AC104854AF37B8C2107F8481@EXMBX01.austin.utexas.edu>

There is a relative new behavior from the scanners.   In order are to work around the automatic scan blocking they have increased the scan rate to so that they can scan 30K-60K address in a second.   This make bro go compute bound, I think it do to creating a recorded for each connection pair,  and it cannot keep up.   

Using PF_RING helps but not all attach hash well and one worker can be be overwhelmed.

Has anyone else seeing this new behavior. 

Bill Jones










From mvv at sentia.nl  Mon Feb 13 01:30:50 2012
From: mvv at sentia.nl (Machiel van Veen)
Date: Mon, 13 Feb 2012 10:30:50 +0100
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <CANpnLHiL3pgnvaiCwTaDTfM9pSu5E9DFH5ptOo9appAZMMH6Nw@mail.gmail.com>
References: <201202031438.09356.mvv@sentia.nl>
	<201202101646.40289.mvv@sentia.nl>
	<CANpnLHiL3pgnvaiCwTaDTfM9pSu5E9DFH5ptOo9appAZMMH6Nw@mail.gmail.com>
Message-ID: <201202131030.50114.mvv@sentia.nl>

On Friday 10 February 2012 22:42:43 Martin Holste wrote:
> What do you see in /proc/net/pf_ring/ ?  If you cat a file matching
> the PID of one of the Bro processes, it should say what the cluster_id
> is.  If they are all 21, then it is working.
> 

It looks like only one worker uses pfring, the clusterid is 21.

$ ls -l /proc/net/pf_ring/
r--r--r-- 1 root root 0 Feb 13 10:15 15489-p1p1.1
dr-xr-xr-x 5 root root 0 Feb 13 10:15 dev
-r--r--r-- 1 root root 0 Feb 13 10:15 info
-r--r--r-- 1 root root 0 Feb 13 10:15 plugins_info

[BroControl] > status
...
worker-1   worker     192.168.42.215 running       15489  2      13 Feb 10:12:02
...

When I set transparent_mode to 2 it shows:

[BroControl] > netstats
  worker-1: 1329124697.533460 recvd=144655 dropped=0 link=144655
  worker-2: 1329124697.733532 recvd=0 dropped=0 link=0
  worker-3: 1329124697.934520 recvd=0 dropped=0 link=0

The other two workers do not connect. The only thing I could find so far which could cause this is quick_mode, I've disabled this option.

Any idea what else could cause this?

Machiel.


From hlin33 at illinois.edu  Mon Feb 13 07:10:10 2012
From: hlin33 at illinois.edu (Hui Lin (Hugo) )
Date: Mon, 13 Feb 2012 09:10:10 -0600
Subject: [Bro] Hui Lin_what policy is included into Bro by default
Message-ID: <CAKq214nC6wb-1Y6J8WPZ5fMBOD2ojH0jUVBogYTdxqMUmU9Efw@mail.gmail.com>

Hi,

I think I post similar questions before to ask disable default log option.

Now, I am wondering what logs under which bro installation directories are
enabled by default. I am little confused on the Bro 2.0 policy script
hierarchy (so perhaps we should include this description in the
documentation?)

For example, I find a ssh bro policy under base/protocols/ssh and
policy/protocols/ssh. I guess the "base" one is for basic ssh event while
the "policy" one include some detection rules, such as bruteforce attacks.
If I want to use them, do I have to include them, or they are already been
included by default?

Best,

Hui


-- 
Hui Lin
Research Assistant
DEPEND Research Group, ECE Department
University of Illinois at Urbana-Champaign
hlin33 at illinois.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120213/ebda82ac/attachment.html 

From mvv at sentia.nl  Mon Feb 13 07:15:07 2012
From: mvv at sentia.nl (Machiel van Veen)
Date: Mon, 13 Feb 2012 16:15:07 +0100
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <201202131030.50114.mvv@sentia.nl>
References: <201202031438.09356.mvv@sentia.nl>
	<CANpnLHiL3pgnvaiCwTaDTfM9pSu5E9DFH5ptOo9appAZMMH6Nw@mail.gmail.com>
	<201202131030.50114.mvv@sentia.nl>
Message-ID: <201202131615.08023.mvv@sentia.nl>

On Monday 13 February 2012 10:30:50 Machiel van Veen wrote:
> On Friday 10 February 2012 22:42:43 Martin Holste wrote:
> > What do you see in /proc/net/pf_ring/ ?  If you cat a file matching
> > the PID of one of the Bro processes, it should say what the cluster_id
> > is.  If they are all 21, then it is working.
> 
...
> The other two workers do not connect. The only thing I could find so far
>  which could cause this is quick_mode, I've disabled this option.
> 
> Any idea what else could cause this?
> 

Good news, it turned out to be a recent PF_RING issue, I've updated the kernel 
module and it works now. A issue with pfring_set_channel_id() which has been 
fixed on the 11th. 

I noticed the PF_RING problem trying "pfcount -i p1p1" while bro was running.

Thanks, Machiel.


From seth at icir.org  Mon Feb 13 07:23:47 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 13 Feb 2012 10:23:47 -0500
Subject: [Bro] Bro 2.0 packets dropped
In-Reply-To: <201202131615.08023.mvv@sentia.nl>
References: <201202031438.09356.mvv@sentia.nl>
	<CANpnLHiL3pgnvaiCwTaDTfM9pSu5E9DFH5ptOo9appAZMMH6Nw@mail.gmail.com>
	<201202131030.50114.mvv@sentia.nl>
	<201202131615.08023.mvv@sentia.nl>
Message-ID: <02C66182-33DD-4460-BD9A-69943ED8C7BF@icir.org>


On Feb 13, 2012, at 10:15 AM, Machiel van Veen wrote:

> Good news, it turned out to be a recent PF_RING issue, I've updated the kernel 
> module and it works now. A issue with pfring_set_channel_id() which has been 
> fixed on the 11th. 


Cool!  I'm glad you figured it out, I was running out of ideas. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jsiwek at illinois.edu  Mon Feb 13 08:03:17 2012
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 13 Feb 2012 16:03:17 +0000
Subject: [Bro] Hui Lin_what policy is included into Bro by default
In-Reply-To: <CAKq214nC6wb-1Y6J8WPZ5fMBOD2ojH0jUVBogYTdxqMUmU9Efw@mail.gmail.com>
References: <CAKq214nC6wb-1Y6J8WPZ5fMBOD2ojH0jUVBogYTdxqMUmU9Efw@mail.gmail.com>
Message-ID: <0C19728C-7B2D-484D-9E45-2B86490C23B3@illinois.edu>

> For example, I find a ssh bro policy under base/protocols/ssh and policy/protocols/ssh. I guess the "base" one is for basic ssh event while the "policy" one include some detection rules, such as bruteforce attacks.

Yeah, the general idea is that scripts under base/ are mostly for basic state tracking and logging, while ones under policy/ do more advanced/specific things.

> If I want to use them, do I have to include them, or they are already been included by default?

Using bro on the command line, everything in base/ is loaded by default (overridden with the -b option), but nothing in policy/.

The site/local.bro script will load most everything in policy/ and by default that's loaded if you use broctl.

+Jon


From seth at icir.org  Mon Feb 13 08:16:23 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 13 Feb 2012 11:16:23 -0500
Subject: [Bro] Hui Lin_what policy is included into Bro by default
In-Reply-To: <CAKq214nC6wb-1Y6J8WPZ5fMBOD2ojH0jUVBogYTdxqMUmU9Efw@mail.gmail.com>
References: <CAKq214nC6wb-1Y6J8WPZ5fMBOD2ojH0jUVBogYTdxqMUmU9Efw@mail.gmail.com>
Message-ID: <4EDF6A33-0602-4B2A-B49F-33751180E7C9@icir.org>

On Feb 13, 2012, at 10:10 AM, Hui Lin (Hugo) wrote:

> Now, I am wondering what logs under which bro installation directories are enabled by default. I am little confused on the Bro 2.0 policy script hierarchy (so perhaps we should include this description in the documentation?)

Yes, that's something that we still need to document very clearly.  Fortunately there is a clear answer!

> For example, I find a ssh bro policy under base/protocols/ssh and policy/protocols/ssh. I guess the "base" one is for basic ssh event while the "policy" one include some detection rules, such as bruteforce attacks.

Script in base/protocols (the "base scripts") take care of a lot of the legwork of enabling and configuring the core analyzers.  They also create one or more log files to log something about the protocol and they create and populate the connection record extensions like c$ssl, c$http, s$smtp.  There is still a bit of detection being done in a few of the base scripts, but over time we will pull the rest of that out into the scripts in the policy/ directory.  The best way to think of what they do is provide very basic non-detection oriented functionality.

You had the basic idea right. :)

>   If I want to use them, do I have to include them, or they are already been included by default?

Every single script in base/ is included by default (but you can even turn those off if you are really interested in it).

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From basusahay at gmail.com  Mon Feb 13 11:01:05 2012
From: basusahay at gmail.com (Rishi Sahay)
Date: Tue, 14 Feb 2012 00:31:05 +0530
Subject: [Bro] Reg: bro.init file error (Very Urgent)
Message-ID: <CABmCyhT3pMBhxvFiqbhy1=gRKXR=pBZREkE_XAfGVa4xdVTYTw@mail.gmail.com>

Hello,

 I was working with BRO IDS on my machine. By mistake I moved the Bro
directory which is in /usr/local/Bro/share/Bro directory on the desktop of
my machine. I moved again that Bro directory in /usr/local/Bro/Share/
directory. But now when I execute policy script it is showing error that
"Can not open Bro.init". Now I am not able to execute any policy scripts on
the BRO IDS. Please, tell me how to recover from this error. Otherwise my
two months work will be lost. Please,help me in this regard. Thanks in
advance.

-- 
Best regards
  Rishikesh Sahay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120214/8bb141af/attachment.html 

From seth at icir.org  Mon Feb 13 11:25:55 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 13 Feb 2012 14:25:55 -0500
Subject: [Bro] Reg: bro.init file error (Very Urgent)
In-Reply-To: <CABmCyhT3pMBhxvFiqbhy1=gRKXR=pBZREkE_XAfGVa4xdVTYTw@mail.gmail.com>
References: <CABmCyhT3pMBhxvFiqbhy1=gRKXR=pBZREkE_XAfGVa4xdVTYTw@mail.gmail.com>
Message-ID: <5E4354B8-8791-4714-B823-4B6016D9783D@icir.org>


On Feb 13, 2012, at 2:01 PM, Rishi Sahay wrote:

>  I was working with BRO IDS on my machine. By mistake I moved the Bro directory which is in /usr/local/Bro/share/Bro directory on the desktop of my machine. I moved again that Bro directory in /usr/local/Bro/Share/ directory. But now when I execute policy script it is showing error that "Can not open Bro.init". Now I am not able to execute any policy scripts on the BRO IDS.

I'm guessing you are running something in the 1.5.x series of releases?  All you should need to do is get a new copy of the release you are using and copy the scripts from policy/ to that directory.

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jsiwek at illinois.edu  Mon Feb 13 11:29:58 2012
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 13 Feb 2012 19:29:58 +0000
Subject: [Bro] Reg: bro.init file error (Very Urgent)
In-Reply-To: <CABmCyhT3pMBhxvFiqbhy1=gRKXR=pBZREkE_XAfGVa4xdVTYTw@mail.gmail.com>
References: <CABmCyhT3pMBhxvFiqbhy1=gRKXR=pBZREkE_XAfGVa4xdVTYTw@mail.gmail.com>
Message-ID: <C25EF62F-1AD9-421C-95D0-5039629BF625@illinois.edu>

>  I was working with BRO IDS on my machine. By mistake I moved the Bro directory which is in /usr/local/Bro/share/Bro directory on the desktop of my machine. I moved again that Bro directory in /usr/local/Bro/Share/ directory. But now when I execute policy script it is showing error that "Can not open Bro.init". 

I'm not sure if it's just the way you're typing the paths, but it could be your casing -- the paths are usually all lowercase by default ("bro" vs. "Bro" and "share" vs. "Share").

And I think if you do a `bro --help` and look at the $BROPATH option, it should tell you the default search paths for scripts that you can compare to the current location to see what's wrong.  If not, you can always try changing your BROPATH environment variable to point to where they currently are located.

+Jon


From tyler at hudakville.com  Mon Feb 13 13:02:00 2012
From: tyler at hudakville.com (Tyler)
Date: Mon, 13 Feb 2012 16:02:00 -0500
Subject: [Bro] http.log missing info
Message-ID: <4F397A48.50004@hudakville.com>

Hello all,

  Forgive my question if its basic, but I was not able to find the
answer I was looking for in the docs, FAQ or mailing list archive.

  I just set up bro and am having it just run locally so I can work
with it. I noticed that my http.log file is not recording all the
information for HTTP sessions. It records the connection info (src,
dst, ports) the status_code and msg and thats it. Anything beyond
that, like the uri, host, UA are missing.

  I also noticed that the request body lengths are set to 0. My
conn.log shows the same (request lengths are set to 0). To me this
means that its not seeing the requests go out. I have verified this is
happening with both live analysis and through a recorded pcap which
does contain the requests.

 Oddly, if I dl the http.pcap from the second exercise from the 2011
Bro workshop, it does work correctly.

 Is there a config option that I did not set correctly? I have
verified that my networks.cfg is set up correctly and have not
modified any other settings.

Tyler


From seth at icir.org  Mon Feb 13 13:58:06 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 13 Feb 2012 16:58:06 -0500
Subject: [Bro] http.log missing info
In-Reply-To: <4F397A48.50004@hudakville.com>
References: <4F397A48.50004@hudakville.com>
Message-ID: <80D7862B-B58F-49A9-888A-61D643C81AAD@icir.org>


On Feb 13, 2012, at 4:02 PM, Tyler wrote:

> Anything beyond
> that, like the uri, host, UA are missing.

You're missing everything from the client. ;)

> Is there a config option that I did not set correctly? I have
> verified that my networks.cfg is set up correctly and have not
> modified any other settings.


I suspect you have checksum offloading onto your NIC.  The behavior when someone runs this on their desktop is that you will see everything from the server since the checksums are set correct, but data from the client (you) will offload checksum creation to the NIC so when libpcap receives the packet it has random data in the checksum field.

Try running Bro with the -C flag to disable checksum validation (but only for testing, you obviously don't want that on live traffic).

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From tyler at hudakville.com  Mon Feb 13 15:27:20 2012
From: tyler at hudakville.com (Tyler)
Date: Mon, 13 Feb 2012 18:27:20 -0500
Subject: [Bro] http.log missing info
In-Reply-To: <80D7862B-B58F-49A9-888A-61D643C81AAD@icir.org>
References: <4F397A48.50004@hudakville.com>
	<80D7862B-B58F-49A9-888A-61D643C81AAD@icir.org>
Message-ID: <4F399C58.20609@hudakville.com>

That was it!  Man, I even read that in the FAQ and didn't think about
it. Thanks!

Tyler



On 02/13/2012 04:58 PM, Seth Hall wrote:
> 
> On Feb 13, 2012, at 4:02 PM, Tyler wrote:
> 
>> Anything beyond
>> that, like the uri, host, UA are missing.
> 
> You're missing everything from the client. ;)
> 
>> Is there a config option that I did not set correctly? I have
>> verified that my networks.cfg is set up correctly and have not
>> modified any other settings.
> 
> 
> I suspect you have checksum offloading onto your NIC.  The behavior when someone runs this on their desktop is that you will see everything from the server since the checksums are set correct, but data from the client (you) will offload checksum creation to the NIC so when libpcap receives the packet it has random data in the checksum field.
> 
> Try running Bro with the -C flag to disable checksum validation (but only for testing, you obviously don't want that on live traffic).
> 
>   .Seth
> 
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
> 
> 


From robin at icir.org  Mon Feb 13 16:14:33 2012
From: robin at icir.org (Robin Sommer)
Date: Mon, 13 Feb 2012 16:14:33 -0800
Subject: [Bro] high cpu usage and strange select(2) behavior
In-Reply-To: <20120209144337.GB6571@yahoo.fr>
References: <20120209144337.GB6571@yahoo.fr>
Message-ID: <20120214001433.GI69532@icir.org>

Coming late to this thread, two remarks:

    - Bro's I/O loop definitly needs some work, there's some stuff in
      there that's clearly suboptimal. High CPU load with low traffic
      is one of them.

    - That said, while some things may look odd, there are reasons for
      why it's done this way. It's mostly platform artifacts: at the
      time the code was written, non-blocking pcap file descriptors
      weren't (correctly) supported on some OSs; and sometimes capture
      performance turned out to be better with a less intuitive
      version of the code.

Reworking this is on the todo list but it needs care to not make
things worse.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From basusahay at gmail.com  Fri Feb 17 02:30:30 2012
From: basusahay at gmail.com (Rishi Sahay)
Date: Fri, 17 Feb 2012 16:00:30 +0530
Subject: [Bro] Feature extraction from tcpdump dataset
Message-ID: <CABmCyhSHhKa_o5u6G5tbjBixaoNrB1m0Q0SoAc3KJB6YXFHwPA@mail.gmail.com>

Hello,

 I have used one BRO script to extract the features from the tcpdump
dataset. But after executing the script I am getting the output related to
only on or two ip addresses. Tcp protocol related features are also
missing. In the script tcp related event is already written. Source bytes
and destination bytes and many other fields are missing in the output.
Output is coming like this.

12623 894270562.332940 53 53 192.168.1.10 172.16.112.20 114.076435 udp 53
SF 0 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3379 894264041.623323 53 53 172.16.112.20 192.168.1.10 188.920237 udp 53 SF
0 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
18751 894276050.128957 53 53 192.168.1.10 192.112.36.4 79.935439 udp 53 S0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
18911 894276319.473667 53 53 192.168.1.10 172.16.112.20 32.000492 udp 53 SF
0 32 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
15254 894272639.391092 3 1 192.168.1.1 192.168.1.10 56.142144 icmp 3 SH 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
124 894261608.818447 53 53 172.16.112.20 192.168.1.10 10.889097 udp 53 SF 0
88 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
10890 894268867.616405 53 53 172.16.112.20 192.168.1.20 86.801842 udp 53 SF
0 106 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
20487 894301329.172629 21836 69 192.168.1.1 255.255.255.255 14.185021 udp
69 S0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0
3168 894264041.623323 53 53 172.16.112.20 192.168.1.10 116.796997 udp 53 SF
0 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
5215 894264041.623323 53 53 172.16.112.20 192.168.1.10 824.392281 udp 53 SF
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
20920 894309698.188181 1100 53 192.168.1.20 192.168.1.10 0.000204 udp 53 S0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
17426 894274962.704143 3 1 192.168.1.1 192.168.1.10 55.393165 icmp 3 SH 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
19407 894279440.127267 53 53 192.168.1.10 192.203.230.10 52.577897 udp 53
S0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
14138 894271782.414871 53 53 172.16.112.20 192.168.1.10 70.741656 udp 53 SF
0 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
11385 894269373.326256 53 53 192.168.1.10 172.16.112.20 92.350040 udp 53 S0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
5503 894265091.295491 53 53 172.16.112.20 194.7.248.153 4.497476 udp 53 S0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
6388 894264946.828720 53 53 172.16.112.20 192.168.1.10 523.448570 udp 53 SF
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7118 894265986.131874 53 53 172.16.112.20 192.168.1.20 0.000000 udp 53 OTH
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
6655 894264946.828720 53 53 172.16.112.20 192.168.1.10 768.043159 udp 53 SF
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7530 894265816.715767 53 53 172.16.112.20 192.168.1.10 291.721001 udp 53 SF
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7720 894266226.030665 53 53 172.16.112.20 135.8.60.182 13.094761 udp 53 S0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1312 894262128.202500 53 53 172.16.112.20 192.168.1.10 472.436405 udp 53 SF
0 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
19348 894279464.545284 53 53 192.168.1.10 128.9.0.107 0.000000 udp 53 OTH 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
19665 894283246.635702 53 53 172.16.112.20 197.218.177.69 4.499342 udp 53
S0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1756 894262128.202500 53 53 172.16.112.20 192.168.1.10 720.434234 udp 53 SF
0 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
13742 894271496.860470 53 53 192.168.1.10 172.16.112.20 0.000000 udp 53 OTH
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3339 894264041.623323 53 53 172.16.112.20 192.168.1.10 135.101538 udp 53 SF
0 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9625 894267896.744671 53 53 172.16.112.20 192.168.1.10 0.187004 udp 53 SF 0
80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
7513 894265816.715767 53 53 172.16.112.20 192.168.1.10 269.665598 udp 53 SF
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Please, tell me some scripts if it is available through which I can extract
all the features from the tcpdump data set. I am newbie to BRO IDS. I don't
know much about the BRO IDS. Please,help me. Thanks in advance.

-- 
Best regards
  Rishikesh Sahay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120217/aacc9eb4/attachment.html 

From relevantusername at gmail.com  Fri Feb 17 06:34:34 2012
From: relevantusername at gmail.com (relevant username)
Date: Fri, 17 Feb 2012 08:34:34 -0600
Subject: [Bro] Fast flux domains and Bro
Message-ID: <CAJMfqO=K8AR5uvQ4r4UMnrBT6w0dFhoKUNbTq5sSZ_iwhmis_w@mail.gmail.com>

Hello,

I was wondering if anyone on the list has any experience using Bro to
detect fast flux domains.

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120217/bb418fe7/attachment.html 

From seth at icir.org  Fri Feb 17 07:19:15 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 17 Feb 2012 10:19:15 -0500
Subject: [Bro] Fast flux domains and Bro
In-Reply-To: <CAJMfqO=K8AR5uvQ4r4UMnrBT6w0dFhoKUNbTq5sSZ_iwhmis_w@mail.gmail.com>
References: <CAJMfqO=K8AR5uvQ4r4UMnrBT6w0dFhoKUNbTq5sSZ_iwhmis_w@mail.gmail.com>
Message-ID: <1CA92977-953D-4793-B173-9670AB869A82@icir.org>


On Feb 17, 2012, at 9:34 AM, relevant username wrote:

> I was wondering if anyone on the list has any experience using Bro to detect fast flux domains.


I wrote a script quite a few years ago, but I haven't touched it in a long time and it likely won't work right on 2.0.  It's a very short script though and could probably be ported fairly easily.  It uses the detection technique outlined in this paper:
	http://pi1.informatik.uni-mannheim.de/filepool/publications/fast-flux-ndss08.pdf

Someone else had a fast flux detection script at that time too, but I don't know if they still have it floating around anywhere or not.  I attached my script to this email.  When it's ported to 2.x we can get it into the contributed scripts repository.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: dns-fastflux.bro
Type: application/octet-stream
Size: 2593 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120217/cd51d335/attachment.obj 
-------------- next part --------------


  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/


From sri at basam.org  Fri Feb 17 07:30:09 2012
From: sri at basam.org (sridhar basam)
Date: Fri, 17 Feb 2012 10:30:09 -0500
Subject: [Bro] Feature extraction from tcpdump dataset
In-Reply-To: <CABmCyhSHhKa_o5u6G5tbjBixaoNrB1m0Q0SoAc3KJB6YXFHwPA@mail.gmail.com>
References: <CABmCyhSHhKa_o5u6G5tbjBixaoNrB1m0Q0SoAc3KJB6YXFHwPA@mail.gmail.com>
Message-ID: <CAGnVnB=KZ_=SWuJEWLL-Gnz0VxYmPd_AV6wqHa9RtPo24Tx6QQ@mail.gmail.com>

On Fri, Feb 17, 2012 at 5:30 AM, Rishi Sahay <basusahay at gmail.com> wrote:
> Hello,
>
> ?I have used one BRO script to extract the features from the tcpdump
> dataset. But after executing the script I am getting the output related to
> only on or two ip addresses. Tcp protocol related features are also missing.
> In the script tcp related event is already written. Source bytes and
> destination bytes and many other fields are missing in the output. Output is
> coming like this.
>

What scripts are you using to generate this output? Also what version
of bro? The output doesn't look like the conn.log from either the 2.0
or 1.5 version of bro. Have you written a script to capture this?

> 12623 894270562.332940 53 53 192.168.1.10 172.16.112.20 114.076435 udp 53 SF
> 0 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 3379 894264041.623323 53 53 172.16.112.20 192.168.1.10 188.920237 udp 53 SF
> 0 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 18751 894276050.128957 53 53 192.168.1.10 192.112.36.4 79.935439 udp 53 S0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 18911 894276319.473667 53 53 192.168.1.10 172.16.112.20 32.000492 udp 53 SF
> 0 32 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 15254 894272639.391092 3 1 192.168.1.1 192.168.1.10 56.142144 icmp 3 SH 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 124 894261608.818447 53 53 172.16.112.20 192.168.1.10 10.889097 udp 53 SF 0
> 88 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 10890 894268867.616405 53 53 172.16.112.20 192.168.1.20 86.801842 udp 53 SF
> 0 106 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 20487 894301329.172629 21836 69 192.168.1.1 255.255.255.255 14.185021 udp 69
> S0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 3168 894264041.623323 53 53 172.16.112.20 192.168.1.10 116.796997 udp 53 SF
> 0 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 5215 894264041.623323 53 53 172.16.112.20 192.168.1.10 824.392281 udp 53 SF
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 20920 894309698.188181 1100 53 192.168.1.20 192.168.1.10 0.000204 udp 53 S0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 17426 894274962.704143 3 1 192.168.1.1 192.168.1.10 55.393165 icmp 3 SH 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 19407 894279440.127267 53 53 192.168.1.10 192.203.230.10 52.577897 udp 53 S0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 14138 894271782.414871 53 53 172.16.112.20 192.168.1.10 70.741656 udp 53 SF
> 0 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 11385 894269373.326256 53 53 192.168.1.10 172.16.112.20 92.350040 udp 53 S0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 5503 894265091.295491 53 53 172.16.112.20 194.7.248.153 4.497476 udp 53 S0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 6388 894264946.828720 53 53 172.16.112.20 192.168.1.10 523.448570 udp 53 SF
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 7118 894265986.131874 53 53 172.16.112.20 192.168.1.20 0.000000 udp 53 OTH 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 6655 894264946.828720 53 53 172.16.112.20 192.168.1.10 768.043159 udp 53 SF
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 7530 894265816.715767 53 53 172.16.112.20 192.168.1.10 291.721001 udp 53 SF
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 7720 894266226.030665 53 53 172.16.112.20 135.8.60.182 13.094761 udp 53 S0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 1312 894262128.202500 53 53 172.16.112.20 192.168.1.10 472.436405 udp 53 SF
> 0 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 19348 894279464.545284 53 53 192.168.1.10 128.9.0.107 0.000000 udp 53 OTH 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 19665 894283246.635702 53 53 172.16.112.20 197.218.177.69 4.499342 udp 53 S0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 1756 894262128.202500 53 53 172.16.112.20 192.168.1.10 720.434234 udp 53 SF
> 0 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 13742 894271496.860470 53 53 192.168.1.10 172.16.112.20 0.000000 udp 53 OTH
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 3339 894264041.623323 53 53 172.16.112.20 192.168.1.10 135.101538 udp 53 SF
> 0 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 9625 894267896.744671 53 53 172.16.112.20 192.168.1.10 0.187004 udp 53 SF 0
> 80 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 7513 894265816.715767 53 53 172.16.112.20 192.168.1.10 269.665598 udp 53 SF
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>
> Please, tell me some scripts if it is available through which I can extract
> all the features from the tcpdump data set. I am newbie to BRO IDS. I don't
> know much about the BRO IDS. Please,help me. Thanks in advance.
>

What sort of data are you trying to extract from the pcap files? By
default bro-2.0 analyses a lot of events out of the box, there should
be *.log files in your working directory. What are the names of the
files you see?

 sridhar


> --
> Best regards
> ? Rishikesh Sahay
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From sychan at lbl.gov  Fri Feb 17 10:47:41 2012
From: sychan at lbl.gov (Stephen Chan)
Date: Fri, 17 Feb 2012 10:47:41 -0800
Subject: [Bro] Fast flux domains and Bro
In-Reply-To: <1CA92977-953D-4793-B173-9670AB869A82@icir.org>
References: <CAJMfqO=K8AR5uvQ4r4UMnrBT6w0dFhoKUNbTq5sSZ_iwhmis_w@mail.gmail.com>
	<1CA92977-953D-4793-B173-9670AB869A82@icir.org>
Message-ID: <CA+n9Yfp=biCN9rOC8EiH3MwFqqeq_V+_dnZay-sV_chTrwe5mQ@mail.gmail.com>

   We wrote a few iterations of FF DNS detectors in Bro several years
back. Our paper is here:

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.185.3991

   We tried a few different approaches, but the one that worked the
best in the end was based on white/black listing ASNs. Scott may
remember it better than I, as he is further along in his recovery from
newborn baby induced memory loss. Jason Lee has probably the freshest
recollection of approach since he worked on the most recent edit of
the paper.

   Steve


On Fri, Feb 17, 2012 at 7:19 AM, Seth Hall <seth at icir.org> wrote:
>
> On Feb 17, 2012, at 9:34 AM, relevant username wrote:
>
>> I was wondering if anyone on the list has any experience using Bro to detect fast flux domains.
>
>
> I wrote a script quite a few years ago, but I haven't touched it in a long time and it likely won't work right on 2.0. ?It's a very short script though and could probably be ported fairly easily. ?It uses the detection technique outlined in this paper:
> ? ? ? ?http://pi1.informatik.uni-mannheim.de/filepool/publications/fast-flux-ndss08.pdf
>
> Someone else had a fast flux detection script at that time too, but I don't know if they still have it floating around anywhere or not. ?I attached my script to this email. ?When it's ported to 2.x we can get it into the contributed scripts repository.
>
>
>
>
> ?.Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From iduckhd at hotmail.com  Sat Feb 18 09:38:21 2012
From: iduckhd at hotmail.com (Ioannis WiCom)
Date: Sat, 18 Feb 2012 17:38:21 +0000
Subject: [Bro] http.log event
Message-ID: <SNT134-W5996F68DC6576D4B141F9BB3600@phx.gbl>


I have enabled the conn.log and http.log in Bro.
I am seeing TCP events in dst_port==80 that do not generate events for http.log.

No Event in HTTP:  http 2120 80 tcp 368 1414142 SF X
Event %1 in HTTP: http 59637 80 tcp 1495 1244 SF X %1

The SF flag indicates Normal Establishment and Termination. Why would Bro record one event in http.log, and not the other one?
Note that both of them are actual HTTP traffic. 

Another example:
No event in HTTP: http 49971 80 tcp 0 924818 SH X

I am guessing in the last one, since there are 0 bytes from the originator, it would not generate an event for http.log.

My question is more general: In which occasions of dst_port==80, an http.log event is being recorded?

thank you,
Yannis 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120218/243aa1f7/attachment.html 

From seth at icir.org  Sat Feb 18 13:52:56 2012
From: seth at icir.org (Seth Hall)
Date: Sat, 18 Feb 2012 16:52:56 -0500
Subject: [Bro] http.log event
In-Reply-To: <SNT134-W5996F68DC6576D4B141F9BB3600@phx.gbl>
References: <SNT134-W5996F68DC6576D4B141F9BB3600@phx.gbl>
Message-ID: <FC25FD3A-C895-4CB0-AD92-ABA440A3BDDD@icir.org>


On Feb 18, 2012, at 12:38 PM, Ioannis WiCom wrote:

> No Event in HTTP:  http 2120 80 tcp 368 1414142 SF X
> Event %1 in HTTP: http 59637 80 tcp 1495 1244 SF X %1

You appear to be running an older version of Bro.  I recommend trying out our new 2.0 version that we released last month.  We made so many changes and fixes that it's very likely the problem you are seeing has been fixed.

You can get it from here (we even have binaries for several platforms):
	http://www.bro-ids.org/download/

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From sconzo at visiblerisk.com  Mon Feb 20 13:41:03 2012
From: sconzo at visiblerisk.com (Mike Sconzo)
Date: Mon, 20 Feb 2012 15:41:03 -0600
Subject: [Bro] Script Question
Message-ID: <CAGZoo6Gb52AAYEpcUR9YM=EU1PDGbnVUzHQo4-SNXOGpHmUU+g@mail.gmail.com>

I'm still trying to get a handle on script writing, but I have a
question on one of the events.

http_all_headers  I've got a script that _should_ be processing the headers
like:

for (i in hlist )
{
  local hdr = hlist[i];
  Log::write(BrowserHeaders::LOG, ...)
  ....
}

I've tried using both the NOTICE facility and creating my own log to
at least log to see if any header values are getting processed.  I'm
not sure if I'm doing it wrong or if http_all_headers doesn't do what
I think it should when I think it should.  I can attach the script if
necessary, but I'm curious how I can get more information or even a
basic way to verify that one of these events is even happening.

On the upside a broctl check returns ok when I have my script loaded,
but just because there aren't any errors doesn't mean I did it right.

Thanks for any input.

-=Mike

-- 
cat ~/.bash_history > documentation.txt


From vallentin at icir.org  Mon Feb 20 15:18:43 2012
From: vallentin at icir.org (Matthias Vallentin)
Date: Mon, 20 Feb 2012 15:18:43 -0800
Subject: [Bro] Script Question
In-Reply-To: <CAGZoo6Gb52AAYEpcUR9YM=EU1PDGbnVUzHQo4-SNXOGpHmUU+g@mail.gmail.com>
References: <CAGZoo6Gb52AAYEpcUR9YM=EU1PDGbnVUzHQo4-SNXOGpHmUU+g@mail.gmail.com>
Message-ID: <CADTpMNYxzDPBT9VBRdRA8fB32ESojiras1GcBxKH=vxuNNFiTg@mail.gmail.com>

> for (i in hlist )
> {
> ?local hdr = hlist[i];
> ?Log::write(BrowserHeaders::LOG, ...)
> ?....
> }

This create a log record for each header. Is that what you want? Or would you
like to record HTTP headers for each request/reply in a single line?

> I've tried using both the NOTICE facility and creating my own log to
> at least log to see if any header values are getting processed. ?I'm
> not sure if I'm doing it wrong or if http_all_headers doesn't do what
> I think it should when I think it should.

You could inserting bare print statements and then running the script on a
small trace using just the Bro binary, e.g.,

    bro -r your_trace your_script

and observe the output on STDOUT.

>?I can attach the script if necessary, but I'm curious how I can get more
> information or even a basic way to verify that one of these events is even
> happening.

That would always help to give you feedback.

> On the upside a broctl check returns ok when I have my script loaded,
> but just because there aren't any errors doesn't mean I did it right.

Right, some bugs (often value initialization) only manifest at runtime and are
not detectable at the time when you do 'broctl check.'

    Matthias



From seth at icir.org  Mon Feb 20 18:03:23 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 20 Feb 2012 21:03:23 -0500
Subject: [Bro] Script Question
In-Reply-To: <CAGZoo6Gb52AAYEpcUR9YM=EU1PDGbnVUzHQo4-SNXOGpHmUU+g@mail.gmail.com>
References: <CAGZoo6Gb52AAYEpcUR9YM=EU1PDGbnVUzHQo4-SNXOGpHmUU+g@mail.gmail.com>
Message-ID: <C010B9B0-14B0-4E7D-8DBE-D718F74BE365@icir.org>


On Feb 20, 2012, at 4:41 PM, Mike Sconzo wrote:

> I'm still trying to get a handle on script writing, but I have a
> question on one of the events.


You may need to send along more of the script for use to help debug it.  One thought I have though, is did you make sure and include the &log attribute on the record type that you are logging?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From basusahay at gmail.com  Tue Feb 21 04:40:45 2012
From: basusahay at gmail.com (Rishi Sahay)
Date: Tue, 21 Feb 2012 18:10:45 +0530
Subject: [Bro] Extraction of IP identification field from tcpdump file
Message-ID: <CABmCyhSC3W6w2cji3M4F+a+RjNML4qDgsC95P-f1080j5y5bkg@mail.gmail.com>

Hello,

 I want to extract the IP identification field from the tcpdump file. I
have extracted header information from the packet in the tcpdump file using
conn.bro script. But IP identification field has not been extracted. Is
there any script available to extract the IP identification field. I am
using BRO IDS 1.5.3. Please, help me in this regard. Thanks in advance.

-- 
Best regards
  Rishikesh Sahay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120221/d37a7534/attachment.html 

From sconzo at visiblerisk.com  Tue Feb 21 07:34:39 2012
From: sconzo at visiblerisk.com (Mike Sconzo)
Date: Tue, 21 Feb 2012 09:34:39 -0600
Subject: [Bro] Script Question
In-Reply-To: <C010B9B0-14B0-4E7D-8DBE-D718F74BE365@icir.org>
References: <CAGZoo6Gb52AAYEpcUR9YM=EU1PDGbnVUzHQo4-SNXOGpHmUU+g@mail.gmail.com>
	<C010B9B0-14B0-4E7D-8DBE-D718F74BE365@icir.org>
Message-ID: <CAGZoo6GMcjxODnM6FbTsPHUe_v9F9+xc5j36BLon+2QUoK4WBA@mail.gmail.com>

| This create a log record for each header. Is that what you want? Or would you
| like to record HTTP headers for each request/reply in a single line?

Yeah, this was done more to figure out what was going on and to wrap
my head around the syntax

| You could inserting bare print statements and then running the script on a
| small trace using just the Bro binary, e.g.,
|
|    bro -r your_trace your_script
|
| and observe the output on STDOUT.
|
| I did include the log attribute, I'll conquer that next.  More
questions first.

This was just what I needed, thanks!

I did include the  &log attribute, but I still have no actual clue
what I'm doing.  The price of learning.

Another question, is there a way to force (enable) headers to be
matched in a case sensitive way?
In other words I'm interested in "normal" looking headers.  Accept: is
pretty standard accross browser implementation where as accept: or
ACCEPT are outliers.  I noticed when I print the headers they are all
uc'd, didnt know if there was a way around this.

Reason behind all of these odd questions.  I've been working on some
passive identification of browsers so I can ask the question of "what
browser tells me it's msie via the user-agent string, but doesn't
behave like it".  With my current implentation I've got about a 72%
accuracy/detection rate, however it's currently implemented in another
product (commercial) that not everybody can afford. So I figured I'd
port it over and learn bro in the process so I could give it back to
the community.

Thanks for putting up with the questions.

-=Mike



On Mon, Feb 20, 2012 at 8:03 PM, Seth Hall <seth at icir.org> wrote:
>
> On Feb 20, 2012, at 4:41 PM, Mike Sconzo wrote:
>
>> I'm still trying to get a handle on script writing, but I have a
>> question on one of the events.
>
>
> You may need to send along more of the script for use to help debug it. ?One thought I have though, is did you make sure and include the &log attribute on the record type that you are logging?
>
> ?.Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>



-- 
cat ~/.bash_history > documentation.txt



From sri at basam.org  Tue Feb 21 08:03:27 2012
From: sri at basam.org (sridhar basam)
Date: Tue, 21 Feb 2012 11:03:27 -0500
Subject: [Bro] Extraction of IP identification field from tcpdump file
In-Reply-To: <CABmCyhSC3W6w2cji3M4F+a+RjNML4qDgsC95P-f1080j5y5bkg@mail.gmail.com>
References: <CABmCyhSC3W6w2cji3M4F+a+RjNML4qDgsC95P-f1080j5y5bkg@mail.gmail.com>
Message-ID: <CAGnVnBmyJnOGOgcrFp0dv-JUjy6=rRX25Sk6it7cKiw9mHJTOA@mail.gmail.com>

On Tue, Feb 21, 2012 at 7:40 AM, Rishi Sahay <basusahay at gmail.com> wrote:
> Hello,
>
> ?I want to extract the IP identification field from the tcpdump file. I have
> extracted header information from the packet in the tcpdump file using
> conn.bro script. But IP identification field has not been extracted. Is
> there any script available to extract the IP identification field. I am
> using BRO IDS 1.5.3. Please, help me in this regard. Thanks in advance.
>

Assuming you mean the 16 bit id value in the IP header. All i could
come up with is via event new_packet.

global new_packet: event(c: connection, p: pkt_hdr);

pkt_hdr$ip$id

Handling new_packet is a costly event in terms of performance. I am
curious, if you don't mind, why you are tracking the ID values. Seems
like a lot to keep track of and print out.

    Sridhar





> --
> Best regards
> ? Rishikesh Sahay
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From nweaver at ICSI.Berkeley.EDU  Tue Feb 21 08:13:33 2012
From: nweaver at ICSI.Berkeley.EDU (Nicholas Weaver)
Date: Tue, 21 Feb 2012 08:13:33 -0800
Subject: [Bro] Extraction of IP identification field from tcpdump file
In-Reply-To: <CAGnVnBmyJnOGOgcrFp0dv-JUjy6=rRX25Sk6it7cKiw9mHJTOA@mail.gmail.com>
References: <CABmCyhSC3W6w2cji3M4F+a+RjNML4qDgsC95P-f1080j5y5bkg@mail.gmail.com>
	<CAGnVnBmyJnOGOgcrFp0dv-JUjy6=rRX25Sk6it7cKiw9mHJTOA@mail.gmail.com>
Message-ID: <E1C8AAAF-7459-40A9-93CC-44BD5ED3A5EB@icsi.berkeley.edu>


On Tue, Feb 21, 2012 at 7:40 AM, Rishi Sahay <basusahay at gmail.com> wrote:
> Hello,
> 
>  I want to extract the IP identification field from the tcpdump file. I have
> extracted header information from the packet in the tcpdump file using
> conn.bro script. But IP identification field has not been extracted. Is
> there any script available to extract the IP identification field. I am
> using BRO IDS 1.5.3. Please, help me in this regard. Thanks in advance.
> 

For this task you might want to consider using IPSumdump

http://www.cs.ucla.edu/~kohler/ipsumdump/

Which is much faster for getting IP header level information from a TCPdump file than bro.




From seth at icir.org  Tue Feb 21 09:48:23 2012
From: seth at icir.org (Seth Hall)
Date: Tue, 21 Feb 2012 12:48:23 -0500
Subject: [Bro] Script Question
In-Reply-To: <CAGZoo6GMcjxODnM6FbTsPHUe_v9F9+xc5j36BLon+2QUoK4WBA@mail.gmail.com>
References: <CAGZoo6Gb52AAYEpcUR9YM=EU1PDGbnVUzHQo4-SNXOGpHmUU+g@mail.gmail.com>
	<C010B9B0-14B0-4E7D-8DBE-D718F74BE365@icir.org>
	<CAGZoo6GMcjxODnM6FbTsPHUe_v9F9+xc5j36BLon+2QUoK4WBA@mail.gmail.com>
Message-ID: <69B13095-8F01-4952-B53A-80020DE6E0F0@icir.org>


On Feb 21, 2012, at 10:34 AM, Mike Sconzo wrote:

> Another question, is there a way to force (enable) headers to be
> matched in a case sensitive way?

Yeah, this has always bothered me a little bit too.  I'd like to change that for 2.1, we have some other changes that we are already planning on making to the HTTP analyzer for 2.1 anyway and we can work this in there.  (in other words, no, with 2.0 you are stuck with upper-cased header names unless you change the HTTP analyzer in the core and rebuild)

> I've been working on some
> passive identification of browsers so I can ask the question of "what
> browser tells me it's msie via the user-agent string, but doesn't
> behave like it".  

Awesome!  Have you checked out my script to detect browsers by the headers they send and the order they send them?  Feel free to take any code from that script if it helps you with your browser detection.

	https://github.com/sethhall/bro_scripts/blob/master/testing/http-watch-header-order.bro

> With my current implentation I've got about a 72%
> accuracy/detection rate

Cool, which detection techniques you have implemented so far?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From vallentin at icir.org  Tue Feb 21 16:07:48 2012
From: vallentin at icir.org (Matthias Vallentin)
Date: Tue, 21 Feb 2012 16:07:48 -0800
Subject: [Bro] Script Question
In-Reply-To: <CAGZoo6GMcjxODnM6FbTsPHUe_v9F9+xc5j36BLon+2QUoK4WBA@mail.gmail.com>
References: <CAGZoo6Gb52AAYEpcUR9YM=EU1PDGbnVUzHQo4-SNXOGpHmUU+g@mail.gmail.com>
	<C010B9B0-14B0-4E7D-8DBE-D718F74BE365@icir.org>
	<CAGZoo6GMcjxODnM6FbTsPHUe_v9F9+xc5j36BLon+2QUoK4WBA@mail.gmail.com>
Message-ID: <CADTpMNbkg1a1i_vD+LG6JBU8t6WLJ1oHudSRdzbU27MC73dDKw@mail.gmail.com>

> Another question, is there a way to force (enable) headers to be
> matched in a case sensitive way?

Seth answered that question already, but just to be clear: Bro
normalizes the headers to upper case. That is, each header name has one
unique (uppercase) representation at the scripting layer.

> So I figured I'd port it over and learn bro in the process so I could
> give it back to the community.

We're looking forward to it!

    Matthias


From baxterw3232 at gmail.com  Thu Feb 23 12:18:11 2012
From: baxterw3232 at gmail.com (Will)
Date: Thu, 23 Feb 2012 14:18:11 -0600
Subject: [Bro] Bro (SOC N' a Box) fly-away kit ideas
Message-ID: <CANAhm=iBAUJxG8rHW-9AukJABKhMwDXsvqafwf+2-bu-Ow3UiA@mail.gmail.com>

Was wondering if anyone has some recommendations on hardware and
configuration for building  BroNSM fly-away or incident response kits.
Whether this be laptops with multiple NIC's, external HD's, and high
horsepower or mini-tower's that can be pre-built and deployed quickly. In
addition to hardware, I am interested in OS and cluster configuration ideas
that might focus on IR vs. a "log the world" approach. Anyhow, thanks in
advance for any advice or recommendations.

-will
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120223/b638b8de/attachment.html 

From mpilking at gmail.com  Thu Feb 23 14:12:46 2012
From: mpilking at gmail.com (Mike Pilkington)
Date: Thu, 23 Feb 2012 16:12:46 -0600
Subject: [Bro] Bro (SOC N' a Box) fly-away kit ideas
In-Reply-To: <CANAhm=iBAUJxG8rHW-9AukJABKhMwDXsvqafwf+2-bu-Ow3UiA@mail.gmail.com>
References: <CANAhm=iBAUJxG8rHW-9AukJABKhMwDXsvqafwf+2-bu-Ow3UiA@mail.gmail.com>
Message-ID: <CABQd7PNgO=txxNYA1Bz8Dd3a__2cWZ0+NvHZazBg=U2EM0Qz3g@mail.gmail.com>

Will, I did something similar, in a virtual sense.  I needed to have
an ISO image that I could have our staff in the regions setup on a
generic system.  I remastered a Security Onion CD (which includes Bro)
and customized as I needed.  Here are my notes from that little
exercise.  Might be useful...


? The purpose of this exercise is to create a customized Security
Onion Live DVD that will allow me to SSH to it upon boot up of the
DVD.  This allows for emergency remote installs or even short-term
Live DVD network analysis (non-NSM) from a generic PC hardware
platform at a remote location.
? The username created in step 2 below will become the hostname of the
Live DVD.  Don't know why this is, but it's important to keep in mind,
particularly with regard to the next note...
? Security Onion (Xubuntu) supports/particpates in dynamic DNS.  So if
your environment supports it too, when your machine boots, it will be
registered with the *username* (not hostname) you create in step 2
below.  This threw me off at first, but now that you know, you can easily
connect to the remote machine by name (username) if you are using
dynamic DNS.

Steps to custom ISO creation:

1. Installed SO to a new VM
2. Created a temp user with command "sudo adduser <username>"
3. Edit /usr/bin/remastersys shell script and comment out these 4
lines which would delete the SSH keys (if these keys get deleted
during the remastersys process, you won't be able to SSH to the Live
DVD):

#rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_rsa_key
#rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_dsa_key.pub
#rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_dsa_key
#rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_rsa_key.pub

4. Further customize the install as you see fit.  For me, I wanted to
update the firewall to allow access only from my network:

sudo ufw delete allow 80/tcp
sudo ufw delete allow 22/tcp
sudo ufw allow from 10.10.10.0/24 to any
sudo ufw status (to verify configuration)

5. Create the new DVD image with the command "sudo remastersys backup
so-customized.iso".  I used the 'backup' option from remastersys so
that the temp user I created would be left as-is.
6. Test your ISO.  You will find it in /home/remastersys/remastersys.

At this point, you can run as a Live OS or you could install it remotely.
If you install it remotely, I suggest updating the SSH keys.

Hope that helps!
Mike


On 2/23/12, Will <baxterw3232 at gmail.com> wrote:
> Was wondering if anyone has some recommendations on hardware and
> configuration for building  BroNSM fly-away or incident response kits.
> Whether this be laptops with multiple NIC's, external HD's, and high
> horsepower or mini-tower's that can be pre-built and deployed quickly. In
> addition to hardware, I am interested in OS and cluster configuration ideas
> that might focus on IR vs. a "log the world" approach. Anyhow, thanks in
> advance for any advice or recommendations.
>
> -will
>



From basusahay at gmail.com  Wed Feb 22 05:01:41 2012
From: basusahay at gmail.com (Rishi Sahay)
Date: Wed, 22 Feb 2012 18:31:41 +0530
Subject: [Bro] Extraction of IP identification field from tcpdump file
In-Reply-To: <CABmCyhS_RKa_68-Y72tTjLcgBc6unpo7pE1bDUXXSJj51j5k3A@mail.gmail.com>
References: <CABmCyhSC3W6w2cji3M4F+a+RjNML4qDgsC95P-f1080j5y5bkg@mail.gmail.com>
	<CAGnVnBmyJnOGOgcrFp0dv-JUjy6=rRX25Sk6it7cKiw9mHJTOA@mail.gmail.com>
	<CABmCyhS_RKa_68-Y72tTjLcgBc6unpo7pE1bDUXXSJj51j5k3A@mail.gmail.com>
Message-ID: <CABmCyhTw1xKDFy0VKCCr7YHxhhB2f-iKTLteX1mNgL_gg9ynVw@mail.gmail.com>

   Hello,

I want to extract the IP identification field from tcpdump dataset. I have
> attached the script which I have got. I am new to BRO IDS. Please, tell me
> how to extract the IP identification field from tcpdump file using this
> script. Actually, I was extracting the 41 features from the tcpdump
> dataset. Thanks in advance.
>
> --
> Best regards
>   Rishikesh Sahay
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120222/558ec3b2/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: extractfeature.bro
Type: application/octet-stream
Size: 75835 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120222/558ec3b2/attachment.obj 

From vern at icir.org  Sun Feb 26 13:04:26 2012
From: vern at icir.org (Vern Paxson)
Date: Sun, 26 Feb 2012 13:04:26 -0800
Subject: [Bro] Extraction of IP identification field from tcpdump file
In-Reply-To: <CABmCyhTw1xKDFy0VKCCr7YHxhhB2f-iKTLteX1mNgL_gg9ynVw@mail.gmail.com>
	(Wed, 22 Feb 2012 18:31:41 +0530).
Message-ID: <20120226210426.7E59A2C4003@rock.ICSI.Berkeley.EDU>

> I want to extract the IP identification field from tcpdump dataset. I have
> attached the script which I have got.

It's unreasonable to expect us to look through an 1,800 line script for a
problem like this.  You really should trim it down to the simplest script
that expresses the problem you're encountering.

That said, I'll note that the only references to $ip_hdr$id (or even $ip_hdr)
are commented-out, so certainly aren't going to work!

		Vern


From baxterw3232 at gmail.com  Mon Feb 27 09:51:58 2012
From: baxterw3232 at gmail.com (Will)
Date: Mon, 27 Feb 2012 11:51:58 -0600
Subject: [Bro] Bro (SOC N' a Box) fly-away kit ideas
In-Reply-To: <CABQd7PNgO=txxNYA1Bz8Dd3a__2cWZ0+NvHZazBg=U2EM0Qz3g@mail.gmail.com>
References: <CANAhm=iBAUJxG8rHW-9AukJABKhMwDXsvqafwf+2-bu-Ow3UiA@mail.gmail.com>
	<CABQd7PNgO=txxNYA1Bz8Dd3a__2cWZ0+NvHZazBg=U2EM0Qz3g@mail.gmail.com>
Message-ID: <CANAhm=hkWkkf+6xEkTqz-Nfm3MsE8Ewew5TW8TrG1Ai1typVuw@mail.gmail.com>

Mike,

Sorry for the delay. I really appreciate you sharing your notes on
this. Having a live disc makes sense, for both a scalability and an
incident response time perspective.  Looking forward to putting
something similar together!

Thanks!

-will

On Thu, Feb 23, 2012 at 4:12 PM, Mike Pilkington <mpilking at gmail.com> wrote:
> Will, I did something similar, in a virtual sense. ?I needed to have
> an ISO image that I could have our staff in the regions setup on a
> generic system. ?I remastered a Security Onion CD (which includes Bro)
> and customized as I needed. ?Here are my notes from that little
> exercise. ?Might be useful...
>
>
> ? The purpose of this exercise is to create a customized Security
> Onion Live DVD that will allow me to SSH to it upon boot up of the
> DVD. ?This allows for emergency remote installs or even short-term
> Live DVD network analysis (non-NSM) from a generic PC hardware
> platform at a remote location.
> ? The username created in step 2 below will become the hostname of the
> Live DVD. ?Don't know why this is, but it's important to keep in mind,
> particularly with regard to the next note...
> ? Security Onion (Xubuntu) supports/particpates in dynamic DNS. ?So if
> your environment supports it too, when your machine boots, it will be
> registered with the *username* (not hostname) you create in step 2
> below. ?This threw me off at first, but now that you know, you can easily
> connect to the remote machine by name (username) if you are using
> dynamic DNS.
>
> Steps to custom ISO creation:
>
> 1. Installed SO to a new VM
> 2. Created a temp user with command "sudo adduser <username>"
> 3. Edit /usr/bin/remastersys shell script and comment out these 4
> lines which would delete the SSH keys (if these keys get deleted
> during the remastersys process, you won't be able to SSH to the Live
> DVD):
>
> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_rsa_key
> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_dsa_key.pub
> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_dsa_key
> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_rsa_key.pub
>
> 4. Further customize the install as you see fit. ?For me, I wanted to
> update the firewall to allow access only from my network:
>
> sudo ufw delete allow 80/tcp
> sudo ufw delete allow 22/tcp
> sudo ufw allow from 10.10.10.0/24 to any
> sudo ufw status (to verify configuration)
>
> 5. Create the new DVD image with the command "sudo remastersys backup
> so-customized.iso". ?I used the 'backup' option from remastersys so
> that the temp user I created would be left as-is.
> 6. Test your ISO. ?You will find it in /home/remastersys/remastersys.
>
> At this point, you can run as a Live OS or you could install it remotely.
> If you install it remotely, I suggest updating the SSH keys.
>
> Hope that helps!
> Mike
>
>
> On 2/23/12, Will <baxterw3232 at gmail.com> wrote:
>> Was wondering if anyone has some recommendations on hardware and
>> configuration for building ?BroNSM fly-away or incident response kits.
>> Whether this be laptops with multiple NIC's, external HD's, and high
>> horsepower or mini-tower's that can be pre-built and deployed quickly. In
>> addition to hardware, I am interested in OS and cluster configuration ideas
>> that might focus on IR vs. a "log the world" approach. Anyhow, thanks in
>> advance for any advice or recommendations.
>>
>> -will
>>



From doug.burks at gmail.com  Mon Feb 27 10:03:54 2012
From: doug.burks at gmail.com (Doug Burks)
Date: Mon, 27 Feb 2012 13:03:54 -0500
Subject: [Bro] Bro (SOC N' a Box) fly-away kit ideas
In-Reply-To: <CANAhm=hkWkkf+6xEkTqz-Nfm3MsE8Ewew5TW8TrG1Ai1typVuw@mail.gmail.com>
References: <CANAhm=iBAUJxG8rHW-9AukJABKhMwDXsvqafwf+2-bu-Ow3UiA@mail.gmail.com>
	<CABQd7PNgO=txxNYA1Bz8Dd3a__2cWZ0+NvHZazBg=U2EM0Qz3g@mail.gmail.com>
	<CANAhm=hkWkkf+6xEkTqz-Nfm3MsE8Ewew5TW8TrG1Ai1typVuw@mail.gmail.com>
Message-ID: <CAK8kjrATq2Zjy++ETATVk2RqEkfoxxikE8Q78vDLAs2ON1X4JQ@mail.gmail.com>

Hi Will,

If you have questions specific to Security Onion, please join our mailing list:
http://groups.google.com/group/security-onion

Thanks,
Doug

On Mon, Feb 27, 2012 at 12:51 PM, Will <baxterw3232 at gmail.com> wrote:
> Mike,
>
> Sorry for the delay. I really appreciate you sharing your notes on
> this. Having a live disc makes sense, for both a scalability and an
> incident response time perspective. ?Looking forward to putting
> something similar together!
>
> Thanks!
>
> -will
>
> On Thu, Feb 23, 2012 at 4:12 PM, Mike Pilkington <mpilking at gmail.com> wrote:
>> Will, I did something similar, in a virtual sense. ?I needed to have
>> an ISO image that I could have our staff in the regions setup on a
>> generic system. ?I remastered a Security Onion CD (which includes Bro)
>> and customized as I needed. ?Here are my notes from that little
>> exercise. ?Might be useful...
>>
>>
>> ? The purpose of this exercise is to create a customized Security
>> Onion Live DVD that will allow me to SSH to it upon boot up of the
>> DVD. ?This allows for emergency remote installs or even short-term
>> Live DVD network analysis (non-NSM) from a generic PC hardware
>> platform at a remote location.
>> ? The username created in step 2 below will become the hostname of the
>> Live DVD. ?Don't know why this is, but it's important to keep in mind,
>> particularly with regard to the next note...
>> ? Security Onion (Xubuntu) supports/particpates in dynamic DNS. ?So if
>> your environment supports it too, when your machine boots, it will be
>> registered with the *username* (not hostname) you create in step 2
>> below. ?This threw me off at first, but now that you know, you can easily
>> connect to the remote machine by name (username) if you are using
>> dynamic DNS.
>>
>> Steps to custom ISO creation:
>>
>> 1. Installed SO to a new VM
>> 2. Created a temp user with command "sudo adduser <username>"
>> 3. Edit /usr/bin/remastersys shell script and comment out these 4
>> lines which would delete the SSH keys (if these keys get deleted
>> during the remastersys process, you won't be able to SSH to the Live
>> DVD):
>>
>> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_rsa_key
>> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_dsa_key.pub
>> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_dsa_key
>> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_rsa_key.pub
>>
>> 4. Further customize the install as you see fit. ?For me, I wanted to
>> update the firewall to allow access only from my network:
>>
>> sudo ufw delete allow 80/tcp
>> sudo ufw delete allow 22/tcp
>> sudo ufw allow from 10.10.10.0/24 to any
>> sudo ufw status (to verify configuration)
>>
>> 5. Create the new DVD image with the command "sudo remastersys backup
>> so-customized.iso". ?I used the 'backup' option from remastersys so
>> that the temp user I created would be left as-is.
>> 6. Test your ISO. ?You will find it in /home/remastersys/remastersys.
>>
>> At this point, you can run as a Live OS or you could install it remotely.
>> If you install it remotely, I suggest updating the SSH keys.
>>
>> Hope that helps!
>> Mike
>>
>>
>> On 2/23/12, Will <baxterw3232 at gmail.com> wrote:
>>> Was wondering if anyone has some recommendations on hardware and
>>> configuration for building ?BroNSM fly-away or incident response kits.
>>> Whether this be laptops with multiple NIC's, external HD's, and high
>>> horsepower or mini-tower's that can be pre-built and deployed quickly. In
>>> addition to hardware, I am interested in OS and cluster configuration ideas
>>> that might focus on IR vs. a "log the world" approach. Anyhow, thanks in
>>> advance for any advice or recommendations.
>>>
>>> -will
>>>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Doug Burks
Security Onion | http://securityonion.blogspot.com
President, Greater Augusta ISSA | http://augusta.issa.org
SANS Augusta 6/11 - 6/16 | http://www.sans.org/augusta-2012-cs/



From mcholste at gmail.com  Mon Feb 27 10:22:19 2012
From: mcholste at gmail.com (Martin Holste)
Date: Mon, 27 Feb 2012 12:22:19 -0600
Subject: [Bro] Bro (SOC N' a Box) fly-away kit ideas
In-Reply-To: <CAK8kjrATq2Zjy++ETATVk2RqEkfoxxikE8Q78vDLAs2ON1X4JQ@mail.gmail.com>
References: <CANAhm=iBAUJxG8rHW-9AukJABKhMwDXsvqafwf+2-bu-Ow3UiA@mail.gmail.com>
	<CABQd7PNgO=txxNYA1Bz8Dd3a__2cWZ0+NvHZazBg=U2EM0Qz3g@mail.gmail.com>
	<CANAhm=hkWkkf+6xEkTqz-Nfm3MsE8Ewew5TW8TrG1Ai1typVuw@mail.gmail.com>
	<CAK8kjrATq2Zjy++ETATVk2RqEkfoxxikE8Q78vDLAs2ON1X4JQ@mail.gmail.com>
Message-ID: <CANpnLHhcU7LRYtLoZo4NNL6pskdVbqWMCAgDua3QM_1rfPegdA@mail.gmail.com>

One other thing to consider:  If you're dealing with more of a
"scalpel" situation in which you already know a fair amount of IP
information you're looking for, you might consider simple pcap
collection with off-line Bro processing back at HQ via upload or NFS
over VPN.  That's obviously not ideal, but it would be a lot easier to
reuse on-site hardware or very small hardware (even a Cisco NAM which
already has access to SPAN traffic).

On Mon, Feb 27, 2012 at 12:03 PM, Doug Burks <doug.burks at gmail.com> wrote:
> Hi Will,
>
> If you have questions specific to Security Onion, please join our mailing list:
> http://groups.google.com/group/security-onion
>
> Thanks,
> Doug
>
> On Mon, Feb 27, 2012 at 12:51 PM, Will <baxterw3232 at gmail.com> wrote:
>> Mike,
>>
>> Sorry for the delay. I really appreciate you sharing your notes on
>> this. Having a live disc makes sense, for both a scalability and an
>> incident response time perspective. ?Looking forward to putting
>> something similar together!
>>
>> Thanks!
>>
>> -will
>>
>> On Thu, Feb 23, 2012 at 4:12 PM, Mike Pilkington <mpilking at gmail.com> wrote:
>>> Will, I did something similar, in a virtual sense. ?I needed to have
>>> an ISO image that I could have our staff in the regions setup on a
>>> generic system. ?I remastered a Security Onion CD (which includes Bro)
>>> and customized as I needed. ?Here are my notes from that little
>>> exercise. ?Might be useful...
>>>
>>>
>>> ? The purpose of this exercise is to create a customized Security
>>> Onion Live DVD that will allow me to SSH to it upon boot up of the
>>> DVD. ?This allows for emergency remote installs or even short-term
>>> Live DVD network analysis (non-NSM) from a generic PC hardware
>>> platform at a remote location.
>>> ? The username created in step 2 below will become the hostname of the
>>> Live DVD. ?Don't know why this is, but it's important to keep in mind,
>>> particularly with regard to the next note...
>>> ? Security Onion (Xubuntu) supports/particpates in dynamic DNS. ?So if
>>> your environment supports it too, when your machine boots, it will be
>>> registered with the *username* (not hostname) you create in step 2
>>> below. ?This threw me off at first, but now that you know, you can easily
>>> connect to the remote machine by name (username) if you are using
>>> dynamic DNS.
>>>
>>> Steps to custom ISO creation:
>>>
>>> 1. Installed SO to a new VM
>>> 2. Created a temp user with command "sudo adduser <username>"
>>> 3. Edit /usr/bin/remastersys shell script and comment out these 4
>>> lines which would delete the SSH keys (if these keys get deleted
>>> during the remastersys process, you won't be able to SSH to the Live
>>> DVD):
>>>
>>> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_rsa_key
>>> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_dsa_key.pub
>>> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_dsa_key
>>> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_rsa_key.pub
>>>
>>> 4. Further customize the install as you see fit. ?For me, I wanted to
>>> update the firewall to allow access only from my network:
>>>
>>> sudo ufw delete allow 80/tcp
>>> sudo ufw delete allow 22/tcp
>>> sudo ufw allow from 10.10.10.0/24 to any
>>> sudo ufw status (to verify configuration)
>>>
>>> 5. Create the new DVD image with the command "sudo remastersys backup
>>> so-customized.iso". ?I used the 'backup' option from remastersys so
>>> that the temp user I created would be left as-is.
>>> 6. Test your ISO. ?You will find it in /home/remastersys/remastersys.
>>>
>>> At this point, you can run as a Live OS or you could install it remotely.
>>> If you install it remotely, I suggest updating the SSH keys.
>>>
>>> Hope that helps!
>>> Mike
>>>
>>>
>>> On 2/23/12, Will <baxterw3232 at gmail.com> wrote:
>>>> Was wondering if anyone has some recommendations on hardware and
>>>> configuration for building ?BroNSM fly-away or incident response kits.
>>>> Whether this be laptops with multiple NIC's, external HD's, and high
>>>> horsepower or mini-tower's that can be pre-built and deployed quickly. In
>>>> addition to hardware, I am interested in OS and cluster configuration ideas
>>>> that might focus on IR vs. a "log the world" approach. Anyhow, thanks in
>>>> advance for any advice or recommendations.
>>>>
>>>> -will
>>>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> --
> Doug Burks
> Security Onion | http://securityonion.blogspot.com
> President, Greater Augusta ISSA | http://augusta.issa.org
> SANS Augusta 6/11 - 6/16 | http://www.sans.org/augusta-2012-cs/
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From baxterw3232 at gmail.com  Mon Feb 27 11:54:03 2012
From: baxterw3232 at gmail.com (Will)
Date: Mon, 27 Feb 2012 13:54:03 -0600
Subject: [Bro] Bro (SOC N' a Box) fly-away kit ideas
In-Reply-To: <CANpnLHhcU7LRYtLoZo4NNL6pskdVbqWMCAgDua3QM_1rfPegdA@mail.gmail.com>
References: <CANAhm=iBAUJxG8rHW-9AukJABKhMwDXsvqafwf+2-bu-Ow3UiA@mail.gmail.com>
	<CABQd7PNgO=txxNYA1Bz8Dd3a__2cWZ0+NvHZazBg=U2EM0Qz3g@mail.gmail.com>
	<CANAhm=hkWkkf+6xEkTqz-Nfm3MsE8Ewew5TW8TrG1Ai1typVuw@mail.gmail.com>
	<CAK8kjrATq2Zjy++ETATVk2RqEkfoxxikE8Q78vDLAs2ON1X4JQ@mail.gmail.com>
	<CANpnLHhcU7LRYtLoZo4NNL6pskdVbqWMCAgDua3QM_1rfPegdA@mail.gmail.com>
Message-ID: <CANAhm=ggLzCOPtHs=-rN4sb7E0C99YS7rHmOW8cm5CtYKc_Ytw@mail.gmail.com>

Good point. I've considered using dumpcap or something similar to
archive and batch traffic back to a central monitoring location where
it could be analyzed any number of ways. Over saturation of a WAN or
MPLS link was the primary concern with back-hauling traffic, but if
broken into small enough jobs combined with compression, I think it
would be manageable. Thanks for the feedback!

On Mon, Feb 27, 2012 at 12:22 PM, Martin Holste <mcholste at gmail.com> wrote:
> One other thing to consider: ?If you're dealing with more of a
> "scalpel" situation in which you already know a fair amount of IP
> information you're looking for, you might consider simple pcap
> collection with off-line Bro processing back at HQ via upload or NFS
> over VPN. ?That's obviously not ideal, but it would be a lot easier to
> reuse on-site hardware or very small hardware (even a Cisco NAM which
> already has access to SPAN traffic).
>
> On Mon, Feb 27, 2012 at 12:03 PM, Doug Burks <doug.burks at gmail.com> wrote:
>> Hi Will,
>>
>> If you have questions specific to Security Onion, please join our mailing list:
>> http://groups.google.com/group/security-onion
>>
>> Thanks,
>> Doug
>>
>> On Mon, Feb 27, 2012 at 12:51 PM, Will <baxterw3232 at gmail.com> wrote:
>>> Mike,
>>>
>>> Sorry for the delay. I really appreciate you sharing your notes on
>>> this. Having a live disc makes sense, for both a scalability and an
>>> incident response time perspective. ?Looking forward to putting
>>> something similar together!
>>>
>>> Thanks!
>>>
>>> -will
>>>
>>> On Thu, Feb 23, 2012 at 4:12 PM, Mike Pilkington <mpilking at gmail.com> wrote:
>>>> Will, I did something similar, in a virtual sense. ?I needed to have
>>>> an ISO image that I could have our staff in the regions setup on a
>>>> generic system. ?I remastered a Security Onion CD (which includes Bro)
>>>> and customized as I needed. ?Here are my notes from that little
>>>> exercise. ?Might be useful...
>>>>
>>>>
>>>> ? The purpose of this exercise is to create a customized Security
>>>> Onion Live DVD that will allow me to SSH to it upon boot up of the
>>>> DVD. ?This allows for emergency remote installs or even short-term
>>>> Live DVD network analysis (non-NSM) from a generic PC hardware
>>>> platform at a remote location.
>>>> ? The username created in step 2 below will become the hostname of the
>>>> Live DVD. ?Don't know why this is, but it's important to keep in mind,
>>>> particularly with regard to the next note...
>>>> ? Security Onion (Xubuntu) supports/particpates in dynamic DNS. ?So if
>>>> your environment supports it too, when your machine boots, it will be
>>>> registered with the *username* (not hostname) you create in step 2
>>>> below. ?This threw me off at first, but now that you know, you can easily
>>>> connect to the remote machine by name (username) if you are using
>>>> dynamic DNS.
>>>>
>>>> Steps to custom ISO creation:
>>>>
>>>> 1. Installed SO to a new VM
>>>> 2. Created a temp user with command "sudo adduser <username>"
>>>> 3. Edit /usr/bin/remastersys shell script and comment out these 4
>>>> lines which would delete the SSH keys (if these keys get deleted
>>>> during the remastersys process, you won't be able to SSH to the Live
>>>> DVD):
>>>>
>>>> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_rsa_key
>>>> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_dsa_key.pub
>>>> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_dsa_key
>>>> #rm -rf $WORKDIR/dummysys/etc/ssh/ssh_host_rsa_key.pub
>>>>
>>>> 4. Further customize the install as you see fit. ?For me, I wanted to
>>>> update the firewall to allow access only from my network:
>>>>
>>>> sudo ufw delete allow 80/tcp
>>>> sudo ufw delete allow 22/tcp
>>>> sudo ufw allow from 10.10.10.0/24 to any
>>>> sudo ufw status (to verify configuration)
>>>>
>>>> 5. Create the new DVD image with the command "sudo remastersys backup
>>>> so-customized.iso". ?I used the 'backup' option from remastersys so
>>>> that the temp user I created would be left as-is.
>>>> 6. Test your ISO. ?You will find it in /home/remastersys/remastersys.
>>>>
>>>> At this point, you can run as a Live OS or you could install it remotely.
>>>> If you install it remotely, I suggest updating the SSH keys.
>>>>
>>>> Hope that helps!
>>>> Mike
>>>>
>>>>
>>>> On 2/23/12, Will <baxterw3232 at gmail.com> wrote:
>>>>> Was wondering if anyone has some recommendations on hardware and
>>>>> configuration for building ?BroNSM fly-away or incident response kits.
>>>>> Whether this be laptops with multiple NIC's, external HD's, and high
>>>>> horsepower or mini-tower's that can be pre-built and deployed quickly. In
>>>>> addition to hardware, I am interested in OS and cluster configuration ideas
>>>>> that might focus on IR vs. a "log the world" approach. Anyhow, thanks in
>>>>> advance for any advice or recommendations.
>>>>>
>>>>> -will
>>>>>
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>>
>> --
>> Doug Burks
>> Security Onion | http://securityonion.blogspot.com
>> President, Greater Augusta ISSA | http://augusta.issa.org
>> SANS Augusta 6/11 - 6/16 | http://www.sans.org/augusta-2012-cs/
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From nweaver at ICSI.Berkeley.EDU  Mon Feb 27 11:58:03 2012
From: nweaver at ICSI.Berkeley.EDU (Nicholas Weaver)
Date: Mon, 27 Feb 2012 11:58:03 -0800
Subject: [Bro] Bro (SOC N' a Box) fly-away kit ideas
In-Reply-To: <CANAhm=hkWkkf+6xEkTqz-Nfm3MsE8Ewew5TW8TrG1Ai1typVuw@mail.gmail.com>
References: <CANAhm=iBAUJxG8rHW-9AukJABKhMwDXsvqafwf+2-bu-Ow3UiA@mail.gmail.com>
	<CABQd7PNgO=txxNYA1Bz8Dd3a__2cWZ0+NvHZazBg=U2EM0Qz3g@mail.gmail.com>
	<CANAhm=hkWkkf+6xEkTqz-Nfm3MsE8Ewew5TW8TrG1Ai1typVuw@mail.gmail.com>
Message-ID: <161BA984-7708-4A7D-9382-A1634097FBE3@icsi.berkeley.edu>


Another thing which may be useful on the hardware side for this kit:

http://www.dual-comm.com/gigabit_port-mirroring-LAN_switch.htm

These are small, cheap (<$200), USB powered Gigabit switches with a built in mirror: all traffic to/from Port 1 is mirrored to port 4.

I purchased one of these and put it inline in my house as an IDS monitor:  I don't know at which point it starts dropping packets (I didn't test that), but I was able to get it up and running trivially.

This is specifically designed for "incident response" type tapping:  You have a notebook and a network you want to monitor, so you just splice this in with a short ethernet cable.