From greencm at gmail.com  Sun Jun  3 09:59:39 2012
From: greencm at gmail.com (Chris Green)
Date: Sun, 3 Jun 2012 11:59:39 -0500
Subject: [Bro] Debugging Policies and Print Statement
In-Reply-To: <20120531005039.GM10373@icir.org>
References: <CAJUMos5PtzQrLG+MSfgY3ivxhjpD7RBfb_GYOXR4+5k7pLGqGA@mail.gmail.com>
	<8998D311-561E-46C9-AC96-10D2D1721DA8@illinois.edu>
	<20120531005039.GM10373@icir.org>
Message-ID: <CAJUMos7CMuefAUE_bapA831o38PnJXPA-yj_81q3n1W2_19ikA@mail.gmail.com>

On Wed, May 30, 2012 at 7:50 PM, Robin Sommer <robin at icir.org> wrote:

>
> I've merged this into git master now.
>

Thanks all;  things seem more predictable now.  Especially helped figuring
out c$$http$mime_type is sniffed and not read from the HTTP header.

-- 
Chris Green <greencm at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120603/d6fba922/attachment.html 

From robin at icir.org  Mon Jun 11 08:44:30 2012
From: robin at icir.org (Robin Sommer)
Date: Mon, 11 Jun 2012 08:44:30 -0700
Subject: [Bro] Bro Exchange 2012
Message-ID: <20120611154430.GG86209@icir.org>

This year we will hold our first "Bro Exchange": A Bro users meeting
aiming to get a large number Bro users together in the same room to
exchange thoughts and talk about how everybody?s using Bro. In
contrast to past workshops, the focus of this event is less on
providing training and more on exchanging experiences and ideas.

The Bro Exchange 2012 will take place on August 7-8, 2012, at the
National Center for Atmospheric Research (NCAR) in Boulder, Colorado.

Registration will open soon. More information here:

    http://www.bro-ids.org/community/exchange2012.html

Hope to see many of you guys in Boulder,

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From roger.larsen at hig.no  Wed Jun 13 00:56:19 2012
From: roger.larsen at hig.no (=?iso-8859-1?Q?Roger_Larsen_-_H=F8gskolen_i_Gj=F8vik?=)
Date: Wed, 13 Jun 2012 09:56:19 +0200
Subject: [Bro] Are there any machine learning functionality in Bro?
Message-ID: <000901cd493a$04b4c8e0$0e1e5aa0$@hig.no>

Dear Bro maillist members,

 

I am doing a school project on Bro capabilities.

I still have not found any ?machine learning? functionality.

The most interesting so far is;

Radix or Patricia Tree algorithm

The Smith-Waterman algorithm and

Threshold Random Walk algorithm (TRW).

 

Are the learning capabilities in Bro well hidden ? or missing?

 

I really appreciate any feedback on this question !

 

 

Thanks,

 

Roger Larsen

Master student in information security

Gj?vik University College ? NORWAY

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120613/b240ec40/attachment.html 

From robin at icir.org  Wed Jun 13 07:45:01 2012
From: robin at icir.org (Robin Sommer)
Date: Wed, 13 Jun 2012 07:45:01 -0700
Subject: [Bro] Are there any machine learning functionality in Bro?
In-Reply-To: <000901cd493a$04b4c8e0$0e1e5aa0$@hig.no>
References: <000901cd493a$04b4c8e0$0e1e5aa0$@hig.no>
Message-ID: <20120613144500.GC91508@icir.org>


On Wed, Jun 13, 2012 at 09:56 +0200, Roger Larsen - H???gskolen i Gj???vik wrote:

> Threshold Random Walk algorithm (TRW).

(TRW doesn't use machine learning.)

> Are the learning capabilities in Bro well hidden ? or missing?

Missing. Not by design, but out of operational concerns: it's
extremely hard to get ML approaches to work reliably. You may be
interested in this paper we wrote a little while ago on this topic:

    http://www.icir.org/robin/papers/oakland10-ml.pdf

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From mcholste at gmail.com  Wed Jun 13 07:56:45 2012
From: mcholste at gmail.com (Martin Holste)
Date: Wed, 13 Jun 2012 09:56:45 -0500
Subject: [Bro] Are there any machine learning functionality in Bro?
In-Reply-To: <20120613144500.GC91508@icir.org>
References: <000901cd493a$04b4c8e0$0e1e5aa0$@hig.no>
	<20120613144500.GC91508@icir.org>
Message-ID: <CANpnLHg3q2Zqa7jZUcSnBsUsxWKym7+8zfk=N=dCvJyu=f1Mfw@mail.gmail.com>

You know a project is awesome when a response indicating that a
feature isn't available is accompanied by a full paper on it.

On Wed, Jun 13, 2012 at 9:45 AM, Robin Sommer <robin at icir.org> wrote:
>
> On Wed, Jun 13, 2012 at 09:56 +0200, Roger Larsen - H?gskolen i Gj?vik wrote:
>
>> Threshold Random Walk algorithm (TRW).
>
> (TRW doesn't use machine learning.)
>
>> Are the learning capabilities in Bro well hidden ? or missing?
>
> Missing. Not by design, but out of operational concerns: it's
> extremely hard to get ML approaches to work reliably. You may be
> interested in this paper we wrote a little while ago on this topic:
>
> ? ?http://www.icir.org/robin/papers/oakland10-ml.pdf
>
> Robin
>
> --
> Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
> ICSI/LBNL ? ?* Fax ? +1 (510) 666-2956 * ? www.icir.org
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From hlin33 at illinois.edu  Mon Jun 18 07:35:27 2012
From: hlin33 at illinois.edu (Hui Lin (Hugo) )
Date: Mon, 18 Jun 2012 09:35:27 -0500
Subject: [Bro]  Hui Lin_SSH Analyzer
Message-ID: <CAKq214nBD0SvxwJ42e+1sMet-6PsJDnoiDmuNUOnDZ4UNcm_vA@mail.gmail.com>

Hi,

In my experiment, I need to use SSH analyzer simply to record a successful
log in. I find that Bro comes with events, heuristic_successful_login,
heuristic_failed_login, in policy file /share/bro/base/protocol/main.bro.

When I test these two events with the default implementation, I find that
the log file always record a failed ssh log in to the system even if I log
in correctly by user/authentication. I want to check when these two events
are called, but I could not find ssh analyzer binpac code.

so I am wondering, how can I correctly record the ssh log in with
user/password authentication and with the user name logged in plain text.

Best,

Hui
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120618/8c7944c5/attachment.html 

From seth at icir.org  Mon Jun 18 08:29:39 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 18 Jun 2012 11:29:39 -0400
Subject: [Bro] Hui Lin_SSH Analyzer
In-Reply-To: <CAKq214nBD0SvxwJ42e+1sMet-6PsJDnoiDmuNUOnDZ4UNcm_vA@mail.gmail.com>
References: <CAKq214nBD0SvxwJ42e+1sMet-6PsJDnoiDmuNUOnDZ4UNcm_vA@mail.gmail.com>
Message-ID: <A92C9E8F-E7D0-4C23-9E65-D45429BB5012@icir.org>


On Jun 18, 2012, at 10:35 AM, Hui Lin (Hugo) wrote:

> When I test these two events with the default implementation, I find that the log file always record a failed ssh log in to the system even if I log in correctly by user/authentication. I want to check when these two events are called, but I could not find ssh analyzer binpac code. 

Those are script-land events.  Currently all events generated by core code (typically the analyzers) are defined in events.bif. You can see in the SSH scripts where those events are generated.

The reason you're seeing a false positive is because the SSH successful login code uses a heuristic to guess if the login was successful or not and sometimes it's wrong.

> so I am wondering, how can I correctly record the ssh log in with user/password authentication and with the user name logged in plain text. 

That information is encrypted in SSH.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From hlin33 at illinois.edu  Mon Jun 18 09:25:03 2012
From: hlin33 at illinois.edu (Hui Lin (Hugo) )
Date: Mon, 18 Jun 2012 11:25:03 -0500
Subject: [Bro] Hui Lin_SSH Analyzer
In-Reply-To: <30c89f863f954f5d93b95a019ce39572@CITESHT3.ad.uillinois.edu>
References: <CAKq214nBD0SvxwJ42e+1sMet-6PsJDnoiDmuNUOnDZ4UNcm_vA@mail.gmail.com>
	<30c89f863f954f5d93b95a019ce39572@CITESHT3.ad.uillinois.edu>
Message-ID: <CAKq214mDxbGXm5T8GGye+bk9AATee6TkMJcKNcWb2ki0xEaR0Q@mail.gmail.com>

On Mon, Jun 18, 2012 at 10:29 AM, Seth Hall <seth at icir.org> wrote:

>
> On Jun 18, 2012, at 10:35 AM, Hui Lin (Hugo) wrote:
>
> > When I test these two events with the default implementation, I find
> that the log file always record a failed ssh log in to the system even if I
> log in correctly by user/authentication. I want to check when these two
> events are called, but I could not find ssh analyzer binpac code.
>
> Those are script-land events.  Currently all events generated by core code
> (typically the analyzers) are defined in events.bif. You can see in the SSH
> scripts where those events are generated.
>

It seems that these two events are included in event.bif.bro any more.


>
> The reason you're seeing a false positive is because the SSH successful
> login code uses a heuristic to guess if the login was successful or not and
> sometimes it's wrong




> > so I am wondering, how can I correctly record the ssh log in with
> user/password authentication and with the user name logged in plain text.
>
> That information is encrypted in SSH.
>

I see.

I accidentally find that there is also syslog policy in Bro. I know that
SSH login to the host machine will be logged in auth.log. I am wondering
whether Bro can log the SSH login through the syslog policy. At least, I am
not successful in my test.

>
>  .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120618/9ce00334/attachment.html 

From seth at icir.org  Mon Jun 18 09:44:12 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 18 Jun 2012 12:44:12 -0400
Subject: [Bro] Hui Lin_SSH Analyzer
In-Reply-To: <CAKq214mDxbGXm5T8GGye+bk9AATee6TkMJcKNcWb2ki0xEaR0Q@mail.gmail.com>
References: <CAKq214nBD0SvxwJ42e+1sMet-6PsJDnoiDmuNUOnDZ4UNcm_vA@mail.gmail.com>
	<30c89f863f954f5d93b95a019ce39572@CITESHT3.ad.uillinois.edu>
	<CAKq214mDxbGXm5T8GGye+bk9AATee6TkMJcKNcWb2ki0xEaR0Q@mail.gmail.com>
Message-ID: <BE2C111C-0F2D-474F-AF36-48E4EAD623A8@icir.org>


On Jun 18, 2012, at 12:25 PM, Hui Lin (Hugo) wrote:

> It seems that these two events are included in event.bif.bro any more.

They never were included in that file since they aren't events from the core.

> I accidentally find that there is also syslog policy in Bro. I know that SSH login to the host machine will be logged in auth.log. I am wondering whether Bro can log the SSH login through the syslog policy. At least, I am not successful in my test. 

That's for analyzing the syslog protocol, you just have to make sure that the host sniffing traffic would see the syslog traffic or you could use the input framework from the upcoming Bro 2.1 (it's in the the master branch already) to read the log file in directly if it's on some host in your cluster.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From will.havlovick at zenimax.com  Mon Jun 18 12:24:52 2012
From: will.havlovick at zenimax.com (Will Havlovick)
Date: Mon, 18 Jun 2012 15:24:52 -0400
Subject: [Bro] Dropped Packets
In-Reply-To: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA0E6@usrkvexch01.zenimax.com>
References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA0E6@usrkvexch01.zenimax.com>
Message-ID: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA624@usrkvexch01.zenimax.com>

Update:

I have found a way to lessen the amount of packets being dropped.

Here is what I have:
Dell r310 - 3.2Ghz - 4GB RAM - Dell hardware RAID controller - two 1TB 7.2k drives in a RAID 1

Test scenario:
Two bro2.0 servers running virtually identical configs with Ubuntu 11.10.
One server for testing and one as a control.
Both monitoring 2 Network Taps of live traffic.

Test 1 : increased RAM to 8GB 
Result : same amount of packets dropped

Test 2 : replaced hard drives with 2 10k drives in a RAID 1
Result : 10% less packet drops  in bro logs as compared to the control server

Test 3 : replaced hard drives with 2 SSD drives in a RAID 1
Result :  80% less packet drops then the control server

Test 4 : switched SSD hard drives to a RAID 0
Result | 90% less packet drops then the control server

I have heard that SSD drives have a shorter life span if it is written to a lot.  So this is probably not the best solution.

But, from now on I will order servers with the fastest possible hard drives which for the Dell r310 are 15K SAS drives.  

When I get the 15K SAS drives in I will run the same tests and put the results out.


Will

-----Original Message-----
From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf Of Will Havlovick
Sent: Thursday, January 12, 2012 2:00 PM
To: 'bro at bro-ids.org'
Subject: [Bro] Dropped Packets

Hi all,

I recently upgraded 3 standalone Bro nodes.  2 of them are Ubuntu and one of them is CentOS 6.2.

On the 2 Ubuntu 11.10 boxes I have a lot of dropped packets in the notice.log
---
PacketFilter::Dropped_Packets   476 packets dropped after filtering, 52258 received, 52258 on link      
PacketFilter::Dropped_Packets   4914 packets dropped after filtering, 52785 received, 52785 on link     
PacketFilter::Dropped_Packets   3061 packets dropped after filtering, 35701 received, 35702 on link     
PacketFilter::Dropped_Packets   3371 packets dropped after filtering, 30573 received, 30591 on link     
---
broctl netstats
       bro: 1326394056.309957 recvd=958721774 dropped=67351350 link=1026073125

I then tried to add this line to the broctl.cfg from http://comments.gmane.org/gmane.comp.security.detection.bro/4146
broargs = -l 9800

Which does not appear to be part of the final release and did not work.

The CentOS box is dropping packets, but not the amounts that the 2 Ubuntu boxes are.  

Is there a way to reduce the amount of dropped packets?

Also, I can provide more data if necessary.

Thank you in advance,


Will

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From mcholste at gmail.com  Mon Jun 18 12:37:19 2012
From: mcholste at gmail.com (Martin Holste)
Date: Mon, 18 Jun 2012 14:37:19 -0500
Subject: [Bro] Dropped Packets
In-Reply-To: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA624@usrkvexch01.zenimax.com>
References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA0E6@usrkvexch01.zenimax.com>
	<6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA624@usrkvexch01.zenimax.com>
Message-ID: <CANpnLHghVYqvt9vyBx2kqNiOS-zuj74XfcFEs7EjHyzA6ug07g@mail.gmail.com>

That's really interesting!  What about using a ramdisk (e.g. /dev/shm)
file system for logs being currently written to, then at the hour mark
(when the logs rollover), putting them on disk?  That should
theoretically take disk performance out of the equation, and I'd be
really interested in your numbers then.

On Mon, Jun 18, 2012 at 2:24 PM, Will Havlovick
<will.havlovick at zenimax.com> wrote:
> Update:
>
> I have found a way to lessen the amount of packets being dropped.
>
> Here is what I have:
> Dell r310 - 3.2Ghz - 4GB RAM - Dell hardware RAID controller - two 1TB 7.2k drives in a RAID 1
>
> Test scenario:
> Two bro2.0 servers running virtually identical configs with Ubuntu 11.10.
> One server for testing and one as a control.
> Both monitoring 2 Network Taps of live traffic.
>
> Test 1 : increased RAM to 8GB
> Result : same amount of packets dropped
>
> Test 2 : replaced hard drives with 2 10k drives in a RAID 1
> Result : 10% less packet drops ?in bro logs as compared to the control server
>
> Test 3 : replaced hard drives with 2 SSD drives in a RAID 1
> Result : ?80% less packet drops then the control server
>
> Test 4 : switched SSD hard drives to a RAID 0
> Result | 90% less packet drops then the control server
>
> I have heard that SSD drives have a shorter life span if it is written to a lot. ?So this is probably not the best solution.
>
> But, from now on I will order servers with the fastest possible hard drives which for the Dell r310 are 15K SAS drives.
>
> When I get the 15K SAS drives in I will run the same tests and put the results out.
>
>
> Will
>
> -----Original Message-----
> From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf Of Will Havlovick
> Sent: Thursday, January 12, 2012 2:00 PM
> To: 'bro at bro-ids.org'
> Subject: [Bro] Dropped Packets
>
> Hi all,
>
> I recently upgraded 3 standalone Bro nodes. ?2 of them are Ubuntu and one of them is CentOS 6.2.
>
> On the 2 Ubuntu 11.10 boxes I have a lot of dropped packets in the notice.log
> ---
> PacketFilter::Dropped_Packets ? 476 packets dropped after filtering, 52258 received, 52258 on link
> PacketFilter::Dropped_Packets ? 4914 packets dropped after filtering, 52785 received, 52785 on link
> PacketFilter::Dropped_Packets ? 3061 packets dropped after filtering, 35701 received, 35702 on link
> PacketFilter::Dropped_Packets ? 3371 packets dropped after filtering, 30573 received, 30591 on link
> ---
> broctl netstats
> ? ? ? bro: 1326394056.309957 recvd=958721774 dropped=67351350 link=1026073125
>
> I then tried to add this line to the broctl.cfg from http://comments.gmane.org/gmane.comp.security.detection.bro/4146
> broargs = -l 9800
>
> Which does not appear to be part of the final release and did not work.
>
> The CentOS box is dropping packets, but not the amounts that the 2 Ubuntu boxes are.
>
> Is there a way to reduce the amount of dropped packets?
>
> Also, I can provide more data if necessary.
>
> Thank you in advance,
>
>
> Will
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From jthoel at gmail.com  Mon Jun 18 12:47:44 2012
From: jthoel at gmail.com (Jeremy Hoel)
Date: Mon, 18 Jun 2012 19:47:44 +0000
Subject: [Bro] Dropped Packets
In-Reply-To: <CANpnLHghVYqvt9vyBx2kqNiOS-zuj74XfcFEs7EjHyzA6ug07g@mail.gmail.com>
References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA0E6@usrkvexch01.zenimax.com>
	<6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA624@usrkvexch01.zenimax.com>
	<CANpnLHghVYqvt9vyBx2kqNiOS-zuj74XfcFEs7EjHyzA6ug07g@mail.gmail.com>
Message-ID: <CAH_p-VN=igCah-pK1xsvK352SM3vA7AuRuiPDfVSMsn-erYFVA@mail.gmail.com>

What was the network speeds you had been seeing during these tests?

On Mon, Jun 18, 2012 at 7:37 PM, Martin Holste <mcholste at gmail.com> wrote:
> That's really interesting! ?What about using a ramdisk (e.g. /dev/shm)
> file system for logs being currently written to, then at the hour mark
> (when the logs rollover), putting them on disk? ?That should
> theoretically take disk performance out of the equation, and I'd be
> really interested in your numbers then.
>
> On Mon, Jun 18, 2012 at 2:24 PM, Will Havlovick
> <will.havlovick at zenimax.com> wrote:
>> Update:
>>
>> I have found a way to lessen the amount of packets being dropped.
>>
>> Here is what I have:
>> Dell r310 - 3.2Ghz - 4GB RAM - Dell hardware RAID controller - two 1TB 7.2k drives in a RAID 1
>>
>> Test scenario:
>> Two bro2.0 servers running virtually identical configs with Ubuntu 11.10.
>> One server for testing and one as a control.
>> Both monitoring 2 Network Taps of live traffic.
>>
>> Test 1 : increased RAM to 8GB
>> Result : same amount of packets dropped
>>
>> Test 2 : replaced hard drives with 2 10k drives in a RAID 1
>> Result : 10% less packet drops ?in bro logs as compared to the control server
>>
>> Test 3 : replaced hard drives with 2 SSD drives in a RAID 1
>> Result : ?80% less packet drops then the control server
>>
>> Test 4 : switched SSD hard drives to a RAID 0
>> Result | 90% less packet drops then the control server
>>
>> I have heard that SSD drives have a shorter life span if it is written to a lot. ?So this is probably not the best solution.
>>
>> But, from now on I will order servers with the fastest possible hard drives which for the Dell r310 are 15K SAS drives.
>>
>> When I get the 15K SAS drives in I will run the same tests and put the results out.
>>
>>
>> Will
>>
>> -----Original Message-----
>> From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf Of Will Havlovick
>> Sent: Thursday, January 12, 2012 2:00 PM
>> To: 'bro at bro-ids.org'
>> Subject: [Bro] Dropped Packets
>>
>> Hi all,
>>
>> I recently upgraded 3 standalone Bro nodes. ?2 of them are Ubuntu and one of them is CentOS 6.2.
>>
>> On the 2 Ubuntu 11.10 boxes I have a lot of dropped packets in the notice.log
>> ---
>> PacketFilter::Dropped_Packets ? 476 packets dropped after filtering, 52258 received, 52258 on link
>> PacketFilter::Dropped_Packets ? 4914 packets dropped after filtering, 52785 received, 52785 on link
>> PacketFilter::Dropped_Packets ? 3061 packets dropped after filtering, 35701 received, 35702 on link
>> PacketFilter::Dropped_Packets ? 3371 packets dropped after filtering, 30573 received, 30591 on link
>> ---
>> broctl netstats
>> ? ? ? bro: 1326394056.309957 recvd=958721774 dropped=67351350 link=1026073125
>>
>> I then tried to add this line to the broctl.cfg from http://comments.gmane.org/gmane.comp.security.detection.bro/4146
>> broargs = -l 9800
>>
>> Which does not appear to be part of the final release and did not work.
>>
>> The CentOS box is dropping packets, but not the amounts that the 2 Ubuntu boxes are.
>>
>> Is there a way to reduce the amount of dropped packets?
>>
>> Also, I can provide more data if necessary.
>>
>> Thank you in advance,
>>
>>
>> Will
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From will.havlovick at zenimax.com  Mon Jun 18 13:01:59 2012
From: will.havlovick at zenimax.com (Will Havlovick)
Date: Mon, 18 Jun 2012 16:01:59 -0400
Subject: [Bro] Dropped Packets
In-Reply-To: <CAH_p-VN=igCah-pK1xsvK352SM3vA7AuRuiPDfVSMsn-erYFVA@mail.gmail.com>
References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA0E6@usrkvexch01.zenimax.com>
	<6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA624@usrkvexch01.zenimax.com>
	<CANpnLHghVYqvt9vyBx2kqNiOS-zuj74XfcFEs7EjHyzA6ug07g@mail.gmail.com>
	<CAH_p-VN=igCah-pK1xsvK352SM3vA7AuRuiPDfVSMsn-erYFVA@mail.gmail.com>
Message-ID: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA626@usrkvexch01.zenimax.com>

Per broctl>capstats,  I was averaging between 15-45mbps.



-----Original Message-----
From: Jeremy Hoel [mailto:jthoel at gmail.com] 
Sent: Monday, June 18, 2012 3:48 PM
To: Martin Holste
Cc: Will Havlovick; bro at bro-ids.org
Subject: Re: [Bro] Dropped Packets

What was the network speeds you had been seeing during these tests?

On Mon, Jun 18, 2012 at 7:37 PM, Martin Holste <mcholste at gmail.com> wrote:
> That's really interesting! ?What about using a ramdisk (e.g. /dev/shm) 
> file system for logs being currently written to, then at the hour mark 
> (when the logs rollover), putting them on disk? ?That should 
> theoretically take disk performance out of the equation, and I'd be 
> really interested in your numbers then.
>
> On Mon, Jun 18, 2012 at 2:24 PM, Will Havlovick 
> <will.havlovick at zenimax.com> wrote:
>> Update:
>>
>> I have found a way to lessen the amount of packets being dropped.
>>
>> Here is what I have:
>> Dell r310 - 3.2Ghz - 4GB RAM - Dell hardware RAID controller - two 
>> 1TB 7.2k drives in a RAID 1
>>
>> Test scenario:
>> Two bro2.0 servers running virtually identical configs with Ubuntu 11.10.
>> One server for testing and one as a control.
>> Both monitoring 2 Network Taps of live traffic.
>>
>> Test 1 : increased RAM to 8GB
>> Result : same amount of packets dropped
>>
>> Test 2 : replaced hard drives with 2 10k drives in a RAID 1 Result : 
>> 10% less packet drops ?in bro logs as compared to the control server
>>
>> Test 3 : replaced hard drives with 2 SSD drives in a RAID 1 Result : ?
>> 80% less packet drops then the control server
>>
>> Test 4 : switched SSD hard drives to a RAID 0 Result | 90% less 
>> packet drops then the control server
>>
>> I have heard that SSD drives have a shorter life span if it is written to a lot. ?So this is probably not the best solution.
>>
>> But, from now on I will order servers with the fastest possible hard drives which for the Dell r310 are 15K SAS drives.
>>
>> When I get the 15K SAS drives in I will run the same tests and put the results out.
>>
>>
>> Will
>>
>> -----Original Message-----
>> From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On 
>> Behalf Of Will Havlovick
>> Sent: Thursday, January 12, 2012 2:00 PM
>> To: 'bro at bro-ids.org'
>> Subject: [Bro] Dropped Packets
>>
>> Hi all,
>>
>> I recently upgraded 3 standalone Bro nodes. ?2 of them are Ubuntu and one of them is CentOS 6.2.
>>
>> On the 2 Ubuntu 11.10 boxes I have a lot of dropped packets in the 
>> notice.log
>> ---
>> PacketFilter::Dropped_Packets ? 476 packets dropped after filtering, 
>> 52258 received, 52258 on link PacketFilter::Dropped_Packets ? 4914 
>> packets dropped after filtering, 52785 received, 52785 on link 
>> PacketFilter::Dropped_Packets ? 3061 packets dropped after filtering, 
>> 35701 received, 35702 on link PacketFilter::Dropped_Packets ? 3371 
>> packets dropped after filtering, 30573 received, 30591 on link
>> ---
>> broctl netstats
>> ? ? ? bro: 1326394056.309957 recvd=958721774 dropped=67351350 
>> link=1026073125
>>
>> I then tried to add this line to the broctl.cfg from 
>> http://comments.gmane.org/gmane.comp.security.detection.bro/4146
>> broargs = -l 9800
>>
>> Which does not appear to be part of the final release and did not work.
>>
>> The CentOS box is dropping packets, but not the amounts that the 2 Ubuntu boxes are.
>>
>> Is there a way to reduce the amount of dropped packets?
>>
>> Also, I can provide more data if necessary.
>>
>> Thank you in advance,
>>
>>
>> Will
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at icir.org  Mon Jun 18 13:22:12 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 18 Jun 2012 16:22:12 -0400
Subject: [Bro] Dropped Packets
In-Reply-To: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA626@usrkvexch01.zenimax.com>
References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA0E6@usrkvexch01.zenimax.com>
	<6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA624@usrkvexch01.zenimax.com>
	<CANpnLHghVYqvt9vyBx2kqNiOS-zuj74XfcFEs7EjHyzA6ug07g@mail.gmail.com>
	<CAH_p-VN=igCah-pK1xsvK352SM3vA7AuRuiPDfVSMsn-erYFVA@mail.gmail.com>
	<6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA626@usrkvexch01.zenimax.com>
Message-ID: <DDA7F049-E5BE-4E98-B47F-9CBAEFD73FC7@icir.org>


On Jun 18, 2012, at 4:01 PM, Will Havlovick wrote:

> Per broctl>capstats,  I was averaging between 15-45mbps.


Hm, that seems like an oddly low amount of traffic to see drops on a box like you have.

Could you also try the current master branch in our repository?  The logging framework has been threaded and it's likely that disk latency issues have been resolved to some degree.  Also, I assume these were running Bro in standalone mode?  When running in a cluster this shouldn't have nearly so much effect because the manager process does all of the log writing and it doesn't do any packet processing.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From JAzoff at albany.edu  Mon Jun 18 13:22:42 2012
From: JAzoff at albany.edu (Justin Azoff)
Date: Mon, 18 Jun 2012 16:22:42 -0400
Subject: [Bro] Dropped Packets
In-Reply-To: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA624@usrkvexch01.zenimax.com>
References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA0E6@usrkvexch01.zenimax.com>
	<6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA624@usrkvexch01.zenimax.com>
Message-ID: <20120618202242.GN23613@datacomm.albany.edu>

On Mon, Jun 18, 2012 at 03:24:52PM -0400, Will Havlovick wrote:
> Update:
> 
> I have found a way to lessen the amount of packets being dropped.
> 
> Here is what I have:
> Dell r310 - 3.2Ghz - 4GB RAM - Dell hardware RAID controller - two 1TB 7.2k drives in a RAID 1
> 
> Test scenario:
> Two bro2.0 servers running virtually identical configs with Ubuntu 11.10.
> One server for testing and one as a control.
> Both monitoring 2 Network Taps of live traffic.
> 
> Test 1 : increased RAM to 8GB 
> Result : same amount of packets dropped
> 
> Test 2 : replaced hard drives with 2 10k drives in a RAID 1
> Result : 10% less packet drops  in bro logs as compared to the control server
> 
> Test 3 : replaced hard drives with 2 SSD drives in a RAID 1
> Result :  80% less packet drops then the control server
> 
> Test 4 : switched SSD hard drives to a RAID 0
> Result | 90% less packet drops then the control server
> 
> I have heard that SSD drives have a shorter life span if it is written to a lot.  So this is probably not the best solution.
> 
> But, from now on I will order servers with the fastest possible hard drives which for the Dell r310 are 15K SAS drives.  
> 
> When I get the 15K SAS drives in I will run the same tests and put the results out.

How much disk IO are these boxes actually doing while the test is
running?

A good tool for showing this is dstat (apt-get install dstat)

    dstat  --disk-tps -a --mem 5


-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From hlein at korelogic.com  Mon Jun 18 13:24:54 2012
From: hlein at korelogic.com (Hank Leininger)
Date: Mon, 18 Jun 2012 16:24:54 -0400
Subject: [Bro] Dropped Packets
In-Reply-To: <CANpnLHghVYqvt9vyBx2kqNiOS-zuj74XfcFEs7EjHyzA6ug07g@mail.gmail.com>
References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA0E6@usrkvexch01.zenimax.com>
	<6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA624@usrkvexch01.zenimax.com>
	<CANpnLHghVYqvt9vyBx2kqNiOS-zuj74XfcFEs7EjHyzA6ug07g@mail.gmail.com>
Message-ID: <20120618202454.GC24357@marklar.spinoli.org>

On Mon, Jun 18, 2012 at 02:37:19PM -0500, Martin Holste wrote:
> That's really interesting!  What about using a ramdisk (e.g. /dev/shm)
> file system for logs being currently written to, then at the hour mark
> (when the logs rollover), putting them on disk?  That should
> theoretically take disk performance out of the equation, and I'd be
> really interested in your numbers then.

Along those lines, you could experiment with various filesystem
robustness-performance tradeoffs.  For instance, assuming you're running
ext4 filesystems, you could try any/all of these mount-time/fstab options:

  noatime,barrier=0,data=writeback

...with the related caveats about what you're giving up (man mount).

Hank

> On Mon, Jun 18, 2012 at 2:24 PM, Will Havlovick
> <will.havlovick at zenimax.com> wrote:
> > Update:
> >
> > I have found a way to lessen the amount of packets being dropped.
> >
> > Here is what I have:
> > Dell r310 - 3.2Ghz - 4GB RAM - Dell hardware RAID controller - two 1TB 7.2k drives in a RAID 1
> >
> > Test scenario:
> > Two bro2.0 servers running virtually identical configs with Ubuntu 11.10.
> > One server for testing and one as a control.
> > Both monitoring 2 Network Taps of live traffic.
> >
> > Test 1 : increased RAM to 8GB
> > Result : same amount of packets dropped
> >
> > Test 2 : replaced hard drives with 2 10k drives in a RAID 1
> > Result : 10% less packet drops ?in bro logs as compared to the control server
> >
> > Test 3 : replaced hard drives with 2 SSD drives in a RAID 1
> > Result : ?80% less packet drops then the control server
> >
> > Test 4 : switched SSD hard drives to a RAID 0
> > Result | 90% less packet drops then the control server
> >
> > I have heard that SSD drives have a shorter life span if it is written to a lot. ?So this is probably not the best solution.
> >
> > But, from now on I will order servers with the fastest possible hard drives which for the Dell r310 are 15K SAS drives.
> >
> > When I get the 15K SAS drives in I will run the same tests and put the results out.
> >
> >
> > Will
> >
> > -----Original Message-----
> > From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf Of Will Havlovick
> > Sent: Thursday, January 12, 2012 2:00 PM
> > To: 'bro at bro-ids.org'
> > Subject: [Bro] Dropped Packets
> >
> > Hi all,
> >
> > I recently upgraded 3 standalone Bro nodes. ?2 of them are Ubuntu and one of them is CentOS 6.2.
> >
> > On the 2 Ubuntu 11.10 boxes I have a lot of dropped packets in the notice.log
> > ---
> > PacketFilter::Dropped_Packets ? 476 packets dropped after filtering, 52258 received, 52258 on link
> > PacketFilter::Dropped_Packets ? 4914 packets dropped after filtering, 52785 received, 52785 on link
> > PacketFilter::Dropped_Packets ? 3061 packets dropped after filtering, 35701 received, 35702 on link
> > PacketFilter::Dropped_Packets ? 3371 packets dropped after filtering, 30573 received, 30591 on link
> > ---
> > broctl netstats
> > ? ? ? bro: 1326394056.309957 recvd=958721774 dropped=67351350 link=1026073125
> >
> > I then tried to add this line to the broctl.cfg from http://comments.gmane.org/gmane.comp.security.detection.bro/4146
> > broargs = -l 9800
> >
> > Which does not appear to be part of the final release and did not work.
> >
> > The CentOS box is dropping packets, but not the amounts that the 2 Ubuntu boxes are.
> >
> > Is there a way to reduce the amount of dropped packets?
> >
> > Also, I can provide more data if necessary.
> >
> > Thank you in advance,
> >
> >
> > Will
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-- 

Hank Leininger <hlein at korelogic.com>
D24D 2C2A F3AC B9AE CD03  B506 2D57 32E1 686B 6DB3
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 447 bytes
Desc: Digital signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120618/b0f34252/attachment.bin 

From tyler.schoenke at colorado.edu  Mon Jun 18 14:35:40 2012
From: tyler.schoenke at colorado.edu (Tyler T. Schoenke)
Date: Mon, 18 Jun 2012 15:35:40 -0600
Subject: [Bro] Dropped Packets
In-Reply-To: <DDA7F049-E5BE-4E98-B47F-9CBAEFD73FC7@icir.org>
References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA0E6@usrkvexch01.zenimax.com>
	<6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA624@usrkvexch01.zenimax.com>
	<CANpnLHghVYqvt9vyBx2kqNiOS-zuj74XfcFEs7EjHyzA6ug07g@mail.gmail.com>
	<CAH_p-VN=igCah-pK1xsvK352SM3vA7AuRuiPDfVSMsn-erYFVA@mail.gmail.com>
	<6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA626@usrkvexch01.zenimax.com>
	<DDA7F049-E5BE-4E98-B47F-9CBAEFD73FC7@icir.org>
Message-ID: <4FDF9F2C.6010501@colorado.edu>

I don't mean to hijack the thread, but I just tried cluster versus
standalone and got some interesting results.  I tuned the cFlow to send
about 45 mbps to an interface on a particular worker.  Based on the
Dropped_Packets in notice.log, I see about 35% drop on the cluster for
this particular worker.  When I run standalone against the same
interface with the same filter I see between 0 and 1% dropped.  Very odd.

Here is some output from dstat -a --mem 5

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-->
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw >
 14   9  76   0   0   1|  26k  430k|   0     0 |   0     0 |  24k  132k>
 11   7  82   0   0   1|   0  8192B|  17M  141k|   0     0 |  65k   91k>
 12   6  81   0   0   1|   0  9011B|  10M   32k|   0     0 |  67k   91k>
 12   6  81   0   0   1|   0   282k|  19M  142k|   0     0 |  69k   90k>
 12   7  81   0   0   0|   0    33k|  13M   35k|   0     0 |  69k   89k>
 11   6  82   0   0   1|   0  1558k|  17M  124k|   0     0 |  63k   92k>
 12   7  81   0   0   0|   0  3403k|  11M   26k|   0     0 |  67k   92k>
 12   7  80   0   0   1|   0  9011B|  23M  158k|   0     0 |  69k   91k>
 12   6  82   0   0   1|   0  9011B|  11M   33k|   0     0 |  67k   90k>
 12   6  80   0   0   1|   0   258k|  23M  125k|   0     0 |  69k   89k>
 13   7  80   0   0   0|   0  9011B|  13M   26k|   0     0 |  66k   91k>
 11   7  81   0   0   0|   0   490k|  17M  155k|   0     0 |  66k   91k>
 12   6  81   0   0   1|   0  4980k|  10M   29k|   0     0 |  67k   91k>^C

My dstat doesn't have the --disk-tps parameter (plug-in) that Justin
mentioned.  Do any of the values look too large?

Tyler

--
Tyler Schoenke
Network Security Manager
IT Security Office
University of Colorado at Boulder

On 6/18/12 2:22 PM, Seth Hall wrote:
> 
> On Jun 18, 2012, at 4:01 PM, Will Havlovick wrote:
> 
>> Per broctl>capstats,  I was averaging between 15-45mbps.
> 
> 
> Hm, that seems like an oddly low amount of traffic to see drops on a
> box like you have.
> 
> Could you also try the current master branch in our repository?  The
> logging framework has been threaded and it's likely that disk latency
> issues have been resolved to some degree.  Also, I assume these were
> running Bro in standalone mode?  When running in a cluster this
> shouldn't have nearly so much effect because the manager process does
> all of the log writing and it doesn't do any packet processing.
> 
> .Seth
> 
> -- Seth Hall International Computer Science Institute (Bro) because
> everyone has a network http://www.bro-ids.org/
> 
> 
> _______________________________________________ Bro mailing list 
> bro at bro-ids.org 
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 


From jones at tacc.utexas.edu  Mon Jun 18 21:43:30 2012
From: jones at tacc.utexas.edu (William Jones)
Date: Tue, 19 Jun 2012 04:43:30 +0000
Subject: [Bro] Dropped Packets
In-Reply-To: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA624@usrkvexch01.zenimax.com>
References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA0E6@usrkvexch01.zenimax.com>
	<6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA624@usrkvexch01.zenimax.com>
Message-ID: <D0BD4A5C03717E40AC104854AF37B8C216AC5737@EXMBX05.austin.utexas.edu>

If you running bro on linux, without pf_ring, try increasing net.core-rmem_default:

For 10GigE nic

net.core.rmem_max =     500000000
net.core.rmem_default = 500000000

For 1 GigE nics

 
net.core.rmem_max =     50000000
net.core.rmem_default = 50000000

Bill Jones



-----Original Message-----
From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf Of Will Havlovick
Sent: Monday, June 18, 2012 2:25 PM
To: 'bro at bro-ids.org'
Subject: Re: [Bro] Dropped Packets

Update:

I have found a way to lessen the amount of packets being dropped.

Here is what I have:
Dell r310 - 3.2Ghz - 4GB RAM - Dell hardware RAID controller - two 1TB 7.2k drives in a RAID 1

Test scenario:
Two bro2.0 servers running virtually identical configs with Ubuntu 11.10.
One server for testing and one as a control.
Both monitoring 2 Network Taps of live traffic.

Test 1 : increased RAM to 8GB
Result : same amount of packets dropped

Test 2 : replaced hard drives with 2 10k drives in a RAID 1 Result : 10% less packet drops  in bro logs as compared to the control server

Test 3 : replaced hard drives with 2 SSD drives in a RAID 1 Result :  80% less packet drops then the control server

Test 4 : switched SSD hard drives to a RAID 0 Result | 90% less packet drops then the control server

I have heard that SSD drives have a shorter life span if it is written to a lot.  So this is probably not the best solution.

But, from now on I will order servers with the fastest possible hard drives which for the Dell r310 are 15K SAS drives.  

When I get the 15K SAS drives in I will run the same tests and put the results out.


Will

-----Original Message-----
From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf Of Will Havlovick
Sent: Thursday, January 12, 2012 2:00 PM
To: 'bro at bro-ids.org'
Subject: [Bro] Dropped Packets

Hi all,

I recently upgraded 3 standalone Bro nodes.  2 of them are Ubuntu and one of them is CentOS 6.2.

On the 2 Ubuntu 11.10 boxes I have a lot of dropped packets in the notice.log
---
PacketFilter::Dropped_Packets   476 packets dropped after filtering, 52258 received, 52258 on link      
PacketFilter::Dropped_Packets   4914 packets dropped after filtering, 52785 received, 52785 on link     
PacketFilter::Dropped_Packets   3061 packets dropped after filtering, 35701 received, 35702 on link     
PacketFilter::Dropped_Packets   3371 packets dropped after filtering, 30573 received, 30591 on link     
---
broctl netstats
       bro: 1326394056.309957 recvd=958721774 dropped=67351350 link=1026073125

I then tried to add this line to the broctl.cfg from http://comments.gmane.org/gmane.comp.security.detection.bro/4146
broargs = -l 9800

Which does not appear to be part of the final release and did not work.

The CentOS box is dropping packets, but not the amounts that the 2 Ubuntu boxes are.  

Is there a way to reduce the amount of dropped packets?

Also, I can provide more data if necessary.

Thank you in advance,


Will

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From will.havlovick at zenimax.com  Tue Jun 19 06:16:31 2012
From: will.havlovick at zenimax.com (Will Havlovick)
Date: Tue, 19 Jun 2012 09:16:31 -0400
Subject: [Bro] Dropped Packets
In-Reply-To: <D0BD4A5C03717E40AC104854AF37B8C216AC5737@EXMBX05.austin.utexas.edu>
References: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA0E6@usrkvexch01.zenimax.com>
	<6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA624@usrkvexch01.zenimax.com>
	<D0BD4A5C03717E40AC104854AF37B8C216AC5737@EXMBX05.austin.utexas.edu>
Message-ID: <6AAC380F84E5004CB6BEE3A040BBBC6D05CC0BA62B@usrkvexch01.zenimax.com>

Thank you all  for the suggestions. 

I will begin testing the different options and email out the results.


Will

-----Original Message-----
From: William Jones [mailto:jones at tacc.utexas.edu] 
Sent: Tuesday, June 19, 2012 12:43 AM
To: Will Havlovick; 'bro at bro-ids.org'
Subject: RE: Dropped Packets

If you running bro on linux, without pf_ring, try increasing net.core-rmem_default:

For 10GigE nic

net.core.rmem_max =     500000000
net.core.rmem_default = 500000000

For 1 GigE nics

 
net.core.rmem_max =     50000000
net.core.rmem_default = 50000000

Bill Jones



-----Original Message-----
From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf Of Will Havlovick
Sent: Monday, June 18, 2012 2:25 PM
To: 'bro at bro-ids.org'
Subject: Re: [Bro] Dropped Packets

Update:

I have found a way to lessen the amount of packets being dropped.

Here is what I have:
Dell r310 - 3.2Ghz - 4GB RAM - Dell hardware RAID controller - two 1TB 7.2k drives in a RAID 1

Test scenario:
Two bro2.0 servers running virtually identical configs with Ubuntu 11.10.
One server for testing and one as a control.
Both monitoring 2 Network Taps of live traffic.

Test 1 : increased RAM to 8GB
Result : same amount of packets dropped

Test 2 : replaced hard drives with 2 10k drives in a RAID 1 Result : 10% less packet drops  in bro logs as compared to the control server

Test 3 : replaced hard drives with 2 SSD drives in a RAID 1 Result :  80% less packet drops then the control server

Test 4 : switched SSD hard drives to a RAID 0 Result | 90% less packet drops then the control server

I have heard that SSD drives have a shorter life span if it is written to a lot.  So this is probably not the best solution.

But, from now on I will order servers with the fastest possible hard drives which for the Dell r310 are 15K SAS drives.  

When I get the 15K SAS drives in I will run the same tests and put the results out.


Will

-----Original Message-----
From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf Of Will Havlovick
Sent: Thursday, January 12, 2012 2:00 PM
To: 'bro at bro-ids.org'
Subject: [Bro] Dropped Packets

Hi all,

I recently upgraded 3 standalone Bro nodes.  2 of them are Ubuntu and one of them is CentOS 6.2.

On the 2 Ubuntu 11.10 boxes I have a lot of dropped packets in the notice.log
---
PacketFilter::Dropped_Packets   476 packets dropped after filtering, 52258 received, 52258 on link      
PacketFilter::Dropped_Packets   4914 packets dropped after filtering, 52785 received, 52785 on link     
PacketFilter::Dropped_Packets   3061 packets dropped after filtering, 35701 received, 35702 on link     
PacketFilter::Dropped_Packets   3371 packets dropped after filtering, 30573 received, 30591 on link     
---
broctl netstats
       bro: 1326394056.309957 recvd=958721774 dropped=67351350 link=1026073125

I then tried to add this line to the broctl.cfg from http://comments.gmane.org/gmane.comp.security.detection.bro/4146
broargs = -l 9800

Which does not appear to be part of the final release and did not work.

The CentOS box is dropping packets, but not the amounts that the 2 Ubuntu boxes are.  

Is there a way to reduce the amount of dropped packets?

Also, I can provide more data if necessary.

Thank you in advance,


Will

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at icir.org  Thu Jun 21 04:46:24 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 21 Jun 2012 07:46:24 -0400
Subject: [Bro] Bro Exchange registration form posted
Message-ID: <6858173E-713C-4947-A6C5-F61DA83B6886@icir.org>

The registration form for the Bro Exchange has been posted and can be found here:
	http://www.regonline.com/broexchange2012

I hope to see you all there!

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From Liam.Randall at gigaco.com  Thu Jun 21 17:47:03 2012
From: Liam.Randall at gigaco.com (Liam Randall)
Date: Thu, 21 Jun 2012 20:47:03 -0400
Subject: [Bro] Bro Exchange registration form posted
In-Reply-To: <6858173E-713C-4947-A6C5-F61DA83B6886@icir.org>
References: <6858173E-713C-4947-A6C5-F61DA83B6886@icir.org>
Message-ID: <2697B86A359F6C43BF6FD44F8403D71741E152@giga-dc001.GigaCo.local>

As far as operation use goes for presentations; this one would be
interesting to hear about:

http://www.informationweek.com/news/security/attacks/240002474

Liam

-----Original Message-----
From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf
Of Seth Hall
Sent: Thursday, June 21, 2012 7:46 AM
To: bro at bro-ids.org
Subject: [Bro] Bro Exchange registration form posted

The registration form for the Bro Exchange has been posted and can be
found here:
	http://www.regonline.com/broexchange2012

I hope to see you all there!

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/


_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From srunnels at gmail.com  Thu Jun 21 18:11:53 2012
From: srunnels at gmail.com (scott runnels)
Date: Thu, 21 Jun 2012 21:11:53 -0400
Subject: [Bro] Bro Exchange registration form posted
In-Reply-To: <6858173E-713C-4947-A6C5-F61DA83B6886@icir.org>
References: <6858173E-713C-4947-A6C5-F61DA83B6886@icir.org>
Message-ID: <79F91C78-F8BA-4AA4-83F8-628E5BC740D1@gmail.com>

The registration form says Tuesday the 7th but doesn't mention the following day.  Is the exchange still two days?  


On Jun 21, 2012, at 7:46 AM, Seth Hall wrote:

> The registration form for the Bro Exchange has been posted and can be found here:
> 	http://www.regonline.com/broexchange2012
> 
> I hope to see you all there!
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From mcholste at gmail.com  Thu Jun 21 18:15:12 2012
From: mcholste at gmail.com (Martin Holste)
Date: Thu, 21 Jun 2012 20:15:12 -0500
Subject: [Bro] Bro Exchange registration form posted
In-Reply-To: <2697B86A359F6C43BF6FD44F8403D71741E152@giga-dc001.GigaCo.local>
References: <6858173E-713C-4947-A6C5-F61DA83B6886@icir.org>
	<2697B86A359F6C43BF6FD44F8403D71741E152@giga-dc001.GigaCo.local>
Message-ID: <CANpnLHjx-AbuEibqvFSXbOkB6tAiT-N5dAD4mEEbh2XyArJYUw@mail.gmail.com>

Nice article.  I'd love to hear about that if someone from NERSC can attend.

On Thu, Jun 21, 2012 at 7:47 PM, Liam Randall <Liam.Randall at gigaco.com> wrote:
> As far as operation use goes for presentations; this one would be
> interesting to hear about:
>
> http://www.informationweek.com/news/security/attacks/240002474
>
> Liam
>
> -----Original Message-----
> From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf
> Of Seth Hall
> Sent: Thursday, June 21, 2012 7:46 AM
> To: bro at bro-ids.org
> Subject: [Bro] Bro Exchange registration form posted
>
> The registration form for the Bro Exchange has been posted and can be
> found here:
> ? ? ? ?http://www.regonline.com/broexchange2012
>
> I hope to see you all there!
>
> ?.Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at icir.org  Thu Jun 21 19:03:59 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 21 Jun 2012 22:03:59 -0400
Subject: [Bro] Bro Exchange registration form posted
In-Reply-To: <CANpnLHjx-AbuEibqvFSXbOkB6tAiT-N5dAD4mEEbh2XyArJYUw@mail.gmail.com>
References: <6858173E-713C-4947-A6C5-F61DA83B6886@icir.org>
	<2697B86A359F6C43BF6FD44F8403D71741E152@giga-dc001.GigaCo.local>
	<CANpnLHjx-AbuEibqvFSXbOkB6tAiT-N5dAD4mEEbh2XyArJYUw@mail.gmail.com>
Message-ID: <EE8A94E9-BADA-454F-BED7-1BD900BEA204@icir.org>


On Jun 21, 2012, at 9:15 PM, Martin Holste wrote:

> Nice article.  I'd love to hear about that if someone from NERSC can attend.


Yes, I just chatted with the person that set it up and it sounds very likely that he will be there and presenting.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From vern at icir.org  Thu Jun 21 19:42:47 2012
From: vern at icir.org (Vern Paxson)
Date: Thu, 21 Jun 2012 19:42:47 -0700
Subject: [Bro] Bro Exchange registration form posted
In-Reply-To: <2697B86A359F6C43BF6FD44F8403D71741E152@giga-dc001.GigaCo.local>
	(Thu, 21 Jun 2012 20:47:03 EDT).
Message-ID: <20120622024247.CC0502C4004@rock.ICSI.Berkeley.EDU>

> As far as operation use goes for presentations; this one would be
> interesting to hear about:
> 
> http://www.informationweek.com/news/security/attacks/240002474

I'll let someone from NERSC answer definitively, but I believe this was
in fact pretty much a non-event.

		Vern


From Liam.Randall at gigaco.com  Thu Jun 21 19:46:28 2012
From: Liam.Randall at gigaco.com (Liam Randall)
Date: Thu, 21 Jun 2012 22:46:28 -0400
Subject: [Bro] Bro Exchange registration form posted
In-Reply-To: <CANpnLHjx-AbuEibqvFSXbOkB6tAiT-N5dAD4mEEbh2XyArJYUw@mail.gmail.com>
References: <6858173E-713C-4947-A6C5-F61DA83B6886@icir.org>
	<2697B86A359F6C43BF6FD44F8403D71741E152@giga-dc001.GigaCo.local>
	<CANpnLHjx-AbuEibqvFSXbOkB6tAiT-N5dAD4mEEbh2XyArJYUw@mail.gmail.com>
Message-ID: <2697B86A359F6C43BF6FD44F8403D71741E153@giga-dc001.GigaCo.local>

Martin,

I hope you're going to throw your hat into the ring (again).

Liam

-----Original Message-----
From: Martin Holste [mailto:mcholste at gmail.com] 
Sent: Thursday, June 21, 2012 9:15 PM
To: Liam Randall
Cc: Seth Hall; bro at bro-ids.org
Subject: Re: [Bro] Bro Exchange registration form posted

Nice article.  I'd love to hear about that if someone from NERSC can attend.

On Thu, Jun 21, 2012 at 7:47 PM, Liam Randall <Liam.Randall at gigaco.com> wrote:
> As far as operation use goes for presentations; this one would be 
> interesting to hear about:
>
> http://www.informationweek.com/news/security/attacks/240002474
>
> Liam
>
> -----Original Message-----
> From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On 
> Behalf Of Seth Hall
> Sent: Thursday, June 21, 2012 7:46 AM
> To: bro at bro-ids.org
> Subject: [Bro] Bro Exchange registration form posted
>
> The registration form for the Bro Exchange has been posted and can be 
> found here:
> ? ? ? ?http://www.regonline.com/broexchange2012
>
> I hope to see you all there!
>
> ?.Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From mcholste at gmail.com  Thu Jun 21 19:56:28 2012
From: mcholste at gmail.com (Martin Holste)
Date: Thu, 21 Jun 2012 21:56:28 -0500
Subject: [Bro] Bro Exchange registration form posted
In-Reply-To: <20120622024247.CC0502C4004@rock.ICSI.Berkeley.EDU>
References: <2697B86A359F6C43BF6FD44F8403D71741E152@giga-dc001.GigaCo.local>
	<20120622024247.CC0502C4004@rock.ICSI.Berkeley.EDU>
Message-ID: <CANpnLHhe+JZwvu1rii+TbJbw7kvbjh1UCkGbtTaScqbZ_BvMmg@mail.gmail.com>

@Vern Just showing off the development and deployment of the
monitoring would be plenty cool enough for me.  I'm interested in how
the heuristics were developed and tuned, and what the challenges were
along the way.  My hunch is that there are a lot of lessons learned
there that could be extended to covering more typical scenarios like
Active Directory abuse.

@Liam I really hope I can finagle some way to get out there, but it'll
have to be on my own dime.  I'm clinging to the hope that I can make
it into a family vacation.  Maybe there could be a "Monitoring Elmo's
World" track for the little one and a "Auditing Pinterest Activity
Using Bro" for the Mrs?

On Thu, Jun 21, 2012 at 9:42 PM, Vern Paxson <vern at icir.org> wrote:
>> As far as operation use goes for presentations; this one would be
>> interesting to hear about:
>>
>> http://www.informationweek.com/news/security/attacks/240002474
>
> I'll let someone from NERSC answer definitively, but I believe this was
> in fact pretty much a non-event.
>
> ? ? ? ? ? ? ? ?Vern
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From middleton.jake at gmail.com  Mon Jun 25 13:26:55 2012
From: middleton.jake at gmail.com (Jake Middleton)
Date: Mon, 25 Jun 2012 15:26:55 -0500
Subject: [Bro] Global IP host ignore
Message-ID: <CAB05aXhBeD7uyWMPFREV8THGJ4dPm55YBjokmCrvFQUDbjsPiw@mail.gmail.com>

I have an install using 8 nodes and a master on a single host.  I'm
monitoring ~2,000 hosts across a split core and would like to add a global
ignore for a handfull of noisy hosts.

What's the best approach to handle this?

Thanks in advance

-- 
-j
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120625/608c76b7/attachment.html 

From hlin33 at illinois.edu  Mon Jun 25 13:34:05 2012
From: hlin33 at illinois.edu (Hui Lin (Hugo) )
Date: Mon, 25 Jun 2012 15:34:05 -0500
Subject: [Bro] Hui Lin_Enable Protocol Analyzer in Bro bare mode
Message-ID: <CAKq214nRLJijEnzm=FXjqmbN++qzNjLJbZjeWaq8tTXaNb=+0Q@mail.gmail.com>

Hi,

I am using Bro bare mode to test my own policy script.

I also like to use a Syslog analyzer to analyze *syslog_message* event. I
define *syslog_message* event in my own script, but this event handler is
not executed under bare mode? I am wondering what scripts should be loaded
to enable Syslog analyzer.

Best,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120625/1c6ffeb5/attachment.html 

From hlin33 at illinois.edu  Mon Jun 25 13:36:48 2012
From: hlin33 at illinois.edu (Hui Lin (Hugo) )
Date: Mon, 25 Jun 2012 15:36:48 -0500
Subject: [Bro] Hui Lin_Bro disable log stream is not working?
Message-ID: <CAKq214kDQVgy84ENxfzktD6dw3i7E32gk5HQ2_dg3BPgx0J8vQ@mail.gmail.com>

Hi,

In order to disable default logs, I follow the documentation and define
bro_init as follows:


event bro_init() &priority = 5
        {
        Log::disable_stream(Conn::LOG);
        Log::disable_stream(Notice::POLICY_LOG);
        Log::disable_stream(PacketFilter::LOG);
        Log::disable_stream(Syslog::LOG);
        }

But, all logs are generated as usual, any comment?

Best,

Hui
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120625/06992809/attachment.html 

From seth at icir.org  Mon Jun 25 13:39:41 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 25 Jun 2012 16:39:41 -0400
Subject: [Bro] Hui Lin_Enable Protocol Analyzer in Bro bare mode
In-Reply-To: <CAKq214nRLJijEnzm=FXjqmbN++qzNjLJbZjeWaq8tTXaNb=+0Q@mail.gmail.com>
References: <CAKq214nRLJijEnzm=FXjqmbN++qzNjLJbZjeWaq8tTXaNb=+0Q@mail.gmail.com>
Message-ID: <CDEB4DC5-5032-4DEB-A8D2-47A8D4453A53@icir.org>


On Jun 25, 2012, at 4:34 PM, Hui Lin (Hugo) wrote:

> I also like to use a Syslog analyzer to analyze syslog_message event. I define syslog_message event in my own script, but this event handler is not executed under bare mode? I am wondering what scripts should be loaded to enable Syslog analyzer.

It's enabled by default.  Can you show the code you are using that isn't working?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Mon Jun 25 13:42:56 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 25 Jun 2012 16:42:56 -0400
Subject: [Bro] Global IP host ignore
In-Reply-To: <CAB05aXhBeD7uyWMPFREV8THGJ4dPm55YBjokmCrvFQUDbjsPiw@mail.gmail.com>
References: <CAB05aXhBeD7uyWMPFREV8THGJ4dPm55YBjokmCrvFQUDbjsPiw@mail.gmail.com>
Message-ID: <4C80895C-94A0-4808-A177-4F524084E49C@icir.org>


On Jun 25, 2012, at 4:26 PM, Jake Middleton wrote:

> I have an install using 8 nodes and a master on a single host.  I'm monitoring ~2,000 hosts across a split core and would like to add a global ignore for a handfull of noisy hosts.
>  
> What's the best approach to handle this?

Unfortunately it's kind of messy right now due to implementation issues in the packet filter framework, but here it goes (it will be fixed in 2.2 probably, I didn't get the rewrite ready for 2.1)?

redef PacketFilter::all_packets = F;
redef capture_filters = [[ "all"] = "ip or not ip"];
redef restrict_filters += [ ["not-high-volume-hosts"] = "not host 192.168.1.100 and not host 192.168.2.100"];

You can just set the restrict filter to whatever you want and put that in local.bro.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From hlin33 at illinois.edu  Mon Jun 25 13:44:37 2012
From: hlin33 at illinois.edu (Hui Lin (Hugo) )
Date: Mon, 25 Jun 2012 15:44:37 -0500
Subject: [Bro] Hui Lin_Enable Protocol Analyzer in Bro bare mode
In-Reply-To: <7de83747e340497e8773b12fdf4928b7@CITESHT1.ad.uillinois.edu>
References: <CAKq214nRLJijEnzm=FXjqmbN++qzNjLJbZjeWaq8tTXaNb=+0Q@mail.gmail.com>
	<7de83747e340497e8773b12fdf4928b7@CITESHT1.ad.uillinois.edu>
Message-ID: <CAKq214=o3BW36ZTGsunjeugG6rT8uSf+mOBvc3A83iT7v47kEw@mail.gmail.com>

Actually it is quite simple. This is my syslog_message event handler.

@load frameworks/communication/listen

....

event syslog_message(c: connection, facility: count, severity: count, msg:
string)
        {

        gUsrID = facility;

        print fmt("syslog %d", facility);

        findSyslog = T ;

        }

gUsrID and findSyslog are two global variables.

I am not sure why it is not executing. I did not see any print on the
console.

Best,

Hui


On Mon, Jun 25, 2012 at 3:39 PM, Seth Hall <seth at icir.org> wrote:

>
> On Jun 25, 2012, at 4:34 PM, Hui Lin (Hugo) wrote:
>
> > I also like to use a Syslog analyzer to analyze syslog_message event. I
> define syslog_message event in my own script, but this event handler is not
> executed under bare mode? I am wondering what scripts should be loaded to
> enable Syslog analyzer.
>
> It's enabled by default.  Can you show the code you are using that isn't
> working?
>
>  .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120625/c9562c6a/attachment.html 

From seth at icir.org  Mon Jun 25 13:45:59 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 25 Jun 2012 16:45:59 -0400
Subject: [Bro] Hui Lin_Bro disable log stream is not working?
In-Reply-To: <CAKq214kDQVgy84ENxfzktD6dw3i7E32gk5HQ2_dg3BPgx0J8vQ@mail.gmail.com>
References: <CAKq214kDQVgy84ENxfzktD6dw3i7E32gk5HQ2_dg3BPgx0J8vQ@mail.gmail.com>
Message-ID: <49AEA343-E827-435C-B685-2F772FC55B4D@icir.org>


On Jun 25, 2012, at 4:36 PM, Hui Lin (Hugo) wrote:

> event bro_init() &priority = 5

Remove the priority.  

Rule of thumb:
	Don't put a priority to your event handlers unless you know why you are giving a priority.

 .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Mon Jun 25 13:55:11 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 25 Jun 2012 16:55:11 -0400
Subject: [Bro] Hui Lin_Enable Protocol Analyzer in Bro bare mode
In-Reply-To: <CAKq214=o3BW36ZTGsunjeugG6rT8uSf+mOBvc3A83iT7v47kEw@mail.gmail.com>
References: <CAKq214nRLJijEnzm=FXjqmbN++qzNjLJbZjeWaq8tTXaNb=+0Q@mail.gmail.com>
	<7de83747e340497e8773b12fdf4928b7@CITESHT1.ad.uillinois.edu>
	<CAKq214=o3BW36ZTGsunjeugG6rT8uSf+mOBvc3A83iT7v47kEw@mail.gmail.com>
Message-ID: <3B0B2333-DAD4-48BA-BD77-C3DE15AB0E70@icir.org>


On Jun 25, 2012, at 4:44 PM, Hui Lin (Hugo) wrote:

> I am not sure why it is not executing. I did not see any print on the console. 

I just noticed that you said you are running in bare mode.  Why is that?  I would only recommend that you do that if you really know what you are doing.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From middleton.jake at gmail.com  Mon Jun 25 13:58:07 2012
From: middleton.jake at gmail.com (Jake Middleton)
Date: Mon, 25 Jun 2012 15:58:07 -0500
Subject: [Bro] Global IP host ignore
In-Reply-To: <4C80895C-94A0-4808-A177-4F524084E49C@icir.org>
References: <CAB05aXhBeD7uyWMPFREV8THGJ4dPm55YBjokmCrvFQUDbjsPiw@mail.gmail.com>
	<4C80895C-94A0-4808-A177-4F524084E49C@icir.org>
Message-ID: <-1374936175144951683@unknownmsgid>

Thanks Seth

I'll try to wrap my head around that and make it work.

-j

>From my iPhone.

On Jun 25, 2012, at 3:42 PM, Seth Hall <seth at icir.org> wrote:

>
> On Jun 25, 2012, at 4:26 PM, Jake Middleton wrote:
>
>> I have an install using 8 nodes and a master on a single host.  I'm monitoring ~2,000 hosts across a split core and would like to add a global ignore for a handfull of noisy hosts.
>>
>> What's the best approach to handle this?
>
> Unfortunately it's kind of messy right now due to implementation issues in the packet filter framework, but here it goes (it will be fixed in 2.2 probably, I didn't get the rewrite ready for 2.1)?
>
> redef PacketFilter::all_packets = F;
> redef capture_filters = [[ "all"] = "ip or not ip"];
> redef restrict_filters += [ ["not-high-volume-hosts"] = "not host 192.168.1.100 and not host 192.168.2.100"];
>
> You can just set the restrict filter to whatever you want and put that in local.bro.
>
>  .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>



From seth at icir.org  Mon Jun 25 14:05:12 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 25 Jun 2012 17:05:12 -0400
Subject: [Bro] Global IP host ignore
In-Reply-To: <-1374936175144951683@unknownmsgid>
References: <CAB05aXhBeD7uyWMPFREV8THGJ4dPm55YBjokmCrvFQUDbjsPiw@mail.gmail.com>
	<4C80895C-94A0-4808-A177-4F524084E49C@icir.org>
	<-1374936175144951683@unknownmsgid>
Message-ID: <9D5522F7-8252-4E8C-9715-159F6B1E543A@icir.org>


On Jun 25, 2012, at 4:58 PM, Jake Middleton wrote:

> I'll try to wrap my head around that and make it work.


Sorry about the complicated-ness of it, this was a particular pain point for the 2.0 release and will continue to be for the 2.1 release it looks like.  This task will be extremely easy to do and understand once my packet filter framework changes are merged in.  Keep the questions coming!

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jsiwek at illinois.edu  Mon Jun 25 14:19:26 2012
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 25 Jun 2012 21:19:26 +0000
Subject: [Bro] Hui Lin_Enable Protocol Analyzer in Bro bare mode
In-Reply-To: <CAKq214nRLJijEnzm=FXjqmbN++qzNjLJbZjeWaq8tTXaNb=+0Q@mail.gmail.com>
References: <CAKq214nRLJijEnzm=FXjqmbN++qzNjLJbZjeWaq8tTXaNb=+0Q@mail.gmail.com>
Message-ID: <F0CB020D-A41C-4F5D-AFF7-9C163EC2F454@illinois.edu>


> I also like to use a Syslog analyzer to analyze syslog_message event. I define syslog_message event in my own script, but this event handler is not executed under bare mode? I am wondering what scripts should be loaded to enable Syslog analyzer.

You could "@load base/protocols/syslog" to enable the analyzer at least for UDP port 514 traffic.  Or you could just "redef dpd_config" like base/protocols/syslog/main.bro does for the ports you need.  Not sure if a DPD signature could/should be added for syslog so that would not be necessary, Seth would probably have an idea.

    Jon


From middleton.jake at gmail.com  Mon Jun 25 14:42:46 2012
From: middleton.jake at gmail.com (Jake Middleton)
Date: Mon, 25 Jun 2012 16:42:46 -0500
Subject: [Bro] Global IP host ignore
In-Reply-To: <9D5522F7-8252-4E8C-9715-159F6B1E543A@icir.org>
References: <CAB05aXhBeD7uyWMPFREV8THGJ4dPm55YBjokmCrvFQUDbjsPiw@mail.gmail.com>
	<4C80895C-94A0-4808-A177-4F524084E49C@icir.org>
	<-1374936175144951683@unknownmsgid>
	<9D5522F7-8252-4E8C-9715-159F6B1E543A@icir.org>
Message-ID: <-3939490598024745734@unknownmsgid>

-j

>From my iPhone.

On Jun 25, 2012, at 4:05 PM, Seth Hall <seth at icir.org> wrote:

>
> On Jun 25, 2012, at 4:58 PM, Jake Middleton wrote:
>
>> I'll try to wrap my head around that and make it work.
>
>
> Sorry about the complicated-ness of it, this was a particular pain point for the 2.0 release and will continue to be for the 2.1 release it looks like.  This task will be extremely easy to do and understand once my packet filter framework changes are merged in.  Keep the questions coming!
>
>  .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>



From hunarame at gmail.com  Tue Jun 26 06:16:50 2012
From: hunarame at gmail.com (Naveed Anwar)
Date: Tue, 26 Jun 2012 18:16:50 +0500
Subject: [Bro] DNS state remains uninitialized in dns_message event
Message-ID: <CANg8oi_YnYK=_0FWBuj_c1LMOQ+VGvmS+59YmdnRAJWkwKK6Vw@mail.gmail.com>

Hi,

I want to capture DNS queries of a pcap but there is an issue with DNS
events. The DNS state in the connection record remains uninitialized for my
DNS queries.

Here's how I'm looking at the DNS state information:

event dns_message(c: connection, is_orig: bool, msg: dns_msg, len: count)
{
         print c;
}

pcap: http://www.sysnet.org.pk/needo/mix1.pcap
bro-output: http://www.sysnet.org.pk/needo/bro.log


-- 
Regards,
Naveed Anwar Bhatti
MS(CS) - FAST NU islamabd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120626/edd5699f/attachment.html 

From seth at icir.org  Tue Jun 26 07:00:55 2012
From: seth at icir.org (Seth Hall)
Date: Tue, 26 Jun 2012 10:00:55 -0400
Subject: [Bro] DNS state remains uninitialized in dns_message event
In-Reply-To: <CANg8oi_YnYK=_0FWBuj_c1LMOQ+VGvmS+59YmdnRAJWkwKK6Vw@mail.gmail.com>
References: <CANg8oi_YnYK=_0FWBuj_c1LMOQ+VGvmS+59YmdnRAJWkwKK6Vw@mail.gmail.com>
Message-ID: <B97744B1-C63F-402F-8558-54CB031104C2@icir.org>


On Jun 26, 2012, at 9:16 AM, Naveed Anwar wrote:

> I want to capture DNS queries of a pcap but there is an issue with DNS events. The DNS state in the connection record remains uninitialized for my DNS queries.
> 
> event dns_message(c: connection, is_orig: bool, msg: dns_msg, len: count)

I don't use the dns_message event in the base scripts for DNS so what is and what is not set when that event fires is currently undefined.  Also, I'm a little unsure about what you suspect is unset in the output from your short script?

If you want to look at the data that ends up being inserted into the logs, you can look at it this way...

event DNS::log_dns(rec: DNS::Info)
	{
	print rec;
	}

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From hunarame at gmail.com  Wed Jun 27 06:25:16 2012
From: hunarame at gmail.com (Naveed Anwar)
Date: Wed, 27 Jun 2012 18:25:16 +0500
Subject: [Bro] DNS state remains uninitialized in dns_message event
In-Reply-To: <B97744B1-C63F-402F-8558-54CB031104C2@icir.org>
References: <CANg8oi_YnYK=_0FWBuj_c1LMOQ+VGvmS+59YmdnRAJWkwKK6Vw@mail.gmail.com>
	<B97744B1-C63F-402F-8558-54CB031104C2@icir.org>
Message-ID: <CANg8oi9M5VOXD3Duw7rr9X6cNP1zaock==4vVNp3DbO+pJqR4Q@mail.gmail.com>

On Tue, Jun 26, 2012 at 7:00 PM, Seth Hall <seth at icir.org> wrote:

> > I don't use the dns_message event in the base scripts for DNS so what is
> and what is not set when that event fires is currently undefined.  > Also,
> I'm a little unsure about what you suspect is unset in the output from your
> short script?
>

Thanks for the quick reply. I was trying to read the c$dns record in the
dns_message event which was uninitialized. Since you've pointed out that
the dns_message event's behavior is currently undefined I'll now be using
dns_query_reply and dns_rejected events to look at the DNS queries.


-- 
Regards,
Naveed Anwar Bhatti
MS(CS) - FAST NU islamabd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120627/51d0e96d/attachment.html 

From sheharbano.k at gmail.com  Thu Jun 28 12:06:29 2012
From: sheharbano.k at gmail.com (Sheharbano Khattak)
Date: Fri, 29 Jun 2012 00:06:29 +0500
Subject: [Bro] Playing with the input framework
Message-ID: <CAA6v-1aMTP8y-miEkhHytJhtf0DS-g_XFZP-ky7S4p28U-pJwQ@mail.gmail.com>

Hi,

I recently finished reading about the new input framework
http://www.icsi.berkeley.edu/~bernhard/papers/loneWolf.pdf and found it
very interesting. As a first step, i tried implementing the example about
reading data into tables mentioned here
http://blog.bro-ids.org/2012/06/upcoming-loading-data-into-bro-with.html.
My bro and source blacklist file look like this:

---------------------------------try.bro----------------------------------------------
module Try;

type Idx: record {
        ip: addr;
};

type Val: record {
        timestamp: time;
        reason: string;
};

global blacklist: table[addr] of Val = table();

event bro_init()
    {
    print "hello";
    Input::add_table([$source="bl.txt", $name="bl_stream", $idx=Idx,
$val=Val, $destination=Try::blacklist]);
    Input::remove("bl_stream");
    print "bye";
    }

event Input::update_finished(name: string, source: string)
    {
        # now all data is in the table
    print "Updated";
        print Try::blacklist;
    }

----------------------------bl.txt---------------------------------------------

#fields ip    timestamp    reason
#types    addr    time    string
192.168.17.1    1333252748    Malware host
192.168.27.2    1330235733    Botnet server
192.168.250.3    1333145108    Virus detected

---------------------------------------------------------------------------------
Initially, i tried "bro -r file.pcap try.bro" but it didn't work. To
provide ample time for reading in the blacklist, i tried "bro -i eth0
try.bro". The output displays hello and bye but the blacklist wasn't
printed even after 5 minutes. I tried giving the absolute source path i.e.
"/home/myname/bl.txt" but to no avail.

Moreover, i purposely gave a wrong input source file and no error was
displayed. I feel an appropriate error message will be helpful if someone
has mistyped the source file name or if it doesn't exist.

Regards,
-- 
Sheharbano Khattak

http://etheryell.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120629/2dc5a767/attachment.html 

From bernhard at ICSI.Berkeley.EDU  Thu Jun 28 12:14:03 2012
From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann)
Date: Thu, 28 Jun 2012 12:14:03 -0700
Subject: [Bro] Playing with the input framework
In-Reply-To: <CAA6v-1aMTP8y-miEkhHytJhtf0DS-g_XFZP-ky7S4p28U-pJwQ@mail.gmail.com>
References: <CAA6v-1aMTP8y-miEkhHytJhtf0DS-g_XFZP-ky7S4p28U-pJwQ@mail.gmail.com>
Message-ID: <C8575418-A865-43C2-9619-0AF7694E3D35@icsi.berkeley.edu>

Hello Sheharabano,

I just tried your example and it seems to work fine. Please note that the fields in the file "bl.txt" need to be separated with tabulators - including the header lines. If you simply copied the example from the website it probably ended up being separated with spaces.

The input frameworks outputs error messages. These are written into the file reporter.log (where most Bro error messages end up).

If the file contains a line that looks like 
"Reporter::ERROR	InputReader/bl.txt: Not enough fields in line?" 
the framework is complaining about missing tabulators-fields (because it is not finding enough tab-separated entries in the line it read).

I hope that helps - if it still does not solve your problem please write again.

Bernhard


On Jun 28, 2012, at 12:06 PM, Sheharbano Khattak wrote:

> Hi,
> 
> I recently finished reading about the new input framework http://www.icsi.berkeley.edu/~bernhard/papers/loneWolf.pdf and found it very interesting. As a first step, i tried implementing the example about reading data into tables mentioned here http://blog.bro-ids.org/2012/06/upcoming-loading-data-into-bro-with.html. My bro and source blacklist file look like this:
> 
> ---------------------------------try.bro----------------------------------------------
> module Try;
> 
> type Idx: record {
>         ip: addr;
> };
> 
> type Val: record {
>         timestamp: time;
>         reason: string;
> };
> 
> global blacklist: table[addr] of Val = table();
> 
> event bro_init() 
>     {
>     print "hello";
>     Input::add_table([$source="bl.txt", $name="bl_stream", $idx=Idx, $val=Val, $destination=Try::blacklist]);
>     Input::remove("bl_stream");    
>     print "bye";
>     }
> 
> event Input::update_finished(name: string, source: string) 
>     {
>         # now all data is in the table
>     print "Updated";
>         print Try::blacklist;
>     }
> 
> ----------------------------bl.txt---------------------------------------------
> 
> #fields ip    timestamp    reason
> #types    addr    time    string
> 192.168.17.1    1333252748    Malware host
> 192.168.27.2    1330235733    Botnet server
> 192.168.250.3    1333145108    Virus detected
> 
> ---------------------------------------------------------------------------------
> Initially, i tried "bro -r file.pcap try.bro" but it didn't work. To provide ample time for reading in the blacklist, i tried "bro -i eth0 try.bro". The output displays hello and bye but the blacklist wasn't printed even after 5 minutes. I tried giving the absolute source path i.e. "/home/myname/bl.txt" but to no avail. 
> 
> Moreover, i purposely gave a wrong input source file and no error was displayed. I feel an appropriate error message will be helpful if someone has mistyped the source file name or if it doesn't exist. 
> 
> Regards,
> -- 
> Sheharbano Khattak
> 
> http://etheryell.com
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120628/8656c37f/attachment.html 

From sheharbano.k at gmail.com  Thu Jun 28 12:28:33 2012
From: sheharbano.k at gmail.com (Sheharbano Khattak)
Date: Fri, 29 Jun 2012 00:28:33 +0500
Subject: [Bro] Playing with the input framework
In-Reply-To: <C8575418-A865-43C2-9619-0AF7694E3D35@icsi.berkeley.edu>
References: <CAA6v-1aMTP8y-miEkhHytJhtf0DS-g_XFZP-ky7S4p28U-pJwQ@mail.gmail.com>
	<C8575418-A865-43C2-9619-0AF7694E3D35@icsi.berkeley.edu>
Message-ID: <CAA6v-1bLpAFa=K7VUKnVAKCWXR2_RC8AJ+FgjsHw4zcQ4MRhSQ@mail.gmail.com>

Thank you. Indeed there was a tabulation problem. On a side note, how do
you decide which errors are directed to the console and which ones are
reported in reporter.log ?

Regards,

On Fri, Jun 29, 2012 at 12:14 AM, Bernhard Amann <bernhard at icsi.berkeley.edu
> wrote:

> Hello Sheharabano,
>
> I just tried your example and it seems to work fine. Please note that the
> fields in the file "bl.txt" need to be separated with tabulators -
> including the header lines. If you simply copied the example from the
> website it probably ended up being separated with spaces.
>
> The input frameworks outputs error messages. These are written into the
> file reporter.log (where most Bro error messages end up).
>
> If the file contains a line that looks like
> "Reporter::ERROR InputReader/bl.txt: Not enough fields in line?"
> the framework is complaining about missing tabulators-fields (because it
> is not finding enough tab-separated entries in the line it read).
>
> I hope that helps - if it still does not solve your problem please write
> again.
>
> Bernhard
>
>
> On Jun 28, 2012, at 12:06 PM, Sheharbano Khattak wrote:
>
> Hi,
>
> I recently finished reading about the new input framework
> http://www.icsi.berkeley.edu/~bernhard/papers/loneWolf.pdf and found it
> very interesting. As a first step, i tried implementing the example about
> reading data into tables mentioned here
> http://blog.bro-ids.org/2012/06/upcoming-loading-data-into-bro-with.html.
> My bro and source blacklist file look like this:
>
>
> ---------------------------------try.bro----------------------------------------------
> module Try;
>
> type Idx: record {
>         ip: addr;
> };
>
> type Val: record {
>         timestamp: time;
>         reason: string;
> };
>
> global blacklist: table[addr] of Val = table();
>
> event bro_init()
>     {
>     print "hello";
>     Input::add_table([$source="bl.txt", $name="bl_stream", $idx=Idx,
> $val=Val, $destination=Try::blacklist]);
>     Input::remove("bl_stream");
>     print "bye";
>     }
>
> event Input::update_finished(name: string, source: string)
>     {
>         # now all data is in the table
>     print "Updated";
>         print Try::blacklist;
>     }
>
>
> ----------------------------bl.txt---------------------------------------------
>
> #fields ip    timestamp    reason
> #types    addr    time    string
> 192.168.17.1    1333252748    Malware host
> 192.168.27.2    1330235733    Botnet server
> 192.168.250.3    1333145108    Virus detected
>
>
> ---------------------------------------------------------------------------------
> Initially, i tried "bro -r file.pcap try.bro" but it didn't work. To
> provide ample time for reading in the blacklist, i tried "bro -i eth0
> try.bro". The output displays hello and bye but the blacklist wasn't
> printed even after 5 minutes. I tried giving the absolute source path i.e.
> "/home/myname/bl.txt" but to no avail.
>
> Moreover, i purposely gave a wrong input source file and no error was
> displayed. I feel an appropriate error message will be helpful if someone
> has mistyped the source file name or if it doesn't exist.
>
> Regards,
> --
> Sheharbano Khattak
>
> http://etheryell.com
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>


-- 
Sheharbano Khattak

http://etheryell.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120629/a7e0d2fa/attachment.html 

From bernhard at ICSI.Berkeley.EDU  Thu Jun 28 12:32:16 2012
From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann)
Date: Thu, 28 Jun 2012 12:32:16 -0700
Subject: [Bro] Playing with the input framework
In-Reply-To: <CAA6v-1bLpAFa=K7VUKnVAKCWXR2_RC8AJ+FgjsHw4zcQ4MRhSQ@mail.gmail.com>
References: <CAA6v-1aMTP8y-miEkhHytJhtf0DS-g_XFZP-ky7S4p28U-pJwQ@mail.gmail.com>
	<C8575418-A865-43C2-9619-0AF7694E3D35@icsi.berkeley.edu>
	<CAA6v-1bLpAFa=K7VUKnVAKCWXR2_RC8AJ+FgjsHw4zcQ4MRhSQ@mail.gmail.com>
Message-ID: <EFEE72A0-0EC4-49E4-B6FC-F285D80CF53A@ICSI.Berkeley.EDU>

Nearly everything ends up in reporter.log.

The only things (I know of) that are directly reported to the command line are fatal errors like wrong command line switches or syntax errors in scripts that are encountered while parsing them. 

Bernhard

On Jun 28, 2012, at 12:28 PM, Sheharbano Khattak wrote:

> Thank you. Indeed there was a tabulation problem. On a side note, how do you decide which errors are directed to the console and which ones are reported in reporter.log ?
> 
> Regards,
> 
> On Fri, Jun 29, 2012 at 12:14 AM, Bernhard Amann <bernhard at icsi.berkeley.edu> wrote:
> Hello Sheharabano,
> 
> I just tried your example and it seems to work fine. Please note that the fields in the file "bl.txt" need to be separated with tabulators - including the header lines. If you simply copied the example from the website it probably ended up being separated with spaces.
> 
> The input frameworks outputs error messages. These are written into the file reporter.log (where most Bro error messages end up).
> 
> If the file contains a line that looks like 
> "Reporter::ERROR	InputReader/bl.txt: Not enough fields in line?" 
> the framework is complaining about missing tabulators-fields (because it is not finding enough tab-separated entries in the line it read).
> 
> I hope that helps - if it still does not solve your problem please write again.
> 
> Bernhard
> 
> 
> On Jun 28, 2012, at 12:06 PM, Sheharbano Khattak wrote:
> 
>> Hi,
>> 
>> I recently finished reading about the new input framework http://www.icsi.berkeley.edu/~bernhard/papers/loneWolf.pdf and found it very interesting. As a first step, i tried implementing the example about reading data into tables mentioned here http://blog.bro-ids.org/2012/06/upcoming-loading-data-into-bro-with.html. My bro and source blacklist file look like this:
>> 
>> ---------------------------------try.bro----------------------------------------------
>> module Try;
>> 
>> type Idx: record {
>>         ip: addr;
>> };
>> 
>> type Val: record {
>>         timestamp: time;
>>         reason: string;
>> };
>> 
>> global blacklist: table[addr] of Val = table();
>> 
>> event bro_init() 
>>     {
>>     print "hello";
>>     Input::add_table([$source="bl.txt", $name="bl_stream", $idx=Idx, $val=Val, $destination=Try::blacklist]);
>>     Input::remove("bl_stream");    
>>     print "bye";
>>     }
>> 
>> event Input::update_finished(name: string, source: string) 
>>     {
>>         # now all data is in the table
>>     print "Updated";
>>         print Try::blacklist;
>>     }
>> 
>> ----------------------------bl.txt---------------------------------------------
>> 
>> #fields ip    timestamp    reason
>> #types    addr    time    string
>> 192.168.17.1    1333252748    Malware host
>> 192.168.27.2    1330235733    Botnet server
>> 192.168.250.3    1333145108    Virus detected
>> 
>> ---------------------------------------------------------------------------------
>> Initially, i tried "bro -r file.pcap try.bro" but it didn't work. To provide ample time for reading in the blacklist, i tried "bro -i eth0 try.bro". The output displays hello and bye but the blacklist wasn't printed even after 5 minutes. I tried giving the absolute source path i.e. "/home/myname/bl.txt" but to no avail. 
>> 
>> Moreover, i purposely gave a wrong input source file and no error was displayed. I feel an appropriate error message will be helpful if someone has mistyped the source file name or if it doesn't exist. 
>> 
>> Regards,
>> -- 
>> Sheharbano Khattak
>> 
>> http://etheryell.com
>> 
>> 
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> 
> 
> -- 
> Sheharbano Khattak
> 
> http://etheryell.com
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120628/ed611d24/attachment.html 

From sheharbano.k at gmail.com  Thu Jun 28 23:47:50 2012
From: sheharbano.k at gmail.com (Sheharbano Khattak)
Date: Fri, 29 Jun 2012 11:47:50 +0500
Subject: [Bro] Re-reading data in the input framework
Message-ID: <CAA6v-1a2EiOj7N0-ada_k1eGesM9FtowrRy5870Ck_7FqDX9YQ@mail.gmail.com>

Hi,

I want to test if a table that holds data from an input source file with
the automatic refresh mode "REREAD" reflects changes applied to the source
file. This is what my file looks like

-----------------------------config.bro---------------------------------
module Config;

type Idx: record {
        parameter: string;
};

type Val: record {
        value: string;
};

export {
    global table_config: table[string] of Val;
}

global config_filename = "/usr/local/bro/share/bro/site/botflex/config.txt";

event bro_init() &priority=20
    {
    Input::add_table([$source=config_filename, $name="config_stream",
$idx=Idx,
              $val=Val, $destination=table_config, $mode=Input::REREAD]);
    Input::remove("config_stream");
    }

event Input::update_finished(name: string, source: string)
    {
    # now all data is in the table
    print "Updated";
    print table_config;
    }

event bro_done()
    {
    print "Ending";
    print table_config;
    }
-------------------------------------------------------------------------------------------------------

I do bro -i eth0 config.bro. The first time, the event
Input::update_finished is triggered with the values/indices defined in
config.txt.
Then i manually make some changes in config.txt and save the changes but
Input::update_finished is not triggered. After waiting for several minutes,
i ctrl+c Bro hoping to see the modification in table_config, still it shows
the old values. There is nothing interesting in reporter.log either. What
am i doing wrong?

Thanks.
-- 
Sheharbano Khattak

http://etheryell.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120629/7f9cf7a7/attachment.html 

From bernhard at ICSI.Berkeley.EDU  Fri Jun 29 01:26:09 2012
From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann)
Date: Fri, 29 Jun 2012 01:26:09 -0700
Subject: [Bro] Re-reading data in the input framework
In-Reply-To: <CAA6v-1a2EiOj7N0-ada_k1eGesM9FtowrRy5870Ck_7FqDX9YQ@mail.gmail.com>
References: <CAA6v-1a2EiOj7N0-ada_k1eGesM9FtowrRy5870Ck_7FqDX9YQ@mail.gmail.com>
Message-ID: <31BE3A39-CFA9-4060-B4D4-7DDB59B0125C@icsi.berkeley.edu>

Hello,

my tutorial probably was not very clear on this point. The line -- Input::remove("config_stream"); -- removes the input stream after the current input stream operation has been executed. Thus, in your script the input stream is closed right after the file has been read for the first time. After the stream is closed, changes to the input file will not be reflected in Bro, no matter the mode you chose.

Removing that single line should fix the problem :)

Bernhard

On Jun 28, 2012, at 11:47 PM, Sheharbano Khattak wrote:

> Hi,
> 
> I want to test if a table that holds data from an input source file with the automatic refresh mode "REREAD" reflects changes applied to the source file. This is what my file looks like
> 
> -----------------------------config.bro---------------------------------
> module Config;
> 
> type Idx: record {
>         parameter: string;
> };
> 
> type Val: record {
>         value: string;
> };
> 
> export {
>     global table_config: table[string] of Val;
> }
> 
> global config_filename = "/usr/local/bro/share/bro/site/botflex/config.txt";
> 
> event bro_init() &priority=20
>     {
>     Input::add_table([$source=config_filename, $name="config_stream", $idx=Idx, 
>               $val=Val, $destination=table_config, $mode=Input::REREAD]);
>     Input::remove("config_stream");
>     }
> 
> event Input::update_finished(name: string, source: string) 
>     {
>     # now all data is in the table
>     print "Updated";
>     print table_config;
>     }
> 
> event bro_done()
>     {
>     print "Ending";
>     print table_config;
>     }
> -------------------------------------------------------------------------------------------------------
> 
> I do bro -i eth0 config.bro. The first time, the event  Input::update_finished is triggered with the values/indices defined in config.txt.
> Then i manually make some changes in config.txt and save the changes but Input::update_finished is not triggered. After waiting for several minutes, i ctrl+c Bro hoping to see the modification in table_config, still it shows the old values. There is nothing interesting in reporter.log either. What am i doing wrong?
> 
> Thanks.
> -- 
> Sheharbano Khattak
> 
> http://etheryell.com
> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120629/3a080ab9/attachment.html 

From sheharbano.k at gmail.com  Fri Jun 29 01:45:51 2012
From: sheharbano.k at gmail.com (Sheharbano Khattak)
Date: Fri, 29 Jun 2012 01:45:51 -0700
Subject: [Bro] Re-reading data in the input framework
In-Reply-To: <31BE3A39-CFA9-4060-B4D4-7DDB59B0125C@icsi.berkeley.edu>
References: <CAA6v-1a2EiOj7N0-ada_k1eGesM9FtowrRy5870Ck_7FqDX9YQ@mail.gmail.com>
	<31BE3A39-CFA9-4060-B4D4-7DDB59B0125C@icsi.berkeley.edu>
Message-ID: <CAA6v-1bUYOqNG+Ro6H6TWXsxBJuxgvJWptb2cHoWkdJ_+FSNyw@mail.gmail.com>

Thanks. I thought it creates a new stream for each read/remove. From your
answer, it appears that each source with REREAD mode gets a dedicated
stream that exists as long as Bro runs.

Regards,

On Fri, Jun 29, 2012 at 1:26 AM, Bernhard Amann
<bernhard at icsi.berkeley.edu>wrote:

> Hello,
>
> my tutorial probably was not very clear on this point. The line
> -- Input::remove("config_stream"); -- removes the input stream after the
> current input stream operation has been executed. Thus, in your script the
> input stream is closed right after the file has been read for the first
> time. After the stream is closed, changes to the input file will not be
> reflected in Bro, no matter the mode you chose.
>
> Removing that single line should fix the problem :)
>
> Bernhard
>
> On Jun 28, 2012, at 11:47 PM, Sheharbano Khattak wrote:
>
> Hi,
>
> I want to test if a table that holds data from an input source file with
> the automatic refresh mode "REREAD" reflects changes applied to the source
> file. This is what my file looks like
>
> -----------------------------config.bro---------------------------------
> module Config;
>
> type Idx: record {
>         parameter: string;
> };
>
> type Val: record {
>         value: string;
> };
>
> export {
>     global table_config: table[string] of Val;
> }
>
> global config_filename =
> "/usr/local/bro/share/bro/site/botflex/config.txt";
>
> event bro_init() &priority=20
>     {
>     Input::add_table([$source=config_filename, $name="config_stream",
> $idx=Idx,
>               $val=Val, $destination=table_config, $mode=Input::REREAD]);
>     Input::remove("config_stream");
>     }
>
> event Input::update_finished(name: string, source: string)
>     {
>     # now all data is in the table
>     print "Updated";
>     print table_config;
>     }
>
> event bro_done()
>     {
>     print "Ending";
>     print table_config;
>     }
>
> -------------------------------------------------------------------------------------------------------
>
> I do bro -i eth0 config.bro. The first time, the event
> Input::update_finished is triggered with the values/indices defined in
> config.txt.
> Then i manually make some changes in config.txt and save the changes but
> Input::update_finished is not triggered. After waiting for several minutes,
> i ctrl+c Bro hoping to see the modification in table_config, still it shows
> the old values. There is nothing interesting in reporter.log either. What
> am i doing wrong?
>
> Thanks.
> --
> Sheharbano Khattak
>
> http://etheryell.com
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>


-- 
Sheharbano Khattak

http://etheryell.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120629/259c99ec/attachment.html 

From bernhard at ICSI.Berkeley.EDU  Fri Jun 29 07:04:14 2012
From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann)
Date: Fri, 29 Jun 2012 07:04:14 -0700
Subject: [Bro] Re-reading data in the input framework
In-Reply-To: <CAA6v-1bUYOqNG+Ro6H6TWXsxBJuxgvJWptb2cHoWkdJ_+FSNyw@mail.gmail.com>
References: <CAA6v-1a2EiOj7N0-ada_k1eGesM9FtowrRy5870Ck_7FqDX9YQ@mail.gmail.com>
	<31BE3A39-CFA9-4060-B4D4-7DDB59B0125C@icsi.berkeley.edu>
	<CAA6v-1bUYOqNG+Ro6H6TWXsxBJuxgvJWptb2cHoWkdJ_+FSNyw@mail.gmail.com>
Message-ID: <EA9C2CD6-0733-44F9-899C-7802CCBC0677@ICSI.Berkeley.EDU>


On Jun 29, 2012, at 1:45 AM, Sheharbano Khattak wrote:

> Thanks. I thought it creates a new stream for each read/remove. From your answer, it appears that each source with REREAD mode gets a dedicated stream that exists as long as Bro runs.

Exactly. The stream exists for the whole duration and the reader for the stream runs in a thread and checks for changes of the file.

Bernhard