From hammadog at gmail.com  Thu May  3 18:10:40 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Thu, 3 May 2012 21:10:40 -0400
Subject: [Bro] Packet Drops
Message-ID: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>

Need some thoughts from the LINUX/BRO gifted....

Hardware:

CPU: two - Intel(R) Xeon(TM) CPU 2.40GHz
MEM: 2gig
NIC's: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI

We  peak around 130mbps and at this time we are running around 10mbps.
 No matter what speed we run at we continue to drop packets.  We have
loaded pf_ring and load balanced across two NIC's based on Martin's
BLOG:  http://ossectools.blogspot.com/2011/09/bro-quickstart-cluster-edition.html

Only change I made was use an additional NIC in the node.cfg and not
the same one.

I have also made the follwoing NIC changes based on some threads I
found on SO and BRO lists.

ethtool -K eth0 rx off
ethtool -K eth0 tx off
ethtool -K eth0 sg off
ethtool -K eth0 tso off
ethtool -K eth0 gso off
ethtool -K eth0 lro off

and

echo 33554432 > /proc/sys/net/core/rmem_default
echo 33554432 > /proc/sys/net/core/rmem_max
echo 10000 > /proc/sys/net/core/netdev_max_backlog

as well as

Changed the MTU size on NIC's to match BRO

Still no love. I then went back to a standalone setup and the packet
drops are not as bad, but again we are running very low bandwidth at
this time.  Any ideas? Update NIC maybe?  Drop Kick G200 in dumpster!
:)

Thanks

Tom


From JAzoff at albany.edu  Thu May  3 18:21:20 2012
From: JAzoff at albany.edu (Justin Azoff)
Date: Thu, 3 May 2012 21:21:20 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
Message-ID: <20120504012120.GY23613@datacomm.albany.edu>

On Thu, May 03, 2012 at 09:10:40PM -0400, Tom OBrion wrote:
> Need some thoughts from the LINUX/BRO gifted....
> 
> Hardware:
> 
> CPU: two - Intel(R) Xeon(TM) CPU 2.40GHz
> MEM: 2gig
> NIC's: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
> 
> We  peak around 130mbps and at this time we are running around 10mbps.
>  No matter what speed we run at we continue to drop packets.  We have
> loaded pf_ring and load balanced across two NIC's based on Martin's
> BLOG:  http://ossectools.blogspot.com/2011/09/bro-quickstart-cluster-edition.html

Can you post the contents of the files in /proc/net/pf_ring/ for the bro
processes?  You should have one per bro worker.


-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From mcholste at gmail.com  Thu May  3 21:26:08 2012
From: mcholste at gmail.com (Martin Holste)
Date: Thu, 3 May 2012 23:26:08 -0500
Subject: [Bro] Packet Drops
In-Reply-To: <20120504012120.GY23613@datacomm.albany.edu>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
Message-ID: <CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>

On moderate hardware, I've found that it takes about one CPU per 100
Mb/sec, so you shouldn't be dropping at anything under that.  You
probably also don't need PF_RING or any special kernel tunings at
anything less than 200-300 Mb/sec, so that shouldn't be the problem
either.  When you say dropped packets, is that per the Bro drop log,
or the nic stats?

On Thu, May 3, 2012 at 8:21 PM, Justin Azoff <JAzoff at albany.edu> wrote:
> On Thu, May 03, 2012 at 09:10:40PM -0400, Tom OBrion wrote:
>> Need some thoughts from the LINUX/BRO gifted....
>>
>> Hardware:
>>
>> CPU: two - Intel(R) Xeon(TM) CPU 2.40GHz
>> MEM: 2gig
>> NIC's: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
>>
>> We ?peak around 130mbps and at this time we are running around 10mbps.
>> ?No matter what speed we run at we continue to drop packets. ?We have
>> loaded pf_ring and load balanced across two NIC's based on Martin's
>> BLOG: ?http://ossectools.blogspot.com/2011/09/bro-quickstart-cluster-edition.html
>
> Can you post the contents of the files in /proc/net/pf_ring/ for the bro
> processes? ?You should have one per bro worker.
>
>
> --
> -- Justin Azoff
> -- Network Security & Performance Analyst
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From bro at wressnegger.info  Thu May  3 23:41:51 2012
From: bro at wressnegger.info (Chris)
Date: Fri, 4 May 2012 08:41:51 +0200
Subject: [Bro] Signature Matching Performance
Message-ID: <2F81E96D-D075-4DAB-B663-1529C20CE04B@wressnegger.info>

Hi,
currently I'm dealing with quite a lot (enough to significantly impact the runtime performance) of signatures in my Bro setup. I understand that signature matching isn't part of Bro's main focus; after reading a response of Robin to the mailing list from 2010 (http://mailman.icsi.berkeley.edu/pipermail/bro/2010-October/004621.html) made me abandon this illusion ;)
However, I wonder if there is a way to speed up things a little? Some points that come to my mind are:
- obviously reduce the number of signatures
- properly anchor the signatures rather than prefixing them with ".*" This seems to be the critical point in my situation. So if you have ideas how to resolve this without giving up matching at arbritrary positions.... ;)
- clusters of Bro instances
- ...

Thanks, Chris.





From seth at icir.org  Fri May  4 00:04:46 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 4 May 2012 03:04:46 -0400
Subject: [Bro] Signature Matching Performance
In-Reply-To: <2F81E96D-D075-4DAB-B663-1529C20CE04B@wressnegger.info>
References: <2F81E96D-D075-4DAB-B663-1529C20CE04B@wressnegger.info>
Message-ID: <BFB582DD-241A-470F-A36A-BE3AA05A94B1@icir.org>


On May 4, 2012, at 2:41 AM, Chris wrote:

> - properly anchor the signatures rather than prefixing them with ".*" This seems to be the critical point in my situation. So if you have ideas how to resolve this without giving up matching at arbritrary positions.... ;)

Could you give us some example signatures?  If they have private data in them, you could defang them a little bit, I'm only asking so that we can see more about how you are using signatures.  In general though, lots of signatures with .* at the beginning are going to be really, really bad.

> - clusters of Bro instances

That's always an option, but it may be more worthwhile to find out if you are using signatures for an appropriate task first. :)

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From bro at wressnegger.info  Fri May  4 00:21:14 2012
From: bro at wressnegger.info (Chris)
Date: Fri, 4 May 2012 09:21:14 +0200
Subject: [Bro] Signature Matching Performance
In-Reply-To: <BFB582DD-241A-470F-A36A-BE3AA05A94B1@icir.org>
References: <2F81E96D-D075-4DAB-B663-1529C20CE04B@wressnegger.info>
	<BFB582DD-241A-470F-A36A-BE3AA05A94B1@icir.org>
Message-ID: <EB593254-8DDE-4BDA-94E7-1D6D9899E23C@wressnegger.info>

On 04.05.2012, at 09:04, Seth Hall <seth at icir.org> wrote:
> On May 4, 2012, at 2:41 AM, Chris wrote:
> 
>> - properly anchor the signatures rather than prefixing them with ".*" This seems to be the critical point in my situation. So if you have ideas how to resolve this without giving up matching at arbritrary positions.... ;)
> 
> Could you give us some example signatures?  If they have private data in them, you could defang them a little bit, I'm only asking so that we can see more about how you are using signatures.  In general though, lots of signatures with .* at the beginning are going to be really, really bad.

Hi,
pretty much any signature is prefixed with .* followed by a potentially short body of actual signature data. The reason for this is that those signatures are automatically generated and do not have much information about their location within the payload. Hence, it might happen that e.g. a signature for a http request type might end up as /.*GET\ / although that obviously isn't what one would usually go for.
Seems like I'm misusing the concept of the signature engine a bit hehe

Chris.



From seth at icir.org  Fri May  4 00:27:36 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 4 May 2012 03:27:36 -0400
Subject: [Bro] Signature Matching Performance
In-Reply-To: <EB593254-8DDE-4BDA-94E7-1D6D9899E23C@wressnegger.info>
References: <2F81E96D-D075-4DAB-B663-1529C20CE04B@wressnegger.info>
	<BFB582DD-241A-470F-A36A-BE3AA05A94B1@icir.org>
	<EB593254-8DDE-4BDA-94E7-1D6D9899E23C@wressnegger.info>
Message-ID: <DF89DD91-90FA-47E8-82C6-3C9FDAF83A4A@icir.org>


On May 4, 2012, at 3:21 AM, Chris wrote:

> Hence, it might happen that e.g. a signature for a http request type might end up as /.*GET\ / although that obviously isn't what one would usually go for.

Are you trying to auto convert snort signatures?  

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From bro at wressnegger.info  Fri May  4 01:00:59 2012
From: bro at wressnegger.info (Chris)
Date: Fri, 4 May 2012 10:00:59 +0200
Subject: [Bro] Signature Matching Performance
In-Reply-To: <DF89DD91-90FA-47E8-82C6-3C9FDAF83A4A@icir.org>
References: <2F81E96D-D075-4DAB-B663-1529C20CE04B@wressnegger.info>
	<BFB582DD-241A-470F-A36A-BE3AA05A94B1@icir.org>
	<EB593254-8DDE-4BDA-94E7-1D6D9899E23C@wressnegger.info>
	<DF89DD91-90FA-47E8-82C6-3C9FDAF83A4A@icir.org>
Message-ID: <9B587D72-C5D4-417D-B8CA-B8F2CB9CADF9@wressnegger.info>

On 04.05.2012, at 09:27, Seth Hall <seth at icir.org> wrote:

> On May 4, 2012, at 3:21 AM, Chris wrote:
> 
>> Hence, it might happen that e.g. a signature for a http request type might end up as /.*GET\ / although that obviously isn't what one would usually go for.
> 
> Are you trying to auto convert snort signatures?

No, I'm just trying to generate some signatures on my own. With a lot of leading .* hehe
Well I guess I have to rethink the idea a bit ;)

Thanks, Chris.



From rmkml at yahoo.fr  Fri May  4 01:37:44 2012
From: rmkml at yahoo.fr (rmkml at yahoo.fr)
Date: Fri, 4 May 2012 08:37:44 +0000 (UTC)
Subject: [Bro] RE : Re:  Signature Matching Performance
Message-ID: <1013339355.9847.1336120726868.JavaMail.seven@ap3.p2.fra.samsungsocialhub.net>

An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120504/6ae881d5/attachment.html 

From hammadog at gmail.com  Fri May  4 03:21:35 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Fri, 4 May 2012 06:21:35 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
Message-ID: <CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>

That is from the netstats via Bro.  Zero dropped packets via Nic stats.

Here is some stats from this AM with the following setup.

root at ptlsecsensor1:/home/secarch# /usr/local/bro/bin/broctl capstats

Interface            kpps       mbps       (10s average)
------------------------------
x.x.x.x/eth0     3.8        20.0

Total                3.8        20.0

worker-0: 1336126625.749682 recvd=263871 dropped=30023 link=293912
worker-1: 1336126625.997021 recvd=262510 dropped=30656 link=293227

[manager]
type=manager
host=x.x.x.x

[proxy-0]
type=proxy
host=x.x.x.x

[worker-0]
type=worker
host=x.x.x.x
interface=eth0

[worker-1]
type=worker
host=x.x.x.x
interface=eth0

We were unsure as the documentation mentioned 80mbps per CPU, so we
thought we would give pf_ring a run.  But at these rates I would not
think we would see drops.

Is netstats not telling the truth?  :)

We are just trying to get an idea of what these old IBM hardware can
do for us and are running into this.

Thanks very much for the assistance.

Tom



On Fri, May 4, 2012 at 12:26 AM, Martin Holste <mcholste at gmail.com> wrote:
> On moderate hardware, I've found that it takes about one CPU per 100
> Mb/sec, so you shouldn't be dropping at anything under that. ?You
> probably also don't need PF_RING or any special kernel tunings at
> anything less than 200-300 Mb/sec, so that shouldn't be the problem
> either. ?When you say dropped packets, is that per the Bro drop log,
> or the nic stats?
>
> On Thu, May 3, 2012 at 8:21 PM, Justin Azoff <JAzoff at albany.edu> wrote:
>> On Thu, May 03, 2012 at 09:10:40PM -0400, Tom OBrion wrote:
>>> Need some thoughts from the LINUX/BRO gifted....
>>>
>>> Hardware:
>>>
>>> CPU: two - Intel(R) Xeon(TM) CPU 2.40GHz
>>> MEM: 2gig
>>> NIC's: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
>>>
>>> We ?peak around 130mbps and at this time we are running around 10mbps.
>>> ?No matter what speed we run at we continue to drop packets. ?We have
>>> loaded pf_ring and load balanced across two NIC's based on Martin's
>>> BLOG: ?http://ossectools.blogspot.com/2011/09/bro-quickstart-cluster-edition.html
>>
>> Can you post the contents of the files in /proc/net/pf_ring/ for the bro
>> processes? ?You should have one per bro worker.
>>
>>
>> --
>> -- Justin Azoff
>> -- Network Security & Performance Analyst
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Tom O'Brion
TEL: 207.210.2167
Skype:

"Life is too short to spend time with people who suck the happy out of you."



From hammadog at gmail.com  Fri May  4 03:28:46 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Fri, 4 May 2012 06:28:46 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
Message-ID: <CANCSO1d-+4UQZuqT0GXYTfq1GRufjexbWV70FdKKBtOezb+kiA@mail.gmail.com>

Via tcpdump

1995 packets captured
1995 packets received by filter
14731 packets dropped by kernel

On Fri, May 4, 2012 at 12:26 AM, Martin Holste <mcholste at gmail.com> wrote:
> On moderate hardware, I've found that it takes about one CPU per 100
> Mb/sec, so you shouldn't be dropping at anything under that. ?You
> probably also don't need PF_RING or any special kernel tunings at
> anything less than 200-300 Mb/sec, so that shouldn't be the problem
> either. ?When you say dropped packets, is that per the Bro drop log,
> or the nic stats?
>
> On Thu, May 3, 2012 at 8:21 PM, Justin Azoff <JAzoff at albany.edu> wrote:
>> On Thu, May 03, 2012 at 09:10:40PM -0400, Tom OBrion wrote:
>>> Need some thoughts from the LINUX/BRO gifted....
>>>
>>> Hardware:
>>>
>>> CPU: two - Intel(R) Xeon(TM) CPU 2.40GHz
>>> MEM: 2gig
>>> NIC's: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
>>>
>>> We ?peak around 130mbps and at this time we are running around 10mbps.
>>> ?No matter what speed we run at we continue to drop packets. ?We have
>>> loaded pf_ring and load balanced across two NIC's based on Martin's
>>> BLOG: ?http://ossectools.blogspot.com/2011/09/bro-quickstart-cluster-edition.html
>>
>> Can you post the contents of the files in /proc/net/pf_ring/ for the bro
>> processes? ?You should have one per bro worker.
>>
>>
>> --
>> -- Justin Azoff
>> -- Network Security & Performance Analyst
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Tom O'Brion
TEL: 207.210.2167
Skype:

"Life is too short to spend time with people who suck the happy out of you."



From hammadog at gmail.com  Fri May  4 03:29:27 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Fri, 4 May 2012 06:29:27 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
Message-ID: <CANCSO1f5USEpZdju9uqt8iFL2N5=u49njtW-st4Po+j-3bJy5Q@mail.gmail.com>

meant to replay all.

This is via tcpdump this morning.

1995 packets captured
1995 packets received by filter
14731 packets dropped by kernel

On Fri, May 4, 2012 at 12:26 AM, Martin Holste <mcholste at gmail.com> wrote:
> On moderate hardware, I've found that it takes about one CPU per 100
> Mb/sec, so you shouldn't be dropping at anything under that. ?You
> probably also don't need PF_RING or any special kernel tunings at
> anything less than 200-300 Mb/sec, so that shouldn't be the problem
> either. ?When you say dropped packets, is that per the Bro drop log,
> or the nic stats?
>
> On Thu, May 3, 2012 at 8:21 PM, Justin Azoff <JAzoff at albany.edu> wrote:
>> On Thu, May 03, 2012 at 09:10:40PM -0400, Tom OBrion wrote:
>>> Need some thoughts from the LINUX/BRO gifted....
>>>
>>> Hardware:
>>>
>>> CPU: two - Intel(R) Xeon(TM) CPU 2.40GHz
>>> MEM: 2gig
>>> NIC's: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
>>>
>>> We ?peak around 130mbps and at this time we are running around 10mbps.
>>> ?No matter what speed we run at we continue to drop packets. ?We have
>>> loaded pf_ring and load balanced across two NIC's based on Martin's
>>> BLOG: ?http://ossectools.blogspot.com/2011/09/bro-quickstart-cluster-edition.html
>>
>> Can you post the contents of the files in /proc/net/pf_ring/ for the bro
>> processes? ?You should have one per bro worker.
>>
>>
>> --
>> -- Justin Azoff
>> -- Network Security & Performance Analyst
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Tom O'Brion
TEL: 207.210.2167
Skype:

"Life is too short to spend time with people who suck the happy out of you."



From hammadog at gmail.com  Fri May  4 03:37:50 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Fri, 4 May 2012 06:37:50 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
Message-ID: <CANCSO1fuaBQ8q3fUJffA6RjwmtnzRVtnJwzWiZtsy0xCpYewGA@mail.gmail.com>

I stop bro and then run tcpdump on the same interface and I get no drops.

Time to retire this old gear..:)

On Fri, May 4, 2012 at 12:26 AM, Martin Holste <mcholste at gmail.com> wrote:
> On moderate hardware, I've found that it takes about one CPU per 100
> Mb/sec, so you shouldn't be dropping at anything under that. ?You
> probably also don't need PF_RING or any special kernel tunings at
> anything less than 200-300 Mb/sec, so that shouldn't be the problem
> either. ?When you say dropped packets, is that per the Bro drop log,
> or the nic stats?
>
> On Thu, May 3, 2012 at 8:21 PM, Justin Azoff <JAzoff at albany.edu> wrote:
>> On Thu, May 03, 2012 at 09:10:40PM -0400, Tom OBrion wrote:
>>> Need some thoughts from the LINUX/BRO gifted....
>>>
>>> Hardware:
>>>
>>> CPU: two - Intel(R) Xeon(TM) CPU 2.40GHz
>>> MEM: 2gig
>>> NIC's: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
>>>
>>> We ?peak around 130mbps and at this time we are running around 10mbps.
>>> ?No matter what speed we run at we continue to drop packets. ?We have
>>> loaded pf_ring and load balanced across two NIC's based on Martin's
>>> BLOG: ?http://ossectools.blogspot.com/2011/09/bro-quickstart-cluster-edition.html
>>
>> Can you post the contents of the files in /proc/net/pf_ring/ for the bro
>> processes? ?You should have one per bro worker.
>>
>>
>> --
>> -- Justin Azoff
>> -- Network Security & Performance Analyst
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From JAzoff at albany.edu  Fri May  4 05:37:42 2012
From: JAzoff at albany.edu (Justin Azoff)
Date: Fri, 4 May 2012 08:37:42 -0400
Subject: [Bro] Signature Matching Performance
In-Reply-To: <9B587D72-C5D4-417D-B8CA-B8F2CB9CADF9@wressnegger.info>
References: <2F81E96D-D075-4DAB-B663-1529C20CE04B@wressnegger.info>
	<BFB582DD-241A-470F-A36A-BE3AA05A94B1@icir.org>
	<EB593254-8DDE-4BDA-94E7-1D6D9899E23C@wressnegger.info>
	<DF89DD91-90FA-47E8-82C6-3C9FDAF83A4A@icir.org>
	<9B587D72-C5D4-417D-B8CA-B8F2CB9CADF9@wressnegger.info>
Message-ID: <20120504123742.GC23613@datacomm.albany.edu>

On Fri, May 04, 2012 at 10:00:59AM +0200, Chris wrote:
> On 04.05.2012, at 09:27, Seth Hall <seth at icir.org> wrote:
> 
> > On May 4, 2012, at 3:21 AM, Chris wrote:
> > 
> >> Hence, it might happen that e.g. a signature for a http request type might end up as /.*GET\ / although that obviously isn't what one would usually go for.
> > 
> > Are you trying to auto convert snort signatures?
> 
> No, I'm just trying to generate some signatures on my own. With a lot of leading .* hehe
> Well I guess I have to rethink the idea a bit ;)

If you have a large list of urls what you want to do is generate a set
of those urls ...

    redef bad_urls += {
        "http://bad.example.com",
        "http://evil.example.com",
        ...
    }

then in a policy somewhere you can simply do

    if(url in bad_urls)
        ...

If you still need a regular expression then you can build up a single pattern
like this:

    redef bad_urls =
          /bad\.example\.com\/some_regex_here/
        | /evil\.example\.com/;

and then use it like this

    if(bad_urls in url)
        ...


both methods will be a huge improvement over building multiple
signatures.

-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From seth at icir.org  Fri May  4 06:58:44 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 4 May 2012 09:58:44 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
	<CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
Message-ID: <4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>


On May 4, 2012, at 6:21 AM, Tom OBrion wrote:

> worker-0: 1336126625.749682 recvd=263871 dropped=30023 link=293912
> worker-1: 1336126625.997021 recvd=262510 dropped=30656 link=293227

Are you running "misc/capture-loss"?  That should provide a much more holistic view of packet loss because it's not relying on anything other than characteristics of the actual traffic to tell you if packets are being lost.  It doesn't tell you where the packet loss is happening and could mean a very large number of things, but it's a good place to start.

> We were unsure as the documentation mentioned 80mbps per CPU, so we
> thought we would give pf_ring a run.  But at these rates I would not
> think we would see drops.

I was really conflicted when I wrote 80Mbps in that documentation.  There is really no good way to figure out what that will be.  With reasonably fast, modern Xeon CPUs people seem to be getting ~150Mbps per core now but you need to take value with a grain of salt too since it depends so heavily on your traffic mix 

> Is netstats not telling the truth?  :)

That question is really hard to answer, especially if you are running pf_ring where the normal Linux packet processing pipeline is being bypassed.

> We are just trying to get an idea of what these old IBM hardware can
> do for us and are running into this.

You didn't mention that it's old hardware. :)  What's the architecture?  How many cores does the box have total?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From brad.shoop at gmail.com  Fri May  4 08:23:09 2012
From: brad.shoop at gmail.com (Brad Shoop)
Date: Fri, 4 May 2012 15:23:09 +0000 (UTC)
Subject: [Bro] Analyzing and Visualizing Bro Logs with Splunk
References: <CAG1VgOb9sJrpsaPtufRSSXoNC7pmM+f_w6J=ACfg-i9jXAB0GQ@mail.gmail.com>
	<20120419153710.GO9794@datacomm.albany.edu>
Message-ID: <loom.20120504T171949-571@post.gmane.org>

Justin Azoff <JAzoff <at> albany.edu> writes:

> 
> On Thu, Apr 19, 2012 at 11:13:20AM -0400, Chris Crawford wrote:
> > Does anybody have the slides or video from "Analyzing and Visualizing
> > Bro Logs with Splunk" talk at Bro Workshop 2011?
> > 
> > -Chris
> 
> Hmm, I thought they were put on the website.. I was difficult and used
> the google HTML5 slideshow template 
> 
> The presentation is attached. Let me know if you have any questions.
> 
> The old metrics scripts I mention were indeed obsoleted by 2.0, but I've
> updated most of them:
> 
> https://github.com/JustinAzoff/bro_scripts/tree/2.0/
> 


If you want to get going quickly, download the Security Onion app
for Splunk and either install it (if it's not a Security Onion system,
you'll want to disable the SOstat scripts) or rename it to a .tar.gz
and extract. If you're already pulling Bro data in, you should be
able to match up the sourcetype names to the props/transforms.conf
then copy the props.conf and transforms.conf files to your Splunk
instance.

That will get you all the field extractions and data into Splunk, and
the Security Onion app will provide some initial dashboards and
panels to give you more ideas.

Brad Shoop



From srunnels at gmail.com  Fri May  4 08:24:04 2012
From: srunnels at gmail.com (scott runnels)
Date: Fri, 4 May 2012 11:24:04 -0400
Subject: [Bro] Learning the Bro Scripting Language Part 3 - Detecting basic
	auth and going from evidence to practical use in Bro
Message-ID: <FE76200D-BFA8-4E60-829A-B45BE0B7D4BB@gmail.com>

I sent the first post of the series to the mailing list and got a decent response from people who were interested in learning Bro's scripting language.  

This post tries to move to a more operationally useful script and something you could actually deploy across an enterprise.  If anyone has any comments or critiques I'd love to hear them.  

http://ryesecurity.blogspot.com/2012/05/learning-bro-scripting-language.html


v/r
Scott Runnels


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120504/6a119498/attachment.html 

From mcholste at gmail.com  Fri May  4 09:34:48 2012
From: mcholste at gmail.com (Martin Holste)
Date: Fri, 4 May 2012 11:34:48 -0500
Subject: [Bro] Learning the Bro Scripting Language Part 3 - Detecting
 basic auth and going from evidence to practical use in Bro
In-Reply-To: <FE76200D-BFA8-4E60-829A-B45BE0B7D4BB@gmail.com>
References: <FE76200D-BFA8-4E60-829A-B45BE0B7D4BB@gmail.com>
Message-ID: <CANpnLHjzJyYFT1Egvj0EAPMx0ZjyHCP-PvQ4L9KisLPT0e8OgQ@mail.gmail.com>

Great post!  Perfect cheat-sheet for incident responders that need to
add something specific.

On Fri, May 4, 2012 at 10:24 AM, scott runnels <srunnels at gmail.com> wrote:
> I sent the first post of the series to the mailing list and got a decent
> response from people who were interested in learning Bro's scripting
> language.
>
> This post tries to move to a more operationally useful script and something
> you could actually deploy across an enterprise. ?If anyone has any comments
> or critiques I'd love to hear them.
>
> http://ryesecurity.blogspot.com/2012/05/learning-bro-scripting-language.html
>
>
> v/r
> Scott Runnels
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From vallentin at icir.org  Fri May  4 12:09:57 2012
From: vallentin at icir.org (Matthias Vallentin)
Date: Fri, 4 May 2012 12:09:57 -0700
Subject: [Bro] Learning the Bro Scripting Language Part 3 - Detecting
 basic auth and going from evidence to practical use in Bro
In-Reply-To: <FE76200D-BFA8-4E60-829A-B45BE0B7D4BB@gmail.com>
References: <FE76200D-BFA8-4E60-829A-B45BE0B7D4BB@gmail.com>
Message-ID: <CADTpMNa8eyPeT5RF2rsZPO46j+G10C2VDdr9Qhx3p3G3CdH06g@mail.gmail.com>

> I sent the first post of the series to the mailing list and got a
> decent response from people who were interested in learning Bro's
> scripting language.

Nice work, Scott!

One small comment: "Three lines of Bro's scripting language and we can
detect a server using Basic Access Authentication!"

It's actually just one line [1]:

    redef HTTP::default_capture_password = T;

This automatically creates a new column password in the http.log with
the password value, if available.

Keep the posts coming!

    Matthias

[1] http://git.bro-ids.org/bro.git/blob/HEAD:/scripts/base/protocols/http/main.bro#l233


From srunnels at gmail.com  Fri May  4 12:24:59 2012
From: srunnels at gmail.com (scott runnels)
Date: Fri, 4 May 2012 15:24:59 -0400
Subject: [Bro] Learning the Bro Scripting Language Part 3 - Detecting
	basic auth and going from evidence to practical use in Bro
In-Reply-To: <CADTpMNa8eyPeT5RF2rsZPO46j+G10C2VDdr9Qhx3p3G3CdH06g@mail.gmail.com>
References: <FE76200D-BFA8-4E60-829A-B45BE0B7D4BB@gmail.com>
	<CADTpMNa8eyPeT5RF2rsZPO46j+G10C2VDdr9Qhx3p3G3CdH06g@mail.gmail.com>
Message-ID: <0E4D572F-9A1D-4FA2-B78E-7A2DC6B1C42D@gmail.com>

Hi Mattias, 

Thank you!  

Yea, that's definitely a little misleading on my part.  I tried to touch on the fact that "Hey, Bro really does this kind of stuff under the hood!" I actually saw the username getting parsed out when I was dumping the connection getting passed into http_header and sent some colorful language at Seth over IM  :)

I'm hoping to try to get as many posts up as I can think of.  I've been working pretty closely with Seth to make sure that I don't do something 'unbroly', that I stick to the already established conventions, and to make sure I don't go about spreading any misinformation.  It's been a great learning experience.  I'll reiterate what I said the post, "Some day, I'll stop being shocked by everything Bro does and just accept that it's wall-to-wall awesome!"  Kind of hard sometimes, though!

v/r
Scott Runnels

On May 4, 2012, at 3:09 PM, Matthias Vallentin wrote:

>> I sent the first post of the series to the mailing list and got a
>> decent response from people who were interested in learning Bro's
>> scripting language.
> 
> Nice work, Scott!
> 
> One small comment: "Three lines of Bro's scripting language and we can
> detect a server using Basic Access Authentication!"
> 
> It's actually just one line [1]:
> 
>    redef HTTP::default_capture_password = T;
> 
> This automatically creates a new column password in the http.log with
> the password value, if available.
> 
> Keep the posts coming!
> 
>    Matthias
> 
> [1] http://git.bro-ids.org/bro.git/blob/HEAD:/scripts/base/protocols/http/main.bro#l233




From vallentin at icir.org  Fri May  4 12:39:50 2012
From: vallentin at icir.org (Matthias Vallentin)
Date: Fri, 4 May 2012 12:39:50 -0700
Subject: [Bro] Learning the Bro Scripting Language Part 3 - Detecting
 basic auth and going from evidence to practical use in Bro
In-Reply-To: <0E4D572F-9A1D-4FA2-B78E-7A2DC6B1C42D@gmail.com>
References: <FE76200D-BFA8-4E60-829A-B45BE0B7D4BB@gmail.com>
	<CADTpMNa8eyPeT5RF2rsZPO46j+G10C2VDdr9Qhx3p3G3CdH06g@mail.gmail.com>
	<0E4D572F-9A1D-4FA2-B78E-7A2DC6B1C42D@gmail.com>
Message-ID: <CADTpMNZCbF2Y3RvfsNDi2zkPg4uzfdRupuf1EL5rOgeK1BjnfA@mail.gmail.com>

> I'm hoping to try to get as many posts up as I can think of.

Terrific!

> I've been working pretty closely with Seth to make sure that I don't
> do something 'unbroly', that I stick to the already established
> conventions, and to make sure I don't go about spreading any
> misinformation.

For sure, you're on the safe path with Seth on your side :-). Speaking
of conventions, one additional Bro idiom comes to mind. Maybe that's
already clear to you, even better then.

As you may have noticed, there are several boolean indicator flags in
the connection record. This introduces a new idiom: selectively enabling
or disabling certain analyses on a *per-connection basis*. For example,
you may only want to exclude logging passwords of users from a specific
subnet. All that this requires is setting HTTP::capture_password to true
for connections that do not originate from the corresponding subnet (or
if the reverse is easier, setting it to false for that specific subnet).

    Matthias


From hammadog at gmail.com  Sat May  5 04:10:28 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Sat, 5 May 2012 07:10:28 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
	<CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
	<4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
Message-ID: <1971884F-1E3F-4EDD-B523-AA0048CF3C53@gmail.com>

Thanks for reply Seth.   Was out of pocket all day yesterday but I will load up capture-loss and see what other details we get.  We got something weird going on. We have another g200 in another location seeing about the amount of traffic, same thing.   It is not running pf_ring.  The boxes I believe are dual processor zeon's old technology.  But no reason they should not Andre the traffic.  I might switch OS as Ubuntu server sometimes has flakey Nic drivers.   Then load up Bro by itself and see how that goes.   

Thanks again

Tom

Sent from my iPad

On May 4, 2012, at 9:58 AM, Seth Hall <seth at icir.org> wrote:

> 
> On May 4, 2012, at 6:21 AM, Tom OBrion wrote:
> 
>> worker-0: 1336126625.749682 recvd=263871 dropped=30023 link=293912
>> worker-1: 1336126625.997021 recvd=262510 dropped=30656 link=293227
> 
> Are you running "misc/capture-loss"?  That should provide a much more holistic view of packet loss because it's not relying on anything other than characteristics of the actual traffic to tell you if packets are being lost.  It doesn't tell you where the packet loss is happening and could mean a very large number of things, but it's a good place to start.
> 
>> We were unsure as the documentation mentioned 80mbps per CPU, so we
>> thought we would give pf_ring a run.  But at these rates I would not
>> think we would see drops.
> 
> I was really conflicted when I wrote 80Mbps in that documentation.  There is really no good way to figure out what that will be.  With reasonably fast, modern Xeon CPUs people seem to be getting ~150Mbps per core now but you need to take value with a grain of salt too since it depends so heavily on your traffic mix 
> 
>> Is netstats not telling the truth?  :)
> 
> That question is really hard to answer, especially if you are running pf_ring where the normal Linux packet processing pipeline is being bypassed.
> 
>> We are just trying to get an idea of what these old IBM hardware can
>> do for us and are running into this.
> 
> You didn't mention that it's old hardware. :)  What's the architecture?  How many cores does the box have total?
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
> 



From daltonporter at yahoo.com  Mon May  7 11:14:32 2012
From: daltonporter at yahoo.com (Dalton Porter)
Date: Mon, 7 May 2012 11:14:32 -0700 (PDT)
Subject: [Bro] bro operational questions
Message-ID: <1336414472.79944.YahooMailNeo@web120703.mail.ne1.yahoo.com>

Hi,
I need to keep bro up and running to process logs continuously.? I was wondering what folks would suggest for doing that.? Does broctl automatically restart the process if it dies?
?
Using broctl, how do I specify snaplen=X? in the config file? I have tried putting variations of this into broctl.cfg, but it's not happy
? BroArgs = snaplen 65535
.
?
Finally, what is the best way to specify the logging output path?? Is this in a config file or do I need to set it in a script?
?? Log::add_filter(HTTP::LOG,[$name="myname", $path="/my/custom/path/basename", ...
Ideally, I would like to set the path on ALL logs with one setting, not just http.
?
Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120507/117bcf97/attachment.html 

From seth at icir.org  Mon May  7 11:34:00 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 7 May 2012 14:34:00 -0400
Subject: [Bro] bro operational questions
In-Reply-To: <1336414472.79944.YahooMailNeo@web120703.mail.ne1.yahoo.com>
References: <1336414472.79944.YahooMailNeo@web120703.mail.ne1.yahoo.com>
Message-ID: <F7637DBE-58DD-484C-87C4-87059A571A0A@icir.org>


On May 7, 2012, at 2:14 PM, Dalton Porter wrote:

> I need to keep bro up and running to process logs continuously.  I was wondering what folks would suggest for doing that.  Does broctl automatically restart the process if it dies?

Yes, it does.  BroControl was built around the need to keep running Bro constantly.  You need to make sure that you have a cron job in your system's crontab to run the "broctl cron" command.  It's documented at this section of our quick start guide:

	http://bro-ids.org/documentation/quickstart.html#a-minimal-starting-configuration

>  Using broctl, how do I specify snaplen=X  in the config file? I have tried putting variations of this into broctl.cfg, but it's not happy
>   BroArgs = snaplen 65535

Into your local.bro add this (then in broctl, do "check", "install", "restart"):

redef snaplen = 65535;

It's not a command line argument (although you can give it that way, it's probably better to keep it as part of your Bro script configuration).

> Finally, what is the best way to specify the logging output path?  Is this in a config file or do I need to set it in a script?
>    Log::add_filter(HTTP::LOG,[$name="myname", $path="/my/custom/path/basename", ?

In broctl.cfg:

logdir=/my/custom/path/basename

The $path field in the logging framework is used as the filename for the various logs.  We didn't use the term "filename" because once we have database output plugins the $path field will be used as the table name.  

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From hammadog at gmail.com  Wed May  9 10:16:21 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Wed, 9 May 2012 13:16:21 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
	<CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
	<4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
Message-ID: <CANCSO1eGihn3Uk7rPLHVLQYWYSdNySvG=SomQZzXLgn0d-HwwA@mail.gmail.com>

Well I finally got some time to work on this dude.

I started with a fresh build of Ubuntu 10.10 server all up to snuff.
Loaded only Bro with a single tap.  Drops started right off the bat.
So I updated my intel driver to the latest and restarted bro.  Drops
still happening.  I loaded capture-loss and I assume you wanted some
date out of the notice.log about the packet drops?

Here is a small snippet of a couple.  They are pretty frequent.

1336583347.593837	-	-	-	-	-	-	PacketFilter::Dropped_Packets	93989
packets dropped after filtering, 249946 received, 249946 on
link	-	-	-	-	bro	Notice::ACTION_LOG	6	3600.000000	F	-	-	-	-	-	-	-	-
1336583357.594487	-	-	-	-	-	-	PacketFilter::Dropped_Packets	73508
packets dropped after filtering, 227808 received, 227808 on
link	-	-	-	-	bro	Notice::ACTION_LOG	6	3600.000000	F	-	-	-	-	-	-	-	-
1336583367.594936	-	-	-	-	-	-	PacketFilter::Dropped_Packets	82349
packets dropped after filtering, 234476 received, 234476 on
link	-	-	-	-	bro	Notice::ACTION_LOG	6	3600.000000	F	-	-	-	-	-	-	-	-

Current traffic on the monitor port:

Interface            kpps       mbps       (10s average)
------------------------------
localhost/eth1       22.8       121.5

Any other thoughts other than drop kick the bay into the dumpster!  hehe

Thanks

Tom


On Fri, May 4, 2012 at 9:58 AM, Seth Hall <seth at icir.org> wrote:
>
> On May 4, 2012, at 6:21 AM, Tom OBrion wrote:
>
>> worker-0: 1336126625.749682 recvd=263871 dropped=30023 link=293912
>> worker-1: 1336126625.997021 recvd=262510 dropped=30656 link=293227
>
> Are you running "misc/capture-loss"? ?That should provide a much more holistic view of packet loss because it's not relying on anything other than characteristics of the actual traffic to tell you if packets are being lost. ?It doesn't tell you where the packet loss is happening and could mean a very large number of things, but it's a good place to start.
>
>> We were unsure as the documentation mentioned 80mbps per CPU, so we
>> thought we would give pf_ring a run. ?But at these rates I would not
>> think we would see drops.
>
> I was really conflicted when I wrote 80Mbps in that documentation. ?There is really no good way to figure out what that will be. ?With reasonably fast, modern Xeon CPUs people seem to be getting ~150Mbps per core now but you need to take value with a grain of salt too since it depends so heavily on your traffic mix
>
>> Is netstats not telling the truth? ?:)
>
> That question is really hard to answer, especially if you are running pf_ring where the normal Linux packet processing pipeline is being bypassed.
>
>> We are just trying to get an idea of what these old IBM hardware can
>> do for us and are running into this.
>
> You didn't mention that it's old hardware. :) ?What's the architecture? ?How many cores does the box have total?
>
> ?.Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>



From hammadog at gmail.com  Wed May  9 10:32:18 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Wed, 9 May 2012 13:32:18 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
	<CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
	<4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
Message-ID: <CANCSO1cNbAKRR+Q_Bc_3z55bq4Z99qEeyJEbzbT_hgGcx1KRvw@mail.gmail.com>

Just for giggles and cause I can.  I am upgrading to latest Ubuntu on
the same box.  WTH

On Fri, May 4, 2012 at 9:58 AM, Seth Hall <seth at icir.org> wrote:
>
> On May 4, 2012, at 6:21 AM, Tom OBrion wrote:
>
>> worker-0: 1336126625.749682 recvd=263871 dropped=30023 link=293912
>> worker-1: 1336126625.997021 recvd=262510 dropped=30656 link=293227
>
> Are you running "misc/capture-loss"? ?That should provide a much more holistic view of packet loss because it's not relying on anything other than characteristics of the actual traffic to tell you if packets are being lost. ?It doesn't tell you where the packet loss is happening and could mean a very large number of things, but it's a good place to start.
>
>> We were unsure as the documentation mentioned 80mbps per CPU, so we
>> thought we would give pf_ring a run. ?But at these rates I would not
>> think we would see drops.
>
> I was really conflicted when I wrote 80Mbps in that documentation. ?There is really no good way to figure out what that will be. ?With reasonably fast, modern Xeon CPUs people seem to be getting ~150Mbps per core now but you need to take value with a grain of salt too since it depends so heavily on your traffic mix
>
>> Is netstats not telling the truth? ?:)
>
> That question is really hard to answer, especially if you are running pf_ring where the normal Linux packet processing pipeline is being bypassed.
>
>> We are just trying to get an idea of what these old IBM hardware can
>> do for us and are running into this.
>
> You didn't mention that it's old hardware. :) ?What's the architecture? ?How many cores does the box have total?
>
> ?.Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>



-- 
Tom O'Brion
TEL: 207.210.2167
Skype:

"Life is too short to spend time with people who suck the happy out of you."



From seth at icir.org  Wed May  9 10:57:01 2012
From: seth at icir.org (Seth Hall)
Date: Wed, 9 May 2012 13:57:01 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <CANCSO1eGihn3Uk7rPLHVLQYWYSdNySvG=SomQZzXLgn0d-HwwA@mail.gmail.com>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
	<CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
	<4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
	<CANCSO1eGihn3Uk7rPLHVLQYWYSdNySvG=SomQZzXLgn0d-HwwA@mail.gmail.com>
Message-ID: <A82B76E6-595E-4D07-8DFA-9429D89223BF@icir.org>


On May 9, 2012, at 1:16 PM, Tom OBrion wrote:

> I loaded capture-loss and I assume you wanted some
> date out of the notice.log about the packet drops?

You will have a new log named capture_loss.log.  Could we see some lines from that?

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From hammadog at gmail.com  Wed May  9 11:07:30 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Wed, 9 May 2012 14:07:30 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <A82B76E6-595E-4D07-8DFA-9429D89223BF@icir.org>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
	<CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
	<4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
	<CANCSO1eGihn3Uk7rPLHVLQYWYSdNySvG=SomQZzXLgn0d-HwwA@mail.gmail.com>
	<A82B76E6-595E-4D07-8DFA-9429D89223BF@icir.org>
Message-ID: <CANCSO1f=zVq0VpdV6Syp=4PVgcHHQAj9vcM8GVMYUrVe2NGDLA@mail.gmail.com>

hmmm...I did not see that one.  I am in the middle of a upgrade.  Soon
as that is done I will send it along.  Maybe the script did not load.

Thanks Seth

Tom

On Wed, May 9, 2012 at 1:57 PM, Seth Hall <seth at icir.org> wrote:
>
> On May 9, 2012, at 1:16 PM, Tom OBrion wrote:
>
>> I loaded capture-loss and I assume you wanted some
>> date out of the notice.log about the packet drops?
>
> You will have a new log named capture_loss.log. ?Could we see some lines from that?
>
> ?.Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>



From hammadog at gmail.com  Wed May  9 11:44:05 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Wed, 9 May 2012 14:44:05 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <A82B76E6-595E-4D07-8DFA-9429D89223BF@icir.org>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
	<CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
	<4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
	<CANCSO1eGihn3Uk7rPLHVLQYWYSdNySvG=SomQZzXLgn0d-HwwA@mail.gmail.com>
	<A82B76E6-595E-4D07-8DFA-9429D89223BF@icir.org>
Message-ID: <CANCSO1ed9UOEv19LWxrf3Onk2RUEtkO9A_YKnHA81JxXeNzxuQ@mail.gmail.com>

Well new version of UBuntu:

Distributor ID:	Ubuntu
Description:	Ubuntu 11.04
Release:	11.04
Codename:	natty

I do not see that log file being created.  I do see:

/usr/local/bro/share/bro/policy/misc/capture-loss.bro

in loaded_scripts.log

Do I feed it to the birds now?  :)

Tom

On Wed, May 9, 2012 at 1:57 PM, Seth Hall <seth at icir.org> wrote:
>
> On May 9, 2012, at 1:16 PM, Tom OBrion wrote:
>
>> I loaded capture-loss and I assume you wanted some
>> date out of the notice.log about the packet drops?
>
> You will have a new log named capture_loss.log. ?Could we see some lines from that?
>
> ?.Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>



From seth at icir.org  Wed May  9 14:04:37 2012
From: seth at icir.org (Seth Hall)
Date: Wed, 9 May 2012 17:04:37 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <CANCSO1ed9UOEv19LWxrf3Onk2RUEtkO9A_YKnHA81JxXeNzxuQ@mail.gmail.com>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
	<CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
	<4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
	<CANCSO1eGihn3Uk7rPLHVLQYWYSdNySvG=SomQZzXLgn0d-HwwA@mail.gmail.com>
	<A82B76E6-595E-4D07-8DFA-9429D89223BF@icir.org>
	<CANCSO1ed9UOEv19LWxrf3Onk2RUEtkO9A_YKnHA81JxXeNzxuQ@mail.gmail.com>
Message-ID: <351F3320-0A2F-4B27-AC17-DA8CAE99602D@icir.org>


On May 9, 2012, at 2:44 PM, Tom OBrion wrote:

> I do not see that log file being created.  I do see:


It takes 5 minutes before the log shows up. :)

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/



From hammadog at gmail.com  Wed May  9 17:32:14 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Wed, 9 May 2012 20:32:14 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <351F3320-0A2F-4B27-AC17-DA8CAE99602D@icir.org>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
	<CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
	<4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
	<CANCSO1eGihn3Uk7rPLHVLQYWYSdNySvG=SomQZzXLgn0d-HwwA@mail.gmail.com>
	<A82B76E6-595E-4D07-8DFA-9429D89223BF@icir.org>
	<CANCSO1ed9UOEv19LWxrf3Onk2RUEtkO9A_YKnHA81JxXeNzxuQ@mail.gmail.com>
	<351F3320-0A2F-4B27-AC17-DA8CAE99602D@icir.org>
Message-ID: <CANCSO1d1bW50KwbWFY0j2+Aad7Thwd4+FswEPyiJ5RStx4Q0eg@mail.gmail.com>

Well I just checked again an have the following in the file.

#separator \x09
#set_separator	,
#empty_field	(empty)
#unset_field	-
#path	capture_loss
#fields	ts	ts_delta	peer	gaps	acks	percent_lost
#types	time	interval	string	count	count	string
1336608708.135106	900.000206	bro	996	721708	0.138%
1336609608.135122	900.000016	bro	805	705801	0.114%

On Wed, May 9, 2012 at 5:04 PM, Seth Hall <seth at icir.org> wrote:
>
> On May 9, 2012, at 2:44 PM, Tom OBrion wrote:
>
>> I do not see that log file being created. ?I do see:
>
>
> It takes 5 minutes before the log shows up. :)
>
> ?.Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>



-- 
Tom O'Brion
TEL: 207.210.2167
Skype:

"Life is too short to spend time with people who suck the happy out of you."



From seth at icir.org  Thu May 10 05:25:54 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 10 May 2012 08:25:54 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <CANCSO1d1bW50KwbWFY0j2+Aad7Thwd4+FswEPyiJ5RStx4Q0eg@mail.gmail.com>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
	<CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
	<4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
	<CANCSO1eGihn3Uk7rPLHVLQYWYSdNySvG=SomQZzXLgn0d-HwwA@mail.gmail.com>
	<A82B76E6-595E-4D07-8DFA-9429D89223BF@icir.org>
	<CANCSO1ed9UOEv19LWxrf3Onk2RUEtkO9A_YKnHA81JxXeNzxuQ@mail.gmail.com>
	<351F3320-0A2F-4B27-AC17-DA8CAE99602D@icir.org>
	<CANCSO1d1bW50KwbWFY0j2+Aad7Thwd4+FswEPyiJ5RStx4Q0eg@mail.gmail.com>
Message-ID: <EDDD559B-42D8-4500-826D-3E162DF9BB96@icir.org>


On May 9, 2012, at 8:32 PM, Tom OBrion wrote:

> #fields	ts	ts_delta	peer	gaps	acks	percent_lost
> #types	time	interval	string	count	count	string
> 1336608708.135106	900.000206	bro	996	721708	0.138%
> 1336609608.135122	900.000016	bro	805	705801	0.114%


Now that actually looks really nice.  Did you say that you are running PF_Ring?  I trust the data from the NIC even less when using any of the various things that bypass the normal OS data flow (but I'm not saying that's a bad thing!).
  
  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From hammadog at gmail.com  Thu May 10 06:05:17 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Thu, 10 May 2012 09:05:17 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <EDDD559B-42D8-4500-826D-3E162DF9BB96@icir.org>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
	<CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
	<4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
	<CANCSO1eGihn3Uk7rPLHVLQYWYSdNySvG=SomQZzXLgn0d-HwwA@mail.gmail.com>
	<A82B76E6-595E-4D07-8DFA-9429D89223BF@icir.org>
	<CANCSO1ed9UOEv19LWxrf3Onk2RUEtkO9A_YKnHA81JxXeNzxuQ@mail.gmail.com>
	<351F3320-0A2F-4B27-AC17-DA8CAE99602D@icir.org>
	<CANCSO1d1bW50KwbWFY0j2+Aad7Thwd4+FswEPyiJ5RStx4Q0eg@mail.gmail.com>
	<EDDD559B-42D8-4500-826D-3E162DF9BB96@icir.org>
Message-ID: <CANCSO1fsTJT7zOt4pke5eFLQEiQ2i8wknnBL+CyKKYWAAuosnw@mail.gmail.com>

hehe

Well that does seem exciting, but at the time we were running around
13mbps and no we are not running pf_ring.  Here is a snipet of the log
when we were running close to 100mbps.

#separator \x09
#set_separator	,
#empty_field	(empty)
#unset_field	-
#path	capture_loss
#fields	ts	ts_delta	peer	gaps	acks	percent_lost
#types	time	interval	string	count	count	string
1336586727.588158	900.000168	bro	289518	644040	44.953%
1336587627.588220	900.000062	bro	306102	746812	40.988%



On Thu, May 10, 2012 at 8:25 AM, Seth Hall <seth at icir.org> wrote:
>
> On May 9, 2012, at 8:32 PM, Tom OBrion wrote:
>
>> #fields ? ? ? ts ? ? ?ts_delta ? ? ? ?peer ? ?gaps ? ?acks ? ?percent_lost
>> #types ? ? ? ?time ? ?interval ? ? ? ?string ?count ? count ? string
>> 1336608708.135106 ? ? 900.000206 ? ? ?bro ? ? 996 ? ? 721708 ?0.138%
>> 1336609608.135122 ? ? 900.000016 ? ? ?bro ? ? 805 ? ? 705801 ?0.114%
>
>
> Now that actually looks really nice. ?Did you say that you are running PF_Ring? ?I trust the data from the NIC even less when using any of the various things that bypass the normal OS data flow (but I'm not saying that's a bad thing!).
>
> ?.Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>



-- 
Tom O'Brion
TEL: 207.210.2167
Skype:

"Life is too short to spend time with people who suck the happy out of you."



From seth at icir.org  Thu May 10 06:51:24 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 10 May 2012 09:51:24 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <CANCSO1fsTJT7zOt4pke5eFLQEiQ2i8wknnBL+CyKKYWAAuosnw@mail.gmail.com>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
	<CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
	<4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
	<CANCSO1eGihn3Uk7rPLHVLQYWYSdNySvG=SomQZzXLgn0d-HwwA@mail.gmail.com>
	<A82B76E6-595E-4D07-8DFA-9429D89223BF@icir.org>
	<CANCSO1ed9UOEv19LWxrf3Onk2RUEtkO9A_YKnHA81JxXeNzxuQ@mail.gmail.com>
	<351F3320-0A2F-4B27-AC17-DA8CAE99602D@icir.org>
	<CANCSO1d1bW50KwbWFY0j2+Aad7Thwd4+FswEPyiJ5RStx4Q0eg@mail.gmail.com>
	<EDDD559B-42D8-4500-826D-3E162DF9BB96@icir.org>
	<CANCSO1fsTJT7zOt4pke5eFLQEiQ2i8wknnBL+CyKKYWAAuosnw@mail.gmail.com>
Message-ID: <3D8EDBC2-50FE-4EA2-97E2-8BCCCCF8D510@icir.org>


On May 10, 2012, at 9:05 AM, Tom OBrion wrote:

> Here is a snipet of the log
> when we were running close to 100mbps.
> 1336586727.588158	900.000168	bro	289518	644040	44.953%
> 1336587627.588220	900.000062	bro	306102	746812	40.988%

Ah, ok.  So you said that it's an old dual Xeon box?  Does that mean you have 2 cores total?

It's very likely that your box is just underpowered if you only have two cores and it's an older cpu architecture.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From eyrich at illinois.edu  Thu May 10 06:52:53 2012
From: eyrich at illinois.edu (James Eyrich)
Date: Thu, 10 May 2012 08:52:53 -0500
Subject: [Bro] How to suppress: is not seeing any packets on interface
Message-ID: <4FABC835.9060305@illinois.edu>

I have a Bro box monitoring traffic on a wireless network.
At night traffic is very light and sporadic and I get a lot of
is not seeing any packets on interface
and
is seeing packets again on interface
messages

what is the best way to suppress these?



-- 
James Eyrich
Security Engineer
National Center for Supercomputer Applications
University of Illinois at Urbana-Champaign
eyrich at illinois.edu
217-265-6867


From hammadog at gmail.com  Thu May 10 07:02:56 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Thu, 10 May 2012 10:02:56 -0400
Subject: [Bro] Packet Drops
In-Reply-To: <3D8EDBC2-50FE-4EA2-97E2-8BCCCCF8D510@icir.org>
References: <CANCSO1di2fe0b0KK85cwiUcQuD=ZhZbcOHFZsdTR805NWgEgeA@mail.gmail.com>
	<20120504012120.GY23613@datacomm.albany.edu>
	<CANpnLHjYYEZuKaMiiamgKkm-0oY5-=trooeGyc2qLS12dbCgsQ@mail.gmail.com>
	<CANCSO1euM+OQH9ss=hAaHcO+TQhg886KErp3QahNbAqznKV7xA@mail.gmail.com>
	<4519D440-F8E4-4ED6-9B63-5501C10B52CA@icir.org>
	<CANCSO1eGihn3Uk7rPLHVLQYWYSdNySvG=SomQZzXLgn0d-HwwA@mail.gmail.com>
	<A82B76E6-595E-4D07-8DFA-9429D89223BF@icir.org>
	<CANCSO1ed9UOEv19LWxrf3Onk2RUEtkO9A_YKnHA81JxXeNzxuQ@mail.gmail.com>
	<351F3320-0A2F-4B27-AC17-DA8CAE99602D@icir.org>
	<CANCSO1d1bW50KwbWFY0j2+Aad7Thwd4+FswEPyiJ5RStx4Q0eg@mail.gmail.com>
	<EDDD559B-42D8-4500-826D-3E162DF9BB96@icir.org>
	<CANCSO1fsTJT7zOt4pke5eFLQEiQ2i8wknnBL+CyKKYWAAuosnw@mail.gmail.com>
	<3D8EDBC2-50FE-4EA2-97E2-8BCCCCF8D510@icir.org>
Message-ID: <CANCSO1fCGFkiDX8bNpfHuxh+e7k7=7C06ia8CGDtZkA9jOu15w@mail.gmail.com>

yeah I suspect you are correct.   we are at 91mbps atm and bro is
consuming pretty much the 1 CPU.

Thanks for all your help peops.  Time to throw a bigger horse at it!

Tom

On Thu, May 10, 2012 at 9:51 AM, Seth Hall <seth at icir.org> wrote:
>
> On May 10, 2012, at 9:05 AM, Tom OBrion wrote:
>
>> Here is a snipet of the log
>> when we were running close to 100mbps.
>> 1336586727.588158 ? ? 900.000168 ? ? ?bro ? ? 289518 ?644040 ?44.953%
>> 1336587627.588220 ? ? 900.000062 ? ? ?bro ? ? 306102 ?746812 ?40.988%
>
> Ah, ok. ?So you said that it's an old dual Xeon box? ?Does that mean you have 2 cores total?
>
> It's very likely that your box is just underpowered if you only have two cores and it's an older cpu architecture.
>
> ?.Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>



From robin at icir.org  Thu May 10 08:05:27 2012
From: robin at icir.org (Robin Sommer)
Date: Thu, 10 May 2012 08:05:27 -0700
Subject: [Bro] How to suppress: is not seeing any packets on interface
In-Reply-To: <4FABC835.9060305@illinois.edu>
References: <4FABC835.9060305@illinois.edu>
Message-ID: <20120510150527.GD25114@icir.org>


On Thu, May 10, 2012 at 08:52 -0500, James Eyrich wrote:

> is not seeing any packets on interface

> what is the best way to suppress these?

The only way currently is to pretend you don't have the capstats tool
installed. Try adding "CapstatsPath=" to broctl.cfg (i.e., set it to
an empty value). You will however also loose the "capstats" command
and the traffic stats in stats.log.

Alternatively, you could hack the source: delete lines 119-123 in
aux/broctl/BroControl/cron.py.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From daltonporter at yahoo.com  Thu May 10 10:56:43 2012
From: daltonporter at yahoo.com (Dalton Porter)
Date: Thu, 10 May 2012 10:56:43 -0700 (PDT)
Subject: [Bro] how to specify output directory?
Message-ID: <1336672603.91644.YahooMailNeo@web120701.mail.ne1.yahoo.com>

Is there a way to specify the output dir when using just the bro 2.0 executable?? I know how to do it when using broctl, but I was wondering if there is a way to do it outside of that.? I don't see a bro option for that using bro --help.
?
Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120510/1fdee38f/attachment.html 

From robin at icir.org  Thu May 10 12:41:42 2012
From: robin at icir.org (Robin Sommer)
Date: Thu, 10 May 2012 12:41:42 -0700
Subject: [Bro] how to specify output directory?
In-Reply-To: <1336672603.91644.YahooMailNeo@web120701.mail.ne1.yahoo.com>
References: <1336672603.91644.YahooMailNeo@web120701.mail.ne1.yahoo.com>
Message-ID: <20120510194142.GD47855@icir.org>


On Thu, May 10, 2012 at 10:56 -0700, Dalton Porter wrote:

> Is there a way to specify the output dir when using just the bro 2.0
> executable?

Bro always logs into the current directory, there's no way to change
that other than cd'ing somewhere else first (that's what broctl does
internally).

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From david.bianco at ge.com  Thu May 10 13:03:58 2012
From: david.bianco at ge.com (Bianco, David (GE Corporate))
Date: Thu, 10 May 2012 20:03:58 +0000
Subject: [Bro] how to specify output directory?
In-Reply-To: <20120510194142.GD47855@icir.org>
References: <1336672603.91644.YahooMailNeo@web120701.mail.ne1.yahoo.com>
	<20120510194142.GD47855@icir.org>
Message-ID: <D48F1970A7F7614BAA72A23693982397044605@ALPENMCH02.e2k.ad.ge.com>

Which is a shame, btw.  I +1 a request for a command line option to change the output directory.

	David

-----Original Message-----
From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf Of Robin Sommer
Sent: Thursday, May 10, 2012 3:42 PM
To: Dalton Porter
Cc: bro at bro-ids.org
Subject: Re: [Bro] how to specify output directory?


On Thu, May 10, 2012 at 10:56 -0700, Dalton Porter wrote:

> Is there a way to specify the output dir when using just the bro 2.0 
> executable?

Bro always logs into the current directory, there's no way to change that other than cd'ing somewhere else first (that's what broctl does internally).

Robin

--
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From vallentin at icir.org  Thu May 10 15:44:19 2012
From: vallentin at icir.org (Matthias Vallentin)
Date: Thu, 10 May 2012 15:44:19 -0700
Subject: [Bro] how to specify output directory?
In-Reply-To: <D48F1970A7F7614BAA72A23693982397044605@ALPENMCH02.e2k.ad.ge.com>
References: <1336672603.91644.YahooMailNeo@web120701.mail.ne1.yahoo.com>
	<20120510194142.GD47855@icir.org>
	<D48F1970A7F7614BAA72A23693982397044605@ALPENMCH02.e2k.ad.ge.com>
Message-ID: <CADTpMNYBdueKsWjm+46A9MmxGikpV_QnQ-jFjQ01vLxE2wJOrw@mail.gmail.com>

> Which is a shame, btw. ?I +1 a request for a command line option to change the output directory.

The best way to help us remember such feature requests is by filing a
ticket, and adding a +1 comment definitely helps to push it up the
priority queue.

     Matthias



From sconzo at visiblerisk.com  Thu May 10 21:12:12 2012
From: sconzo at visiblerisk.com (Mike Sconzo)
Date: Thu, 10 May 2012 23:12:12 -0500
Subject: [Bro] Scripting Question
Message-ID: <CAGZoo6EVjFBfd4-wdXD_5y=ALZomZL04yznpnD4XV2qAz5ku2Q@mail.gmail.com>

I've written the attached scripts, and for some reason the event
http_all_headers or http_request doesn't seem to be firing.  I've
tried a couple different pcaps to test on, tried using
HTTP::http_all_headers as the event, and now I'm pretty much out of
ideas.

In httpsetup.bro it's a simple event that sets c$http$method so I can
use this elsewhere.

in suspicious_post.bro I have a basic set of rules to look at some
POST behavior, but the only thing that seems to fire is the init_bro
(I used a print statmet to test as I haven't fully figured out -d).  I
also have what

I'm running bro -r test.pcap ./suspicious_post.bro and everything
seems to load ok.  I even tried loading via local.bro and running it
as part of the daemonized process, but that doesn't fire even after I
generate traffic that I know one of the cases _should_ fire on.  Any
thoughts or information on what I'm doing wrong would be appreciated.

Thanks,
-=Mike
-- 
cat ~/.bash_history > documentation.txt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: suspicious_post.bro
Type: application/octet-stream
Size: 1657 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120510/d7d0bd3a/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: httpsetup.bro
Type: application/octet-stream
Size: 261 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120510/d7d0bd3a/attachment-0001.obj 

From daltonporter at yahoo.com  Thu May 10 21:54:52 2012
From: daltonporter at yahoo.com (Dalton Porter)
Date: Thu, 10 May 2012 21:54:52 -0700 (PDT)
Subject: [Bro] Scripting Question
In-Reply-To: <CAGZoo6EVjFBfd4-wdXD_5y=ALZomZL04yznpnD4XV2qAz5ku2Q@mail.gmail.com>
References: <CAGZoo6EVjFBfd4-wdXD_5y=ALZomZL04yznpnD4XV2qAz5ku2Q@mail.gmail.com>
Message-ID: <1336712092.407.YahooMailNeo@web120706.mail.ne1.yahoo.com>

Mike, did you try adding the -C option?  (no-checksums)
I had something similar happen to me.  It's worth a try.

 

________________________________
 From: Mike Sconzo <sconzo at visiblerisk.com>
To: bro at bro-ids.org 
Sent: Friday, May 11, 2012 12:12 AM
Subject: [Bro] Scripting Question
  
I've written the attached scripts, and for some reason the event
http_all_headers or http_request doesn't seem to be firing.? I've
tried a couple different pcaps to test on, tried using
HTTP::http_all_headers as the event, and now I'm pretty much out of
ideas.

In httpsetup.bro it's a simple event that sets c$http$method so I can
use this elsewhere.

in suspicious_post.bro I have a basic set of rules to look at some
POST behavior, but the only thing that seems to fire is the init_bro
(I used a print statmet to test as I haven't fully figured out -d).? I
also have what

I'm running bro -r test.pcap ./suspicious_post.bro and everything
seems to load ok.? I even tried loading via local.bro and running it
as part of the daemonized process, but that doesn't fire even after I
generate traffic that I know one of the cases _should_ fire on.? Any
thoughts or information on what I'm doing wrong would be appreciated.

Thanks,
-=Mike
-- 
cat ~/.bash_history > documentation.txt

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120510/544ac44e/attachment.html 

From asma.mtz at gmail.com  Sun May 13 02:55:39 2012
From: asma.mtz at gmail.com (Asma Mumtaz)
Date: Sun, 13 May 2012 14:55:39 +0500
Subject: [Bro] A problem during using 'set'
Message-ID: <CALpj_VzDrHr4B3ZUX86HkgzbctDLq4VZY78SuYTKGOiqjn2PNg@mail.gmail.com>

Hi,

I was testing my code written in Bro, and I was noticing some wrong values.
When I debugged the code, the following section had the problem (the bold
part)

if(vals[index2] == "length"){
                local length_vals = split(vals[index2 + 1], /-/);
                *local lengths: set[count];*
                *print cat("here - prev length = ", |lengths|);*
                for(index3 in length_vals)
                    if(length_vals[index3] != "") add
lengths[to_count(length_vals[index3])];
                *print |lengths|;*
            } #finished lengths
here is the output.

here - prev length = 0
1
here - prev length = 1
2
here - prev length = 2
3
here - prev length = 3
4
here - prev length = 4
31
here - prev length = 31
34
here - prev length = 34
46
here - prev length = 46
48

It means that whenever I declare local lengths: set[count]. It is not
initializing a new 'set', but keeps the state of previous set of values
entered and updated it. Why is it doing so?
regards,
Asma
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120513/38dee022/attachment.html 

From asma.mtz at gmail.com  Sun May 13 03:10:03 2012
From: asma.mtz at gmail.com (Asma Mumtaz)
Date: Sun, 13 May 2012 15:10:03 +0500
Subject: [Bro] A problem during using 'set'
In-Reply-To: <CALpj_VzDrHr4B3ZUX86HkgzbctDLq4VZY78SuYTKGOiqjn2PNg@mail.gmail.com>
References: <CALpj_VzDrHr4B3ZUX86HkgzbctDLq4VZY78SuYTKGOiqjn2PNg@mail.gmail.com>
Message-ID: <CALpj_VzCmE4b_nE13A84z3WhfwLgzL_cf4BYO09JnSNA3J373Q@mail.gmail.com>

And the problem remains same, even if I use 'table[count] of count'.

regards,
Asma

On Sun, May 13, 2012 at 2:55 PM, Asma Mumtaz <asma.mtz at gmail.com> wrote:

> Hi,
>
> I was testing my code written in Bro, and I was noticing some wrong
> values. When I debugged the code, the following section had the problem
> (the bold part)
>
> if(vals[index2] == "length"){
>                 local length_vals = split(vals[index2 + 1], /-/);
>                 *local lengths: set[count];*
>                 *print cat("here - prev length = ", |lengths|);*
>                 for(index3 in length_vals)
>                     if(length_vals[index3] != "") add
> lengths[to_count(length_vals[index3])];
>                 *print |lengths|;*
>             } #finished lengths
> here is the output.
>
> here - prev length = 0
> 1
> here - prev length = 1
> 2
> here - prev length = 2
> 3
> here - prev length = 3
> 4
> here - prev length = 4
> 31
> here - prev length = 31
> 34
> here - prev length = 34
> 46
> here - prev length = 46
> 48
>
> It means that whenever I declare local lengths: set[count]. It is not
> initializing a new 'set', but keeps the state of previous set of values
> entered and updated it. Why is it doing so?
> regards,
> Asma
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120513/b09f8bbd/attachment.html 

From seth at icir.org  Sun May 13 07:05:50 2012
From: seth at icir.org (Seth Hall)
Date: Sun, 13 May 2012 10:05:50 -0400
Subject: [Bro] A problem during using 'set'
In-Reply-To: <CALpj_VzDrHr4B3ZUX86HkgzbctDLq4VZY78SuYTKGOiqjn2PNg@mail.gmail.com>
References: <CALpj_VzDrHr4B3ZUX86HkgzbctDLq4VZY78SuYTKGOiqjn2PNg@mail.gmail.com>
Message-ID: <3649D99A-B165-482A-A560-7315E875F110@icir.org>


On May 13, 2012, at 5:55 AM, Asma Mumtaz wrote:

> It means that whenever I declare local lengths: set[count]. It is not initializing a new 'set', but keeps the state of previous set of values entered and updated it. Why is it doing so?

You aren't assigning a value to the variable, you are only declaring the type of the variable.  Try this:

 local lengths: set[count] = set();

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From vern at icir.org  Sun May 13 10:09:52 2012
From: vern at icir.org (Vern Paxson)
Date: Sun, 13 May 2012 10:09:52 -0700
Subject: [Bro] A problem during using 'set'
In-Reply-To: <CALpj_VzDrHr4B3ZUX86HkgzbctDLq4VZY78SuYTKGOiqjn2PNg@mail.gmail.com>
	(Sun, 13 May 2012 14:55:39 +0500).
Message-ID: <20120513170952.998C42C400A@rock.ICSI.Berkeley.EDU>

> if(vals[index2] == "length"){
>                 local length_vals = split(vals[index2 + 1], /-/);
>                 *local lengths: set[count];*

Contrary to most languages, "local" declarations in Bro persist throughout
the entire function/event handler in which they're declared.  They don't
get discarded once the block they're declared in goes out of scope.
I believe this explains the behavior you're encountering

Arguably this should be changed, but doing so would be a fair amount
of work.

		Vern


From daltonporter at yahoo.com  Tue May 15 11:38:07 2012
From: daltonporter at yahoo.com (Dalton Porter)
Date: Tue, 15 May 2012 11:38:07 -0700 (PDT)
Subject: [Bro] http response encoded length ?
Message-ID: <1337107087.55897.YahooMailNeo@web120705.mail.ne1.yahoo.com>

Hi,
How can I get the length of an http response that is gzip ENcoded and where the response does not have a "content-length" header?
It seems that? stat$body_length has the length of the DEcoded data.? 
?
Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120515/8854be3a/attachment.html 

From oguzyarimtepe at gmail.com  Wed May 16 13:14:03 2012
From: oguzyarimtepe at gmail.com (Oguz Yarimtepe)
Date: Wed, 16 May 2012 23:14:03 +0300
Subject: [Bro] saving the binary information at pcap
Message-ID: <20120516231403.aff5c270d1aa12375d6b14d4@gmail.com>

Hi,

I set the default_extract variable as

const default_extract = T &redef;

at the contents.bro script to get the dat files including tcp reassembly contents. Is there a way at the Python binding side so that i can save the binaries as seperate files in the created files? The dat files include many responses. I can read the file and try to parse the content out of by looking at the orig file. But maybe there is a better way at the binding side

Cheers.

-- 
Oguz Yarimtepe <oguzyarimtepe at gmail.com>
http://about.me/oguzy


From seth at icir.org  Wed May 16 13:36:07 2012
From: seth at icir.org (Seth Hall)
Date: Wed, 16 May 2012 16:36:07 -0400
Subject: [Bro] http response encoded length ?
In-Reply-To: <1337107087.55897.YahooMailNeo@web120705.mail.ne1.yahoo.com>
References: <1337107087.55897.YahooMailNeo@web120705.mail.ne1.yahoo.com>
Message-ID: <6192887F-8CFC-45C7-95A8-0AFE312F8471@icir.org>


On May 15, 2012, at 2:38 PM, Dalton Porter wrote:

> How can I get the length of an http response that is gzip ENcoded and where the response does not have a "content-length" header?

I don't believe that data is available right now.  What do you need it for?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Wed May 16 13:37:00 2012
From: seth at icir.org (Seth Hall)
Date: Wed, 16 May 2012 16:37:00 -0400
Subject: [Bro] saving the binary information at pcap
In-Reply-To: <20120516231403.aff5c270d1aa12375d6b14d4@gmail.com>
References: <20120516231403.aff5c270d1aa12375d6b14d4@gmail.com>
Message-ID: <F087B247-01C9-4D41-B8F0-0A25474DD8EA@icir.org>


On May 16, 2012, at 4:14 PM, Oguz Yarimtepe wrote:

> Is there a way at the Python binding side so that i can save the binaries as seperate files in the created files?


What are you referring to as binaries?  You are going to need to explain what you are trying to accomplish in more detail.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From oguzyarimtepe at gmail.com  Wed May 16 22:53:56 2012
From: oguzyarimtepe at gmail.com (Oguz Yarimtepe)
Date: Thu, 17 May 2012 08:53:56 +0300
Subject: [Bro] saving the binary information at pcap
In-Reply-To: <F087B247-01C9-4D41-B8F0-0A25474DD8EA@icir.org>
References: <20120516231403.aff5c270d1aa12375d6b14d4@gmail.com>
	<F087B247-01C9-4D41-B8F0-0A25474DD8EA@icir.org>
Message-ID: <20120517085356.5ced01e056bf48df4056de64@gmail.com>

Hi,

On Wed, 16 May 2012 16:37:00 -0400
Seth Hall <seth at icir.org> wrote:

> What are you referring to as binaries?  You are going to need to explain what you are trying to accomplish in more detail.

When i run 

bro -r somepcapfile base/protocols/conn/ (the aim is to make the contents.bro loaded, but i might be writing the wrong path now, i couldn't remembe the whole path of contencts.bro directory installed)

i got some dat files. They have names_
contencts_192.168.1.10_4356_193.255.98.2_80_orig.dat
contencts_192.168.1.10_4356_193.255.98.2_80_resp.dat

Each has the result of tcp reassembly sessions. I saved my port 80 traffic when i browse to an address to www.milliyet.com.tr, so the results has images, js files, returned HTMLs eveything that can a web site has.

By traversing each file, i can save the contents separetely. It seems the response dat files has the saved information like images, htmlfiles, texts in plain format. 

Is there a way to tell Bro that ok don't save this response as a single file, but save the images here, js files here, etc. Or can i use Brocolli Python binding for it?

-- 
Oguz Yarimtepe <oguzyarimtepe at gmail.com>


From sheharbano.k at gmail.com  Thu May 17 13:26:51 2012
From: sheharbano.k at gmail.com (Sheharbano Khattak)
Date: Fri, 18 May 2012 01:26:51 +0500
Subject: [Bro] Disabling DPD
Message-ID: <CAA6v-1YH2SkuWbf-54YBXv2jCrDWdAqmLe=UPSUMKJdTss_-8g@mail.gmail.com>

Hi,

For the purpose of testing my scripts, sometimes i am interested in looking
at results only from a particular protocol. However, everything is enabled
by default. So i have HTTP, SMTP, FTP... analysis going on when i am only
interested in, say, DNS. I redefined PacketFilter::all_packets=F in
bro_init but to no avail. Then i made the same change in the actual script
file instead of redefining all_packets in my script, still the same
behavior. Please correct me if i am doing it wrong :-)

Regards,

-- 
Sheharbano Khattak

http://etheryell.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120518/fad92047/attachment.html 

From JAzoff at albany.edu  Thu May 17 13:44:39 2012
From: JAzoff at albany.edu (Justin Azoff)
Date: Thu, 17 May 2012 16:44:39 -0400
Subject: [Bro] Disabling DPD
In-Reply-To: <CAA6v-1YH2SkuWbf-54YBXv2jCrDWdAqmLe=UPSUMKJdTss_-8g@mail.gmail.com>
References: <CAA6v-1YH2SkuWbf-54YBXv2jCrDWdAqmLe=UPSUMKJdTss_-8g@mail.gmail.com>
Message-ID: <20120517204439.GZ23613@datacomm.albany.edu>

On Fri, May 18, 2012 at 01:26:51AM +0500, Sheharbano Khattak wrote:
> Hi,
> 
> For the purpose of testing my scripts, sometimes i am interested in looking at
> results only from a particular protocol. However, everything is enabled by
> default. So i have HTTP, SMTP, FTP... analysis going on when i am only
> interested in, say, DNS. I redefined PacketFilter::all_packets=F in bro_init
> but to no avail. Then i made the same change in the actual script file instead
> of redefining all_packets in my script, still the same behavior. Please correct
> me if i am doing it wrong :-)

try:

    redef PacketFilter::all_packets = F;
    redef capture_filters = [[ "only-dns"] = "port 53"];



-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From seth at icir.org  Sat May 19 21:35:15 2012
From: seth at icir.org (Seth Hall)
Date: Sun, 20 May 2012 00:35:15 -0400
Subject: [Bro] saving the binary information at pcap
In-Reply-To: <20120517085356.5ced01e056bf48df4056de64@gmail.com>
References: <20120516231403.aff5c270d1aa12375d6b14d4@gmail.com>
	<F087B247-01C9-4D41-B8F0-0A25474DD8EA@icir.org>
	<20120517085356.5ced01e056bf48df4056de64@gmail.com>
Message-ID: <C7667AC5-81B3-438D-9FCE-D7F74AC1A658@icir.org>


On May 17, 2012, at 1:53 AM, Oguz Yarimtepe wrote:

> Is there a way to tell Bro that ok don't save this response as a single file, but save the images here, js files here, etc. Or can i use Brocolli Python binding for it?


You are looking at the wrong extraction. :)

This will extract windows executables from HTTP traffic:
redef HTTP::extract_file_types += /application\/x-dosexec/;

If you have different criteria for extracting files, it's possible to do your own thing by setting a boolean value in the c$http record.  You just need to make sure that you set it before any data has begun to transfer.  In your case, you might want to do this...

event http_header(c: connection, is_orig: bool, name: string, value: string)
	{
	if ( name == "HOST" && value == "www.milliyet.com.tr" )
		c$http$extract_file = T;
	}

The above code will make Bro extract all files from the site you mentioned in your previous email.  This will all be changing when we get the file analysis framework released though, but should be easier and more generic for all protocols.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Sat May 19 21:48:45 2012
From: seth at icir.org (Seth Hall)
Date: Sun, 20 May 2012 00:48:45 -0400
Subject: [Bro] http response encoded length ?
In-Reply-To: <1337201790.72559.YahooMailNeo@web120704.mail.ne1.yahoo.com>
References: <1337107087.55897.YahooMailNeo@web120705.mail.ne1.yahoo.com>
	<6192887F-8CFC-45C7-95A8-0AFE312F8471@icir.org>
	<1337201790.72559.YahooMailNeo@web120704.mail.ne1.yahoo.com>
Message-ID: <BCC1EC2E-C3D8-46B6-B118-9E2E025A2796@icir.org>


On May 16, 2012, at 4:56 PM, Dalton Porter wrote:

> One of the requirements for my project is to measure the bytes sent and received on the wire.  I'm thinking there must be a counter -  maybe in conn that could be used for this.resp_ip_bytes  looks promising, but I have not figured out how to use it yet.

Ah!  Ok, that's available but you have to be aware of what you are measuring.  In your conn.log there are several fields that represent the data you're looking for.

orig_bytes, resp_bytes
These are payload bytes for data sent by the originator and responder.  

orig_ip_bytes, resp_ip_bytes
These are byte counts including the IP header.  If you are looking for the total amount of data being sent across your border to the "internet", then this is likely the measurement you want.

These fields available several ways, one easy way that is a nice analog to log processing is to access it through the logging framework event as the data is being logged.

event Conn::log_conn(rec: Conn::Info)
	{
	print rec$orig_ip_bytes + rec$resp_ip_bytes;
	}

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From christopher.p.crawford at gmail.com  Mon May 21 06:38:46 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Mon, 21 May 2012 09:38:46 -0400
Subject: [Bro] pcap_next Question
Message-ID: <CAG1VgOapQQbSmnOkZOO9dhY54XdaNKXy+Y3+ks6M4Hn_9UXFNQ@mail.gmail.com>

Bro 2 has been crashing for me regularly and frequently for several
months. [ http://bit.ly/JJQVVf ]

Although I configured Bro in a way that works for me, it would still
be nice to use it as it is intended to be used.

I studied a number of crash dumps, and have looked through the code.
I was seeing crashes with the Bro 2.0 release, but I am now using a
version of Bro 2.0 from the git repositories that I checked out on
April 30.  I saw very similar crashes in both versions.

Line 78 in PktSrc.cc is consistently related to issues in the
backtraces I'm getting from bro core dumps.  I really haven't written
much code with libpcap, so there's probably a good reason to use
pcap_next() there.  I'm just wondering, why not use pcap_next_ex()
there and do a bit of error checking before passing packet data along?
 The way it is right now, it looks like the code just trusts that
pcap_next() read a packet successfully and then hands it off.

I think that in my case, something is going wrong with the call to
pcap_next() -- it's returning a pointer that doesn't make any sense.
If there was a little error checking around pcap_next() by using
pcap_next_ex() instead, maybe that would prevent the crash I'm seeing.

On the other hand, maybe there is some code that does some error
checking on if the value returned by pcap_next() makes sense and I'm
just not finding it.

Can anyone help me understand the choice to use pcap_next() vs pcap_next_ext()?


From jsiwek at illinois.edu  Mon May 21 09:48:31 2012
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 21 May 2012 16:48:31 +0000
Subject: [Bro] pcap_next Question
In-Reply-To: <CAG1VgOapQQbSmnOkZOO9dhY54XdaNKXy+Y3+ks6M4Hn_9UXFNQ@mail.gmail.com>
References: <CAG1VgOapQQbSmnOkZOO9dhY54XdaNKXy+Y3+ks6M4Hn_9UXFNQ@mail.gmail.com>
Message-ID: <BC3C8AB0-B4C0-41A1-A2E7-39F71CE2C664@illinois.edu>


> Line 78 in PktSrc.cc is consistently related to issues in the
> backtraces I'm getting from bro core dumps.  I really haven't written
> much code with libpcap, so there's probably a good reason to use
> pcap_next() there.  I'm just wondering, why not use pcap_next_ex()
> there and do a bit of error checking before passing packet data along?
> The way it is right now, it looks like the code just trusts that
> pcap_next() read a packet successfully and then hands it off.

pcap_next() returns NULL if an error occurs or no packets are read from a live capture.  The call to it in PktSrc::ExtractNextPacket() that you mention does look like it checks the validity of the return value in several places and its own return value is based on it (which is also checked whenever it's called).

But I don't know why pcap_next_ex() isn't used to get information about errors so some text can be relayed to the user, maybe that function didn't exist at the time the code was written.

> 
> I think that in my case, something is going wrong with the call to
> pcap_next() -- it's returning a pointer that doesn't make any sense.
> If there was a little error checking around pcap_next() by using
> pcap_next_ex() instead, maybe that would prevent the crash I'm seeing.

Do you have a stack trace you can send?  If pcap_next() were returning a bogus pointer, I don't think you'd see the call to it in the stack, you'd be at a later point in the code where it attempted to access it and crashes.  That is, if pcap_next() is in your stack trace, something bad is probably happening within the pcap library and the caller would never have the opportunity to check the return value.

+Jon


From robin at icir.org  Mon May 21 10:19:44 2012
From: robin at icir.org (Robin Sommer)
Date: Mon, 21 May 2012 10:19:44 -0700
Subject: [Bro] pcap_next Question
In-Reply-To: <BC3C8AB0-B4C0-41A1-A2E7-39F71CE2C664@illinois.edu>
References: <CAG1VgOapQQbSmnOkZOO9dhY54XdaNKXy+Y3+ks6M4Hn_9UXFNQ@mail.gmail.com>
	<BC3C8AB0-B4C0-41A1-A2E7-39F71CE2C664@illinois.edu>
Message-ID: <20120521171944.GM56062@icir.org>


On Mon, May 21, 2012 at 16:48 +0000, Siwek, Jonathan Luke wrote:

> about errors so some text can be relayed to the user, maybe that
> function didn't exist at the time the code was written.

Yes, I think that's the reason, we were trying to stay compatible to
older pcaps for a while. It should be safe to switch to pcap_next_ex()
now, but I'd like to understand the underlying problem here. I don't
think we have seen this problem reported from elsewhere yet. Could
this be an artifact of the VM setting?

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From christopher.p.crawford at gmail.com  Mon May 21 10:32:43 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Mon, 21 May 2012 13:32:43 -0400
Subject: [Bro] pcap_next Question
In-Reply-To: <BC3C8AB0-B4C0-41A1-A2E7-39F71CE2C664@illinois.edu>
References: <CAG1VgOapQQbSmnOkZOO9dhY54XdaNKXy+Y3+ks6M4Hn_9UXFNQ@mail.gmail.com>
	<BC3C8AB0-B4C0-41A1-A2E7-39F71CE2C664@illinois.edu>
Message-ID: <CAG1VgOaUqPjXpCTUJexqPwiwDQYr7wmPfSiEjW-z1YTcUi7yrQ@mail.gmail.com>

On Mon, May 21, 2012 at 12:48 PM, Siwek, Jonathan Luke
<jsiwek at illinois.edu> wrote:

> Do you have a stack trace you can send? ?If pcap_next() were returning a bogus pointer, I don't think you'd see the call to it in the stack, you'd be at a later point in the code where it attempted to access it and crashes. ?That is, if pcap_next() is in your stack trace, something bad is probably happening within the pcap library and the caller would never have the opportunity to check the return value.

I've attached a fairly recent backtrace from a core dump.

As you can see, the value being passed as pkt to net_packet_dispatch
triggers an "Address out of bounds" error.  PktSrc::Process calls
PktSrc::ExtractNextPacket which calls pcap_next.  The return value
from pcap_next sets the value for data, which PktSrc::Process
ultimately passes into net_packet_arrival as pkt.  So, either
pcap_next is returning an out of bounds pointer, or something happens
to data between the point in time when pcap_next returns a values and
PktSrc::Process calls net_packet_dispatch.

I've tried to identify what type of traffic might cause this to
happen, but unfortunately, nothing is jumping out at me.

Whatever it is, I don't think that data should be an out of bounds
address.  That's why I'm thinking that some preemptive error checking
may help.
-------------- next part --------------
Core was generated by `/usr/local/bro2/bin/bro -i eth1 -U .status -p broctl -p broctl-live -p local -p'.
Program terminated with signal 6, Aborted.
(gdb) bt
#0  0x00007f56d94a9a75 in raise () from /lib/libc.so.6
#1  0x00007f56d94ad5c0 in abort () from /lib/libc.so.6
#2  0x00007f56d9d5f8c5 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#3  0x00007f56d9d5dcf6 in ?? () from /usr/lib/libstdc++.so.6
#4  0x00007f56d9d5dd23 in std::terminate() () from /usr/lib/libstdc++.so.6
#5  0x00007f56d9d5de1e in __cxa_throw () from /usr/lib/libstdc++.so.6
#6  0x00007f56d9d5e2ad in operator new(unsigned long) () from /usr/lib/libstdc++.so.6
#7  0x00007f56d9d5e369 in operator new[](unsigned long) () from /usr/lib/libstdc++.so.6
#8  0x000000000060cfec in DataBlock (this=0xbcd2180, data=0x156b <Address 0x156b out of bounds>, size=6, arg_seq=<value optimized out>, arg_prev=0xa, arg_next=0x7f56db24d720) at /path/to/bro/src/Reassem.cc:23
#9  0x000000000060d3f1 in Reassembler::NewBlock (this=0xabb1c50, t=<value optimized out>, seq=<value optimized out>, len=<value optimized out>, data=0x7f56d8b3a05c <Address 0x7f56d8b3a05c out of bounds>) at /path/to/bro/src/Reassem.cc:85
#10 0x000000000059b912 in FragReassembler (this=0xabb1c50, arg_s=<value optimized out>, ip=0x7fff30688760, pkt=0x7f56d8b3a058 <Address 0x7f56d8b3a058 out of bounds>, k=0x3baac90, t=<value optimized out>) at /path/to/bro/src/Frag.cc:63
#11 0x000000000063337e in NetSessions::NextFragment (this=0x33c5880, t=<value optimized out>, ip=0x7fff30688760, pkt=<value optimized out>) at /path/to/bro/src/Sessions.cc:733
#12 0x0000000000634581 in NetSessions::DoNextPacket (this=0x33c5880, t=<value optimized out>, hdr=0x338e7b0, ip_hdr=0x7fff30688760, pkt=0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>, hdr_size=<value optimized out>) at /path/to/bro/src/Sessions.cc:458
#13 0x00000000006362ca in NetSessions::NextPacket (this=0x33c5880, t=<value optimized out>, hdr=0x338e7b0, pkt=0x7f56d8b3a058 <Address 0x7f56d8b3a058 out of bounds>, hdr_size=0, pkt_elem=<value optimized out>) at /path/to/bro/src/Sessions.cc:279
#14 0x00000000005ef855 in net_packet_dispatch (t=<value optimized out>, hdr=0x338e7b0, pkt=0x7f56d8b3a058 <Address 0x7f56d8b3a058 out of bounds>, hdr_size=0, src_ps=0x338e770, pkt_elem=0x0) at /path/to/bro/src/Net.cc:352
#15 0x00000000005ff637 in PktSrc::Process (this=0x338e770) at /path/to/bro/src/PktSrc.cc:273
#16 0x00000000005efbbb in net_run () at /path/to/bro/src/Net.cc:445
#17 0x0000000000514aab in main (argc=<value optimized out>, argv=<value optimized out>) at /path/to/bro/src/main.cc:1034

From jsiwek at illinois.edu  Mon May 21 12:45:25 2012
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 21 May 2012 19:45:25 +0000
Subject: [Bro] pcap_next Question
In-Reply-To: <CAG1VgOaUqPjXpCTUJexqPwiwDQYr7wmPfSiEjW-z1YTcUi7yrQ@mail.gmail.com>
References: <CAG1VgOapQQbSmnOkZOO9dhY54XdaNKXy+Y3+ks6M4Hn_9UXFNQ@mail.gmail.com>
	<BC3C8AB0-B4C0-41A1-A2E7-39F71CE2C664@illinois.edu>
	<CAG1VgOaUqPjXpCTUJexqPwiwDQYr7wmPfSiEjW-z1YTcUi7yrQ@mail.gmail.com>
Message-ID: <523E76AD-5308-4D8A-9E96-F586755483E3@illinois.edu>


> So, either
> pcap_next is returning an out of bounds pointer, or something happens
> to data between the point in time when pcap_next returns a values and
> PktSrc::Process calls net_packet_dispatch.

Could be, but then I'd expect a segfault much earlier, at least before NetSessions::DoNextPacket() when the packet data is accessed.  In the end, your crash doesn't look related to accessing that data at all, it's an unhandled exception thrown from operator new[] (failure to allocate memory).

Another possibility could be that since GDB is working with optimized code (the reason why some arguments are "<value optimized out>"), the "out of bounds" indicators don't necessarily indicate a problem yet.  If you `./configure --enable-debug` then rebuild/reinstall to disable optimizations, you can see if stack traces for future crashes start reporting valid addresses for the pointer.

+Jon


From seth at icir.org  Tue May 22 10:17:55 2012
From: seth at icir.org (Seth Hall)
Date: Tue, 22 May 2012 13:17:55 -0400
Subject: [Bro] Bro Exchange 2012!
Message-ID: <FB79DBC0-84C5-4CDF-B299-D71386DFA1E0@icir.org>

http://blog.bro-ids.org/2012/05/announcing-bro-exchange-2012.html

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/



From seth at icir.org  Tue May 22 17:55:17 2012
From: seth at icir.org (Seth Hall)
Date: Tue, 22 May 2012 20:55:17 -0400
Subject: [Bro] Bro Exchange 2012!
In-Reply-To: <FB79DBC0-84C5-4CDF-B299-D71386DFA1E0@icir.org>
References: <FB79DBC0-84C5-4CDF-B299-D71386DFA1E0@icir.org>
Message-ID: <BF3CE4C6-2314-4DB0-BF71-2E4457A38FDA@icir.org>


On May 22, 2012, at 1:17 PM, Seth Hall wrote:

> http://blog.bro-ids.org/2012/05/announcing-bro-exchange-2012.html


I made a mistake with the dates earlier and we don't actually have solid dates quite yet.  I've updated the blog post and I will send out updates again when we have the dates solid.  Please consider the posting as a request for presentations.

Thanks,
  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From sheharbano.k at gmail.com  Wed May 23 03:05:43 2012
From: sheharbano.k at gmail.com (Sheharbano Khattak)
Date: Wed, 23 May 2012 15:05:43 +0500
Subject: [Bro] Event for syn-ack packet
Message-ID: <CAA6v-1ZHjS_u6KUJSBWKH2XkAn=XVAhrNx7ENg8rpaVxjhUZ4g@mail.gmail.com>

Hi,

I want to identify hosts within our monitored network that reply to certain
external IP addresses. The reply could be as short as a syn-ack. The event
connection_established is too late as it doesn't matter whether the
connection was established. All that matters is whether any of our hosts
replied to the external IP even if that means a single syn-ack packet. Do
we have an event that could be used to capture this information?

Regards,

-- 
Sheharbano Khattak

http://etheryell.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120523/33995580/attachment.html 

From seth at icir.org  Wed May 23 09:29:36 2012
From: seth at icir.org (Seth Hall)
Date: Wed, 23 May 2012 12:29:36 -0400
Subject: [Bro] Event for syn-ack packet
In-Reply-To: <CAA6v-1ZHjS_u6KUJSBWKH2XkAn=XVAhrNx7ENg8rpaVxjhUZ4g@mail.gmail.com>
References: <CAA6v-1ZHjS_u6KUJSBWKH2XkAn=XVAhrNx7ENg8rpaVxjhUZ4g@mail.gmail.com>
Message-ID: <480591AC-5631-48ED-B5B7-49E811F4032B@icir.org>


On May 23, 2012, at 6:05 AM, Sheharbano Khattak wrote:

> The reply could be as short as a syn-ack. The event connection_established is too late as it doesn't matter whether the connection was established.

Are you trying to reduce your latency in detecting something?  I guess I don't understand why connection_established is too late.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jsiwek at illinois.edu  Wed May 23 09:37:32 2012
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Wed, 23 May 2012 16:37:32 +0000
Subject: [Bro] Event for syn-ack packet
In-Reply-To: <CAA6v-1ZHjS_u6KUJSBWKH2XkAn=XVAhrNx7ENg8rpaVxjhUZ4g@mail.gmail.com>
References: <CAA6v-1ZHjS_u6KUJSBWKH2XkAn=XVAhrNx7ENg8rpaVxjhUZ4g@mail.gmail.com>
Message-ID: <C4AC77BD-F1B2-492A-AFF7-35F500AD8CAE@illinois.edu>

> 
> I want to identify hosts within our monitored network that reply to certain external IP addresses. The reply could be as short as a syn-ack. The event connection_established is too late as it doesn't matter whether the connection was established. All that matters is whether any of our hosts replied to the external IP even if that means a single syn-ack packet. Do we have an event that could be used to capture this information?

I think the main ways to get that information would be to inspect a connection record's "resp$state" or "history" fields to determine if a reply was made, and I'd say within a handler for "connection_state_remove" is a good place to do that if timing isn't critical.  A non-zero value of "resp$state" (not TCP_INACTIVE or not UDP_INACTIVE) or any lower-case letter in "history" would mean there was a reply.

Part of the default set of scripts that get loaded, base/protocols/conn/main.bro, already does some interpretation of endpoint states (at the time of handling "connection_state_remove") and puts that in the "Conn::Info::conn_sate" field.  Along with that, it also picks up the connection records "history" field and will log both.  There's more description of the meaning of those fields in that script's comments or online at: http://www.bro-ids.org/documentation/scripts/base/protocols/conn/main.html

So a simple example you can try (relying on that default "conn" scripts):

@load base/protocols/conn

global bad_stuff: set[addr] = { 1.2.3.4, 5.6.7.8 }

event Conn::log_conn(rec: Conn::Info)
    {
    if ( rec$id$orig_h in bad_stuff && /[a-z]/ in rec$history )
        {
        # do something like raise a notice that will generate email, etc.
        }
    }


From sheharbano.k at gmail.com  Wed May 23 09:37:48 2012
From: sheharbano.k at gmail.com (Sheharbano Khattak)
Date: Wed, 23 May 2012 21:37:48 +0500
Subject: [Bro] Event for syn-ack packet
In-Reply-To: <480591AC-5631-48ED-B5B7-49E811F4032B@icir.org>
References: <CAA6v-1ZHjS_u6KUJSBWKH2XkAn=XVAhrNx7ENg8rpaVxjhUZ4g@mail.gmail.com>
	<480591AC-5631-48ED-B5B7-49E811F4032B@icir.org>
Message-ID: <CAA6v-1bfEqsVtNcZLswX53f0ktnOKUn4TGH5qJUzgfBpwd10Kg@mail.gmail.com>

I have a list of C&C servers and i want to detect which hosts in our
network talk to them. The first case is simple, check all outbound
connections in which the destination is C&C IP. In the second case when a
C&C server takes the initiative and tries to connect to an internal host.
In some cases, it may not proceed to establishing a full connection. Hear
the syn-ack and leave it at that. Come back later or maybe that's it's idea
of 'i am at your service' messages. I need this event for the second case
as the connection may never be established at all, at least for the time
for which i have pcap trace.

On Wed, May 23, 2012 at 9:29 PM, Seth Hall <seth at icir.org> wrote:

>
> On May 23, 2012, at 6:05 AM, Sheharbano Khattak wrote:
>
> > The reply could be as short as a syn-ack. The event
> connection_established is too late as it doesn't matter whether the
> connection was established.
>
> Are you trying to reduce your latency in detecting something?  I guess I
> don't understand why connection_established is too late.
>
>  .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>


-- 
Sheharbano Khattak

http://etheryell.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120523/a2aeed83/attachment.html 

From vern at icir.org  Wed May 23 09:42:57 2012
From: vern at icir.org (Vern Paxson)
Date: Wed, 23 May 2012 09:42:57 -0700
Subject: [Bro] Event for syn-ack packet
In-Reply-To: <CAA6v-1bfEqsVtNcZLswX53f0ktnOKUn4TGH5qJUzgfBpwd10Kg@mail.gmail.com>
	(Wed, 23 May 2012 21:37:48 +0500).
Message-ID: <20120523164257.DE8872C4022@rock.ICSI.Berkeley.EDU>

To clarify, a SYN-ACK in response to a SYN is enough for Bro to generate
connection_established.  It doesn't actually look for a full 3-way handshake
(i.e., an ACK of the SYN-ACK).  Does that help?  Alternatively, if you have
traces you can share that demonstrate a failure to get the
connection_established event, then we can look into just what's going on.

		Vern


From sheharbano.k at gmail.com  Wed May 23 09:58:44 2012
From: sheharbano.k at gmail.com (Sheharbano Khattak)
Date: Wed, 23 May 2012 21:58:44 +0500
Subject: [Bro] Event for syn-ack packet
In-Reply-To: <20120523164257.DE8872C4022@rock.ICSI.Berkeley.EDU>
References: <CAA6v-1bfEqsVtNcZLswX53f0ktnOKUn4TGH5qJUzgfBpwd10Kg@mail.gmail.com>
	<20120523164257.DE8872C4022@rock.ICSI.Berkeley.EDU>
Message-ID: <CAA6v-1bLxyioGr6-BGx3VqzNRT5FRs7b_HW7FLPcMh0u9EAKpA@mail.gmail.com>

Thanks. I thought the event connection_established was generated after the
initial 3-way handshake is completed as mentioned here:

http://bro-ids.org/documentation/scripts/base/event.bif.html#id-connection_established

On Wed, May 23, 2012 at 9:42 PM, Vern Paxson <vern at icir.org> wrote:

> To clarify, a SYN-ACK in response to a SYN is enough for Bro to generate
> connection_established.  It doesn't actually look for a full 3-way
> handshake
> (i.e., an ACK of the SYN-ACK).  Does that help?  Alternatively, if you have
> traces you can share that demonstrate a failure to get the
> connection_established event, then we can look into just what's going on.
>
>                Vern
>



-- 
Sheharbano Khattak

http://etheryell.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120523/bad0e5d5/attachment.html 

From vern at icir.org  Wed May 23 10:02:02 2012
From: vern at icir.org (Vern Paxson)
Date: Wed, 23 May 2012 10:02:02 -0700
Subject: [Bro] Event for syn-ack packet
In-Reply-To: <CAA6v-1bLxyioGr6-BGx3VqzNRT5FRs7b_HW7FLPcMh0u9EAKpA@mail.gmail.com>
	(Wed, 23 May 2012 21:58:44 +0500).
Message-ID: <20120523170202.4C2442C401F@rock.ICSI.Berkeley.EDU>

> Thanks. I thought the event connection_established was generated after the
> initial 3-way handshake is completed as mentioned here:

Yeah, that's in fact a documentation glitch :-(.  That describes what
probably *should* be done, but in fact the event is generated on seeing
the SYN-ACK (I just double-checked the code).  I wrote it that way eons
ago when Bro often operated on TCP streams that had been filtered to
SYN/FIN/RST packets only, which meant it wouldn't see the pure ACK completing
the handshake.

		Vern


From jsiwek at illinois.edu  Wed May 23 10:06:51 2012
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Wed, 23 May 2012 17:06:51 +0000
Subject: [Bro] Event for syn-ack packet
In-Reply-To: <20120523164257.DE8872C4022@rock.ICSI.Berkeley.EDU>
References: <20120523164257.DE8872C4022@rock.ICSI.Berkeley.EDU>
Message-ID: <F5A5AB10-23A2-4C7B-96FC-6F8CA45DC802@illinois.edu>


On May 23, 2012, at 11:42 AM, Vern Paxson wrote:

> To clarify, a SYN-ACK in response to a SYN is enough for Bro to generate
> connection_established.  It doesn't actually look for a full 3-way handshake
> (i.e., an ACK of the SYN-ACK).  Does that help?

Ok, my confusion was that the comment for that event in event.bif was "The event is raised when the initial 3-way TCP handshake has successfully finished for a connection.", but actually testing it out it seems to be generated for just syn/syn-ack exchanges with nothing further.  I'll update that comment unless there's some other subtlety about why it's worded that way.

One caveat could still be that connection_established is TCP-specific, the example I gave could be used for UDP "connections", too.


From sheharbano.k at gmail.com  Wed May 23 10:07:07 2012
From: sheharbano.k at gmail.com (Sheharbano Khattak)
Date: Wed, 23 May 2012 22:07:07 +0500
Subject: [Bro] Event for syn-ack packet
In-Reply-To: <20120523170202.4C2442C401F@rock.ICSI.Berkeley.EDU>
References: <CAA6v-1bLxyioGr6-BGx3VqzNRT5FRs7b_HW7FLPcMh0u9EAKpA@mail.gmail.com>
	<20120523170202.4C2442C401F@rock.ICSI.Berkeley.EDU>
Message-ID: <CAA6v-1YuxGRVT3g9HAAi52ZuCf7R-CQ81oCU2hU63ZDaWfkiBQ@mail.gmail.com>

If someone tries to open up several half open connections to our host, how
will we know if we don't distinguish between SYN-ACK and ACK ? This implies
that a connection for which an ACK was never heard would still be treated
as an established connection.

On Wed, May 23, 2012 at 10:02 PM, Vern Paxson <vern at icir.org> wrote:

> > Thanks. I thought the event connection_established was generated after
> the
> > initial 3-way handshake is completed as mentioned here:
>
> Yeah, that's in fact a documentation glitch :-(.  That describes what
> probably *should* be done, but in fact the event is generated on seeing
> the SYN-ACK (I just double-checked the code).  I wrote it that way eons
> ago when Bro often operated on TCP streams that had been filtered to
> SYN/FIN/RST packets only, which meant it wouldn't see the pure ACK
> completing
> the handshake.
>
>                Vern
>



-- 
Sheharbano Khattak

http://etheryell.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120523/d4c91e4b/attachment.html 

From vern at icir.org  Wed May 23 10:47:44 2012
From: vern at icir.org (Vern Paxson)
Date: Wed, 23 May 2012 10:47:44 -0700
Subject: [Bro] Event for syn-ack packet
In-Reply-To: <F5A5AB10-23A2-4C7B-96FC-6F8CA45DC802@illinois.edu> (Wed,
	23 May 2012 17:06:51 -0000).
Message-ID: <20120523174744.2CCC12C401F@rock.ICSI.Berkeley.EDU>

> Ok, my confusion was that the comment for that event in event.bif was "The event is raised when the initial 3-way TCP handshake has successfully finished for a connection."

Yeah, I think you can pin the blame on me for that comment.

> I'll update that comment unless there's some other subtlety about why it's worded that way.

Updating it would be good.

> One caveat could still be that connection_established is TCP-specific, the example I gave could be used for UDP "connections", too.

I don't believe we generate connection_established in a UDP context.  There
it's instead udp_request and udp_reply.  I'm not sure what example you're
referring to.

		Vern


From vern at icir.org  Wed May 23 10:55:49 2012
From: vern at icir.org (Vern Paxson)
Date: Wed, 23 May 2012 10:55:49 -0700
Subject: [Bro] Event for syn-ack packet
In-Reply-To: <CAA6v-1YuxGRVT3g9HAAi52ZuCf7R-CQ81oCU2hU63ZDaWfkiBQ@mail.gmail.com>
	(Wed, 23 May 2012 22:07:07 +0500).
Message-ID: <20120523175549.3EAB52C401F@rock.ICSI.Berkeley.EDU>

> If someone tries to open up several half open connections to our host, how
> will we know if we don't distinguish between SYN-ACK and ACK ?

I'm not sure I understand your concern here.  Connections are identified
by their five-tuple.  If the five-tuple for an active connection is reused,
the two instances will be treated as a single connection; that would be
the case regardless of whether the connection has seen a 2-packet SYN
exchange or a full 3-way handshake.

In terms of TCP semantics, a connection that's only had a 2-packet SYN
exchange is still active, and shouldn't be reused.  If the handshake
never completes, it will eventually be torn down with a RST - which will
also cause Bro to consider it no longer active.

		Vern


From jsiwek at illinois.edu  Wed May 23 12:12:15 2012
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Wed, 23 May 2012 19:12:15 +0000
Subject: [Bro] Event for syn-ack packet
In-Reply-To: <20120523174744.2CCC12C401F@rock.ICSI.Berkeley.EDU>
References: <20120523174744.2CCC12C401F@rock.ICSI.Berkeley.EDU>
Message-ID: <E3927C06-BACF-42E3-948B-2E6386F6CC65@illinois.edu>


>> One caveat could still be that connection_established is TCP-specific, the example I gave could be used for UDP "connections", too.
> 
> I don't believe we generate connection_established in a UDP context.  There
> it's instead udp_request and udp_reply.  I'm not sure what example you're
> referring to.

Right, I meant the example of checking a connection's "history" field for any lower-case letters should indicate the responding side sent some type of packet during the connection lifetime and wouldn't need special handling for UDP separately from TCP.

And if it's important to distinguish half-open TCP connections from ones that complete a 3-way handshake, "history" looks like it could do that, too.


From vern at icir.org  Wed May 23 13:21:41 2012
From: vern at icir.org (Vern Paxson)
Date: Wed, 23 May 2012 13:21:41 -0700
Subject: [Bro] Event for syn-ack packet
In-Reply-To: <E3927C06-BACF-42E3-948B-2E6386F6CC65@illinois.edu> (Wed,
	23 May 2012 19:12:15 -0000).
Message-ID: <20120523202141.1F88C2C4023@rock.ICSI.Berkeley.EDU>

> Right, I meant the example of checking a connection's "history" field
> for any lower-case letters should indicate the responding side sent some
> type of packet ...

Ah, right.  Yes, that's indeed the right way to discern exactly what occurred.

		Vern


From krkhan at inspirated.com  Sun May 27 04:49:20 2012
From: krkhan at inspirated.com (Kamran Riaz Khan)
Date: Sun, 27 May 2012 16:49:20 +0500
Subject: [Bro] Using expire attributes for nested compound types
Message-ID: <4FC214C0.4030804@inspirated.com>

Hello,

If I have a table of set, it's pretty straight-forward to add expiring
timers for the parent table:

--
global test_table: table[count] of set[string]
	&read_expire = 1 day &expire_func = test_expire;
--

However, I can't figure out a way to define {read,write,create}_expire
attributes for the *child*, i.e., set[string].

I have tried defining the timers when adding the table entry:

--
test_table[0] = set("Hello", "World") &create_expire = 1 usec;
--

But the string entries do not expire 1 usec after creation. Any ideas on
how to accomplish this?

Regards,
-- 
Kamran Riaz Khan.

http://inspirated.com/



From krkhan at inspirated.com  Sun May 27 05:01:14 2012
From: krkhan at inspirated.com (Kamran Riaz Khan)
Date: Sun, 27 May 2012 17:01:14 +0500
Subject: [Bro] Using expire attributes for nested compound types
In-Reply-To: <4FC214C0.4030804@inspirated.com>
References: <4FC214C0.4030804@inspirated.com>
Message-ID: <4FC2178A.7000403@inspirated.com>

On 05/27/2012 04:49 PM, Kamran Riaz Khan wrote:
> If I have a table of set, it's pretty straight-forward to add expiring
> timers for the parent table:
> 
> --
> global test_table: table[count] of set[string]
> 	&read_expire = 1 day &expire_func = test_expire;
> --
> 
> However, I can't figure out a way to define {read,write,create}_expire
> attributes for the *child*, i.e., set[string].
> 
> I have tried defining the timers when adding the table entry:
> 
> --
> test_table[0] = set("Hello", "World") &create_expire = 1 usec;
> --
> 
> But the string entries do not expire 1 usec after creation. Any ideas on
> how to accomplish this?

Okay, this works:

--
local entry: set[string] &create_expire = 1 day;

add entry["Hello"];
add entry["World"];
test_table[0] = entry;
--

Regards,
-- 
Kamran Riaz Khan.

http://inspirated.com/



From robin at icir.org  Sun May 27 08:32:47 2012
From: robin at icir.org (Robin Sommer)
Date: Sun, 27 May 2012 08:32:47 -0700
Subject: [Bro] Using expire attributes for nested compound types
In-Reply-To: <4FC2178A.7000403@inspirated.com>
References: <4FC214C0.4030804@inspirated.com> <4FC2178A.7000403@inspirated.com>
Message-ID: <20120527153247.GA50219@icir.org>


On Sun, May 27, 2012 at 17:01 +0500, Kamran Riaz Khan wrote:

> local entry: set[string] &create_expire = 1 day;

Using a local is indeed the work-around I'd have suggested. There's a
bit of an inconsistency currently in terms of how the interpreter
associates attributes with values.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From greencm at gmail.com  Mon May 28 12:48:06 2012
From: greencm at gmail.com (Chris Green)
Date: Mon, 28 May 2012 14:48:06 -0500
Subject: [Bro] Debugging Policies and Print Statement
Message-ID: <CAJUMos5PtzQrLG+MSfgY3ivxhjpD7RBfb_GYOXR4+5k7pLGqGA@mail.gmail.com>

Good day,

I'm using bro version 2.0-372 with a new git checkout from master as of
2012-05-27 on Centos6.   I'm having an issue where the print statement in
the policy debugger displays nothing. Breakpoints and stack tracks seem to
work correctly but not being able to inspect variables is making playing
with policies frustrating.

Is this a known issue or is there some type of preprocessing make
inspection work?

Thanks,
Chris

In bro_init() at /usr/local/bro/share/bro/base/utils/site.bro:155
155        local_dns_suffix_regex = set_to_regex(local_zones,
"(^\\.?|\\.)(~~)$");
(Bro [0]) break http_entity_data
Setting breakpoint on http_entity_data:
There are multiple definitions of that event handler.
Please choose one of the following options:
[1] /usr/local/bro/share/bro/base/protocols/http/./file-ident.bro:79
[2] /usr/local/bro/share/bro/base/protocols/http/./file-hash.bro:37
[3] /usr/local/bro/share/bro/base/protocols/http/./file-extract.bro:32
[4] /usr/local/bro/share/bro/base/protocols/http/./file-ident.bro:85
[a] All of the above
[n] None of the above
Enter your choice: 2
Breakpoint 1 set at http_entity_data at
/usr/local/bro/share/bro/base/protocols/http/./file-hash.bro:37
(Bro [1]) c
Continuing.
Breakpoint 1, http_entity_data(c = '[id=[orig_h=192.168.1.13,
orig_p=50402/tcp, resp_h=209.132.180.131, resp_p=80/tcp], orig=[size=121,
state=4, num_pkts=4, num_bytes_ip=216, flow_label=0], resp=[size=3095,
state=4, num_pkts=4, num_bytes_ip=1963, flow_label=0],
start_time=1338087055.218736, duration=0.27459, service={

}, addl=, hot=0, history=ShACad, uid=cTLeNAfgQo2, dpd=<uninitialized>,
conn=[ts=1338087055.218736, uid=cTLeNAfgQo2, id=[orig_h=192.168.1.13,
orig_p=50402/tcp, resp_h=209.132.180.131, resp_p=80/tcp], proto=tcp,
service=<uninitialized>, duration=<uninitialized>,
orig_bytes=<uninitialized>, resp_bytes=<uninitialized>,
conn_state=<uninitialized>, local_orig=T, missed_bytes=121,
history=<uninitialized>, orig_pkts=<uninitialized>,
orig_ip_bytes=<uninitialized>, resp_pkts=<uninitialized>,
resp_ip_bytes=<uninitialized>], extract_orig=F, extract_resp=F,
dns=<uninitialized>, dns_state=<uninitialized>, ftp=<uninitialized>,
http=[ts=1338087055.413572, uid=cTLeNAfgQo2, id=[orig_h=192.168.1.13,
orig_p=50402/tcp, resp_h=209.132.180.131, resp_p=80/tcp], trans_depth=0,
method=<uninitialized>, host=<uninitialized>, uri=<uninitialized>,
referrer=<uninitialized>, user_agent=<uninitialized>, request_body_len=0,
response_body_len=0, status_code=200, status_msg=OK,
info_code=<uninitialized>, info_msg=<uninitialized>,
filename=<uninitialized>, tags={

}, username=<uninitialized>, password=<uninitialized>, capture_password=F,
proxied=<uninitialized>, mime_type=application/x-dosexec, first_chunk=T,
md5=<uninitialized>, calc_md5=F, calculating_md5=F,
extraction_file=<uninitialized>, extract_file=F], http_state=[pending={
    [0] = [ts=1338087055.413572, uid=cTLeNAfgQo2, id=[orig_h=192.168.1.13,
orig_p=50402/tcp, resp_h=209.132.180.131, resp_p=80/tcp], trans_depth=0,
method=<uninitialized>, host=<uninitialized>, uri=<uninitialized>,
referrer=<uninitialized>, user_agent=<uninitialized>, request_body_len=0,
response_body_len=0, status_code=<uninitialized>,
status_msg=<uninitialized>, info_code=<uninitialized>,
info_msg=<uninitialized>, filename=<uninitialized>, tags={

    }, username=<uninitialized>, password=<uninitialized>,
capture_password=F, proxied=<uninitialized>, mime_type=<uninitialized>,
first_chunk=T, md5=<uninitialized>, calc_md5=F, calculating_md5=F,
extraction_file=<uninitialized>, extract_file=F],
    [1] = [ts=1338087055.413572, uid=cTLeNAfgQo2, id=[orig_h=192.168.1.13,
orig_p=50402/tcp, resp_h=209.132.180.131, resp_p=80/tcp], trans_depth=0,
method=<uninitialized>, host=<uninitialized>, uri=<uninitialized>,
referrer=<uninitialized>, user_agent=<uninitialized>, request_body_len=0,
response_body_len=0, status_code=200, status_msg=OK,
info_code=<uninitialized>, info_msg=<uninitialized>,
filename=<uninitialized>, tags={

    }, username=<uninitialized>, password=<uninitialized>,
capture_password=F, proxied=<uninitialized>,
mime_type=application/x-dosexec, first_chunk=T, md5=<uninitialized>,
calc_md5=F, calculating_md5=F, extraction_file=<uninitialized>,
extract_file=F]
}, current_request=0, current_response=1], irc=<uninitialized>,
smtp=<uninitialized>, smtp_state=<uninitialized>, ssh=<uninitialized>,
ssl=<uninitialized>, syslog=<uninitialized>, known_services_done=F]',
is_orig = 'F', length = '1500', data =
'MZ\x90\0^C\0\0\0^D\0\0\0\xff\xff\0\0\xb8\0\0\0\0\0\0\0@\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x80\0\0\0^N\x1f\xba^N\0\xb4^I\xcd!\xb8^AL\xcd!This
program cannot be run in DOS
mode.^M^M^J$\0\0\0\0\0\0\0PE\0\0L^A^C\0`\xdf\x9cO\0\xe2!\0\0\0\0\0\xe0\0/^C^K^A^B^V\0\xd0^I\0\0\0^A\0\0\xc0^Y\0^P\x85#\0\0\xd0^Y\0\0\xa0#\0\0\0@\0\0^P\0\0\0^B\0\0^D\0\0\0^A\0\0\0^D\0\0\0\0\0\0\0\0\xa0$\0\0^P\0\0\0\0\0\0^B\0\0\x80\0\0
\0\0^P\0\0\0\0^P\0\0^P\0\0\0\0\0\0^P\0\0\0\0\0\0\0\0\0\0\0^T\x9a$\0l^B\0\0\0\xa0#\0^T\xfa\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xf0\x90#\0^X\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0UPX0\0\0\0\0\0\xc0^Y\0\0^P\0\0\0\0\0\0\0^B\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x80\0\0\xe0UPX1\0\0\0\0\0\xd0^I\0\0\xd0^Y\0\0\xc2^I\0\0^B\0\0\0\0\0\0\0\0\0\0\0\0\0\0@
\0\0\xe0.rsrc\0\0\0\0\0^A\0\0\xa0#\0\0\xfe\0\0\0\xc4^I\0\0\0\0\0\0\0\0\0\0\0\0\0@\0\0\xc03.07\0UPX!^M^I^N^G\x8c*]\xc2\xf4\x96\xc3\xc8\xd3e#\0\xf2\xb4^I\0\x1d\xe2!\0&\x1e\0\xb0^Z^C\0*\xa2X\xa54\x1ec\x1d\xbe^Wo\xc6UdK\xb5\xd9/kuB"\x9e^Q\xa5e\xe73\x1f\xc6Z\x91^Q\x8e\xd8\x97^BF\xba^P\xbdp\x92\xb4"7/\x92,\xf3\x9eg\x90\xfe^T\xfa|\xb7\xba\x8d0^T,\xf6K\x9a\xdf`:\x9e\xceJ\xe8\x98u\xa4(\xe2(\xd0H)^C\x90\x95>q\xca
\xfa^D\x963\xff2\x86QD'\xe0\xf5\x9a,\x9d^H4V\xda\xd5\xd0\xb5[^E|\x8e\xfd\x83\xd8^U\xa9\xd7^F^S\xa7\xe9\x91\xc7
~\xd3\xc7^B\x90\x1en\xbb\xe6\xa2\xb5\x8e\xe7\xf7\xbe\xc5\x86g\xe2^W\xff\xd2RG\xd6\xb2P\xd9\x8a\x8f\xf5^D]B|i\x9c\x97W\xf8^NL\x8a\x9dsk\xdac\xb5^V\xe3\xa0\xf14\x94^K\xb8\xff^ER\xfa\xe4\xf8\x955\xdd\xa1\xa8\0^Y^Qr/^E\x9a\xb4\xee}\xde\xb7@
\xf2\x92^N/\xe4L\xc5\xd3\xebT\xcb\xa1&\x90^A\xbcC\xd9\xdaW\x9f^A^Vbt\x9f\xb1\x81\x1dX^Mx^\xeb6^H{v\xeeA\xeeAA\x9eD\x81E\xe1\xdeA\xa8\x1f^U\xe6\xd2\xbcr\xf1cIc\xac^TB\x8d\xd8\x8a\xf5\xd1L\xac\xd9\xad\xbb%\xc8=^J\x87\x1b\x89\xa46\x8d;d\xfd\xd8$\xf7=Y^I\xdev\x85]h^BNT\x9c^UL8\xb9%M^Y\x83\xc0C^G\xa1\xd9L^S\x854\xdfi^A\xf8;\xe9\x90^Hj@
^\x1fO^S:\xfc%\xf9*\\x98\x1e\xf2b\xcc\xb3^Y^Ga\xa0\xf5tw;\xddg^?^CG\x8f\x88\xef^B\xbb\xcd=^Thn\x1b\xa3^O\xf1^E\xb5\xa1\0@
\x85\xe0\x8b\xf5^GW\xc0\x94\x8a\xee\xa7^T\x99\xd7qq'6\xa9X\xe4#;\xefg4D\x82^G\l\xbbH\x82\xbc\xc2G\xf7\xca\x9f\xa5K@
\x1e\xe7lB^O\xe3\xbd+\xf4\xa1\x89\xd1@
\xc4\xc6\xd6\x1b^U^Fpf\x89h\xc3G\xf3\x88o^Fg\xe7#q\x9d\x1d\xd9\xa1,\x81\xc6\x9bF^R^P^\xd5H)()^T8\xebT\x86\xc0\xd8\x1c_\x95\xe6\xab\x82l=\x82v\x90\xb2g\xf6O\xecE(\xa0\xceR\6)^OX[\xea\xcf\x89^Qd\xbeg^Q\xc7\xc0\xed\x90FG\x80:/!b\x1d^S\xa8\x89\x95\x8e>\xea\xe1s|^?,\xf2\xe6d\xb8\xee)\xd9Ti\xefO\xa2\xdb\xab\xeb\xf1\xe6\xc70W\x89\xdam\xbd\x8ewiS\xc5\x95^G^C\xadM\xbf^K\xdb\xed\xa0\x97K\xe9\x1c\xf2^?\x92k\x92Vk\x89>\xb9l\xff\x80\x97^UXN\xbf\x90[\xe6AH)X\xe99\xce\xb5\xf0>^E'\xb1C^J\xe0\x82\xc1\xffj3u\xe8\xbb\xaeyE#\xb8\xbe\xa7\xa9\xeb\x93{\xa3\x90^Y\xe1,[.Y\xf3^Z\xa8\xa5\x99\x96\xc5sI=\xa5\xf0~\xc1^M\xe2\xc7RK^XF\x80\x1bhX\xdc\xc4P\xcb^P^W^Z\xe2\x8a\xdb'@a
;\xa9\xa0^G"+d\x9f\xa0\xd1j\x993\x83^A\xaa\x80\xfc\xc3\x8f\xf3\xf2\xd1\x96kG\xa8\xf7\xb1\xf4\xe6\xd4\x9d\xc4Y\xbc\xd9\xc1\xaf^D^N\xce^Q\xf6\xd9H^E^?\xe6\xfc\xe6\x8210\x1f\x86yrO\x96s\xffU^R3^V\xa6^UD\xb3^Nt\xed"\x92\xa3``\xd4^Fn\xdb^Lf\xd5\xc2^K1^E\xdf\xa6\xe7\xbc\xb8'=\xa9\xeft\xeb;O1R\x1er\xccC&\x99\xba\x85t\xfa\xc8\xe7^D\xdc5\x1bN at M\x8cFrB2\xe3\x83^So^U\xcf\x9cu\xde9\xe7xV"$Q\^K\xef^U^M\xbd\x99\xd5I^EgS\xfe\xaf\xa3^I\x80P\xad!\x88\x97;\x85\xf1\x95\xed^Y^Im\xd9\x93\x9e4O\xe4Y^S\x88\x88^F\xa8'^U_}\xa1\xcbh.;\xe6\xc7\xac^KSGvF<\xe4i\xe7GN\0rV\xbd\xcf}`\x84\xa5^G\xe6\xa1U\xc6\x91,^Ct\xe0A\x99\x9d^O\xffj\xe8\xf4n\x86^^J\xf1\x91.\xaa\xc0^Y?\x8bV^N\xc1\x8aJPhh\xfa\xbc\xf3\x9e\xd1yX\0\x1f$^Ml(\xe6\xbd^Ji\xd9')
at /usr/local/bro/share/bro/base/protocols/http/./file-hash.bro:37
In http_entity_data(c = '[id=[orig_h=192.168.1.13, orig_p=50402/tcp,
resp_h=209.132.180.131, resp_p=80/tcp], orig=[size=121, state=4,
num_pkts=4, num_bytes_ip=216, flow_label=0], resp=[size=3095, state=4,
num_pkts=4, num_bytes_ip=1963, flow_label=0], start_time=1338087055.218736,
duration=0.27459, service={

}, addl=, hot=0, history=ShACad, uid=cTLeNAfgQo2, dpd=<uninitialized>,
conn=[ts=1338087055.218736, uid=cTLeNAfgQo2, id=[orig_h=192.168.1.13,
orig_p=50402/tcp, resp_h=209.132.180.131, resp_p=80/tcp], proto=tcp,
service=<uninitialized>, duration=<uninitialized>,
orig_bytes=<uninitialized>, resp_bytes=<uninitialized>,
conn_state=<uninitialized>, local_orig=T, missed_bytes=121,
history=<uninitialized>, orig_pkts=<uninitialized>,
orig_ip_bytes=<uninitialized>, resp_pkts=<uninitialized>,
resp_ip_bytes=<uninitialized>], extract_orig=F, extract_resp=F,
dns=<uninitialized>, dns_state=<uninitialized>, ftp=<uninitialized>,
http=[ts=1338087055.413572, uid=cTLeNAfgQo2, id=[orig_h=192.168.1.13,
orig_p=50402/tcp, resp_
37        if ( is_orig || ! c?$http ) return;
(Bro [2])  print c
(Bro [3])  print is_orig


-- 
Chris Green <greencm at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120528/b766e424/attachment.html 

From robin at icir.org  Tue May 29 08:25:02 2012
From: robin at icir.org (Robin Sommer)
Date: Tue, 29 May 2012 08:25:02 -0700
Subject: [Bro] Debugging Policies and Print Statement
In-Reply-To: <CAJUMos5PtzQrLG+MSfgY3ivxhjpD7RBfb_GYOXR4+5k7pLGqGA@mail.gmail.com>
References: <CAJUMos5PtzQrLG+MSfgY3ivxhjpD7RBfb_GYOXR4+5k7pLGqGA@mail.gmail.com>
Message-ID: <20120529152502.GF79263@icir.org>


On Mon, May 28, 2012 at 14:48 -0500, you wrote:

> the policy debugger displays nothing. Breakpoints and stack tracks seem to
> work correctly but not being able to inspect variables is making playing
> with policies frustrating.

The debugger hasn't seen much attention for a while unfortunately, and
some of its functionality isn't working at the moment. We have a
ticket tracking that, http://tracker.bro-ids.org/bro/ticket/678, but
it doesn't look like we'll get to it in time for the next release.

Fwiw, I often just insert print statements into the code while
developing/debugging.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From jsiwek at illinois.edu  Tue May 29 10:48:57 2012
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Tue, 29 May 2012 17:48:57 +0000
Subject: [Bro] Debugging Policies and Print Statement
In-Reply-To: <CAJUMos5PtzQrLG+MSfgY3ivxhjpD7RBfb_GYOXR4+5k7pLGqGA@mail.gmail.com>
References: <CAJUMos5PtzQrLG+MSfgY3ivxhjpD7RBfb_GYOXR4+5k7pLGqGA@mail.gmail.com>
Message-ID: <8998D311-561E-46C9-AC96-10D2D1721DA8@illinois.edu>


> I'm using bro version 2.0-372 with a new git checkout from master as of 2012-05-27 on Centos6.   I'm having an issue where the print statement in the policy debugger displays nothing.

It seems that parameters of functions/events take on the module scoping in which they're declared.  So in your case, breaking in the "http_entity_data" in file-hash.bro is inside the "HTTP" module (there's the "module HTTP;" statement near the top of that script).  So qualifying local variables or event parameters with the module name in the script debugger should work:

print HTTP::is_orig
print HTTP::c
print HTTP::c$id

> Breakpoints and stack tracks seem to work correctly

I also noticed that the context string for stack traces is using a static buffer size, so it's getting truncated sometimes and the filename:line won't be there.  I can fix that and also give some better error messages for when the 'print' command fails to parse the given args and put that on the fastpath branch, but I'm not sure how bad it will be to try to change local identifiers to not inherit module scoping, or even if that would be "right".

    Jon


From christopher.p.crawford at gmail.com  Wed May 30 07:39:33 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Wed, 30 May 2012 10:39:33 -0400
Subject: [Bro] broctl Email Reports
Message-ID: <CAG1VgObyz84xBbaUOn5vuPvvy6huV_RA19Aw51nQFp757Cobgw@mail.gmail.com>

I like that broctl will roll logs over every hour.  My default
broctl.cfg file includes:

# Rotation interval in seconds for log files on manager/standalone node.
LogRotationInterval = 3600

I don't like getting an email from broctl every hour, though.  Is
there a way to get a daily report, instead of an hourly report?


Related --

The Bro README [1] claims:

"BroControl sends four types of mails to the address given in MailTo:

1. When logs are rotated (per default once a day), a list of all
alarms during the last rotation interval is sent. This can be disabled
by setting MailAlarms=0."

But elsewhere in the README:

"LogRotationInterval (int, default 3600)
    The frequency of log rotation in seconds for the manager/standalone node."

This is confusing to me -- maybe someone can help me understand.  Are
they talking about two different things?

[1] http://www.bro-ids.org/documentation/components/broctl/README.html


From mcholste at gmail.com  Wed May 30 07:48:33 2012
From: mcholste at gmail.com (Martin Holste)
Date: Wed, 30 May 2012 09:48:33 -0500
Subject: [Bro] broctl Email Reports
In-Reply-To: <CAG1VgObyz84xBbaUOn5vuPvvy6huV_RA19Aw51nQFp757Cobgw@mail.gmail.com>
References: <CAG1VgObyz84xBbaUOn5vuPvvy6huV_RA19Aw51nQFp757Cobgw@mail.gmail.com>
Message-ID: <CANpnLHgaDsM-NGtF7tNB9x-wUPFpQmG04SYO5_oXKbq4X7jaEg@mail.gmail.com>

You sound like a perfect candidate for someone who wants to get their
logs into a frontend for reporting like Splunk or my ELSA project.  I
have a how-to available here:
ossectools.blogspot.com/2011/09/bro-quickstart-cluster-edition.html .
This will let you do reporting and alerting at whatever interval
you're looking for.

On Wed, May 30, 2012 at 9:39 AM, Chris Crawford
<christopher.p.crawford at gmail.com> wrote:
> I like that broctl will roll logs over every hour. ?My default
> broctl.cfg file includes:
>
> # Rotation interval in seconds for log files on manager/standalone node.
> LogRotationInterval = 3600
>
> I don't like getting an email from broctl every hour, though. ?Is
> there a way to get a daily report, instead of an hourly report?
>
>
> Related --
>
> The Bro README [1] claims:
>
> "BroControl sends four types of mails to the address given in MailTo:
>
> 1. When logs are rotated (per default once a day), a list of all
> alarms during the last rotation interval is sent. This can be disabled
> by setting MailAlarms=0."
>
> But elsewhere in the README:
>
> "LogRotationInterval (int, default 3600)
> ? ?The frequency of log rotation in seconds for the manager/standalone node."
>
> This is confusing to me -- maybe someone can help me understand. ?Are
> they talking about two different things?
>
> [1] http://www.bro-ids.org/documentation/components/broctl/README.html
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From robin at icir.org  Wed May 30 11:27:55 2012
From: robin at icir.org (Robin Sommer)
Date: Wed, 30 May 2012 11:27:55 -0700
Subject: [Bro] broctl Email Reports
In-Reply-To: <CAG1VgObyz84xBbaUOn5vuPvvy6huV_RA19Aw51nQFp757Cobgw@mail.gmail.com>
References: <CAG1VgObyz84xBbaUOn5vuPvvy6huV_RA19Aw51nQFp757Cobgw@mail.gmail.com>
Message-ID: <20120530182755.GA97952@icir.org>


On Wed, May 30, 2012 at 10:39 -0400, you wrote:

> I don't like getting an email from broctl every hour, though.  Is
> there a way to get a daily report, instead of an hourly report?

It's indeed coupled to log rotation currently, but you can change that
by redefining the rotation interval for the alarm summaries. Try this
in local.bro:

    event bro_init()
        {
        local f = Log::get_filter(Notice::ALARM_LOG, "alarm-mail");
        f$interv = 1day;
        Log::add_filter(Notice::ALARM_LOG, f);
        }

> 1. When logs are rotated (per default once a day),

Ah, that's outdated, the default log rotation used to be once a day,
but is now once an hour.

> "LogRotationInterval (int, default 3600)

We should add a second option here that defines the rotation interval
for the alarm summaries separately.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From christopher.p.crawford at gmail.com  Wed May 30 12:04:05 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Wed, 30 May 2012 15:04:05 -0400
Subject: [Bro] broctl Email Reports
In-Reply-To: <20120530182755.GA97952@icir.org>
References: <CAG1VgObyz84xBbaUOn5vuPvvy6huV_RA19Aw51nQFp757Cobgw@mail.gmail.com>
	<20120530182755.GA97952@icir.org>
Message-ID: <CAG1VgOZWxDzr4J5TN91Sikk8cgWUOBKPEoHeMuf2fS82LoL0Hw@mail.gmail.com>

Thanks Robin - I think this is exactly what I was looking for.

I tried adding what you recommended to my local.bro and then did a

broctl install
broctl update

but the connection summary email is still going out on the hour.

Is there something else that I need to do?

-Chris

On Wed, May 30, 2012 at 2:27 PM, Robin Sommer <robin at icir.org> wrote:
>
> On Wed, May 30, 2012 at 10:39 -0400, you wrote:
>
>> I don't like getting an email from broctl every hour, though. ?Is
>> there a way to get a daily report, instead of an hourly report?
>
> It's indeed coupled to log rotation currently, but you can change that
> by redefining the rotation interval for the alarm summaries. Try this
> in local.bro:
>
> ? ?event bro_init()
> ? ? ? ?{
> ? ? ? ?local f = Log::get_filter(Notice::ALARM_LOG, "alarm-mail");
> ? ? ? ?f$interv = 1day;
> ? ? ? ?Log::add_filter(Notice::ALARM_LOG, f);
> ? ? ? ?}
>
>> 1. When logs are rotated (per default once a day),
>
> Ah, that's outdated, the default log rotation used to be once a day,
> but is now once an hour.
>
>> "LogRotationInterval (int, default 3600)
>
> We should add a second option here that defines the rotation interval
> for the alarm summaries separately.
>
> Robin
>
> --
> Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
> ICSI/LBNL ? ?* Fax ? +1 (510) 666-2956 * ? www.icir.org



From seth at icir.org  Wed May 30 12:16:29 2012
From: seth at icir.org (Seth Hall)
Date: Wed, 30 May 2012 15:16:29 -0400
Subject: [Bro] broctl Email Reports
In-Reply-To: <CAG1VgOZWxDzr4J5TN91Sikk8cgWUOBKPEoHeMuf2fS82LoL0Hw@mail.gmail.com>
References: <CAG1VgObyz84xBbaUOn5vuPvvy6huV_RA19Aw51nQFp757Cobgw@mail.gmail.com>
	<20120530182755.GA97952@icir.org>
	<CAG1VgOZWxDzr4J5TN91Sikk8cgWUOBKPEoHeMuf2fS82LoL0Hw@mail.gmail.com>
Message-ID: <E880D16C-598D-4244-8725-3B7E03B1CB91@icir.org>


On May 30, 2012, at 3:04 PM, Chris Crawford wrote:

> I tried adding what you recommended to my local.bro and then did a
> 
> broctl install
> broctl update
> 
> but the connection summary email is still going out on the hour.


Try restarting.  bro_init handlers aren't invoked when updating.  I'm not even sure that the logging framework is modifiable at runtime (we had a discussion about this a long time ago).

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From robin at icir.org  Wed May 30 17:50:39 2012
From: robin at icir.org (Robin Sommer)
Date: Wed, 30 May 2012 17:50:39 -0700
Subject: [Bro] Debugging Policies and Print Statement
In-Reply-To: <8998D311-561E-46C9-AC96-10D2D1721DA8@illinois.edu>
References: <CAJUMos5PtzQrLG+MSfgY3ivxhjpD7RBfb_GYOXR4+5k7pLGqGA@mail.gmail.com>
	<8998D311-561E-46C9-AC96-10D2D1721DA8@illinois.edu>
Message-ID: <20120531005039.GM10373@icir.org>


On Tue, May 29, 2012 at 17:48 +0000, Siwek, Jonathan Luke wrote:

> I also noticed that the context string for stack traces is using a
> static buffer size, so it's getting truncated sometimes and the
> filename:line won't be there.  I can fix that and also give some
> better error messages for when the 'print' command fails to parse the
> given args and put that on the fastpath branch,

I've merged this into git master now.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From christopher.p.crawford at gmail.com  Thu May 31 06:22:53 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Thu, 31 May 2012 09:22:53 -0400
Subject: [Bro] broctl Email Reports
In-Reply-To: <E880D16C-598D-4244-8725-3B7E03B1CB91@icir.org>
References: <CAG1VgObyz84xBbaUOn5vuPvvy6huV_RA19Aw51nQFp757Cobgw@mail.gmail.com>
	<20120530182755.GA97952@icir.org>
	<CAG1VgOZWxDzr4J5TN91Sikk8cgWUOBKPEoHeMuf2fS82LoL0Hw@mail.gmail.com>
	<E880D16C-598D-4244-8725-3B7E03B1CB91@icir.org>
Message-ID: <CAG1VgOYGNiAFJ-tAQVyDOJVJm+uO7VoLYr4+AmJiXf2OCRKUBA@mail.gmail.com>

Hmm...I restarted bro and it's still sending Connection Summary
reports every hour.

On Wed, May 30, 2012 at 3:16 PM, Seth Hall <seth at icir.org> wrote:
>
> On May 30, 2012, at 3:04 PM, Chris Crawford wrote:
>
>> I tried adding what you recommended to my local.bro and then did a
>>
>> broctl install
>> broctl update
>>
>> but the connection summary email is still going out on the hour.
>
>
> Try restarting. ?bro_init handlers aren't invoked when updating. ?I'm not even sure that the logging framework is modifiable at runtime (we had a discussion about this a long time ago).
>
> ?.Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>



From robin at icir.org  Thu May 31 07:44:14 2012
From: robin at icir.org (Robin Sommer)
Date: Thu, 31 May 2012 07:44:14 -0700
Subject: [Bro] broctl Email Reports
In-Reply-To: <CAG1VgOYGNiAFJ-tAQVyDOJVJm+uO7VoLYr4+AmJiXf2OCRKUBA@mail.gmail.com>
References: <CAG1VgObyz84xBbaUOn5vuPvvy6huV_RA19Aw51nQFp757Cobgw@mail.gmail.com>
	<20120530182755.GA97952@icir.org>
	<CAG1VgOZWxDzr4J5TN91Sikk8cgWUOBKPEoHeMuf2fS82LoL0Hw@mail.gmail.com>
	<E880D16C-598D-4244-8725-3B7E03B1CB91@icir.org>
	<CAG1VgOYGNiAFJ-tAQVyDOJVJm+uO7VoLYr4+AmJiXf2OCRKUBA@mail.gmail.com>
Message-ID: <20120531144414.GE48884@icir.org>


On Thu, May 31, 2012 at 09:22 -0400, Chris Crawford wrote:

> Hmm...I restarted bro and it's still sending Connection Summary
> reports every hour.

Ah, ok, I thought your question was only about the alarm summaries
(they should now come once a day). The connection summaries can't
really be detached from the rotation because that's a post-processor
working on the conn.log file at the time it's closed and archived. If
you want them daily (but keep rotating conn.log hourly), you'd need to
do that externally, like with a cron job running over the archived
conn.logs.

(The tool that generates the summaries is "trace-summary", it can be
used standalone as well).

Robin
-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org