From jlay at slave-tothe-box.net  Sun Aug  3 04:44:14 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Sun, 03 Aug 2014 05:44:14 -0600
Subject: [Bro] Brocontrol revisited
Message-ID: <1407066254.2671.7.camel@JamesiMac>

Morning all (least where I'm at),

So...when I very first started with bro, I noticed that running
brocontrol vs. just bro I noticed a large difference in cpu usage.  This
morning I decided to revisit brocontrol.  When running bro commandline I
see:

top - 05:31:35 up 5 min,  2 users,  load average: 0.30, 0.70, 0.38
Tasks:  95 total,   2 running,  93 sleeping,   0 stopped,   0 zombie
%Cpu(s): 11.3 us, 23.8 sy,  0.0 ni, 64.5 id,  0.0 wa,  0.4 hi,  0.0 si,
0.0 st
KiB Mem:   3082108 total,  2562596 used,   519512 free,    29828 buffers
KiB Swap:  3002364 total,        0 used,  3002364 free.  2212828 cached
Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+
COMMAND 
1735 root      20   0  761264  62972   8696 R 27.5  2.0   0:16.45 bro 

However when running broctrl running standalone on the same machine I
see:

top - 05:34:44 up 8 min,  2 users,  load average: 0.68, 0.60, 0.40
Tasks: 100 total,   3 running,  97 sleeping,   0 stopped,   0 zombie
%Cpu(s): 10.3 us, 33.7 sy,  3.2 ni, 52.1 id,  0.7 wa,  0.0 hi,  0.0 si,
0.0 st
KiB Mem:   3082108 total,  2578976 used,   503132 free,    30604 buffers
KiB Swap:  3002364 total,        0 used,  3002364 free.  2216848 cached
Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+
COMMAND 
 2356 root      20   0  529220  56588   8624 R 24.8  1.8   0:04.44 bro 
 2358 root      25   5  168756  50788   2896 R 15.6  1.6   0:02.24 bro 

I like brocontrol's ease of use and auto-reports, but not at the cost of
an additional bro process that eats %15 CPU usage.  Any explanation for
this?  Thank you.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140803/557f177f/attachment.html 

From jsiwek at illinois.edu  Mon Aug  4 07:51:31 2014
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Mon, 4 Aug 2014 14:51:31 +0000
Subject: [Bro] Brocontrol revisited
In-Reply-To: <1407066254.2671.7.camel@JamesiMac>
References: <1407066254.2671.7.camel@JamesiMac>
Message-ID: <294643E9-5324-4C81-A28E-77C42B7F5F13@illinois.edu>


On Aug 3, 2014, at 6:44 AM, James Lay <jlay at slave-tothe-box.net> wrote:

> I like brocontrol's ease of use and auto-reports, but not at the cost of an additional bro process that eats %15 CPU usage.  Any explanation for this?  Thank you.

Even in standalone mode, BroControl currently will have Bro listen for remote connections as some functionality of BroControl depends on that.  Bro will fork a process to do the listening which is the additional bro process.  The communication between parent, child, and peers use somewhat suboptimal I/O loops that rely on small timeouts which can be the reason for the extra CPU usage.  From what I understand, the reason for it being that way is historical (i.e. there were reasons for doing it that way on older systems).  I don?t know of any way to workaround it at this time, but improving/fixing the underlying problem is on the roadmap.

- Jon


From jlay at slave-tothe-box.net  Mon Aug  4 08:10:10 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Mon, 04 Aug 2014 09:10:10 -0600
Subject: [Bro] Brocontrol revisited
In-Reply-To: <294643E9-5324-4C81-A28E-77C42B7F5F13@illinois.edu>
References: <1407066254.2671.7.camel@JamesiMac>
	<294643E9-5324-4C81-A28E-77C42B7F5F13@illinois.edu>
Message-ID: <0aa11c3ae174d6975417a994a6a77b9a@localhost>

On 2014-08-04 08:51, Siwek, Jon wrote:
> On Aug 3, 2014, at 6:44 AM, James Lay <jlay at slave-tothe-box.net> 
> wrote:
>
>> I like brocontrol's ease of use and auto-reports, but not at the 
>> cost of an additional bro process that eats %15 CPU usage.  Any 
>> explanation for this?  Thank you.
>
> Even in standalone mode, BroControl currently will have Bro listen
> for remote connections as some functionality of BroControl depends on
> that.  Bro will fork a process to do the listening which is the
> additional bro process.  The communication between parent, child, and
> peers use somewhat suboptimal I/O loops that rely on small timeouts
> which can be the reason for the extra CPU usage.  From what I
> understand, the reason for it being that way is historical (i.e. 
> there
> were reasons for doing it that way on older systems).  I don?t know 
> of
> any way to workaround it at this time, but improving/fixing the
> underlying problem is on the roadmap.
>
> - Jon

Thanks a bunch Jon...that's a great response that really helps my 
understanding.

James


From pachinko.tw at gmail.com  Mon Aug  4 08:36:07 2014
From: pachinko.tw at gmail.com (Po-Ching Lin)
Date: Mon, 04 Aug 2014 23:36:07 +0800
Subject: [Bro] scripts output with a pager on broctl
Message-ID: <53DFA867.2040900@gmail.com>


How about displaying scripts output on broctl with a pager? The output
spans over several pages long, and it would be easier to check (and also
search) the output with a pager. Just a suggestion :-)

Po-Ching



From coen_bakkers at symantec.com  Mon Aug  4 13:45:46 2014
From: coen_bakkers at symantec.com (Coen Bakkers)
Date: Mon, 4 Aug 2014 13:45:46 -0700
Subject: [Bro] reassembling DNS traffic
Message-ID: <977D60F74C0DA94DBB5807CAF9EC2BB3012464CF5E@EDO1XCHEVSPIN43.SYMC.SYMANTEC.COM>

Is Bro capable of reassembling DNS traffic that is being captured over a TX and a RX tap interface if they are bridged?

Regards,

Coen

-----Original Message-----
From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of bro-request at bro.org
Sent: Montag, 4. August 2014 21:00
To: bro at bro.org
Subject: Bro Digest, Vol 100, Issue 2

Send Bro mailing list submissions to
	bro at bro.org

To subscribe or unsubscribe via the World Wide Web, visit
	http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
or, via email, send a message with subject or body 'help' to
	bro-request at bro.org

You can reach the person managing the list at
	bro-owner at bro.org

When replying, please edit your Subject line so it is more specific than "Re: Contents of Bro digest..."


Today's Topics:

   1. Re: Brocontrol revisited (Siwek, Jon)
   2. Re: Brocontrol revisited (James Lay)
   3. scripts output with a pager on broctl (Po-Ching Lin)


----------------------------------------------------------------------

Message: 1
Date: Mon, 4 Aug 2014 14:51:31 +0000
From: "Siwek, Jon" <jsiwek at illinois.edu>
Subject: Re: [Bro] Brocontrol revisited
To: James Lay <jlay at slave-tothe-box.net>
Cc: Bro-IDS <bro at bro.org>
Message-ID: <294643E9-5324-4C81-A28E-77C42B7F5F13 at illinois.edu>
Content-Type: text/plain; charset="Windows-1252"


On Aug 3, 2014, at 6:44 AM, James Lay <jlay at slave-tothe-box.net> wrote:

> I like brocontrol's ease of use and auto-reports, but not at the cost of an additional bro process that eats %15 CPU usage.  Any explanation for this?  Thank you.

Even in standalone mode, BroControl currently will have Bro listen for remote connections as some functionality of BroControl depends on that.  Bro will fork a process to do the listening which is the additional bro process.  The communication between parent, child, and peers use somewhat suboptimal I/O loops that rely on small timeouts which can be the reason for the extra CPU usage.  From what I understand, the reason for it being that way is historical (i.e. there were reasons for doing it that way on older systems).  I don?t know of any way to workaround it at this time, but improving/fixing the underlying problem is on the roadmap.

- Jon


------------------------------

Message: 2
Date: Mon, 04 Aug 2014 09:10:10 -0600
From: James Lay <jlay at slave-tothe-box.net>
Subject: Re: [Bro] Brocontrol revisited
To: Bro-IDS <bro at bro.org>
Message-ID: <0aa11c3ae174d6975417a994a6a77b9a at localhost>
Content-Type: text/plain; charset=UTF-8; format=flowed

On 2014-08-04 08:51, Siwek, Jon wrote:
> On Aug 3, 2014, at 6:44 AM, James Lay <jlay at slave-tothe-box.net>
> wrote:
>
>> I like brocontrol's ease of use and auto-reports, but not at the cost 
>> of an additional bro process that eats %15 CPU usage.  Any 
>> explanation for this?  Thank you.
>
> Even in standalone mode, BroControl currently will have Bro listen for 
> remote connections as some functionality of BroControl depends on 
> that.  Bro will fork a process to do the listening which is the 
> additional bro process.  The communication between parent, child, and 
> peers use somewhat suboptimal I/O loops that rely on small timeouts 
> which can be the reason for the extra CPU usage.  From what I 
> understand, the reason for it being that way is historical (i.e.
> there
> were reasons for doing it that way on older systems).  I don?t know of 
> any way to workaround it at this time, but improving/fixing the 
> underlying problem is on the roadmap.
>
> - Jon

Thanks a bunch Jon...that's a great response that really helps my understanding.

James


------------------------------

Message: 3
Date: Mon, 04 Aug 2014 23:36:07 +0800
From: Po-Ching Lin <pachinko.tw at gmail.com>
Subject: [Bro] scripts output with a pager on broctl
To: bro at bro.org
Message-ID: <53DFA867.2040900 at gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed


How about displaying scripts output on broctl with a pager? The output spans over several pages long, and it would be easier to check (and also
search) the output with a pager. Just a suggestion :-)

Po-Ching



------------------------------

_______________________________________________
Bro mailing list
Bro at bro.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


End of Bro Digest, Vol 100, Issue 2
***********************************



From nithen at gmail.com  Tue Aug  5 06:28:31 2014
From: nithen at gmail.com (nithen)
Date: Tue, 5 Aug 2014 15:28:31 +0200
Subject: [Bro] Question on quick start documentation SSH:Login example.
Message-ID: <CACNThYdqJRpKyp_YF3MgqzGf0PgVD6CUmT-tXfBAjRKGo0i0wg@mail.gmail.com>

Hi,

Sorry for directing such a simple question to the mailing list - but
I'm really stuck and would appreciate your help.

I am running 2 separate instances of Bro (on separate hardware):
1. Bro 2.2 on FreeBSD 10
2. Bro 2.3 on FreeBSD 10

I am following the Quick Start documentation found here:
http://www.bro.org/sphinx/quickstart/index.html

I can't get the deployment customization example on "SSH:Login" to work.

I have performed the following:
1. Checked my installation is working.
2. Checked my email (mailto) is working.
3. Checked my networks.cfg includes my test SSH server and excludes my client.
4. Checked for previous posts on the issue.

Here is the code that is to be written into local.bro (only change was
the IP Addresses):
<snip>
const watched_servers: set[addr] = {
     192.168.1.100,
     192.168.1.101,
     192.168.1.102,
 } &redef;

hook Notice::policy(n: Notice::Info)
    {
    if ( n$note == SSH::SUCCESSFUL_LOGIN && n$id$resp_h in watched_servers )
         add n$actions[Notice::ACTION_EMAIL];
    }
</snip>

Thank you,
Nithen


From JAzoff at albany.edu  Tue Aug  5 07:31:22 2014
From: JAzoff at albany.edu (Justin Azoff)
Date: Tue, 5 Aug 2014 10:31:22 -0400
Subject: [Bro] Question on quick start documentation SSH:Login example.
In-Reply-To: <CACNThYdqJRpKyp_YF3MgqzGf0PgVD6CUmT-tXfBAjRKGo0i0wg@mail.gmail.com>
References: <CACNThYdqJRpKyp_YF3MgqzGf0PgVD6CUmT-tXfBAjRKGo0i0wg@mail.gmail.com>
Message-ID: <20140805143122.GB23785@datacomm.albany.edu>

On Tue, Aug 05, 2014 at 03:28:31PM +0200, nithen wrote:
> hook Notice::policy(n: Notice::Info)
>     {
>     if ( n$note == SSH::SUCCESSFUL_LOGIN && n$id$resp_h in watched_servers )
>          add n$actions[Notice::ACTION_EMAIL];
>     }

Looks like the quickstart is a bit off here..

That used to use SSH::Login but was changed to SSH::SUCCESSFUL_LOGIN,
but SSH::SUCCESSFUL_LOGIN is a Intel::Where and not a Notice::Type.

There's no ssh login notice anymore, so I don't think there is an easy
fix here to accomplish the same thing.

-- 
-- Justin Azoff


From jsiwek at illinois.edu  Tue Aug  5 07:59:22 2014
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Tue, 5 Aug 2014 14:59:22 +0000
Subject: [Bro] Question on quick start documentation SSH:Login example.
In-Reply-To: <CACNThYdqJRpKyp_YF3MgqzGf0PgVD6CUmT-tXfBAjRKGo0i0wg@mail.gmail.com>
References: <CACNThYdqJRpKyp_YF3MgqzGf0PgVD6CUmT-tXfBAjRKGo0i0wg@mail.gmail.com>
Message-ID: <8DF3C970-A563-4DBC-B2F4-358E112D2F86@illinois.edu>


On Aug 5, 2014, at 8:28 AM, nithen <nithen at gmail.com> wrote:

> I am following the Quick Start documentation found here:
> http://www.bro.org/sphinx/quickstart/index.html
> 
> I can't get the deployment customization example on "SSH:Login" to work.

That documentation is not correct anymore, sorry about that.  Will see about getting it fixed, but I put an example at [1] that should work to accomplish the same thing.  The ?SSH:: heuristic_successful_login? event is somewhat delayed, so just be aware of that if you?re looking for immediate feedback to check whether it?s working.  And another gotcha is that the event only triggers after a certain amount of data is transmitted so just logging in/out real quick may not be detected.  (I?m realizing this example is no longer that straightforward and probably doesn?t belong in the quick-start guide anymore).

- Jon

[1] https://gist.github.com/jsiwek/2a7692aa9f24e197ca9c


From lists at g-clef.net  Tue Aug  5 09:57:18 2014
From: lists at g-clef.net (Aaron Gee-Clough)
Date: Tue, 05 Aug 2014 12:57:18 -0400
Subject: [Bro] reassembling DNS traffic
References: <977D60F74C0DA94DBB5807CAF9EC2BB3012464CF5E@EDO1XCHEVSPIN43.SYMC.SYMANTEC.COM>
Message-ID: <53E10CEE.3050408@g-clef.net>


If the TX and RX interfaces are bonded together, and bro pointed to that 
bonded interface rather than the individual ones, then it works fine. 
Bro never knows the difference, as the TX and RX are presented to it on 
the bonded interface.  (I'm doing this right now...works well.)

aaron

On 08/04/2014 04:45 PM, Coen Bakkers wrote:
>
> Is Bro capable of reassembling DNS traffic that is being captured
> over a TX and a RX tap interface if they are bridged?
>
> Regards,
>
> Coen
>
> -----Original Message----- From: bro-bounces at bro.org
> [mailto:bro-bounces at bro.org] On Behalf Of bro-request at bro.org Sent:
> Montag, 4. August 2014 21:00 To: bro at bro.org Subject: Bro Digest, Vol
> 100, Issue 2
>
> Send Bro mailing list submissions to bro at bro.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro or, via email,
> send a message with subject or body 'help' to bro-request at bro.org
>
> You can reach the person managing the list at bro-owner at bro.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Bro digest..."
>
>
> Today's Topics:
>
> 1. Re: Brocontrol revisited (Siwek, Jon) 2. Re: Brocontrol revisited
> (James Lay) 3. scripts output with a pager on broctl (Po-Ching Lin)
>
>
> ----------------------------------------------------------------------
>
>  Message: 1 Date: Mon, 4 Aug 2014 14:51:31 +0000 From: "Siwek, Jon"
> <jsiwek at illinois.edu> Subject: Re: [Bro] Brocontrol revisited To:
> James Lay <jlay at slave-tothe-box.net> Cc: Bro-IDS <bro at bro.org>
> Message-ID: <294643E9-5324-4C81-A28E-77C42B7F5F13 at illinois.edu>
> Content-Type: text/plain; charset="Windows-1252"
>
>
> On Aug 3, 2014, at 6:44 AM, James Lay <jlay at slave-tothe-box.net>
> wrote:
>
>> I like brocontrol's ease of use and auto-reports, but not at the
>> cost of an additional bro process that eats %15 CPU usage.  Any
>> explanation for this?  Thank you.
>
> Even in standalone mode, BroControl currently will have Bro listen
> for remote connections as some functionality of BroControl depends on
> that.  Bro will fork a process to do the listening which is the
> additional bro process.  The communication between parent, child, and
> peers use somewhat suboptimal I/O loops that rely on small timeouts
> which can be the reason for the extra CPU usage.  From what I
> understand, the reason for it being that way is historical (i.e.
> there were reasons for doing it that way on older systems).  I don?t
> know of any way to workaround it at this time, but improving/fixing
> the underlying problem is on the roadmap.
>
> - Jon
>
>
> ------------------------------
>
> Message: 2 Date: Mon, 04 Aug 2014 09:10:10 -0600 From: James Lay
> <jlay at slave-tothe-box.net> Subject: Re: [Bro] Brocontrol revisited
> To: Bro-IDS <bro at bro.org> Message-ID:
> <0aa11c3ae174d6975417a994a6a77b9a at localhost> Content-Type:
> text/plain; charset=UTF-8; format=flowed
>
> On 2014-08-04 08:51, Siwek, Jon wrote:
>> On Aug 3, 2014, at 6:44 AM, James Lay <jlay at slave-tothe-box.net>
>> wrote:
>>
>>> I like brocontrol's ease of use and auto-reports, but not at the
>>> cost of an additional bro process that eats %15 CPU usage.  Any
>>> explanation for this?  Thank you.
>>
>> Even in standalone mode, BroControl currently will have Bro listen
>> for remote connections as some functionality of BroControl depends
>> on that.  Bro will fork a process to do the listening which is the
>> additional bro process.  The communication between parent, child,
>> and peers use somewhat suboptimal I/O loops that rely on small
>> timeouts which can be the reason for the extra CPU usage.  From
>> what I understand, the reason for it being that way is historical
>> (i.e. there were reasons for doing it that way on older systems).
>> I don?t know of any way to workaround it at this time, but
>> improving/fixing the underlying problem is on the roadmap.
>>
>> - Jon
>
> Thanks a bunch Jon...that's a great response that really helps my
> understanding.
>
> James
>
>
> ------------------------------
>
> Message: 3 Date: Mon, 04 Aug 2014 23:36:07 +0800 From: Po-Ching Lin
> <pachinko.tw at gmail.com> Subject: [Bro] scripts output with a pager on
> broctl To: bro at bro.org Message-ID: <53DFA867.2040900 at gmail.com>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
>
> How about displaying scripts output on broctl with a pager? The
> output spans over several pages long, and it would be easier to check
> (and also search) the output with a pager. Just a suggestion :-)
>
> Po-Ching
>
>
>
> ------------------------------
>
> _______________________________________________ Bro mailing list
> Bro at bro.org http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> End of Bro Digest, Vol 100, Issue 2
> ***********************************
>
> _______________________________________________ Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>


From GehanaBooth at cmail.carleton.ca  Wed Aug  6 06:12:48 2014
From: GehanaBooth at cmail.carleton.ca (Gehana Booth)
Date: Wed, 6 Aug 2014 09:12:48 -0400
Subject: [Bro] Packet Level Analysis
Message-ID: <CACkaxDT9VyqVuxDy-HVQtbqLp6-0oO68NRKxn5wfKSDqCeEdiA@mail.gmail.com>

Hello,

This is probably a very silly question, but I just wanted to get some
opinions. Is it possible/feasible to do packet level analysis with bro
(e.g., looking at the entire packet as a string to find similar patterns
between packets)? Or is bro too high-level to make this an option, as it
seems that the relevant events (new_packet, packet_contents, etc.) are
exceedingly slow.

If this is possible, however, would I be able to do this in bro scripts or
would I need to do something like write the module in C/C++ to hook into
bro?

Cheers,
Gehana
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140806/2f89a6bc/attachment.html 

From davidvasil at gmail.com  Wed Aug  6 06:35:13 2014
From: davidvasil at gmail.com (David Vasil)
Date: Wed, 6 Aug 2014 08:35:13 -0500
Subject: [Bro] Filtering out hosts from notices
Message-ID: <CAFBiiTydYzMGiMoFANi=bJO2S6HWV=L3uHf4=YLP5yPuhnoFOQ@mail.gmail.com>

I'm sure this has been documented somewhere, but I have been unable to find
it thus far.  How do you define suppression criteria for individual
notices?  For example, SSH::Interesting_Hostname_Login is triggering for me
quite a bit on an ftp server that also provides sftp access; I'd like to
suppress all notices to this system (e.g. ip: 192.168.0.100, hostname:
ftp.mydomain.org).  Would this be a redef of 'interesting_hostnames' to
something like:

(/^d?ns[0-9]*\./ |
/^smtp[0-9]*\./ |
/^mail[0-9]*\./ |
/^pop[0-9]*\./  |
/^imap[0-9]*\./ |
/^www[0-9]*\./  |
/^ftp[0-9]*\./) & !(ftp.mydomain.org) &redef;

Thanks!
-David Vasil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140806/a2c3ed69/attachment.html 

From hosom at battelle.org  Wed Aug  6 07:49:33 2014
From: hosom at battelle.org (Hosom, Stephen M)
Date: Wed, 6 Aug 2014 14:49:33 +0000
Subject: [Bro] Packet Level Analysis
In-Reply-To: <CACkaxDT9VyqVuxDy-HVQtbqLp6-0oO68NRKxn5wfKSDqCeEdiA@mail.gmail.com>
References: <CACkaxDT9VyqVuxDy-HVQtbqLp6-0oO68NRKxn5wfKSDqCeEdiA@mail.gmail.com>
Message-ID: <E5A2071C6FFFF640B714C224139C9B33109C4E@WP-MBX2.milky-way.battelle.org>

The sorts of places where I see this being useful are well served by the Signatures framework.

The traceroute detector in policy/misc is a pretty good example of this ?sort? of thing.

From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Gehana Booth
Sent: Wednesday, August 06, 2014 9:13 AM
To: bro at bro.org
Subject: [Bro] Packet Level Analysis

Hello,

This is probably a very silly question, but I just wanted to get some opinions. Is it possible/feasible to do packet level analysis with bro (e.g., looking at the entire packet as a string to find similar patterns between packets)? Or is bro too high-level to make this an option, as it seems that the relevant events (new_packet, packet_contents, etc.) are exceedingly slow.

If this is possible, however, would I be able to do this in bro scripts or would I need to do something like write the module in C/C++ to hook into bro?

Cheers,
Gehana
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140806/37b1707f/attachment.html 

From seth at icir.org  Wed Aug  6 09:20:05 2014
From: seth at icir.org (Seth Hall)
Date: Wed, 6 Aug 2014 12:20:05 -0400
Subject: [Bro] Packet Level Analysis
In-Reply-To: <CACkaxDT9VyqVuxDy-HVQtbqLp6-0oO68NRKxn5wfKSDqCeEdiA@mail.gmail.com>
References: <CACkaxDT9VyqVuxDy-HVQtbqLp6-0oO68NRKxn5wfKSDqCeEdiA@mail.gmail.com>
Message-ID: <96A744F5-BD2A-4611-8252-B992B688633D@icir.org>


On Aug 6, 2014, at 9:12 AM, Gehana Booth <GehanaBooth at cmail.carleton.ca> wrote:

> Or is bro too high-level to make this an option, as it seems that the relevant events (new_packet, packet_contents, etc.) are exceedingly slow.

Bro is unfortunately too high level to this right now.  There are a few things being worked on that might provide better interfaces for doing this analysis but they aren't functional yet. (bro script compiler and binpac++)

> would I need to do something like write the module in C/C++ to hook into bro?

You could certainly write something like that.  Our analyzers are abstracted in our repository so it should be fairly easy to see how they're constructed and to write your own, assuming you're comfortable with c/c++.  We definitely recognize that falling back to c/c++ is suboptimal though, but at the moment it's all we have to solve your problem well.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140806/f7e6d8c8/attachment.bin 

From jonathon.s.wright at gmail.com  Wed Aug  6 12:53:16 2014
From: jonathon.s.wright at gmail.com (Jonathon Wright)
Date: Wed, 6 Aug 2014 09:53:16 -1000
Subject: [Bro] Bro 2.2 File Extraction (RHEL 6.5)
Message-ID: <CAGX1DMYo2BRwbwayFygV9CkfrWO6DgoYzHNebySmY-Ee1hqEhg@mail.gmail.com>

Hey Bro List,



 I'm trying to setup the File Extraction using Bro 2.2 on a RHEL 6.5

 system and its not functioning properly (no files are being extracted

 from the pcap).



 Here is what I've tried:



 I put whatever.bro into the directory:

 /opt/bro/share/bro/site

 I edited "local.bro" and told it to "load whatever.bro"

 I verified all configuration syntax: broctl check

 I addressed any errors (none)

 I install the script: broctl install

 Then bounced bro: broctl restart


 To test the bro file extraction capabilities, my "whatever.bro" scrip

 contains the following:

 -----------START

 #This produces logs only, no extracted files

     event file_new(f: fa_file)

     {

         Files::add_analyzer(f, Files::ANALYZER_EXTRACT);

     }

 -----------END



 My (produced from tcpdump) pcap contains a five minute section of

 traffic where I downloaded a few hp printer drivers to test. Wireshark

 was able to extract the files, so we know the pcap file integrity is good.

 I ran this on command line to have Bro extract the hp printer driver

 files from same pcap file:



 bro -C -r my_pcap_file



 Logs are produced in the pwd, but no extracted files.





 Any ideas?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140806/679e5c02/attachment.html 

From seth at icir.org  Wed Aug  6 13:07:30 2014
From: seth at icir.org (Seth Hall)
Date: Wed, 6 Aug 2014 16:07:30 -0400
Subject: [Bro] Bro 2.2 File Extraction (RHEL 6.5)
In-Reply-To: <CAGX1DMYo2BRwbwayFygV9CkfrWO6DgoYzHNebySmY-Ee1hqEhg@mail.gmail.com>
References: <CAGX1DMYo2BRwbwayFygV9CkfrWO6DgoYzHNebySmY-Ee1hqEhg@mail.gmail.com>
Message-ID: <14E53734-31D8-4A9E-8C8E-60B9E08441C2@icir.org>


On Aug 6, 2014, at 3:53 PM, Jonathon Wright <jonathon.s.wright at gmail.com> wrote:

> I verified all configuration syntax: broctl check
> 
>  bro -C -r my_pcap_file

Two separate things are going on here.  Broctl is really focused around running Bro on live traffic and orchestrating all of the complexity involved in that.  You are then separately trying to run the Bro binary on a trace file and get output.

Your whatever.bro script is installed and ready to be used when Bro is run with broctl.  Since you're just running Bro directly here though, you will want to load your script on the command line like this:

	bro -C -r my_pcap_file whatever.bro

You could also load the full local.bro script if you want that functionality too like this:

	bro -C -r my_pcap_file local.bro whatever.bro 

Does that explain things better?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140806/f74905d3/attachment.bin 

From jonathon.s.wright at gmail.com  Wed Aug  6 13:17:15 2014
From: jonathon.s.wright at gmail.com (Jonathon Wright)
Date: Wed, 6 Aug 2014 10:17:15 -1000
Subject: [Bro] Bro 2.2 File Extraction (RHEL 6.5)
In-Reply-To: <14E53734-31D8-4A9E-8C8E-60B9E08441C2@icir.org>
References: <CAGX1DMYo2BRwbwayFygV9CkfrWO6DgoYzHNebySmY-Ee1hqEhg@mail.gmail.com>
	<14E53734-31D8-4A9E-8C8E-60B9E08441C2@icir.org>
Message-ID: <CAGX1DMZNzbPTB3iWFgYRLjrgsUJ2Nkf0g0rkMO9aEDKCCbRr2g@mail.gmail.com>

Yes it does!

What I'm trying to do is "Verify that broctl is configured for File
Extraction properly". My method was to test broctl by using bro on the CLI.
Your explanation is good information.

I'm going to try that now and update the list on results.


On Wed, Aug 6, 2014 at 10:07 AM, Seth Hall <seth at icir.org> wrote:

>
> On Aug 6, 2014, at 3:53 PM, Jonathon Wright <jonathon.s.wright at gmail.com>
> wrote:
>
> > I verified all configuration syntax: broctl check
> >
> >  bro -C -r my_pcap_file
>
> Two separate things are going on here.  Broctl is really focused around
> running Bro on live traffic and orchestrating all of the complexity
> involved in that.  You are then separately trying to run the Bro binary on
> a trace file and get output.
>
> Your whatever.bro script is installed and ready to be used when Bro is run
> with broctl.  Since you're just running Bro directly here though, you will
> want to load your script on the command line like this:
>
>         bro -C -r my_pcap_file whatever.bro
>
> You could also load the full local.bro script if you want that
> functionality too like this:
>
>         bro -C -r my_pcap_file local.bro whatever.bro
>
> Does that explain things better?
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140806/ea2096e0/attachment.html 

From nithen at gmail.com  Wed Aug  6 13:33:54 2014
From: nithen at gmail.com (nithen)
Date: Wed, 6 Aug 2014 22:33:54 +0200
Subject: [Bro] Question on quick start documentation SSH:Login example.
In-Reply-To: <8DF3C970-A563-4DBC-B2F4-358E112D2F86@illinois.edu>
References: <CACNThYdqJRpKyp_YF3MgqzGf0PgVD6CUmT-tXfBAjRKGo0i0wg@mail.gmail.com>
	<8DF3C970-A563-4DBC-B2F4-358E112D2F86@illinois.edu>
Message-ID: <CACNThYdOmX37sO0Zy6sffOqL6LbG9kPHGh68W_j-rzY7rbJieg@mail.gmail.com>

Thank you Jon and Justin. I really appreciate your help!

Jon, I could not get your script working - so I took a step back to
check my installation. I wanted to confirm that my default scripts
work.

I setup the following lab:

Kali Linux -> Bro SPAN -> Metasploitable

Using: FreeBSD + Bro 2.3 (compiled from source)

Test: trigger /usr/local/bro/share/bro/policy/protocols/ssh/detect-bruteforcing.bro

Verified: loaded_scripts.log (script is loaded), ssh.log (ssh login
attempts there).

So here is an extract of the ssh.log:
<snip>
1407355776.833081	CNjybf25kbwTIpD9D6	192.168.88.2	58904	192.168.88.101	22	undetermined	INBOUND	SSH-2.0-MEDUSA_1.0	-	-	-
1407355784.647680	CGYsSAwShJeTcT2t8	192.168.88.2	58905	192.168.88.101	22	undetermined	INBOUND	SSH-2.0-MEDUSA_1.0	-	-	-
</snip>

I checked the threshold in the Bro script:
<snip>
const password_guesses_limit: double = 30
</snip>

I hit the SSH server over 500 incorrect root logins - however no alerts noted.

Any ideas on where I should start investigating? Do you require more
information?

Thank you,
Nithen


From jonathon.s.wright at gmail.com  Wed Aug  6 13:41:43 2014
From: jonathon.s.wright at gmail.com (Jonathon Wright)
Date: Wed, 6 Aug 2014 10:41:43 -1000
Subject: [Bro] Bro 2.2 File Extraction (RHEL 6.5)
In-Reply-To: <CAGX1DMZNzbPTB3iWFgYRLjrgsUJ2Nkf0g0rkMO9aEDKCCbRr2g@mail.gmail.com>
References: <CAGX1DMYo2BRwbwayFygV9CkfrWO6DgoYzHNebySmY-Ee1hqEhg@mail.gmail.com>
	<14E53734-31D8-4A9E-8C8E-60B9E08441C2@icir.org>
	<CAGX1DMZNzbPTB3iWFgYRLjrgsUJ2Nkf0g0rkMO9aEDKCCbRr2g@mail.gmail.com>
Message-ID: <CAGX1DMZy4NoEw69RsSXdT4qm3c0nsdiPStAGU_TqptTg_01VZQ@mail.gmail.com>

Too easy, that worked! It created the extracted files in the 'pwd'. I
checked the md5 they matched from the wireshark pcap file. I'll run another
test on a tcpdump file and verify the md5 as well.

Three questions then:
1. Can I safely assume, based on these test results, that broctl
will perform the same way as bro?
2. If so, where will broctl place the 'extracted_files' directory?
3. Lastly, whats the best way to investigate these files (I'm capturing all
exe downloads on HTTP)? For example, the directory 'extracted_files' will
be full of HTTP-blahblah names. How would I correlate those file names
to its actual file name? Is that information stored in the conn.log,
files.log, http.log, packet_filter.log,  & weird.log?

Thanks for your time.

JW


On Wed, Aug 6, 2014 at 10:17 AM, Jonathon Wright <
jonathon.s.wright at gmail.com> wrote:

> Yes it does!
>
> What I'm trying to do is "Verify that broctl is configured for File
> Extraction properly". My method was to test broctl by using bro on the CLI.
> Your explanation is good information.
>
> I'm going to try that now and update the list on results.
>
>
> On Wed, Aug 6, 2014 at 10:07 AM, Seth Hall <seth at icir.org> wrote:
>
>>
>> On Aug 6, 2014, at 3:53 PM, Jonathon Wright <jonathon.s.wright at gmail.com>
>> wrote:
>>
>> > I verified all configuration syntax: broctl check
>> >
>> >  bro -C -r my_pcap_file
>>
>> Two separate things are going on here.  Broctl is really focused around
>> running Bro on live traffic and orchestrating all of the complexity
>> involved in that.  You are then separately trying to run the Bro binary on
>> a trace file and get output.
>>
>> Your whatever.bro script is installed and ready to be used when Bro is
>> run with broctl.  Since you're just running Bro directly here though, you
>> will want to load your script on the command line like this:
>>
>>         bro -C -r my_pcap_file whatever.bro
>>
>> You could also load the full local.bro script if you want that
>> functionality too like this:
>>
>>         bro -C -r my_pcap_file local.bro whatever.bro
>>
>> Does that explain things better?
>>
>>   .Seth
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140806/9479bb0e/attachment.html 

From jlay at slave-tothe-box.net  Wed Aug  6 13:45:50 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 06 Aug 2014 14:45:50 -0600
Subject: [Bro] Question on quick start documentation SSH:Login example.
In-Reply-To: <CACNThYdOmX37sO0Zy6sffOqL6LbG9kPHGh68W_j-rzY7rbJieg@mail.gmail.com>
References: <CACNThYdqJRpKyp_YF3MgqzGf0PgVD6CUmT-tXfBAjRKGo0i0wg@mail.gmail.com>
	<8DF3C970-A563-4DBC-B2F4-358E112D2F86@illinois.edu>
	<CACNThYdOmX37sO0Zy6sffOqL6LbG9kPHGh68W_j-rzY7rbJieg@mail.gmail.com>
Message-ID: <08e4076b60b93c7c61ff0832604da34d@localhost>

On 2014-08-06 14:33, nithen wrote:
> Thank you Jon and Justin. I really appreciate your help!
>
> Jon, I could not get your script working - so I took a step back to
> check my installation. I wanted to confirm that my default scripts
> work.
>
> I setup the following lab:
>
> Kali Linux -> Bro SPAN -> Metasploitable
>
> Using: FreeBSD + Bro 2.3 (compiled from source)
>
> Test: trigger
> /usr/local/bro/share/bro/policy/protocols/ssh/detect-bruteforcing.bro
>
> Verified: loaded_scripts.log (script is loaded), ssh.log (ssh login
> attempts there).
>
> So here is an extract of the ssh.log:
> <snip>
> 
> 1407355776.833081	CNjybf25kbwTIpD9D6	192.168.88.2	58904	192.168.88.101	22	undetermined	INBOUND	SSH-2.0-MEDUSA_1.0	-	-	-
> 
> 1407355784.647680	CGYsSAwShJeTcT2t8	192.168.88.2	58905	192.168.88.101	22	undetermined	INBOUND	SSH-2.0-MEDUSA_1.0	-	-	-
> </snip>
>
> I checked the threshold in the Bro script:
> <snip>
> const password_guesses_limit: double = 30
> </snip>
>
> I hit the SSH server over 500 incorrect root logins - however no
> alerts noted.
>
> Any ideas on where I should start investigating? Do you require more
> information?
>
> Thank you,
> Nithen
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

 From the script:

# Generate the notice.
NOTICE([$note=Password_Guessing,
     $msg=fmt("%s appears to be guessing SSH passwords (seen in %d 
connections).", key$host, r$num),
     $sub=sub_msg,
     $src=key$host,
     $identifier=cat(key$host)]);
}]);

Would that be in the ssh.log or the notice.log?

James


From nithen at gmail.com  Wed Aug  6 14:02:33 2014
From: nithen at gmail.com (nithen)
Date: Wed, 6 Aug 2014 23:02:33 +0200
Subject: [Bro] Question on quick start documentation SSH:Login example.
In-Reply-To: <08e4076b60b93c7c61ff0832604da34d@localhost>
References: <CACNThYdqJRpKyp_YF3MgqzGf0PgVD6CUmT-tXfBAjRKGo0i0wg@mail.gmail.com>
	<8DF3C970-A563-4DBC-B2F4-358E112D2F86@illinois.edu>
	<CACNThYdOmX37sO0Zy6sffOqL6LbG9kPHGh68W_j-rzY7rbJieg@mail.gmail.com>
	<08e4076b60b93c7c61ff0832604da34d@localhost>
Message-ID: <CACNThYfmxWaNBFwbnKAG2a7XO6OGq_RxOB6vyYw5CVrn9pLqKg@mail.gmail.com>

On 8/6/14, James Lay wrote:
> Would that be in the ssh.log or the notice.log?

Thanks James, I checked both (actually all logs) and mail -> my novice
assessment was notice.log to answer your question - am I correct?

N


From nithen at gmail.com  Wed Aug  6 14:02:32 2014
From: nithen at gmail.com (nithen)
Date: Wed, 6 Aug 2014 23:02:32 +0200
Subject: [Bro] Question on quick start documentation SSH:Login example.
In-Reply-To: <08e4076b60b93c7c61ff0832604da34d@localhost>
References: <CACNThYdqJRpKyp_YF3MgqzGf0PgVD6CUmT-tXfBAjRKGo0i0wg@mail.gmail.com>
	<8DF3C970-A563-4DBC-B2F4-358E112D2F86@illinois.edu>
	<CACNThYdOmX37sO0Zy6sffOqL6LbG9kPHGh68W_j-rzY7rbJieg@mail.gmail.com>
	<08e4076b60b93c7c61ff0832604da34d@localhost>
Message-ID: <CACNThYfXLORat7e1K5gLq0H+jWSnAtr87dKpqEGkb6Gmh_W5aw@mail.gmail.com>

On 8/6/14, James Lay wrote:
> Would that be in the ssh.log or the notice.log?

Thanks James, I checked both (actually all logs) and mail -> my novice
assessment was notice.log to answer your question - am I correct?

N


From seth at icir.org  Wed Aug  6 14:09:13 2014
From: seth at icir.org (Seth Hall)
Date: Wed, 6 Aug 2014 17:09:13 -0400
Subject: [Bro] Bro 2.2 File Extraction (RHEL 6.5)
In-Reply-To: <CAGX1DMZy4NoEw69RsSXdT4qm3c0nsdiPStAGU_TqptTg_01VZQ@mail.gmail.com>
References: <CAGX1DMYo2BRwbwayFygV9CkfrWO6DgoYzHNebySmY-Ee1hqEhg@mail.gmail.com>
	<14E53734-31D8-4A9E-8C8E-60B9E08441C2@icir.org>
	<CAGX1DMZNzbPTB3iWFgYRLjrgsUJ2Nkf0g0rkMO9aEDKCCbRr2g@mail.gmail.com>
	<CAGX1DMZy4NoEw69RsSXdT4qm3c0nsdiPStAGU_TqptTg_01VZQ@mail.gmail.com>
Message-ID: <A8A79A3D-E050-411D-BDAB-B8CE8E117269@icir.org>


On Aug 6, 2014, at 4:41 PM, Jonathon Wright <jonathon.s.wright at gmail.com> wrote:

> Too easy, that worked! It created the extracted files in the 'pwd'. I checked the md5 they matched from the wireshark pcap file.

Great!

> 1. Can I safely assume, based on these test results, that broctl will perform the same way as bro?

Generally yes.  Broctl is just a control harness for Bro that runs it in a certain way.

> 2. If so, where will broctl place the 'extracted_files' directory?

Unfortunately that will in the <prefix>/spool/{node-name} directory.  You can set it to something system-wide though like this...

redef FilesExtract::prefix = "/extract/here/";

That directory will just need to exist and multiple Bro processes will write extracted files there.

> 3. Lastly, whats the best way to investigate these files (I'm capturing all exe downloads on HTTP)? For example, the directory 'extracted_files' will be full of HTTP-blahblah names. How would I correlate those file names to its actual file name? Is that information stored in the conn.log, files.log, http.log, packet_filter.log,  & weird.log?

Unfortunately again, that's something where you may want to write a script that can take the file names and inspect the logs.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140806/09562a64/attachment.bin 

From jlay at slave-tothe-box.net  Wed Aug  6 14:16:23 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 06 Aug 2014 15:16:23 -0600
Subject: [Bro] Question on quick start documentation SSH:Login example.
In-Reply-To: <CACNThYfXLORat7e1K5gLq0H+jWSnAtr87dKpqEGkb6Gmh_W5aw@mail.gmail.com>
References: <CACNThYdqJRpKyp_YF3MgqzGf0PgVD6CUmT-tXfBAjRKGo0i0wg@mail.gmail.com>
	<8DF3C970-A563-4DBC-B2F4-358E112D2F86@illinois.edu>
	<CACNThYdOmX37sO0Zy6sffOqL6LbG9kPHGh68W_j-rzY7rbJieg@mail.gmail.com>
	<08e4076b60b93c7c61ff0832604da34d@localhost>
	<CACNThYfXLORat7e1K5gLq0H+jWSnAtr87dKpqEGkb6Gmh_W5aw@mail.gmail.com>
Message-ID: <0deb86a2ebe7ef6961a823b28f13622d@localhost>

On 2014-08-06 15:02, nithen wrote:
> On 8/6/14, James Lay wrote:
>> Would that be in the ssh.log or the notice.log?
>
> Thanks James, I checked both (actually all logs) and mail -> my 
> novice
> assessment was notice.log to answer your question - am I correct?
>
> N

Good call on checking.  Per the html:

http://www.bro.org/sphinx/scripts/policy/protocols/ssh/detect-bruteforcing.bro.html

It looks like unless it's redefined, these should show up in 
notice.log...but I'm a noob, so someone smarter then me on this list 
should be able to verify that.

James


From jsiwek at illinois.edu  Wed Aug  6 14:35:15 2014
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Wed, 6 Aug 2014 21:35:15 +0000
Subject: [Bro] Question on quick start documentation SSH:Login example.
In-Reply-To: <CACNThYdOmX37sO0Zy6sffOqL6LbG9kPHGh68W_j-rzY7rbJieg@mail.gmail.com>
References: <CACNThYdqJRpKyp_YF3MgqzGf0PgVD6CUmT-tXfBAjRKGo0i0wg@mail.gmail.com>
	<8DF3C970-A563-4DBC-B2F4-358E112D2F86@illinois.edu>
	<CACNThYdOmX37sO0Zy6sffOqL6LbG9kPHGh68W_j-rzY7rbJieg@mail.gmail.com>
Message-ID: <74E82FA3-BD81-40D9-96FA-388BF5E535D3@illinois.edu>


On Aug 6, 2014, at 3:33 PM, nithen <nithen at gmail.com> wrote:

> So here is an extract of the ssh.log:
> <snip>
> 1407355776.833081	CNjybf25kbwTIpD9D6	192.168.88.2	58904	192.168.88.101	22	undetermined	INBOUND	SSH-2.0-MEDUSA_1.0	-	-	-

The ?undetermined? is saying it doesn?t even have a guess as to whether the ssh log in failed or was successful so either type of analysis you?ve tried so far won?t notice anything interesting happening because they?re only concerned about ssh logins with a status of ?success? or ?failure".  I suggest trying to read scripts/base/protocols/ssh/main.bro and understand the criteria it uses to flip the login status to either ?failure? or ?success?, then try to look at conn.log to see which criteria aren?t being met.

- Jon


From jonathon.s.wright at gmail.com  Wed Aug  6 17:58:41 2014
From: jonathon.s.wright at gmail.com (Jonathon Wright)
Date: Wed, 6 Aug 2014 14:58:41 -1000
Subject: [Bro] Bro 2.2 File Extraction (RHEL 6.5)
In-Reply-To: <A8A79A3D-E050-411D-BDAB-B8CE8E117269@icir.org>
References: <CAGX1DMYo2BRwbwayFygV9CkfrWO6DgoYzHNebySmY-Ee1hqEhg@mail.gmail.com>
	<14E53734-31D8-4A9E-8C8E-60B9E08441C2@icir.org>
	<CAGX1DMZNzbPTB3iWFgYRLjrgsUJ2Nkf0g0rkMO9aEDKCCbRr2g@mail.gmail.com>
	<CAGX1DMZy4NoEw69RsSXdT4qm3c0nsdiPStAGU_TqptTg_01VZQ@mail.gmail.com>
	<A8A79A3D-E050-411D-BDAB-B8CE8E117269@icir.org>
Message-ID: <CAGX1DMaayZhOqiF4WM948ay5AYSst5j4q04RirKK2NXTn6z3Eg@mail.gmail.com>

That's great information. When you say " you can set it to something
system-wide though like this"
What file do I edit, or is that entry something I put at the top of my
"whatever.bro" ?

No problem about writing a script. We are a big perl/php/shell shop, I
guess my question is, what files would I need to parse / correlate to
determine the correct / original name of the exe?

Thanks again for your help!


On Wed, Aug 6, 2014 at 11:09 AM, Seth Hall <seth at icir.org> wrote:

>
> On Aug 6, 2014, at 4:41 PM, Jonathon Wright <jonathon.s.wright at gmail.com>
> wrote:
>
> > Too easy, that worked! It created the extracted files in the 'pwd'. I
> checked the md5 they matched from the wireshark pcap file.
>
> Great!
>
> > 1. Can I safely assume, based on these test results, that broctl will
> perform the same way as bro?
>
> Generally yes.  Broctl is just a control harness for Bro that runs it in a
> certain way.
>
> > 2. If so, where will broctl place the 'extracted_files' directory?
>
> Unfortunately that will in the <prefix>/spool/{node-name} directory.  You
> can set it to something system-wide though like this...
>
> redef FilesExtract::prefix = "/extract/here/";
>
> That directory will just need to exist and multiple Bro processes will
> write extracted files there.
>
> > 3. Lastly, whats the best way to investigate these files (I'm capturing
> all exe downloads on HTTP)? For example, the directory 'extracted_files'
> will be full of HTTP-blahblah names. How would I correlate those file names
> to its actual file name? Is that information stored in the conn.log,
> files.log, http.log, packet_filter.log,  & weird.log?
>
> Unfortunately again, that's something where you may want to write a script
> that can take the file names and inspect the logs.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140806/cb545e23/attachment.html 

From seth at icir.org  Wed Aug  6 18:49:59 2014
From: seth at icir.org (Seth Hall)
Date: Wed, 6 Aug 2014 21:49:59 -0400
Subject: [Bro] Bro 2.2 File Extraction (RHEL 6.5)
In-Reply-To: <CAGX1DMaayZhOqiF4WM948ay5AYSst5j4q04RirKK2NXTn6z3Eg@mail.gmail.com>
References: <CAGX1DMYo2BRwbwayFygV9CkfrWO6DgoYzHNebySmY-Ee1hqEhg@mail.gmail.com>
	<14E53734-31D8-4A9E-8C8E-60B9E08441C2@icir.org>
	<CAGX1DMZNzbPTB3iWFgYRLjrgsUJ2Nkf0g0rkMO9aEDKCCbRr2g@mail.gmail.com>
	<CAGX1DMZy4NoEw69RsSXdT4qm3c0nsdiPStAGU_TqptTg_01VZQ@mail.gmail.com>
	<A8A79A3D-E050-411D-BDAB-B8CE8E117269@icir.org>
	<CAGX1DMaayZhOqiF4WM948ay5AYSst5j4q04RirKK2NXTn6z3Eg@mail.gmail.com>
Message-ID: <8C2AEFE1-3BE2-4D8E-AAA8-2DAF70999B1C@icir.org>


On Aug 6, 2014, at 8:58 PM, Jonathon Wright <jonathon.s.wright at gmail.com> wrote:

> That's great information. When you say " you can set it to something system-wide though like this"
> What file do I edit, or is that entry something I put at the top of my "whatever.bro" ?

You could add that directly to local.bro or add it to your whatever.bro script and load that script in local.bro.  

I guess my comment about "system-wide" was far too non-specific. :)

What I meant is that if you're running a number of worker (traffic sniffing) processes on a single host they will each have their own spool directory which will cause them all to write files to separate subdirectories of their spool/ directory.  If you set the prefix to be an absolute path it will cause all of the processes to write their files to that same directory but I don't know what your deployment looks like so I may be giving unhelpful advice.

>  No problem about writing a script. We are a big perl/php/shell shop, I guess my question is, what files would I need to parse / correlate to determine the correct / original name of the exe?

Ah!  That's complicated.  You can refer to the "filename" field in the files log.  For any files that were extracted, you should be able to find the name of the file that was written to disk in the "extracted" field in the files.log.  So, take the filename you have on disk, search for that in the files.log, then look at the "filename" field.

One gotcha here though.  We have taken a somewhat tough line on what we consider a "filename".  The basic gist is that in order to be a filename it must be something explicitly declared as a filename.  In other words, we don't yank path components from HTTP requests to assign as file names.  If we did, you'd very likely extract a bunch of files named index.asp and others like that.  HTTP actually declares a header field where filenames can be explicitly passed through.  Those are extracted and given as filenames in files.log.  Other protocols provide file names in various ways as well.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140806/d90aa098/attachment.bin 

From jonathon.s.wright at gmail.com  Wed Aug  6 19:27:52 2014
From: jonathon.s.wright at gmail.com (Jonathon Wright)
Date: Wed, 6 Aug 2014 16:27:52 -1000
Subject: [Bro] Bro 2.2 File Extraction (RHEL 6.5)
In-Reply-To: <8C2AEFE1-3BE2-4D8E-AAA8-2DAF70999B1C@icir.org>
References: <CAGX1DMYo2BRwbwayFygV9CkfrWO6DgoYzHNebySmY-Ee1hqEhg@mail.gmail.com>
	<14E53734-31D8-4A9E-8C8E-60B9E08441C2@icir.org>
	<CAGX1DMZNzbPTB3iWFgYRLjrgsUJ2Nkf0g0rkMO9aEDKCCbRr2g@mail.gmail.com>
	<CAGX1DMZy4NoEw69RsSXdT4qm3c0nsdiPStAGU_TqptTg_01VZQ@mail.gmail.com>
	<A8A79A3D-E050-411D-BDAB-B8CE8E117269@icir.org>
	<CAGX1DMaayZhOqiF4WM948ay5AYSst5j4q04RirKK2NXTn6z3Eg@mail.gmail.com>
	<8C2AEFE1-3BE2-4D8E-AAA8-2DAF70999B1C@icir.org>
Message-ID: <CAGX1DMYtoDDsXsyNwQDYdb+Ok_gyPgSM+VPigums2dKJBBNd3A@mail.gmail.com>

Very Very interesting.

No worries about specifics, I usually ask if I'm still unsure, but thanks
for the clarification!
Historically, the standard bro "communication.log, wierd.log, etc." logs
that are created under dated directories are what we currently use. We are
now adding the HTTP / exe file carving to our requirements and my thought
was how to know what the original .exe filename was since we keep a db of
md5's of known exe's from the OS that are used for comparisons. The problem
is, I won't know what file/md5_value to compare it too since I wont know
the original filename. Hope that makes sense.

For example, if a user downloads something.exe (via http), bro will create
a HTTP-blahblah file name. My problem at that point, is how do I know what
the user tried to download, was it "notepad.exe" or "maliciousIntent.exe"?
I will only have a directory full of HTTP-blahblah names, correct? That was
where I was trying to go. Perhaps I misunderstood your response and you
already answered me? If so, apolgies, but I still seem to be missing the
connection of the bro created file name when its carved and the actual
filename of the exe that the user attempted to download.



On Wed, Aug 6, 2014 at 3:49 PM, Seth Hall <seth at icir.org> wrote:

>
> On Aug 6, 2014, at 8:58 PM, Jonathon Wright <jonathon.s.wright at gmail.com>
> wrote:
>
> > That's great information. When you say " you can set it to something
> system-wide though like this"
> > What file do I edit, or is that entry something I put at the top of my
> "whatever.bro" ?
>
> You could add that directly to local.bro or add it to your whatever.bro
> script and load that script in local.bro.
>
> I guess my comment about "system-wide" was far too non-specific. :)
>
> What I meant is that if you're running a number of worker (traffic
> sniffing) processes on a single host they will each have their own spool
> directory which will cause them all to write files to separate
> subdirectories of their spool/ directory.  If you set the prefix to be an
> absolute path it will cause all of the processes to write their files to
> that same directory but I don't know what your deployment looks like so I
> may be giving unhelpful advice.
>
> >  No problem about writing a script. We are a big perl/php/shell shop, I
> guess my question is, what files would I need to parse / correlate to
> determine the correct / original name of the exe?
>
> Ah!  That's complicated.  You can refer to the "filename" field in the
> files log.  For any files that were extracted, you should be able to find
> the name of the file that was written to disk in the "extracted" field in
> the files.log.  So, take the filename you have on disk, search for that in
> the files.log, then look at the "filename" field.
>
> One gotcha here though.  We have taken a somewhat tough line on what we
> consider a "filename".  The basic gist is that in order to be a filename it
> must be something explicitly declared as a filename.  In other words, we
> don't yank path components from HTTP requests to assign as file names.  If
> we did, you'd very likely extract a bunch of files named index.asp and
> others like that.  HTTP actually declares a header field where filenames
> can be explicitly passed through.  Those are extracted and given as
> filenames in files.log.  Other protocols provide file names in various ways
> as well.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140806/dea3854e/attachment.html 

From seth at icir.org  Wed Aug  6 21:14:26 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 7 Aug 2014 00:14:26 -0400
Subject: [Bro] Bro 2.2 File Extraction (RHEL 6.5)
In-Reply-To: <CAGX1DMYtoDDsXsyNwQDYdb+Ok_gyPgSM+VPigums2dKJBBNd3A@mail.gmail.com>
References: <CAGX1DMYo2BRwbwayFygV9CkfrWO6DgoYzHNebySmY-Ee1hqEhg@mail.gmail.com>
	<14E53734-31D8-4A9E-8C8E-60B9E08441C2@icir.org>
	<CAGX1DMZNzbPTB3iWFgYRLjrgsUJ2Nkf0g0rkMO9aEDKCCbRr2g@mail.gmail.com>
	<CAGX1DMZy4NoEw69RsSXdT4qm3c0nsdiPStAGU_TqptTg_01VZQ@mail.gmail.com>
	<A8A79A3D-E050-411D-BDAB-B8CE8E117269@icir.org>
	<CAGX1DMaayZhOqiF4WM948ay5AYSst5j4q04RirKK2NXTn6z3Eg@mail.gmail.com>
	<8C2AEFE1-3BE2-4D8E-AAA8-2DAF70999B1C@icir.org>
	<CAGX1DMYtoDDsXsyNwQDYdb+Ok_gyPgSM+VPigums2dKJBBNd3A@mail.gmail.com>
Message-ID: <0ECE8B4D-F1E4-4BB9-9E66-1DA8F6B1929A@icir.org>


On Aug 6, 2014, at 10:27 PM, Jonathon Wright <jonathon.s.wright at gmail.com> wrote:

> The problem is, I won't know what file/md5_value to compare it too since I wont know the original filename. Hope that makes sense.

If you're running Bro with broctl, you will already have hashes (md5 and sha1) for every file transferred in your files.log.  

> For example, if a user downloads something.exe (via http), bro will create a HTTP-blahblah file name. My problem at that point, is how do I know what the user tried to download, was it "notepad.exe" or "maliciousIntent.exe"? I will only have a directory full of HTTP-blahblah names, correct? That was where I was trying to go. Perhaps I misunderstood your response and you already answered me?

# Look at the extracted files.
$ ls ./extract_files
	extract-HTTP-FsRNbD323oiMhWA761

# Look at the line in files.log that maps to that file.
$ grep extract-HTTP-FsRNbD323oiMhWA761 files.log
1407384770.727269	FsRNbD323oiMhWA761	1.2.3.4	5.6.7.8	CuTpVT1LB2eQP0eMP4	HTTP	0	EXTRACT	application/x-dosexec	-	1.151308	49152	49152	0	0	F	-	-	-	-	extract-HTTP-FsRNbD323oiMhWA761

# Look for the HTTP request that maps to that file.
$ grep FsRNbD323oiMhWA761 http.log
1407384770.568614	CuTpVT1LB2eQP0eMP4	5.6.7.8	1066	1.2.3.4	80	1	GET	1.2.3.4	/lprx.php	-	-	0	49152200	OK	-	-	-	(empty)	-	-	-	-	-	FsRNbD323oiMhWA761	application/x-dosexec

You can see in that example that the best file name we could have possibly hoped to extract for that connection would be "lprx.php" which I don't think is what you want.  That is real traffic (with modified field data) from a compromised host downloading an update to the malware installed on it.

> If so, apolgies, but I still seem to be missing the connection of the bro created file name when its carved and the actual filename of the exe that the user attempted to download.

Ah, ok.  I can explain a bit more here.  Before arriving at the current model, I spent a lot of time thinking about how to flexibly name files.  What I realized is that I don't want aspects of the network traffic to be able to affect the name of the file being written to disk (by default at least, you can do whatever you want in your own scripts).  There could be maliciously named files or attempts to play with the path to write into sensitive areas of the file system.  By giving the files being written to disk names that were totally fabricated by the Bro process we sidestep any of these potential issues.  You can use the name of the extracted file to then pivot back into the logs.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140807/20999eb6/attachment.bin 

From seth at icir.org  Wed Aug  6 21:26:55 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 7 Aug 2014 00:26:55 -0400
Subject: [Bro] Question on quick start documentation SSH:Login example.
In-Reply-To: <74E82FA3-BD81-40D9-96FA-388BF5E535D3@illinois.edu>
References: <CACNThYdqJRpKyp_YF3MgqzGf0PgVD6CUmT-tXfBAjRKGo0i0wg@mail.gmail.com>
	<8DF3C970-A563-4DBC-B2F4-358E112D2F86@illinois.edu>
	<CACNThYdOmX37sO0Zy6sffOqL6LbG9kPHGh68W_j-rzY7rbJieg@mail.gmail.com>
	<74E82FA3-BD81-40D9-96FA-388BF5E535D3@illinois.edu>
Message-ID: <82183F97-884A-49D0-81C1-DC1C1D956A55@icir.org>


On Aug 6, 2014, at 5:35 PM, Siwek, Jon <jsiwek at illinois.edu> wrote:

> The ?undetermined? is saying it doesn?t even have a guess as to whether the ssh log in failed or was successful so either type of analysis you?ve tried so far won?t notice anything interesting happening because they?re only concerned about ssh logins with a status of ?success? or ?failure". 

This is where I twist Vlad's arm hard to finish his work on his rewritten SSH analyzer so that we can get rid of my crummy success determiner for SSH connections.  His new one appears to do a greatly improved job at determining success and failure for logins.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140807/adf4625e/attachment.bin 

From Jim.Zhai at ontario.ca  Thu Aug  7 08:17:33 2014
From: Jim.Zhai at ontario.ca (Zhai, Jim (MGS))
Date: Thu, 7 Aug 2014 15:17:33 +0000
Subject: [Bro] bro werid.log are very high
Message-ID: <0922FA096972B14B96C2905E97BB56020C1DE52E@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>

Just wondering why werid.log are very high volume. There is a lot of "possible_split_routing" in werid.log. How to get rid of this issue?

Regards,

Jim Zhai

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140807/3fb3a08a/attachment.html 

From Jim.Zhai at ontario.ca  Thu Aug  7 08:37:24 2014
From: Jim.Zhai at ontario.ca (Zhai, Jim (MGS))
Date: Thu, 7 Aug 2014 15:37:24 +0000
Subject: [Bro] capture_loss are very high
Message-ID: <0922FA096972B14B96C2905E97BB56020C1DE554@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>

We have update the bro from 2.2 to 2.3 and we noticed that the capture_loss increased a lot, over 60% . What will be the reason?

Regards,
Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140807/e3ab7cd8/attachment.html 

From seth at icir.org  Thu Aug  7 08:41:49 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 7 Aug 2014 11:41:49 -0400
Subject: [Bro] bro werid.log are very high
In-Reply-To: <0922FA096972B14B96C2905E97BB56020C1DE52E@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
References: <0922FA096972B14B96C2905E97BB56020C1DE52E@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
Message-ID: <33E02C54-D85A-4C09-B343-E7C9740F555D@icir.org>


On Aug 7, 2014, at 11:17 AM, Zhai, Jim (MGS) <Jim.Zhai at ontario.ca> wrote:

> Just wondering why werid.log are very high volume. There is a lot of ?possible_split_routing? in werid.log. How to get rid of this issue?

It's very possible that you have split routing on your network.  In other words, you might only be seeing one direction of traffic because the other direction of traffic is going on a route that you aren't seeing (another router for example).

Are you loading the misc/capture-loss.bro script?  It's possible that could be cause by a high degree of packet loss as well.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140807/0a92e967/attachment.bin 

From Jim.Zhai at ontario.ca  Thu Aug  7 08:50:32 2014
From: Jim.Zhai at ontario.ca (Zhai, Jim (MGS))
Date: Thu, 7 Aug 2014 15:50:32 +0000
Subject: [Bro] bro werid.log are very high
In-Reply-To: <33E02C54-D85A-4C09-B343-E7C9740F555D@icir.org>
References: <0922FA096972B14B96C2905E97BB56020C1DE52E@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
	<33E02C54-D85A-4C09-B343-E7C9740F555D@icir.org>
Message-ID: <0922FA096972B14B96C2905E97BB56020C1DE57F@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>

Thanks Seth. We do have very high loss degree loss as well, over 60%. We use the bridge-utils to bridge two interface eth1 and eth2 which does split the traffic. We currently just monitoring br0 interface. We recently upgrade bro from 2.2 to 2.3 The capture loss used to be very low on 2.2. But the wried.log remain the same.  Just wondering if software bridge setting works in this situation?

Regards,

Jim Zhai


-----Original Message-----
From: Seth Hall [mailto:seth at icir.org] 
Sent: August-07-14 11:42 AM
To: Zhai, Jim (MGS)
Cc: bro at bro.org
Subject: Re: [Bro] bro werid.log are very high


On Aug 7, 2014, at 11:17 AM, Zhai, Jim (MGS) <Jim.Zhai at ontario.ca> wrote:

> Just wondering why werid.log are very high volume. There is a lot of "possible_split_routing" in werid.log. How to get rid of this issue?

It's very possible that you have split routing on your network.  In other words, you might only be seeing one direction of traffic because the other direction of traffic is going on a route that you aren't seeing (another router for example).

Are you loading the misc/capture-loss.bro script?  It's possible that could be cause by a high degree of packet loss as well.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From jlay at slave-tothe-box.net  Thu Aug  7 09:26:25 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 07 Aug 2014 10:26:25 -0600
Subject: [Bro] Quick smtp-url-extraction question
Message-ID: <67f16fa2c2516e86a55f15ffb0123616@localhost>

Hey all,

So here's the run:

sudo bro -C -r ../captures/email.pcapng 
/usr/local/bro/share/bro/policy/frameworks/intel/seen/smtp-url-extraction.bro

and list of files generated:
-rw-r--r-- 1 root root 12419 Aug  7 10:18 conn.log
-rw-r--r-- 1 root root     0 Aug  7 10:18 debug.log
-rw-r--r-- 1 root root 12586 Aug  7 10:18 files.log
-rw-r--r-- 1 root root   253 Aug  7 10:18 packet_filter.log
-rw-r--r-- 1 root root 39557 Aug  7 10:18 smtp.log
-rw-r--r-- 1 root root  7936 Aug  7 10:18 ssl.log
-rw-r--r-- 1 root root  8608 Aug  7 10:18 x509.log

For the life of me I'm unable to find where the links might be at.  One 
of the links in the pcap has 88EX336W4062X11N55206638L1122194955 in 
it...this string shows up no where in any of the logs...is there a step 
I'm missing with this?  Thank you.

James


From seth at icir.org  Thu Aug  7 10:17:35 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 7 Aug 2014 13:17:35 -0400
Subject: [Bro] bro werid.log are very high
In-Reply-To: <0922FA096972B14B96C2905E97BB56020C1DE57F@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
References: <0922FA096972B14B96C2905E97BB56020C1DE52E@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
	<33E02C54-D85A-4C09-B343-E7C9740F555D@icir.org>
	<0922FA096972B14B96C2905E97BB56020C1DE57F@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
Message-ID: <AA5CBB96-EF8D-4C9E-A34D-88A416209108@icir.org>


On Aug 7, 2014, at 11:50 AM, Zhai, Jim (MGS) <Jim.Zhai at ontario.ca> wrote:

> Thanks Seth. We do have very high loss degree loss as well, over 60%.

You're determining that number from capture-loss.log or something else?

> We use the bridge-utils to bridge two interface eth1 and eth2 which does split the traffic.

Did you mean that it merges the traffic?

> We recently upgrade bro from 2.2 to 2.3 The capture loss used to be very low on 2.2. But the wried.log remain the same.  Just wondering if software bridge setting works in this situation?

Yeah, that should work fine.  It sounds like you might want to come up with a solution to your packet loss first.  Unfortunately I can't give you an answer without knowing more about your network and what your deploy looks like.  In most cases, 2.3 should actually be more efficient than 2.2.  There was some work done around identifying some major inefficiencies and addressing them.

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140807/1cce2940/attachment.bin 

From Jim.Zhai at ontario.ca  Thu Aug  7 10:22:56 2014
From: Jim.Zhai at ontario.ca (Zhai, Jim (MGS))
Date: Thu, 7 Aug 2014 17:22:56 +0000
Subject: [Bro] bro werid.log are very high
In-Reply-To: <AA5CBB96-EF8D-4C9E-A34D-88A416209108@icir.org>
References: <0922FA096972B14B96C2905E97BB56020C1DE52E@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
	<33E02C54-D85A-4C09-B343-E7C9740F555D@icir.org>
	<0922FA096972B14B96C2905E97BB56020C1DE57F@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
	<AA5CBB96-EF8D-4C9E-A34D-88A416209108@icir.org>
Message-ID: <0922FA096972B14B96C2905E97BB56020C1DE602@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>

>You're determining that number from capture-loss.log or something else?
Yes, we find this from capture-loss.log.   It used to be very low. But after upgrade 2.3 today, it jumps to 67%

>Did you mean that it merges the traffic?

Inbound and outbound merges

Regards,

Jim Zhai


-----Original Message-----
From: Seth Hall [mailto:seth at icir.org] 
Sent: August-07-14 1:18 PM
To: Zhai, Jim (MGS)
Cc: bro at bro.org
Subject: Re: [Bro] bro werid.log are very high


On Aug 7, 2014, at 11:50 AM, Zhai, Jim (MGS) <Jim.Zhai at ontario.ca> wrote:

> Thanks Seth. We do have very high loss degree loss as well, over 60%.

You're determining that number from capture-loss.log or something else?

> We use the bridge-utils to bridge two interface eth1 and eth2 which does split the traffic.

Did you mean that it merges the traffic?

> We recently upgrade bro from 2.2 to 2.3 The capture loss used to be very low on 2.2. But the wried.log remain the same.  Just wondering if software bridge setting works in this situation?

Yeah, that should work fine.  It sounds like you might want to come up with a solution to your packet loss first.  Unfortunately I can't give you an answer without knowing more about your network and what your deploy looks like.  In most cases, 2.3 should actually be more efficient than 2.2.  There was some work done around identifying some major inefficiencies and addressing them.

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From seth at icir.org  Thu Aug  7 10:26:28 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 7 Aug 2014 13:26:28 -0400
Subject: [Bro] Quick smtp-url-extraction question
In-Reply-To: <67f16fa2c2516e86a55f15ffb0123616@localhost>
References: <67f16fa2c2516e86a55f15ffb0123616@localhost>
Message-ID: <D36825AC-89C7-4AF3-B1C2-091D0FE2E151@icir.org>


On Aug 7, 2014, at 12:26 PM, James Lay <jlay at slave-tothe-box.net> wrote:

> sudo bro -C -r ../captures/email.pcapng 
> /usr/local/bro/share/bro/policy/frameworks/intel/seen/smtp-url-extraction.bro

Ah!  Perhaps a poorly named script.  That's only extracting the URLs and feeding them into the intel framework.

Would you like a script that extracts and logs them?  I ran one of those in production before, it was useful to be able to see what links were flying around for sure.

I'm thinking for fields we could have...

	ts
	uid
	fuid
	trans_depth
	link

That should provide enough information to link back to the connection it happened over and which "file" (or body content since they're effectively the same in smtp) it was seen within.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140807/b5239d3c/attachment.bin 

From seth at icir.org  Thu Aug  7 10:30:12 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 7 Aug 2014 13:30:12 -0400
Subject: [Bro] bro werid.log are very high
In-Reply-To: <0922FA096972B14B96C2905E97BB56020C1DE602@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
References: <0922FA096972B14B96C2905E97BB56020C1DE52E@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
	<33E02C54-D85A-4C09-B343-E7C9740F555D@icir.org>
	<0922FA096972B14B96C2905E97BB56020C1DE57F@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
	<AA5CBB96-EF8D-4C9E-A34D-88A416209108@icir.org>
	<0922FA096972B14B96C2905E97BB56020C1DE602@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
Message-ID: <DD5221BE-1FBB-4ABE-8809-24B736FF7A4F@icir.org>


On Aug 7, 2014, at 1:22 PM, Zhai, Jim (MGS) <Jim.Zhai at ontario.ca> wrote:

>> You're determining that number from capture-loss.log or something else?
> Yes, we find this from capture-loss.log.   It used to be very low. But after upgrade 2.3 today, it jumps to 67%

Hm, some of the TCP handling was rewritten for 2.3.  It's possible you're running into edge cases that weren't handled correctly.

Would it be possible for you to privately provide us with some of your conn.log and weird.log files?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140807/69414d0c/attachment.bin 

From jlay at slave-tothe-box.net  Thu Aug  7 10:30:49 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 07 Aug 2014 11:30:49 -0600
Subject: [Bro] Quick smtp-url-extraction question
In-Reply-To: <D36825AC-89C7-4AF3-B1C2-091D0FE2E151@icir.org>
References: <67f16fa2c2516e86a55f15ffb0123616@localhost>
	<D36825AC-89C7-4AF3-B1C2-091D0FE2E151@icir.org>
Message-ID: <175259a1168f271b0ef7759c72f9bf39@localhost>

On 2014-08-07 11:26, Seth Hall wrote:
> On Aug 7, 2014, at 12:26 PM, James Lay <jlay at slave-tothe-box.net> 
> wrote:
>
>> sudo bro -C -r ../captures/email.pcapng
>> 
>> /usr/local/bro/share/bro/policy/frameworks/intel/seen/smtp-url-extraction.bro
>
> Ah!  Perhaps a poorly named script.  That's only extracting the URLs
> and feeding them into the intel framework.
>
> Would you like a script that extracts and logs them?  I ran one of
> those in production before, it was useful to be able to see what 
> links
> were flying around for sure.
>
> I'm thinking for fields we could have...
>
> 	ts
> 	uid
> 	fuid
> 	trans_depth
> 	link
>
> That should provide enough information to link back to the connection
> it happened over and which "file" (or body content since they're
> effectively the same in smtp) it was seen within.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/

I would absolutely love a script that would log urls....we all know 
that quoted-printable and bas364 shenanigans may get missed, but every 
little bit helps..thanks a bunch Seth.

James


From seth at icir.org  Thu Aug  7 10:39:24 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 7 Aug 2014 13:39:24 -0400
Subject: [Bro] Quick smtp-url-extraction question
In-Reply-To: <175259a1168f271b0ef7759c72f9bf39@localhost>
References: <67f16fa2c2516e86a55f15ffb0123616@localhost>
	<D36825AC-89C7-4AF3-B1C2-091D0FE2E151@icir.org>
	<175259a1168f271b0ef7759c72f9bf39@localhost>
Message-ID: <EB78E1F8-41CD-4051-90C1-497E8905B84F@icir.org>


On Aug 7, 2014, at 1:30 PM, James Lay <jlay at slave-tothe-box.net> wrote:

> I would absolutely love a script that would log urls....we all know that quoted-printable and bas364 shenanigans may get missed

Much of that should be handled automatically by the mime analyzer (I'm not sure of the limits of that offhand).

> , but every little bit helps..thanks a bunch Seth.

I'll see if I can get to it soon.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140807/01f1515d/attachment.bin 

From hhoffman at ip-solutions.net  Thu Aug  7 11:37:57 2014
From: hhoffman at ip-solutions.net (Harry Hoffman)
Date: Thu, 7 Aug 2014 14:37:57 -0400
Subject: [Bro] Bro and myricom woes
Message-ID: <F80D6F50-9697-440C-B86D-77DBC00D31EC@ip-solutions.net>

Hi All,

Thought I?d write in to seek some guidance from the list.

I?ve got bro running on RHEL 6.5 sitting on a box with 20 cores and 64 GB of RAM and a RAID 6 configuration through 1.2TB disks on a LSI raid card. This is a Cisco UCS 1u server.

I?m running with myricom?s sniffer 10G software (v3) in an x16 slot set at GEN II in the BIOS (I don?t have a x8 slot to put it in).

I?ve tried running both bro out of git and bro 2.3.

/usr/local/bro/etc/node.conf looks like:
drop ring full
[manager]
type=manager
host=localhost
#
[proxy-1]
type=proxy
host=localhost
#
[worker-1]
type=worker
host=localhost
interface=eth2
lb_method=myricom
lb_procs=14
pin_cpus=2,3,4,5,6,7,8,9,10,11,12,13,14,15
#

Policies are set to default for what broctl uses.

I?ve been playing around with the myricom environmentals and have the following exported before running broctl start
SNF_DATARING_SIZE = 17179869184
SNF_DESCRING_SIZE = 4294967296

Running tcpdump to output to /dev/null I see no drops in packet capture (either through myri_counters looking at the SNF ring drop full) or from tcpdump itself. Writing full packet capture to disk using tcpdump -i eth2 -s 0 -C 500 -w /usr/local/bro/logs/testing shows roughly a 2% drop via myri_counters SNF drop ring full.

Average traffic on the interface is .5 Mpps. I?m using some of the features of our gigamon to slice packets and only keep up to 32 bytes of certain packets discarding the rest of the payload (I mention this in the event that bro might have some difficulty in dealing with packets that are shorter then their advertised length).

Of all of the workers running 1 worker is always pegged at 100% and the rest of the workers use roughly 50-65% of their CPU at any given point in time.
when running broctl stop all workers stop fine except the one pegging 100% of the cpu. That was is forcibly terminated.

The increase of counters for SNF drop ring full indicates that the application (bro in this case) is not getting to the packets fast enough. broctl seems to be reporting roughly 80% packet loss.

I?ve got 1490 networks defined in /usr/local/bro/etc/networks.conf but I?m under the impression that this only matters for the email reports.

I?m happy to provide any other information but would really like to have bro work smoothly here so am hoping that someone can chime in with experiences in dealing with such problems.


many thanks!

Cheers,
Harry




From seth at icir.org  Thu Aug  7 14:04:14 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 7 Aug 2014 17:04:14 -0400
Subject: [Bro] Bro and myricom woes
In-Reply-To: <F80D6F50-9697-440C-B86D-77DBC00D31EC@ip-solutions.net>
References: <F80D6F50-9697-440C-B86D-77DBC00D31EC@ip-solutions.net>
Message-ID: <10A1FC55-47A8-456E-8812-08B53F3012B1@icir.org>


On Aug 7, 2014, at 2:37 PM, Harry Hoffman <hhoffman at ip-solutions.net> wrote:

> I?m running with myricom?s sniffer 10G software (v3) in an x16 slot set at GEN II in the BIOS (I don?t have a x8 slot to put it in).

I don't actually have docs from Myricom yet, but I hope our myricom plugin still support their software in v3. :)

> I?m using some of the features of our gigamon to slice packets and only keep up to 32 bytes of certain packets discarding

Ding!  You got it.  This completely breaks Bro and you really don't want to do it.  Try removing that trimming config and see how Bro reacts.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140807/af96d92a/attachment.bin 

From hhoffman at ip-solutions.net  Thu Aug  7 14:20:02 2014
From: hhoffman at ip-solutions.net (Harry Hoffman)
Date: Thu, 07 Aug 2014 17:20:02 -0400
Subject: [Bro] Bro and myricom woes
Message-ID: <20140807212004.65BFC2F384@pb-smtp0.pobox.com>

Aha! Thanks, Seth.

I guess I need to think through this a bit.

Just curious, why does Bro break? Or what makes bro break?

Also, myricom needs to update some of their docs for v3 but all seems to still function just fine.

Cheers,
Harry

On Aug 7, 2014 5:04 PM, Seth Hall <seth at icir.org> wrote:
>
>
> On Aug 7, 2014, at 2:37 PM, Harry Hoffman <hhoffman at ip-solutions.net> wrote: 
>
> > I?m running with myricom?s sniffer 10G software (v3) in an x16 slot set at GEN II in the BIOS (I don?t have a x8 slot to put it in). 
>
> I don't actually have docs from Myricom yet, but I hope our myricom plugin still support their software in v3. :) 
>
> > I?m using some of the features of our gigamon to slice packets and only keep up to 32 bytes of certain packets discarding 
>
> Ding!? You got it.? This completely breaks Bro and you really don't want to do it.? Try removing that trimming config and see how Bro reacts. 
>
> ? .Seth 
>
> -- 
> Seth Hall 
> International Computer Science Institute 
> (Bro) because everyone has a network 
> http://www.bro.org/ 
>



From jlay at slave-tothe-box.net  Thu Aug  7 16:50:20 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 07 Aug 2014 17:50:20 -0600
Subject: [Bro] Quick smtp-url-extraction question
In-Reply-To: <EB78E1F8-41CD-4051-90C1-497E8905B84F@icir.org>
References: <67f16fa2c2516e86a55f15ffb0123616@localhost>
	<D36825AC-89C7-4AF3-B1C2-091D0FE2E151@icir.org>
	<175259a1168f271b0ef7759c72f9bf39@localhost>
	<EB78E1F8-41CD-4051-90C1-497E8905B84F@icir.org>
Message-ID: <1407455420.2576.5.camel@JamesiMac>

On Thu, 2014-08-07 at 13:39 -0400, Seth Hall wrote:

> On Aug 7, 2014, at 1:30 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> 
> > I would absolutely love a script that would log urls....we all know that quoted-printable and bas364 shenanigans may get missed
> 
> Much of that should be handled automatically by the mime analyzer (I'm not sure of the limits of that offhand).
> 
> > , but every little bit helps..thanks a bunch Seth.
> 
> I'll see if I can get to it soon.
> 
>   .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 


Thanks again Seth.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140807/a2b32dc4/attachment.html 

From jonathon.s.wright at gmail.com  Thu Aug  7 17:16:17 2014
From: jonathon.s.wright at gmail.com (Jonathon Wright)
Date: Thu, 7 Aug 2014 14:16:17 -1000
Subject: [Bro] Bro 2.2 File Extraction (RHEL 6.5)
In-Reply-To: <0ECE8B4D-F1E4-4BB9-9E66-1DA8F6B1929A@icir.org>
References: <CAGX1DMYo2BRwbwayFygV9CkfrWO6DgoYzHNebySmY-Ee1hqEhg@mail.gmail.com>
	<14E53734-31D8-4A9E-8C8E-60B9E08441C2@icir.org>
	<CAGX1DMZNzbPTB3iWFgYRLjrgsUJ2Nkf0g0rkMO9aEDKCCbRr2g@mail.gmail.com>
	<CAGX1DMZy4NoEw69RsSXdT4qm3c0nsdiPStAGU_TqptTg_01VZQ@mail.gmail.com>
	<A8A79A3D-E050-411D-BDAB-B8CE8E117269@icir.org>
	<CAGX1DMaayZhOqiF4WM948ay5AYSst5j4q04RirKK2NXTn6z3Eg@mail.gmail.com>
	<8C2AEFE1-3BE2-4D8E-AAA8-2DAF70999B1C@icir.org>
	<CAGX1DMYtoDDsXsyNwQDYdb+Ok_gyPgSM+VPigums2dKJBBNd3A@mail.gmail.com>
	<0ECE8B4D-F1E4-4BB9-9E66-1DA8F6B1929A@icir.org>
Message-ID: <CAGX1DMbJwG72+Q9SUeRFh2qcZqF5QvBZHE8ZhF0vrHSraHua4A@mail.gmail.com>

This is awesome information, I've been busy "playing" with my extract.bro
and scripts to see what kind of process I can come up with. Thanks again
for all the assistance!!


On Wed, Aug 6, 2014 at 6:14 PM, Seth Hall <seth at icir.org> wrote:

>
> On Aug 6, 2014, at 10:27 PM, Jonathon Wright <jonathon.s.wright at gmail.com>
> wrote:
>
> > The problem is, I won't know what file/md5_value to compare it too since
> I wont know the original filename. Hope that makes sense.
>
> If you're running Bro with broctl, you will already have hashes (md5 and
> sha1) for every file transferred in your files.log.
>
> > For example, if a user downloads something.exe (via http), bro will
> create a HTTP-blahblah file name. My problem at that point, is how do I
> know what the user tried to download, was it "notepad.exe" or
> "maliciousIntent.exe"? I will only have a directory full of HTTP-blahblah
> names, correct? That was where I was trying to go. Perhaps I misunderstood
> your response and you already answered me?
>
> # Look at the extracted files.
> $ ls ./extract_files
>         extract-HTTP-FsRNbD323oiMhWA761
>
> # Look at the line in files.log that maps to that file.
> $ grep extract-HTTP-FsRNbD323oiMhWA761 files.log
> 1407384770.727269       FsRNbD323oiMhWA761      1.2.3.4 5.6.7.8
> CuTpVT1LB2eQP0eMP4      HTTP    0       EXTRACT application/x-dosexec   -
>     1.151308        49152   49152   0       0       F       -       -
> -       -       extract-HTTP-FsRNbD323oiMhWA761
>
> # Look for the HTTP request that maps to that file.
> $ grep FsRNbD323oiMhWA761 http.log
> 1407384770.568614       CuTpVT1LB2eQP0eMP4      5.6.7.8 1066    1.2.3.4 80
>      1       GET     1.2.3.4 /lprx.php       -       -       0
> 49152200        OK      -       -       -       (empty) -       -       -
>     -       -       FsRNbD323oiMhWA761      application/x-dosexec
>
> You can see in that example that the best file name we could have possibly
> hoped to extract for that connection would be "lprx.php" which I don't
> think is what you want.  That is real traffic (with modified field data)
> from a compromised host downloading an update to the malware installed on
> it.
>
> > If so, apolgies, but I still seem to be missing the connection of the
> bro created file name when its carved and the actual filename of the exe
> that the user attempted to download.
>
> Ah, ok.  I can explain a bit more here.  Before arriving at the current
> model, I spent a lot of time thinking about how to flexibly name files.
>  What I realized is that I don't want aspects of the network traffic to be
> able to affect the name of the file being written to disk (by default at
> least, you can do whatever you want in your own scripts).  There could be
> maliciously named files or attempts to play with the path to write into
> sensitive areas of the file system.  By giving the files being written to
> disk names that were totally fabricated by the Bro process we sidestep any
> of these potential issues.  You can use the name of the extracted file to
> then pivot back into the logs.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140807/1e216d23/attachment.html 

From doris at bro.org  Thu Aug  7 17:59:19 2014
From: doris at bro.org (Doris Schioberg)
Date: Thu, 07 Aug 2014 17:59:19 -0700
Subject: [Bro] Meet the Bro Teaching Community
Message-ID: <53E420E7.9010901@bro.org>

We are happy to announce the newly started Bro Teaching Community,
a community project of educators interested in collaboratively
exploring Bro's use as a teaching tool, and sharing experiences and 
material.
The goal is to create a knowledge base and
resource collection for educators, ranging from example curricula and
slide sets to exercises for all purposes and skills levels.

We invite you to participate in our open discussion every
Tuesday at 10:00 AM PDT. In these meetings we discuss planned curricula,
practical and technical topics around exercises, slide sets, and general
questions related to teaching security, networks and systems with Bro

For details see www.bro.org/teaching/ or contact us directly via 
teaching at bro.org

The Bro Team

-- 
Doris Schioberg
Bro Outreach Coordinator
International Computer Science Institute (ICSI Berkeley)
Phone: +1 (510) 289-8406 * doris at bro.org


From doris at bro.org  Thu Aug  7 18:37:45 2014
From: doris at bro.org (Doris Schioberg)
Date: Thu, 07 Aug 2014 18:37:45 -0700
Subject: [Bro] Meet the Bro Teaching Community
In-Reply-To: <53E420E7.9010901@bro.org>
References: <53E420E7.9010901@bro.org>
Message-ID: <53E429E9.30201@bro.org>

Please contact info at bro for the initial contact or subscribe to 
teaching at bro.org.


On 8/7/14, 5:59 PM, Doris Schioberg wrote:
> We are happy to announce the newly started Bro Teaching Community,
> a community project of educators interested in collaboratively
> exploring Bro's use as a teaching tool, and sharing experiences and
> material.
> The goal is to create a knowledge base and
> resource collection for educators, ranging from example curricula and
> slide sets to exercises for all purposes and skills levels.
>
> We invite you to participate in our open discussion every
> Tuesday at 10:00 AM PDT. In these meetings we discuss planned curricula,
> practical and technical topics around exercises, slide sets, and general
> questions related to teaching security, networks and systems with Bro
>
> For details see www.bro.org/teaching/ or contact us directly via
> teaching at bro.org
>
> The Bro Team
>

-- 
Doris Schioberg
Bro Outreach Coordinator
International Computer Science Institute (ICSI Berkeley)
Phone: +1 (510) 289-8406 * doris at bro.org


From seth at icir.org  Thu Aug  7 20:16:56 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 7 Aug 2014 23:16:56 -0400
Subject: [Bro] Bro and myricom woes
In-Reply-To: <20140807212004.65BFC2F384@pb-smtp0.pobox.com>
References: <20140807212004.65BFC2F384@pb-smtp0.pobox.com>
Message-ID: <47839373-033A-477D-B41D-FBF19FC2F3E2@icir.org>


On Aug 7, 2014, at 5:20 PM, Harry Hoffman <hhoffman at ip-solutions.net> wrote:

> Just curious, why does Bro break? Or what makes bro break?

Bro does deep packet inspection and if the packets are artificially cropped and Bro can't see all of the traffic there ends up being a lot of missed bytes in connections.  In order to correctly analyze many connections, all of the packet data needs to be there.

> Also, myricom needs to update some of their docs for v3 but all seems to still function just fine.

They actually added a feature we requested.  It should be possible to load balance traffic multiple times to multiple tools now.  We still need to adapt our Myricom broctl plugin to support the way that works.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140807/db877182/attachment.bin 

From Jim.Zhai at ontario.ca  Fri Aug  8 05:51:08 2014
From: Jim.Zhai at ontario.ca (Zhai, Jim (MGS))
Date: Fri, 8 Aug 2014 12:51:08 +0000
Subject: [Bro] report log for error message
Message-ID: <0922FA096972B14B96C2905E97BB56020C1DEB3A@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>

Got a lot of ERROR in report log for the smtp.  "....Reporter::ERROR	field value missing [SMTPurl::c$smtp$from]	....." Is that some way to ignore this record?

-Jim



From hosom at battelle.org  Fri Aug  8 05:55:08 2014
From: hosom at battelle.org (Hosom, Stephen M)
Date: Fri, 8 Aug 2014 12:55:08 +0000
Subject: [Bro] report log for error message
In-Reply-To: <0922FA096972B14B96C2905E97BB56020C1DEB3A@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
References: <0922FA096972B14B96C2905E97BB56020C1DEB3A@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
Message-ID: <E5A2071C6FFFF640B714C224139C9B3310A528@WP-MBX2.milky-way.battelle.org>

Jim, 

It's normally best to prevent these errors by checking to determine if the value exists before using it. 

For example:

	if ( c$smtp?$from )
		## do stuff

Lots of errors within Bro scripts can cause some pretty interesting problems with your cluster.

-----Original Message-----
From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Zhai, Jim (MGS)
Sent: Friday, August 08, 2014 8:51 AM
To: bro at bro.org
Subject: [Bro] report log for error message

Got a lot of ERROR in report log for the smtp.  "....Reporter::ERROR	field value missing [SMTPurl::c$smtp$from]	....." Is that some way to ignore this record?

-Jim

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From Jim.Zhai at ontario.ca  Fri Aug  8 05:56:20 2014
From: Jim.Zhai at ontario.ca (Zhai, Jim (MGS))
Date: Fri, 8 Aug 2014 12:56:20 +0000
Subject: [Bro] report log for error message
In-Reply-To: <E5A2071C6FFFF640B714C224139C9B3310A528@WP-MBX2.milky-way.battelle.org>
References: <0922FA096972B14B96C2905E97BB56020C1DEB3A@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
	<E5A2071C6FFFF640B714C224139C9B3310A528@WP-MBX2.milky-way.battelle.org>
Message-ID: <0922FA096972B14B96C2905E97BB56020C1DEB4F@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>

Great Thanks,
-Jim 

-----Original Message-----
From: Hosom, Stephen M [mailto:hosom at battelle.org] 
Sent: August-08-14 8:55 AM
To: Zhai, Jim (MGS); bro at bro.org
Subject: RE: report log for error message

Jim, 

It's normally best to prevent these errors by checking to determine if the value exists before using it. 

For example:

	if ( c$smtp?$from )
		## do stuff

Lots of errors within Bro scripts can cause some pretty interesting problems with your cluster.

-----Original Message-----
From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Zhai, Jim (MGS)
Sent: Friday, August 08, 2014 8:51 AM
To: bro at bro.org
Subject: [Bro] report log for error message

Got a lot of ERROR in report log for the smtp.  "....Reporter::ERROR	field value missing [SMTPurl::c$smtp$from]	....." Is that some way to ignore this record?

-Jim

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From johanna at icir.org  Fri Aug  8 05:56:40 2014
From: johanna at icir.org (Johanna Amann)
Date: Fri, 08 Aug 2014 05:56:40 -0700
Subject: [Bro] report log for error message
In-Reply-To: <0922FA096972B14B96C2905E97BB56020C1DEB3A@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
References: <0922FA096972B14B96C2905E97BB56020C1DEB3A@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>
Message-ID: <5C1FD1B4-1213-4B01-AFA2-7EF1ACA4B78B@icir.org>

In your scripts, you can (and should) check the existence of optional 
values (like from in smtp) using the ?$ operator.

In this case, if c$smtp?$from returns true, the field is set and you can 
access it.

Johanna

On 8 Aug 2014, at 5:51, Zhai, Jim (MGS) wrote:

> Got a lot of ERROR in report log for the smtp.  
> "....Reporter::ERROR	field value missing [SMTPurl::c$smtp$from]	....." 
> Is that some way to ignore this record?
>
> -Jim
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From julien.t43 at gmail.com  Sun Aug 10 07:47:49 2014
From: julien.t43 at gmail.com (Julien T)
Date: Sun, 10 Aug 2014 10:47:49 -0400
Subject: [Bro] Bro 2.3 and Ubuntu 14.04
Message-ID: <CAF-0m9j9pKNjPWKRQks=t4C1rZeFJGtV+U86CyhOjekYCg4XUA@mail.gmail.com>

Hello,

I'm trying to install Bro on a Ubuntu 14.04.1. I want to use debian package
from
http://www.bro.org/download/index.html

But the package seems to ask for
libc6 (<< 2.12) whereas I have 2.19-0ubuntu6.1
libpython2.6 which is in database but not available

Any debian source package to compile?
Or my only option is source.

Any reason Bro which is kind of old didn't make is way to official
debian/ubuntu repository?

Thanks.
Cheers,

Julien
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140810/0be434ba/attachment.html 

From slagell at illinois.edu  Sun Aug 10 08:21:10 2014
From: slagell at illinois.edu (Slagell, Adam J)
Date: Sun, 10 Aug 2014 15:21:10 +0000
Subject: [Bro] Bro 2.3 and Ubuntu 14.04
In-Reply-To: <CAF-0m9j9pKNjPWKRQks=t4C1rZeFJGtV+U86CyhOjekYCg4XUA@mail.gmail.com>
References: <CAF-0m9j9pKNjPWKRQks=t4C1rZeFJGtV+U86CyhOjekYCg4XUA@mail.gmail.com>
Message-ID: <1458545B-4820-482C-872B-D17EA969CE4C@illinois.edu>


On Aug 10, 2014, at 9:47 AM, Julien T <julien.t43 at gmail.com<mailto:julien.t43 at gmail.com>> wrote:

Any debian source package to compile?
Or my only option is source.

No. Just the source tarball.


Any reason Bro which is kind of old didn't make is way to official debian/ubuntu repository?

No one has volunteered to be the Debian package maintainer.

------

Adam J. Slagell
Chief Information Security Officer
Assistant Director, Cybersecurity Directorate
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info<http://www.slagell.info>

"Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure."









-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140810/3a0705f7/attachment.html 

From fengqingleiyue at 163.com  Wed Aug 13 03:44:55 2014
From: fengqingleiyue at 163.com (fql)
Date: Wed, 13 Aug 2014 18:44:55 +0800 (CST)
Subject: [Bro] How to use Broccoli to pull event from Bro
Message-ID: <20925cde.c537.147cef875b1.Coremail.fengqingleiyue@163.com>

Hi Everyone; 
    we are now using Bro to decode netflow and packet , and we found that it 's pretty good product , it gave use such a big impression , but one thing we feel a little  truble is that, the log writer can only write all the network activity into some log files (http.log,conn.log), i went though some documents on www.bro.org , and found that it supports DataSeries , ElasticSearch & SQLIte database as extral outputs  , unforunately none of this features match our requriment . however i found another thing called "Broccoli " can talk to Bro . i wrote some C program which can sent event to Bro and get the event which i defined from Bro , Now i have a question , can i use "Broccoli" to pull the event which looks like the lines in the log files [e.g "conn.log"] or send the content of these logs in some format like Syslog to some server , if anyone know how to do it , please tell me , because i was haunnted by this question for a log time .  
Thanks you for your time on my eamil . 


Regards,
Fql 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140813/d5ab3835/attachment.html 

From jsiwek at illinois.edu  Wed Aug 13 07:30:02 2014
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Wed, 13 Aug 2014 14:30:02 +0000
Subject: [Bro] How to use Broccoli to pull event from Bro
In-Reply-To: <20925cde.c537.147cef875b1.Coremail.fengqingleiyue@163.com>
References: <20925cde.c537.147cef875b1.Coremail.fengqingleiyue@163.com>
Message-ID: <50F4788C-6F68-4723-B0AA-9F0FB28CC77A@illinois.edu>


On Aug 13, 2014, at 5:44 AM, fql <fengqingleiyue at 163.com> wrote:

> Now i have a question , can i use "Broccoli" to pull the event which looks like the lines in the log files [e.g "conn.log?]

On the Bro side, there is usually an event that corresponds to a given log line, e.g. the ?Conn::log_conn? event [1].  On the Broccoli side, there?s a general outline of what needs to be done to receive events at [2], which you should be able to follow to receive events whose parameters correspond to the fields of log files, e.g. ?Conn::log_conn? or some other event that you?ve defined yourself in order to pick a subset of the fields that are interesting to you.

- Jon

[1] http://www.bro.org/sphinx-git/scripts/base/protocols/conn/main.bro.html#id-Conn::log_conn
[2] http://www.bro.org/sphinx-git/components/broccoli/broccoli-manual.html#receiving-events


From seth at icir.org  Wed Aug 13 07:41:43 2014
From: seth at icir.org (Seth Hall)
Date: Wed, 13 Aug 2014 10:41:43 -0400
Subject: [Bro] How to use Broccoli to pull event from Bro
In-Reply-To: <20925cde.c537.147cef875b1.Coremail.fengqingleiyue@163.com>
References: <20925cde.c537.147cef875b1.Coremail.fengqingleiyue@163.com>
Message-ID: <957E610F-BE97-45D4-8C60-02D85A4E5AF4@icir.org>


On Aug 13, 2014, at 6:44 AM, fql <fengqingleiyue at 163.com> wrote:

>  i went though some documents on www.bro.org , and found that it supports DataSeries , ElasticSearch & SQLIte database as extral outputs  , unforunately none of this features match our requriment . however i found another thing called "Broccoli " can talk to Bro .

I don't think you want to use Broccoli for this.  If you look into the Bro source code, you will see that our log writers are abstracted and you can write your own log writer.

May I ask what your requirements are?  Where and how do you want to be able to write logs?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140813/c4537fec/attachment.bin 

From monahbaki at gmail.com  Wed Aug 13 10:54:24 2014
From: monahbaki at gmail.com (Monah Baki)
Date: Wed, 13 Aug 2014 13:54:24 -0400
Subject: [Bro] http.log and dns.log missing
Message-ID: <CALP3=x_ufVakN9b9qHjhkaHMKLoahqbYbt9+QEFzo=Zt7BB9Cg@mail.gmail.com>

Running  Bro 2.3 on Redhat ES. If I run tcpdump, I do see 80 and 53
requests, and of all the logs in /opt/bro/logs/current, I see them
increase, but the http and dns log is missing. How can I troubleshoot
as to why I am missing the two files?


Thanks


From cryptowave at gmail.com  Wed Aug 13 12:16:42 2014
From: cryptowave at gmail.com (JRH)
Date: Wed, 13 Aug 2014 15:16:42 -0400
Subject: [Bro] [OT] Open Position
Message-ID: <CABwOY18fYhHQeO8hvMjbjyjY2XSjWHE47JLGUs52vVUrzwahOg@mail.gmail.com>

Do you or do you have any friends looking for a position deploying,
configuring, and customizing BRO in the Washington DC metro area? Minimum
of a SECRET clearance is required.

If you are interested in more specifics, please drop me a note off-list.
Thanks to the BRO moderators for letting me try to recruit one of you smart
folks from the list.


-john
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140813/19d07255/attachment.html 

From gfaulkner.nsm at gmail.com  Wed Aug 13 20:33:33 2014
From: gfaulkner.nsm at gmail.com (Gary Faulkner)
Date: Wed, 13 Aug 2014 22:33:33 -0500
Subject: [Bro] Anyone using PF_RING ZC with Bro yet?
Message-ID: <53EC2E0D.4050203@gmail.com>

Hello,

I have a couple new machines to set up and I am curious if anyone has 
upgraded from PF_RING DNA + Libzero to PF_RING ZC for use with Bro and 
what your experience has been? Is it safe or preferred to upgrade to ZC 
or to stick with the DNA/Libzero approach at this time?

Regards,
Gary


From seth at icir.org  Wed Aug 13 20:42:34 2014
From: seth at icir.org (Seth Hall)
Date: Wed, 13 Aug 2014 23:42:34 -0400
Subject: [Bro] Anyone using PF_RING ZC with Bro yet?
In-Reply-To: <53EC2E0D.4050203@gmail.com>
References: <53EC2E0D.4050203@gmail.com>
Message-ID: <9B0898A1-0A49-4DB5-8E19-D43BDCC83686@icir.org>


On Aug 13, 2014, at 11:33 PM, Gary Faulkner <gfaulkner.nsm at gmail.com> wrote:

> I have a couple new machines to set up and I am curious if anyone has 
> upgraded from PF_RING DNA + Libzero to PF_RING ZC for use with Bro and 
> what your experience has been? Is it safe or preferred to upgrade to ZC 
> or to stick with the DNA/Libzero approach at this time?

The PF_Ring plugin in 2.3 should support ZC interfaces from the ZC traffic balancing tool they provide.  One problem with it though is that the new ZC tool only support balancing the traffic to a single tool unlike the DNA load balancing tool which can load balance traffic multiple times out to different tools.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140813/69793d35/attachment.bin 

From gfaulkner.nsm at gmail.com  Wed Aug 13 21:14:25 2014
From: gfaulkner.nsm at gmail.com (Gary Faulkner)
Date: Wed, 13 Aug 2014 23:14:25 -0500
Subject: [Bro] Anyone using PF_RING ZC with Bro yet?
In-Reply-To: <9B0898A1-0A49-4DB5-8E19-D43BDCC83686@icir.org>
References: <53EC2E0D.4050203@gmail.com>
	<9B0898A1-0A49-4DB5-8E19-D43BDCC83686@icir.org>
Message-ID: <53EC37A1.1090401@gmail.com>

Seth,

Thanks for the reply. I remembered you commenting on that and asked 
Alfredo from NTOP if it was supported yet and he indicated that you can 
actually do the multiple app thing now. He also mentioned that the 
daemon mode option isn't implemented within the new script.

For example I asked about doing something like this, but in ZC:

pfdnacluster_master -i dna0,dna1 -d -n 12,1 -c 21


Alfredo indicated I should be able to get similar results with the new 
script like this (excepting no built in -d mode):

zbalance_ipc -i zc:ethX,zc:ethY -n 12,1 -m 1 -c 21

That said I also seem to recall someone else on the bro list having some other issues such as with jumbo frames or missing packets, but don't know if those ever got resolved. ZC is initially tempting because you only need the one ZC license instead of separate licenses for the DNA driver and Libzero, plus not having to go back for ZC later, but that's only helpful if it is working well for people.

Regards,
Gary




On 8/13/2014 10:42 PM, Seth Hall wrote:
> On Aug 13, 2014, at 11:33 PM, Gary Faulkner <gfaulkner.nsm at gmail.com> wrote:
>
>> I have a couple new machines to set up and I am curious if anyone has
>> upgraded from PF_RING DNA + Libzero to PF_RING ZC for use with Bro and
>> what your experience has been? Is it safe or preferred to upgrade to ZC
>> or to stick with the DNA/Libzero approach at this time?
> The PF_Ring plugin in 2.3 should support ZC interfaces from the ZC traffic balancing tool they provide.  One problem with it though is that the new ZC tool only support balancing the traffic to a single tool unlike the DNA load balancing tool which can load balance traffic multiple times out to different tools.
>
>    .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>



From seth at icir.org  Wed Aug 13 22:03:30 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 14 Aug 2014 01:03:30 -0400
Subject: [Bro] Anyone using PF_RING ZC with Bro yet?
In-Reply-To: <53EC37A1.1090401@gmail.com>
References: <53EC2E0D.4050203@gmail.com>
	<9B0898A1-0A49-4DB5-8E19-D43BDCC83686@icir.org>
	<53EC37A1.1090401@gmail.com>
Message-ID: <92EBDCF3-1583-44BF-8028-28AA43062DE7@icir.org>


On Aug 14, 2014, at 12:14 AM, Gary Faulkner <gfaulkner.nsm at gmail.com> wrote:

> zbalance_ipc -i zc:ethX,zc:ethY -n 12,1 -m 1 -c 21

Ah nice!  It's been several months since I've looked at this.  I think that to get this config working in Bro you should be able to use lb_method=pf_ring with interface=zc:21 (in Bro 2.3).

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140814/453c666a/attachment.bin 

From John_Lankau at sra.com  Thu Aug 14 05:57:52 2014
From: John_Lankau at sra.com (Lankau, John)
Date: Thu, 14 Aug 2014 12:57:52 +0000
Subject: [Bro] Quick smtp-url-extraction question
In-Reply-To: <1407455420.2576.5.camel@JamesiMac>
References: <67f16fa2c2516e86a55f15ffb0123616@localhost>
	<D36825AC-89C7-4AF3-B1C2-091D0FE2E151@icir.org>
	<175259a1168f271b0ef7759c72f9bf39@localhost>
	<EB78E1F8-41CD-4051-90C1-497E8905B84F@icir.org>
	<1407455420.2576.5.camel@JamesiMac>
Message-ID: <0464E9BF13BEE74EA662DE35911D64232B9CE62A@SRAexMBX03.sra.com>

Seth,

+100

I just wanted to add that I think that script that logs SMTP URLs would get a lot of use in our environment as well.  It?s been an elusive data point, but one we really would like to have.  We?ve been having high-level discussions on how to implement something that does this exact process in our office, so I?d be very interested in using this script once it?s ready as well.

Thanks!
--John

From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of James Lay
Sent: Thursday, August 07, 2014 7:50 PM
To: bro at bro-ids.org
Subject: Re: [Bro] Quick smtp-url-extraction question

On Thu, 2014-08-07 at 13:39 -0400, Seth Hall wrote:



On Aug 7, 2014, at 1:30 PM, James Lay <jlay at slave-tothe-box.net<mailto:jlay at slave-tothe-box.net>> wrote:



> I would absolutely love a script that would log urls....we all know that quoted-printable and bas364 shenanigans may get missed



Much of that should be handled automatically by the mime analyzer (I'm not sure of the limits of that offhand).



> , but every little bit helps..thanks a bunch Seth.



I'll see if I can get to it soon.



  .Seth



--

Seth Hall

International Computer Science Institute

(Bro) because everyone has a network

http://www.bro.org/



Thanks again Seth.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140814/af424e6c/attachment.html 

From hosom at battelle.org  Thu Aug 14 06:51:30 2014
From: hosom at battelle.org (Hosom, Stephen M)
Date: Thu, 14 Aug 2014 13:51:30 +0000
Subject: [Bro] Quick smtp-url-extraction question
In-Reply-To: <0464E9BF13BEE74EA662DE35911D64232B9CE62A@SRAexMBX03.sra.com>
References: <67f16fa2c2516e86a55f15ffb0123616@localhost>
	<D36825AC-89C7-4AF3-B1C2-091D0FE2E151@icir.org>
	<175259a1168f271b0ef7759c72f9bf39@localhost>
	<EB78E1F8-41CD-4051-90C1-497E8905B84F@icir.org>
	<1407455420.2576.5.camel@JamesiMac>
	<0464E9BF13BEE74EA662DE35911D64232B9CE62A@SRAexMBX03.sra.com>
Message-ID: <E5A2071C6FFFF640B714C224139C9B3310C0EB@WP-MBX2.milky-way.battelle.org>

All,

I submitted a pull request last week for this. You could technically grab the script and run it. Since I?m not part of the Bro team though, I can?t promise that this will continue to work.

https://github.com/bro/bro/pull/10

I run a variation of this script in my production environment right now. Keep in mind that it is normally a bad plan to extend an internal Bro module. Since there?s a pretty high demand for it, if you?d like to modify this to not extend the internal SMTP modules and be separate, it is a relatively short task (about 15 minutes).

Lastly, this is provided as-is with no warranty, etc. etc.

Thanks,

Stephen

From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Lankau, John
Sent: Thursday, August 14, 2014 8:58 AM
To: James Lay; bro at bro-ids.org
Subject: Re: [Bro] Quick smtp-url-extraction question

Seth,

+100

I just wanted to add that I think that script that logs SMTP URLs would get a lot of use in our environment as well.  It?s been an elusive data point, but one we really would like to have.  We?ve been having high-level discussions on how to implement something that does this exact process in our office, so I?d be very interested in using this script once it?s ready as well.

Thanks!
--John

From: bro-bounces at bro.org<mailto:bro-bounces at bro.org> [mailto:bro-bounces at bro.org] On Behalf Of James Lay
Sent: Thursday, August 07, 2014 7:50 PM
To: bro at bro-ids.org<mailto:bro at bro-ids.org>
Subject: Re: [Bro] Quick smtp-url-extraction question

On Thu, 2014-08-07 at 13:39 -0400, Seth Hall wrote:



On Aug 7, 2014, at 1:30 PM, James Lay <jlay at slave-tothe-box.net<mailto:jlay at slave-tothe-box.net>> wrote:



> I would absolutely love a script that would log urls....we all know that quoted-printable and bas364 shenanigans may get missed



Much of that should be handled automatically by the mime analyzer (I'm not sure of the limits of that offhand).



> , but every little bit helps..thanks a bunch Seth.



I'll see if I can get to it soon.



  .Seth



--

Seth Hall

International Computer Science Institute

(Bro) because everyone has a network

http://www.bro.org/



Thanks again Seth.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140814/4995f29b/attachment.html 

From asharma at lbl.gov  Thu Aug 14 07:30:52 2014
From: asharma at lbl.gov (Aashish Sharma)
Date: Thu, 14 Aug 2014 07:30:52 -0700
Subject: [Bro] Quick smtp-url-extraction question
In-Reply-To: <E5A2071C6FFFF640B714C224139C9B3310C0EB@WP-MBX2.milky-way.battelle.org>
References: <67f16fa2c2516e86a55f15ffb0123616@localhost>
	<D36825AC-89C7-4AF3-B1C2-091D0FE2E151@icir.org>
	<175259a1168f271b0ef7759c72f9bf39@localhost>
	<EB78E1F8-41CD-4051-90C1-497E8905B84F@icir.org>
	<1407455420.2576.5.camel@JamesiMac>
	<0464E9BF13BEE74EA662DE35911D64232B9CE62A@SRAexMBX03.sra.com>
	<E5A2071C6FFFF640B714C224139C9B3310C0EB@WP-MBX2.milky-way.battelle.org>
Message-ID: <20140814143050.GB2497@yaksha.lbl.gov>

OK. Here is smtp-url-extraction scripts attached with this email. I apologize for the delays in sending. 

These scripts have been running for >  1 1/2 years so I can say they are fairly stable and should not cause any issues. 

1) Please configure site.bro (attached) as per your site specifics and add it to your site/local.bro file.

2) If you are running bro-2.2 or below please use: smtp-url-extraction.bro 

3) if you are running bro-2.3, use smtp-url-extraction-bloom.bro - it uses bloom filters to check against URL's in the http stream. So its less taxing on memory compared to (2). 

This script should log urls embedded in smtp traffic into a file called smtpurl_links.log. Also there are configuration variables such as suspicious_text_in_url, suspicious_text_in_body etc. You can look into smtp-embedded-url.bro (and -bloom.bro) to see kinds of notices it would generate. 

This script is part of a bigger smtp suite. I will try to collect other scripts and send those out as well. 

Please let me know if you have any questions or have issues running these scripts. 

Thanks, 
Aashish 
LBNL 

On Thu, Aug 14, 2014 at 01:51:30PM +0000, Hosom, Stephen M wrote:
> 
>    All,
> 
> 
>    I submitted a pull request last week for this. You could technically grab
>    the script and run it. Since I?m not part of the Bro team though, I can?t
>    promise that this will continue to work.
> 
> 
>    [1]https://github.com/bro/bro/pull/10
> 
> 
>    I run a variation of this script in my production environment right now.
>    Keep  in mind that it is normally a bad plan to extend an internal Bro
>    module. Since there?s a pretty high demand for it, if you?d like to modify
>    this  to not extend the internal SMTP modules and be separate, it is a
>    relatively short task (about 15 minutes).
> 
> 
>    Lastly, this is provided as-is with no warranty, etc. etc.
> 
> 
>    Thanks,
> 
>    Stephen
> 
> 
>    From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Lankau,
>    John
>    Sent: Thursday, August 14, 2014 8:58 AM
>    To: James Lay; bro at bro-ids.org
>    Subject: Re: [Bro] Quick smtp-url-extraction question
> 
> 
>    Seth,
> 
> 
>    +100
> 
> 
>    I just wanted to add that I think that script that logs SMTP URLs would get
>    a lot of use in our environment as well.  It?s been an elusive data point,
>    but  one  we  really would like to have.  We?ve been having high-level
>    discussions on how to implement something that does this exact process in
>    our office, so I?d be very interested in using this script once it?s ready
>    as well.
> 
> 
>    Thanks!
> 
>    --John
> 
> 
>    From: [2]bro-bounces at bro.org [[3]mailto:bro-bounces at bro.org] On Behalf Of
>    James Lay
>    Sent: Thursday, August 07, 2014 7:50 PM
>    To: [4]bro at bro-ids.org
>    Subject: Re: [Bro] Quick smtp-url-extraction question
> 
> 
>    On Thu, 2014-08-07 at 13:39 -0400, Seth Hall wrote:
> 
> On Aug 7, 2014, at 1:30 PM, James Lay <[5]jlay at slave-tothe-box.net> wrote:
> 
> > I would absolutely love a script that would log urls....we all know that quot
> ed-printable and bas364 shenanigans may get missed
> 
> Much of that should be handled automatically by the mime analyzer (I'm not sure
>  of the limits of that offhand).
> 
> > , but every little bit helps..thanks a bunch Seth.
> 
> I'll see if I can get to it soon.
> 
>   .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> [6]http://www.bro.org/
> 
> 
>    Thanks again Seth.
>    James
> 
> References
> 
>    1. https://github.com/bro/bro/pull/10
>    2. mailto:bro-bounces at bro.org
>    3. mailto:bro-bounces at bro.org
>    4. mailto:bro at bro-ids.org
>    5. mailto:jlay at slave-tothe-box.net
>    6. http://www.bro.org/

> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


-- 
Aashish Sharma	(asharma at lbl.gov) 				 
Cyber Security, 
Lawrence Berkeley National Laboratory  
http://go.lbl.gov/pgp-aashish 
Office: (510)-495-2680  Cell: (510)-612-7971
-------------- next part --------------
module SMTPurl;

export {

	    redef enum Log::ID += { Links_LOG };

	    type Info: record {
                ## When the email was seen.
                ts:   time    &log;
                ## Unique ID for the connection.
                uid:  string  &log;
                ## Connection details.
                id:   conn_id &log;
                ## url that was discovered.
		host: string &log &optional ; 
                url:  string  &log &optional;

        };


        redef enum Notice::Type += {
                ## Indicates that an MD5 sum was calculated for a MIME message.
                SMTP_Embeded_Malicious_URL,
		SMTP_Link_in_EMAIL_Clicked, 
		SMTP_Link_REFERRER_Clicked, 
		SMTP_Linked_BINARY_Download, 
		SMTP_Dotted_URL, 	
		SMTP_Suspicious_File_URL, 
		SMTP_Suspicious_Embedded_Text, 
		SMTP_WatchedFileType, 
		SMTP_Click_Here_Seen
	}; 
        

#		global url_dotted_pattern: pattern = /href.*http:\/\/([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}.*\"/ ; 
		global url_dotted_pattern: pattern = /([^"#]+)/; 

		const url_regex = /^([a-zA-Z\-]{3,5})(:\/\/[^\/?#"'\r\n><]*)([^?#"'\r\n><]*)([^[:blank:]\r\n"'><]*|\??[^"'\r\n><]*)/ &redef;

		global mail_links: table [string] of string &synchronized &create_expire=12 hrs &redef ; 
		global link_already_seen: set[string] &redef ; 
		global referrer_link_already_seen: set[string] ; 
		
		const suspicious_file_types: pattern = /\.rar$|\.exe$|\.zip$/ &redef; 
		const ignore_file_types: pattern = /\.gif$|\.png$|\.jpg$|\.xml$|\.PNG$|\.jpeg$|\.css$/ &redef; 

		redef link_already_seen += { "example.com", }; 
		
		const ignore_mail_originators: set[subnet] += { 1.2.3.4/24} &redef; 
		
		const ignore_mailfroms : pattern += /bro@|alerts/ &redef ; 
		const ignore_mails_to: set[string] = {"reports at example.com", } &redef ; 
		const ignore_site_links: pattern = /http:\/\/.*\.example\.gov\/|http:\/\/.*\.example\.net/ &redef ; 

		
		const suspicious_text_in_url = /googledoc|googledocs|wrait\.ru|webs\.com|jimdo\.com|yolasite\.com\// &redef ; 
		const suspicious_text_in_body = /[Pp][Ee][Rr][Ss][Oo][Nn][Aa][Ll] [Ee][Mm][Aa][Ll]|[Pp][Aa][Ss][Ss][Ww][Oo][Rr][Dd]|[Uu][Ss][Ee][Rr] [Nn][Aa][Mm][Ee]|[Uu][Ss][Ee][Rr][Nn][Aa][Mm][Ee]/ &redef ; 

	#redef Notice::policy += {
		  #[$pred(n: Notice::Info) = { return n$note == SMTPurl::SMTP_Embeded_Malicious_URL; }, $action = Notice::ACTION_EMAIL],  
		  #####[$pred(n: Notice::Info) = { return n$note == SMTPurl::SMTP_Click_Here_Seen; }, $action = Notice::ACTION_EMAIL],   ## too many false +ve
	#} ; 
} 

redef record connection += {
        smtp_url: Info &optional;
};


event bro_init() &priority=5
{
        Log::create_stream(SMTPurl::Links_LOG, [$columns=Info]);

} 


function extract_host(name: string): string
{
        local split_on_slash = split(name, /\//);
        local num_slash = |split_on_slash|;

## ash
        return split_on_slash[3];
}



## Extracts URLs discovered in arbitrary text.
function find_all_urls(s: string): string_set
    {
    return find_all(s, url_regex);
    }


## Extracts URLs discovered in arbitrary text without
## the URL scheme included.
function find_all_urls_without_scheme(s: string): string_set
{
	local urls = find_all_urls(s);
	local return_urls: set[string] = set();
	for ( url in urls )
		{
		local no_scheme = sub(url, /^([a-zA-Z\-]{3,5})(:\/\/)/, "");
		add return_urls[no_scheme];
		}

	return return_urls;
}



function log_smtp_urls(c:connection, url:string)
{
		local info: Info; 

		info$ts = c$smtp$ts;
               	info$uid = c$smtp$uid ;
                info$id = c$id ;
               	info$url = url;
		info$host = extract_host(url) ;  

              	c$smtp_url = info;
               
		Log::write(SMTPurl::Links_LOG, c$smtp_url);

} 


event mime_segment_data(c: connection, length: count, data: string) &priority=-5
{

	if(c$smtp?$mailfrom && ignore_mailfroms  in c$smtp$mailfrom)
	{	
	
		return ; 
	} 

	if (c$smtp?$to) 
	{  
		for (to in c$smtp$to) 
		{ 
			if (to in ignore_mails_to)
                        { 
				return ; 
			} 
		} 
	} 

	if ( ! c?$smtp ) return;

	#if (c$smtp?$to in ignore_mails_to) return ; 
	if (c$id$orig_h in ignore_mail_originators) return; 


	local mail_info:string; 

	if (c$smtp?$to && c$smtp?$subject) { 
                mail_info =  fmt ("uid=%s from=%s to=%s subject=%s", c$smtp$uid, c$smtp$from, c$smtp$to, c$smtp$subject);
        }   
        else { 
		mail_info =  fmt ("uid=%s from=%s", c$smtp$uid, c$smtp$from);
        } 

	local urls = find_all_urls(data) ; 

	for (link in urls){
#		local link =  sub(a,/(http|https):\/\//,"");
		if (link !in mail_links && ignore_file_types !in link )
		  { 
			mail_links[link] = mail_info ; 
			log_smtp_urls(c, link); 
			
			if ( suspicious_file_types in link)
			{ 
				NOTICE([$note=SMTP_WatchedFileType, $msg=fmt("Suspicious filetype embeded in URL %s from  %s", link, c$id$orig_h), $conn=c]); 
			} 
			
			if ( suspicious_text_in_url in link)
			{ 
				NOTICE([$note=SMTP_Embeded_Malicious_URL, $msg=fmt("Suspicious text embeded in URL %s from  %s", link, c$smtp$uid), $conn=c]); 
			} 
			
			if ( suspicious_text_in_body in data && /[Cc][Ll][Ii][Cc][Kk] [Hh][Ee][Rr][Ee]/ in data)
			{ 
				NOTICE([$note=SMTP_Click_Here_Seen, $msg=fmt("Click Here seen in the email %s from  %s", link, c$smtp$uid), $conn=c]); 
			} 

			if (/([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}.*/ in link )
			{ 
				#local url = split_all(data, /href.*\"http:\/\/([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}.*\"/); 
				#url[2]= sub(url[2], /^href=3D\"|href=\"/, "");
				#url[2]= sub(url[2], /\"$/, "");
				NOTICE([$note=SMTP_Dotted_URL, $msg=fmt("Embeded IP in URL %s from  %s", link, c$id$orig_h), $conn=c]);
			} 

		 } ## check link in mail_links 
	} 	## for  
}
 

event http_message_done(c: connection, is_orig: bool, stat: http_message_stat) &priority=-3
{ 
	local str = HTTP::build_url_http(c$http); 

	if (str in SMTPurl::mail_links && str !in SMTPurl::link_already_seen && ignore_file_types !in str && ignore_site_links !in str)
	{ 		
		NOTICE([$note=SMTPurl::SMTP_Link_in_EMAIL_Clicked, $msg=fmt("URL %s [%s]", str, SMTPurl::mail_links[str]), $conn=c]);
		add SMTPurl::link_already_seen[str] ; 
	} 	
	
	if (c$http?$referrer) 
	{ 

	local ref = c$http$referrer; 
		
		if (ref in SMTPurl::mail_links && ref !in SMTPurl::referrer_link_already_seen && ignore_file_types !in ref && ignore_site_links !in ref)
		{  
		fmt("Added %s from %s", SMTPurl::mail_links[ref],  ref); 
		} 
	} 

	## aashish

#                if (c$http?$md5 && str in SMTPurl::mail_links )
#                {
#                	NOTICE([$note=SMTP_Linked_BINARY_Download, $msg=fmt("%s %s %s", c$id$orig_h, c$http$md5, str),
#				$sub=c$http$md5, $conn=c, $URL=str]);
#                }

} 
-------------- next part --------------
module SMTPurl;

export {

	    redef enum Log::ID += { Links_LOG };

	    type Info: record {
                ## When the email was seen.
                ts:   time    &log;
                ## Unique ID for the connection.
                uid:  string  &log;
                ## Connection details.
                id:   conn_id &log;
                ## url that was discovered.
		host: string &log &optional ; 
                url:  string  &log &optional;

        };


        redef enum Notice::Type += {
                ## Indicates that an MD5 sum was calculated for a MIME message.
                SMTP_Embeded_Malicious_URL,
		SMTP_Link_in_EMAIL_Clicked, 
		SMTP_Link_REFERRER_Clicked, 
		SMTP_Linked_BINARY_Download, 
		SMTP_Dotted_URL, 	
		SMTP_Suspicious_File_URL, 
		SMTP_Suspicious_Embedded_Text, 
		SMTP_WatchedFileType, 
		SMTP_Click_Here_Seen
	}; 
        

		global url_dotted_pattern: pattern = /([^"#]+)/; 

		const url_regex = /^([a-zA-Z\-]{3,5})(:\/\/[^\/?#"'\r\n><]*)([^?#"'\r\n><]*)([^[:blank:]\r\n"'><]*|\??[^"'\r\n><]*)/ &redef;


		global mail_links = bloomfilter_basic_init(0.00000001, 10000000) ; 
	
		global link_already_seen: set[string] &redef ; 
		global referrer_link_already_seen: set[string] ; 
		
		const suspicious_file_types: pattern = /\.rar$|\.exe$|\.zip$/ &redef; 
		const ignore_file_types: pattern = /\.gif$|\.png$|\.jpg$|\.xml$|\.PNG$|\.jpeg$|\.css$/ &redef; 

		redef link_already_seen += { "example.net"} ;
                const ignore_mail_originators: set[subnet] += { 1.2.3.4/24, 2.3.4.0/24} &redef;
                const ignore_mailfroms : pattern += /bro@|alerts|reports/ &redef ;
                const ignore_mails_to: set[string] = {"alerts at example.com", "notices at example.com",} &redef ;
                const ignore_site_links: pattern = /http:\/\/.*\.example\.come\/|http:\/\/.*\.example\.net/ &redef ;

		const suspicious_text_in_url = /googledoc|googledocs|wrait\.ru|webs\.com|jimdo\.com|yolasite\.com\// &redef ; 
		const suspicious_text_in_body = /[Pp][Ee][Rr][Ss][Oo][Nn][Aa][Ll] [Ee][Mm][Aa][Ll]|[Pp][Aa][Ss][Ss][Ww][Oo][Rr][Dd]|[Uu][Ss][Ee][Rr] [Nn][Aa][Mm][Ee]|[Uu][Ss][Ee][Rr][Nn][Aa][Mm][Ee]/ &redef ; 

} 

redef record connection += {
        smtp_url: Info &optional;
};


event bro_init() &priority=5
{
        Log::create_stream(SMTPurl::Links_LOG, [$columns=Info]);

} 


function extract_host(name: string): string
{
        local split_on_slash = split(name, /\//);
        local num_slash = |split_on_slash|;

## ash
        return split_on_slash[3];
}



## Extracts URLs discovered in arbitrary text.
function find_all_urls(s: string): string_set
    {
    return find_all(s, url_regex);
    }


## Extracts URLs discovered in arbitrary text without
## the URL scheme included.
function find_all_urls_without_scheme(s: string): string_set
{
	local urls = find_all_urls(s);
	local return_urls: set[string] = set();
	for ( url in urls )
		{
		local no_scheme = sub(url, /^([a-zA-Z\-]{3,5})(:\/\/)/, "");
		add return_urls[no_scheme];
		}

	return return_urls;
}



function log_smtp_urls(c:connection, url:string)
{
		local info: Info; 

		info$ts = c$smtp$ts;
               	info$uid = c$smtp$uid ;
                info$id = c$id ;
               	info$url = url;
		info$host = extract_host(url) ;  

              	c$smtp_url = info;
               
		Log::write(SMTPurl::Links_LOG, c$smtp_url);

} 


event mime_segment_data(c: connection, length: count, data: string) &priority=-5
{

	if(c$smtp?$mailfrom && ignore_mailfroms  in c$smtp$mailfrom)
	{	
		return ; 
	} 

	if (c$smtp?$to) 
	{  
		for (to in c$smtp$to) 
		{ 
			if (to in ignore_mails_to) 
                        { 
				return ; 
			} 
		} 
	} 

	if ( ! c?$smtp ) return;

	#if (c$smtp?$to in ignore_mails_to) return ; 
	if (c$id$orig_h in ignore_mail_originators) return; 


	local mail_info:string; 

	if (c$smtp?$to && c$smtp?$subject) { 
                mail_info =  fmt ("uid=%s from=%s to=%s subject=%s", c$smtp$uid, c$smtp$from, c$smtp$to, c$smtp$subject);
        }   
        else { 
		mail_info =  fmt ("uid=%s from=%s", c$smtp$uid, c$smtp$from);
        } 

	local urls = find_all_urls(data) ; 

	for (link in urls){
#		local link =  sub(a,/(http|https):\/\//,"");


		#local _bf_lookup = bloomfilter_lookup(mail_links, link);

		#if (link !in mail_links && ignore_file_types !in link )
		#if ((_bf_lookup ==  0) && ignore_file_types !in link )
		if ( ignore_file_types !in link )
		  { 
		#	mail_links[link] = mail_info ; 
			
			bloomfilter_add(mail_links, link); 
			log_smtp_urls(c, link); 
			
			if ( suspicious_file_types in link)
			{ 
				NOTICE([$note=SMTP_WatchedFileType, $msg=fmt("Suspicious filetype embeded in URL %s from  %s", link, c$id$orig_h), $conn=c]); 
			} 
			
			if ( suspicious_text_in_url in link)
			{ 
				NOTICE([$note=SMTP_Embeded_Malicious_URL, $msg=fmt("Suspicious text embeded in URL %s from  %s", link, c$smtp$uid), $conn=c]); 
			} 
			
			if ( suspicious_text_in_body in data && /[Cc][Ll][Ii][Cc][Kk] [Hh][Ee][Rr][Ee]/ in data)
			{ 
				NOTICE([$note=SMTP_Click_Here_Seen, $msg=fmt("Click Here seen in the email %s from  %s", link, c$smtp$uid), $conn=c]); 
			} 

			if (/([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}.*/ in link )
			{ 
				#local url = split_all(data, /href.*\"http:\/\/([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}.*\"/); 
				#url[2]= sub(url[2], /^href=3D\"|href=\"/, "");
				#url[2]= sub(url[2], /\"$/, "");
				NOTICE([$note=SMTP_Dotted_URL, $msg=fmt("Embeded IP in URL %s from  %s", link, c$id$orig_h), $conn=c]);
			} 

		 } ## check link in mail_links 
	} 	## for  
}
 

event log_smtp(rec: Info)
{ 
	#print fmt ("log_smtp: INfo: %s", Info); 
} 

#event SMTP::log_mime (rec: SMTP::EntityInfo)
#{
##	print fmt ("log_mine Log_mime: %s", rec); 
#} 

event mime_begin_entity(c: connection) 
{
#print fmt ("mime_begin_entity: %s %s %s %s", c$smtp$from, c$smtp$to, c$smtp$subject, c$smtp$reply_to);
} 

event http_message_done(c: connection, is_orig: bool, stat: http_message_stat) &priority=-3
{ 
	local str = HTTP::build_url_http(c$http); 

	local _bf_lookup_http= bloomfilter_lookup(SMTPurl::mail_links, str); 



	if ((_bf_lookup_http >0) && str !in SMTPurl::link_already_seen && ignore_file_types !in str && ignore_site_links !in str)
	{ 		
		#NOTICE([$note=SMTPurl::SMTP_Link_in_EMAIL_Clicked, $msg=fmt("URL %s [%s]", str, SMTPurl::mail_links[str]), $conn=c]);
	
		NOTICE([$note=SMTPurl::SMTP_Link_in_EMAIL_Clicked, $msg=fmt("URL %s ", str), $conn=c]);
		add SMTPurl::link_already_seen[str] ; 
	} 	
	
	if (c$http?$referrer) 
	{ 

	local ref = c$http$referrer; 
	
		local _bf_lookup_ref = bloomfilter_lookup(SMTPurl::mail_links, ref);		
		#if (ref in SMTPurl::mail_links && ref !in SMTPurl::referrer_link_already_seen && ignore_file_types !in ref && ignore_site_links !in ref)
		if ((_bf_lookup_ref > 0) && ref !in SMTPurl::referrer_link_already_seen && ignore_file_types !in ref && ignore_site_links !in ref)
		{  
		fmt("Added from %s", ref); 
		} 
	} 

	## aashish: need to port to file analysis framework 

#                if (c$http?$md5 && str in SMTPurl::mail_links )
#                {
#                	NOTICE([$note=SMTP_Linked_BINARY_Download, $msg=fmt("%s %s %s", c$id$orig_h, c$http$md5, str),
#				$sub=c$http$md5, $conn=c, $URL=str]);
#		} 	

} 
-------------- next part --------------
@load ./smtp-embedded-url-bloom.bro 

### smtp-embedded-url analysis

## Ignore HTTP tracking if the links from these domains are seen/clicked

redef SMTPurl::link_already_seen += { "example.come","example.org", };
redef SMTPurl::ignore_site_links: pattern = /.*\.example\.com\/|.*\.example\.net/ ;

## Careful: Since Bro watches all the emails (including the alerts it sends, this
## can create an Email storm because an alert including a malicious URL can cause another alert email
## ignore email going to these addresses.

redef SMTPurl::ignore_mails_to: set[string] = {"bro-alerts at example.com", "alerts at example.com", "reports at example.com"}; 

# Ignore emails from the following sender
redef SMTPurl::ignore_mailfroms += /bro@|alerts@|security@|reports/;

### Ignore emails originating from these subnets
## For IP address please use x.y.w.z/32

redef SMTPurl::ignore_mail_originators: set[subnet] += { 1.2.3.4/24, 1.2.3.5/24, } &redef;

### ignore further processing on the following file types embedded in the url - too much volume not useful dataset
redef SMTPurl::ignore_file_types: pattern = /\.gif$|\.png$|\.jpg$|\.xml$|\.PNG$|\.jpeg$|\.css$/ ;

## alert on these file types: generates SMTP_WatchedFileType
redef SMTPurl::suspicious_file_types: pattern = /\.doc$|\.docx|\.xlsx|\.xls|\.rar$|\.exe$|\.zip$/ ;


### Alert on text in URI : generates SMTP_Embeded_Malicious_URL
redef SMTPurl::suspicious_text_in_url = /googledoc|googledocs|ph\.ly\/|webs\.com\/|jimdo\.com/ &redef ;
#redef SMTPurl::suspicious_text_in_url = /googledoc|googledocs|ph\.ly\/|webs\.com\/|jimdo\.com|http(s)?:\/\/.*\/.*(\.edu|\.gov|\.com).*/ &redef ;

## Alert on the text in the body of the message: generates
redef SMTPurl::suspicious_text_in_body = /[Pp][Ee][Rr][Ss][Oo][Nn][Aa][Ll] [Ee][Mm][Aa][Ll]|[Pp][Aa][Ss][Ss][Ww][Oo][Rr][Dd]|[Uu][Ss][Ee][Rr] [Nn][Aa][Mm][Ee]|[Uu][Ss][Ee][Rr][Nn][Aa][Mm][Ee]/ &redef ;
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140814/51a7444a/attachment.bin 

From seth at icir.org  Thu Aug 14 08:31:08 2014
From: seth at icir.org (Seth Hall)
Date: Thu, 14 Aug 2014 11:31:08 -0400
Subject: [Bro] Quick smtp-url-extraction question
In-Reply-To: <20140814143050.GB2497@yaksha.lbl.gov>
References: <67f16fa2c2516e86a55f15ffb0123616@localhost>
	<D36825AC-89C7-4AF3-B1C2-091D0FE2E151@icir.org>
	<175259a1168f271b0ef7759c72f9bf39@localhost>
	<EB78E1F8-41CD-4051-90C1-497E8905B84F@icir.org>
	<1407455420.2576.5.camel@JamesiMac>
	<0464E9BF13BEE74EA662DE35911D64232B9CE62A@SRAexMBX03.sra.com>
	<E5A2071C6FFFF640B714C224139C9B3310C0EB@WP-MBX2.milky-way.battelle.org>
	<20140814143050.GB2497@yaksha.lbl.gov>
Message-ID: <FAC7D8FA-E172-4FFA-B06F-C6FA6B964808@icir.org>


On Aug 14, 2014, at 10:30 AM, Aashish Sharma <asharma at lbl.gov> wrote:

> 3) if you are running bro-2.3, use smtp-url-extraction-bloom.bro - it uses bloom filters to check against URL's in the http stream. So its less taxing on memory compared to (2). 

Thanks, Aashish.

I've been working on this script for a while this morning just doing general clean up and documentation.  Right now I'm getting ready to add cluster support to it.  I'll 

Stephen, cool if I close your pull request since I think that Aashish's script has more functionality?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140814/8e577d2a/attachment.bin 

From liburdi.joshua at gmail.com  Thu Aug 14 09:06:35 2014
From: liburdi.joshua at gmail.com (Josh Liburdi)
Date: Thu, 14 Aug 2014 09:06:35 -0700
Subject: [Bro] Quick smtp-url-extraction question
In-Reply-To: <20140814143050.GB2497@yaksha.lbl.gov>
References: <67f16fa2c2516e86a55f15ffb0123616@localhost>
	<D36825AC-89C7-4AF3-B1C2-091D0FE2E151@icir.org>
	<175259a1168f271b0ef7759c72f9bf39@localhost>
	<EB78E1F8-41CD-4051-90C1-497E8905B84F@icir.org>
	<1407455420.2576.5.camel@JamesiMac>
	<0464E9BF13BEE74EA662DE35911D64232B9CE62A@SRAexMBX03.sra.com>
	<E5A2071C6FFFF640B714C224139C9B3310C0EB@WP-MBX2.milky-way.battelle.org>
	<20140814143050.GB2497@yaksha.lbl.gov>
Message-ID: <CANrCRiGqQAj2jBPfG=YOrSkLaUy6=mkpxDrdHu-EkyzeLsWzZQ@mail.gmail.com>

Aashish,

I'm curious why you suggested only using the bloom filter version of
this script in Bro 2.3-- is there a reason one wouldn't want to use it
in Bro 2.2?

Josh

On Thu, Aug 14, 2014 at 7:30 AM, Aashish Sharma <asharma at lbl.gov> wrote:
> OK. Here is smtp-url-extraction scripts attached with this email. I apologize for the delays in sending.
>
> These scripts have been running for >  1 1/2 years so I can say they are fairly stable and should not cause any issues.
>
> 1) Please configure site.bro (attached) as per your site specifics and add it to your site/local.bro file.
>
> 2) If you are running bro-2.2 or below please use: smtp-url-extraction.bro
>
> 3) if you are running bro-2.3, use smtp-url-extraction-bloom.bro - it uses bloom filters to check against URL's in the http stream. So its less taxing on memory compared to (2).
>
> This script should log urls embedded in smtp traffic into a file called smtpurl_links.log. Also there are configuration variables such as suspicious_text_in_url, suspicious_text_in_body etc. You can look into smtp-embedded-url.bro (and -bloom.bro) to see kinds of notices it would generate.
>
> This script is part of a bigger smtp suite. I will try to collect other scripts and send those out as well.
>
> Please let me know if you have any questions or have issues running these scripts.
>
> Thanks,
> Aashish
> LBNL
>
> On Thu, Aug 14, 2014 at 01:51:30PM +0000, Hosom, Stephen M wrote:
>>
>>    All,
>>
>>
>>    I submitted a pull request last week for this. You could technically grab
>>    the script and run it. Since I?m not part of the Bro team though, I can?t
>>    promise that this will continue to work.
>>
>>
>>    [1]https://github.com/bro/bro/pull/10
>>
>>
>>    I run a variation of this script in my production environment right now.
>>    Keep  in mind that it is normally a bad plan to extend an internal Bro
>>    module. Since there?s a pretty high demand for it, if you?d like to modify
>>    this  to not extend the internal SMTP modules and be separate, it is a
>>    relatively short task (about 15 minutes).
>>
>>
>>    Lastly, this is provided as-is with no warranty, etc. etc.
>>
>>
>>    Thanks,
>>
>>    Stephen
>>
>>
>>    From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Lankau,
>>    John
>>    Sent: Thursday, August 14, 2014 8:58 AM
>>    To: James Lay; bro at bro-ids.org
>>    Subject: Re: [Bro] Quick smtp-url-extraction question
>>
>>
>>    Seth,
>>
>>
>>    +100
>>
>>
>>    I just wanted to add that I think that script that logs SMTP URLs would get
>>    a lot of use in our environment as well.  It?s been an elusive data point,
>>    but  one  we  really would like to have.  We?ve been having high-level
>>    discussions on how to implement something that does this exact process in
>>    our office, so I?d be very interested in using this script once it?s ready
>>    as well.
>>
>>
>>    Thanks!
>>
>>    --John
>>
>>
>>    From: [2]bro-bounces at bro.org [[3]mailto:bro-bounces at bro.org] On Behalf Of
>>    James Lay
>>    Sent: Thursday, August 07, 2014 7:50 PM
>>    To: [4]bro at bro-ids.org
>>    Subject: Re: [Bro] Quick smtp-url-extraction question
>>
>>
>>    On Thu, 2014-08-07 at 13:39 -0400, Seth Hall wrote:
>>
>> On Aug 7, 2014, at 1:30 PM, James Lay <[5]jlay at slave-tothe-box.net> wrote:
>>
>> > I would absolutely love a script that would log urls....we all know that quot
>> ed-printable and bas364 shenanigans may get missed
>>
>> Much of that should be handled automatically by the mime analyzer (I'm not sure
>>  of the limits of that offhand).
>>
>> > , but every little bit helps..thanks a bunch Seth.
>>
>> I'll see if I can get to it soon.
>>
>>   .Seth
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> [6]http://www.bro.org/
>>
>>
>>    Thanks again Seth.
>>    James
>>
>> References
>>
>>    1. https://github.com/bro/bro/pull/10
>>    2. mailto:bro-bounces at bro.org
>>    3. mailto:bro-bounces at bro.org
>>    4. mailto:bro at bro-ids.org
>>    5. mailto:jlay at slave-tothe-box.net
>>    6. http://www.bro.org/
>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> --
> Aashish Sharma  (asharma at lbl.gov)
> Cyber Security,
> Lawrence Berkeley National Laboratory
> http://go.lbl.gov/pgp-aashish
> Office: (510)-495-2680  Cell: (510)-612-7971
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From asharma at lbl.gov  Thu Aug 14 09:30:26 2014
From: asharma at lbl.gov (Aashish Sharma)
Date: Thu, 14 Aug 2014 09:30:26 -0700
Subject: [Bro] Quick smtp-url-extraction question
In-Reply-To: <CANrCRiGqQAj2jBPfG=YOrSkLaUy6=mkpxDrdHu-EkyzeLsWzZQ@mail.gmail.com>
References: <67f16fa2c2516e86a55f15ffb0123616@localhost>
	<D36825AC-89C7-4AF3-B1C2-091D0FE2E151@icir.org>
	<175259a1168f271b0ef7759c72f9bf39@localhost>
	<EB78E1F8-41CD-4051-90C1-497E8905B84F@icir.org>
	<1407455420.2576.5.camel@JamesiMac>
	<0464E9BF13BEE74EA662DE35911D64232B9CE62A@SRAexMBX03.sra.com>
	<E5A2071C6FFFF640B714C224139C9B3310C0EB@WP-MBX2.milky-way.battelle.org>
	<20140814143050.GB2497@yaksha.lbl.gov>
	<CANrCRiGqQAj2jBPfG=YOrSkLaUy6=mkpxDrdHu-EkyzeLsWzZQ@mail.gmail.com>
Message-ID: <20140814163024.GD2497@yaksha.lbl.gov>

Bloomfilter code in bro-2.2 or below has had some hash collision issues. 

Matthias's fixes became part of bro-2.3 release (from CHANGE log): 

- Switch to double hashing for Bloomfilters for better performance.  (Matthias Vallentin)
- Bugfix to use full digest length instead of just one byte for Bloomfilter's universal hash function. Addresses BIT-1140.  (Matthias Vallentin)

Please see: https://bro-tracker.atlassian.net/browse/BIT-1140

If you run smtp-embedded-url-bloom.bro in bro-2.2 world, You will see a huge number of false positives for "SMTP_Link_in_EMAIL_Clicked" 

smtp-embedded-url.bro has exact same functionality, except that it maintains a table of smtp urls and checks http requests against it. So less efficient on memory. I expire the contents of the table in 12hours thus a little limited on visibility too. But still I'd say the code works quite alright, so if you cannot quite immidiately upgrade to bro-2.3, feel free to use: smtp-embedded-url.bro script. 

Hope this helps, 
Aashish 


On Thu, Aug 14, 2014 at 09:06:35AM -0700, Josh Liburdi wrote:
> Aashish,
> 
> I'm curious why you suggested only using the bloom filter version of
> this script in Bro 2.3-- is there a reason one wouldn't want to use it
> in Bro 2.2?
> 
> Josh
> 
> On Thu, Aug 14, 2014 at 7:30 AM, Aashish Sharma <asharma at lbl.gov> wrote:
> > OK. Here is smtp-url-extraction scripts attached with this email. I apologize for the delays in sending.
> >
> > These scripts have been running for >  1 1/2 years so I can say they are fairly stable and should not cause any issues.
> >
> > 1) Please configure site.bro (attached) as per your site specifics and add it to your site/local.bro file.
> >
> > 2) If you are running bro-2.2 or below please use: smtp-url-extraction.bro
> >
> > 3) if you are running bro-2.3, use smtp-url-extraction-bloom.bro - it uses bloom filters to check against URL's in the http stream. So its less taxing on memory compared to (2).
> >
> > This script should log urls embedded in smtp traffic into a file called smtpurl_links.log. Also there are configuration variables such as suspicious_text_in_url, suspicious_text_in_body etc. You can look into smtp-embedded-url.bro (and -bloom.bro) to see kinds of notices it would generate.
> >
> > This script is part of a bigger smtp suite. I will try to collect other scripts and send those out as well.
> >
> > Please let me know if you have any questions or have issues running these scripts.
> >
> > Thanks,
> > Aashish
> > LBNL
> >
> > On Thu, Aug 14, 2014 at 01:51:30PM +0000, Hosom, Stephen M wrote:
> >>
> >>    All,
> >>
> >>
> >>    I submitted a pull request last week for this. You could technically grab
> >>    the script and run it. Since I?m not part of the Bro team though, I can?t
> >>    promise that this will continue to work.
> >>
> >>
> >>    [1]https://github.com/bro/bro/pull/10
> >>
> >>
> >>    I run a variation of this script in my production environment right now.
> >>    Keep  in mind that it is normally a bad plan to extend an internal Bro
> >>    module. Since there?s a pretty high demand for it, if you?d like to modify
> >>    this  to not extend the internal SMTP modules and be separate, it is a
> >>    relatively short task (about 15 minutes).
> >>
> >>
> >>    Lastly, this is provided as-is with no warranty, etc. etc.
> >>
> >>
> >>    Thanks,
> >>
> >>    Stephen
> >>
> >>
> >>    From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Lankau,
> >>    John
> >>    Sent: Thursday, August 14, 2014 8:58 AM
> >>    To: James Lay; bro at bro-ids.org
> >>    Subject: Re: [Bro] Quick smtp-url-extraction question
> >>
> >>
> >>    Seth,
> >>
> >>
> >>    +100
> >>
> >>
> >>    I just wanted to add that I think that script that logs SMTP URLs would get
> >>    a lot of use in our environment as well.  It?s been an elusive data point,
> >>    but  one  we  really would like to have.  We?ve been having high-level
> >>    discussions on how to implement something that does this exact process in
> >>    our office, so I?d be very interested in using this script once it?s ready
> >>    as well.
> >>
> >>
> >>    Thanks!
> >>
> >>    --John
> >>
> >>
> >>    From: [2]bro-bounces at bro.org [[3]mailto:bro-bounces at bro.org] On Behalf Of
> >>    James Lay
> >>    Sent: Thursday, August 07, 2014 7:50 PM
> >>    To: [4]bro at bro-ids.org
> >>    Subject: Re: [Bro] Quick smtp-url-extraction question
> >>
> >>
> >>    On Thu, 2014-08-07 at 13:39 -0400, Seth Hall wrote:
> >>
> >> On Aug 7, 2014, at 1:30 PM, James Lay <[5]jlay at slave-tothe-box.net> wrote:
> >>
> >> > I would absolutely love a script that would log urls....we all know that quot
> >> ed-printable and bas364 shenanigans may get missed
> >>
> >> Much of that should be handled automatically by the mime analyzer (I'm not sure
> >>  of the limits of that offhand).
> >>
> >> > , but every little bit helps..thanks a bunch Seth.
> >>
> >> I'll see if I can get to it soon.
> >>
> >>   .Seth
> >>
> >> --
> >> Seth Hall
> >> International Computer Science Institute
> >> (Bro) because everyone has a network
> >> [6]http://www.bro.org/
> >>
> >>
> >>    Thanks again Seth.
> >>    James
> >>
> >> References
> >>
> >>    1. https://github.com/bro/bro/pull/10
> >>    2. mailto:bro-bounces at bro.org
> >>    3. mailto:bro-bounces at bro.org
> >>    4. mailto:bro at bro-ids.org
> >>    5. mailto:jlay at slave-tothe-box.net
> >>    6. http://www.bro.org/
> >
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> >
> > --
> > Aashish Sharma  (asharma at lbl.gov)
> > Cyber Security,
> > Lawrence Berkeley National Laboratory
> > http://go.lbl.gov/pgp-aashish
> > Office: (510)-495-2680  Cell: (510)-612-7971
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-- 
Aashish Sharma	(asharma at lbl.gov) 				 
Cyber Security, 
Lawrence Berkeley National Laboratory  
http://go.lbl.gov/pgp-aashish 
Office: (510)-495-2680  Cell: (510)-612-7971
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140814/74f518db/attachment.bin 

From liburdi.joshua at gmail.com  Thu Aug 14 09:41:59 2014
From: liburdi.joshua at gmail.com (Josh Liburdi)
Date: Thu, 14 Aug 2014 09:41:59 -0700
Subject: [Bro] Quick smtp-url-extraction question
In-Reply-To: <20140814163024.GD2497@yaksha.lbl.gov>
References: <67f16fa2c2516e86a55f15ffb0123616@localhost>
	<D36825AC-89C7-4AF3-B1C2-091D0FE2E151@icir.org>
	<175259a1168f271b0ef7759c72f9bf39@localhost>
	<EB78E1F8-41CD-4051-90C1-497E8905B84F@icir.org>
	<1407455420.2576.5.camel@JamesiMac>
	<0464E9BF13BEE74EA662DE35911D64232B9CE62A@SRAexMBX03.sra.com>
	<E5A2071C6FFFF640B714C224139C9B3310C0EB@WP-MBX2.milky-way.battelle.org>
	<20140814143050.GB2497@yaksha.lbl.gov>
	<CANrCRiGqQAj2jBPfG=YOrSkLaUy6=mkpxDrdHu-EkyzeLsWzZQ@mail.gmail.com>
	<20140814163024.GD2497@yaksha.lbl.gov>
Message-ID: <CANrCRiH7rstYgSCm+znJq4U=s9rwMV0DZcXNzqm601eb7qfimA@mail.gmail.com>

I didn't realize there were hash collision issues with 2.2 and below--
thanks for the info!

Josh

On Thu, Aug 14, 2014 at 9:30 AM, Aashish Sharma <asharma at lbl.gov> wrote:
> Bloomfilter code in bro-2.2 or below has had some hash collision issues.
>
> Matthias's fixes became part of bro-2.3 release (from CHANGE log):
>
> - Switch to double hashing for Bloomfilters for better performance.  (Matthias Vallentin)
> - Bugfix to use full digest length instead of just one byte for Bloomfilter's universal hash function. Addresses BIT-1140.  (Matthias Vallentin)
>
> Please see: https://bro-tracker.atlassian.net/browse/BIT-1140
>
> If you run smtp-embedded-url-bloom.bro in bro-2.2 world, You will see a huge number of false positives for "SMTP_Link_in_EMAIL_Clicked"
>
> smtp-embedded-url.bro has exact same functionality, except that it maintains a table of smtp urls and checks http requests against it. So less efficient on memory. I expire the contents of the table in 12hours thus a little limited on visibility too. But still I'd say the code works quite alright, so if you cannot quite immidiately upgrade to bro-2.3, feel free to use: smtp-embedded-url.bro script.
>
> Hope this helps,
> Aashish
>
>
> On Thu, Aug 14, 2014 at 09:06:35AM -0700, Josh Liburdi wrote:
>> Aashish,
>>
>> I'm curious why you suggested only using the bloom filter version of
>> this script in Bro 2.3-- is there a reason one wouldn't want to use it
>> in Bro 2.2?
>>
>> Josh
>>
>> On Thu, Aug 14, 2014 at 7:30 AM, Aashish Sharma <asharma at lbl.gov> wrote:
>> > OK. Here is smtp-url-extraction scripts attached with this email. I apologize for the delays in sending.
>> >
>> > These scripts have been running for >  1 1/2 years so I can say they are fairly stable and should not cause any issues.
>> >
>> > 1) Please configure site.bro (attached) as per your site specifics and add it to your site/local.bro file.
>> >
>> > 2) If you are running bro-2.2 or below please use: smtp-url-extraction.bro
>> >
>> > 3) if you are running bro-2.3, use smtp-url-extraction-bloom.bro - it uses bloom filters to check against URL's in the http stream. So its less taxing on memory compared to (2).
>> >
>> > This script should log urls embedded in smtp traffic into a file called smtpurl_links.log. Also there are configuration variables such as suspicious_text_in_url, suspicious_text_in_body etc. You can look into smtp-embedded-url.bro (and -bloom.bro) to see kinds of notices it would generate.
>> >
>> > This script is part of a bigger smtp suite. I will try to collect other scripts and send those out as well.
>> >
>> > Please let me know if you have any questions or have issues running these scripts.
>> >
>> > Thanks,
>> > Aashish
>> > LBNL
>> >
>> > On Thu, Aug 14, 2014 at 01:51:30PM +0000, Hosom, Stephen M wrote:
>> >>
>> >>    All,
>> >>
>> >>
>> >>    I submitted a pull request last week for this. You could technically grab
>> >>    the script and run it. Since I?m not part of the Bro team though, I can?t
>> >>    promise that this will continue to work.
>> >>
>> >>
>> >>    [1]https://github.com/bro/bro/pull/10
>> >>
>> >>
>> >>    I run a variation of this script in my production environment right now.
>> >>    Keep  in mind that it is normally a bad plan to extend an internal Bro
>> >>    module. Since there?s a pretty high demand for it, if you?d like to modify
>> >>    this  to not extend the internal SMTP modules and be separate, it is a
>> >>    relatively short task (about 15 minutes).
>> >>
>> >>
>> >>    Lastly, this is provided as-is with no warranty, etc. etc.
>> >>
>> >>
>> >>    Thanks,
>> >>
>> >>    Stephen
>> >>
>> >>
>> >>    From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Lankau,
>> >>    John
>> >>    Sent: Thursday, August 14, 2014 8:58 AM
>> >>    To: James Lay; bro at bro-ids.org
>> >>    Subject: Re: [Bro] Quick smtp-url-extraction question
>> >>
>> >>
>> >>    Seth,
>> >>
>> >>
>> >>    +100
>> >>
>> >>
>> >>    I just wanted to add that I think that script that logs SMTP URLs would get
>> >>    a lot of use in our environment as well.  It?s been an elusive data point,
>> >>    but  one  we  really would like to have.  We?ve been having high-level
>> >>    discussions on how to implement something that does this exact process in
>> >>    our office, so I?d be very interested in using this script once it?s ready
>> >>    as well.
>> >>
>> >>
>> >>    Thanks!
>> >>
>> >>    --John
>> >>
>> >>
>> >>    From: [2]bro-bounces at bro.org [[3]mailto:bro-bounces at bro.org] On Behalf Of
>> >>    James Lay
>> >>    Sent: Thursday, August 07, 2014 7:50 PM
>> >>    To: [4]bro at bro-ids.org
>> >>    Subject: Re: [Bro] Quick smtp-url-extraction question
>> >>
>> >>
>> >>    On Thu, 2014-08-07 at 13:39 -0400, Seth Hall wrote:
>> >>
>> >> On Aug 7, 2014, at 1:30 PM, James Lay <[5]jlay at slave-tothe-box.net> wrote:
>> >>
>> >> > I would absolutely love a script that would log urls....we all know that quot
>> >> ed-printable and bas364 shenanigans may get missed
>> >>
>> >> Much of that should be handled automatically by the mime analyzer (I'm not sure
>> >>  of the limits of that offhand).
>> >>
>> >> > , but every little bit helps..thanks a bunch Seth.
>> >>
>> >> I'll see if I can get to it soon.
>> >>
>> >>   .Seth
>> >>
>> >> --
>> >> Seth Hall
>> >> International Computer Science Institute
>> >> (Bro) because everyone has a network
>> >> [6]http://www.bro.org/
>> >>
>> >>
>> >>    Thanks again Seth.
>> >>    James
>> >>
>> >> References
>> >>
>> >>    1. https://github.com/bro/bro/pull/10
>> >>    2. mailto:bro-bounces at bro.org
>> >>    3. mailto:bro-bounces at bro.org
>> >>    4. mailto:bro at bro-ids.org
>> >>    5. mailto:jlay at slave-tothe-box.net
>> >>    6. http://www.bro.org/
>> >
>> >> _______________________________________________
>> >> Bro mailing list
>> >> bro at bro-ids.org
>> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> >
>> >
>> > --
>> > Aashish Sharma  (asharma at lbl.gov)
>> > Cyber Security,
>> > Lawrence Berkeley National Laboratory
>> > http://go.lbl.gov/pgp-aashish
>> > Office: (510)-495-2680  Cell: (510)-612-7971
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> --
> Aashish Sharma  (asharma at lbl.gov)
> Cyber Security,
> Lawrence Berkeley National Laboratory
> http://go.lbl.gov/pgp-aashish
> Office: (510)-495-2680  Cell: (510)-612-7971



From hosom at battelle.org  Thu Aug 14 09:58:44 2014
From: hosom at battelle.org (Hosom, Stephen M)
Date: Thu, 14 Aug 2014 16:58:44 +0000
Subject: [Bro] Quick smtp-url-extraction question
In-Reply-To: <FAC7D8FA-E172-4FFA-B06F-C6FA6B964808@icir.org>
References: <67f16fa2c2516e86a55f15ffb0123616@localhost>
	<D36825AC-89C7-4AF3-B1C2-091D0FE2E151@icir.org>
	<175259a1168f271b0ef7759c72f9bf39@localhost>
	<EB78E1F8-41CD-4051-90C1-497E8905B84F@icir.org>
	<1407455420.2576.5.camel@JamesiMac>
	<0464E9BF13BEE74EA662DE35911D64232B9CE62A@SRAexMBX03.sra.com>
	<E5A2071C6FFFF640B714C224139C9B3310C0EB@WP-MBX2.milky-way.battelle.org>
	<20140814143050.GB2497@yaksha.lbl.gov>
	<FAC7D8FA-E172-4FFA-B06F-C6FA6B964808@icir.org>
Message-ID: <E5A2071C6FFFF640B714C224139C9B3310C1F5@WP-MBX2.milky-way.battelle.org>

Please close it and use Aashish's. Mine is actually a variation of his suite, so getting it from him works. 

-----Original Message-----
From: Seth Hall [mailto:seth at icir.org] 
Sent: Thursday, August 14, 2014 11:31 AM
To: Aashish Sharma
Cc: Hosom, Stephen M; Lankau, John; bro at bro-ids.org
Subject: Re: [Bro] Quick smtp-url-extraction question


On Aug 14, 2014, at 10:30 AM, Aashish Sharma <asharma at lbl.gov> wrote:

> 3) if you are running bro-2.3, use smtp-url-extraction-bloom.bro - it uses bloom filters to check against URL's in the http stream. So its less taxing on memory compared to (2). 

Thanks, Aashish.

I've been working on this script for a while this morning just doing general clean up and documentation.  Right now I'm getting ready to add cluster support to it.  I'll 

Stephen, cool if I close your pull request since I think that Aashish's script has more functionality?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From dave at dechellis.com  Thu Aug 14 18:26:22 2014
From: dave at dechellis.com (Dave DeChellis)
Date: Thu, 14 Aug 2014 20:26:22 -0500 (EST)
Subject: [Bro] Question on file hashes and cyrmu db
Message-ID: <1796319538.1573830.1408065982756.open-xchange@bosoxweb03.eigbox.net>

Hello,

I'm helping to customize an existing deployment of Bro and while I think I'm
collecting all the file info correctly, I'm not hitting any matches when I run
the hashes against cymru's database.   I was wondering if someone could confirm
that none of these hashes match either.   I've run them against the DNS,Whois
and web queries and had no luck.  I work at a very open place and I find it
almost impossible that not one of the 1.7M hashes match.   In the event there
are no matches, could someone point me to some sample pcap files so I can test
my scripts?

If someone wanted to help cross correlate my findings, I could send offline a
.gz of 1.7M hashes from a few hours of collection.


Thanks again for any help or assistance
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140814/130136a8/attachment.html 

From doug.burks at gmail.com  Fri Aug 15 03:07:50 2014
From: doug.burks at gmail.com (Doug Burks)
Date: Fri, 15 Aug 2014 06:07:50 -0400
Subject: [Bro] Question on file hashes and cyrmu db
In-Reply-To: <1796319538.1573830.1408065982756.open-xchange@bosoxweb03.eigbox.net>
References: <1796319538.1573830.1408065982756.open-xchange@bosoxweb03.eigbox.net>
Message-ID: <CAK8kjrCXZGymeQDLmBZLvbTei2ZeyQsnv=jd2DfJP51f4=KNvA@mail.gmail.com>

On Thu, Aug 14, 2014 at 9:26 PM, Dave DeChellis <dave at dechellis.com> wrote:
> In the
> event there are no matches, could someone point me to some sample pcap files
> so I can test my scripts?

Hi Dave,

The following pcap generates a TeamCymruMalwareHashRegistry::Match for me:
https://github.com/markofu/workshop/blob/master/samples/pcaps/netforensics_evidence05.pcap


-- 
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com


From dave at dechellis.com  Fri Aug 15 04:47:39 2014
From: dave at dechellis.com (Dave DeChellis)
Date: Fri, 15 Aug 2014 07:47:39 -0400
Subject: [Bro] Question on file hashes and cyrmu db
Message-ID: <20140815114751.9BD482C400C@rock.ICSI.Berkeley.EDU>

Thank you Doug,

The bad hash shows up in my files.log but nowhere else - time to troubleshoot the MHR bro script.

Thanks!
Dave

On Aug 15, 2014 6:07 AM, Doug Burks <doug.burks at gmail.com> wrote:
>
> On Thu, Aug 14, 2014 at 9:26 PM, Dave DeChellis <dave at dechellis.com> wrote: 
> > In the 
> > event there are no matches, could someone point me to some sample pcap files 
> > so I can test my scripts? 
>
> Hi Dave, 
>
> The following pcap generates a TeamCymruMalwareHashRegistry::Match for me: 
> https://github.com/markofu/workshop/blob/master/samples/pcaps/netforensics_evidence05.pcap 
>
>
> -- 
> Doug Burks 
> Need Security Onion Training or Commercial Support? 
> http://securityonionsolutions.com 



From jlay at slave-tothe-box.net  Fri Aug 15 04:59:46 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 15 Aug 2014 05:59:46 -0600
Subject: [Bro] Append instead of overwrite
In-Reply-To: <ac42a46e9e3bef952fb63791b6e974df@localhost>
References: <ac42a46e9e3bef952fb63791b6e974df@localhost>
Message-ID: <1408103986.2820.1.camel@JamesiMac>

On Thu, 2014-07-17 at 09:33 -0600, James Lay wrote:

> Hey All,
> 
> So I run bro instead of broctl.  Currently, if I stop a running bro, 
> and start it again, bro overwrites any previous log files...is there a 
> way to change this behavior?  Thank you.
> 
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


Any takers on this?   Would be really nice not to have to move the
current logs every time I want to test something.  Thanks all.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140815/12866cad/attachment.html 

From seth at icir.org  Fri Aug 15 08:44:59 2014
From: seth at icir.org (Seth Hall)
Date: Fri, 15 Aug 2014 11:44:59 -0400
Subject: [Bro] Question on file hashes and cyrmu db
In-Reply-To: <1796319538.1573830.1408065982756.open-xchange@bosoxweb03.eigbox.net>
References: <1796319538.1573830.1408065982756.open-xchange@bosoxweb03.eigbox.net>
Message-ID: <E744E6C5-C00C-4ADA-A5E5-FDDF21B0BBF4@icir.org>


On Aug 14, 2014, at 9:26 PM, Dave DeChellis <dave at dechellis.com> wrote:

> I've run them against the DNS,Whois and web queries and had no luck.  I work at a very open place and I find it almost impossible that not one of the 1.7M hashes match.

Most of those hashes are likely just web pages your user's are visiting so it think it's very possible that none of them would match.

I see that the pcap file Doug pointed you to isn't working for you either.  It's very possible that you're using a DNS server that isn't very fast and Bro is finishing reading the tracefile before you get a DNS response which will cause you to not have a match.  Try this...

bro -r netforensics_evidence05.pcap frameworks/files/detect-MHR exit_only_after_terminate=T

Wait for a few seconds and then hit ctrl-c and see if you get a notice.  That "exit_only_after_terminate" bit I added at the end will ensure that Bro doesn't terminate as soon as it reaches the end of the tracefile, giving your DNS server a bit of time to respond.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140815/fe9e6bc5/attachment.bin 

From seth at icir.org  Fri Aug 15 08:46:54 2014
From: seth at icir.org (Seth Hall)
Date: Fri, 15 Aug 2014 11:46:54 -0400
Subject: [Bro] Append instead of overwrite
In-Reply-To: <1408103986.2820.1.camel@JamesiMac>
References: <ac42a46e9e3bef952fb63791b6e974df@localhost>
	<1408103986.2820.1.camel@JamesiMac>
Message-ID: <7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>


On Aug 15, 2014, at 7:59 AM, James Lay <jlay at slave-tothe-box.net> wrote:

>> So I run bro instead of broctl.  Currently, if I stop a running bro, 
>> and start it again, bro overwrites any previous log files...is there a 
>> way to change this behavior?  Thank you.

How would you like it to behave instead?  

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140815/30931f6c/attachment.bin 

From jlay at slave-tothe-box.net  Fri Aug 15 08:53:38 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 15 Aug 2014 09:53:38 -0600
Subject: [Bro] Append instead of overwrite
In-Reply-To: <7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>
References: <ac42a46e9e3bef952fb63791b6e974df@localhost>
	<1408103986.2820.1.camel@JamesiMac>
	<7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>
Message-ID: <6c254c70aa052255c9e04c76dd8f2737@localhost>

On 2014-08-15 09:46, Seth Hall wrote:
> On Aug 15, 2014, at 7:59 AM, James Lay <jlay at slave-tothe-box.net> 
> wrote:
>
>>> So I run bro instead of broctl.  Currently, if I stop a running 
>>> bro,
>>> and start it again, bro overwrites any previous log files...is 
>>> there a
>>> way to change this behavior?  Thank you.
>
> How would you like it to behave instead?
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/

To give me an option to append instead of overwrite.  I imagine that 
since broctl does all the file management that this could be a command 
line option...

bro -i eth0 -n local.bro

where -n would be a no overwrite option.  In a nutshell "if the files 
don't exist, create them, if they do, just append, without the header, 
to the current file".  It could just be a single check on start.

How's that?  Thanks Seth.

James


From jlay at slave-tothe-box.net  Fri Aug 15 09:28:54 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 15 Aug 2014 10:28:54 -0600
Subject: [Bro] Append instead of overwrite
In-Reply-To: <7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>
References: <ac42a46e9e3bef952fb63791b6e974df@localhost>
	<1408103986.2820.1.camel@JamesiMac>
	<7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>
Message-ID: <dc6d369463771087f5a230a1ed80e8f1@localhost>

On 2014-08-15 09:46, Seth Hall wrote:
> On Aug 15, 2014, at 7:59 AM, James Lay <jlay at slave-tothe-box.net> 
> wrote:
>
>>> So I run bro instead of broctl.  Currently, if I stop a running 
>>> bro,
>>> and start it again, bro overwrites any previous log files...is 
>>> there a
>>> way to change this behavior?  Thank you.
>
> How would you like it to behave instead?
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/

Seth,

Additionally, it would be wonderful to have bro re-load it's local.bro 
(or whatever) on SIGHUP.  During testing my process is:

killall bro
move log files
make changes to scripts
bro -i eth0 local

Repeat.  It's pretty tedious.  Would be nice too see:

make changes to scripts
killal -HUP bro

That would reload bro local.bro and not overwrite the current log 
files.

Just some more thoughts...thanks Seth.

James



From johanna at icir.org  Fri Aug 15 09:37:07 2014
From: johanna at icir.org (Johanna Amann)
Date: Fri, 15 Aug 2014 09:37:07 -0700
Subject: [Bro] Append instead of overwrite
In-Reply-To: <6c254c70aa052255c9e04c76dd8f2737@localhost>
References: <ac42a46e9e3bef952fb63791b6e974df@localhost>
	<1408103986.2820.1.camel@JamesiMac>
	<7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>
	<6c254c70aa052255c9e04c76dd8f2737@localhost>
Message-ID: <45B8B6CC-E231-483C-8E2A-C15CAA128513@icir.org>

The problem with that approach is, that Bro would have to check that the 
mapping in the files still match. If you change the scripts in-between, 
the order or even the number of columns in the log-files might be 
different. Which would mean that the header do not fit the file content 
anymore.

hat might give you really difficult to parse log-files if you do it by 
accident.

Johanna

On 15 Aug 2014, at 8:53, James Lay wrote:

> On 2014-08-15 09:46, Seth Hall wrote:
>> On Aug 15, 2014, at 7:59 AM, James Lay <jlay at slave-tothe-box.net>
>> wrote:
>>
>>>> So I run bro instead of broctl.  Currently, if I stop a running
>>>> bro,
>>>> and start it again, bro overwrites any previous log files...is
>>>> there a
>>>> way to change this behavior?  Thank you.
>>
>> How would you like it to behave instead?
>>
>> .Seth
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>
> To give me an option to append instead of overwrite.  I imagine that
> since broctl does all the file management that this could be a command
> line option...
>
> bro -i eth0 -n local.bro
>
> where -n would be a no overwrite option.  In a nutshell "if the files
> don't exist, create them, if they do, just append, without the header,
> to the current file".  It could just be a single check on start.
>
> How's that?  Thanks Seth.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From jlay at slave-tothe-box.net  Fri Aug 15 09:54:28 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 15 Aug 2014 10:54:28 -0600
Subject: [Bro] Append instead of overwrite
In-Reply-To: <45B8B6CC-E231-483C-8E2A-C15CAA128513@icir.org>
References: <ac42a46e9e3bef952fb63791b6e974df@localhost>
	<1408103986.2820.1.camel@JamesiMac>
	<7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>
	<6c254c70aa052255c9e04c76dd8f2737@localhost>
	<45B8B6CC-E231-483C-8E2A-C15CAA128513@icir.org>
Message-ID: <058b744de1ea04d204e5fab2c6247e0c@localhost>

On 2014-08-15 10:37, Johanna Amann wrote:
> The problem with that approach is, that Bro would have to check that
> the mapping in the files still match. If you change the scripts
> in-between, the order or even the number of columns in the log-files
> might be different. Which would mean that the header do not fit the
> file content anymore.
>
> hat might give you really difficult to parse log-files if you do it
> by accident.
>
> Johanna
>
> On 15 Aug 2014, at 8:53, James Lay wrote:
>
>> On 2014-08-15 09:46, Seth Hall wrote:
>>> On Aug 15, 2014, at 7:59 AM, James Lay <jlay at slave-tothe-box.net>
>>> wrote:
>>>
>>>>> So I run bro instead of broctl.  Currently, if I stop a running
>>>>> bro,
>>>>> and start it again, bro overwrites any previous log files...is
>>>>> there a
>>>>> way to change this behavior?  Thank you.
>>>
>>> How would you like it to behave instead?
>>>
>>> .Seth
>>>
>>> --
>>> Seth Hall
>>> International Computer Science Institute
>>> (Bro) because everyone has a network
>>> http://www.bro.org/
>>
>> To give me an option to append instead of overwrite.  I imagine that
>> since broctl does all the file management that this could be a 
>> command
>> line option...
>>
>> bro -i eth0 -n local.bro
>>
>> where -n would be a no overwrite option.  In a nutshell "if the 
>> files
>> don't exist, create them, if they do, just append, without the 
>> header,
>> to the current file".  It could just be a single check on start.
>>
>> How's that?  Thanks Seth.
>>
>> James

That makes sense, thanks Johanna.  I'm guessing that not a lot of folks 
run bro outside of brocontrol in a production environment, and to be 
honest, if the cpu usage gets reduced in subsequent versions then I'll 
hop on the brocontrol boat and enjoy all the benefits.  But until then 
bro commandline is where I sit.  I agree that creating broken log files 
is not gonna work for anyone, which is why maybe having to specify if 
via command line and not make it a default would be the way to go.  But 
maybe not.  Thanks again Johanna and everyone really...bro is a crucial 
part of my continuous monitoring...I find more uses for it every day.  I 
just which I was smart enough to give back to the community.

James



From seth at icir.org  Fri Aug 15 11:13:39 2014
From: seth at icir.org (Seth Hall)
Date: Fri, 15 Aug 2014 14:13:39 -0400
Subject: [Bro] Append instead of overwrite
In-Reply-To: <6c254c70aa052255c9e04c76dd8f2737@localhost>
References: <ac42a46e9e3bef952fb63791b6e974df@localhost>
	<1408103986.2820.1.camel@JamesiMac>
	<7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>
	<6c254c70aa052255c9e04c76dd8f2737@localhost>
Message-ID: <5527F075-5D51-4B11-9A20-76B45BAAC647@icir.org>


On Aug 15, 2014, at 11:53 AM, James Lay <jlay at slave-tothe-box.net> wrote:

> To give me an option to append instead of overwrite.  I imagine that 
> since broctl does all the file management that this could be a command 
> line option...

Ah!  You just want to have file management (and perhaps full rotation?) added as a standalone script and not something that is added by broctl?

Johanna is right that with our current logging scheme we can't really append log files for multiple reasons but I could certainly pull together something that would give you decent log rotation without running broctl.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140815/2588f89f/attachment.bin 

From jlay at slave-tothe-box.net  Fri Aug 15 11:24:18 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 15 Aug 2014 12:24:18 -0600
Subject: [Bro] Append instead of overwrite
In-Reply-To: <5527F075-5D51-4B11-9A20-76B45BAAC647@icir.org>
References: <ac42a46e9e3bef952fb63791b6e974df@localhost>
	<1408103986.2820.1.camel@JamesiMac>
	<7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>
	<6c254c70aa052255c9e04c76dd8f2737@localhost>
	<5527F075-5D51-4B11-9A20-76B45BAAC647@icir.org>
Message-ID: <d9b3cad69c9f144144420dd165f6ea8c@localhost>

On 2014-08-15 12:13, Seth Hall wrote:
> On Aug 15, 2014, at 11:53 AM, James Lay <jlay at slave-tothe-box.net> 
> wrote:
>
>> To give me an option to append instead of overwrite.  I imagine that
>> since broctl does all the file management that this could be a 
>> command
>> line option...
>
> Ah!  You just want to have file management (and perhaps full
> rotation?) added as a standalone script and not something that is
> added by broctl?
>
> Johanna is right that with our current logging scheme we can't really
> append log files for multiple reasons but I could certainly pull
> together something that would give you decent log rotation without
> running broctl.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/

Seth, that would absolutely rock!

James


From doug.burks at gmail.com  Sat Aug 16 04:51:05 2014
From: doug.burks at gmail.com (Doug Burks)
Date: Sat, 16 Aug 2014 07:51:05 -0400
Subject: [Bro] What logs have changed from Bro 2.2 to Bro 2.3?
Message-ID: <CAK8kjrBcYLx6BkgaWX7GyMCgmT_BJZs8pMW_gj_NzwrWPG2ypg@mail.gmail.com>

Good morning all,

We're in the process of moving from Bro 2.2 to Bro 2.3 and I'm trying
to determine the logs that have changed.  Is this documented
somewhere?  I see from the Release notes that snmp.log and radius.log
are new logs:
http://www.bro.org/sphinx-git/install/release-notes.html#id3

Looking at some actual 2.3 output, it looks like the format of ssl.log
has changed and there is also a new x509.log.

Are there any other log changes that we need to be aware of?

Thanks!

-- 
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com


From npratley at redhat.com  Tue Aug 19 17:14:37 2014
From: npratley at redhat.com (Nick Pratley)
Date: Wed, 20 Aug 2014 10:14:37 +1000
Subject: [Bro] Bro on RHEL 7
Message-ID: <53F3E86D.8090502@redhat.com>

Hi, does anyone know whether Bro is likely to be packaged for RHEL 7 in
the near future? Installing the current RPM gives:

error: Failed dependencies:
	libpython2.6.so.1.0()(64bit) is needed by bro-2.3.1-1.x86_64

Presumably because RHEL 7 doesn't include python 2.6 any more (it's on 2.7).

I can submit an RFE but if it's already on the roadmap there's no need.
If this is a low priority let me know and I'll see if I can assist.

- Nick


From Bjorn.Samvik at netclean.com  Wed Aug 20 00:30:27 2014
From: Bjorn.Samvik at netclean.com (=?utf-8?B?QmrDtnJuIFNhbXZpaw==?=)
Date: Wed, 20 Aug 2014 07:30:27 +0000
Subject: [Bro] RSS and packet reordering
Message-ID: <53F44E1D.5040009@netclean.com>

Hello,

I'm developing a program that dumps selected traffic to a tap device 
which bro is listening on. This works great. However to increase 
performance I'm trying to use receive side scaling, splitting the packet 
stream into multiple queues according to ips, ports etc. This results in 
packet reordering which confuses bro and the data is not analyzed  and 
assembled correctly. Other programs such as wireshark and tcpflow are 
able to assemble the traffic correctly so all data is there. Typically 
small packets such as acks seems to arrive before larger packets.

I have been searching for bro configurations that affect the tcp 
reassembly process but have so far not found anything that makes the 
situation better. Is there any particular configurations I should look at?

Anyone have experience with RSS and have any ideas how the packet 
reordering issue can be mitigated?

Thank you
/Bj?rn

(DoNotAddDisclaimer)



From john_steve at 163.com  Wed Aug 20 01:42:09 2014
From: john_steve at 163.com (john)
Date: Wed, 20 Aug 2014 16:42:09 +0800 (CST)
Subject: [Bro] There is some memory leak in the Broccoli
Message-ID: <6dbe280f.14b85.147f2949647.Coremail.john_steve@163.com>

Hi Everyone ;
            These days i was writting some C code using Broccoli API to pull events from the Bro , and i run my program for about 10 hours to test it's stability , it worked fine , however i find another issue , it's seems there is some 
memory leak in my program , the ram usage by my program went to  157MB ,  in order to test where the memory leak problem comes from , i remove all my logical from the Broccoli client i wrote , and test it , finally i found that the mem-leak problem comes from the Broccoli API part ,
i use "valgrind [ http://valgrind.org/ ] " to check the mem-leak problem , it gave me some result like this:


==9063== 972 (104 direct, 868 indirect) bytes in 1 blocks are definitely lost in loss record 212 of 236
==9063==    at 0x4A0580F: calloc (vg_replace_malloc.c:618)
==9063==    by 0x4C1FB67: __bro_attrs_new (bro_attrs.c:52)
==9063==    by 0x4C2CA8A: __bro_sobject_create (bro_sobject.c:109)
==9063==    by 0x4C2D767: __bro_sobject_unserialize (bro_sobject.c:426)
==9063==    by 0x4C36217: __bro_table_val_read (bro_val.c:1682)
==9063==    by 0x4C2D7F5: __bro_sobject_unserialize (bro_sobject.c:435)
==9063==    by 0x4C35996: __bro_record_val_read (bro_val.c:1471)
==9063==    by 0x4C2D7F5: __bro_sobject_unserialize (bro_sobject.c:435)
==9063==    by 0x4C22E63: __bro_event_unserialize (bro_event.c:202)
==9063==    by 0x4C26858: io_process_serialization (bro_io.c:239)
==9063==    by 0x4C28AFB: __bro_io_process_input (bro_io.c:1043)
==9063==    by 0x4C1CB39: bro_conn_process_input (bro.c:781)
==9063== 
==9063== LEAK SUMMARY:
==9063==    definitely lost: 104 bytes in 1 blocks
==9063==    indirectly lost: 868 bytes in 16 blocks
==9063==      possibly lost: 0 bytes in 0 blocks
==9063==    still reachable: 102,002 bytes in 2,836 blocks
==9063==         suppressed: 0 bytes in 0 blocks
==9063== Reachable blocks (those to which a pointer was found) are not shown.
==9063== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9063== 
==9063== For counts of detected and suppressed errors, rerun with: -v
==9063== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 4 from 4)


, and i seems the mem-leak problem come from the "bro_conn_process_input"[http://www.bro.org/sphinx/broccoli-api/broccoli_8h.html#a9ca697fd4737c4e0dc3f4a2bf98fa85c]"   function . 
I add the test program i wrote  as an attach file "memory_check.c " , you can compile it and run it , from the system monitor , the ram usage by the program is increasing . 
by the way , i am working the centos-6.5 64bit and gcc  version is "gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)"  and the system monitor is "gnome-system-monitor". 
if anyone know the cause to this problem , please tell me or correct the way i use the Broccoli API . 




Regards ,
John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140820/f874367e/attachment.html 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: memory_check.c
Url: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140820/f874367e/attachment.c 

From jsiwek at illinois.edu  Wed Aug 20 13:35:23 2014
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Wed, 20 Aug 2014 20:35:23 +0000
Subject: [Bro] There is some memory leak in the Broccoli
In-Reply-To: <6dbe280f.14b85.147f2949647.Coremail.john_steve@163.com>
References: <6dbe280f.14b85.147f2949647.Coremail.john_steve@163.com>
Message-ID: <B128DA2D-6A7E-41FF-9A96-E6D57DCDF2B6@illinois.edu>


On Aug 20, 2014, at 3:42 AM, john <john_steve at 163.com> wrote:

> found that the mem-leak problem comes from the Broccoli API part

Yes, thanks for the test case, the leak is in broccoli.  There?s a patch at [1] if you?d like to try it, or if you?re using the git repos, that?s now on the master branches of bro/broccoli.

- Jon

[1] https://github.com/bro/broccoli/commit/29995a9fc1f719d1b408c114e06a4c7b773a1470


From john_steve at 163.com  Wed Aug 20 16:45:23 2014
From: john_steve at 163.com (john)
Date: Thu, 21 Aug 2014 07:45:23 +0800 (CST)
Subject: [Bro] There is some memory leak in the Broccoli
In-Reply-To: <B128DA2D-6A7E-41FF-9A96-E6D57DCDF2B6@illinois.edu>
References: <6dbe280f.14b85.147f2949647.Coremail.john_steve@163.com>
	<B128DA2D-6A7E-41FF-9A96-E6D57DCDF2B6@illinois.edu>
Message-ID: <1c70d99f.327b.147f5cf8397.Coremail.john_steve@163.com>

Ok, thank you for replay, otherwise i need to dig into the source code to find the mem-leak location! Thank you very much!
? 2014-08-21 04:35:23?"Siwek, Jon" <jsiwek at illinois.edu> ???

On Aug 20, 2014, at 3:42 AM, john <john_steve at 163.com> wrote:

> found that the mem-leak problem comes from the Broccoli API part

Yes, thanks for the test case, the leak is in broccoli.  There?s a patch at [1] if you?d like to try it, or if you?re using the git repos, that?s now on the master branches of bro/broccoli.

- Jon

[1] https://github.com/bro/broccoli/commit/29995a9fc1f719d1b408c114e06a4c7b773a1470



From hhoffman at ip-solutions.net  Thu Aug 21 08:26:18 2014
From: hhoffman at ip-solutions.net (Harry Hoffman)
Date: Thu, 21 Aug 2014 11:26:18 -0400
Subject: [Bro] Myricom and Bro... show of hands for successful deployments
	on 10G links (with > 5Gpbs)
Message-ID: <15BDB174-6FB4-4B0B-85C9-899990C9863A@ip-solutions.net>

Hi All,

So, I?m writing to hopefully get a show of hands from those of your out there who?ve employed Myricom cards to capture packets on your 10G links.

I?ll start by saying that while the myricom cards we have in place do a fine job of capturing I?ve been unable to find the secret sauce that allows both capture and writing to disk in a way that doesn?t drop significant amounts of packets using either bro, tcpdump, snort, suricata.

For those of you out there using myricom cards in conjunction with your favorite tools (bro of course ;-)  ) can you let me know what data rate your Myricom cards are seeing and what (assuming some) percentage of packets you are dropping?

If you aren?t dropping anything I?d love to know more about your setup! :-) 

Cheers,
Harry


From Jim.Zhai at ontario.ca  Thu Aug 21 12:36:55 2014
From: Jim.Zhai at ontario.ca (Zhai, Jim (TBS))
Date: Thu, 21 Aug 2014 19:36:55 +0000
Subject: [Bro] override const in weird.bro
Message-ID: <0922FA096972B14B96C2905E97BB56020C1F3B51@CTSPIGDCAPMXS27.cihs.ad.gov.on.ca>

Hello,

Is there a way to override one of the const actions value in the "weird.bro" from ACTION_LOG to ACTION_IGNORE in policy folder?

 Thanks,
-Jim

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140821/93b7092a/attachment.html 

From dnthayer at illinois.edu  Thu Aug 21 12:36:02 2014
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Thu, 21 Aug 2014 14:36:02 -0500
Subject: [Bro] Append instead of overwrite
In-Reply-To: <5527F075-5D51-4B11-9A20-76B45BAAC647@icir.org>
References: <ac42a46e9e3bef952fb63791b6e974df@localhost>	<1408103986.2820.1.camel@JamesiMac>	<7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>	<6c254c70aa052255c9e04c76dd8f2737@localhost>
	<5527F075-5D51-4B11-9A20-76B45BAAC647@icir.org>
Message-ID: <53F64A22.8000209@illinois.edu>

On 08/15/2014 01:13 PM, Seth Hall wrote:
>
> On Aug 15, 2014, at 11:53 AM, James Lay <jlay at slave-tothe-box.net> wrote:
>
>> To give me an option to append instead of overwrite.  I imagine that
>> since broctl does all the file management that this could be a command
>> line option...
>
> Ah!  You just want to have file management (and perhaps full rotation?) added as a standalone script and not something that is added by broctl?
>
> Johanna is right that with our current logging scheme we can't really append log files for multiple reasons but I could certainly pull together something that would give you decent log rotation without running broctl.
>
>    .Seth
>

To get basic log rotation working without running broctl, you only
need to add this in one of your Bro scripts:

redef Log::default_rotation_interval = 3600 secs;

However, that does not compress the rotated logs, and it will not
move them to another directory.  If you want those features, then
you need to have broctl installed, and you need to add this line also:

redef Log::default_rotation_postprocessor_cmd = "archive-log";

The "archive-log" script will be executed by Bro (so it either needs
to be in Bro's PATH or you need to give the pathname).

In order to get the archive-log script to work, you need to
edit broctl.cfg as needed and run "broctl install".  Then start Bro
manually and when Bro runs archive-log it should have all the
info it needs.


From jlay at slave-tothe-box.net  Thu Aug 21 12:57:16 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 21 Aug 2014 13:57:16 -0600
Subject: [Bro] Append instead of overwrite
In-Reply-To: <53F64A22.8000209@illinois.edu>
References: "<ac42a46e9e3bef952fb63791b6e974df@localhost>	<1408103986.2820.1.camel@JamesiMac>	<7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>"
	<6c254c70aa052255c9e04c76dd8f2737@localhost>
	<5527F075-5D51-4B11-9A20-76B45BAAC647@icir.org>
	<53F64A22.8000209@illinois.edu>
Message-ID: <fd9f3c786189b83559e52e9abe42d3b9@localhost>

On 2014-08-21 13:36, Daniel Thayer wrote:
> On 08/15/2014 01:13 PM, Seth Hall wrote:
>>
>> On Aug 15, 2014, at 11:53 AM, James Lay <jlay at slave-tothe-box.net> 
>> wrote:
>>
>>> To give me an option to append instead of overwrite.  I imagine 
>>> that
>>> since broctl does all the file management that this could be a 
>>> command
>>> line option...
>>
>> Ah!  You just want to have file management (and perhaps full 
>> rotation?) added as a standalone script and not something that is 
>> added by broctl?
>>
>> Johanna is right that with our current logging scheme we can't 
>> really append log files for multiple reasons but I could certainly 
>> pull together something that would give you decent log rotation 
>> without running broctl.
>>
>>    .Seth
>>
>
> To get basic log rotation working without running broctl, you only
> need to add this in one of your Bro scripts:
>
> redef Log::default_rotation_interval = 3600 secs;
>
> However, that does not compress the rotated logs, and it will not
> move them to another directory.  If you want those features, then
> you need to have broctl installed, and you need to add this line 
> also:
>
> redef Log::default_rotation_postprocessor_cmd = "archive-log";
>
> The "archive-log" script will be executed by Bro (so it either needs
> to be in Bro's PATH or you need to give the pathname).
>
> In order to get the archive-log script to work, you need to
> edit broctl.cfg as needed and run "broctl install".  Then start Bro
> manually and when Bro runs archive-log it should have all the
> info it needs.

Excellent...thanks so much Daniel...I will start testing that out this 
weekend.

James


From jlay at slave-tothe-box.net  Thu Aug 21 14:13:25 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 21 Aug 2014 15:13:25 -0600
Subject: [Bro] Append instead of overwrite
In-Reply-To: <53F64A22.8000209@illinois.edu>
References: "<ac42a46e9e3bef952fb63791b6e974df@localhost>	<1408103986.2820.1.camel@JamesiMac>	<7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>"
	<6c254c70aa052255c9e04c76dd8f2737@localhost>
	<5527F075-5D51-4B11-9A20-76B45BAAC647@icir.org>
	<53F64A22.8000209@illinois.edu>
Message-ID: <63f0c4d714e3b3d8b57fa6cc27d20501@localhost>

On 2014-08-21 13:36, Daniel Thayer wrote:
> On 08/15/2014 01:13 PM, Seth Hall wrote:
>>
>> On Aug 15, 2014, at 11:53 AM, James Lay <jlay at slave-tothe-box.net> 
>> wrote:
>>
>>> To give me an option to append instead of overwrite.  I imagine 
>>> that
>>> since broctl does all the file management that this could be a 
>>> command
>>> line option...
>>
>> Ah!  You just want to have file management (and perhaps full 
>> rotation?) added as a standalone script and not something that is 
>> added by broctl?
>>
>> Johanna is right that with our current logging scheme we can't 
>> really append log files for multiple reasons but I could certainly 
>> pull together something that would give you decent log rotation 
>> without running broctl.
>>
>>    .Seth
>>
>
> To get basic log rotation working without running broctl, you only
> need to add this in one of your Bro scripts:
>
> redef Log::default_rotation_interval = 3600 secs;
>
> However, that does not compress the rotated logs, and it will not
> move them to another directory.  If you want those features, then
> you need to have broctl installed, and you need to add this line 
> also:
>
> redef Log::default_rotation_postprocessor_cmd = "archive-log";
>
> The "archive-log" script will be executed by Bro (so it either needs
> to be in Bro's PATH or you need to give the pathname).
>
> In order to get the archive-log script to work, you need to
> edit broctl.cfg as needed and run "broctl install".  Then start Bro
> manually and when Bro runs archive-log it should have all the
> info it needs.

Yea so I lied I tested this already :D  This works really well.  I'm 
assuming that the number of seconds in "redef 3600 secs" and 
"LogRotationInterval = 3600" in broctl.conf have to match up.  And as I 
have bro symlinked to /usr/local/bin/bro I also had to symlink:

lrwxrwxrwx 1 root root      47 Aug 21 14:24 /usr/local/bin/archive-log 
-> /usr/local/bro/share/broctl/scripts/archive-log
lrwxrwxrwx 1 root root      52 Aug 21 14:38 
/usr/local/bin/broctl-config.sh -> 
/usr/local/bro/share/broctl/scripts/broctl-config.sh

But after that it ran like a champ.  My last question is if I have 
these rotate every 24 hours, if I say...start this at 15:00, will it 
rotate at 15:00?  Thank you.

James


From jlay at slave-tothe-box.net  Thu Aug 21 15:11:14 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 21 Aug 2014 16:11:14 -0600
Subject: [Bro] =?utf-8?q?Quick_pf=5Fring_question?=
Message-ID: <ad15961af9084456e2a7b17bbe6fb75c@localhost>

Hey all!

So...where/how does one utilize pf_ring via command-line/local.bro?  
I'm not having much luck finding the info...thanks for any help.

James


From dnthayer at illinois.edu  Thu Aug 21 15:07:00 2014
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Thu, 21 Aug 2014 17:07:00 -0500
Subject: [Bro] Append instead of overwrite
In-Reply-To: <63f0c4d714e3b3d8b57fa6cc27d20501@localhost>
References: "<ac42a46e9e3bef952fb63791b6e974df@localhost>	<1408103986.2820.1.camel@JamesiMac>	<7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>"	<6c254c70aa052255c9e04c76dd8f2737@localhost>	<5527F075-5D51-4B11-9A20-76B45BAAC647@icir.org>	<53F64A22.8000209@illinois.edu>
	<63f0c4d714e3b3d8b57fa6cc27d20501@localhost>
Message-ID: <53F66D84.8030608@illinois.edu>

On 08/21/2014 04:13 PM, James Lay wrote:
> On 2014-08-21 13:36, Daniel Thayer wrote:
>> On 08/15/2014 01:13 PM, Seth Hall wrote:
>>>
>>> On Aug 15, 2014, at 11:53 AM, James Lay <jlay at slave-tothe-box.net>
>>> wrote:
>>>
>>>> To give me an option to append instead of overwrite.  I imagine
>>>> that
>>>> since broctl does all the file management that this could be a
>>>> command
>>>> line option...
>>>
>>> Ah!  You just want to have file management (and perhaps full
>>> rotation?) added as a standalone script and not something that is
>>> added by broctl?
>>>
>>> Johanna is right that with our current logging scheme we can't
>>> really append log files for multiple reasons but I could certainly
>>> pull together something that would give you decent log rotation
>>> without running broctl.
>>>
>>>     .Seth
>>>
>>
>> To get basic log rotation working without running broctl, you only
>> need to add this in one of your Bro scripts:
>>
>> redef Log::default_rotation_interval = 3600 secs;
>>
>> However, that does not compress the rotated logs, and it will not
>> move them to another directory.  If you want those features, then
>> you need to have broctl installed, and you need to add this line
>> also:
>>
>> redef Log::default_rotation_postprocessor_cmd = "archive-log";
>>
>> The "archive-log" script will be executed by Bro (so it either needs
>> to be in Bro's PATH or you need to give the pathname).
>>
>> In order to get the archive-log script to work, you need to
>> edit broctl.cfg as needed and run "broctl install".  Then start Bro
>> manually and when Bro runs archive-log it should have all the
>> info it needs.
>
> Yea so I lied I tested this already :D  This works really well.  I'm
> assuming that the number of seconds in "redef 3600 secs" and
> "LogRotationInterval = 3600" in broctl.conf have to match up.  And as I


Those values don't really need to match (but it might be best to
keep them in sync just to avoid confusion).  Since you're not
starting Bro with broctl, then the only broctl config options
that will be used are the ones that the archive-log script uses
(you can look in that script to see which variables it uses,
if you're curious).


> But after that it ran like a champ.  My last question is if I have
> these rotate every 24 hours, if I say...start this at 15:00, will it
> rotate at 15:00?  Thank you.
>
> James

In that case I think it will rotate at midnight.


From jlay at slave-tothe-box.net  Thu Aug 21 15:29:21 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 21 Aug 2014 16:29:21 -0600
Subject: [Bro] Append instead of overwrite
In-Reply-To: <53F66D84.8030608@illinois.edu>
References: "\"<ac42a46e9e3bef952fb63791b6e974df@localhost>	<1408103986.2820.1.camel@JamesiMac>	<7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>\"	<6c254c70aa052255c9e04c76dd8f2737@localhost>	<5527F075-5D51-4B11-9A20-76B45BAAC647@icir.org>"
	<53F64A22.8000209@illinois.edu>
	<63f0c4d714e3b3d8b57fa6cc27d20501@localhost>
	<53F66D84.8030608@illinois.edu>
Message-ID: <12dac000e0b84604ab8cce62f14d65f8@localhost>

On 2014-08-21 16:07, Daniel Thayer wrote:
> On 08/21/2014 04:13 PM, James Lay wrote:
>> On 2014-08-21 13:36, Daniel Thayer wrote:
>>> On 08/15/2014 01:13 PM, Seth Hall wrote:
>>>>
>>>> On Aug 15, 2014, at 11:53 AM, James Lay <jlay at slave-tothe-box.net>
>>>> wrote:
>>>>
>>>>> To give me an option to append instead of overwrite.  I imagine
>>>>> that
>>>>> since broctl does all the file management that this could be a
>>>>> command
>>>>> line option...
>>>>
>>>> Ah!  You just want to have file management (and perhaps full
>>>> rotation?) added as a standalone script and not something that is
>>>> added by broctl?
>>>>
>>>> Johanna is right that with our current logging scheme we can't
>>>> really append log files for multiple reasons but I could certainly
>>>> pull together something that would give you decent log rotation
>>>> without running broctl.
>>>>
>>>>     .Seth
>>>>
>>>
>>> To get basic log rotation working without running broctl, you only
>>> need to add this in one of your Bro scripts:
>>>
>>> redef Log::default_rotation_interval = 3600 secs;
>>>
>>> However, that does not compress the rotated logs, and it will not
>>> move them to another directory.  If you want those features, then
>>> you need to have broctl installed, and you need to add this line
>>> also:
>>>
>>> redef Log::default_rotation_postprocessor_cmd = "archive-log";
>>>
>>> The "archive-log" script will be executed by Bro (so it either 
>>> needs
>>> to be in Bro's PATH or you need to give the pathname).
>>>
>>> In order to get the archive-log script to work, you need to
>>> edit broctl.cfg as needed and run "broctl install".  Then start Bro
>>> manually and when Bro runs archive-log it should have all the
>>> info it needs.
>>
>> Yea so I lied I tested this already :D  This works really well.  I'm
>> assuming that the number of seconds in "redef 3600 secs" and
>> "LogRotationInterval = 3600" in broctl.conf have to match up.  And 
>> as I
>
>
> Those values don't really need to match (but it might be best to
> keep them in sync just to avoid confusion).  Since you're not
> starting Bro with broctl, then the only broctl config options
> that will be used are the ones that the archive-log script uses
> (you can look in that script to see which variables it uses,
> if you're curious).
>
>
>> But after that it ran like a champ.  My last question is if I have
>> these rotate every 24 hours, if I say...start this at 15:00, will it
>> rotate at 15:00?  Thank you.
>>
>> James
>
> In that case I think it will rotate at midnight.

That's beautiful...thanks so much Daniel!

James


From jlay at slave-tothe-box.net  Fri Aug 22 07:17:49 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 22 Aug 2014 08:17:49 -0600
Subject: [Bro] =?utf-8?q?Quick_pf=5Fring_question?=
In-Reply-To: <ad15961af9084456e2a7b17bbe6fb75c@localhost>
References: <ad15961af9084456e2a7b17bbe6fb75c@localhost>
Message-ID: <43a344db79e39bae76142c36937caccd@localhost>

On 2014-08-21 16:11, James Lay wrote:
> Hey all!
>
> So...where/how does one utilize pf_ring via command-line/local.bro?
> I'm not having much luck finding the info...thanks for any help.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

Any takers on this?  I've got my ldd for this looking good and really 
wanting to test this...is broctrl my only option for use with pf_ring?  
Thanks all...appreciate the help.

James


From jlay at slave-tothe-box.net  Fri Aug 22 10:04:16 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 22 Aug 2014 11:04:16 -0600
Subject: [Bro] =?utf-8?q?Quick_pf=5Fring_question?=
In-Reply-To: <ad15961af9084456e2a7b17bbe6fb75c@localhost>
References: <ad15961af9084456e2a7b17bbe6fb75c@localhost>
Message-ID: <2cd5586dc73c3a4e5ba3188f4e08acf3@localhost>

On 2014-08-21 16:11, James Lay wrote:
> Hey all!
>
> So...where/how does one utilize pf_ring via command-line/local.bro?
> I'm not having much luck finding the info...thanks for any help.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

Comparison of running bro linked to system libpcap to bro links to 
pf_ring via command line:

sudo bro --no-checksums -i eth0 local "Site::local_nets += { 
192.168.1.0/24 }"


system libpcap: libpcap.so.0.8 => 
/usr/lib/x86_64-linux-gnu/libpcap.so.0.8 (0x00007f3221f6c000)

top - 10:43:19 up 20:36,  2 users,  load average: 0.42, 0.17, 0.29
Tasks:  99 total,   1 running,  98 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.8 us, 27.6 sy,  0.0 ni, 65.2 id,  0.0 wa,  0.4 hi,  0.0 si, 
0.0 st
KiB Mem:   3082108 total,  2808360 used,   273748 free,    94848 
buffers
KiB Swap:  3002364 total,     1112 used,  3001252 free.  2310580 cached 
Mem

   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ 
COMMAND
  2801 root      20   0  522304  55964   7500 S 27.5  1.8   0:11.61 bro


pf_ring: libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 
(0x00007fe6c42b5000)

top - 10:54:13 up 20:47,  2 users,  load average: 0.05, 0.10, 0.21
Tasks:  99 total,   2 running,  97 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.1 us, 26.9 sy,  0.0 ni, 67.0 id,  0.0 wa,  0.0 hi,  0.0 si, 
0.0 st
KiB Mem:   3082108 total,  2992864 used,    89244 free,    75388 
buffers
KiB Swap:  3002364 total,     1568 used,  3000796 free.  1996052 cached 
Mem

   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ 
COMMAND
  5580 root      20   0  899404 576276 529768 R 29.5 18.7   0:24.32 bro


This is on a low usage link with like...one active connection (my ssh 
session).  Memory is higher and CPU just a smidgeon, so I'm guessing 
something is working.  Thought I'd fire this out for anyone thinking of 
moving to pf_ring.

James


From jxbatchelor at gmail.com  Fri Aug 22 10:09:21 2014
From: jxbatchelor at gmail.com (Jason Batchelor)
Date: Fri, 22 Aug 2014 12:09:21 -0500
Subject: [Bro] Protocol Analyzer Template
Message-ID: <CANWz5X_hRgAG44mtexJaiGWmZiayHL+w1_UGTHNrNhbSpOi98A@mail.gmail.com>

Hello:

I am interested in writing a protocol analyzer, however, I really did not
know exactly where to start.

I checked out the presentation here:
https://www.youtube.com/watch?v=1eDIl9y6ZnM

It was fantastic, and helped me understand more about what the requirements
are.

Toward the end of the presentation there is mention of a script that auto
generates the basic files you need to create your analyzer. Unfortunately,
the deck states it is yet to be released. Does anyone know if this has
happened yet?

Additionally, I noticed that some of the directories/files the presenter
mentions are not present in my installation. For example:

src/analyzers/protocol (not present)

I do not see any .pac files either.

I may be (likely) missing something. If so, please kindly point it out to
me. If not, were there changes made that would make much of the location
information provided in the presentation irrelevant? Could someone kindly
issue a refresher or point me to one?

Many thanks,
Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140822/f8ee7b70/attachment.html 

From rotsted at reservoir.com  Fri Aug 22 11:10:37 2014
From: rotsted at reservoir.com (Robert Rotsted)
Date: Fri, 22 Aug 2014 11:10:37 -0700
Subject: [Bro] Producer Consumer Ratio Script Released
Message-ID: <CALu66+4rVTgrZxYUzShP4O5TvejZMrFUcAZ_haFa3GLDtX4Xcw@mail.gmail.com>

Hi all,

As announced at BroCon, Reservoir Labs will be releasing a few Bro
scripts to the community that we hope you will enjoy!

The first script was released a few minutes ago. It implements the
Producer Consumer Ratio described by Carter Bullard and John Gerth at
FloCon 2014.

This script is located in the following Git repo:
https://github.com/reservoirlabs/bro-producer-consumer-ratio

If you have any questions or comments feel free to reach out.

Best,

Bob

-- 
Bob Rotsted
Senior Engineer
Reservoir Labs, Inc.


From jlay at slave-tothe-box.net  Fri Aug 22 11:21:21 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 22 Aug 2014 12:21:21 -0600
Subject: [Bro] Producer Consumer Ratio Script Released
In-Reply-To: <CALu66+4rVTgrZxYUzShP4O5TvejZMrFUcAZ_haFa3GLDtX4Xcw@mail.gmail.com>
References: <CALu66+4rVTgrZxYUzShP4O5TvejZMrFUcAZ_haFa3GLDtX4Xcw@mail.gmail.com>
Message-ID: <4e29ddbe309c21fcb5e690a793d34ff1@localhost>

On 2014-08-22 12:10, Robert Rotsted wrote:
> Hi all,
>
> As announced at BroCon, Reservoir Labs will be releasing a few Bro
> scripts to the community that we hope you will enjoy!
>
> The first script was released a few minutes ago. It implements the
> Producer Consumer Ratio described by Carter Bullard and John Gerth at
> FloCon 2014.
>
> This script is located in the following Git repo:
> https://github.com/reservoirlabs/bro-producer-consumer-ratio
>
> If you have any questions or comments feel free to reach out.
>
> Best,
>
> Bob

Testing this in dev now....I'll have to tweak my logstash for the new 
column, but it looks pretty tasty...thank you.

James


From vlad at grigorescu.org  Fri Aug 22 11:28:15 2014
From: vlad at grigorescu.org (Vlad Grigorescu)
Date: Fri, 22 Aug 2014 13:28:15 -0500
Subject: [Bro] Protocol Analyzer Template
In-Reply-To: <CANWz5X_hRgAG44mtexJaiGWmZiayHL+w1_UGTHNrNhbSpOi98A@mail.gmail.com>
References: <CANWz5X_hRgAG44mtexJaiGWmZiayHL+w1_UGTHNrNhbSpOi98A@mail.gmail.com>
Message-ID: <CANUJ_Fm9rPkMfNkaOM=Ofr9JTGJ7RVu5NzYHEZbURr5GEXv_qw@mail.gmail.com>

Hi Jason,

The scripts are available here:
https://github.com/grigorescu/binpac_quickstart

Please note that these won't work with current git master, due to the
recently added plugin support (more specifically, the files that are
generated are correct, just the paths are wrong).

It will work with Bro 2.3, though. Updating this to work with master is on
my todo list.

  --Vlad


On Fri, Aug 22, 2014 at 12:09 PM, Jason Batchelor <jxbatchelor at gmail.com>
wrote:

> Hello:
>
> I am interested in writing a protocol analyzer, however, I really did not
> know exactly where to start.
>
> I checked out the presentation here:
> https://www.youtube.com/watch?v=1eDIl9y6ZnM
>
> It was fantastic, and helped me understand more about what the
> requirements are.
>
> Toward the end of the presentation there is mention of a script that auto
> generates the basic files you need to create your analyzer. Unfortunately,
> the deck states it is yet to be released. Does anyone know if this has
> happened yet?
>
> Additionally, I noticed that some of the directories/files the presenter
> mentions are not present in my installation. For example:
>
> src/analyzers/protocol (not present)
>
> I do not see any .pac files either.
>
> I may be (likely) missing something. If so, please kindly point it out to
> me. If not, were there changes made that would make much of the location
> information provided in the presentation irrelevant? Could someone kindly
> issue a refresher or point me to one?
>
> Many thanks,
> Jason
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140822/988bb4ef/attachment.html 

From jlay at slave-tothe-box.net  Fri Aug 22 11:34:04 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 22 Aug 2014 12:34:04 -0600
Subject: [Bro] Producer Consumer Ratio Script Released
In-Reply-To: <4e29ddbe309c21fcb5e690a793d34ff1@localhost>
References: <CALu66+4rVTgrZxYUzShP4O5TvejZMrFUcAZ_haFa3GLDtX4Xcw@mail.gmail.com>
	<4e29ddbe309c21fcb5e690a793d34ff1@localhost>
Message-ID: <c48b056ea4eb4d50edd75b621f7765ed@localhost>

On 2014-08-22 12:21, James Lay wrote:
> On 2014-08-22 12:10, Robert Rotsted wrote:
>> Hi all,
>>
>> As announced at BroCon, Reservoir Labs will be releasing a few Bro
>> scripts to the community that we hope you will enjoy!
>>
>> The first script was released a few minutes ago. It implements the
>> Producer Consumer Ratio described by Carter Bullard and John Gerth 
>> at
>> FloCon 2014.
>>
>> This script is located in the following Git repo:
>> https://github.com/reservoirlabs/bro-producer-consumer-ratio
>>
>> If you have any questions or comments feel free to reach out.
>>
>> Best,
>>
>> Bob
>
> Testing this in dev now....I'll have to tweak my logstash for the new
> column, but it looks pretty tasty...thank you.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

Snazzy:

[12:31:45 analysis:~/current$] head pcr.log
#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   pcr
#open   2014-08-22-12-29-23
#fields ts      src     pcr     summary_interval
#types  time    addr    double  interval
1408732163.900788       192.168.1.253   1.0     60.000000
1408732223.903164       192.168.1.6     -1.0    60.000000

[12:32:11 analysis:~/current$] head conn.log
#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   conn
#open   2014-08-22-12-28-55
#fields ts      uid     id.orig_h       id.orig_p       id.resp_h       
id.resp_p       proto   service duration        orig_bytes      
resp_bytes      conn_state      local_orig      missed_bytes    history 
orig_pkts       orig_ip_bytes   resp_pkts       resp_ip_bytes   
tunnel_parents  pcr
#types  time    string  addr    port    addr    port    enum    string  
interval        count   count   string  bool    count   string  count   
count   count   count   set[string]     double
1408732115.010821       CWQXXpe4ylnesmVAi       x.x.x.x.x        5353   
ff02::fb        5353    udp     dns     3.003461        129     0       
S0      F       0       D       3       273     0       0       (empty) 
-
1408732115.010972       C3ZDEP2vCdSWZkGsnd      192.168.1.253   5353    
224.0.0.251     5353    udp     dns     3.003392        129     0       
S0      T       0       D       3       213     0       0       (empty) 
1.0

James


From vlad at grigorescu.org  Fri Aug 22 11:44:53 2014
From: vlad at grigorescu.org (Vlad Grigorescu)
Date: Fri, 22 Aug 2014 13:44:53 -0500
Subject: [Bro] Myricom and Bro... show of hands for successful
 deployments on 10G links (with > 5Gpbs)
In-Reply-To: <15BDB174-6FB4-4B0B-85C9-899990C9863A@ip-solutions.net>
References: <15BDB174-6FB4-4B0B-85C9-899990C9863A@ip-solutions.net>
Message-ID: <CANUJ_FkukrPvQz7a5Zmo_myXop_5ivWWChN65zqPrncP33fwOA@mail.gmail.com>

Hi Harry,

Can you expand on "allowing both capture and writing to disk?" Carnegie
Mellon runs a Bro cluster with Myricom NICS, which works well. However, the
manager is on a box that doesn't have any workers on it (and thus doesn't
receive any traffic), so I haven't had any I/O contention from network
traffic and log writing. Is that what you're referring to?

We're seeing about 16 Gbps and dropping < 1% (around 0.1% most of the time,
I believe). That's split up over 4 rather beefy boxes, though.

  --Vlad


On Thu, Aug 21, 2014 at 10:26 AM, Harry Hoffman <hhoffman at ip-solutions.net>
wrote:

> Hi All,
>
> So, I?m writing to hopefully get a show of hands from those of your out
> there who?ve employed Myricom cards to capture packets on your 10G links.
>
> I?ll start by saying that while the myricom cards we have in place do a
> fine job of capturing I?ve been unable to find the secret sauce that allows
> both capture and writing to disk in a way that doesn?t drop significant
> amounts of packets using either bro, tcpdump, snort, suricata.
>
> For those of you out there using myricom cards in conjunction with your
> favorite tools (bro of course ;-)  ) can you let me know what data rate
> your Myricom cards are seeing and what (assuming some) percentage of
> packets you are dropping?
>
> If you aren?t dropping anything I?d love to know more about your setup! :-)
>
> Cheers,
> Harry
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140822/da514f62/attachment.html 

From anthony.kasza at gmail.com  Fri Aug 22 11:56:31 2014
From: anthony.kasza at gmail.com (anthony kasza)
Date: Fri, 22 Aug 2014 11:56:31 -0700
Subject: [Bro] Producer Consumer Ratio Script Released
In-Reply-To: <CALu66+4rVTgrZxYUzShP4O5TvejZMrFUcAZ_haFa3GLDtX4Xcw@mail.gmail.com>
References: <CALu66+4rVTgrZxYUzShP4O5TvejZMrFUcAZ_haFa3GLDtX4Xcw@mail.gmail.com>
Message-ID: <CAEZw2bwrWZiKmZ1_MwYGuc8Ci5Sexwu6TpKxUtkLoR1FvH5+tQ@mail.gmail.com>

Awesome! Thanks for releasing this, Bob.

-AK
On Aug 22, 2014 11:12 AM, "Robert Rotsted" <rotsted at reservoir.com> wrote:

> Hi all,
>
> As announced at BroCon, Reservoir Labs will be releasing a few Bro
> scripts to the community that we hope you will enjoy!
>
> The first script was released a few minutes ago. It implements the
> Producer Consumer Ratio described by Carter Bullard and John Gerth at
> FloCon 2014.
>
> This script is located in the following Git repo:
> https://github.com/reservoirlabs/bro-producer-consumer-ratio
>
> If you have any questions or comments feel free to reach out.
>
> Best,
>
> Bob
>
> --
> Bob Rotsted
> Senior Engineer
> Reservoir Labs, Inc.
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140822/dd032663/attachment.html 

From kyle.creyts at gmail.com  Fri Aug 22 12:09:46 2014
From: kyle.creyts at gmail.com (Kyle Creyts)
Date: Fri, 22 Aug 2014 12:09:46 -0700
Subject: [Bro] Myricom and Bro... show of hands for successful
 deployments on 10G links (with > 5Gpbs)
In-Reply-To: <CANUJ_FkukrPvQz7a5Zmo_myXop_5ivWWChN65zqPrncP33fwOA@mail.gmail.com>
References: <15BDB174-6FB4-4B0B-85C9-899990C9863A@ip-solutions.net>
	<CANUJ_FkukrPvQz7a5Zmo_myXop_5ivWWChN65zqPrncP33fwOA@mail.gmail.com>
Message-ID: <CA+TcGd_5cN1TXkfcZCg2yO8kV3mAvF2TEEsmmEjfw5CpAu+cKg@mail.gmail.com>

check your memory bandwidth:
http://www.ntop.org/pf_ring/not-all-servers-are-alike-with-dna/

On Fri, Aug 22, 2014 at 11:44 AM, Vlad Grigorescu <vlad at grigorescu.org> wrote:
> Hi Harry,
>
> Can you expand on "allowing both capture and writing to disk?" Carnegie
> Mellon runs a Bro cluster with Myricom NICS, which works well. However, the
> manager is on a box that doesn't have any workers on it (and thus doesn't
> receive any traffic), so I haven't had any I/O contention from network
> traffic and log writing. Is that what you're referring to?
>
> We're seeing about 16 Gbps and dropping < 1% (around 0.1% most of the time,
> I believe). That's split up over 4 rather beefy boxes, though.
>
>   --Vlad
>
>
> On Thu, Aug 21, 2014 at 10:26 AM, Harry Hoffman <hhoffman at ip-solutions.net>
> wrote:
>>
>> Hi All,
>>
>> So, I?m writing to hopefully get a show of hands from those of your out
>> there who?ve employed Myricom cards to capture packets on your 10G links.
>>
>> I?ll start by saying that while the myricom cards we have in place do a
>> fine job of capturing I?ve been unable to find the secret sauce that allows
>> both capture and writing to disk in a way that doesn?t drop significant
>> amounts of packets using either bro, tcpdump, snort, suricata.
>>
>> For those of you out there using myricom cards in conjunction with your
>> favorite tools (bro of course ;-)  ) can you let me know what data rate your
>> Myricom cards are seeing and what (assuming some) percentage of packets you
>> are dropping?
>>
>> If you aren?t dropping anything I?d love to know more about your setup!
>> :-)
>>
>> Cheers,
>> Harry
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Kyle Creyts

Information Assurance Professional
Founder BSidesDetroit



From hhoffman at ip-solutions.net  Fri Aug 22 13:27:25 2014
From: hhoffman at ip-solutions.net (Harry Hoffman)
Date: Fri, 22 Aug 2014 16:27:25 -0400
Subject: [Bro] Myricom and Bro... show of hands for successful
	deployments on 10G links (with > 5Gpbs)
In-Reply-To: <CANUJ_FkukrPvQz7a5Zmo_myXop_5ivWWChN65zqPrncP33fwOA@mail.gmail.com>
References: <15BDB174-6FB4-4B0B-85C9-899990C9863A@ip-solutions.net>
	<CANUJ_FkukrPvQz7a5Zmo_myXop_5ivWWChN65zqPrncP33fwOA@mail.gmail.com>
Message-ID: <FEB75270-75E5-4991-81DF-16C59E96D100@ip-solutions.net>

Hi Vlad,

Absolutely. Sorry if that was vague or cryptic.

What I meant was that using the myricom test utilities I can capture everything on the wire. These utilities don?t write to disk so they only show that there?s not a issue with nic to memory transfers.

Once I fire up bro one worker consistently pegs a core at 100% and I drop greater then 1/2 of packets. The rate of drop isn?t as severe with tools like tcpdump but I assume that the difference in processing that bro does with packets.

All of these is running on a Dell R710 with 2 Xeon CPUs at 2.8GHz with 6 cores each (HT disabled)  and 96GB of RAM and two SSD drives for data each 700GB in size. We moved to the Dell specifically to test whether or not using SSD drives gave a performance boost in writing to disk.

We?re using the myricom tools (/opt/snf/bin/myri_counters) to determine dropped packets, via SNF drop ring full,  due to the application (tcpdump, bro, etc) being too slow to grab packets from the ring buffer.

As an initial, memory only test, we?ve run /opt/snf/bin/tests/snf_simple_recv and /opt/snf/bin/tests/snf_multi_recv. Both run without any drops and output shows an avg of 7Gbps on the wire. Running either test for extended periods of time does not cause the values in ?SNF drop ring full? to increment.

/usr/local/bro/etc/node.cfg looks like (as you can see we?re attempting to tweak performance via the various SNF env variables. There?s no difference noticed using pin_cpus:

[manager]
type=manager
host=localhost
#
[proxy-1]
type=proxy
host=localhost
#
[worker-1]
type=worker
host=localhost
interface=p1p1
lb_method=myricom
lb_procs=10
#pin_cpus=2,3,4,5,6,7,8,9,10,11
env_vars="SNF_DEBUG_MASK=0x03,SNF_DATARING_SIZE=1024MB,SNF_DESCRING_SIZE=256MB?



We do have a rather large networks.cfg with 1439 entries in it.

Any suggestions would be greatly appreciated!  We had a ?beefy-er Cisco 1u UCS box with 2 x 8 cores and the same memory and that didn?t hold up either. There were no free pci slots to test SSD drives which is why we?re testing with the Dell.

Cheers,
Harry


On Aug 22, 2014, at 2:44 PM, Vlad Grigorescu <vlad at grigorescu.org> wrote:

> Hi Harry,
> 
> Can you expand on "allowing both capture and writing to disk?" Carnegie Mellon runs a Bro cluster with Myricom NICS, which works well. However, the manager is on a box that doesn't have any workers on it (and thus doesn't receive any traffic), so I haven't had any I/O contention from network traffic and log writing. Is that what you're referring to?
> 
> We're seeing about 16 Gbps and dropping < 1% (around 0.1% most of the time, I believe). That's split up over 4 rather beefy boxes, though.
> 
>   --Vlad
> 
> 
> On Thu, Aug 21, 2014 at 10:26 AM, Harry Hoffman <hhoffman at ip-solutions.net> wrote:
> Hi All,
> 
> So, I?m writing to hopefully get a show of hands from those of your out there who?ve employed Myricom cards to capture packets on your 10G links.
> 
> I?ll start by saying that while the myricom cards we have in place do a fine job of capturing I?ve been unable to find the secret sauce that allows both capture and writing to disk in a way that doesn?t drop significant amounts of packets using either bro, tcpdump, snort, suricata.
> 
> For those of you out there using myricom cards in conjunction with your favorite tools (bro of course ;-)  ) can you let me know what data rate your Myricom cards are seeing and what (assuming some) percentage of packets you are dropping?
> 
> If you aren?t dropping anything I?d love to know more about your setup! :-)
> 
> Cheers,
> Harry
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140822/e9ceafb7/attachment.html 

From hhoffman at ip-solutions.net  Fri Aug 22 13:31:08 2014
From: hhoffman at ip-solutions.net (Harry Hoffman)
Date: Fri, 22 Aug 2014 16:31:08 -0400
Subject: [Bro] Myricom and Bro... show of hands for successful
	deployments on 10G links (with > 5Gpbs)
In-Reply-To: <CA+TcGd_5cN1TXkfcZCg2yO8kV3mAvF2TEEsmmEjfw5CpAu+cKg@mail.gmail.com>
References: <15BDB174-6FB4-4B0B-85C9-899990C9863A@ip-solutions.net>
	<CANUJ_FkukrPvQz7a5Zmo_myXop_5ivWWChN65zqPrncP33fwOA@mail.gmail.com>
	<CA+TcGd_5cN1TXkfcZCg2yO8kV3mAvF2TEEsmmEjfw5CpAu+cKg@mail.gmail.com>
Message-ID: <E5B69322-A8DB-4657-9402-C2C01170A9A4@ip-solutions.net>

Thanks, Kyle!

Very informative article. I?m installing numactl now and will test.

I do note that they say they are doing close to line rate with a Dell R710 so that?s promising :-)

Cheers,
Harry


On Aug 22, 2014, at 3:09 PM, Kyle Creyts <kyle.creyts at gmail.com> wrote:

> check your memory bandwidth:
> http://www.ntop.org/pf_ring/not-all-servers-are-alike-with-dna/
> 
> On Fri, Aug 22, 2014 at 11:44 AM, Vlad Grigorescu <vlad at grigorescu.org> wrote:
>> Hi Harry,
>> 
>> Can you expand on "allowing both capture and writing to disk?" Carnegie
>> Mellon runs a Bro cluster with Myricom NICS, which works well. However, the
>> manager is on a box that doesn't have any workers on it (and thus doesn't
>> receive any traffic), so I haven't had any I/O contention from network
>> traffic and log writing. Is that what you're referring to?
>> 
>> We're seeing about 16 Gbps and dropping < 1% (around 0.1% most of the time,
>> I believe). That's split up over 4 rather beefy boxes, though.
>> 
>>  --Vlad
>> 
>> 
>> On Thu, Aug 21, 2014 at 10:26 AM, Harry Hoffman <hhoffman at ip-solutions.net>
>> wrote:
>>> 
>>> Hi All,
>>> 
>>> So, I?m writing to hopefully get a show of hands from those of your out
>>> there who?ve employed Myricom cards to capture packets on your 10G links.
>>> 
>>> I?ll start by saying that while the myricom cards we have in place do a
>>> fine job of capturing I?ve been unable to find the secret sauce that allows
>>> both capture and writing to disk in a way that doesn?t drop significant
>>> amounts of packets using either bro, tcpdump, snort, suricata.
>>> 
>>> For those of you out there using myricom cards in conjunction with your
>>> favorite tools (bro of course ;-)  ) can you let me know what data rate your
>>> Myricom cards are seeing and what (assuming some) percentage of packets you
>>> are dropping?
>>> 
>>> If you aren?t dropping anything I?d love to know more about your setup!
>>> :-)
>>> 
>>> Cheers,
>>> Harry
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> 
>> 
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> 
> -- 
> Kyle Creyts
> 
> Information Assurance Professional
> Founder BSidesDetroit




From jlay at slave-tothe-box.net  Sat Aug 23 04:43:54 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Sat, 23 Aug 2014 05:43:54 -0600
Subject: [Bro] Append instead of overwrite
In-Reply-To: <53F66D84.8030608@illinois.edu>
References: "<ac42a46e9e3bef952fb63791b6e974df@localhost>
	<1408103986.2820.1.camel@JamesiMac>
	<7D632EEB-2163-46E3-8A3B-C2C706FEABDF@icir.org>
	"	<6c254c70aa052255c9e04c76dd8f2737@localhost>
	<5527F075-5D51-4B11-9A20-76B45BAAC647@icir.org>
	<53F64A22.8000209@illinois.edu>
	<63f0c4d714e3b3d8b57fa6cc27d20501@localhost>
	<53F66D84.8030608@illinois.edu>
Message-ID: <1408794234.2785.4.camel@JamesiMac>

On Thu, 2014-08-21 at 17:07 -0500, Daniel Thayer wrote:

> On 08/21/2014 04:13 PM, James Lay wrote:
> > On 2014-08-21 13:36, Daniel Thayer wrote:
> >> On 08/15/2014 01:13 PM, Seth Hall wrote:
> >>>
> >>> On Aug 15, 2014, at 11:53 AM, James Lay <jlay at slave-tothe-box.net>
> >>> wrote:
> >>>
> >>>> To give me an option to append instead of overwrite.  I imagine
> >>>> that
> >>>> since broctl does all the file management that this could be a
> >>>> command
> >>>> line option...
> >>>
> >>> Ah!  You just want to have file management (and perhaps full
> >>> rotation?) added as a standalone script and not something that is
> >>> added by broctl?
> >>>
> >>> Johanna is right that with our current logging scheme we can't
> >>> really append log files for multiple reasons but I could certainly
> >>> pull together something that would give you decent log rotation
> >>> without running broctl.
> >>>
> >>>     .Seth
> >>>
> >>
> >> To get basic log rotation working without running broctl, you only
> >> need to add this in one of your Bro scripts:
> >>
> >> redef Log::default_rotation_interval = 3600 secs;
> >>
> >> However, that does not compress the rotated logs, and it will not
> >> move them to another directory.  If you want those features, then
> >> you need to have broctl installed, and you need to add this line
> >> also:
> >>
> >> redef Log::default_rotation_postprocessor_cmd = "archive-log";
> >>
> >> The "archive-log" script will be executed by Bro (so it either needs
> >> to be in Bro's PATH or you need to give the pathname).
> >>
> >> In order to get the archive-log script to work, you need to
> >> edit broctl.cfg as needed and run "broctl install".  Then start Bro
> >> manually and when Bro runs archive-log it should have all the
> >> info it needs.
> >
> > Yea so I lied I tested this already :D  This works really well.  I'm
> > assuming that the number of seconds in "redef 3600 secs" and
> > "LogRotationInterval = 3600" in broctl.conf have to match up.  And as I
> 
> 
> Those values don't really need to match (but it might be best to
> keep them in sync just to avoid confusion).  Since you're not
> starting Bro with broctl, then the only broctl config options
> that will be used are the ones that the archive-log script uses
> (you can look in that script to see which variables it uses,
> if you're curious).
> 
> 
> > But after that it ran like a champ.  My last question is if I have
> > these rotate every 24 hours, if I say...start this at 15:00, will it
> > rotate at 15:00?  Thank you.
> >
> > James
> 
> In that case I think it will rotate at midnight.


FYI...this absolutely rotated at midnight...which is just
perfect...thanks again.

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140823/5ecba51c/attachment.html 

From fabian at affolter-engineering.ch  Sat Aug 23 06:38:07 2014
From: fabian at affolter-engineering.ch (Fabian Affolter)
Date: Sat, 23 Aug 2014 15:38:07 +0200
Subject: [Bro] Bro on RHEL 7
In-Reply-To: <53F3E86D.8090502@redhat.com>
References: <53F3E86D.8090502@redhat.com>
Message-ID: <53F8993F.5000002@affolter-engineering.ch>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 08/20/2014 02:14 AM, Nick Pratley wrote:
> Hi, does anyone know whether Bro is likely to be packaged for RHEL
> 7 in the near future? Installing the current RPM gives:

Right know I'm working on the Fedora package of Bro 2.3 and the plan
is to push that package to EPEL as well.

The package needs a couple of changes because of given locations/paths
and patches to work around the mixture of autotools elements and cmake
which confuses the build system. I don't make an promises but I hope
to be ready by the end of September.

> If this is a low priority let me know and I'll see if I can
> assist.

Once if it's done, a test run be nice.  This will probably break things.

Kind regards,

Fabian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iEYEARECAAYFAlP4mTwACgkQ4jzS3TakOX/zawCghE81TJqZrmQiJ58hNgQjzj4/
IhYAnRd7z6q3rmzd0NiTHKYpBgfWZ482
=Oz9u
-----END PGP SIGNATURE-----


From michalpurzynski1 at gmail.com  Sat Aug 23 07:38:59 2014
From: michalpurzynski1 at gmail.com (Michal Purzynski)
Date: Sat, 23 Aug 2014 09:38:59 -0500
Subject: [Bro] Myricom and Bro... show of hands for successful
 deployments on 10G links (with > 5Gpbs)
In-Reply-To: <FEB75270-75E5-4991-81DF-16C59E96D100@ip-solutions.net>
References: <15BDB174-6FB4-4B0B-85C9-899990C9863A@ip-solutions.net>	<CANUJ_FkukrPvQz7a5Zmo_myXop_5ivWWChN65zqPrncP33fwOA@mail.gmail.com>
	<FEB75270-75E5-4991-81DF-16C59E96D100@ip-solutions.net>
Message-ID: <53F8A783.10301@gmail.com>

On 8/22/14, 3:27 PM, Harry Hoffman wrote:
> Hi Vlad,
>
> Absolutely. Sorry if that was vague or cryptic.
>
> What I meant was that using the myricom test utilities I can capture 
> everything on the wire. These utilities don't write to disk so they 
> only show that there's not a issue with nic to memory transfers.
>
> Once I fire up bro one worker consistently pegs a core at 100% and I 
> drop greater then 1/2 of packets. The rate of drop isn't as severe 
> with tools like tcpdump but I assume that the difference in processing 
> that bro does with packets.
That most likely means you are not using the Myricom API to capture 
packets. I've seen the symptoms you're describing.

Please send the output of

ldd `which bro` | egrep -i '(myri|snf)
>
> All of these is running on a Dell R710 with 2 Xeon CPUs at 2.8GHz with 
> 6 cores each (HT disabled)  and 96GB of RAM and two SSD drives for 
> data each 700GB in size. We moved to the Dell specifically to test 
> whether or not using SSD drives gave a performance boost in writing to 
> disk.
>
> We're using the myricom tools (/opt/snf/bin/myri_counters) to 
> determine dropped packets, via SNF drop ring full,  due to the 
> application (tcpdump, bro, etc) being too slow to grab packets from 
> the ring buffer.
>
> As an initial, memory only test, we've run 
> /opt/snf/bin/tests/snf_simple_recv and 
> /opt/snf/bin/tests/snf_multi_recv. Both run without any drops and 
> output shows an avg of 7Gbps on the wire. Running either test for 
> extended periods of time does not cause the values in "SNF drop ring 
> full" to increment.
>
> /usr/local/bro/etc/node.cfg looks like (as you can see we're 
> attempting to tweak performance via the various SNF env variables. 
> There's no difference noticed using pin_cpus:
>
> [manager]
> type=manager
> host=localhost
> #
> [proxy-1]
> type=proxy
> host=localhost
> #
> [worker-1]
> type=worker
> host=localhost
> interface=p1p1
> lb_method=myricom
> lb_procs=10
> #pin_cpus=2,3,4,5,6,7,8,9,10,11
> env_vars="SNF_DEBUG_MASK=0x03,SNF_DATARING_SIZE=1024MB,SNF_DESCRING_SIZE=256MB"
Oh man that's way too small, I'll check in more details later but I'm 
running with a few GB large dataring.

Fortunately the ring is shared accross all processes, so a 16GB ring 
times 16 processes does not use 256GB of RAM ;)

There's some info in the Myricom docs on how the dataring size and 
descring size should related to each other, I believe it was 4:1.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140823/b98de260/attachment.html 

From bkellogg at dresser-rand.com  Sat Aug 23 21:18:47 2014
From: bkellogg at dresser-rand.com (Kellogg, Brian D (OLN))
Date: Sun, 24 Aug 2014 04:18:47 +0000
Subject: [Bro] Python file to build and modify Intel files
Message-ID: <BDF3793B557F624699D4C68A3B8FCF821ECF0F@DRTEXW238VM.na.d-rco.com>

I created this Python script so that I wouldn't have to modify our custom intel files by hand.  I've only really used the add and remove IP portions of the script, but all of the other intel options are present.  It can work on existing intel files or create new ones.

This is written around the SecurityOnion installation of Bro so you will have to change a variable or two to get it to work if Bro is installed in another directory.

Sharing in case anyone else finds use in it.

The adding and removing of IPs may not be to everyone's liking.  I thought of using subnetting to do this, but I find that the most IPs I add to an intel file at one time is a /24 and that is rare.  Usually it is one or just a couple that get added in my experience.

I haven't added validation for other types of intel additions yet, just IPs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140824/8ec57da5/attachment.html 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: modIntel.txt
Url: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140824/8ec57da5/attachment.txt 

From hhoffman at ip-solutions.net  Sun Aug 24 06:16:06 2014
From: hhoffman at ip-solutions.net (Harry Hoffman)
Date: Sun, 24 Aug 2014 09:16:06 -0400
Subject: [Bro] Myricom and Bro... show of hands for successful
	deployments on 10G links (with > 5Gpbs)
In-Reply-To: <53F8A783.10301@gmail.com>
References: <15BDB174-6FB4-4B0B-85C9-899990C9863A@ip-solutions.net>	<CANUJ_FkukrPvQz7a5Zmo_myXop_5ivWWChN65zqPrncP33fwOA@mail.gmail.com>
	<FEB75270-75E5-4991-81DF-16C59E96D100@ip-solutions.net>
	<53F8A783.10301@gmail.com>
Message-ID: <23EFBB05-C678-41E3-921D-0606B56C2A76@ip-solutions.net>

Hi Michal,

I am indeed linked to the pcap supplied by myricom. As described I see the snf counters increasing when bro is running (via myri_counters). Bro also shows up in the myri_endpoint_info

# ldd /usr/local/bro/bin/bro
	linux-vdso.so.1 =>  (0x00007fffaabd1000)
	libpcap.so.1 => /opt/snf/lib/libpcap.so.1 (0x00007f6c0ebf1000)
        ...


Thanks for the pointer about the shared rings. I?d mistakenly (?) believed the opposite and will consult the documentation.

Any other thoughts?

Cheers,
Harry


On Aug 23, 2014, at 10:38 AM, Michal Purzynski <michalpurzynski1 at gmail.com> wrote:

> On 8/22/14, 3:27 PM, Harry Hoffman wrote:
>> Hi Vlad,
>> 
>> Absolutely. Sorry if that was vague or cryptic.
>> 
>> What I meant was that using the myricom test utilities I can capture everything on the wire. These utilities don?t write to disk so they only show that there?s not a issue with nic to memory transfers.
>> 
>> Once I fire up bro one worker consistently pegs a core at 100% and I drop greater then 1/2 of packets. The rate of drop isn?t as severe with tools like tcpdump but I assume that the difference in processing that bro does with packets.
> That most likely means you are not using the Myricom API to capture packets. I've seen the symptoms you're describing.
> 
> Please send the output of 
> 
> ldd `which bro` | egrep -i '(myri|snf)
>> 
>> All of these is running on a Dell R710 with 2 Xeon CPUs at 2.8GHz with 6 cores each (HT disabled)  and 96GB of RAM and two SSD drives for data each 700GB in size. We moved to the Dell specifically to test whether or not using SSD drives gave a performance boost in writing to disk.
>> 
>> We?re using the myricom tools (/opt/snf/bin/myri_counters) to determine dropped packets, via SNF drop ring full,  due to the application (tcpdump, bro, etc) being too slow to grab packets from the ring buffer.
>> 
>> As an initial, memory only test, we?ve run /opt/snf/bin/tests/snf_simple_recv and /opt/snf/bin/tests/snf_multi_recv. Both run without any drops and output shows an avg of 7Gbps on the wire. Running either test for extended periods of time does not cause the values in ?SNF drop ring full? to increment.
>> 
>> /usr/local/bro/etc/node.cfg looks like (as you can see we?re attempting to tweak performance via the various SNF env variables. There?s no difference noticed using pin_cpus:
>> 
>> [manager]
>> type=manager
>> host=localhost
>> #
>> [proxy-1]
>> type=proxy
>> host=localhost
>> #
>> [worker-1]
>> type=worker
>> host=localhost
>> interface=p1p1
>> lb_method=myricom
>> lb_procs=10
>> #pin_cpus=2,3,4,5,6,7,8,9,10,11
>> env_vars="SNF_DEBUG_MASK=0x03,SNF_DATARING_SIZE=1024MB,SNF_DESCRING_SIZE=256MB?
> Oh man that's way too small, I'll check in more details later but I'm running with a few GB large dataring.
> 
> Fortunately the ring is shared accross all processes, so a 16GB ring times 16 processes does not use 256GB of RAM ;)
> 
> There's some info in the Myricom docs on how the dataring size and descring size should related to each other, I believe it was 4:1.
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140824/97f90645/attachment.html 

From pachinko.tw at gmail.com  Sun Aug 24 07:34:15 2014
From: pachinko.tw at gmail.com (Po-Ching Lin)
Date: Sun, 24 Aug 2014 22:34:15 +0800
Subject: [Bro] BroArgs in broctl.cfg
Message-ID: <53F9F7E7.7090102@gmail.com>


I find broctl is unable to install the following line in broctl.cfg because of the parentheses.

BroArgs = -C -f "(src net 192.168.0.0/24 and dst port 25) or (src port 25 and dst net 192.168.0.0/24)"

The error message is as follows:

[BroControl] > check
bro scripts failed.
    /usr/local/bro/share/broctl/scripts/broctl-config.sh: line 69: syntax error near unexpected token `('
    /usr/local/bro/share/broctl/scripts/broctl-config.sh: line 69: `broargs="-C -f "(src net 192.168.0.0/24 and dst port 25) or (src port 25 and dst net 192.168.0.0/24)""'

What should I do if I want to use parentheses in the BPF string? Thanks.

Po-Ching




From dnthayer at illinois.edu  Sun Aug 24 08:46:20 2014
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Sun, 24 Aug 2014 10:46:20 -0500
Subject: [Bro] BroArgs in broctl.cfg
In-Reply-To: <53F9F7E7.7090102@gmail.com>
References: <53F9F7E7.7090102@gmail.com>
Message-ID: <53FA08CC.6040909@illinois.edu>

Try changing the double quotes to single quotes, like this:

BroArgs = -C -f '(src net 192.168.0.0/24 and dst port 25) or (src port 
25 and dst net 192.168.0.0/24)'


On 08/24/2014 09:34 AM, Po-Ching Lin wrote:
>
> I find broctl is unable to install the following line in broctl.cfg because of the parentheses.
>
> BroArgs = -C -f "(src net 192.168.0.0/24 and dst port 25) or (src port 25 and dst net 192.168.0.0/24)"
>
> The error message is as follows:
>
> [BroControl] > check
> bro scripts failed.
>      /usr/local/bro/share/broctl/scripts/broctl-config.sh: line 69: syntax error near unexpected token `('
>      /usr/local/bro/share/broctl/scripts/broctl-config.sh: line 69: `broargs="-C -f "(src net 192.168.0.0/24 and dst port 25) or (src port 25 and dst net 192.168.0.0/24)""'
>
> What should I do if I want to use parentheses in the BPF string? Thanks.
>
> Po-Ching
>


From npratley at redhat.com  Sun Aug 24 21:50:56 2014
From: npratley at redhat.com (Nick Pratley)
Date: Mon, 25 Aug 2014 14:50:56 +1000
Subject: [Bro] Bro on RHEL 7
In-Reply-To: <53F8993F.5000002@affolter-engineering.ch>
References: <53F3E86D.8090502@redhat.com>
	<53F8993F.5000002@affolter-engineering.ch>
Message-ID: <53FAC0B0.8050308@redhat.com>

On 08/23/2014 11:38 PM, Fabian Affolter wrote:
> On 08/20/2014 02:14 AM, Nick Pratley wrote:
>> Hi, does anyone know whether Bro is likely to be packaged for RHEL
>> 7 in the near future? Installing the current RPM gives:
> 
> Right know I'm working on the Fedora package of Bro 2.3 and the plan
> is to push that package to EPEL as well.
> 
> The package needs a couple of changes because of given locations/paths
> and patches to work around the mixture of autotools elements and cmake
> which confuses the build system. I don't make an promises but I hope
> to be ready by the end of September.
> 
>> If this is a low priority let me know and I'll see if I can
>> assist.
> 
> Once if it's done, a test run be nice.  This will probably break things.


Thanks Fabian, I'll be happy to do a test installation then, just let me
know.

- Nick


From seth at icir.org  Mon Aug 25 07:06:13 2014
From: seth at icir.org (Seth Hall)
Date: Mon, 25 Aug 2014 10:06:13 -0400
Subject: [Bro] RSS and packet reordering
In-Reply-To: <53F44E1D.5040009@netclean.com>
References: <53F44E1D.5040009@netclean.com>
Message-ID: <A91FD3D8-5B37-4C3D-A79D-863E951D86F5@icir.org>


On Aug 20, 2014, at 3:30 AM, Bj?rn Samvik <Bjorn.Samvik at netclean.com> wrote:

> Anyone have experience with RSS and have any ideas how the packet 
> reordering issue can be mitigated?

Are you using straight RSS without any modifications?  By default the Toeplitz hash that RSS uses is not a bidirectional hash.  i.e., each flow of a connection ends up on a different NIC queue.  As a general rule though, you should try to avoid anything that could result in packet reordering.

Another question, how are you making Bro attach to the various NIC hardware queues?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140825/c5f78561/attachment.bin 

From seth at icir.org  Mon Aug 25 10:35:19 2014
From: seth at icir.org (Seth Hall)
Date: Mon, 25 Aug 2014 13:35:19 -0400
Subject: [Bro] Quick pf_ring question
In-Reply-To: <ad15961af9084456e2a7b17bbe6fb75c@localhost>
References: <ad15961af9084456e2a7b17bbe6fb75c@localhost>
Message-ID: <FCBF9B0D-E130-4A93-9515-47967D793CD0@icir.org>


On Aug 21, 2014, at 6:11 PM, James Lay <jlay at slave-tothe-box.net> wrote:

> Hey all!
> 
> So...where/how does one utilize pf_ring via command-line/local.bro?  
> I'm not having much luck finding the info...thanks for any help.

You could take a look at the pf_ring plugin in BroControl.  There are some special environment variables that need to be set.  

The main one you probably are concerned with is: PCAP_PF_RING_CLUSTER_ID.  Set this to some numeric value and use the same value for each worker you are running and the traffic should be balanced across all of your processes.

You should also probably set the PCAP_PF_RING_USE_CLUSTER_PER_FLOW to 1 as well.

Since you're running Bro manually, it might look like this:

PCAP_PF_RING_USE_CLUSTER_PER_FLOW=1 PCAP_PF_RING_CLUSTER_ID=21 bro <your args>

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140825/dbb63e48/attachment.bin 

From jlay at slave-tothe-box.net  Mon Aug 25 10:37:59 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Mon, 25 Aug 2014 11:37:59 -0600
Subject: [Bro] =?utf-8?q?Quick_pf=5Fring_question?=
In-Reply-To: <FCBF9B0D-E130-4A93-9515-47967D793CD0@icir.org>
References: <ad15961af9084456e2a7b17bbe6fb75c@localhost>
	<FCBF9B0D-E130-4A93-9515-47967D793CD0@icir.org>
Message-ID: <03addafd3bbc3b686798fe2a9ee02e5f@localhost>

On 2014-08-25 11:35, Seth Hall wrote:
> On Aug 21, 2014, at 6:11 PM, James Lay <jlay at slave-tothe-box.net> 
> wrote:
>
>> Hey all!
>>
>> So...where/how does one utilize pf_ring via command-line/local.bro?
>> I'm not having much luck finding the info...thanks for any help.
>
> You could take a look at the pf_ring plugin in BroControl.  There are
> some special environment variables that need to be set.
>
> The main one you probably are concerned with is:
> PCAP_PF_RING_CLUSTER_ID.  Set this to some numeric value and use the
> same value for each worker you are running and the traffic should be
> balanced across all of your processes.
>
> You should also probably set the PCAP_PF_RING_USE_CLUSTER_PER_FLOW to
> 1 as well.
>
> Since you're running Bro manually, it might look like this:
>
> PCAP_PF_RING_USE_CLUSTER_PER_FLOW=1 PCAP_PF_RING_CLUSTER_ID=21 bro
> <your args>
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/


That's awesome...thanks for the info Seth.

James


From Bjorn.Samvik at netclean.com  Tue Aug 26 00:24:50 2014
From: Bjorn.Samvik at netclean.com (=?Windows-1252?Q?Bj=F6rn_Samvik?=)
Date: Tue, 26 Aug 2014 07:24:50 +0000
Subject: [Bro] RSS and packet reordering
In-Reply-To: <A91FD3D8-5B37-4C3D-A79D-863E951D86F5@icir.org>
References: <53F44E1D.5040009@netclean.com>
	<A91FD3D8-5B37-4C3D-A79D-863E951D86F5@icir.org>
Message-ID: <53FC35C7.5090703@netclean.com>

Hello Seth,

Thanks for the answer.

I'm using pfring with rss rehashing so it's bidirectional.

 From bor's perspective there is only a tap device with one queue . I'm 
developing a program that basically acts as a filter between the network 
stream and bro, this program uses RSS to be able to cope with 10GBit/s. 
Bro only sees a small fraction of the entire network stream.

I would like to disable the RSS which is working extremely well. 
Unfortunately this gives me performance issues since I'm not able to 
process all the traffic.

So potential solutions I'm investigating.
1. In some way mitigate the reordering issue at network card/pfring 
level without disabling RSS, have so far not found a solution for this.
2. Tell bro to reorder the packets, I guess bro already does this but 
gives up if the packets are to much out of order. Is there any 
parameters I can change to tell bro to try harder?
3. One option is of cause to reorder the packets myself before sending 
them to bro.


Thanks
Bj?rn

(DoNotAddDisclaimer)




On 2014-08-25 16:06, Seth Hall wrote:
> On Aug 20, 2014, at 3:30 AM, Bj?rn Samvik <Bjorn.Samvik at netclean.com> wrote:
>
>> Anyone have experience with RSS and have any ideas how the packet
>> reordering issue can be mitigated?
> 
> Are you using straight RSS without any modifications?  By default the Toeplitz hash that RSS uses is not a bidirectional hash.  i.e., each flow of a connection ends up on a different NIC queue.  As a general rule though, you should try to avoid anything that could result in packet reordering.
>
> Another question, how are you making Bro attach to the various NIC hardware queues?
>
>    .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>


From seth at icir.org  Tue Aug 26 05:54:19 2014
From: seth at icir.org (Seth Hall)
Date: Tue, 26 Aug 2014 08:54:19 -0400
Subject: [Bro] RSS and packet reordering
In-Reply-To: <53FC35C7.5090703@netclean.com>
References: <53F44E1D.5040009@netclean.com>
	<A91FD3D8-5B37-4C3D-A79D-863E951D86F5@icir.org>
	<53FC35C7.5090703@netclean.com>
Message-ID: <7B41E292-BB15-4F91-B386-D4D295A571A0@icir.org>


On Aug 26, 2014, at 3:24 AM, Bj?rn Samvik <Bjorn.Samvik at netclean.com> wrote:

> I'm using pfring with rss rehashing so it's bidirectional.

Ahh, that explains a lot.

> From bor's perspective there is only a tap device with one queue . I'm 
> developing a program that basically acts as a filter between the network 
> stream and bro, this program uses RSS to be able to cope with 10GBit/s. 
> Bro only sees a small fraction of the entire network stream.

You may want to wait a bit on this. :)  

We should be announcing a new tool soon that does this and is *very* flexible.  It uses netmap to get packets into user space and then you essentially get "packet bricks" to build your own packet handling pipeline, very similar to the "Click! Software Router" if you're familiar with that.  Our intern this summer has been working really hard on getting this tool ready and I think that everyone is really going to like it.

> 1. In some way mitigate the reordering issue at network card/pfring 
> level without disabling RSS, have so far not found a solution for this.

With the way that PF_Ring is doing this I don't think there is a solution for it.

> 2. Tell bro to reorder the packets, I guess bro already does this but 
> gives up if the packets are to much out of order. Is there any 
> parameters I can change to tell bro to try harder?

This is kind of mess and has a ton of edge cases since the packets arrive without timestamps we wouldn't even have a mechanism to reorder them.  The TCP reassembler ends up being how they're reordered but that can fail for a number of reasons due to the reordering.

> 3. One option is of cause to reorder the packets myself before sending 
> them to bro.

Yes, but you're in the position where you still wouldn't know how to reorder them without timestamps.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140826/9a894943/attachment.bin 

From jessebowling at gmail.com  Tue Aug 26 07:45:45 2014
From: jessebowling at gmail.com (Jesse Bowling)
Date: Tue, 26 Aug 2014 10:45:45 -0400
Subject: [Bro] RSS and packet reordering
In-Reply-To: <7B41E292-BB15-4F91-B386-D4D295A571A0@icir.org>
References: <53F44E1D.5040009@netclean.com>
	<A91FD3D8-5B37-4C3D-A79D-863E951D86F5@icir.org>
	<53FC35C7.5090703@netclean.com>
	<7B41E292-BB15-4F91-B386-D4D295A571A0@icir.org>
Message-ID: <99C7194A-AC6D-43D1-9EEA-E32EAA8C2999@gmail.com>


On Aug 26, 2014, at 8:54 AM, Seth Hall <seth at icir.org> wrote:

> We should be announcing a new tool soon that does this and is *very* flexible.  It uses netmap to get packets into user space and then you essentially get "packet bricks" to build your own packet handling pipeline, very similar to the "Click! Software Router" if you're familiar with that.  Our intern this summer has been working really hard on getting this tool ready and I think that everyone is really going to like it.

Interesting! Will this tool be flexible enough to also feed packets to other programs on the same box such as argus, snort, etc?

Cheers,

Jesse
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 204 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140826/d0b433cc/attachment.bin 

From seth at icir.org  Tue Aug 26 07:55:32 2014
From: seth at icir.org (Seth Hall)
Date: Tue, 26 Aug 2014 10:55:32 -0400
Subject: [Bro] RSS and packet reordering
In-Reply-To: <99C7194A-AC6D-43D1-9EEA-E32EAA8C2999@gmail.com>
References: <53F44E1D.5040009@netclean.com>
	<A91FD3D8-5B37-4C3D-A79D-863E951D86F5@icir.org>
	<53FC35C7.5090703@netclean.com>
	<7B41E292-BB15-4F91-B386-D4D295A571A0@icir.org>
	<99C7194A-AC6D-43D1-9EEA-E32EAA8C2999@gmail.com>
Message-ID: <67BA89CA-029E-4147-B2B5-4FDF07786CC7@icir.org>


On Aug 26, 2014, at 10:45 AM, Jesse Bowling <jessebowling at gmail.com> wrote:

> Interesting! Will this tool be flexible enough to also feed packets to other programs on the same box such as argus, snort, etc?

Yep, that was a specific design requirement.  It uses the newish netmap feature "pipes" to give you specially named interfaces that you can open in anything using the netmap libpcap support.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140826/2e591aa3/attachment.bin 

From dngr7512 at gmail.com  Wed Aug 27 10:29:27 2014
From: dngr7512 at gmail.com (daniel nagar)
Date: Wed, 27 Aug 2014 20:29:27 +0300
Subject: [Bro] connecting to bro with broccoli
Message-ID: <CAAa3MOdKwDTR1mHo=PvXfrtZPziU0wC8g_8eyUqiUZ7cVZ19Vg@mail.gmail.com>

Hello, new here
I'm using bro 2.2 and I connect to bro using broccoli to receive events.
I can manage connecting to bro-worker and receive events, not sure if it's
the correct way to receive event from bro but connecting to the manager
port didn't retrieve any event whatsoever,
the problem is that when I receive events at speeds higher than 2Mbps the
parent of the bro-worker (not the broccoli application) memory expands
rapidly and can reach 10Gb in a minute.
Disconnecting the broccoli application immediately frees all memory of the
worker (10Gb to 100Mb in less than a second).

Daniel.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140827/065a99de/attachment.html 

From jxbatchelor at gmail.com  Wed Aug 27 14:38:29 2014
From: jxbatchelor at gmail.com (Jason Batchelor)
Date: Wed, 27 Aug 2014 16:38:29 -0500
Subject: [Bro] File Extraction Directory
Message-ID: <CANWz5X9+TgqcNWQPv=DcwiinCD7OoqsDwjujNFg3UVS8+4QOQg@mail.gmail.com>

Hello all:

Quick question, can you change the default file extraction directory for
files being extracted in a script. After some poking I came across where
this was specified in /opt/bro/share/bro/base/files/extract/main.bro with
the following:

export {
   ## The prefix where files are extracted to.
   const prefix = "./extract_files/" &redef;
If I try to do something like this in my script:

# Set extraction folder
redef prefix = "/var/opt/bro/spool/extract_files";
I am met with the following:
error in ./scripts/file-ext.bro, line 22: "redef" used but not previously
defined (prefix)
internal warning in ./scripts/file-ext.bro, line 22: Can't document redef
of prefix, identifier lookup failed
Can someone help me understand how to define this attribute or otherwise
influence where the files are extracted to? I would rather not manually
define it in the main.bro file.

Thanks,
Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140827/ca79b753/attachment.html 

From jsiwek at illinois.edu  Wed Aug 27 15:01:47 2014
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Wed, 27 Aug 2014 22:01:47 +0000
Subject: [Bro] File Extraction Directory
In-Reply-To: <CANWz5X9+TgqcNWQPv=DcwiinCD7OoqsDwjujNFg3UVS8+4QOQg@mail.gmail.com>
References: <CANWz5X9+TgqcNWQPv=DcwiinCD7OoqsDwjujNFg3UVS8+4QOQg@mail.gmail.com>
Message-ID: <B588D93A-6543-4854-9774-AF58177E20FE@illinois.edu>


On Aug 27, 2014, at 4:38 PM, Jason Batchelor <jxbatchelor at gmail.com> wrote:

> redef prefix = "/var/opt/bro/spool/extract_files?;

It?s missing the namespace scoping, try using ?FileExtract::prefix?.

- Jon


From johanna at icir.org  Wed Aug 27 15:07:50 2014
From: johanna at icir.org (Johanna Amann)
Date: Wed, 27 Aug 2014 15:07:50 -0700
Subject: [Bro] File Extraction Directory
In-Reply-To: <CANWz5X9+TgqcNWQPv=DcwiinCD7OoqsDwjujNFg3UVS8+4QOQg@mail.gmail.com>
References: <CANWz5X9+TgqcNWQPv=DcwiinCD7OoqsDwjujNFg3UVS8+4QOQg@mail.gmail.com>
Message-ID: <AC8715C1-A7E1-4A2A-B013-93E90D5CB5A9@icir.org>

Hi,

redef FileExtract::prefix = "/var/opt/bro/spool/extract_files/"; should work.

Johanna

On 27 Aug 2014, at 14:38, Jason Batchelor wrote:

> Hello all:
>
> Quick question, can you change the default file extraction directory for
> files being extracted in a script. After some poking I came across where
> this was specified in /opt/bro/share/bro/base/files/extract/main.bro with
> the following:
>
> export {
>  ## The prefix where files are extracted to.
>  const prefix = "./extract_files/" &redef;
> If I try to do something like this in my script:
>
> # Set extraction folder
> redef prefix = "/var/opt/bro/spool/extract_files";
> I am met with the following:
> error in ./scripts/file-ext.bro, line 22: "redef" used but not previously
> defined (prefix)
> internal warning in ./scripts/file-ext.bro, line 22: Can't document redef
> of prefix, identifier lookup failed
> Can someone help me understand how to define this attribute or otherwise
> influence where the files are extracted to? I would rather not manually
> define it in the main.bro file.
>
> Thanks,
> Jason
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From inetjunkmail at gmail.com  Thu Aug 28 07:50:11 2014
From: inetjunkmail at gmail.com (inetjunkmail)
Date: Thu, 28 Aug 2014 10:50:11 -0400
Subject: [Bro] Adding options to bro managed by broctl
Message-ID: <CAAJ38WXmoxcOX0c1Ci1eDWqi6QXq2qxayTDfnC+Fd1W=caU3Xg@mail.gmail.com>

We'd like to start bro with a tcpdump filter of, for example, -f 'net
1.0.0.0/24 or port 443'.  We use broctl to manage the bro process.  Where
is the appropriate place to add the desired filter so that broctl appends
it to any instantiation of a bro process?

Thanks for your help
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140828/c449348a/attachment.html 

From jlay at slave-tothe-box.net  Thu Aug 28 08:07:49 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 28 Aug 2014 09:07:49 -0600
Subject: [Bro] Adding options to bro managed by broctl
In-Reply-To: <CAAJ38WXmoxcOX0c1Ci1eDWqi6QXq2qxayTDfnC+Fd1W=caU3Xg@mail.gmail.com>
References: <CAAJ38WXmoxcOX0c1Ci1eDWqi6QXq2qxayTDfnC+Fd1W=caU3Xg@mail.gmail.com>
Message-ID: <c1b78eb4fada872cc6fd4a69b5615b07@localhost>

On 2014-08-28 08:50, inetjunkmail wrote:
> Wed like to start bro with a tcpdump filter of, for example, -f net
> 1.0.0.0/24 [1] or port 443. ?We use broctl to manage the bro process.
> ?Where is the appropriate place to add the desired filter so that
> broctl appends it to any instantiation of a bro process?
>
> Thanks for your help
>
> Links:
> ------
> [1] http://1.0.0.0/24

Add:

broargs = -f 'net 1.0.0.0/24 or port 443'

to your broctl.cfg file.

James


From monahbaki at gmail.com  Thu Aug 28 10:18:31 2014
From: monahbaki at gmail.com (Monah Baki)
Date: Thu, 28 Aug 2014 13:18:31 -0400
Subject: [Bro] Running racluster but with a time frame
Message-ID: <CALP3=x8tLPDaXtkkTNFVOaiHWC0GuPhfR-LcdfwdC2Xa626bXA@mail.gmail.com>

Hi all,


I need to run the following command "racluster -r
argus.2014.08.19.10.30.01.0.gz -s stime daddr -s stime saddr daddr
trans" but to display only events from 10:00am to 10:15am.


How can I accomplish this?



Thanks
Monah


From jessebowling at gmail.com  Thu Aug 28 10:50:34 2014
From: jessebowling at gmail.com (Jesse Bowling)
Date: Thu, 28 Aug 2014 13:50:34 -0400
Subject: [Bro] Running racluster but with a time frame
In-Reply-To: <CALP3=x8tLPDaXtkkTNFVOaiHWC0GuPhfR-LcdfwdC2Xa626bXA@mail.gmail.com>
References: <CALP3=x8tLPDaXtkkTNFVOaiHWC0GuPhfR-LcdfwdC2Xa626bXA@mail.gmail.com>
Message-ID: <528B2C4B-3BDE-41B7-9AA7-67B997213A8A@gmail.com>

Hi Monah,

You probably meant to email the argus listserv, or possibly the security onion listserv...But since you asked, you should be able to:

> racluster -r argus.2014.08.19.10.30.01.0.gz -s stime daddr -s stime saddr daddr trans -t 10h+15m

Much more detail can be found in the man page for ra...It?s quite a flexible option.

Cheers,

Jesse

On Aug 28, 2014, at 1:18 PM, Monah Baki <monahbaki at gmail.com> wrote:

> Hi all,
> 
> 
> I need to run the following command "racluster -r
> argus.2014.08.19.10.30.01.0.gz -s stime daddr -s stime saddr daddr
> trans" but to display only events from 10:00am to 10:15am.
> 
> 
> How can I accomplish this?
> 
> 
> 
> Thanks
> Monah
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From blackhole.em at gmail.com  Thu Aug 28 14:52:12 2014
From: blackhole.em at gmail.com (Joe Blow)
Date: Thu, 28 Aug 2014 17:52:12 -0400
Subject: [Bro] Fwd: Configure error linking libpcap and pthread
In-Reply-To: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>
References: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>
Message-ID: <CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>

Hey all,

I'm having a really tough time getting PF_RING working with Bro in a
threaded fashion.  I have PF_RING compiled and working fine (tcpdump test
works fine with Transparent mode = 2):

PF_RING Version          : 6.0.2 ($Revision: exported$)
Total rings              : 0

Standard (non DNA) Options
Ring slots               : 4096
Slot version             : 16
Capture TX               : No [RX only]
IP Defragment            : No
Socket Mode              : Standard
Transparent mode         : No [mode 2]
Total plugins            : 0
Cluster Fragment Queue   : 0
Cluster Fragment Discard : 0

Bro is version 2.3 (sha1 - 79397be0e351165d44047b044d29b5e6580532cc
bro-2.3.tar.gz)
OS is CentOS 6.4 running 2.6.32-358.11.1.el6.x86_64

When I try and configure against my PF_RING libraries, I get this:

./configure --with-pcap=/opt/pfring
Build Directory : build
Source Directory: /root/src/bro-2.3
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found sed: /bin/sed
-- Found Perl: /usr/bin/perl
-- Found FLEX: 2.5.35
-- Found BISON: /usr/bin/bison
-- Found PCAP: /opt/pfring/lib/libpcap.so
-- Performing Test PCAP_LINKS_SOLO
-- Performing Test PCAP_LINKS_SOLO - Failed
-- Looking for include files CMAKE_HAVE_PTHREAD_H
-- Looking for include files CMAKE_HAVE_PTHREAD_H - found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Performing Test PCAP_NEEDS_THREADS
-- Performing Test PCAP_NEEDS_THREADS - Failed
CMake Error at cmake/FindPCAP.cmake:61 (message):
  Couldn't determine how to link against libpcap
Call Stack (most recent call first):
  cmake/FindRequiredPackage.cmake:26 (find_package)
  CMakeLists.txt:52 (FindRequiredPackage)


-- Configuring incomplete, errors occurred!

I'm banging my head against this, but I believe this is because bro can't
find the threading library to link to.  Could someone point me in the right
direction?  Do I need other threading libraries? Static linking?

Cheers,

JB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140828/d51d2231/attachment.html 

From jlay at slave-tothe-box.net  Thu Aug 28 14:59:49 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 28 Aug 2014 15:59:49 -0600
Subject: [Bro] Fwd: Configure error linking libpcap and pthread
In-Reply-To: <CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>
References: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>
	<CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>
Message-ID: <d2bce584e1d9ca8c6c74137976d795fc@localhost>

On 2014-08-28 15:52, Joe Blow wrote:
> Hey all,
>
> Im having a really tough time getting PF_RING working with Bro in a
> threaded fashion.? I have PF_RING compiled and working fine (tcpdump
> test works fine with Transparent mode = 2):
>
> PF_RING Version????????? : 6.0.2 ($Revision: exported$)
> Total rings????????????? : 0
>
> Standard (non DNA) Options
> Ring slots?????????????? : 4096
> Slot version???????????? : 16
> Capture TX?????????????? : No [RX only]
>  IP Defragment??????????? : No
> Socket Mode????????????? : Standard
> Transparent mode???????? : No [mode 2]
> Total plugins??????????? : 0
> Cluster Fragment Queue?? : 0
> Cluster Fragment Discard : 0
>
>  Bro is version 2.3 (sha1 - 79397be0e351165d44047b044d29b5e6580532cc?
> bro-2.3.tar.gz)
>
> OS is CentOS 6.4 running 2.6.32-358.11.1.el6.x86_64
>
> When I try and configure against my PF_RING libraries, I get this:
>
> ./configure --with-pcap=/opt/pfring
> Build Directory : build
>  Source Directory: /root/src/bro-2.3
> -- The C compiler identification is GNU
> -- The CXX compiler identification is GNU
> -- Check for working C compiler: /usr/bin/gcc
> -- Check for working C compiler: /usr/bin/gcc -- works
>  -- Detecting C compiler ABI info
> -- Detecting C compiler ABI info - done
> -- Check for working CXX compiler: /usr/bin/c++
> -- Check for working CXX compiler: /usr/bin/c++ -- works
> -- Detecting CXX compiler ABI info
>  -- Detecting CXX compiler ABI info - done
> -- Found sed: /bin/sed
> -- Found Perl: /usr/bin/perl
> -- Found FLEX: 2.5.35
> -- Found BISON: /usr/bin/bison
> -- Found PCAP: /opt/pfring/lib/libpcap.so
> -- Performing Test PCAP_LINKS_SOLO
>  -- Performing Test PCAP_LINKS_SOLO - Failed
> -- Looking for include files CMAKE_HAVE_PTHREAD_H
> -- Looking for include files CMAKE_HAVE_PTHREAD_H - found
> -- Looking for pthread_create in pthreads
> -- Looking for pthread_create in pthreads - not found
>  -- Looking for pthread_create in pthread
> -- Looking for pthread_create in pthread - found
> -- Found Threads: TRUE
> -- Performing Test PCAP_NEEDS_THREADS
> -- Performing Test PCAP_NEEDS_THREADS - Failed
> CMake Error at cmake/FindPCAP.cmake:61 (message):
>  ? Couldnt determine how to link against libpcap
> Call Stack (most recent call first):
> ? cmake/FindRequiredPackage.cmake:26 (find_package)
> ? CMakeLists.txt:52 (FindRequiredPackage)
>
> -- Configuring incomplete, errors occurred!
>
> Im banging my head against this, but I believe this is because bro
> cant find the threading library to link to.? Could someone point me
> in the right direction?? Do I need other threading libraries? Static
> linking?
>
> Cheers,
>
> JB

Did you follow this:

http://www.bro.org/documentation/load-balancing.html

James


From blackhole.em at gmail.com  Thu Aug 28 15:27:48 2014
From: blackhole.em at gmail.com (Joe Blow)
Date: Thu, 28 Aug 2014 18:27:48 -0400
Subject: [Bro] Fwd: Configure error linking libpcap and pthread
In-Reply-To: <d2bce584e1d9ca8c6c74137976d795fc@localhost>
References: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>
	<CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>
	<d2bce584e1d9ca8c6c74137976d795fc@localhost>
Message-ID: <CAMCSCoWuTsgnELPznEbsbsbLE-CwfucmBDdbYfjMseMYK7moKw@mail.gmail.com>

Similar, except I actually use the PF_RING_aware drivers, and transparent
mode = 2.  So before I perform step 1 I make and make install in
PF_RING_aware/non-ZC-drivers/2.6.x/broadcom/netxtreme2-5.2.50/bnx2.  Then
load the module with modprobe, then I compile PF_RING without issues, and
compile tcpdump to work on the new PF_RING.  That works fine with tcpdump,
but I can't seem to compile Bro.

Other than that nuance (and the fact that i'm running PF_RING 6.0.2, as
mentioned above, not 5.6.2 like the guide) it should be the same.

Any ideas?

Cheers,

JB


On Thu, Aug 28, 2014 at 5:59 PM, James Lay <jlay at slave-tothe-box.net> wrote:

> On 2014-08-28 15:52, Joe Blow wrote:
> > Hey all,
> >
> > Im having a really tough time getting PF_RING working with Bro in a
> > threaded fashion.  I have PF_RING compiled and working fine (tcpdump
> > test works fine with Transparent mode = 2):
> >
> > PF_RING Version          : 6.0.2 ($Revision: exported$)
> > Total rings              : 0
> >
> > Standard (non DNA) Options
> > Ring slots               : 4096
> > Slot version             : 16
> > Capture TX               : No [RX only]
> >  IP Defragment            : No
> > Socket Mode              : Standard
> > Transparent mode         : No [mode 2]
> > Total plugins            : 0
> > Cluster Fragment Queue   : 0
> > Cluster Fragment Discard : 0
> >
> >  Bro is version 2.3 (sha1 - 79397be0e351165d44047b044d29b5e6580532cc
> > bro-2.3.tar.gz)
> >
> > OS is CentOS 6.4 running 2.6.32-358.11.1.el6.x86_64
> >
> > When I try and configure against my PF_RING libraries, I get this:
> >
> > ./configure --with-pcap=/opt/pfring
> > Build Directory : build
> >  Source Directory: /root/src/bro-2.3
> > -- The C compiler identification is GNU
> > -- The CXX compiler identification is GNU
> > -- Check for working C compiler: /usr/bin/gcc
> > -- Check for working C compiler: /usr/bin/gcc -- works
> >  -- Detecting C compiler ABI info
> > -- Detecting C compiler ABI info - done
> > -- Check for working CXX compiler: /usr/bin/c++
> > -- Check for working CXX compiler: /usr/bin/c++ -- works
> > -- Detecting CXX compiler ABI info
> >  -- Detecting CXX compiler ABI info - done
> > -- Found sed: /bin/sed
> > -- Found Perl: /usr/bin/perl
> > -- Found FLEX: 2.5.35
> > -- Found BISON: /usr/bin/bison
> > -- Found PCAP: /opt/pfring/lib/libpcap.so
> > -- Performing Test PCAP_LINKS_SOLO
> >  -- Performing Test PCAP_LINKS_SOLO - Failed
> > -- Looking for include files CMAKE_HAVE_PTHREAD_H
> > -- Looking for include files CMAKE_HAVE_PTHREAD_H - found
> > -- Looking for pthread_create in pthreads
> > -- Looking for pthread_create in pthreads - not found
> >  -- Looking for pthread_create in pthread
> > -- Looking for pthread_create in pthread - found
> > -- Found Threads: TRUE
> > -- Performing Test PCAP_NEEDS_THREADS
> > -- Performing Test PCAP_NEEDS_THREADS - Failed
> > CMake Error at cmake/FindPCAP.cmake:61 (message):
> >    Couldnt determine how to link against libpcap
> > Call Stack (most recent call first):
> >   cmake/FindRequiredPackage.cmake:26 (find_package)
> >   CMakeLists.txt:52 (FindRequiredPackage)
> >
> > -- Configuring incomplete, errors occurred!
> >
> > Im banging my head against this, but I believe this is because bro
> > cant find the threading library to link to.  Could someone point me
> > in the right direction?  Do I need other threading libraries? Static
> > linking?
> >
> > Cheers,
> >
> > JB
>
> Did you follow this:
>
> http://www.bro.org/documentation/load-balancing.html
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140828/33ac622e/attachment.html 

From ben.bt.wood at gmail.com  Thu Aug 28 15:40:41 2014
From: ben.bt.wood at gmail.com (ben.bt.wood at gmail.com)
Date: Thu, 28 Aug 2014 18:40:41 -0400
Subject: [Bro] Fwd: Configure error linking libpcap and pthread
In-Reply-To: <CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>
References: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>
	<CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>
Message-ID: <53ffaff9.094e320a.4f66.17f8@mx.google.com>

I once had an issue with this. My bad workaround was to remove any non PF_RING libpcap. I was noticing some problems with the compiler choosing the wrong one. So using the big hammer, I removed it.

-----Original Message-----
From: "Joe Blow" <blackhole.em at gmail.com>
Sent: ?8/?28/?2014 5:54 PM
To: "bro at bro-ids.org" <bro at bro-ids.org>
Subject: [Bro] Fwd: Configure error linking libpcap and pthread

Hey all,

I'm having a really tough time getting PF_RING working with Bro in a threaded fashion.  I have PF_RING compiled and working fine (tcpdump test works fine with Transparent mode = 2):

PF_RING Version          : 6.0.2 ($Revision: exported$)
Total rings              : 0

Standard (non DNA) Options
Ring slots               : 4096
Slot version             : 16
Capture TX               : No [RX only]
IP Defragment            : No
Socket Mode              : Standard
Transparent mode         : No [mode 2]
Total plugins            : 0
Cluster Fragment Queue   : 0
Cluster Fragment Discard : 0


Bro is version 2.3 (sha1 - 79397be0e351165d44047b044d29b5e6580532cc  bro-2.3.tar.gz)

OS is CentOS 6.4 running 2.6.32-358.11.1.el6.x86_64



When I try and configure against my PF_RING libraries, I get this:

./configure --with-pcap=/opt/pfring
Build Directory : build
Source Directory: /root/src/bro-2.3
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found sed: /bin/sed
-- Found Perl: /usr/bin/perl
-- Found FLEX: 2.5.35
-- Found BISON: /usr/bin/bison
-- Found PCAP: /opt/pfring/lib/libpcap.so
-- Performing Test PCAP_LINKS_SOLO
-- Performing Test PCAP_LINKS_SOLO - Failed
-- Looking for include files CMAKE_HAVE_PTHREAD_H
-- Looking for include files CMAKE_HAVE_PTHREAD_H - found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Performing Test PCAP_NEEDS_THREADS
-- Performing Test PCAP_NEEDS_THREADS - Failed
CMake Error at cmake/FindPCAP.cmake:61 (message):
  Couldn't determine how to link against libpcap
Call Stack (most recent call first):
  cmake/FindRequiredPackage.cmake:26 (find_package)
  CMakeLists.txt:52 (FindRequiredPackage)


-- Configuring incomplete, errors occurred!


I'm banging my head against this, but I believe this is because bro can't find the threading library to link to.  Could someone point me in the right direction?  Do I need other threading libraries? Static linking?


Cheers,

JB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140828/b0d9244b/attachment.html 

From jlay at slave-tothe-box.net  Thu Aug 28 15:42:33 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 28 Aug 2014 16:42:33 -0600
Subject: [Bro] Fwd: Configure error linking libpcap and pthread
In-Reply-To: <CAMCSCoWuTsgnELPznEbsbsbLE-CwfucmBDdbYfjMseMYK7moKw@mail.gmail.com>
References: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>
	<CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>
	<d2bce584e1d9ca8c6c74137976d795fc@localhost>
	<CAMCSCoWuTsgnELPznEbsbsbLE-CwfucmBDdbYfjMseMYK7moKw@mail.gmail.com>
Message-ID: <a89c4ad312bd827d3151c609b4d288f8@localhost>

On 2014-08-28 16:27, Joe Blow wrote:
> Similar, except I actually use the PF_RING_aware drivers, and
> transparent mode = 2.? So before I perform step 1 I make and make
> install in
> PF_RING_aware/non-ZC-drivers/2.6.x/broadcom/netxtreme2-5.2.50/bnx2.?
> Then load the module with modprobe, then I compile PF_RING without
> issues, and compile tcpdump to work on the new PF_RING.? That works
> fine with tcpdump, but I cant seem to compile Bro.
>
> Other than that nuance (and the fact that im running PF_RING 6.0.2, 
> as
> mentioned above, not 5.6.2 like the guide) it should be the same.
>
> Any ideas?
>
> Cheers,
>
> JB
>
> On Thu, Aug 28, 2014 at 5:59 PM, James Lay <jlay at slave-tothe-box.net
> [4]> wrote:
>
>> On 2014-08-28 15:52, Joe Blow wrote:
>> > Hey all,
>>
>>>
>> > Im having a really tough time getting PF_RING working with Bro in
>> a
>> > threaded fashion.? I have PF_RING compiled and working fine
>> (tcpdump
>> > test works fine with Transparent mode = 2):
>> >
>> > PF_RING Version????????? : 6.0.2 ($Revision: exported$)
>> > Total rings????????????? : 0
>> >
>> > Standard (non DNA) Options
>> > Ring slots?????????????? : 4096
>> > Slot version???????????? : 16
>> > Capture TX?????????????? : No [RX only]
>> >? IP Defragment??????????? : No
>> > Socket Mode????????????? : Standard
>> > Transparent mode???????? : No [mode 2]
>> > Total plugins??????????? : 0
>> > Cluster Fragment Queue?? : 0
>> > Cluster Fragment Discard : 0
>> >
>> >? Bro is version 2.3 (sha1 -
>> 79397be0e351165d44047b044d29b5e6580532cc?
>> > bro-2.3.tar.gz)
>> >
>> > OS is CentOS 6.4 running 2.6.32-358.11.1.el6.x86_64
>> >
>> > When I try and configure against my PF_RING libraries, I get
>> this:
>> >
>> > ./configure --with-pcap=/opt/pfring
>> > Build Directory : build
>> >? Source Directory: /root/src/bro-2.3
>> > -- The C compiler identification is GNU
>> > -- The CXX compiler identification is GNU
>> > -- Check for working C compiler: /usr/bin/gcc
>> > -- Check for working C compiler: /usr/bin/gcc -- works
>> >? -- Detecting C compiler ABI info
>> > -- Detecting C compiler ABI info - done
>> > -- Check for working CXX compiler: /usr/bin/c++
>> > -- Check for working CXX compiler: /usr/bin/c++ -- works
>> > -- Detecting CXX compiler ABI info
>> >? -- Detecting CXX compiler ABI info - done
>> > -- Found sed: /bin/sed
>> > -- Found Perl: /usr/bin/perl
>> > -- Found FLEX: 2.5.35
>> > -- Found BISON: /usr/bin/bison
>> > -- Found PCAP: /opt/pfring/lib/libpcap.so
>> > -- Performing Test PCAP_LINKS_SOLO
>> >? -- Performing Test PCAP_LINKS_SOLO - Failed
>> > -- Looking for include files CMAKE_HAVE_PTHREAD_H
>> > -- Looking for include files CMAKE_HAVE_PTHREAD_H - found
>> > -- Looking for pthread_create in pthreads
>> > -- Looking for pthread_create in pthreads - not found
>> >? -- Looking for pthread_create in pthread
>> > -- Looking for pthread_create in pthread - found
>> > -- Found Threads: TRUE
>> > -- Performing Test PCAP_NEEDS_THREADS
>> > -- Performing Test PCAP_NEEDS_THREADS - Failed
>> > CMake Error at cmake/FindPCAP.cmake:61 (message):
>> >? ? Couldnt determine how to link against libpcap
>>
>>> Call Stack (most recent call first):
>> > ? cmake/FindRequiredPackage.cmake:26 (find_package)
>> > ? CMakeLists.txt:52 (FindRequiredPackage)
>> >
>> > -- Configuring incomplete, errors occurred!
>> >
>> > Im banging my head against this, but I believe this is because
>> bro
>> > cant find the threading library to link to.? Could someone point
>> me
>>
>>> in the right direction?? Do I need other threading libraries?
>> Static
>> > linking?
>> >
>> > Cheers,
>> >
>> > JB
>>
>> Did you follow this:
>>
>> http://www.bro.org/documentation/load-balancing.html [1]
>>
>> James

Hrmm...I just tested this now with PF_RING 6.0.1:

Build Directory : build
Source Directory: /home/dev/bro-2.3
-- Found sed: /bin/sed
-- Found Perl: /usr/bin/perl (found version "5.18.2")
-- Found FLEX: 2.5.35
-- Found BISON: /usr/bin/bison
-- Found PCAP: /opt/pfring/lib/libpcap.so
-- Performing Test PCAP_LINKS_SOLO
-- Performing Test PCAP_LINKS_SOLO - Success
-- Looking for pcap_get_pfring_id
-- Looking for pcap_get_pfring_id - found
etc...
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE


I didn't see the PCAP_NEEDS_THREADS however.  Machine info:

Linux ubuntu-test 3.13.0-34-generic #60-Ubuntu SMP Wed Aug 13 15:49:09 
UTC 2014 i686 i686 i686 GNU/Linux

Hope that's at least a little more intel.

James






From blackhole.em at gmail.com  Thu Aug 28 16:06:16 2014
From: blackhole.em at gmail.com (Joe Blow)
Date: Thu, 28 Aug 2014 19:06:16 -0400
Subject: [Bro] Fwd: Configure error linking libpcap and pthread
In-Reply-To: <CAK8kjrCXx76+SS8xuhEnWuM2CVDxy9HGjMrU3YhYtcPAgVB7eA@mail.gmail.com>
References: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>
	<CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>
	<CAK8kjrCXx76+SS8xuhEnWuM2CVDxy9HGjMrU3YhYtcPAgVB7eA@mail.gmail.com>
Message-ID: <CAMCSCoVZrJ=W05Ly+fXKhR4X2QgZPkohPAKFOf_Zj9TZg02ZZw@mail.gmail.com>

Doug Burks was quick to point out that i didn't export LIBS or LDFLAGS.

I would have NEVER guessed this... thanks a thousand times over for this
tidbit.  Configure finished just fine.  Making now.  Will update once i've
got it up and load balanced.

<code>

export LDFLAGS="-Wl,--no-as-needed -lrt"

export LIBS="-lrt -lnuma"
</code>

Cheers,

JB


On Thu, Aug 28, 2014 at 6:52 PM, Doug Burks <doug.burks at gmail.com> wrote:

> Hi Joe,
>
> When I packaged Bro 2.3 and PF_RING 6.0.2, I had to do the following:
>
> export LDFLAGS := $(LDFLAGS) -Wl,--no-as-needed -lrt
> export LIBS := $(LIBS) -lrt -lnuma
>
> Depending on your configuration, you may also need to include
> -lpthread in your LIBS.
>
> On Thu, Aug 28, 2014 at 5:52 PM, Joe Blow <blackhole.em at gmail.com> wrote:
> > Hey all,
> >
> > I'm having a really tough time getting PF_RING working with Bro in a
> > threaded fashion.  I have PF_RING compiled and working fine (tcpdump test
> > works fine with Transparent mode = 2):
> >
> > PF_RING Version          : 6.0.2 ($Revision: exported$)
> > Total rings              : 0
> >
> > Standard (non DNA) Options
> > Ring slots               : 4096
> > Slot version             : 16
> > Capture TX               : No [RX only]
> > IP Defragment            : No
> > Socket Mode              : Standard
> > Transparent mode         : No [mode 2]
> > Total plugins            : 0
> > Cluster Fragment Queue   : 0
> > Cluster Fragment Discard : 0
> >
> > Bro is version 2.3 (sha1 - 79397be0e351165d44047b044d29b5e6580532cc
> > bro-2.3.tar.gz)
> > OS is CentOS 6.4 running 2.6.32-358.11.1.el6.x86_64
> >
> > When I try and configure against my PF_RING libraries, I get this:
> >
> > ./configure --with-pcap=/opt/pfring
> > Build Directory : build
> > Source Directory: /root/src/bro-2.3
> > -- The C compiler identification is GNU
> > -- The CXX compiler identification is GNU
> > -- Check for working C compiler: /usr/bin/gcc
> > -- Check for working C compiler: /usr/bin/gcc -- works
> > -- Detecting C compiler ABI info
> > -- Detecting C compiler ABI info - done
> > -- Check for working CXX compiler: /usr/bin/c++
> > -- Check for working CXX compiler: /usr/bin/c++ -- works
> > -- Detecting CXX compiler ABI info
> > -- Detecting CXX compiler ABI info - done
> > -- Found sed: /bin/sed
> > -- Found Perl: /usr/bin/perl
> > -- Found FLEX: 2.5.35
> > -- Found BISON: /usr/bin/bison
> > -- Found PCAP: /opt/pfring/lib/libpcap.so
> > -- Performing Test PCAP_LINKS_SOLO
> > -- Performing Test PCAP_LINKS_SOLO - Failed
> > -- Looking for include files CMAKE_HAVE_PTHREAD_H
> > -- Looking for include files CMAKE_HAVE_PTHREAD_H - found
> > -- Looking for pthread_create in pthreads
> > -- Looking for pthread_create in pthreads - not found
> > -- Looking for pthread_create in pthread
> > -- Looking for pthread_create in pthread - found
> > -- Found Threads: TRUE
> > -- Performing Test PCAP_NEEDS_THREADS
> > -- Performing Test PCAP_NEEDS_THREADS - Failed
> > CMake Error at cmake/FindPCAP.cmake:61 (message):
> >   Couldn't determine how to link against libpcap
> > Call Stack (most recent call first):
> >   cmake/FindRequiredPackage.cmake:26 (find_package)
> >   CMakeLists.txt:52 (FindRequiredPackage)
> >
> >
> > -- Configuring incomplete, errors occurred!
> >
> > I'm banging my head against this, but I believe this is because bro can't
> > find the threading library to link to.  Could someone point me in the
> right
> > direction?  Do I need other threading libraries? Static linking?
> >
> > Cheers,
> >
> > JB
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> --
> Doug Burks
> Need Security Onion Training or Commercial Support?
> http://securityonionsolutions.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140828/dc5fdfd1/attachment.html 

From blackhole.em at gmail.com  Fri Aug 29 07:53:20 2014
From: blackhole.em at gmail.com (Joe Blow)
Date: Fri, 29 Aug 2014 10:53:20 -0400
Subject: [Bro] Fwd: Configure error linking libpcap and pthread
In-Reply-To: <CAMCSCoVZrJ=W05Ly+fXKhR4X2QgZPkohPAKFOf_Zj9TZg02ZZw@mail.gmail.com>
References: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>
	<CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>
	<CAK8kjrCXx76+SS8xuhEnWuM2CVDxy9HGjMrU3YhYtcPAgVB7eA@mail.gmail.com>
	<CAMCSCoVZrJ=W05Ly+fXKhR4X2QgZPkohPAKFOf_Zj9TZg02ZZw@mail.gmail.com>
Message-ID: <CAMCSCoUA2OPb2kcpJ29oks_uE03QPd3HJ6FCfcrkLaf0oz1mtw@mail.gmail.com>

So i've gone and recompiled with PF_RING 6.  I have everything working and
using PF_RING correctly, but i'm still seeing packet loss (around 25% on a
400-450mb/s link).   I was only ever able to get Bro working with
"Transparent mode = 0" and not 2 or 1.  I might be doing something
completely wrong, but whenever i start BRO, i only ever see one thread
peaking at 100%. Here is my node configuration:

[worker-0]
type=worker
host=10.10.10.10
interface=eth3
lb_method=pf_ring
lb_procs=12

Any ideas as to why i'm only getting one thread seeing the bro traffic?
Excuse my ignorance.

Cheers,

JB


On Thu, Aug 28, 2014 at 7:06 PM, Joe Blow <blackhole.em at gmail.com> wrote:

> Doug Burks was quick to point out that i didn't export LIBS or LDFLAGS.
>
> I would have NEVER guessed this... thanks a thousand times over for this
> tidbit.  Configure finished just fine.  Making now.  Will update once i've
> got it up and load balanced.
>
> <code>
>
> export LDFLAGS="-Wl,--no-as-needed -lrt"
>
> export LIBS="-lrt -lnuma"
> </code>
>
> Cheers,
>
> JB
>
>
> On Thu, Aug 28, 2014 at 6:52 PM, Doug Burks <doug.burks at gmail.com> wrote:
>
>> Hi Joe,
>>
>> When I packaged Bro 2.3 and PF_RING 6.0.2, I had to do the following:
>>
>> export LDFLAGS := $(LDFLAGS) -Wl,--no-as-needed -lrt
>> export LIBS := $(LIBS) -lrt -lnuma
>>
>> Depending on your configuration, you may also need to include
>> -lpthread in your LIBS.
>>
>> On Thu, Aug 28, 2014 at 5:52 PM, Joe Blow <blackhole.em at gmail.com> wrote:
>> > Hey all,
>> >
>> > I'm having a really tough time getting PF_RING working with Bro in a
>> > threaded fashion.  I have PF_RING compiled and working fine (tcpdump
>> test
>> > works fine with Transparent mode = 2):
>> >
>> > PF_RING Version          : 6.0.2 ($Revision: exported$)
>> > Total rings              : 0
>> >
>> > Standard (non DNA) Options
>> > Ring slots               : 4096
>> > Slot version             : 16
>> > Capture TX               : No [RX only]
>> > IP Defragment            : No
>> > Socket Mode              : Standard
>> > Transparent mode         : No [mode 2]
>> > Total plugins            : 0
>> > Cluster Fragment Queue   : 0
>> > Cluster Fragment Discard : 0
>> >
>> > Bro is version 2.3 (sha1 - 79397be0e351165d44047b044d29b5e6580532cc
>> > bro-2.3.tar.gz)
>> > OS is CentOS 6.4 running 2.6.32-358.11.1.el6.x86_64
>> >
>> > When I try and configure against my PF_RING libraries, I get this:
>> >
>> > ./configure --with-pcap=/opt/pfring
>> > Build Directory : build
>> > Source Directory: /root/src/bro-2.3
>> > -- The C compiler identification is GNU
>> > -- The CXX compiler identification is GNU
>> > -- Check for working C compiler: /usr/bin/gcc
>> > -- Check for working C compiler: /usr/bin/gcc -- works
>> > -- Detecting C compiler ABI info
>> > -- Detecting C compiler ABI info - done
>> > -- Check for working CXX compiler: /usr/bin/c++
>> > -- Check for working CXX compiler: /usr/bin/c++ -- works
>> > -- Detecting CXX compiler ABI info
>> > -- Detecting CXX compiler ABI info - done
>> > -- Found sed: /bin/sed
>> > -- Found Perl: /usr/bin/perl
>> > -- Found FLEX: 2.5.35
>> > -- Found BISON: /usr/bin/bison
>> > -- Found PCAP: /opt/pfring/lib/libpcap.so
>> > -- Performing Test PCAP_LINKS_SOLO
>> > -- Performing Test PCAP_LINKS_SOLO - Failed
>> > -- Looking for include files CMAKE_HAVE_PTHREAD_H
>> > -- Looking for include files CMAKE_HAVE_PTHREAD_H - found
>> > -- Looking for pthread_create in pthreads
>> > -- Looking for pthread_create in pthreads - not found
>> > -- Looking for pthread_create in pthread
>> > -- Looking for pthread_create in pthread - found
>> > -- Found Threads: TRUE
>> > -- Performing Test PCAP_NEEDS_THREADS
>> > -- Performing Test PCAP_NEEDS_THREADS - Failed
>> > CMake Error at cmake/FindPCAP.cmake:61 (message):
>> >   Couldn't determine how to link against libpcap
>> > Call Stack (most recent call first):
>> >   cmake/FindRequiredPackage.cmake:26 (find_package)
>> >   CMakeLists.txt:52 (FindRequiredPackage)
>> >
>> >
>> > -- Configuring incomplete, errors occurred!
>> >
>> > I'm banging my head against this, but I believe this is because bro
>> can't
>> > find the threading library to link to.  Could someone point me in the
>> right
>> > direction?  Do I need other threading libraries? Static linking?
>> >
>> > Cheers,
>> >
>> > JB
>> >
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>>
>> --
>> Doug Burks
>> Need Security Onion Training or Commercial Support?
>> http://securityonionsolutions.com
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140829/1a0e3270/attachment.html 

From doug.burks at gmail.com  Fri Aug 29 07:58:27 2014
From: doug.burks at gmail.com (Doug Burks)
Date: Fri, 29 Aug 2014 10:58:27 -0400
Subject: [Bro] Fwd: Configure error linking libpcap and pthread
In-Reply-To: <CAMCSCoUA2OPb2kcpJ29oks_uE03QPd3HJ6FCfcrkLaf0oz1mtw@mail.gmail.com>
References: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>
	<CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>
	<CAK8kjrCXx76+SS8xuhEnWuM2CVDxy9HGjMrU3YhYtcPAgVB7eA@mail.gmail.com>
	<CAMCSCoVZrJ=W05Ly+fXKhR4X2QgZPkohPAKFOf_Zj9TZg02ZZw@mail.gmail.com>
	<CAMCSCoUA2OPb2kcpJ29oks_uE03QPd3HJ6FCfcrkLaf0oz1mtw@mail.gmail.com>
Message-ID: <CAK8kjrDqa3seYxdMoOu=KcWqcQ-B3Z-3jOv6vWG1H90TfNaZ6A@mail.gmail.com>

It's possible that Bro is not actually using PF_RING and is actually
falling back to standard libpcap.  Have you checked /proc/net/pf_ring/
to see if there is evidence of Bro using PF_RING?

On Fri, Aug 29, 2014 at 10:53 AM, Joe Blow <blackhole.em at gmail.com> wrote:
> So i've gone and recompiled with PF_RING 6.  I have everything working and
> using PF_RING correctly, but i'm still seeing packet loss (around 25% on a
> 400-450mb/s link).   I was only ever able to get Bro working with
> "Transparent mode = 0" and not 2 or 1.  I might be doing something
> completely wrong, but whenever i start BRO, i only ever see one thread
> peaking at 100%. Here is my node configuration:
>
> [worker-0]
> type=worker
> host=10.10.10.10
> interface=eth3
> lb_method=pf_ring
> lb_procs=12
>
> Any ideas as to why i'm only getting one thread seeing the bro traffic?
> Excuse my ignorance.
>
> Cheers,
>
> JB
>
>
> On Thu, Aug 28, 2014 at 7:06 PM, Joe Blow <blackhole.em at gmail.com> wrote:
>>
>> Doug Burks was quick to point out that i didn't export LIBS or LDFLAGS.
>>
>> I would have NEVER guessed this... thanks a thousand times over for this
>> tidbit.  Configure finished just fine.  Making now.  Will update once i've
>> got it up and load balanced.
>>
>> <code>
>>
>> export LDFLAGS="-Wl,--no-as-needed -lrt"
>>
>> export LIBS="-lrt -lnuma"
>>
>> </code>
>>
>> Cheers,
>>
>> JB
>>
>>
>> On Thu, Aug 28, 2014 at 6:52 PM, Doug Burks <doug.burks at gmail.com> wrote:
>>>
>>> Hi Joe,
>>>
>>> When I packaged Bro 2.3 and PF_RING 6.0.2, I had to do the following:
>>>
>>> export LDFLAGS := $(LDFLAGS) -Wl,--no-as-needed -lrt
>>> export LIBS := $(LIBS) -lrt -lnuma
>>>
>>> Depending on your configuration, you may also need to include
>>> -lpthread in your LIBS.
>>>
>>> On Thu, Aug 28, 2014 at 5:52 PM, Joe Blow <blackhole.em at gmail.com> wrote:
>>> > Hey all,
>>> >
>>> > I'm having a really tough time getting PF_RING working with Bro in a
>>> > threaded fashion.  I have PF_RING compiled and working fine (tcpdump
>>> > test
>>> > works fine with Transparent mode = 2):
>>> >
>>> > PF_RING Version          : 6.0.2 ($Revision: exported$)
>>> > Total rings              : 0
>>> >
>>> > Standard (non DNA) Options
>>> > Ring slots               : 4096
>>> > Slot version             : 16
>>> > Capture TX               : No [RX only]
>>> > IP Defragment            : No
>>> > Socket Mode              : Standard
>>> > Transparent mode         : No [mode 2]
>>> > Total plugins            : 0
>>> > Cluster Fragment Queue   : 0
>>> > Cluster Fragment Discard : 0
>>> >
>>> > Bro is version 2.3 (sha1 - 79397be0e351165d44047b044d29b5e6580532cc
>>> > bro-2.3.tar.gz)
>>> > OS is CentOS 6.4 running 2.6.32-358.11.1.el6.x86_64
>>> >
>>> > When I try and configure against my PF_RING libraries, I get this:
>>> >
>>> > ./configure --with-pcap=/opt/pfring
>>> > Build Directory : build
>>> > Source Directory: /root/src/bro-2.3
>>> > -- The C compiler identification is GNU
>>> > -- The CXX compiler identification is GNU
>>> > -- Check for working C compiler: /usr/bin/gcc
>>> > -- Check for working C compiler: /usr/bin/gcc -- works
>>> > -- Detecting C compiler ABI info
>>> > -- Detecting C compiler ABI info - done
>>> > -- Check for working CXX compiler: /usr/bin/c++
>>> > -- Check for working CXX compiler: /usr/bin/c++ -- works
>>> > -- Detecting CXX compiler ABI info
>>> > -- Detecting CXX compiler ABI info - done
>>> > -- Found sed: /bin/sed
>>> > -- Found Perl: /usr/bin/perl
>>> > -- Found FLEX: 2.5.35
>>> > -- Found BISON: /usr/bin/bison
>>> > -- Found PCAP: /opt/pfring/lib/libpcap.so
>>> > -- Performing Test PCAP_LINKS_SOLO
>>> > -- Performing Test PCAP_LINKS_SOLO - Failed
>>> > -- Looking for include files CMAKE_HAVE_PTHREAD_H
>>> > -- Looking for include files CMAKE_HAVE_PTHREAD_H - found
>>> > -- Looking for pthread_create in pthreads
>>> > -- Looking for pthread_create in pthreads - not found
>>> > -- Looking for pthread_create in pthread
>>> > -- Looking for pthread_create in pthread - found
>>> > -- Found Threads: TRUE
>>> > -- Performing Test PCAP_NEEDS_THREADS
>>> > -- Performing Test PCAP_NEEDS_THREADS - Failed
>>> > CMake Error at cmake/FindPCAP.cmake:61 (message):
>>> >   Couldn't determine how to link against libpcap
>>> > Call Stack (most recent call first):
>>> >   cmake/FindRequiredPackage.cmake:26 (find_package)
>>> >   CMakeLists.txt:52 (FindRequiredPackage)
>>> >
>>> >
>>> > -- Configuring incomplete, errors occurred!
>>> >
>>> > I'm banging my head against this, but I believe this is because bro
>>> > can't
>>> > find the threading library to link to.  Could someone point me in the
>>> > right
>>> > direction?  Do I need other threading libraries? Static linking?
>>> >
>>> > Cheers,
>>> >
>>> > JB
>>> >
>>> >
>>> > _______________________________________________
>>> > Bro mailing list
>>> > bro at bro-ids.org
>>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>
>>>
>>>
>>> --
>>> Doug Burks
>>> Need Security Onion Training or Commercial Support?
>>> http://securityonionsolutions.com
>>
>>
>



-- 
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com


From jlay at slave-tothe-box.net  Fri Aug 29 08:21:03 2014
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 29 Aug 2014 09:21:03 -0600
Subject: [Bro] Fwd: Configure error linking libpcap and pthread
In-Reply-To: <CAK8kjrDqa3seYxdMoOu=KcWqcQ-B3Z-3jOv6vWG1H90TfNaZ6A@mail.gmail.com>
References: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>
	<CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>
	<CAK8kjrCXx76+SS8xuhEnWuM2CVDxy9HGjMrU3YhYtcPAgVB7eA@mail.gmail.com>
	<CAMCSCoVZrJ=W05Ly+fXKhR4X2QgZPkohPAKFOf_Zj9TZg02ZZw@mail.gmail.com>
	<CAMCSCoUA2OPb2kcpJ29oks_uE03QPd3HJ6FCfcrkLaf0oz1mtw@mail.gmail.com>
	<CAK8kjrDqa3seYxdMoOu=KcWqcQ-B3Z-3jOv6vWG1H90TfNaZ6A@mail.gmail.com>
Message-ID: <f92bf7654c0210d3979df27df5ff849d@localhost>

On 2014-08-29 08:58, Doug Burks wrote:
> It's possible that Bro is not actually using PF_RING and is actually
> falling back to standard libpcap.  Have you checked 
> /proc/net/pf_ring/
> to see if there is evidence of Bro using PF_RING?
>
> On Fri, Aug 29, 2014 at 10:53 AM, Joe Blow <blackhole.em at gmail.com> 
> wrote:
>> So i've gone and recompiled with PF_RING 6.  I have everything 
>> working and
>> using PF_RING correctly, but i'm still seeing packet loss (around 
>> 25% on a
>> 400-450mb/s link).   I was only ever able to get Bro working with
>> "Transparent mode = 0" and not 2 or 1.  I might be doing something
>> completely wrong, but whenever i start BRO, i only ever see one 
>> thread
>> peaking at 100%. Here is my node configuration:
>>
>> [worker-0]
>> type=worker
>> host=10.10.10.10
>> interface=eth3
>> lb_method=pf_ring
>> lb_procs=12
>>
>> Any ideas as to why i'm only getting one thread seeing the bro 
>> traffic?
>> Excuse my ignorance.
>>
>> Cheers,
>>
>> JB

ldd your bro bin as well:

[09:20:17 ubuntu-test:/usr/local/bro/bin$] ldd bro
         linux-gate.so.1 =>  (0xb778b000)
         libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 (0xb7730000)

James


From blackhole.em at gmail.com  Fri Aug 29 08:31:30 2014
From: blackhole.em at gmail.com (Joe Blow)
Date: Fri, 29 Aug 2014 11:31:30 -0400
Subject: [Bro] Fwd: Configure error linking libpcap and pthread
In-Reply-To: <CAK8kjrDqa3seYxdMoOu=KcWqcQ-B3Z-3jOv6vWG1H90TfNaZ6A@mail.gmail.com>
References: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>
	<CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>
	<CAK8kjrCXx76+SS8xuhEnWuM2CVDxy9HGjMrU3YhYtcPAgVB7eA@mail.gmail.com>
	<CAMCSCoVZrJ=W05Ly+fXKhR4X2QgZPkohPAKFOf_Zj9TZg02ZZw@mail.gmail.com>
	<CAMCSCoUA2OPb2kcpJ29oks_uE03QPd3HJ6FCfcrkLaf0oz1mtw@mail.gmail.com>
	<CAK8kjrDqa3seYxdMoOu=KcWqcQ-B3Z-3jOv6vWG1H90TfNaZ6A@mail.gmail.com>
Message-ID: <CAMCSCoV2LDhcFdTQ=x8+Jrpi7YCPfzOJKAMwcw0+Q5-CaOL5mg@mail.gmail.com>

It sure is.  Here is what it's telling me from the proc fs:

# cat /proc/net/pf_ring/53559-eth3.103
Bound Device(s)    : eth3
Active             : 1
Breed              : Non-DNA
Sampling Rate      : 1
Capture Direction  : RX+TX
Socket Mode        : RX+TX
Appl. Name         : <unknown>
IP Defragment      : No
BPF Filtering      : Enabled
# Sw Filt. Rules   : 0
# Hw Filt. Rules   : 0
Poll Pkt Watermark : 1
Num Poll Calls     : 1
Channel Id Mask    : 0xFFFFFFFF
Cluster Id         : 0
Slot Version       : 16 [6.0.2]
Min Num Slots      : 32768
Bucket Len         : 8192
Slot Len           : 8232 [bucket+header]
Tot Memory         : 269758464
Tot Packets        : 220334266
Tot Pkt Lost       : 74243221
Tot Insert         : 146091045
Tot Read           : 145749734
Insert Offset      : 136479200
Remove Offset      : 136550784
TX: Send Ok        : 0
TX: Send Errors    : 0
Reflect: Fwd Ok    : 0
Reflect: Fwd Errors: 0
Num Free Slots     : 0


This is where i'm seeing tons of the packet loss.  I've got snort running
with PF_RING on the same box with 8 threads, 0 packet loss.  Any ideas?

Cheers,

JB


On Fri, Aug 29, 2014 at 10:58 AM, Doug Burks <doug.burks at gmail.com> wrote:

> It's possible that Bro is not actually using PF_RING and is actually
> falling back to standard libpcap.  Have you checked /proc/net/pf_ring/
> to see if there is evidence of Bro using PF_RING?
>
> On Fri, Aug 29, 2014 at 10:53 AM, Joe Blow <blackhole.em at gmail.com> wrote:
> > So i've gone and recompiled with PF_RING 6.  I have everything working
> and
> > using PF_RING correctly, but i'm still seeing packet loss (around 25% on
> a
> > 400-450mb/s link).   I was only ever able to get Bro working with
> > "Transparent mode = 0" and not 2 or 1.  I might be doing something
> > completely wrong, but whenever i start BRO, i only ever see one thread
> > peaking at 100%. Here is my node configuration:
> >
> > [worker-0]
> > type=worker
> > host=10.10.10.10
> > interface=eth3
> > lb_method=pf_ring
> > lb_procs=12
> >
> > Any ideas as to why i'm only getting one thread seeing the bro traffic?
> > Excuse my ignorance.
> >
> > Cheers,
> >
> > JB
> >
> >
> > On Thu, Aug 28, 2014 at 7:06 PM, Joe Blow <blackhole.em at gmail.com>
> wrote:
> >>
> >> Doug Burks was quick to point out that i didn't export LIBS or LDFLAGS.
> >>
> >> I would have NEVER guessed this... thanks a thousand times over for this
> >> tidbit.  Configure finished just fine.  Making now.  Will update once
> i've
> >> got it up and load balanced.
> >>
> >> <code>
> >>
> >> export LDFLAGS="-Wl,--no-as-needed -lrt"
> >>
> >> export LIBS="-lrt -lnuma"
> >>
> >> </code>
> >>
> >> Cheers,
> >>
> >> JB
> >>
> >>
> >> On Thu, Aug 28, 2014 at 6:52 PM, Doug Burks <doug.burks at gmail.com>
> wrote:
> >>>
> >>> Hi Joe,
> >>>
> >>> When I packaged Bro 2.3 and PF_RING 6.0.2, I had to do the following:
> >>>
> >>> export LDFLAGS := $(LDFLAGS) -Wl,--no-as-needed -lrt
> >>> export LIBS := $(LIBS) -lrt -lnuma
> >>>
> >>> Depending on your configuration, you may also need to include
> >>> -lpthread in your LIBS.
> >>>
> >>> On Thu, Aug 28, 2014 at 5:52 PM, Joe Blow <blackhole.em at gmail.com>
> wrote:
> >>> > Hey all,
> >>> >
> >>> > I'm having a really tough time getting PF_RING working with Bro in a
> >>> > threaded fashion.  I have PF_RING compiled and working fine (tcpdump
> >>> > test
> >>> > works fine with Transparent mode = 2):
> >>> >
> >>> > PF_RING Version          : 6.0.2 ($Revision: exported$)
> >>> > Total rings              : 0
> >>> >
> >>> > Standard (non DNA) Options
> >>> > Ring slots               : 4096
> >>> > Slot version             : 16
> >>> > Capture TX               : No [RX only]
> >>> > IP Defragment            : No
> >>> > Socket Mode              : Standard
> >>> > Transparent mode         : No [mode 2]
> >>> > Total plugins            : 0
> >>> > Cluster Fragment Queue   : 0
> >>> > Cluster Fragment Discard : 0
> >>> >
> >>> > Bro is version 2.3 (sha1 - 79397be0e351165d44047b044d29b5e6580532cc
> >>> > bro-2.3.tar.gz)
> >>> > OS is CentOS 6.4 running 2.6.32-358.11.1.el6.x86_64
> >>> >
> >>> > When I try and configure against my PF_RING libraries, I get this:
> >>> >
> >>> > ./configure --with-pcap=/opt/pfring
> >>> > Build Directory : build
> >>> > Source Directory: /root/src/bro-2.3
> >>> > -- The C compiler identification is GNU
> >>> > -- The CXX compiler identification is GNU
> >>> > -- Check for working C compiler: /usr/bin/gcc
> >>> > -- Check for working C compiler: /usr/bin/gcc -- works
> >>> > -- Detecting C compiler ABI info
> >>> > -- Detecting C compiler ABI info - done
> >>> > -- Check for working CXX compiler: /usr/bin/c++
> >>> > -- Check for working CXX compiler: /usr/bin/c++ -- works
> >>> > -- Detecting CXX compiler ABI info
> >>> > -- Detecting CXX compiler ABI info - done
> >>> > -- Found sed: /bin/sed
> >>> > -- Found Perl: /usr/bin/perl
> >>> > -- Found FLEX: 2.5.35
> >>> > -- Found BISON: /usr/bin/bison
> >>> > -- Found PCAP: /opt/pfring/lib/libpcap.so
> >>> > -- Performing Test PCAP_LINKS_SOLO
> >>> > -- Performing Test PCAP_LINKS_SOLO - Failed
> >>> > -- Looking for include files CMAKE_HAVE_PTHREAD_H
> >>> > -- Looking for include files CMAKE_HAVE_PTHREAD_H - found
> >>> > -- Looking for pthread_create in pthreads
> >>> > -- Looking for pthread_create in pthreads - not found
> >>> > -- Looking for pthread_create in pthread
> >>> > -- Looking for pthread_create in pthread - found
> >>> > -- Found Threads: TRUE
> >>> > -- Performing Test PCAP_NEEDS_THREADS
> >>> > -- Performing Test PCAP_NEEDS_THREADS - Failed
> >>> > CMake Error at cmake/FindPCAP.cmake:61 (message):
> >>> >   Couldn't determine how to link against libpcap
> >>> > Call Stack (most recent call first):
> >>> >   cmake/FindRequiredPackage.cmake:26 (find_package)
> >>> >   CMakeLists.txt:52 (FindRequiredPackage)
> >>> >
> >>> >
> >>> > -- Configuring incomplete, errors occurred!
> >>> >
> >>> > I'm banging my head against this, but I believe this is because bro
> >>> > can't
> >>> > find the threading library to link to.  Could someone point me in the
> >>> > right
> >>> > direction?  Do I need other threading libraries? Static linking?
> >>> >
> >>> > Cheers,
> >>> >
> >>> > JB
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > Bro mailing list
> >>> > bro at bro-ids.org
> >>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >>>
> >>>
> >>>
> >>> --
> >>> Doug Burks
> >>> Need Security Onion Training or Commercial Support?
> >>> http://securityonionsolutions.com
> >>
> >>
> >
>
>
>
> --
> Doug Burks
> Need Security Onion Training or Commercial Support?
> http://securityonionsolutions.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140829/97aeb0c9/attachment.html 

From n.siow at go.wustl.edu  Fri Aug 29 09:03:33 2014
From: n.siow at go.wustl.edu (Nicholas SIow)
Date: Fri, 29 Aug 2014 11:03:33 -0500
Subject: [Bro] Dropped packets in PF_RING install
Message-ID: <etPan.5400a455.507ed7ab.11c@lanfear>

Hi Bro,


We have an install of bro running on a single machine with PF_RING load balancing.

Previously we were seeing a huge amount of dropped traffic ? in the realm of ~90% average packet loss per hour. The history column in our `conn.log` was trash as expected, with only one or two letters per connection.

After some tweaking (adding memory & upping # of bro processes & changing PF_RING buffer size), the logs look much better and the packet loss is drastically reduced, to about 0.5%-1% loss per hour. However, both `broctl netstats` and `cat /proc/net/pf_ring/*eth0*` report some packet loss still.

Is the sub-1% packet loss we?re seeing expected/optimal or are there additional tweaks that we could add to push this down to 0%?

### some notes ###

> both `tcpdump -nn -s0 -vv -i eth0 -w /dev/null` and the pfcount.c utility from pf_ring report 0% packet loss. It?s not until we start using bro that we start seeing dropped packets.

> we?re currently using 16 bro processes pinned to 16 of 32 total processors

> PF_RING buffer size is currently 65536

> packet loss does seem to go down during low-traffic hours but during the day when traffic is 2.5-3 gbps is when the dropped packet count peaks (while still being a small percentage of the overall traffic)


Let me know if you guys have any thoughts on this, thanks!


- - -
Nicholas Siow
Washington University in St. Louis :: Information Security?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140829/ccf38878/attachment.html 

From blackhole.em at gmail.com  Fri Aug 29 09:20:05 2014
From: blackhole.em at gmail.com (Joe Blow)
Date: Fri, 29 Aug 2014 12:20:05 -0400
Subject: [Bro] Dropped packets in PF_RING install
In-Reply-To: <etPan.5400a455.507ed7ab.11c@lanfear>
References: <etPan.5400a455.507ed7ab.11c@lanfear>
Message-ID: <CAMCSCoX2BWa-cop0M2ML=nquU36LcuKk6SNNhKdmTUBpj9S0fg@mail.gmail.com>

Can you paste your node.cfg here?  I'm having similar problems, but my
packet loss is much, much higher.

Cheers,

JB


On Fri, Aug 29, 2014 at 12:03 PM, Nicholas SIow <n.siow at go.wustl.edu> wrote:

> Hi Bro,
>
>
> We have an install of bro running on a single machine with PF_RING load
> balancing.
>
> Previously we were seeing a huge amount of dropped traffic ? in the realm
> of ~90% average packet loss per hour. The history column in our `conn.log`
> was trash as expected, with only one or two letters per connection.
>
> After some tweaking (adding memory & upping # of bro processes & changing
> PF_RING buffer size), the logs look much better and the packet loss is
> drastically reduced, to about 0.5%-1% loss per hour. However, both `broctl
> netstats` and `cat /proc/net/pf_ring/*eth0*` report some packet loss still.
>
> Is the sub-1% packet loss we?re seeing expected/optimal or are there
> additional tweaks that we could add to push this down to 0%?
>
> ### some notes ###
>
> > both `tcpdump -nn -s0 -vv -i eth0 -w /dev/null` and the pfcount.c
> utility from pf_ring report 0% packet loss. It?s not until we start using
> bro that we start seeing dropped packets.
>
> > we?re currently using 16 bro processes pinned to 16 of 32 total
> processors
>
> > PF_RING buffer size is currently 65536
>
> > packet loss does seem to go down during low-traffic hours but during the
> day when traffic is 2.5-3 gbps is when the dropped packet count peaks
> (while still being a small percentage of the overall traffic)
>
>
> Let me know if you guys have any thoughts on this, thanks!
>
>
> - - -
> Nicholas Siow
> Washington University in St. Louis :: Information Security
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140829/05de9992/attachment.html 

From luke at geekempire.com  Fri Aug 29 09:22:54 2014
From: luke at geekempire.com (Mike Reeves)
Date: Fri, 29 Aug 2014 12:22:54 -0400
Subject: [Bro] Dropped packets in PF_RING install
In-Reply-To: <etPan.5400a455.507ed7ab.11c@lanfear>
References: <etPan.5400a455.507ed7ab.11c@lanfear>
Message-ID: <CADC84wSxTjrH8LFAxWQLbGtwTeOWHN=tpkrJKGGuem+a7xEVMA@mail.gmail.com>

Are the pins to actual CPUs or hyper threads? How much throughput are you
dealing with?

On Friday, August 29, 2014, Nicholas SIow <n.siow at go.wustl.edu> wrote:

> Hi Bro,
>
>
> We have an install of bro running on a single machine with PF_RING load
> balancing.
>
> Previously we were seeing a huge amount of dropped traffic ? in the realm
> of ~90% average packet loss per hour. The history column in our `conn.log`
> was trash as expected, with only one or two letters per connection.
>
> After some tweaking (adding memory & upping # of bro processes & changing
> PF_RING buffer size), the logs look much better and the packet loss is
> drastically reduced, to about 0.5%-1% loss per hour. However, both `broctl
> netstats` and `cat /proc/net/pf_ring/*eth0*` report some packet loss still.
>
> Is the sub-1% packet loss we?re seeing expected/optimal or are there
> additional tweaks that we could add to push this down to 0%?
>
> ### some notes ###
>
> > both `tcpdump -nn -s0 -vv -i eth0 -w /dev/null` and the pfcount.c
> utility from pf_ring report 0% packet loss. It?s not until we start using
> bro that we start seeing dropped packets.
>
> > we?re currently using 16 bro processes pinned to 16 of 32 total
> processors
>
> > PF_RING buffer size is currently 65536
>
> > packet loss does seem to go down during low-traffic hours but during the
> day when traffic is 2.5-3 gbps is when the dropped packet count peaks
> (while still being a small percentage of the overall traffic)
>
>
> Let me know if you guys have any thoughts on this, thanks!
>
>
> - - -
> Nicholas Siow
> Washington University in St. Louis :: Information Security
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140829/288c0ce8/attachment.html 

From n.siow at go.wustl.edu  Fri Aug 29 09:23:46 2014
From: n.siow at go.wustl.edu (Nicholas SIow)
Date: Fri, 29 Aug 2014 11:23:46 -0500
Subject: [Bro] Dropped packets in PF_RING install
In-Reply-To: <CAMCSCoX2BWa-cop0M2ML=nquU36LcuKk6SNNhKdmTUBpj9S0fg@mail.gmail.com>
References: <etPan.5400a455.507ed7ab.11c@lanfear>
	<CAMCSCoX2BWa-cop0M2ML=nquU36LcuKk6SNNhKdmTUBpj9S0fg@mail.gmail.com>
Message-ID: <etPan.5400a912.5bd062c2.11c@lanfear>

Sure. It?s pretty standard and is more or less copied from the bro page on Load Balancing ?

```
[manager]
type=manager
host=localhost
#
[proxy-1]
type=proxy
host=localhost
#
[worker-1]
type=worker
host=localhost
interface=eth0
lb_method=pf_ring
lb_procs=16
pin_cpus=2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17
```

On August 29, 2014 at 11:20:08 AM, Joe Blow (blackhole.em at gmail.com) wrote:

Can you paste your node.cfg here?? I'm having similar problems, but my packet loss is much, much higher.

Cheers,

JB


On Fri, Aug 29, 2014 at 12:03 PM, Nicholas SIow <n.siow at go.wustl.edu> wrote:
Hi Bro,


We have an install of bro running on a single machine with PF_RING load balancing.

Previously we were seeing a huge amount of dropped traffic ? in the realm of ~90% average packet loss per hour. The history column in our `conn.log` was trash as expected, with only one or two letters per connection.

After some tweaking (adding memory & upping # of bro processes & changing PF_RING buffer size), the logs look much better and the packet loss is drastically reduced, to about 0.5%-1% loss per hour. However, both `broctl netstats` and `cat /proc/net/pf_ring/*eth0*` report some packet loss still.

Is the sub-1% packet loss we?re seeing expected/optimal or are there additional tweaks that we could add to push this down to 0%?

### some notes ###

> both `tcpdump -nn -s0 -vv -i eth0 -w /dev/null` and the pfcount.c utility from pf_ring report 0% packet loss. It?s not until we start using bro that we start seeing dropped packets.

> we?re currently using 16 bro processes pinned to 16 of 32 total processors

> PF_RING buffer size is currently 65536

> packet loss does seem to go down during low-traffic hours but during the day when traffic is 2.5-3 gbps is when the dropped packet count peaks (while still being a small percentage of the overall traffic)


Let me know if you guys have any thoughts on this, thanks!


- - -
Nicholas Siow
Washington University in St. Louis :: Information Security?

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140829/59f70faa/attachment.html 

From n.siow at go.wustl.edu  Fri Aug 29 09:26:44 2014
From: n.siow at go.wustl.edu (Nicholas SIow)
Date: Fri, 29 Aug 2014 11:26:44 -0500
Subject: [Bro] Dropped packets in PF_RING install
In-Reply-To: <CADC84wSxTjrH8LFAxWQLbGtwTeOWHN=tpkrJKGGuem+a7xEVMA@mail.gmail.com>
References: <etPan.5400a455.507ed7ab.11c@lanfear>
	<CADC84wSxTjrH8LFAxWQLbGtwTeOWHN=tpkrJKGGuem+a7xEVMA@mail.gmail.com>
Message-ID: <etPan.5400a9c5.216231b.11c@lanfear>

Actual CPUs, based on /proc/cpuinfo.

During the day I?m usually seeing traffic in the realm of 2.75-3.25 gbps.




On August 29, 2014 at 11:23:03 AM, Mike Reeves (luke at geekempire.com) wrote:

Are the pins to actual CPUs or hyper threads? How much throughput are you dealing with??

On Friday, August 29, 2014, Nicholas SIow <n.siow at go.wustl.edu> wrote:
Hi Bro,


We have an install of bro running on a single machine with PF_RING load balancing.

Previously we were seeing a huge amount of dropped traffic ? in the realm of ~90% average packet loss per hour. The history column in our `conn.log` was trash as expected, with only one or two letters per connection.

After some tweaking (adding memory & upping # of bro processes & changing PF_RING buffer size), the logs look much better and the packet loss is drastically reduced, to about 0.5%-1% loss per hour. However, both `broctl netstats` and `cat /proc/net/pf_ring/*eth0*` report some packet loss still.

Is the sub-1% packet loss we?re seeing expected/optimal or are there additional tweaks that we could add to push this down to 0%?

### some notes ###

> both `tcpdump -nn -s0 -vv -i eth0 -w /dev/null` and the pfcount.c utility from pf_ring report 0% packet loss. It?s not until we start using bro that we start seeing dropped packets.

> we?re currently using 16 bro processes pinned to 16 of 32 total processors

> PF_RING buffer size is currently 65536

> packet loss does seem to go down during low-traffic hours but during the day when traffic is 2.5-3 gbps is when the dropped packet count peaks (while still being a small percentage of the overall traffic)


Let me know if you guys have any thoughts on this, thanks!


- - -
Nicholas Siow
Washington University in St. Louis :: Information Security?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140829/afdb0b02/attachment.html 

From doug.burks at gmail.com  Fri Aug 29 11:49:12 2014
From: doug.burks at gmail.com (Doug Burks)
Date: Fri, 29 Aug 2014 14:49:12 -0400
Subject: [Bro] Fwd: Configure error linking libpcap and pthread
In-Reply-To: <CAMCSCoV2LDhcFdTQ=x8+Jrpi7YCPfzOJKAMwcw0+Q5-CaOL5mg@mail.gmail.com>
References: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>
	<CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>
	<CAK8kjrCXx76+SS8xuhEnWuM2CVDxy9HGjMrU3YhYtcPAgVB7eA@mail.gmail.com>
	<CAMCSCoVZrJ=W05Ly+fXKhR4X2QgZPkohPAKFOf_Zj9TZg02ZZw@mail.gmail.com>
	<CAMCSCoUA2OPb2kcpJ29oks_uE03QPd3HJ6FCfcrkLaf0oz1mtw@mail.gmail.com>
	<CAK8kjrDqa3seYxdMoOu=KcWqcQ-B3Z-3jOv6vWG1H90TfNaZ6A@mail.gmail.com>
	<CAMCSCoV2LDhcFdTQ=x8+Jrpi7YCPfzOJKAMwcw0+Q5-CaOL5mg@mail.gmail.com>
Message-ID: <CAK8kjrAOM3KQnD+N_Aqz=dyVu67=qGR45U5Xf_fsjHFpT-+b+g@mail.gmail.com>

Based on the following lines, it looks like Bro is running in standalone mode:
Appl. Name         : <unknown>
Cluster Id         : 0

If it were running in cluster mode, I would expect to see something
like the following instead:
Appl. Name         : bro-eth3
Cluster Id         : 21

Have you double-checked your node.cfg?

Have you tried the following?
sudo broctl install && sudo broctl restart

On Fri, Aug 29, 2014 at 11:31 AM, Joe Blow <blackhole.em at gmail.com> wrote:
> It sure is.  Here is what it's telling me from the proc fs:
>
> # cat /proc/net/pf_ring/53559-eth3.103
> Bound Device(s)    : eth3
> Active             : 1
> Breed              : Non-DNA
> Sampling Rate      : 1
> Capture Direction  : RX+TX
> Socket Mode        : RX+TX
> Appl. Name         : <unknown>
> IP Defragment      : No
> BPF Filtering      : Enabled
> # Sw Filt. Rules   : 0
> # Hw Filt. Rules   : 0
> Poll Pkt Watermark : 1
> Num Poll Calls     : 1
> Channel Id Mask    : 0xFFFFFFFF
> Cluster Id         : 0
> Slot Version       : 16 [6.0.2]
> Min Num Slots      : 32768
> Bucket Len         : 8192
> Slot Len           : 8232 [bucket+header]
> Tot Memory         : 269758464
> Tot Packets        : 220334266
> Tot Pkt Lost       : 74243221
> Tot Insert         : 146091045
> Tot Read           : 145749734
> Insert Offset      : 136479200
> Remove Offset      : 136550784
> TX: Send Ok        : 0
> TX: Send Errors    : 0
> Reflect: Fwd Ok    : 0
> Reflect: Fwd Errors: 0
> Num Free Slots     : 0
>
>
> This is where i'm seeing tons of the packet loss.  I've got snort running
> with PF_RING on the same box with 8 threads, 0 packet loss.  Any ideas?
>
> Cheers,
>
> JB
>
>
> On Fri, Aug 29, 2014 at 10:58 AM, Doug Burks <doug.burks at gmail.com> wrote:
>>
>> It's possible that Bro is not actually using PF_RING and is actually
>> falling back to standard libpcap.  Have you checked /proc/net/pf_ring/
>> to see if there is evidence of Bro using PF_RING?
>>
>> On Fri, Aug 29, 2014 at 10:53 AM, Joe Blow <blackhole.em at gmail.com> wrote:
>> > So i've gone and recompiled with PF_RING 6.  I have everything working
>> > and
>> > using PF_RING correctly, but i'm still seeing packet loss (around 25% on
>> > a
>> > 400-450mb/s link).   I was only ever able to get Bro working with
>> > "Transparent mode = 0" and not 2 or 1.  I might be doing something
>> > completely wrong, but whenever i start BRO, i only ever see one thread
>> > peaking at 100%. Here is my node configuration:
>> >
>> > [worker-0]
>> > type=worker
>> > host=10.10.10.10
>> > interface=eth3
>> > lb_method=pf_ring
>> > lb_procs=12
>> >
>> > Any ideas as to why i'm only getting one thread seeing the bro traffic?
>> > Excuse my ignorance.
>> >
>> > Cheers,
>> >
>> > JB
>> >
>> >
>> > On Thu, Aug 28, 2014 at 7:06 PM, Joe Blow <blackhole.em at gmail.com>
>> > wrote:
>> >>
>> >> Doug Burks was quick to point out that i didn't export LIBS or LDFLAGS.
>> >>
>> >> I would have NEVER guessed this... thanks a thousand times over for
>> >> this
>> >> tidbit.  Configure finished just fine.  Making now.  Will update once
>> >> i've
>> >> got it up and load balanced.
>> >>
>> >> <code>
>> >>
>> >> export LDFLAGS="-Wl,--no-as-needed -lrt"
>> >>
>> >> export LIBS="-lrt -lnuma"
>> >>
>> >> </code>
>> >>
>> >> Cheers,
>> >>
>> >> JB
>> >>
>> >>
>> >> On Thu, Aug 28, 2014 at 6:52 PM, Doug Burks <doug.burks at gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi Joe,
>> >>>
>> >>> When I packaged Bro 2.3 and PF_RING 6.0.2, I had to do the following:
>> >>>
>> >>> export LDFLAGS := $(LDFLAGS) -Wl,--no-as-needed -lrt
>> >>> export LIBS := $(LIBS) -lrt -lnuma
>> >>>
>> >>> Depending on your configuration, you may also need to include
>> >>> -lpthread in your LIBS.
>> >>>
>> >>> On Thu, Aug 28, 2014 at 5:52 PM, Joe Blow <blackhole.em at gmail.com>
>> >>> wrote:
>> >>> > Hey all,
>> >>> >
>> >>> > I'm having a really tough time getting PF_RING working with Bro in a
>> >>> > threaded fashion.  I have PF_RING compiled and working fine (tcpdump
>> >>> > test
>> >>> > works fine with Transparent mode = 2):
>> >>> >
>> >>> > PF_RING Version          : 6.0.2 ($Revision: exported$)
>> >>> > Total rings              : 0
>> >>> >
>> >>> > Standard (non DNA) Options
>> >>> > Ring slots               : 4096
>> >>> > Slot version             : 16
>> >>> > Capture TX               : No [RX only]
>> >>> > IP Defragment            : No
>> >>> > Socket Mode              : Standard
>> >>> > Transparent mode         : No [mode 2]
>> >>> > Total plugins            : 0
>> >>> > Cluster Fragment Queue   : 0
>> >>> > Cluster Fragment Discard : 0
>> >>> >
>> >>> > Bro is version 2.3 (sha1 - 79397be0e351165d44047b044d29b5e6580532cc
>> >>> > bro-2.3.tar.gz)
>> >>> > OS is CentOS 6.4 running 2.6.32-358.11.1.el6.x86_64
>> >>> >
>> >>> > When I try and configure against my PF_RING libraries, I get this:
>> >>> >
>> >>> > ./configure --with-pcap=/opt/pfring
>> >>> > Build Directory : build
>> >>> > Source Directory: /root/src/bro-2.3
>> >>> > -- The C compiler identification is GNU
>> >>> > -- The CXX compiler identification is GNU
>> >>> > -- Check for working C compiler: /usr/bin/gcc
>> >>> > -- Check for working C compiler: /usr/bin/gcc -- works
>> >>> > -- Detecting C compiler ABI info
>> >>> > -- Detecting C compiler ABI info - done
>> >>> > -- Check for working CXX compiler: /usr/bin/c++
>> >>> > -- Check for working CXX compiler: /usr/bin/c++ -- works
>> >>> > -- Detecting CXX compiler ABI info
>> >>> > -- Detecting CXX compiler ABI info - done
>> >>> > -- Found sed: /bin/sed
>> >>> > -- Found Perl: /usr/bin/perl
>> >>> > -- Found FLEX: 2.5.35
>> >>> > -- Found BISON: /usr/bin/bison
>> >>> > -- Found PCAP: /opt/pfring/lib/libpcap.so
>> >>> > -- Performing Test PCAP_LINKS_SOLO
>> >>> > -- Performing Test PCAP_LINKS_SOLO - Failed
>> >>> > -- Looking for include files CMAKE_HAVE_PTHREAD_H
>> >>> > -- Looking for include files CMAKE_HAVE_PTHREAD_H - found
>> >>> > -- Looking for pthread_create in pthreads
>> >>> > -- Looking for pthread_create in pthreads - not found
>> >>> > -- Looking for pthread_create in pthread
>> >>> > -- Looking for pthread_create in pthread - found
>> >>> > -- Found Threads: TRUE
>> >>> > -- Performing Test PCAP_NEEDS_THREADS
>> >>> > -- Performing Test PCAP_NEEDS_THREADS - Failed
>> >>> > CMake Error at cmake/FindPCAP.cmake:61 (message):
>> >>> >   Couldn't determine how to link against libpcap
>> >>> > Call Stack (most recent call first):
>> >>> >   cmake/FindRequiredPackage.cmake:26 (find_package)
>> >>> >   CMakeLists.txt:52 (FindRequiredPackage)
>> >>> >
>> >>> >
>> >>> > -- Configuring incomplete, errors occurred!
>> >>> >
>> >>> > I'm banging my head against this, but I believe this is because bro
>> >>> > can't
>> >>> > find the threading library to link to.  Could someone point me in
>> >>> > the
>> >>> > right
>> >>> > direction?  Do I need other threading libraries? Static linking?
>> >>> >
>> >>> > Cheers,
>> >>> >
>> >>> > JB
>> >>> >
>> >>> >
>> >>> > _______________________________________________
>> >>> > Bro mailing list
>> >>> > bro at bro-ids.org
>> >>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Doug Burks
>> >>> Need Security Onion Training or Commercial Support?
>> >>> http://securityonionsolutions.com
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Doug Burks
>> Need Security Onion Training or Commercial Support?
>> http://securityonionsolutions.com
>
>



-- 
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com


From dnthayer at illinois.edu  Fri Aug 29 12:05:11 2014
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Fri, 29 Aug 2014 14:05:11 -0500
Subject: [Bro] Fwd: Configure error linking libpcap and pthread
In-Reply-To: <CAK8kjrAOM3KQnD+N_Aqz=dyVu67=qGR45U5Xf_fsjHFpT-+b+g@mail.gmail.com>
References: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>	<CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>	<CAK8kjrCXx76+SS8xuhEnWuM2CVDxy9HGjMrU3YhYtcPAgVB7eA@mail.gmail.com>	<CAMCSCoVZrJ=W05Ly+fXKhR4X2QgZPkohPAKFOf_Zj9TZg02ZZw@mail.gmail.com>	<CAMCSCoUA2OPb2kcpJ29oks_uE03QPd3HJ6FCfcrkLaf0oz1mtw@mail.gmail.com>	<CAK8kjrDqa3seYxdMoOu=KcWqcQ-B3Z-3jOv6vWG1H90TfNaZ6A@mail.gmail.com>	<CAMCSCoV2LDhcFdTQ=x8+Jrpi7YCPfzOJKAMwcw0+Q5-CaOL5mg@mail.gmail.com>
	<CAK8kjrAOM3KQnD+N_Aqz=dyVu67=qGR45U5Xf_fsjHFpT-+b+g@mail.gmail.com>
Message-ID: <5400CEE7.9050601@illinois.edu>

Another thing to check is to search the output of "broctl config"
for "pfringclusterid" (it must be set to a non-zero value if you
want to use PF_RING).



On 08/29/2014 01:49 PM, Doug Burks wrote:
> Based on the following lines, it looks like Bro is running in standalone mode:
> Appl. Name         : <unknown>
> Cluster Id         : 0
>
> If it were running in cluster mode, I would expect to see something
> like the following instead:
> Appl. Name         : bro-eth3
> Cluster Id         : 21
>
> Have you double-checked your node.cfg?
>
> Have you tried the following?
> sudo broctl install && sudo broctl restart
>
> On Fri, Aug 29, 2014 at 11:31 AM, Joe Blow <blackhole.em at gmail.com> wrote:
>> It sure is.  Here is what it's telling me from the proc fs:
>>
>> # cat /proc/net/pf_ring/53559-eth3.103
>> Bound Device(s)    : eth3
>> Active             : 1
>> Breed              : Non-DNA
>> Sampling Rate      : 1
>> Capture Direction  : RX+TX
>> Socket Mode        : RX+TX
>> Appl. Name         : <unknown>
>> IP Defragment      : No
>> BPF Filtering      : Enabled
>> # Sw Filt. Rules   : 0
>> # Hw Filt. Rules   : 0
>> Poll Pkt Watermark : 1
>> Num Poll Calls     : 1
>> Channel Id Mask    : 0xFFFFFFFF
>> Cluster Id         : 0
>> Slot Version       : 16 [6.0.2]
>> Min Num Slots      : 32768
>> Bucket Len         : 8192
>> Slot Len           : 8232 [bucket+header]
>> Tot Memory         : 269758464
>> Tot Packets        : 220334266
>> Tot Pkt Lost       : 74243221
>> Tot Insert         : 146091045
>> Tot Read           : 145749734
>> Insert Offset      : 136479200
>> Remove Offset      : 136550784
>> TX: Send Ok        : 0
>> TX: Send Errors    : 0
>> Reflect: Fwd Ok    : 0
>> Reflect: Fwd Errors: 0
>> Num Free Slots     : 0
>>
>>
>> This is where i'm seeing tons of the packet loss.  I've got snort running
>> with PF_RING on the same box with 8 threads, 0 packet loss.  Any ideas?
>>
>> Cheers,
>>
>> JB
>>
>>
>> On Fri, Aug 29, 2014 at 10:58 AM, Doug Burks <doug.burks at gmail.com> wrote:
>>>
>>> It's possible that Bro is not actually using PF_RING and is actually
>>> falling back to standard libpcap.  Have you checked /proc/net/pf_ring/
>>> to see if there is evidence of Bro using PF_RING?
>>>
>>> On Fri, Aug 29, 2014 at 10:53 AM, Joe Blow <blackhole.em at gmail.com> wrote:
>>>> So i've gone and recompiled with PF_RING 6.  I have everything working
>>>> and
>>>> using PF_RING correctly, but i'm still seeing packet loss (around 25% on
>>>> a
>>>> 400-450mb/s link).   I was only ever able to get Bro working with
>>>> "Transparent mode = 0" and not 2 or 1.  I might be doing something
>>>> completely wrong, but whenever i start BRO, i only ever see one thread
>>>> peaking at 100%. Here is my node configuration:
>>>>
>>>> [worker-0]
>>>> type=worker
>>>> host=10.10.10.10
>>>> interface=eth3
>>>> lb_method=pf_ring
>>>> lb_procs=12
>>>>
>>>> Any ideas as to why i'm only getting one thread seeing the bro traffic?
>>>> Excuse my ignorance.
>>>>
>>>> Cheers,
>>>>
>>>> JB
>>>>
>>>>
>>>> On Thu, Aug 28, 2014 at 7:06 PM, Joe Blow <blackhole.em at gmail.com>
>>>> wrote:
>>>>>
>>>>> Doug Burks was quick to point out that i didn't export LIBS or LDFLAGS.
>>>>>
>>>>> I would have NEVER guessed this... thanks a thousand times over for
>>>>> this
>>>>> tidbit.  Configure finished just fine.  Making now.  Will update once
>>>>> i've
>>>>> got it up and load balanced.
>>>>>
>>>>> <code>
>>>>>
>>>>> export LDFLAGS="-Wl,--no-as-needed -lrt"
>>>>>
>>>>> export LIBS="-lrt -lnuma"
>>>>>
>>>>> </code>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> JB
>>>>>
>>>>>
>>>>> On Thu, Aug 28, 2014 at 6:52 PM, Doug Burks <doug.burks at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi Joe,
>>>>>>
>>>>>> When I packaged Bro 2.3 and PF_RING 6.0.2, I had to do the following:
>>>>>>
>>>>>> export LDFLAGS := $(LDFLAGS) -Wl,--no-as-needed -lrt
>>>>>> export LIBS := $(LIBS) -lrt -lnuma
>>>>>>
>>>>>> Depending on your configuration, you may also need to include
>>>>>> -lpthread in your LIBS.
>>>>>>
>>>>>> On Thu, Aug 28, 2014 at 5:52 PM, Joe Blow <blackhole.em at gmail.com>
>>>>>> wrote:
>>>>>>> Hey all,
>>>>>>>
>>>>>>> I'm having a really tough time getting PF_RING working with Bro in a
>>>>>>> threaded fashion.  I have PF_RING compiled and working fine (tcpdump
>>>>>>> test
>>>>>>> works fine with Transparent mode = 2):
>>>>>>>
>>>>>>> PF_RING Version          : 6.0.2 ($Revision: exported$)
>>>>>>> Total rings              : 0
>>>>>>>
>>>>>>> Standard (non DNA) Options
>>>>>>> Ring slots               : 4096
>>>>>>> Slot version             : 16
>>>>>>> Capture TX               : No [RX only]
>>>>>>> IP Defragment            : No
>>>>>>> Socket Mode              : Standard
>>>>>>> Transparent mode         : No [mode 2]
>>>>>>> Total plugins            : 0
>>>>>>> Cluster Fragment Queue   : 0
>>>>>>> Cluster Fragment Discard : 0
>>>>>>>
>>>>>>> Bro is version 2.3 (sha1 - 79397be0e351165d44047b044d29b5e6580532cc
>>>>>>> bro-2.3.tar.gz)
>>>>>>> OS is CentOS 6.4 running 2.6.32-358.11.1.el6.x86_64
>>>>>>>
>>>>>>> When I try and configure against my PF_RING libraries, I get this:
>>>>>>>
>>>>>>> ./configure --with-pcap=/opt/pfring
>>>>>>> Build Directory : build
>>>>>>> Source Directory: /root/src/bro-2.3
>>>>>>> -- The C compiler identification is GNU
>>>>>>> -- The CXX compiler identification is GNU
>>>>>>> -- Check for working C compiler: /usr/bin/gcc
>>>>>>> -- Check for working C compiler: /usr/bin/gcc -- works
>>>>>>> -- Detecting C compiler ABI info
>>>>>>> -- Detecting C compiler ABI info - done
>>>>>>> -- Check for working CXX compiler: /usr/bin/c++
>>>>>>> -- Check for working CXX compiler: /usr/bin/c++ -- works
>>>>>>> -- Detecting CXX compiler ABI info
>>>>>>> -- Detecting CXX compiler ABI info - done
>>>>>>> -- Found sed: /bin/sed
>>>>>>> -- Found Perl: /usr/bin/perl
>>>>>>> -- Found FLEX: 2.5.35
>>>>>>> -- Found BISON: /usr/bin/bison
>>>>>>> -- Found PCAP: /opt/pfring/lib/libpcap.so
>>>>>>> -- Performing Test PCAP_LINKS_SOLO
>>>>>>> -- Performing Test PCAP_LINKS_SOLO - Failed
>>>>>>> -- Looking for include files CMAKE_HAVE_PTHREAD_H
>>>>>>> -- Looking for include files CMAKE_HAVE_PTHREAD_H - found
>>>>>>> -- Looking for pthread_create in pthreads
>>>>>>> -- Looking for pthread_create in pthreads - not found
>>>>>>> -- Looking for pthread_create in pthread
>>>>>>> -- Looking for pthread_create in pthread - found
>>>>>>> -- Found Threads: TRUE
>>>>>>> -- Performing Test PCAP_NEEDS_THREADS
>>>>>>> -- Performing Test PCAP_NEEDS_THREADS - Failed
>>>>>>> CMake Error at cmake/FindPCAP.cmake:61 (message):
>>>>>>>    Couldn't determine how to link against libpcap
>>>>>>> Call Stack (most recent call first):
>>>>>>>    cmake/FindRequiredPackage.cmake:26 (find_package)
>>>>>>>    CMakeLists.txt:52 (FindRequiredPackage)
>>>>>>>
>>>>>>>
>>>>>>> -- Configuring incomplete, errors occurred!
>>>>>>>
>>>>>>> I'm banging my head against this, but I believe this is because bro
>>>>>>> can't
>>>>>>> find the threading library to link to.  Could someone point me in
>>>>>>> the
>>>>>>> right
>>>>>>> direction?  Do I need other threading libraries? Static linking?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> JB
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Bro mailing list
>>>>>>> bro at bro-ids.org
>>>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Doug Burks
>>>>>> Need Security Onion Training or Commercial Support?
>>>>>> http://securityonionsolutions.com
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Doug Burks
>>> Need Security Onion Training or Commercial Support?
>>> http://securityonionsolutions.com
>>
>>
>
>
>


From blackhole.em at gmail.com  Fri Aug 29 12:27:57 2014
From: blackhole.em at gmail.com (Joe Blow)
Date: Fri, 29 Aug 2014 15:27:57 -0400
Subject: [Bro] Fwd: Configure error linking libpcap and pthread
In-Reply-To: <5400CEE7.9050601@illinois.edu>
References: <CAMCSCoXVYMZ78tN1=cbiofa+_2VJ1vWrCjAAhK3H6HFO=ZOYDw@mail.gmail.com>
	<CAMCSCoVkf2abJPZTf1G0fPZSM+wm487XC0WpKbYP5zCCwFgZQA@mail.gmail.com>
	<CAK8kjrCXx76+SS8xuhEnWuM2CVDxy9HGjMrU3YhYtcPAgVB7eA@mail.gmail.com>
	<CAMCSCoVZrJ=W05Ly+fXKhR4X2QgZPkohPAKFOf_Zj9TZg02ZZw@mail.gmail.com>
	<CAMCSCoUA2OPb2kcpJ29oks_uE03QPd3HJ6FCfcrkLaf0oz1mtw@mail.gmail.com>
	<CAK8kjrDqa3seYxdMoOu=KcWqcQ-B3Z-3jOv6vWG1H90TfNaZ6A@mail.gmail.com>
	<CAMCSCoV2LDhcFdTQ=x8+Jrpi7YCPfzOJKAMwcw0+Q5-CaOL5mg@mail.gmail.com>
	<CAK8kjrAOM3KQnD+N_Aqz=dyVu67=qGR45U5Xf_fsjHFpT-+b+g@mail.gmail.com>
	<5400CEE7.9050601@illinois.edu>
Message-ID: <CAMCSCoWXNYmbGJkkKUZ9mm=Cr9m=JPQFxy+68noWtG3vZ2uT0Q@mail.gmail.com>

Doug - I fixed my node config up and ran those commands.  There were some
incorrect configs in the node.cfg file, which i was able to check with the
broctl config command.

Everything seems to be working stellar now.  Thanks tons for all the help
everyone!

Cheers,

JB


On Fri, Aug 29, 2014 at 3:05 PM, Daniel Thayer <dnthayer at illinois.edu>
wrote:

> Another thing to check is to search the output of "broctl config"
> for "pfringclusterid" (it must be set to a non-zero value if you
> want to use PF_RING).
>
>
>
>
> On 08/29/2014 01:49 PM, Doug Burks wrote:
>
>> Based on the following lines, it looks like Bro is running in standalone
>> mode:
>> Appl. Name         : <unknown>
>> Cluster Id         : 0
>>
>> If it were running in cluster mode, I would expect to see something
>> like the following instead:
>> Appl. Name         : bro-eth3
>> Cluster Id         : 21
>>
>> Have you double-checked your node.cfg?
>>
>> Have you tried the following?
>> sudo broctl install && sudo broctl restart
>>
>> On Fri, Aug 29, 2014 at 11:31 AM, Joe Blow <blackhole.em at gmail.com>
>> wrote:
>>
>>> It sure is.  Here is what it's telling me from the proc fs:
>>>
>>> # cat /proc/net/pf_ring/53559-eth3.103
>>> Bound Device(s)    : eth3
>>> Active             : 1
>>> Breed              : Non-DNA
>>> Sampling Rate      : 1
>>> Capture Direction  : RX+TX
>>> Socket Mode        : RX+TX
>>> Appl. Name         : <unknown>
>>> IP Defragment      : No
>>> BPF Filtering      : Enabled
>>> # Sw Filt. Rules   : 0
>>> # Hw Filt. Rules   : 0
>>> Poll Pkt Watermark : 1
>>> Num Poll Calls     : 1
>>> Channel Id Mask    : 0xFFFFFFFF
>>> Cluster Id         : 0
>>> Slot Version       : 16 [6.0.2]
>>> Min Num Slots      : 32768
>>> Bucket Len         : 8192
>>> Slot Len           : 8232 [bucket+header]
>>> Tot Memory         : 269758464
>>> Tot Packets        : 220334266
>>> Tot Pkt Lost       : 74243221
>>> Tot Insert         : 146091045
>>> Tot Read           : 145749734
>>> Insert Offset      : 136479200
>>> Remove Offset      : 136550784
>>> TX: Send Ok        : 0
>>> TX: Send Errors    : 0
>>> Reflect: Fwd Ok    : 0
>>> Reflect: Fwd Errors: 0
>>> Num Free Slots     : 0
>>>
>>>
>>> This is where i'm seeing tons of the packet loss.  I've got snort running
>>> with PF_RING on the same box with 8 threads, 0 packet loss.  Any ideas?
>>>
>>> Cheers,
>>>
>>> JB
>>>
>>>
>>> On Fri, Aug 29, 2014 at 10:58 AM, Doug Burks <doug.burks at gmail.com>
>>> wrote:
>>>
>>>>
>>>> It's possible that Bro is not actually using PF_RING and is actually
>>>> falling back to standard libpcap.  Have you checked /proc/net/pf_ring/
>>>> to see if there is evidence of Bro using PF_RING?
>>>>
>>>> On Fri, Aug 29, 2014 at 10:53 AM, Joe Blow <blackhole.em at gmail.com>
>>>> wrote:
>>>>
>>>>> So i've gone and recompiled with PF_RING 6.  I have everything working
>>>>> and
>>>>> using PF_RING correctly, but i'm still seeing packet loss (around 25%
>>>>> on
>>>>> a
>>>>> 400-450mb/s link).   I was only ever able to get Bro working with
>>>>> "Transparent mode = 0" and not 2 or 1.  I might be doing something
>>>>> completely wrong, but whenever i start BRO, i only ever see one thread
>>>>> peaking at 100%. Here is my node configuration:
>>>>>
>>>>> [worker-0]
>>>>> type=worker
>>>>> host=10.10.10.10
>>>>> interface=eth3
>>>>> lb_method=pf_ring
>>>>> lb_procs=12
>>>>>
>>>>> Any ideas as to why i'm only getting one thread seeing the bro traffic?
>>>>> Excuse my ignorance.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> JB
>>>>>
>>>>>
>>>>> On Thu, Aug 28, 2014 at 7:06 PM, Joe Blow <blackhole.em at gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Doug Burks was quick to point out that i didn't export LIBS or
>>>>>> LDFLAGS.
>>>>>>
>>>>>> I would have NEVER guessed this... thanks a thousand times over for
>>>>>> this
>>>>>> tidbit.  Configure finished just fine.  Making now.  Will update once
>>>>>> i've
>>>>>> got it up and load balanced.
>>>>>>
>>>>>> <code>
>>>>>>
>>>>>> export LDFLAGS="-Wl,--no-as-needed -lrt"
>>>>>>
>>>>>> export LIBS="-lrt -lnuma"
>>>>>>
>>>>>> </code>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> JB
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 28, 2014 at 6:52 PM, Doug Burks <doug.burks at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi Joe,
>>>>>>>
>>>>>>> When I packaged Bro 2.3 and PF_RING 6.0.2, I had to do the following:
>>>>>>>
>>>>>>> export LDFLAGS := $(LDFLAGS) -Wl,--no-as-needed -lrt
>>>>>>> export LIBS := $(LIBS) -lrt -lnuma
>>>>>>>
>>>>>>> Depending on your configuration, you may also need to include
>>>>>>> -lpthread in your LIBS.
>>>>>>>
>>>>>>> On Thu, Aug 28, 2014 at 5:52 PM, Joe Blow <blackhole.em at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey all,
>>>>>>>>
>>>>>>>> I'm having a really tough time getting PF_RING working with Bro in a
>>>>>>>> threaded fashion.  I have PF_RING compiled and working fine (tcpdump
>>>>>>>> test
>>>>>>>> works fine with Transparent mode = 2):
>>>>>>>>
>>>>>>>> PF_RING Version          : 6.0.2 ($Revision: exported$)
>>>>>>>> Total rings              : 0
>>>>>>>>
>>>>>>>> Standard (non DNA) Options
>>>>>>>> Ring slots               : 4096
>>>>>>>> Slot version             : 16
>>>>>>>> Capture TX               : No [RX only]
>>>>>>>> IP Defragment            : No
>>>>>>>> Socket Mode              : Standard
>>>>>>>> Transparent mode         : No [mode 2]
>>>>>>>> Total plugins            : 0
>>>>>>>> Cluster Fragment Queue   : 0
>>>>>>>> Cluster Fragment Discard : 0
>>>>>>>>
>>>>>>>> Bro is version 2.3 (sha1 - 79397be0e351165d44047b044d29b5e6580532cc
>>>>>>>> bro-2.3.tar.gz)
>>>>>>>> OS is CentOS 6.4 running 2.6.32-358.11.1.el6.x86_64
>>>>>>>>
>>>>>>>> When I try and configure against my PF_RING libraries, I get this:
>>>>>>>>
>>>>>>>> ./configure --with-pcap=/opt/pfring
>>>>>>>> Build Directory : build
>>>>>>>> Source Directory: /root/src/bro-2.3
>>>>>>>> -- The C compiler identification is GNU
>>>>>>>> -- The CXX compiler identification is GNU
>>>>>>>> -- Check for working C compiler: /usr/bin/gcc
>>>>>>>> -- Check for working C compiler: /usr/bin/gcc -- works
>>>>>>>> -- Detecting C compiler ABI info
>>>>>>>> -- Detecting C compiler ABI info - done
>>>>>>>> -- Check for working CXX compiler: /usr/bin/c++
>>>>>>>> -- Check for working CXX compiler: /usr/bin/c++ -- works
>>>>>>>> -- Detecting CXX compiler ABI info
>>>>>>>> -- Detecting CXX compiler ABI info - done
>>>>>>>> -- Found sed: /bin/sed
>>>>>>>> -- Found Perl: /usr/bin/perl
>>>>>>>> -- Found FLEX: 2.5.35
>>>>>>>> -- Found BISON: /usr/bin/bison
>>>>>>>> -- Found PCAP: /opt/pfring/lib/libpcap.so
>>>>>>>> -- Performing Test PCAP_LINKS_SOLO
>>>>>>>> -- Performing Test PCAP_LINKS_SOLO - Failed
>>>>>>>> -- Looking for include files CMAKE_HAVE_PTHREAD_H
>>>>>>>> -- Looking for include files CMAKE_HAVE_PTHREAD_H - found
>>>>>>>> -- Looking for pthread_create in pthreads
>>>>>>>> -- Looking for pthread_create in pthreads - not found
>>>>>>>> -- Looking for pthread_create in pthread
>>>>>>>> -- Looking for pthread_create in pthread - found
>>>>>>>> -- Found Threads: TRUE
>>>>>>>> -- Performing Test PCAP_NEEDS_THREADS
>>>>>>>> -- Performing Test PCAP_NEEDS_THREADS - Failed
>>>>>>>> CMake Error at cmake/FindPCAP.cmake:61 (message):
>>>>>>>>    Couldn't determine how to link against libpcap
>>>>>>>> Call Stack (most recent call first):
>>>>>>>>    cmake/FindRequiredPackage.cmake:26 (find_package)
>>>>>>>>    CMakeLists.txt:52 (FindRequiredPackage)
>>>>>>>>
>>>>>>>>
>>>>>>>> -- Configuring incomplete, errors occurred!
>>>>>>>>
>>>>>>>> I'm banging my head against this, but I believe this is because bro
>>>>>>>> can't
>>>>>>>> find the threading library to link to.  Could someone point me in
>>>>>>>> the
>>>>>>>> right
>>>>>>>> direction?  Do I need other threading libraries? Static linking?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> JB
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Bro mailing list
>>>>>>>> bro at bro-ids.org
>>>>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Doug Burks
>>>>>>> Need Security Onion Training or Commercial Support?
>>>>>>> http://securityonionsolutions.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Doug Burks
>>>> Need Security Onion Training or Commercial Support?
>>>> http://securityonionsolutions.com
>>>>
>>>
>>>
>>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140829/e4aa6d31/attachment.html 

From jxbatchelor at gmail.com  Fri Aug 29 13:00:00 2014
From: jxbatchelor at gmail.com (Jason Batchelor)
Date: Fri, 29 Aug 2014 15:00:00 -0500
Subject: [Bro] File Extraction Directory
In-Reply-To: <AC8715C1-A7E1-4A2A-B013-93E90D5CB5A9@icir.org>
References: <CANWz5X9+TgqcNWQPv=DcwiinCD7OoqsDwjujNFg3UVS8+4QOQg@mail.gmail.com>
	<AC8715C1-A7E1-4A2A-B013-93E90D5CB5A9@icir.org>
Message-ID: <CANWz5X-78_dFY12z-PuV2DVjwXsqMO4jNUedmTiUjCWEGmhfQQ@mail.gmail.com>

That's it :)

Thanks!


On Wed, Aug 27, 2014 at 5:07 PM, Johanna Amann <johanna at icir.org> wrote:

> Hi,
>
> redef FileExtract::prefix = "/var/opt/bro/spool/extract_files/"; should
> work.
>
> Johanna
>
> On 27 Aug 2014, at 14:38, Jason Batchelor wrote:
>
> > Hello all:
> >
> > Quick question, can you change the default file extraction directory for
> > files being extracted in a script. After some poking I came across where
> > this was specified in /opt/bro/share/bro/base/files/extract/main.bro with
> > the following:
> >
> > export {
> >  ## The prefix where files are extracted to.
> >  const prefix = "./extract_files/" &redef;
> > If I try to do something like this in my script:
> >
> > # Set extraction folder
> > redef prefix = "/var/opt/bro/spool/extract_files";
> > I am met with the following:
> > error in ./scripts/file-ext.bro, line 22: "redef" used but not previously
> > defined (prefix)
> > internal warning in ./scripts/file-ext.bro, line 22: Can't document redef
> > of prefix, identifier lookup failed
> > Can someone help me understand how to define this attribute or otherwise
> > influence where the files are extracted to? I would rather not manually
> > define it in the main.bro file.
> >
> > Thanks,
> > Jason
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140829/7b18792d/attachment.html 

From seth at icir.org  Sun Aug 31 09:07:55 2014
From: seth at icir.org (Seth Hall)
Date: Sun, 31 Aug 2014 12:07:55 -0400
Subject: [Bro] Adding options to bro managed by broctl
In-Reply-To: <c1b78eb4fada872cc6fd4a69b5615b07@localhost>
References: <CAAJ38WXmoxcOX0c1Ci1eDWqi6QXq2qxayTDfnC+Fd1W=caU3Xg@mail.gmail.com>
	<c1b78eb4fada872cc6fd4a69b5615b07@localhost>
Message-ID: <49FAE13F-E71B-4F01-AAB3-F846D8DA5030@icir.org>


On Aug 28, 2014, at 11:07 AM, James Lay <jlay at slave-tothe-box.net> wrote:

> broargs = -f 'net 1.0.0.0/24 or port 443'
> 
> to your broctl.cfg file.

That will work, but technically it might be a bit better to do something like this...

redef capture_filters += {
	["watched network"] = "net 1.0.0.0/24",
	["https"] = "port 443"
};

If you build up what you want to capture this way it gives Bro the chance to automatically build your BPF filters for you, including checking each component of your filter for mistakes which it will then detect at startup and tell you which component of your filter failed.  If you use the above lines to indicate the traffic you'd like to allow into Bro, you can also set restriction filters to limit something a bit.  For instance, in that 1.0.0.0/24 subnet you might want to ignore a single host.  You could implement that by adding the following lines...

redef restrict_filters += {
	["unmonitored host"] = "host 1.0.0.54"
};

The filter that would ultimately be constructed by those lines is...
	((port 443) or (net 1.0.0.0/24)) and (host 1.0.0.54)

One thing to be careful with this though is that generally when you take the stance that you are doing filtering you have to be really careful to understand your traffic.  If you have any traffic with MPLS or VLAN tags, the filters I gave won't allow that traffic through.  If you're interested in doing ARP analysis you won't see those packets either.  Same goes for IPv6.

Filtering is an area where we've tried to make things simple by running a fully open filter, there are a lot of dragons when you stray from that path. :) 

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140831/093fafd2/attachment.bin 

From seth at icir.org  Sun Aug 31 09:13:21 2014
From: seth at icir.org (Seth Hall)
Date: Sun, 31 Aug 2014 12:13:21 -0400
Subject: [Bro] connecting to bro with broccoli
In-Reply-To: <CAAa3MOdKwDTR1mHo=PvXfrtZPziU0wC8g_8eyUqiUZ7cVZ19Vg@mail.gmail.com>
References: <CAAa3MOdKwDTR1mHo=PvXfrtZPziU0wC8g_8eyUqiUZ7cVZ19Vg@mail.gmail.com>
Message-ID: <B305B9CB-1C1F-4700-85BE-36EAF12D0EE5@icir.org>


On Aug 27, 2014, at 1:29 PM, daniel nagar <dngr7512 at gmail.com> wrote:

> I'm using bro 2.2 and I connect to bro using broccoli to receive events.
> I can manage connecting to bro-worker and receive events, not sure if it's the correct way to receive event from bro but connecting to the manager port didn't retrieve any event whatsoever,

Could you give more information about what events you are sending around?  Are you receiving events from Bro or sending them to Bro?  

What configuration have you done in Bro to send or receive these events?

> the problem is that when I receive events at speeds higher than 2Mbps the parent of the bro-worker (not the broccoli application) memory expands rapidly and can reach 10Gb in a minute.

Another interesting number might be events per second.  I'm even a little unclear what you mean by 2Mbps.  Do you mean that the data rate of your connection between your broccoli application and Bro is 2Mbps?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140831/db35bbfc/attachment.bin 

From dngr7512 at gmail.com  Sun Aug 31 11:31:52 2014
From: dngr7512 at gmail.com (daniel nagar)
Date: Sun, 31 Aug 2014 21:31:52 +0300
Subject: [Bro] connecting to bro with broccoli
In-Reply-To: <B305B9CB-1C1F-4700-85BE-36EAF12D0EE5@icir.org>
References: <CAAa3MOdKwDTR1mHo=PvXfrtZPziU0wC8g_8eyUqiUZ7cVZ19Vg@mail.gmail.com>
	<B305B9CB-1C1F-4700-85BE-36EAF12D0EE5@icir.org>
Message-ID: <CAAa3MOf8jjyFYsXpPsrx10AcSANaZp9yYd3bLMYm1jr71FZtgg@mail.gmail.com>

>
> Could you give more information about what events you are sending around?
> Are you receiving events from Bro or sending them to Bro?

I'm capturing HTTP events, only receiving, not sending

What configuration have you done in Bro to send or receive these events?

I've used the default settings, same configurations as in the examples that
come with broccoli

I was sending out many HTTP requests which causes raising of many events
per request/response, capstats showed I was at 2~3 Mbps transfer rate on
the interface but when I checked the transfer rate of events between bro
and my broccoli client I was at 600~700Mbps, the events seem to be too
large, even when using compact events, and my broccoli client ended up
using 100% cpu of the core it was on, maybe enabling parallelism of this
section could give better results at events processing.


I've figured out the memory expansion problem, it seems that the
"ChunkQueue" in "ChunkedIO" does not have a limit and I was sending events
at higher speeds than my broccoli client could process so the queue just
kept growing.
I updated the queue so it will drop chunks when it reaches a certain limit
of chunks in the queue and now the memory stays steady at 1.5GB even at
high speeds of events.

This is a temporary fix in my opinion, a more robust communication
framework is needed such as using an external queue (such as ActiveMQ /
ZeroMQ) for transferring events/chunks.



Daniel.




On Sun, Aug 31, 2014 at 7:13 PM, Seth Hall <seth at icir.org> wrote:

>
> On Aug 27, 2014, at 1:29 PM, daniel nagar <dngr7512 at gmail.com> wrote:
>
> > I'm using bro 2.2 and I connect to bro using broccoli to receive events.
> > I can manage connecting to bro-worker and receive events, not sure if
> it's the correct way to receive event from bro but connecting to the
> manager port didn't retrieve any event whatsoever,
>
> Could you give more information about what events you are sending around?
> Are you receiving events from Bro or sending them to Bro?
>
> What configuration have you done in Bro to send or receive these events?
>
> > the problem is that when I receive events at speeds higher than 2Mbps
> the parent of the bro-worker (not the broccoli application) memory expands
> rapidly and can reach 10Gb in a minute.
>
> Another interesting number might be events per second.  I'm even a little
> unclear what you mean by 2Mbps.  Do you mean that the data rate of your
> connection between your broccoli application and Bro is 2Mbps?
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140831/eb2a0829/attachment.html