[Bro] Bro programming intro

[Tritium Cat]

Thanks for your comments, good to reflect on over coffee.  FWIW I am
running the cluster model with plenty of RAM and CPU to spare with a near
default config.

I do not believe what I am trying to learn and implement is premature
optimization but thanks for the reminder, I've made that mistake a few
times.


Another thing I would like to do is tag every orig_h and resp_h with
additional identifiers relative to the prefix, sorta like BGP ASNs.  I
usually use a Patricia-Trie for this.  Is there a special data type and BiF
I should consider ?

>From the documentation on extending logging [1] it seems that is a bit
beyond the scripting layer.  I read about the input framework [2] and it
seems it might work for this application but I'm not sure if this is best.
 What do you suggest ?  I would like all logs that have an orig_h and/or
resp_p to include the tags.


[1] http://www.bro.org/documentation/logging.html#extending
[2] http://www.bro.org/documentation/input.html

Thanks !

--TC



On Tue, Mar 19, 2013 at 9:51 AM, Clark, Gilbert <gc355804 at ohio.edu> wrote:

> Hi:
>
> One paper I can think of off-hand that appeared in RAID a few years back
> and may be relevant here:
>
> http://www.icir.org/vern/papers/autoconf-raid08.pdf
>
> On principle, there isn't a hard and fast rule for exactly how much state
> you can allocate or how many cycles you can spend processing individual
> packets: it's really going to depend on load, and will likely need to be
> experimentally determined (and constantly tuned).  One way to do this is to
> record snippets of the traffic you normally see at the border and run bro
> against that same set of traffic over and over (bro's —pseudo-realtime
> option can help get a realistic sample of what happens if you want to run
> this locally, or you can alternatively replay traces onto a local link to
> possibly get some more realistic results) while modifying the scripts bro
> has loaded to see what happens.  This kind of testing can help you identify
> the limits of what a specific bro configuration can handle in your
> environment.
>
> Also, I'd like to point out that some folks simply use bro for offline
> trace analysis and the like; while bro does well as a real-time tool, it
> doesn't necessarily have to be.
>
> "I am skeptical of how much emphasis is placed on doing things within
> BroIDS."
>
> Relevant cliche: Premature optimization is the root of much evil.
>
> The emphasis, I think, isn't placed on doing things within bro.  Instead,
> I believe the emphasis is placed on doing things that work for the folks
> who are deploying bro into their environments.  Most of the discussion here
> focuses on the scripting layer here because, in my humble opinion, that is
> the most straightforward interface to bro's event engine, and it's fast
> enough to do what folks need it to.  In the event it isn't, there's
> always the cluster model … there's at least one piece of pretty cool
> hardware I know of that rewrites destination MAC addresses and allows you
> to load-balance across a cluster pretty effectively, and PF_RING can also
> even let you cluster on a local box.  Often, investing in clustering winds
> up being cheaper (in both time and money) than trying to throw more experts
> at the problem to squeeze that last 15 Mbps out of a single node.
>
> Then again, trying to squeeze more out of a single node is always a fun
> programming challenge to solve :)
>
> Normal disclaimer applies: just my $0.02, I'm not an expert, etc.
>
> --Gilbert Clark
>
> From: Tritium Cat <tritium.cat at gmail.com>
> Date: Tue, 19 Mar 2013 11:46:03 -0400
> To: Seth Hall <seth at icir.org>
> Cc: "bro at bro-ids.org" <bro at bro-ids.org>
>
> Subject: Re: [Bro] Bro programming intro
>
> Ok. I am skeptical of how much emphasis is placed on doing things within
> BroIDS.  Simply buffering uids per Notice? seems much easier and less
> resource intensive than storing additional? samples.  Where is the limit
> with tracking too much state or using too many cycles within the "IDS" ?  I
> am weary of inadvertently creating DoS conditions with a philosophy that
> may encompass every script I write in Bro.
>
> I am still interested in a list of key papers on the internals if anyone
> has a few.
>
>
>
> On Tue, Mar 19, 2013 at 8:13 AM, Seth Hall <seth at icir.org> wrote:
>
>>
>> On Mar 18, 2013, at 8:03 PM, Tritium Cat <tritium.cat at gmail.com> wrote:
>>
>> > I want to modify the SQL Injection detection in
>> policy/protocols/http/detect-sqli.bro to include a vector that tracks the
>> associated http request uids and includes them in an additional log field.
>>  After getting it working I would like to apply it generally to other
>> Notices such as SSH Password_Guessing.
>>
>> The upcoming release actually results in this script getting rewritten a
>> bit because of a rewrite of the metrics (now measurement) framework.  The
>> new version actually keeps samples of the requests.  It will be relatively
>> easy to write your own script that tracks uid's instead of urls but the
>> benefit to sampling the urls is that if you have Bro send you email for the
>> notice it will add those sample urls to the email (it's been very
>> convenient for determining if something is a false positive without even
>> searching logs).
>>
>> Otherwise, with the metrics framework in 2.1 there isn't a good way to do
>> it.
>>
>>  .Seth
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130319/75f7ee49/attachment.html