From ghostpepper007 at gmail.com  Fri Mar  1 09:56:45 2013
From: ghostpepper007 at gmail.com (Ghost Pepper)
Date: Fri, 1 Mar 2013 12:56:45 -0500
Subject: [Bro] napatech nt4e-4t
Message-ID: <CAL0mE9orEqKtUNUg+2yWGeKLiYbZcoEkWpUJRc6nqPxg0EmrVg@mail.gmail.com>

Does anyone know if bro-ids capturing from a napatech card is known not to
work?
I've successfully compiled against the napatech provided libpcap, and ldd
shows linkage to
/opt/napatech3/lib/{ libpcap.so.1, libntapi.so, libntos.so }

But it keeps giving a "fatal error ... problem with interface nt3g1 -
pcap_setnoblock SIOCGIFADDR: nt3g1: No such device"

"tcpdump -D" shows the device and I can sniff with it.

Thought I'd check.

Thanks,
kb
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130301/7121c6b2/attachment.html 

From vallentin at icir.org  Sun Mar  3 17:36:09 2013
From: vallentin at icir.org (Matthias Vallentin)
Date: Sun, 3 Mar 2013 17:36:09 -0800
Subject: [Bro] Fwd: Bro SSL analyzer
In-Reply-To: <-1029011899094315077@unknownmsgid>
References: <-1029011899094315077@unknownmsgid>
Message-ID: <CADTpMNa2p9jmRVymeBZK4WadX5wR9sb5mbmCVfH922AgdJ1i9w@mail.gmail.com>

CC'ing the Bro mailing list for broader review.

    Matthias


---------- Forwarded message ----------
From: Ahir Reddy <ahirreddy at gmail.com>
Date: Sun, Mar 3, 2013 at 5:32 PM
Subject: Bro SSL analyzer
To: Matthias Vallentin <vallentin at icir.org>


Hi,

I was wondering if you have some insight into the SSL analyzer. I'm
having some issues detecting SSL alerts (in this case they are
transmitted after a FIN packet is seen). I've been trying to make
changes to SSL.cc, but I can't figure out what effectively closes the
SSL analyzer. I have already subclassed the TCP analyzer to detect RST
packets that appear after FINs, and I've been trying to do something
similar for the SSL Analyzer.

Ahir


From seth at icir.org  Mon Mar  4 06:15:45 2013
From: seth at icir.org (Seth Hall)
Date: Mon, 4 Mar 2013 09:15:45 -0500
Subject: [Bro] napatech nt4e-4t
In-Reply-To: <CAL0mE9orEqKtUNUg+2yWGeKLiYbZcoEkWpUJRc6nqPxg0EmrVg@mail.gmail.com>
References: <CAL0mE9orEqKtUNUg+2yWGeKLiYbZcoEkWpUJRc6nqPxg0EmrVg@mail.gmail.com>
Message-ID: <7F928CC1-FF10-4A63-A13D-A1D4C90D5B94@icir.org>


On Mar 1, 2013, at 12:56 PM, Ghost Pepper <ghostpepper007 at gmail.com> wrote:

> Does anyone know if bro-ids capturing from a napatech card is known not to work?
> I've successfully compiled against the napatech provided libpcap, and ldd shows linkage to
> /opt/napatech3/lib/{ libpcap.so.1, libntapi.so, libntos.so }

I've never heard that anyone has tried.  Does their libpcap wrapper support the nonblocking functionality in libpcap?  There are relatively few applications that use libpcap in nonblocking mode so it's certainly possible that they haven't implemented it.

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From ghostpepper007 at gmail.com  Mon Mar  4 08:55:16 2013
From: ghostpepper007 at gmail.com (Ghost Pepper)
Date: Mon, 4 Mar 2013 11:55:16 -0500
Subject: [Bro] napatech nt4e-4t
In-Reply-To: <7F928CC1-FF10-4A63-A13D-A1D4C90D5B94@icir.org>
References: <CAL0mE9orEqKtUNUg+2yWGeKLiYbZcoEkWpUJRc6nqPxg0EmrVg@mail.gmail.com>
	<7F928CC1-FF10-4A63-A13D-A1D4C90D5B94@icir.org>
Message-ID: <CAL0mE9on54Lfp2CGuaJerCzvw_r0XN9JyT5ktU_a5mfwV6e1NQ@mail.gmail.com>

I'm affraid I'm out of my league on the nonblocking issue.  I see
references to pcap_getnonblock and pcap_setnonblock in the pcap.h file
provided with the Napatech Software Suite.  Beyond that, at the moment I'm
not sure how to investigate further.  I may use pf_ring as an immediate
solution, and communicate with the Npulse team in the longer term, unless
of course someone saves the day with stories of success.

Thanks for your feedback!






On Mon, Mar 4, 2013 at 9:15 AM, Seth Hall <seth at icir.org> wrote:

>
> On Mar 1, 2013, at 12:56 PM, Ghost Pepper <ghostpepper007 at gmail.com>
> wrote:
>
> > Does anyone know if bro-ids capturing from a napatech card is known not
> to work?
> > I've successfully compiled against the napatech provided libpcap, and
> ldd shows linkage to
> > /opt/napatech3/lib/{ libpcap.so.1, libntapi.so, libntos.so }
>
> I've never heard that anyone has tried.  Does their libpcap wrapper
> support the nonblocking functionality in libpcap?  There are relatively few
> applications that use libpcap in nonblocking mode so it's certainly
> possible that they haven't implemented it.
>
>   .Seth
>
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130304/53caadbe/attachment.html 

From lists at g-clef.net  Wed Mar  6 13:35:18 2013
From: lists at g-clef.net (Aaron Gee-Clough)
Date: Wed, 06 Mar 2013 16:35:18 -0500
Subject: [Bro] working with Broccoli
Message-ID: <5137B696-00043FBD@straylight.g-clef.net>

Hello, everyone,  

I started this question over on the security onion list, but it was pointed
out that this list is probably the better venue for Bro questions. Anyway,
here's what I'm trying to do: I'm trying to pull all of the dns resolution
events for several DNS servers off the wire (from several different
locations, so there will be multiple listening boxes involved), so that I can
put the data in a database for alerting/reporting/etc. That separate database
(Arcsight) isn't something I can replace/remove at present (that's a
political decsion that I can't change), but I do want to get this DNS
information into it as cleanly as possible.  

My first assumption was that I would use Security Onion & Bro for this, since
the dns.log file contains exactly what I'm looking for. I had considered
simply dumping the dns.log file (or syslog sending it out), but broccoli came
to my attention as a way to make this process easier (allowing me to have one
central script to format the data for arcsight rather than having to teach
arcsight how to read bro logs)....but I've had a lot of trouble getting the
broccoli python bindings to do what I'm expecting of them.  

?  

Here's where I am at the moment:  

1) I have added the following to the bro config:  
>  
>
>@load policy/frameworks/communication/listen
>@load base/protocols/dns/main
>
>global api_test_log = open_log_file("/tmp/apitest.log");
>
>redef Communication::nodes += {
> ["api"] = [$host=127.0.0.1, $connect=F]
>};
>
>global dns_via_api: event(orig_h: addr, resp_h: addr, start_time: time, ans:
>dns_answer, reply: strin
>g);
> 
>event DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply:
>string) &priority=-4
> {
> event dns_via_api(c$id$orig_h, c$id$resp_h, c$start_time, ans, reply);
> print api_test_log, fmt("%s dns answer received, creating custom event.",?
>reply);
> }  

  

?  

The idea here is to add a custom event that only has the specific DNS bits
I'm interested in, and in theory my python script would subscribe to that
custom event. The api_test_log file is only there for me to verify that the
custom event is firing...once this is working properly I would remove that
bit. This seems to be working (I see records in the test log) from the bro
side, but I still haven't got the python side doing what I want yet.  

2) I have a python script that does this:  
>  
>
>import broccoli  
>
>dns_answer = broccoli.record_type("answer_type", "query", "qtype", "qclass",
>"TTL")
>
>@broccoli.event(broccoli.addr, broccoli.addr, broccoli.time, dns_answer,
>broccoli.string)
>def dns_via_api(orig_h, resp_h, start_time, ans, reply):
> print reply
>
>bc = broccoli.Connection("127.0.0.1:47763", connect=False)
>bc.subscribe("dns_via_api", dns_via_api)
>bc.connect()
>
>while True:
> bc.processInput()
> time.sleep(1)  

  

This gets *something* (which is progress), but the script crashes with:  
>  
>
>File "/opt/bro/lib/broctl/broccoli.py", line 87, in wrapped_f
> new_args += [instantiate(btype, val, type)]
> File "/opt/bro/lib/broctl/broccoli.py", line 408, in instantiate
> return _Factories[src_type](val, dst_type)
> File "/opt/bro/lib/broctl/broccoli.py", line 282, in _factory
> return record(dst_type, vals)
> File "/opt/bro/lib/broctl/broccoli.py", line 297, in __init__
> for (key, val) in zip(type.fields, vals):
>AttributeError: 'NoneType' object has no attribute 'fields'  

  

?  

Could any of you give me a pointer as to what I've missed here?  

?  

Thank you.  

?  

aaron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130306/4a057acb/attachment.html 

From jsiwek at illinois.edu  Wed Mar  6 14:31:59 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Wed, 6 Mar 2013 22:31:59 +0000
Subject: [Bro] working with Broccoli
In-Reply-To: <5137B696-00043FBD@straylight.g-clef.net>
References: <5137B696-00043FBD@straylight.g-clef.net>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D116F41815@CITESMBX2.ad.uillinois.edu>


> import broccoli
> 
> dns_answer = broccoli.record_type("answer_type", "query", "qtype", "qclass", "TTL")
> 
> @broccoli.event(broccoli.addr, broccoli.addr, broccoli.time, dns_answer, broccoli.string)
> def dns_via_api(orig_h, resp_h, start_time, ans, reply):
> print reply
> 
> bc = broccoli.Connection("127.0.0.1:47763", connect=False)
> bc.subscribe("dns_via_api", dns_via_api)
> bc.connect()
> 
> while True:
> bc.processInput()
> time.sleep(1)

Try removing the bc.subscribe() call.  I think it's only for use with non-decorated event callbacks, but you've already got the @broccoli.event decoration there.

    Jon


From mkolkebeck at gmail.com  Wed Mar  6 16:09:10 2013
From: mkolkebeck at gmail.com (Mike Kolkebeck)
Date: Wed, 6 Mar 2013 18:09:10 -0600
Subject: [Bro] smb crashing workers
Message-ID: <2D264538-D1FD-464A-959D-128B385FA5EA@gmail.com>

I know the smb analyzer still has a few bugs in Bro-2.1 , but is there any way to prevent/catch the bro workers from infrequently crashing (~2-3 times per day, under peak periods)?
I've included partial output from stderr.log, which I believe is a good representation of each crash output.

Also, probably a dumb question, but when a worker crashes, will it completely stop logging activity until it is restarted (by broctl cron)? I assume so. Would there be any harm in running broctl cron every minute, as opposed to 5 minutes?

Unfortunately I'm not able to identify the cause of the crash, other than bugs in the code, so any guidance or available tools to investigate the cause would also be helpful.

Thanks!
Mike


stderr.log excerpt:
--------------------------
bro: /home/xuser1/bro-2.1/build/src/smb_pac.cc:517: int binpac::SMB::SMB_unicode_string::Parse(const binpac::uint8*, const binpac::uint8*, int): Assertion `t_dataptr_after_s <= t_end_of_data' failed.
/usr/local/bro/share/broctl/scripts/run-bro: line 60: 12342 Aborted                 (core dumped) nohup $mybro $@
listening on eth2, capture length 8192 bytes


From seth at icir.org  Wed Mar  6 17:32:16 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 6 Mar 2013 20:32:16 -0500
Subject: [Bro] smb crashing workers
In-Reply-To: <2D264538-D1FD-464A-959D-128B385FA5EA@gmail.com>
References: <2D264538-D1FD-464A-959D-128B385FA5EA@gmail.com>
Message-ID: <2CF596D0-7D50-400F-93AA-827B85FC8226@icir.org>


On Mar 6, 2013, at 7:09 PM, Mike Kolkebeck <mkolkebeck at gmail.com> wrote:

> I know the smb analyzer still has a few bugs in Bro-2.1 , but is there any way to prevent/catch the bro workers from infrequently crashing (~2-3 times per day, under peak periods)?

Yep, don't enable the SMB analyzer in 2.1. :)  

We don't have any scripts that enable it or use it because it's not ready and very incomplete in the 2.1 release.  Are you getting some logs or anything from it?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From lists at g-clef.net  Thu Mar  7 07:10:15 2013
From: lists at g-clef.net (Aaron Gee-Clough)
Date: Thu, 07 Mar 2013 10:10:15 -0500
Subject: [Bro] working with Broccoli
References: <5137B696-00043FBD@straylight.g-clef.net>
	<B3E27C90B7D2DD4CAEF09215A29EB0D116F41815@CITESMBX2.ad.uillinois.edu>
Message-ID: <5138ADD7-0004405F@straylight.g-clef.net>

?  
>  Wed Mar 06 2013 05:31:59 PM EST from  "Siwek, Jonathan Luke"
><jsiwek at illinois.edu>  Subject: Re: [Bro] working with Broccoli
>
>    
>>import broccoli
>> 
>> dns_answer = broccoli.record_type("answer_type", "query", "qtype",
>>"qclass", "TTL")
>> 
>> @broccoli.event(broccoli.addr, broccoli.addr, broccoli.time, dns_answer,
>>broccoli.string)
>> def dns_via_api(orig_h, resp_h, start_time, ans, reply):
>> print reply
>> 
>> bc = broccoli.Connection("127.0.0.1:47763", connect=False)
>> bc.subscribe("dns_via_api", dns_via_api)
>> bc.connect()
>> 
>> while True:
>> bc.processInput()
>> time.sleep(1)
>> 
>>

>  Try removing the bc.subscribe() call.  I think it's only for use with
>non-decorated event callbacks, but you've already got the @broccoli.event
>decoration there.
> 
> Jon
> 
>
>
>  

  

?  

That was it. the callbacks are working beautifully now. Thank you very much. 


?  

aaron
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130307/5b84d946/attachment.html 

From jessebowling at gmail.com  Fri Mar  8 10:51:27 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Fri, 8 Mar 2013 13:51:27 -0500
Subject: [Bro] Dropping all packets, but not crashed?
Message-ID: <CAGFhEnknFdNVfixp_Ei=hZm9is-sS5ELW9T=XckXLZ67DTJEYA@mail.gmail.com>

I noticed today while reviewing my notice.log that one worker thread has
been consistently dropping all packets that it received...The status
indicated that it was running, and a restart of the worker did not indicate
that anything was crashed or that it exited oddly...After using broctl to
restart the worker, no more notices...

I imagine it's too late to gather more info about this now, but if the
situation should present itself again, how would I gather the most debug
information to try to find out why? Are there settings I should turn on
now, or commands I should run at the time? strace, gdb, etc?

Is it too late to get more info about why this was happening?

I also just happened to visit the securityonion page and notice this at the
top:

"An issue was recently discovered in Bro 2.1 when monitoring multiple
interfaces with PF_RING that could result in traffic loss.  This issue is
targeted for resolution in Bro 2.2.  In the meantime, if you're monitoring
multiple interfaces with Bro, please disable Bro's PF_RING load balancing
as follows:"

This could perhaps describe my situation....Anyone have any more specifics
on this?

Cheers,

Jesse

-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130308/d3cc1359/attachment.html 

From scampbell at lbl.gov  Fri Mar  8 11:00:44 2013
From: scampbell at lbl.gov (Scott Campbell)
Date: Fri, 08 Mar 2013 13:00:44 -0600
Subject: [Bro] Dropping all packets, but not crashed?
In-Reply-To: <CAGFhEnknFdNVfixp_Ei=hZm9is-sS5ELW9T=XckXLZ67DTJEYA@mail.gmail.com>
References: <CAGFhEnknFdNVfixp_Ei=hZm9is-sS5ELW9T=XckXLZ67DTJEYA@mail.gmail.com>
Message-ID: <513A355C.4000506@lbl.gov>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

We saw a very similar thing here - there ended up being an issue with
PF_RING < 5.5.2 where corrupted VLAN tagged packets caused the exact
situation you describe.  We were seeing this 2-3 times a day.

I upgraded the PF_RING to 5.2.2 and the issue went away.  This problem
is listed in the ChangeLog as well.

cheers,
scott

On 3/8/13 12:51 PM, Jesse Bowling wrote:
> I noticed today while reviewing my notice.log that one worker
> thread has been consistently dropping all packets that it
> received...The status indicated that it was running, and a restart
> of the worker did not indicate that anything was crashed or that it
> exited oddly...After using broctl to restart the worker, no more
> notices...
> 
> I imagine it's too late to gather more info about this now, but if
> the situation should present itself again, how would I gather the
> most debug information to try to find out why? Are there settings I
> should turn on now, or commands I should run at the time? strace,
> gdb, etc?
> 
> Is it too late to get more info about why this was happening?
> 
> I also just happened to visit the securityonion page and notice
> this at the top:
> 
> "An issue was recently discovered in Bro 2.1 when monitoring
> multiple interfaces with PF_RING that could result in traffic loss.
> This issue is targeted for resolution in Bro 2.2.  In the meantime,
> if you're monitoring multiple interfaces with Bro, please disable
> Bro's PF_RING load balancing as follows:"
> 
> This could perhaps describe my situation....Anyone have any more
> specifics on this?
> 
> Cheers,
> 
> Jesse
> 
> 
> 
> _______________________________________________ Bro mailing list 
> bro at bro-ids.org 
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iD8DBQFROjVcK2Plq8B7ZBwRAtDWAJ9YDKTJ4M8bxzyEiSbLpI1ycWj3/QCgrUeU
sqoCq0UeO7DA1JQ1H/tOweo=
=wLLG
-----END PGP SIGNATURE-----


From jessebowling at gmail.com  Fri Mar  8 11:03:06 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Fri, 8 Mar 2013 14:03:06 -0500
Subject: [Bro] Dropping all packets, but not crashed?
In-Reply-To: <513A355C.4000506@lbl.gov>
References: <CAGFhEnknFdNVfixp_Ei=hZm9is-sS5ELW9T=XckXLZ67DTJEYA@mail.gmail.com>
	<513A355C.4000506@lbl.gov>
Message-ID: <CAGFhEnmO-ckhiC_9OhJAxj0Q06_JHESWv1qnPp_sstT29gAj1A@mail.gmail.com>

Thanks Scott!

I'm due for an upgrade on PF_RING so knowing this might be related is more
fuel to the fire.

Cheers,

Jesse

On Fri, Mar 8, 2013 at 2:00 PM, Scott Campbell <scampbell at lbl.gov> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> We saw a very similar thing here - there ended up being an issue with
> PF_RING < 5.5.2 where corrupted VLAN tagged packets caused the exact
> situation you describe.  We were seeing this 2-3 times a day.
>
> I upgraded the PF_RING to 5.2.2 and the issue went away.  This problem
> is listed in the ChangeLog as well.
>
> cheers,
> scott
>
> On 3/8/13 12:51 PM, Jesse Bowling wrote:
> > I noticed today while reviewing my notice.log that one worker
> > thread has been consistently dropping all packets that it
> > received...The status indicated that it was running, and a restart
> > of the worker did not indicate that anything was crashed or that it
> > exited oddly...After using broctl to restart the worker, no more
> > notices...
> >
> > I imagine it's too late to gather more info about this now, but if
> > the situation should present itself again, how would I gather the
> > most debug information to try to find out why? Are there settings I
> > should turn on now, or commands I should run at the time? strace,
> > gdb, etc?
> >
> > Is it too late to get more info about why this was happening?
> >
> > I also just happened to visit the securityonion page and notice
> > this at the top:
> >
> > "An issue was recently discovered in Bro 2.1 when monitoring
> > multiple interfaces with PF_RING that could result in traffic loss.
> > This issue is targeted for resolution in Bro 2.2.  In the meantime,
> > if you're monitoring multiple interfaces with Bro, please disable
> > Bro's PF_RING load balancing as follows:"
> >
> > This could perhaps describe my situation....Anyone have any more
> > specifics on this?
> >
> > Cheers,
> >
> > Jesse
> >
> >
> >
> > _______________________________________________ Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (Darwin)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iD8DBQFROjVcK2Plq8B7ZBwRAtDWAJ9YDKTJ4M8bxzyEiSbLpI1ycWj3/QCgrUeU
> sqoCq0UeO7DA1JQ1H/tOweo=
> =wLLG
> -----END PGP SIGNATURE-----
>



-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130308/260cb5bc/attachment.html 

From giralt at reservoir.com  Mon Mar 11 01:38:14 2013
From: giralt at reservoir.com (Jordi Ros-Giralt)
Date: Mon, 11 Mar 2013 09:38:14 +0100
Subject: [Bro] Dropping all packets, but not crashed?
In-Reply-To: <CAGFhEnmO-ckhiC_9OhJAxj0Q06_JHESWv1qnPp_sstT29gAj1A@mail.gmail.com>
References: <CAGFhEnknFdNVfixp_Ei=hZm9is-sS5ELW9T=XckXLZ67DTJEYA@mail.gmail.com>
	<513A355C.4000506@lbl.gov>
	<CAGFhEnmO-ckhiC_9OhJAxj0Q06_JHESWv1qnPp_sstT29gAj1A@mail.gmail.com>
Message-ID: <CAL_2ynbTmXgLSZryOLnoudbPuX8rR9htEJ0ri557PmA-Tor_Lw@mail.gmail.com>

Hi Jesse,

Regarding the note reported in the securityonion page, that issue would
happen only when using PF_RING and multiple input interfaces.  The issue
comes because currently Bro does not support multiple PF_RING clusters (it
supports configuring one single PF_RING cluster).  If you are running a
single interface, then this would not be your problem, otherwise, this
could be a cause of packet drops.

Seth filed a ticket (Ticket #943) so this issue is being tracked already.

Cheers,
Jordi


On Fri, Mar 8, 2013 at 8:03 PM, Jesse Bowling <jessebowling at gmail.com>wrote:

> Thanks Scott!
>
> I'm due for an upgrade on PF_RING so knowing this might be related is more
> fuel to the fire.
>
> Cheers,
>
> Jesse
>
>
> On Fri, Mar 8, 2013 at 2:00 PM, Scott Campbell <scampbell at lbl.gov> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> We saw a very similar thing here - there ended up being an issue with
>> PF_RING < 5.5.2 where corrupted VLAN tagged packets caused the exact
>> situation you describe.  We were seeing this 2-3 times a day.
>>
>> I upgraded the PF_RING to 5.2.2 and the issue went away.  This problem
>> is listed in the ChangeLog as well.
>>
>> cheers,
>> scott
>>
>> On 3/8/13 12:51 PM, Jesse Bowling wrote:
>> > I noticed today while reviewing my notice.log that one worker
>> > thread has been consistently dropping all packets that it
>> > received...The status indicated that it was running, and a restart
>> > of the worker did not indicate that anything was crashed or that it
>> > exited oddly...After using broctl to restart the worker, no more
>> > notices...
>> >
>> > I imagine it's too late to gather more info about this now, but if
>> > the situation should present itself again, how would I gather the
>> > most debug information to try to find out why? Are there settings I
>> > should turn on now, or commands I should run at the time? strace,
>> > gdb, etc?
>> >
>> > Is it too late to get more info about why this was happening?
>> >
>> > I also just happened to visit the securityonion page and notice
>> > this at the top:
>> >
>> > "An issue was recently discovered in Bro 2.1 when monitoring
>> > multiple interfaces with PF_RING that could result in traffic loss.
>> > This issue is targeted for resolution in Bro 2.2.  In the meantime,
>> > if you're monitoring multiple interfaces with Bro, please disable
>> > Bro's PF_RING load balancing as follows:"
>> >
>> > This could perhaps describe my situation....Anyone have any more
>> > specifics on this?
>> >
>> > Cheers,
>> >
>> > Jesse
>> >
>> >
>> >
>> > _______________________________________________ Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> >
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.9 (Darwin)
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>> iD8DBQFROjVcK2Plq8B7ZBwRAtDWAJ9YDKTJ4M8bxzyEiSbLpI1ycWj3/QCgrUeU
>> sqoCq0UeO7DA1JQ1H/tOweo=
>> =wLLG
>> -----END PGP SIGNATURE-----
>>
>
>
>
> --
> Jesse Bowling
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130311/322da52f/attachment.html 

From jones at tacc.utexas.edu  Wed Mar 13 12:23:21 2013
From: jones at tacc.utexas.edu (William Jones)
Date: Wed, 13 Mar 2013 19:23:21 +0000
Subject: [Bro] Centos 6.x drops packerts do to the use of the linux ring
 ineterface inthe system pcap library
Message-ID: <D0BD4A5C03717E40AC104854AF37B8C225D68F59@EXMBX01.austin.utexas.edu>

I believe that this is do to  not enough buffering in the ring.  Does anyone know how to tune the linux native pcap to increase the buffering in the linux  ring interface.

Bill Jones
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130313/2682de08/attachment.html 

From jlay at slave-tothe-box.net  Wed Mar 13 14:13:22 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 13 Mar 2013 15:13:22 -0600
Subject: [Bro] Newb with a couple questions
Message-ID: <75a03494b26a7182774c30241dfaf118@localhost>

Hey all!

I am wondering how to filter protocols from the conn.log?  I've been 
looking at:

http://www.bro.org/documentation/logging.html#filtering

But that seems to point to filtering columns, and not protocols.  
Thanks for any insight.

James


From anthony.kasza at gmail.com  Wed Mar 13 15:14:38 2013
From: anthony.kasza at gmail.com (anthony kasza)
Date: Wed, 13 Mar 2013 17:14:38 -0500
Subject: [Bro] Newb with a couple questions
In-Reply-To: <75a03494b26a7182774c30241dfaf118@localhost>
References: <75a03494b26a7182774c30241dfaf118@localhost>
Message-ID: <CAEZw2bzn_Dg2yadE5WyiL1sJWET68uX13ay8ZsaCM+WhWabj8g@mail.gmail.com>

Depending on what you are trying to accomplish, you can filter the
data by protocol after it's been written to the conn.log file with
bro-cut or awk.

-Anthony

On Wed, Mar 13, 2013 at 4:13 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> Hey all!
>
> I am wondering how to filter protocols from the conn.log?  I've been
> looking at:
>
> http://www.bro.org/documentation/logging.html#filtering
>
> But that seems to point to filtering columns, and not protocols.
> Thanks for any insight.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From jlay at slave-tothe-box.net  Wed Mar 13 15:24:07 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 13 Mar 2013 16:24:07 -0600
Subject: [Bro] Newb with a couple questions
In-Reply-To: <CAEZw2bzn_Dg2yadE5WyiL1sJWET68uX13ay8ZsaCM+WhWabj8g@mail.gmail.com>
References: <75a03494b26a7182774c30241dfaf118@localhost>
	<CAEZw2bzn_Dg2yadE5WyiL1sJWET68uX13ay8ZsaCM+WhWabj8g@mail.gmail.com>
Message-ID: <9f197289ccd5b032183d0aaf4f392031@localhost>

On 2013-03-13 16:14, anthony kasza wrote:
> Depending on what you are trying to accomplish, you can filter the
> data by protocol after it's been written to the conn.log file with
> bro-cut or awk.
>
> -Anthony
>

Hi Anthony,

Ideally the protocols would be dropped before logging.  I already have 
dns and http logging using Bro, so seeing them in the connections log 
seems a tad redundant.  Thanks for the quick response.

James



From seth at icir.org  Wed Mar 13 17:33:04 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 13 Mar 2013 20:33:04 -0400
Subject: [Bro] Newb with a couple questions
In-Reply-To: <75a03494b26a7182774c30241dfaf118@localhost>
References: <75a03494b26a7182774c30241dfaf118@localhost>
Message-ID: <9F62C845-DE4E-4470-8397-8F0B791F68CE@icir.org>


On Mar 13, 2013, at 5:13 PM, James Lay <jlay at slave-tothe-box.net> wrote:

> But that seems to point to filtering columns, and not protocols.  
> Thanks for any insight.


I wrote a blog post about log filtering a while ago that should help you out:
	http://blog.bro.org/2012/02/filtering-logs-with-bro.html

Does that help?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From mfw113 at psu.edu  Thu Mar 14 03:47:58 2013
From: mfw113 at psu.edu (MICHAEL WAITE)
Date: Thu, 14 Mar 2013 06:47:58 -0400 (EDT)
Subject: [Bro] Newb with a couple questions
In-Reply-To: <9f197289ccd5b032183d0aaf4f392031@localhost>
References: <75a03494b26a7182774c30241dfaf118@localhost>
	<CAEZw2bzn_Dg2yadE5WyiL1sJWET68uX13ay8ZsaCM+WhWabj8g@mail.gmail.com>
	<9f197289ccd5b032183d0aaf4f392031@localhost>
Message-ID: <EB3C4197-7439-49AA-8574-F8F0C7B61570@psu.edu>

I would not call the conn log redundant. The http and conn log are very different and have different data in them. Rather they complement each other. 

-Mike

On Mar 13, 2013, at 18:31, James Lay <jlay at slave-tothe-box.net> wrote:

> On 2013-03-13 16:14, anthony kasza wrote:
>> Depending on what you are trying to accomplish, you can filter the
>> data by protocol after it's been written to the conn.log file with
>> bro-cut or awk.
>> 
>> -Anthony
> 
> Hi Anthony,
> 
> Ideally the protocols would be dropped before logging.  I already have 
> dns and http logging using Bro, so seeing them in the connections log 
> seems a tad redundant.  Thanks for the quick response.
> 
> James
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From jlay at slave-tothe-box.net  Thu Mar 14 03:53:41 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 14 Mar 2013 04:53:41 -0600
Subject: [Bro] Newb with a couple questions
In-Reply-To: <EB3C4197-7439-49AA-8574-F8F0C7B61570@psu.edu>
References: <75a03494b26a7182774c30241dfaf118@localhost>
	<CAEZw2bzn_Dg2yadE5WyiL1sJWET68uX13ay8ZsaCM+WhWabj8g@mail.gmail.com>
	<9f197289ccd5b032183d0aaf4f392031@localhost>
	<EB3C4197-7439-49AA-8574-F8F0C7B61570@psu.edu>
Message-ID: <84857463-9A49-4E53-B9A2-B1D9DF004E40@slave-tothe-box.net>


On Mar 14, 2013, at 4:47 AM, MICHAEL WAITE <mfw113 at psu.edu> wrote:

> I would not call the conn log redundant. The http and conn log are very different and have different data in them. Rather they complement each other. 
> 
> -Mike
> 
> On Mar 13, 2013, at 18:31, James Lay <jlay at slave-tothe-box.net> wrote:
> 
>> On 2013-03-13 16:14, anthony kasza wrote:
>>> Depending on what you are trying to accomplish, you can filter the
>>> data by protocol after it's been written to the conn.log file with
>>> bro-cut or awk.
>>> 
>>> -Anthony
>> 
>> Hi Anthony,
>> 
>> Ideally the protocols would be dropped before logging.  I already have 
>> dns and http logging using Bro, so seeing them in the connections log 
>> seems a tad redundant.  Thanks for the quick response.
>> 
>> James
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

A fair point.  And I'll give that a go Seth thank you?gonna be busy the rest of this week, so I'll report my results on Monday.  Thanks for the assistance all.

James


From rjenkins at rmjconsulting.net  Thu Mar 14 03:54:20 2013
From: rjenkins at rmjconsulting.net (Ron Jenkins)
Date: Thu, 14 Mar 2013 10:54:20 +0000
Subject: [Bro] Newb with a couple questions
In-Reply-To: <EB3C4197-7439-49AA-8574-F8F0C7B61570@psu.edu>
References: <75a03494b26a7182774c30241dfaf118@localhost>
	<CAEZw2bzn_Dg2yadE5WyiL1sJWET68uX13ay8ZsaCM+WhWabj8g@mail.gmail.com>
	<9f197289ccd5b032183d0aaf4f392031@localhost>,
	<EB3C4197-7439-49AA-8574-F8F0C7B61570@psu.edu>
Message-ID: <7E3DC799-4D13-4A84-B229-949F20D0DE4C@rmjconsulting.net>

And very much needed if there is a comprise on the network. 



Ron Jenkins (SnortCP,VCP 3 / 4,MCNE,MCPS,MCNPS,CCNA)
RMJ Consulting, LLC.
"Bringing Companies and Solutions Together"
Owner / Senior Architect
Physical Address
11715 Bricksome Ave STE B-7
Baton Rouge, LA 70816
Mail Address
7575 Jefferson Hwy #103
Baton Rouge, LA 70806
Toll. 855-448-5214
Direct. 225-448-5214
Fax. 225-448-5324
Cell. 225-931-1632
Email. rjenkins at rmjconsulting.net
Web. http://www.rmjconsulting.net<http://www.rmjconsulting.net/>
http://www.linkedin.com/in/ronmjenkins

On Mar 14, 2013, at 5:50 AM, "MICHAEL WAITE" <mfw113 at psu.edu> wrote:

> I would not call the conn log redundant. The http and conn log are very different and have different data in them. Rather they complement each other. 
> 
> -Mike
> 
> On Mar 13, 2013, at 18:31, James Lay <jlay at slave-tothe-box.net> wrote:
> 
>> On 2013-03-13 16:14, anthony kasza wrote:
>>> Depending on what you are trying to accomplish, you can filter the
>>> data by protocol after it's been written to the conn.log file with
>>> bro-cut or awk.
>>> 
>>> -Anthony
>> 
>> Hi Anthony,
>> 
>> Ideally the protocols would be dropped before logging.  I already have 
>> dns and http logging using Bro, so seeing them in the connections log 
>> seems a tad redundant.  Thanks for the quick response.
>> 
>> James
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From rjenkins at rmjconsulting.net  Thu Mar 14 03:54:59 2013
From: rjenkins at rmjconsulting.net (Ron Jenkins)
Date: Thu, 14 Mar 2013 10:54:59 +0000
Subject: [Bro] Newb with a couple questions
In-Reply-To: <EB3C4197-7439-49AA-8574-F8F0C7B61570@psu.edu>
References: <75a03494b26a7182774c30241dfaf118@localhost>
	<CAEZw2bzn_Dg2yadE5WyiL1sJWET68uX13ay8ZsaCM+WhWabj8g@mail.gmail.com>
	<9f197289ccd5b032183d0aaf4f392031@localhost>,
	<EB3C4197-7439-49AA-8574-F8F0C7B61570@psu.edu>
Message-ID: <57794A7B-956A-4DDD-AFB0-49A8F2FC1FE1@rmjconsulting.net>

No such thing as too much logging. 

Ron Jenkins (SnortCP,VCP 3 / 4,MCNE,MCPS,MCNPS,CCNA)
RMJ Consulting, LLC.
"Bringing Companies and Solutions Together"
Owner / Senior Architect
Physical Address
11715 Bricksome Ave STE B-7
Baton Rouge, LA 70816
Mail Address
7575 Jefferson Hwy #103
Baton Rouge, LA 70806
Toll. 855-448-5214
Direct. 225-448-5214
Fax. 225-448-5324
Cell. 225-931-1632
Email. rjenkins at rmjconsulting.net
Web. http://www.rmjconsulting.net<http://www.rmjconsulting.net/>
http://www.linkedin.com/in/ronmjenkins

On Mar 14, 2013, at 5:50 AM, "MICHAEL WAITE" <mfw113 at psu.edu> wrote:

> I would not call the conn log redundant. The http and conn log are very different and have different data in them. Rather they complement each other. 
> 
> -Mike
> 
> On Mar 13, 2013, at 18:31, James Lay <jlay at slave-tothe-box.net> wrote:
> 
>> On 2013-03-13 16:14, anthony kasza wrote:
>>> Depending on what you are trying to accomplish, you can filter the
>>> data by protocol after it's been written to the conn.log file with
>>> bro-cut or awk.
>>> 
>>> -Anthony
>> 
>> Hi Anthony,
>> 
>> Ideally the protocols would be dropped before logging.  I already have 
>> dns and http logging using Bro, so seeing them in the connections log 
>> seems a tad redundant.  Thanks for the quick response.
>> 
>> James
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at icir.org  Thu Mar 14 06:09:22 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 14 Mar 2013 09:09:22 -0400
Subject: [Bro] Newb with a couple questions
In-Reply-To: <EB3C4197-7439-49AA-8574-F8F0C7B61570@psu.edu>
References: <75a03494b26a7182774c30241dfaf118@localhost>
	<CAEZw2bzn_Dg2yadE5WyiL1sJWET68uX13ay8ZsaCM+WhWabj8g@mail.gmail.com>
	<9f197289ccd5b032183d0aaf4f392031@localhost>
	<EB3C4197-7439-49AA-8574-F8F0C7B61570@psu.edu>
Message-ID: <9BA351E0-305E-48D5-B688-8238462306F5@icir.org>


On Mar 14, 2013, at 6:47 AM, MICHAEL WAITE <mfw113 at psu.edu> wrote:

> I would not call the conn log redundant. The http and conn log are very different and have different data in them. Rather they complement each other. 


Additionally, the conn log seems to be getting more important over time.  I've run into several sites already that aren't maintaining a conn.log and they might see tunnels being identified on their network (with the tunnel.log) but they don't know if any connections happened over the tunnel because that is indicated in the tunnel log.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From rjenkins at rmjconsulting.net  Thu Mar 14 06:21:26 2013
From: rjenkins at rmjconsulting.net (Ron Jenkins)
Date: Thu, 14 Mar 2013 13:21:26 +0000
Subject: [Bro] Newb with a couple questions
In-Reply-To: <9BA351E0-305E-48D5-B688-8238462306F5@icir.org>
References: <75a03494b26a7182774c30241dfaf118@localhost>
	<CAEZw2bzn_Dg2yadE5WyiL1sJWET68uX13ay8ZsaCM+WhWabj8g@mail.gmail.com>
	<9f197289ccd5b032183d0aaf4f392031@localhost>
	<EB3C4197-7439-49AA-8574-F8F0C7B61570@psu.edu>,
	<9BA351E0-305E-48D5-B688-8238462306F5@icir.org>
Message-ID: <D01E6CC5-A261-42FA-A491-92B52CF1D076@rmjconsulting.net>

Question

Is the tunnel log on by default?

Thx

Ron Jenkins (SnortCP,VCP 3 / 4,MCNE,CNE6,MCPS,MCNPS,CCNA)
RMJ Consulting, LLC.
"Bringing Companies and Solutions Together"
Owner / Senior Architect
Physical Address
11715 Bricksome Ave STE B-7
Baton Rouge, LA 70816
Mail Address
7575 Jefferson Hwy #103
Baton Rouge, LA 70806
Office. 225-448-5214
Fax. 225-448-5324
Cell. 225-931-1632
Email. rjenkins at rmjconsulting.net
Web. http://www.rmjconsulting.net<http://www.rmjconsulting.net/>

On Mar 14, 2013, at 8:19 AM, "Seth Hall" <seth at icir.org> wrote:

> 
> On Mar 14, 2013, at 6:47 AM, MICHAEL WAITE <mfw113 at psu.edu> wrote:
> 
>> I would not call the conn log redundant. The http and conn log are very different and have different data in them. Rather they complement each other.
> 
> 
> Additionally, the conn log seems to be getting more important over time.  I've run into several sites already that aren't maintaining a conn.log and they might see tunnels being identified on their network (with the tunnel.log) but they don't know if any connections happened over the tunnel because that is indicated in the tunnel log.
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at icir.org  Thu Mar 14 06:25:19 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 14 Mar 2013 09:25:19 -0400
Subject: [Bro] Newb with a couple questions
In-Reply-To: <D01E6CC5-A261-42FA-A491-92B52CF1D076@rmjconsulting.net>
References: <75a03494b26a7182774c30241dfaf118@localhost>
	<CAEZw2bzn_Dg2yadE5WyiL1sJWET68uX13ay8ZsaCM+WhWabj8g@mail.gmail.com>
	<9f197289ccd5b032183d0aaf4f392031@localhost>
	<EB3C4197-7439-49AA-8574-F8F0C7B61570@psu.edu>,
	<9BA351E0-305E-48D5-B688-8238462306F5@icir.org>
	<D01E6CC5-A261-42FA-A491-92B52CF1D076@rmjconsulting.net>
Message-ID: <79B6682F-9FA8-45D8-8762-A28C7162F56F@icir.org>


On Mar 14, 2013, at 9:21 AM, Ron Jenkins <rjenkins at rmjconsulting.net> wrote:

> Is the tunnel log on by default?


Yep.  2.1 automatically identifies and decapsulates Teredo, IP-in-IP (6to4, etc), AYIYA, GTP (further fixes to this are in git now), and SOCKS.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From rjenkins at rmjconsulting.net  Thu Mar 14 06:26:36 2013
From: rjenkins at rmjconsulting.net (Ron Jenkins)
Date: Thu, 14 Mar 2013 13:26:36 +0000
Subject: [Bro] Newb with a couple questions
In-Reply-To: <79B6682F-9FA8-45D8-8762-A28C7162F56F@icir.org>
References: <75a03494b26a7182774c30241dfaf118@localhost>
	<CAEZw2bzn_Dg2yadE5WyiL1sJWET68uX13ay8ZsaCM+WhWabj8g@mail.gmail.com>
	<9f197289ccd5b032183d0aaf4f392031@localhost>
	<EB3C4197-7439-49AA-8574-F8F0C7B61570@psu.edu>,
	<9BA351E0-305E-48D5-B688-8238462306F5@icir.org>
	<D01E6CC5-A261-42FA-A491-92B52CF1D076@rmjconsulting.net>,
	<79B6682F-9FA8-45D8-8762-A28C7162F56F@icir.org>
Message-ID: <37E8C80D-459A-4BC4-A5E8-C70B16B4A1B9@rmjconsulting.net>

Nice!


Thx

Ron Jenkins (SnortCP,VCP 3 / 4,MCNE,CNE6,MCPS,MCNPS,CCNA)
RMJ Consulting, LLC.
"Bringing Companies and Solutions Together"
Owner / Senior Architect
Physical Address
11715 Bricksome Ave STE B-7
Baton Rouge, LA 70816
Mail Address
7575 Jefferson Hwy #103
Baton Rouge, LA 70806
Office. 225-448-5214
Fax. 225-448-5324
Cell. 225-931-1632
Email. rjenkins at rmjconsulting.net
Web. http://www.rmjconsulting.net<http://www.rmjconsulting.net/>

On Mar 14, 2013, at 8:25 AM, "Seth Hall" <seth at icir.org> wrote:

> 
> On Mar 14, 2013, at 9:21 AM, Ron Jenkins <rjenkins at rmjconsulting.net> wrote:
> 
>> Is the tunnel log on by default?
> 
> 
> Yep.  2.1 automatically identifies and decapsulates Teredo, IP-in-IP (6to4, etc), AYIYA, GTP (further fixes to this are in git now), and SOCKS.
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 



From jp.bourget at gmail.com  Thu Mar 14 12:43:33 2013
From: jp.bourget at gmail.com (JP Bourget)
Date: Thu, 14 Mar 2013 15:43:33 -0400
Subject: [Bro] printing metrics to screen
Message-ID: <CAGb0YipfnpRt0PuaLRtJcF7YhNgb5zj1hCNC_Pu+Ct27NTk1vw@mail.gmail.com>

I've tried to figure this out for awhile...

How can I print a metric I'm tracking to screen to confirm it's doing what
I think it's doing?

Thanks!

-- 
JP
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130314/f6996862/attachment.html 

From srunnels at gmail.com  Thu Mar 14 12:58:58 2013
From: srunnels at gmail.com (Scott Runnels)
Date: Thu, 14 Mar 2013 15:58:58 -0400
Subject: [Bro] printing metrics to screen
In-Reply-To: <CAGb0YipfnpRt0PuaLRtJcF7YhNgb5zj1hCNC_Pu+Ct27NTk1vw@mail.gmail.com>
References: <CAGb0YipfnpRt0PuaLRtJcF7YhNgb5zj1hCNC_Pu+Ct27NTk1vw@mail.gmail.com>
Message-ID: <CANYzqZ6UWWSUPka=BvFGbHBEjp2NDt37TPSrwM2CgZvHs8MUHw@mail.gmail.com>

Hi JP,

Are you running bro from the command as part of your development cycle?
 i.e. you make changes, run a test pcap through bro via bro -r
mytestpcap.pcap myscript.bro ?

If so you can just use print and fmt() to generate output.

v/r
Scott

Scott Runnels



On Thu, Mar 14, 2013 at 3:43 PM, JP Bourget <jp.bourget at gmail.com> wrote:

> I've tried to figure this out for awhile...
>
> How can I print a metric I'm tracking to screen to confirm it's doing what
> I think it's doing?
>
> Thanks!
>
> --
> JP
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130314/8bea80a1/attachment.html 

From seth at icir.org  Thu Mar 14 13:37:00 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 14 Mar 2013 16:37:00 -0400
Subject: [Bro] printing metrics to screen
In-Reply-To: <CAGb0YipfnpRt0PuaLRtJcF7YhNgb5zj1hCNC_Pu+Ct27NTk1vw@mail.gmail.com>
References: <CAGb0YipfnpRt0PuaLRtJcF7YhNgb5zj1hCNC_Pu+Ct27NTk1vw@mail.gmail.com>
Message-ID: <BE272A0F-9611-4517-B109-4BDC09E61205@icir.org>


On Mar 14, 2013, at 3:43 PM, JP Bourget <jp.bourget at gmail.com> wrote:

> How can I print a metric I'm tracking to screen to confirm it's doing what I think it's doing? 


When you say "metric", are you referring to something you're doing with the Metrics framework?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From jp.bourget at gmail.com  Thu Mar 14 13:53:10 2013
From: jp.bourget at gmail.com (JP Bourget)
Date: Thu, 14 Mar 2013 16:53:10 -0400
Subject: [Bro] printing metrics to screen
In-Reply-To: <BE272A0F-9611-4517-B109-4BDC09E61205@icir.org>
References: <CAGb0YipfnpRt0PuaLRtJcF7YhNgb5zj1hCNC_Pu+Ct27NTk1vw@mail.gmail.com>
	<BE272A0F-9611-4517-B109-4BDC09E61205@icir.org>
Message-ID: <CAGb0Yio_bF12UtTabk_ATuHeWsxEJXu8ioWKFscPRFXDRbhdmg@mail.gmail.com>

Sorry didn't reply all -

Seth - I'm trying to figure out how to print out Metrics values stored with
the Metrics Framework. So if I'm watching something, how can I figure out
the state or values of everything in order to understand that what I think
is happening is happening.

See below also
JP


said differently - how do I access the values of metrics at the end of
running a bro script to see if what I think is happening is happening -
before I rewrite it to assume it will trigger an alarm.


On Thu, Mar 14, 2013 at 4:05 PM, JP Bourget <jp.bourget at gmail.com> wrote:

> I am for now - so I could do:
>
> event bro_done()
>
>         {
>
> print fmt("DNS NX: %s", DNS_NX_Count);
> }
>
>
> I'm trying to understand a.) syntax of printing out a metric (is it just
> like any other variable? or do we have a table or multidimensional array
> that we are keeping track based on source IP)
>
> and B: if a is true -would printing out the metric just show the values
> that trigger the metric? I'm trying to figure out what happens.
>
>

On Thu, Mar 14, 2013 at 4:37 PM, Seth Hall <seth at icir.org> wrote:

>
> On Mar 14, 2013, at 3:43 PM, JP Bourget <jp.bourget at gmail.com> wrote:
>
> > How can I print a metric I'm tracking to screen to confirm it's doing
> what I think it's doing?
>
>
> When you say "metric", are you referring to something you're doing with
> the Metrics framework?
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>


-- 
JP
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130314/862184d7/attachment.html 

From seth at icir.org  Thu Mar 14 14:04:02 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 14 Mar 2013 17:04:02 -0400
Subject: [Bro] printing metrics to screen
In-Reply-To: <CAGb0Yio_bF12UtTabk_ATuHeWsxEJXu8ioWKFscPRFXDRbhdmg@mail.gmail.com>
References: <CAGb0YipfnpRt0PuaLRtJcF7YhNgb5zj1hCNC_Pu+Ct27NTk1vw@mail.gmail.com>
	<BE272A0F-9611-4517-B109-4BDC09E61205@icir.org>
	<CAGb0Yio_bF12UtTabk_ATuHeWsxEJXu8ioWKFscPRFXDRbhdmg@mail.gmail.com>
Message-ID: <19ED640A-6FDD-4F29-97A3-392BDBA900CA@icir.org>


On Mar 14, 2013, at 4:53 PM, JP Bourget <jp.bourget at gmail.com> wrote:

> Seth - I'm trying to figure out how to print out Metrics values stored with the Metrics Framework. So if I'm watching something, how can I figure out the state or values of everything in order to understand that what I think is happening is happening. 

The metrics (soon measurement) framework is undergoing heavy rewrites right now and I'm hoping to get the latest refactor pushed out very soon, I'm working on it a lot right now.  One of the big changes has been to change it into more of a utility intended to help you accomplish goals instead of trying to accomplish everything on it's own.  What's in 2.0 and 2.1 was written to solve very few situations.  What I'm working on now has a much broader range of applicability and capability.

I guess what I'm trying to say is that I think what you're trying to do isn't possible with the version of the metrics framework you're using. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From tritium.cat at gmail.com  Mon Mar 18 17:03:55 2013
From: tritium.cat at gmail.com (Tritium Cat)
Date: Mon, 18 Mar 2013 17:03:55 -0700
Subject: [Bro] Bro programming intro
Message-ID: <CAMPgRd7Q3+RZ_g3JoqWTuEXeFNH9wsXU4Y1qyYYsoCcBGmMuYw@mail.gmail.com>

Hello.

I want to modify the SQL Injection detection in
policy/protocols/http/detect-sqli.bro to include a vector that tracks the
associated http request uids and includes them in an additional log field.
 After getting it working I would like to apply it generally to other
Notices such as SSH Password_Guessing.

How this should be implemented ?  I do not understand how the timing and
garbage collection or expiration of Vals? works.  I do not know what is
possible from the scripting layer versus modifying the base or policy
scripts.

Reading the source and docs helps but I could use some pointers to help
accelerate the process.

Thanks !

--TC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130318/209883d7/attachment.html 

From seth at icir.org  Tue Mar 19 08:13:26 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 19 Mar 2013 11:13:26 -0400
Subject: [Bro] Bro programming intro
In-Reply-To: <CAMPgRd7Q3+RZ_g3JoqWTuEXeFNH9wsXU4Y1qyYYsoCcBGmMuYw@mail.gmail.com>
References: <CAMPgRd7Q3+RZ_g3JoqWTuEXeFNH9wsXU4Y1qyYYsoCcBGmMuYw@mail.gmail.com>
Message-ID: <FA096504-8769-42DC-B7F5-0BAF2026B14E@icir.org>


On Mar 18, 2013, at 8:03 PM, Tritium Cat <tritium.cat at gmail.com> wrote:

> I want to modify the SQL Injection detection in policy/protocols/http/detect-sqli.bro to include a vector that tracks the associated http request uids and includes them in an additional log field.  After getting it working I would like to apply it generally to other Notices such as SSH Password_Guessing.

The upcoming release actually results in this script getting rewritten a bit because of a rewrite of the metrics (now measurement) framework.  The new version actually keeps samples of the requests.  It will be relatively easy to write your own script that tracks uid's instead of urls but the benefit to sampling the urls is that if you have Bro send you email for the notice it will add those sample urls to the email (it's been very convenient for determining if something is a false positive without even searching logs).

Otherwise, with the metrics framework in 2.1 there isn't a good way to do it.

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From rjenkins at rmjconsulting.net  Tue Mar 19 08:29:55 2013
From: rjenkins at rmjconsulting.net (Ron Jenkins)
Date: Tue, 19 Mar 2013 15:29:55 +0000
Subject: [Bro] Bro programming intro
In-Reply-To: <FA096504-8769-42DC-B7F5-0BAF2026B14E@icir.org>
References: <CAMPgRd7Q3+RZ_g3JoqWTuEXeFNH9wsXU4Y1qyYYsoCcBGmMuYw@mail.gmail.com>
	<FA096504-8769-42DC-B7F5-0BAF2026B14E@icir.org>
Message-ID: <B6D41C594B5D2E44BE282C07CD71CEE4162BB83C@rmjcs-fs001>

When are you all planning the next release version?


Thanks!


Ron Jenkins (SnortCP, VCP (3/4), MCNE, CNE6, MCP,CCNA)
RMJ Consulting, LLC. "Bringing Companies and Solutions Together"
Makers of Active Response System(ARS) & Log Siphon
Owner / Senior Architect
Physical Address
11715 Bricksome Ave STE B-7
Baton Rouge, LA 70816
Mail Address
7575 Jefferson Hwy #103
Baton Rouge, LA 70806
Toll: 855-448-5214
Direct. 225-448-5214
Fax. 225-448-5324
Cell. 225-931-1632
Email. rjenkins at rmjconsulting.net
Web. http://www.rmjconsulting.net
ARS. http://www.rmjars.com
Log Siphon. http://www.logsiphon.com
Linkedin. http://www.linkedin.com/profile/view?id=28564151&trk=tab_pro


-----Original Message-----
From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Seth Hall
Sent: Tuesday, March 19, 2013 10:13 AM
To: Tritium Cat
Cc: bro at bro-ids.org
Subject: Re: [Bro] Bro programming intro


On Mar 18, 2013, at 8:03 PM, Tritium Cat <tritium.cat at gmail.com> wrote:

> I want to modify the SQL Injection detection in policy/protocols/http/detect-sqli.bro to include a vector that tracks the associated http request uids and includes them in an additional log field.  After getting it working I would like to apply it generally to other Notices such as SSH Password_Guessing.

The upcoming release actually results in this script getting rewritten a bit because of a rewrite of the metrics (now measurement) framework.  The new version actually keeps samples of the requests.  It will be relatively easy to write your own script that tracks uid's instead of urls but the benefit to sampling the urls is that if you have Bro send you email for the notice it will add those sample urls to the email (it's been very convenient for determining if something is a false positive without even searching logs).

Otherwise, with the metrics framework in 2.1 there isn't a good way to do it.

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at icir.org  Tue Mar 19 08:34:16 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 19 Mar 2013 11:34:16 -0400
Subject: [Bro] Bro programming intro
In-Reply-To: <B6D41C594B5D2E44BE282C07CD71CEE4162BB83C@rmjcs-fs001>
References: <CAMPgRd7Q3+RZ_g3JoqWTuEXeFNH9wsXU4Y1qyYYsoCcBGmMuYw@mail.gmail.com>
	<FA096504-8769-42DC-B7F5-0BAF2026B14E@icir.org>
	<B6D41C594B5D2E44BE282C07CD71CEE4162BB83C@rmjcs-fs001>
Message-ID: <B5A149B4-56C2-4386-BD99-E5494DAD8D45@icir.org>


On Mar 19, 2013, at 11:29 AM, Ron Jenkins <rjenkins at rmjconsulting.net> wrote:

> When are you all planning the next release version?

I hate answering this way, but when it's ready. :)

We have several things we're trying to finish up now.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From rjenkins at rmjconsulting.net  Tue Mar 19 08:34:41 2013
From: rjenkins at rmjconsulting.net (Ron Jenkins)
Date: Tue, 19 Mar 2013 15:34:41 +0000
Subject: [Bro] Bro programming intro
In-Reply-To: <B5A149B4-56C2-4386-BD99-E5494DAD8D45@icir.org>
References: <CAMPgRd7Q3+RZ_g3JoqWTuEXeFNH9wsXU4Y1qyYYsoCcBGmMuYw@mail.gmail.com>
	<FA096504-8769-42DC-B7F5-0BAF2026B14E@icir.org>
	<B6D41C594B5D2E44BE282C07CD71CEE4162BB83C@rmjcs-fs001>
	<B5A149B4-56C2-4386-BD99-E5494DAD8D45@icir.org>
Message-ID: <B6D41C594B5D2E44BE282C07CD71CEE4162BB950@rmjcs-fs001>

Fully understand.


Thanks!

-----Original Message-----
From: Seth Hall [mailto:seth at icir.org] 
Sent: Tuesday, March 19, 2013 10:34 AM
To: Ron Jenkins
Cc: 'Seth Hall'; Tritium Cat; bro at bro-ids.org
Subject: Re: [Bro] Bro programming intro


On Mar 19, 2013, at 11:29 AM, Ron Jenkins <rjenkins at rmjconsulting.net> wrote:

> When are you all planning the next release version?

I hate answering this way, but when it's ready. :)

We have several things we're trying to finish up now.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From tritium.cat at gmail.com  Tue Mar 19 08:46:03 2013
From: tritium.cat at gmail.com (Tritium Cat)
Date: Tue, 19 Mar 2013 08:46:03 -0700
Subject: [Bro] Bro programming intro
In-Reply-To: <FA096504-8769-42DC-B7F5-0BAF2026B14E@icir.org>
References: <CAMPgRd7Q3+RZ_g3JoqWTuEXeFNH9wsXU4Y1qyYYsoCcBGmMuYw@mail.gmail.com>
	<FA096504-8769-42DC-B7F5-0BAF2026B14E@icir.org>
Message-ID: <CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>

Ok. I am skeptical of how much emphasis is placed on doing things within
BroIDS.  Simply buffering uids per Notice? seems much easier and less
resource intensive than storing additional? samples.  Where is the limit
with tracking too much state or using too many cycles within the "IDS" ?  I
am weary of inadvertently creating DoS conditions with a philosophy that
may encompass every script I write in Bro.

I am still interested in a list of key papers on the internals if anyone
has a few.



On Tue, Mar 19, 2013 at 8:13 AM, Seth Hall <seth at icir.org> wrote:

>
> On Mar 18, 2013, at 8:03 PM, Tritium Cat <tritium.cat at gmail.com> wrote:
>
> > I want to modify the SQL Injection detection in
> policy/protocols/http/detect-sqli.bro to include a vector that tracks the
> associated http request uids and includes them in an additional log field.
>  After getting it working I would like to apply it generally to other
> Notices such as SSH Password_Guessing.
>
> The upcoming release actually results in this script getting rewritten a
> bit because of a rewrite of the metrics (now measurement) framework.  The
> new version actually keeps samples of the requests.  It will be relatively
> easy to write your own script that tracks uid's instead of urls but the
> benefit to sampling the urls is that if you have Bro send you email for the
> notice it will add those sample urls to the email (it's been very
> convenient for determining if something is a false positive without even
> searching logs).
>
> Otherwise, with the metrics framework in 2.1 there isn't a good way to do
> it.
>
>  .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130319/d007f13e/attachment.html 

From vladg at cmu.edu  Tue Mar 19 09:12:38 2013
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Tue, 19 Mar 2013 16:12:38 +0000
Subject: [Bro] Bro programming intro
In-Reply-To: <32694_1363707976_r2JFkBLO019088_CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
References: <CAMPgRd7Q3+RZ_g3JoqWTuEXeFNH9wsXU4Y1qyYYsoCcBGmMuYw@mail.gmail.com>
	<FA096504-8769-42DC-B7F5-0BAF2026B14E@icir.org>,
	<32694_1363707976_r2JFkBLO019088_CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
Message-ID: <1202BE242E080642B0CD0AD0A03E8552AC1FB2@PGH-MSGMB-03.andrew.ad.cmu.edu>


________________________________________
From:Tritium Cat [tritium.cat at gmail.com]

> Simply buffering uids per Notice? seems much easier and less resource intensive than storing additional? samples.

It's also much less useful. If I get an e-mail with a list of UIDs, I have to go query my http log before I can determine what action to take. If I get samples, I can make that decision immediately.

I don't understand how tracking UIDs would be less resource intensive. Many SQL scanners I see attempt thousands of requests over separate UIDs. The way samples work is that you specify a number of samples per source IP. I believe the default is 5. I'd much rather have Bro maintain 5 samples per source instead of thousands of UIDs.

> Where is the limit with tracking too much state or using too many cycles within the "IDS" ?

One side note: Bro hasn't been labelled as an IDS for a while. Network Security Monitor strikes closer to what Bro has become.

>  I am weary of inadvertently creating DoS conditions with a philosophy that may encompass every script I write in Bro.

A fair concern, and one I think I addressed above. I would note that I haven't had any such problems with the scripts that ship with Bro.

> I am still interested in a list of key papers on the internals if anyone has a few.

http://bro.org/research

 --Vlad





From tritium.cat at gmail.com  Tue Mar 19 09:41:24 2013
From: tritium.cat at gmail.com (Tritium Cat)
Date: Tue, 19 Mar 2013 09:41:24 -0700
Subject: [Bro] Bro programming intro
In-Reply-To: <1202BE242E080642B0CD0AD0A03E8552AC1FB2@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <CAMPgRd7Q3+RZ_g3JoqWTuEXeFNH9wsXU4Y1qyYYsoCcBGmMuYw@mail.gmail.com>
	<FA096504-8769-42DC-B7F5-0BAF2026B14E@icir.org>
	<32694_1363707976_r2JFkBLO019088_CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E8552AC1FB2@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <CAMPgRd58+qGnO6kZC-5uNT3zAy583N+iWms4vrBgj+Zis5n8_g@mail.gmail.com>

The ability to work with items outside of BroNSM to me is useful and easier
than rewriting a BroNSM script and restarting a cluster when I want to look
at something differently or trim logs.  Searching for items and *guessing*
which requests are related is more time consuming.  Long term I can see
tweaking a Bro script to perform better.  I am very selective with using
email as an alert mechanism.

Using samples makes sense, as does uids, samples involve content and sound
larger than a simple int32?, but limiting those is fine as well, just as
you would the UIDs.  How do you plan to implement the sampling ?  By time
or by unique requests ?  Can an attack tool run a number of SQL injection
attempts and end the last 5 with something benign ?  I'd rather analyze the
specifics outside of BroNSM before going back and tweaking BroNSM.

Thanks for the /research link.



On Tue, Mar 19, 2013 at 9:12 AM, Vlad Grigorescu <vladg at cmu.edu> wrote:

>
> ________________________________________
> From:Tritium Cat [tritium.cat at gmail.com]
>
> > Simply buffering uids per Notice? seems much easier and less resource
> intensive than storing additional? samples.
>
> It's also much less useful. If I get an e-mail with a list of UIDs, I have
> to go query my http log before I can determine what action to take. If I
> get samples, I can make that decision immediately.
>
> I don't understand how tracking UIDs would be less resource intensive.
> Many SQL scanners I see attempt thousands of requests over separate UIDs.
> The way samples work is that you specify a number of samples per source IP.
> I believe the default is 5. I'd much rather have Bro maintain 5 samples per
> source instead of thousands of UIDs.
>
> > Where is the limit with tracking too much state or using too many cycles
> within the "IDS" ?
>
> One side note: Bro hasn't been labelled as an IDS for a while. Network
> Security Monitor strikes closer to what Bro has become.
>
> >  I am weary of inadvertently creating DoS conditions with a philosophy
> that may encompass every script I write in Bro.
>
> A fair concern, and one I think I addressed above. I would note that I
> haven't had any such problems with the scripts that ship with Bro.
>
> > I am still interested in a list of key papers on the internals if anyone
> has a few.
>
> http://bro.org/research
>
>  --Vlad
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130319/a3eb76d0/attachment.html 

From tritium.cat at gmail.com  Tue Mar 19 09:50:34 2013
From: tritium.cat at gmail.com (Tritium Cat)
Date: Tue, 19 Mar 2013 09:50:34 -0700
Subject: [Bro] Bro programming intro
In-Reply-To: <CAMPgRd58+qGnO6kZC-5uNT3zAy583N+iWms4vrBgj+Zis5n8_g@mail.gmail.com>
References: <CAMPgRd7Q3+RZ_g3JoqWTuEXeFNH9wsXU4Y1qyYYsoCcBGmMuYw@mail.gmail.com>
	<FA096504-8769-42DC-B7F5-0BAF2026B14E@icir.org>
	<32694_1363707976_r2JFkBLO019088_CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E8552AC1FB2@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<CAMPgRd58+qGnO6kZC-5uNT3zAy583N+iWms4vrBgj+Zis5n8_g@mail.gmail.com>
Message-ID: <CAMPgRd68R-SHY-cm5WZKWfxzCSs0-2ey9BV4HnLev1kAKM-jLA@mail.gmail.com>

I have to point out what I interpret as not answering the question about
how to buffer data across time.  Not sure how to interpret that other than
"go figure it out yourself" or wait for $next_release where it will exist
in an altered form :p   Maybe what I'm doing is stupid but maybe it will be
clever.



On Tue, Mar 19, 2013 at 9:41 AM, Tritium Cat <tritium.cat at gmail.com> wrote:

>
> The ability to work with items outside of BroNSM to me is useful and
> easier than rewriting a BroNSM script and restarting a cluster when I want
> to look at something differently or trim logs.  Searching for items and
> *guessing* which requests are related is more time consuming.  Long term I
> can see tweaking a Bro script to perform better.  I am very selective with
> using email as an alert mechanism.
>
> Using samples makes sense, as does uids, samples involve content and sound
> larger than a simple int32?, but limiting those is fine as well, just as
> you would the UIDs.  How do you plan to implement the sampling ?  By time
> or by unique requests ?  Can an attack tool run a number of SQL injection
> attempts and end the last 5 with something benign ?  I'd rather analyze the
> specifics outside of BroNSM before going back and tweaking BroNSM.
>
> Thanks for the /research link.
>
>
>
> On Tue, Mar 19, 2013 at 9:12 AM, Vlad Grigorescu <vladg at cmu.edu> wrote:
>
>>
>> ________________________________________
>> From:Tritium Cat [tritium.cat at gmail.com]
>>
>> > Simply buffering uids per Notice? seems much easier and less resource
>> intensive than storing additional? samples.
>>
>> It's also much less useful. If I get an e-mail with a list of UIDs, I
>> have to go query my http log before I can determine what action to take. If
>> I get samples, I can make that decision immediately.
>>
>> I don't understand how tracking UIDs would be less resource intensive.
>> Many SQL scanners I see attempt thousands of requests over separate UIDs.
>> The way samples work is that you specify a number of samples per source IP.
>> I believe the default is 5. I'd much rather have Bro maintain 5 samples per
>> source instead of thousands of UIDs.
>>
>> > Where is the limit with tracking too much state or using too many
>> cycles within the "IDS" ?
>>
>> One side note: Bro hasn't been labelled as an IDS for a while. Network
>> Security Monitor strikes closer to what Bro has become.
>>
>> >  I am weary of inadvertently creating DoS conditions with a philosophy
>> that may encompass every script I write in Bro.
>>
>> A fair concern, and one I think I addressed above. I would note that I
>> haven't had any such problems with the scripts that ship with Bro.
>>
>> > I am still interested in a list of key papers on the internals if
>> anyone has a few.
>>
>> http://bro.org/research
>>
>>  --Vlad
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130319/b48d654c/attachment.html 

From gc355804 at ohio.edu  Tue Mar 19 09:51:25 2013
From: gc355804 at ohio.edu (Clark, Gilbert)
Date: Tue, 19 Mar 2013 12:51:25 -0400
Subject: [Bro] Bro programming intro
In-Reply-To: <CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
Message-ID: <CD6E015D.269A4%gc355804@ohio.edu>

Hi:

One paper I can think of off-hand that appeared in RAID a few years back and may be relevant here:

http://www.icir.org/vern/papers/autoconf-raid08.pdf

On principle, there isn't a hard and fast rule for exactly how much state you can allocate or how many cycles you can spend processing individual packets: it's really going to depend on load, and will likely need to be experimentally determined (and constantly tuned).  One way to do this is to record snippets of the traffic you normally see at the border and run bro against that same set of traffic over and over (bro's ?pseudo-realtime option can help get a realistic sample of what happens if you want to run this locally, or you can alternatively replay traces onto a local link to possibly get some more realistic results) while modifying the scripts bro has loaded to see what happens.  This kind of testing can help you identify the limits of what a specific bro configuration can handle in your environment.

Also, I'd like to point out that some folks simply use bro for offline trace analysis and the like; while bro does well as a real-time tool, it doesn't necessarily have to be.

"I am skeptical of how much emphasis is placed on doing things within BroIDS."

Relevant cliche: Premature optimization is the root of much evil.

The emphasis, I think, isn't placed on doing things within bro.  Instead, I believe the emphasis is placed on doing things that work for the folks who are deploying bro into their environments.  Most of the discussion here focuses on the scripting layer here because, in my humble opinion, that is the most straightforward interface to bro's event engine, and it's fast enough to do what folks need it to.  In the event it isn't, there's always the cluster model ? there's at least one piece of pretty cool hardware I know of that rewrites destination MAC addresses and allows you to load-balance across a cluster pretty effectively, and PF_RING can also even let you cluster on a local box.  Often, investing in clustering winds up being cheaper (in both time and money) than trying to throw more experts at the problem to squeeze that last 15 Mbps out of a single node.

Then again, trying to squeeze more out of a single node is always a fun programming challenge to solve :)

Normal disclaimer applies: just my $0.02, I'm not an expert, etc.

--Gilbert Clark

From: Tritium Cat <tritium.cat at gmail.com<mailto:tritium.cat at gmail.com>>
Date: Tue, 19 Mar 2013 11:46:03 -0400
To: Seth Hall <seth at icir.org<mailto:seth at icir.org>>
Cc: "bro at bro-ids.org<mailto:bro at bro-ids.org>" <bro at bro-ids.org<mailto:bro at bro-ids.org>>
Subject: Re: [Bro] Bro programming intro

Ok. I am skeptical of how much emphasis is placed on doing things within BroIDS.  Simply buffering uids per Notice? seems much easier and less resource intensive than storing additional? samples.  Where is the limit with tracking too much state or using too many cycles within the "IDS" ?  I am weary of inadvertently creating DoS conditions with a philosophy that may encompass every script I write in Bro.

I am still interested in a list of key papers on the internals if anyone has a few.



On Tue, Mar 19, 2013 at 8:13 AM, Seth Hall <seth at icir.org<mailto:seth at icir.org>> wrote:

On Mar 18, 2013, at 8:03 PM, Tritium Cat <tritium.cat at gmail.com<mailto:tritium.cat at gmail.com>> wrote:

> I want to modify the SQL Injection detection in policy/protocols/http/detect-sqli.bro to include a vector that tracks the associated http request uids and includes them in an additional log field.  After getting it working I would like to apply it generally to other Notices such as SSH Password_Guessing.

The upcoming release actually results in this script getting rewritten a bit because of a rewrite of the metrics (now measurement) framework.  The new version actually keeps samples of the requests.  It will be relatively easy to write your own script that tracks uid's instead of urls but the benefit to sampling the urls is that if you have Bro send you email for the notice it will add those sample urls to the email (it's been very convenient for determining if something is a false positive without even searching logs).

Otherwise, with the metrics framework in 2.1 there isn't a good way to do it.

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130319/ba9711d3/attachment.html 

From seth at icir.org  Tue Mar 19 10:34:05 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 19 Mar 2013 13:34:05 -0400
Subject: [Bro] Bro programming intro
In-Reply-To: <CAMPgRd58+qGnO6kZC-5uNT3zAy583N+iWms4vrBgj+Zis5n8_g@mail.gmail.com>
References: <CAMPgRd7Q3+RZ_g3JoqWTuEXeFNH9wsXU4Y1qyYYsoCcBGmMuYw@mail.gmail.com>
	<FA096504-8769-42DC-B7F5-0BAF2026B14E@icir.org>
	<32694_1363707976_r2JFkBLO019088_CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E8552AC1FB2@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<CAMPgRd58+qGnO6kZC-5uNT3zAy583N+iWms4vrBgj+Zis5n8_g@mail.gmail.com>
Message-ID: <F61F3AFC-10EE-4F78-B5E6-41BEC4C0C069@icir.org>


On Mar 19, 2013, at 12:41 PM, Tritium Cat <tritium.cat at gmail.com> wrote:

> The ability to work with items outside of BroNSM to me is useful and easier than rewriting a BroNSM script and restarting a cluster when I want to look at something differently or trim logs.

We're even moving away from "NSM" now.  
	http://blog.bro.org/2013/03/broorg-new-home-for-bro.html

> I am very selective with using email as an alert mechanism.

I fully believe this will expand over time.  Email is just the obvious way that we support right now.  Is there some other specific tool you would like to see Bro integrated with?

> Using samples makes sense, as does uids, samples involve content and sound larger than a simple int32?, but limiting those is fine as well, just as you would the UIDs.  How do you plan to implement the sampling ?  By time or by unique requests ?  Can an attack tool run a number of SQL injection attempts and end the last 5 with something benign ?  I'd rather analyze the specifics outside of BroNSM before going back and tweaking BroNSM.

If you run on a cluster it would become very hard for an attacker to end up sending just 5 at the end that would be forwarded to the analyst.  Samples are collected on each worker and then interleaved and size limited again when the measurement results are merged at the end.  I'm sure there are ways an attacker could mess with analysts still, but it's not as obvious as just sending a few benign requests at some specific period.

Regardless, this is just Bro scripts that are tracking the content and they can typically be modified fairly easily.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From keqhe at cs.wisc.edu  Tue Mar 19 10:49:42 2013
From: keqhe at cs.wisc.edu (keqhe at cs.wisc.edu)
Date: Tue, 19 Mar 2013 12:49:42 -0500
Subject: [Bro] About Bro Processing Speed
Message-ID: <cb0017022a01f8aff25e969a4a499943.squirrel@webmail.cs.wisc.edu>


Hello,

Does any one know Bro's processing speed? i.e., can it support monitoring
1 Gbps or 10Gbps link?

Thanks!




From hhoffman at ip-solutions.net  Tue Mar 19 11:06:59 2013
From: hhoffman at ip-solutions.net (Harry Hoffman)
Date: Tue, 19 Mar 2013 14:06:59 -0400
Subject: [Bro] Bro programming intro
In-Reply-To: <F61F3AFC-10EE-4F78-B5E6-41BEC4C0C069@icir.org>
References: <CAMPgRd7Q3+RZ_g3JoqWTuEXeFNH9wsXU4Y1qyYYsoCcBGmMuYw@mail.gmail.com>
	<FA096504-8769-42DC-B7F5-0BAF2026B14E@icir.org>
	<32694_1363707976_r2JFkBLO019088_CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E8552AC1FB2@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<CAMPgRd58+qGnO6kZC-5uNT3zAy583N+iWms4vrBgj+Zis5n8_g@mail.gmail.com>
	<F61F3AFC-10EE-4F78-B5E6-41BEC4C0C069@icir.org>
Message-ID: <5148A943.9060209@ip-solutions.net>

Wow!!! What the heck did it cost to get that domain name???

Congrats. Educause SPC is coming up and I'm sure Bro is going to be a
hot topic of conversation :-)

Cheers,
Harry

On 03/19/2013 01:34 PM, Seth Hall wrote:
> 
> On Mar 19, 2013, at 12:41 PM, Tritium Cat <tritium.cat at gmail.com> wrote:
> 
>> The ability to work with items outside of BroNSM to me is useful and easier than rewriting a BroNSM script and restarting a cluster when I want to look at something differently or trim logs.
> 
> We're even moving away from "NSM" now.  
> 	http://blog.bro.org/2013/03/broorg-new-home-for-bro.html
> 
>> I am very selective with using email as an alert mechanism.
> 
> I fully believe this will expand over time.  Email is just the obvious way that we support right now.  Is there some other specific tool you would like to see Bro integrated with?
> 
>> Using samples makes sense, as does uids, samples involve content and sound larger than a simple int32?, but limiting those is fine as well, just as you would the UIDs.  How do you plan to implement the sampling ?  By time or by unique requests ?  Can an attack tool run a number of SQL injection attempts and end the last 5 with something benign ?  I'd rather analyze the specifics outside of BroNSM before going back and tweaking BroNSM.
> 
> If you run on a cluster it would become very hard for an attacker to end up sending just 5 at the end that would be forwarded to the analyst.  Samples are collected on each worker and then interleaved and size limited again when the measurement results are merged at the end.  I'm sure there are ways an attacker could mess with analysts still, but it's not as obvious as just sending a few benign requests at some specific period.
> 
> Regardless, this is just Bro scripts that are tracking the content and they can typically be modified fairly easily.
> 
>   .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 


From tritium.cat at gmail.com  Tue Mar 19 11:18:45 2013
From: tritium.cat at gmail.com (Tritium Cat)
Date: Tue, 19 Mar 2013 11:18:45 -0700
Subject: [Bro] Bro programming intro
In-Reply-To: <CD6E015D.269A4%gc355804@ohio.edu>
References: <CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
	<CD6E015D.269A4%gc355804@ohio.edu>
Message-ID: <CAMPgRd7HkJKrBpk-hSnT1VbvPCSVmh7Tap1kKCSppGqiiOba7A@mail.gmail.com>

Thanks for your comments, good to reflect on over coffee.  FWIW I am
running the cluster model with plenty of RAM and CPU to spare with a near
default config.

I do not believe what I am trying to learn and implement is premature
optimization but thanks for the reminder, I've made that mistake a few
times.


Another thing I would like to do is tag every orig_h and resp_h with
additional identifiers relative to the prefix, sorta like BGP ASNs.  I
usually use a Patricia-Trie for this.  Is there a special data type and BiF
I should consider ?

>From the documentation on extending logging [1] it seems that is a bit
beyond the scripting layer.  I read about the input framework [2] and it
seems it might work for this application but I'm not sure if this is best.
 What do you suggest ?  I would like all logs that have an orig_h and/or
resp_p to include the tags.


[1] http://www.bro.org/documentation/logging.html#extending
[2] http://www.bro.org/documentation/input.html

Thanks !

--TC



On Tue, Mar 19, 2013 at 9:51 AM, Clark, Gilbert <gc355804 at ohio.edu> wrote:

> Hi:
>
> One paper I can think of off-hand that appeared in RAID a few years back
> and may be relevant here:
>
> http://www.icir.org/vern/papers/autoconf-raid08.pdf
>
> On principle, there isn't a hard and fast rule for exactly how much state
> you can allocate or how many cycles you can spend processing individual
> packets: it's really going to depend on load, and will likely need to be
> experimentally determined (and constantly tuned).  One way to do this is to
> record snippets of the traffic you normally see at the border and run bro
> against that same set of traffic over and over (bro's ?pseudo-realtime
> option can help get a realistic sample of what happens if you want to run
> this locally, or you can alternatively replay traces onto a local link to
> possibly get some more realistic results) while modifying the scripts bro
> has loaded to see what happens.  This kind of testing can help you identify
> the limits of what a specific bro configuration can handle in your
> environment.
>
> Also, I'd like to point out that some folks simply use bro for offline
> trace analysis and the like; while bro does well as a real-time tool, it
> doesn't necessarily have to be.
>
> "I am skeptical of how much emphasis is placed on doing things within
> BroIDS."
>
> Relevant cliche: Premature optimization is the root of much evil.
>
> The emphasis, I think, isn't placed on doing things within bro.  Instead,
> I believe the emphasis is placed on doing things that work for the folks
> who are deploying bro into their environments.  Most of the discussion here
> focuses on the scripting layer here because, in my humble opinion, that is
> the most straightforward interface to bro's event engine, and it's fast
> enough to do what folks need it to.  In the event it isn't, there's
> always the cluster model ? there's at least one piece of pretty cool
> hardware I know of that rewrites destination MAC addresses and allows you
> to load-balance across a cluster pretty effectively, and PF_RING can also
> even let you cluster on a local box.  Often, investing in clustering winds
> up being cheaper (in both time and money) than trying to throw more experts
> at the problem to squeeze that last 15 Mbps out of a single node.
>
> Then again, trying to squeeze more out of a single node is always a fun
> programming challenge to solve :)
>
> Normal disclaimer applies: just my $0.02, I'm not an expert, etc.
>
> --Gilbert Clark
>
> From: Tritium Cat <tritium.cat at gmail.com>
> Date: Tue, 19 Mar 2013 11:46:03 -0400
> To: Seth Hall <seth at icir.org>
> Cc: "bro at bro-ids.org" <bro at bro-ids.org>
>
> Subject: Re: [Bro] Bro programming intro
>
> Ok. I am skeptical of how much emphasis is placed on doing things within
> BroIDS.  Simply buffering uids per Notice? seems much easier and less
> resource intensive than storing additional? samples.  Where is the limit
> with tracking too much state or using too many cycles within the "IDS" ?  I
> am weary of inadvertently creating DoS conditions with a philosophy that
> may encompass every script I write in Bro.
>
> I am still interested in a list of key papers on the internals if anyone
> has a few.
>
>
>
> On Tue, Mar 19, 2013 at 8:13 AM, Seth Hall <seth at icir.org> wrote:
>
>>
>> On Mar 18, 2013, at 8:03 PM, Tritium Cat <tritium.cat at gmail.com> wrote:
>>
>> > I want to modify the SQL Injection detection in
>> policy/protocols/http/detect-sqli.bro to include a vector that tracks the
>> associated http request uids and includes them in an additional log field.
>>  After getting it working I would like to apply it generally to other
>> Notices such as SSH Password_Guessing.
>>
>> The upcoming release actually results in this script getting rewritten a
>> bit because of a rewrite of the metrics (now measurement) framework.  The
>> new version actually keeps samples of the requests.  It will be relatively
>> easy to write your own script that tracks uid's instead of urls but the
>> benefit to sampling the urls is that if you have Bro send you email for the
>> notice it will add those sample urls to the email (it's been very
>> convenient for determining if something is a false positive without even
>> searching logs).
>>
>> Otherwise, with the metrics framework in 2.1 there isn't a good way to do
>> it.
>>
>>  .Seth
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130319/75f7ee49/attachment.html 

From mike.patterson at uwaterloo.ca  Tue Mar 19 11:20:08 2013
From: mike.patterson at uwaterloo.ca (Mike Patterson)
Date: Tue, 19 Mar 2013 14:20:08 -0400
Subject: [Bro] About Bro Processing Speed
In-Reply-To: <cb0017022a01f8aff25e969a4a499943.squirrel@webmail.cs.wisc.edu>
References: <cb0017022a01f8aff25e969a4a499943.squirrel@webmail.cs.wisc.edu>
Message-ID: <11E31E35-134B-482C-AD46-C4AD2FA1BA64@uwaterloo.ca>

"It depends."

On - hardware, mostly, and configuration.

I keep meaning to write this up, but on *my* configuration:
* 16 cores of model name	: Intel(R) Xeon(R) CPU           X5677  @ 3.47GHz
* 72GB of RAM
* Endace DAG (9.2)
* some config magic by Seth, which I'd be happy to share.

6 workers keep up with ~2.5-3Gbps peaks, no problem. 

On lesser hardware, your mileage will definitely vary. The secret sauce appears to be the DAG. Hot CPUs doesn't hurt either. :) You'll probably find that an Endace will cost as much as the server you'd put it in. I think it's worthwhile, but your budget is yours.

It doesn't actually consume all of the above resources - I'm running other things on the box too - but bro itself consumes ~4.5GB resident per worker, and can be counted on to pin most of its allocated cores at peak loads.

Mike

On 2013-03-19, at 1:49 PM, keqhe at cs.wisc.edu wrote:

> 
> Hello,
> 
> Does any one know Bro's processing speed? i.e., can it support monitoring
> 1 Gbps or 10Gbps link?
> 
> Thanks!
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From vladg at cmu.edu  Tue Mar 19 11:35:10 2013
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Tue, 19 Mar 2013 18:35:10 +0000
Subject: [Bro] About Bro Processing Speed
In-Reply-To: <13824_1363717681_r2JIRxO0014643_11E31E35-134B-482C-AD46-C4AD2FA1BA64@uwaterloo.ca>
References: <cb0017022a01f8aff25e969a4a499943.squirrel@webmail.cs.wisc.edu>
	<13824_1363717681_r2JIRxO0014643_11E31E35-134B-482C-AD46-C4AD2FA1BA64@uwaterloo.ca>
Message-ID: <1202BE242E080642B0CD0AD0A03E8552AC24BC@PGH-MSGMB-03.andrew.ad.cmu.edu>

Just to throw another data point out there:

> * 16 physical cores of model name	: Intel(R) Xeon(R) CPU E5-2680 @ 2.70 GHz
> * 96GB of RAM
> * Myricom NIC

28 workers (I have Hyperthreading turned on) keep up with a 6-7 Gbps average, and I've seen them do fine with short peaks of 9 Gbps or so. The Myricom cards definitely won't break the bank: card + SR optics + perpetual license is $895.

  --Vlad

On Mar 19, 2013, at 2:20 PM, Mike Patterson <mike.patterson at uwaterloo.ca>
 wrote:

> "It depends."
> 
> On - hardware, mostly, and configuration.
> 
> I keep meaning to write this up, but on *my* configuration:
> * 16 cores of model name	: Intel(R) Xeon(R) CPU           X5677  @ 3.47GHz
> * 72GB of RAM
> * Endace DAG (9.2)
> * some config magic by Seth, which I'd be happy to share.
> 
> 6 workers keep up with ~2.5-3Gbps peaks, no problem. 
> 
> On lesser hardware, your mileage will definitely vary. The secret sauce appears to be the DAG. Hot CPUs doesn't hurt either. :) You'll probably find that an Endace will cost as much as the server you'd put it in. I think it's worthwhile, but your budget is yours.
> 
> It doesn't actually consume all of the above resources - I'm running other things on the box too - but bro itself consumes ~4.5GB resident per worker, and can be counted on to pin most of its allocated cores at peak loads.
> 
> Mike
> 
> On 2013-03-19, at 1:49 PM, keqhe at cs.wisc.edu wrote:
> 
>> 
>> Hello,
>> 
>> Does any one know Bro's processing speed? i.e., can it support monitoring
>> 1 Gbps or 10Gbps link?
>> 
>> Thanks!
>> 
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From jsiwek at illinois.edu  Tue Mar 19 11:56:18 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Tue, 19 Mar 2013 18:56:18 +0000
Subject: [Bro] Bro programming intro
In-Reply-To: <CAMPgRd7HkJKrBpk-hSnT1VbvPCSVmh7Tap1kKCSppGqiiOba7A@mail.gmail.com>
References: <CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
	<CD6E015D.269A4%gc355804@ohio.edu>
	<CAMPgRd7HkJKrBpk-hSnT1VbvPCSVmh7Tap1kKCSppGqiiOba7A@mail.gmail.com>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D116F47B5E@CITESMBX2.ad.uillinois.edu>


> Another thing I would like to do is tag every orig_h and resp_h with additional identifiers relative to the prefix, sorta like BGP ASNs.  I usually use a Patricia-Trie for this.  Is there a special data type and BiF I should consider ? 

When indexed by the `subnet` type, the `set` and `table` types should be using a Patricia-Trie internally.  So you might be able to use something like a `table[subnet] of MyTag` for mapping orig_h/resp_h to however you want to define the `MyTag` type (probably an enum would work).

> From the documentation on extending logging [1] it seems that is a bit beyond the scripting layer.  I read about the input framework [2] and it seems it might work for this application but I'm not sure if this is best.  What do you suggest ?  I would like all logs that have an orig_h and/or resp_p to include the tags. 

My opinion would be that extending the logging would be easier, but I don't know all the details of how you want to use it.  The way I'm thinking, you'd basically do the same thing as the documentation describes, maybe start with conn.log:

(1) add a field to the Conn::Info record for the orig/resp tags
(2) pick a time at which to lookup the orig_h/resp_h in your tag table and assign them to the fields in a Conn::Info instance.  Handling either the Conn::log_conn or connection_state_remove event to do this are some ideas.

Then you can see if it makes sense to extend other logs in a similar way or whether conn.log is adequate.

    Jon


From seth at icir.org  Tue Mar 19 12:41:48 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 19 Mar 2013 15:41:48 -0400
Subject: [Bro] Bro programming intro
In-Reply-To: <CAMPgRd7HkJKrBpk-hSnT1VbvPCSVmh7Tap1kKCSppGqiiOba7A@mail.gmail.com>
References: <CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
	<CD6E015D.269A4%gc355804@ohio.edu>
	<CAMPgRd7HkJKrBpk-hSnT1VbvPCSVmh7Tap1kKCSppGqiiOba7A@mail.gmail.com>
Message-ID: <AF97AD1E-E3A2-468E-844D-DEB1612D0ACF@icir.org>


On Mar 19, 2013, at 2:18 PM, Tritium Cat <tritium.cat at gmail.com> wrote:

> Another thing I would like to do is tag every orig_h and resp_h with additional identifiers relative to the prefix, sorta like BGP ASNs.  I usually use a Patricia-Trie for this.  Is there a special data type and BiF I should consider ? 

You got the right section in the logging framework docs.  I'll give an example if you want to add ASN like you mentioned as an example...

redef record Conn::Log += {
	orig_asn: count &log &optional;
	resp_asn: count &log &optional;
};

event connection_established(c: connection)
	{
	c$conn$orig_asn = lookup_asn(c$id$orig_h);
 	c$conn$resp_asn = lookup_asn(c$id$resp_h);
	}

You need to have the MaxMind ASN database in place for the lookup_asn function to work.  Anyway, it's pretty easy. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From seth at icir.org  Tue Mar 19 12:47:32 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 19 Mar 2013 15:47:32 -0400
Subject: [Bro] About Bro Processing Speed
In-Reply-To: <11E31E35-134B-482C-AD46-C4AD2FA1BA64@uwaterloo.ca>
References: <cb0017022a01f8aff25e969a4a499943.squirrel@webmail.cs.wisc.edu>
	<11E31E35-134B-482C-AD46-C4AD2FA1BA64@uwaterloo.ca>
Message-ID: <0349FA9E-3401-4984-8995-26797B09A372@icir.org>


On Mar 19, 2013, at 2:20 PM, Mike Patterson <mike.patterson at uwaterloo.ca> wrote:

> I keep meaning to write this up, but on *my* configuration:
> * 16 cores of model name	: Intel(R) Xeon(R) CPU           X5677  @ 3.47GHz


Oh!  I'm not sure why but I never paid attention to the speed of your cores.  I suspect that has a huge impact.  From what we were looking at it seems like you are capable of handling at least 500Mbps/core.

I'm not sure what the cost differential is between the mid-range processors and what you have in that box, but I'm starting to wonder if the world has been flipped upside down and suddenly the high-end processors are worth the extra expense now (for Bro at least).

As Vlad pointed out, I suspect that the DAG card doesn't really give any performance benefit over the Myricom nics.  Although, both of those give huge benefits over something that isn't skipping the network stack and getting rid of interrupts.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From tritium.cat at gmail.com  Tue Mar 19 13:29:06 2013
From: tritium.cat at gmail.com (Tritium Cat)
Date: Tue, 19 Mar 2013 13:29:06 -0700
Subject: [Bro] Bro programming intro
In-Reply-To: <AF97AD1E-E3A2-468E-844D-DEB1612D0ACF@icir.org>
References: <CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
	<CD6E015D.269A4%gc355804@ohio.edu>
	<CAMPgRd7HkJKrBpk-hSnT1VbvPCSVmh7Tap1kKCSppGqiiOba7A@mail.gmail.com>
	<AF97AD1E-E3A2-468E-844D-DEB1612D0ACF@icir.org>
Message-ID: <CAMPgRd4nK64VF7PL6qdFDd+fv=rL0p5r3gV6+Fwv8NOiLaw6QA@mail.gmail.com>

Thanks for the detail and examples.  Makes more sense when you consider a
single Bro process.



On Tue, Mar 19, 2013 at 12:41 PM, Seth Hall <seth at icir.org> wrote:

>
> On Mar 19, 2013, at 2:18 PM, Tritium Cat <tritium.cat at gmail.com> wrote:
>
> > Another thing I would like to do is tag every orig_h and resp_h with
> additional identifiers relative to the prefix, sorta like BGP ASNs.  I
> usually use a Patricia-Trie for this.  Is there a special data type and BiF
> I should consider ?
>
> You got the right section in the logging framework docs.  I'll give an
> example if you want to add ASN like you mentioned as an example...
>
> redef record Conn::Log += {
>         orig_asn: count &log &optional;
>         resp_asn: count &log &optional;
> };
>
> event connection_established(c: connection)
>         {
>         c$conn$orig_asn = lookup_asn(c$id$orig_h);
>         c$conn$resp_asn = lookup_asn(c$id$resp_h);
>         }
>
> You need to have the MaxMind ASN database in place for the lookup_asn
> function to work.  Anyway, it's pretty easy. :)
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130319/fca4ba02/attachment.html 

From seth at icir.org  Tue Mar 19 13:51:04 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 19 Mar 2013 16:51:04 -0400
Subject: [Bro] Bro programming intro
In-Reply-To: <CAMPgRd4nK64VF7PL6qdFDd+fv=rL0p5r3gV6+Fwv8NOiLaw6QA@mail.gmail.com>
References: <CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
	<CD6E015D.269A4%gc355804@ohio.edu>
	<CAMPgRd7HkJKrBpk-hSnT1VbvPCSVmh7Tap1kKCSppGqiiOba7A@mail.gmail.com>
	<AF97AD1E-E3A2-468E-844D-DEB1612D0ACF@icir.org>
	<CAMPgRd4nK64VF7PL6qdFDd+fv=rL0p5r3gV6+Fwv8NOiLaw6QA@mail.gmail.com>
Message-ID: <0FC4218F-C104-4DD1-B6FE-2B72D6B8A8D8@icir.org>


On Mar 19, 2013, at 4:29 PM, Tritium Cat <tritium.cat at gmail.com> wrote:

> Makes more sense when you consider a single Bro process.

I'm not sure what you mean by this?  Logs don't really have anything to do with multiple processes in most cases.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From tritium.cat at gmail.com  Tue Mar 19 14:23:29 2013
From: tritium.cat at gmail.com (Tritium Cat)
Date: Tue, 19 Mar 2013 14:23:29 -0700
Subject: [Bro] Bro programming intro
In-Reply-To: <0FC4218F-C104-4DD1-B6FE-2B72D6B8A8D8@icir.org>
References: <CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
	<CD6E015D.269A4%gc355804@ohio.edu>
	<CAMPgRd7HkJKrBpk-hSnT1VbvPCSVmh7Tap1kKCSppGqiiOba7A@mail.gmail.com>
	<AF97AD1E-E3A2-468E-844D-DEB1612D0ACF@icir.org>
	<CAMPgRd4nK64VF7PL6qdFDd+fv=rL0p5r3gV6+Fwv8NOiLaw6QA@mail.gmail.com>
	<0FC4218F-C104-4DD1-B6FE-2B72D6B8A8D8@icir.org>
Message-ID: <CAMPgRd5BET9+b_t-qW+rpn=3F_-q0==NPSL0M1NwcaSFTUKL5g@mail.gmail.com>

Don't worry, neither do I.  Good to know !




On Tue, Mar 19, 2013 at 1:51 PM, Seth Hall <seth at icir.org> wrote:

>
> On Mar 19, 2013, at 4:29 PM, Tritium Cat <tritium.cat at gmail.com> wrote:
>
> > Makes more sense when you consider a single Bro process.
>
> I'm not sure what you mean by this?  Logs don't really have anything to do
> with multiple processes in most cases.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130319/bb37080b/attachment.html 

From roncking at gmail.com  Tue Mar 19 17:47:23 2013
From: roncking at gmail.com (Ron King)
Date: Tue, 19 Mar 2013 19:47:23 -0500
Subject: [Bro] sending event log output to a database
Message-ID: <CAN2L1VqthCSvAjfq0udo5Xn48gty0N0zFo8igNLKtPi6S4BFww@mail.gmail.com>

Hi, I want to send event log data directly to a nosql database. Where
in the code should I look in order to add this capability?

Regards,

Ron


From seth at icir.org  Tue Mar 19 18:14:24 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 19 Mar 2013 21:14:24 -0400
Subject: [Bro] sending event log output to a database
In-Reply-To: <CAN2L1VqthCSvAjfq0udo5Xn48gty0N0zFo8igNLKtPi6S4BFww@mail.gmail.com>
References: <CAN2L1VqthCSvAjfq0udo5Xn48gty0N0zFo8igNLKtPi6S4BFww@mail.gmail.com>
Message-ID: <5A151EA6-F105-49BC-B0DD-C334E3A636AE@icir.org>


On Mar 19, 2013, at 8:47 PM, Ron King <roncking at gmail.com> wrote:

> Hi, I want to send event log data directly to a nosql database. Where
> in the code should I look in order to add this capability?


We already support directly writing to ElasticSearch.  Look in src/logging/writers.

What database did you want to add support for?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From seth at icir.org  Tue Mar 19 19:03:15 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 19 Mar 2013 22:03:15 -0400
Subject: [Bro] Bro programming intro
In-Reply-To: <5148A943.9060209@ip-solutions.net>
References: <CAMPgRd7Q3+RZ_g3JoqWTuEXeFNH9wsXU4Y1qyYYsoCcBGmMuYw@mail.gmail.com>
	<FA096504-8769-42DC-B7F5-0BAF2026B14E@icir.org>
	<32694_1363707976_r2JFkBLO019088_CAMPgRd6dDUE-UG0yKZhnJVB5S5dT=GFn3HHQhW8OjkSUDpBuCg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E8552AC1FB2@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<CAMPgRd58+qGnO6kZC-5uNT3zAy583N+iWms4vrBgj+Zis5n8_g@mail.gmail.com>
	<F61F3AFC-10EE-4F78-B5E6-41BEC4C0C069@icir.org>
	<5148A943.9060209@ip-solutions.net>
Message-ID: <7149B974-07EB-4849-9E68-99B7AF44854B@icir.org>


On Mar 19, 2013, at 2:06 PM, Harry Hoffman <hhoffman at ip-solutions.net> wrote:

> Wow!!! What the heck did it cost to get that domain name???

Not as much as we were afraid it would. :)

> Congrats. Educause SPC is coming up and I'm sure Bro is going to be a
> hot topic of conversation :-)


I'll see you there then.  I'm going to be participating in the "Ask the Expert" session during the REN-ISAC event following the SPC.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From carlopmart at gmail.com  Wed Mar 20 00:25:35 2013
From: carlopmart at gmail.com (C. L. Martinez)
Date: Wed, 20 Mar 2013 07:25:35 +0000
Subject: [Bro] Detecting software components that do strange dns queries
Message-ID: <CAEjQA5LTJdDi3CFQwpwvqQZR6omkvCckpVRwxeTeGxUeFPfhsw@mail.gmail.com>

Hi all,

 Is it possible to detect what software components do "strange"
queries?? For example, in our network, we detected queries to
"abnormal" domains like these:

1363608064.778525|VmUnpNRkiF5|192.168.65.160|2933|10.196.0.67|53|udp|54891|gqtpngnqt.com|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
1363608064.792823|JT4SuPtIQ3k|192.168.65.160|2940|10.196.0.67|53|udp|3431|wvxzfmyw.cc|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
1363608064.794325|tYWZyjP18fd|192.168.65.160|2941|10.196.0.67|53|udp|15204|shlghhw.org|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
1363608079.436835|TO6u5Zqbx1|192.168.65.160|2962|10.196.0.67|53|udp|50810|xqqkwjqdbhh.ws|1|C_INTERNET|1|A|0|NOERROR|F|F|T|T|0|149.20.56.32,149.20.56.33,149.20.56.34|6024.000000,6024.000000,6024.000000

.. and a lot of more.

Any ideas how to accomplish this??


From lysemose at gmail.com  Wed Mar 20 01:03:56 2013
From: lysemose at gmail.com (Heine Lysemose)
Date: Wed, 20 Mar 2013 09:03:56 +0100
Subject: [Bro] Detecting software components that do strange dns queries
In-Reply-To: <CAEjQA5LTJdDi3CFQwpwvqQZR6omkvCckpVRwxeTeGxUeFPfhsw@mail.gmail.com>
References: <CAEjQA5LTJdDi3CFQwpwvqQZR6omkvCckpVRwxeTeGxUeFPfhsw@mail.gmail.com>
Message-ID: <CAN4C-D=_tajwviFDGMjF6Q_QiJv1=V4baR77_71-zY0_JOV3hA@mail.gmail.com>

Hi

Maybe this could help you...
http://code.google.com/p/security-onion/wiki/DNSAnomalyDetection

/Lysemose


On Wed, Mar 20, 2013 at 8:25 AM, C. L. Martinez <carlopmart at gmail.com>wrote:

> Hi all,
>
>  Is it possible to detect what software components do "strange"
> queries?? For example, in our network, we detected queries to
> "abnormal" domains like these:
>
> 1363608064.778525|VmUnpNRkiF5|192.168.65.160|2933|10.196.0.67|53|udp|54891|
> gqtpngnqt.com|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
>
> 1363608064.792823|JT4SuPtIQ3k|192.168.65.160|2940|10.196.0.67|53|udp|3431|wvxzfmyw.cc|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
> 1363608064.794325|tYWZyjP18fd|192.168.65.160|2941|10.196.0.67|53|udp|15204|
> shlghhw.org|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
> 1363608079.436835|TO6u5Zqbx1|192.168.65.160|2962|10.196.0.67|53|udp|50810|
> xqqkwjqdbhh.ws
> |1|C_INTERNET|1|A|0|NOERROR|F|F|T|T|0|149.20.56.32,149.20.56.33,149.20.56.34|6024.000000,6024.000000,6024.000000
>
> .. and a lot of more.
>
> Any ideas how to accomplish this??
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130320/734c9e19/attachment.html 

From nicolas.retrain at cea.fr  Wed Mar 20 03:31:25 2013
From: nicolas.retrain at cea.fr (nicolas.retrain at cea.fr)
Date: Wed, 20 Mar 2013 11:31:25 +0100
Subject: [Bro] Add a new protocol
Message-ID: <51498FFD.6050008@cea.fr>

Hi

I need to detect protocols which are not support in bro yet, so I am 
considering adding them into Bro.
However I am stuck at the beginning, I can't found where or how are 
declared all files.h into CMake (so compile with my news files fails 
because .o aren't generated).

Also, if you have some good articles or advices which could help me in 
adding protocols, it would be nice.
Thanks for your help.

Regards,

Nicolas




From nicolas.retrain at cea.fr  Wed Mar 20 03:45:52 2013
From: nicolas.retrain at cea.fr (nicolas.retrain at cea.fr)
Date: Wed, 20 Mar 2013 11:45:52 +0100
Subject: [Bro] [PROVENANCE  INTERNET]  Add a new protocol
In-Reply-To: <29614_1363775607_r2KAXR2T018248_51498FFD.6050008@cea.fr>
References: <29614_1363775607_r2KAXR2T018248_51498FFD.6050008@cea.fr>
Message-ID: <51499360.8050504@cea.fr>

Le 20/03/2013 11:31, nicolas.retrain at cea.fr a ?crit :
> Hi
>
> I need to detect protocols which are not support in bro yet, so I am
> considering adding them into Bro.
> However I am stuck at the beginning, I can't found where or how are
> declared all files.h into CMake (so compile with my news files fails
> because .o aren't generated).
I finally solved it, of course 2 min after sending the mail, sorry for 
the inconvenience.
I am still waiting for advices, thanks!
> Also, if you have some good articles or advices which could help me in
> adding protocols, it would be nice.
> Thanks for your help.
>
> Regards,
>
> Nicolas
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From sconzo at visiblerisk.com  Wed Mar 20 05:41:06 2013
From: sconzo at visiblerisk.com (Mike Sconzo)
Date: Wed, 20 Mar 2013 07:41:06 -0500
Subject: [Bro] Detecting software components that do strange dns queries
In-Reply-To: <CAN4C-D=_tajwviFDGMjF6Q_QiJv1=V4baR77_71-zY0_JOV3hA@mail.gmail.com>
References: <CAEjQA5LTJdDi3CFQwpwvqQZR6omkvCckpVRwxeTeGxUeFPfhsw@mail.gmail.com>
	<CAN4C-D=_tajwviFDGMjF6Q_QiJv1=V4baR77_71-zY0_JOV3hA@mail.gmail.com>
Message-ID: <CAGZoo6H9PzdZBSA8hdcmSW2HaTQj1mzh+4Df5RRMN+_wn0pWJQ@mail.gmail.com>

Are you asking from a host perspective (now that you've seen this
traffic on a network, what is causing it on the host) or from a
network perspective (how do I find suspicious queries like the in
network traffic)?

-=Mike

On Wed, Mar 20, 2013 at 3:03 AM, Heine Lysemose <lysemose at gmail.com> wrote:
> Hi
>
> Maybe this could help you...
> http://code.google.com/p/security-onion/wiki/DNSAnomalyDetection
>
> /Lysemose
>
>
> On Wed, Mar 20, 2013 at 8:25 AM, C. L. Martinez <carlopmart at gmail.com>
> wrote:
>>
>> Hi all,
>>
>>  Is it possible to detect what software components do "strange"
>> queries?? For example, in our network, we detected queries to
>> "abnormal" domains like these:
>>
>>
>> 1363608064.778525|VmUnpNRkiF5|192.168.65.160|2933|10.196.0.67|53|udp|54891|gqtpngnqt.com|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
>>
>> 1363608064.792823|JT4SuPtIQ3k|192.168.65.160|2940|10.196.0.67|53|udp|3431|wvxzfmyw.cc|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
>>
>> 1363608064.794325|tYWZyjP18fd|192.168.65.160|2941|10.196.0.67|53|udp|15204|shlghhw.org|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
>>
>> 1363608079.436835|TO6u5Zqbx1|192.168.65.160|2962|10.196.0.67|53|udp|50810|xqqkwjqdbhh.ws|1|C_INTERNET|1|A|0|NOERROR|F|F|T|T|0|149.20.56.32,149.20.56.33,149.20.56.34|6024.000000,6024.000000,6024.000000
>>
>> .. and a lot of more.
>>
>> Any ideas how to accomplish this??
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
cat ~/.bash_history > documentation.txt


From keqhe at cs.wisc.edu  Wed Mar 20 07:42:02 2013
From: keqhe at cs.wisc.edu (keqhe at cs.wisc.edu)
Date: Wed, 20 Mar 2013 09:42:02 -0500
Subject: [Bro] About Bro Processing Speed
In-Reply-To: <1202BE242E080642B0CD0AD0A03E8552AC24BC@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <cb0017022a01f8aff25e969a4a499943.squirrel@webmail.cs.wisc.edu>
	<13824_1363717681_r2JIRxO0014643_11E31E35-134B-482C-AD46-C4AD2FA1BA64@uwaterloo.ca>
	<1202BE242E080642B0CD0AD0A03E8552AC24BC@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <9b49757c8ff1b372e315f3a99c08dee9.squirrel@webmail.cs.wisc.edu>

Thanks for the helpful information!
> Just to throw another data point out there:
>
>> * 16 physical cores of model name	: Intel(R) Xeon(R) CPU E5-2680 @ 2.70
>> GHz
>> * 96GB of RAM
>> * Myricom NIC
>
> 28 workers (I have Hyperthreading turned on) keep up with a 6-7 Gbps
> average, and I've seen them do fine with short peaks of 9 Gbps or so. The
> Myricom cards definitely won't break the bank: card + SR optics +
> perpetual license is $895.
>
>   --Vlad
>
> On Mar 19, 2013, at 2:20 PM, Mike Patterson <mike.patterson at uwaterloo.ca>
>  wrote:
>
>> "It depends."
>>
>> On - hardware, mostly, and configuration.
>>
>> I keep meaning to write this up, but on *my* configuration:
>> * 16 cores of model name	: Intel(R) Xeon(R) CPU           X5677  @
>> 3.47GHz
>> * 72GB of RAM
>> * Endace DAG (9.2)
>> * some config magic by Seth, which I'd be happy to share.
>>
>> 6 workers keep up with ~2.5-3Gbps peaks, no problem.
>>
>> On lesser hardware, your mileage will definitely vary. The secret sauce
>> appears to be the DAG. Hot CPUs doesn't hurt either. :) You'll probably
>> find that an Endace will cost as much as the server you'd put it in. I
>> think it's worthwhile, but your budget is yours.
>>
>> It doesn't actually consume all of the above resources - I'm running
>> other things on the box too - but bro itself consumes ~4.5GB resident
>> per worker, and can be counted on to pin most of its allocated cores at
>> peak loads.
>>
>> Mike
>>
>> On 2013-03-19, at 1:49 PM, keqhe at cs.wisc.edu wrote:
>>
>>>
>>> Hello,
>>>
>>> Does any one know Bro's processing speed? i.e., can it support
>>> monitoring
>>> 1 Gbps or 10Gbps link?
>>>
>>> Thanks!
>>>
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>




From jsiwek at illinois.edu  Wed Mar 20 07:57:11 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Wed, 20 Mar 2013 14:57:11 +0000
Subject: [Bro] Add a new protocol
In-Reply-To: <51498FFD.6050008@cea.fr>
References: <51498FFD.6050008@cea.fr>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D116F5193E@chimbx2.ad.uillinois.edu>


> However I am stuck at the beginning, I can't found where or how are 
> declared all files.h into CMake (so compile with my news files fails 
> because .o aren't generated).

Adding source code files to the "bro_SRCS" list in src/CMakeLists.txt is typically the right place.

> Also, if you have some good articles or advices which could help me in 
> adding protocols, it would be nice.

The docs at [1] and [2] may be helpful.

    Jon

[1] http://bro.org/development/dpd.html#adding-analyzers
[2] http://bro.org/development/binpac-sample-analyzer.html


From lyeph at outlook.com  Wed Mar 20 12:55:16 2013
From: lyeph at outlook.com (lyeph lyeph)
Date: Wed, 20 Mar 2013 15:55:16 -0400
Subject: [Bro] Capturing DHCP events
Message-ID: <BAY002-W935C02B117468F9031C870B4EA0@phx.gbl>

Hello,
I'm looking into capturing DHCP events with Bro. I am hoping I am just overlooking a bro script that already handles the collection of DHCP events. The only one I have been able to find so far is at http://svn.icir.org/bro/trunk/bro/policy/dhcp.bro, but it does not appear to work with Bro 2.1. Can anyone point me in a better direction?  		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130320/02c0f2bd/attachment.html 

From vladg at cmu.edu  Wed Mar 20 14:14:48 2013
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Wed, 20 Mar 2013 21:14:48 +0000
Subject: [Bro] Capturing DHCP events
In-Reply-To: <1797_1363809323_r2KJtMfE005654_BAY002-W935C02B117468F9031C870B4EA0@phx.gbl>
References: <1797_1363809323_r2KJtMfE005654_BAY002-W935C02B117468F9031C870B4EA0@phx.gbl>
Message-ID: <1202BE242E080642B0CD0AD0A03E8552AC62FD@PGH-MSGMB-03.andrew.ad.cmu.edu>

I have an experimental git branch with DHCP support here: <https://github.com/grigorescu/bro/tree/remotes/origin/topic/vladg/dhcp>.

In order to use it, you'll need to do the following:

> git clone --recursive git://git.bro.org/bro
> cd bro
> git remote add dhcp_github https://github.com/grigorescu/bro.git
> git pull -s recursive -X theirs dhcp_github remotes/origin/topic/vladg/dhcp

Build Bro with the usual ./configure, make, make install

To enable the DHCP script, add this to local.bro:

> @load base/protocols/dhcp

You should then see a dhcp.log. Here's an example from mine:

> #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	mac	assigned_ip	lease_time	trans_id
> #types	time	string	addr	port	addr	port	string	addr	interval	count
> 1363813938.019865	4qCaj285Yr1	128.237.242.XX	68	128.237.224.X	67	58:bd:a3:ff:ee:dd	128.237.242.WWW	240.000000	3420159137
> 1363813938.020145	XLPtua3OiKg	128.237.237.YY	68	128.237.224.Y	67	90:27:e4:99:aa:cc	128.237.237.ZZ	240.000000	3009200041

Please let me know if you run into any problems with the script. I've been running it in production for about a month, and it works well for me.

  --Vlad


On Mar 20, 2013, at 3:55 PM, lyeph lyeph <lyeph at outlook.com>
 wrote:

> Hello,
> 
> I'm looking into capturing DHCP events with Bro. I am hoping I am just overlooking a bro script that already handles the collection of DHCP events. The only one I have been able to find so far is at http://svn.icir.org/bro/trunk/bro/policy/dhcp.bro, but it does not appear to work with Bro 2.1. Can anyone point me in a better direction? 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From jswan at sugf.com  Wed Mar 20 14:20:33 2013
From: jswan at sugf.com (Swan, Jay)
Date: Wed, 20 Mar 2013 21:20:33 +0000
Subject: [Bro] running bro -C from broctl
Message-ID: <20130320212034.7EFCD2C4003@rock.ICSI.Berkeley.EDU>

Is there a way to make Bro run with the -C flag from broctl?

I'm aware that this isn't advisable in production but I have a test system where it seems to be necessary.

Thanks,
Jay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130320/d75bc341/attachment.html 

From jsiwek at illinois.edu  Wed Mar 20 14:39:28 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Wed, 20 Mar 2013 21:39:28 +0000
Subject: [Bro] running bro -C from broctl
In-Reply-To: <20130320212034.7EFCD2C4003@rock.ICSI.Berkeley.EDU>
References: <20130320212034.7EFCD2C4003@rock.ICSI.Berkeley.EDU>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D116F53E1F@chimbx2.ad.uillinois.edu>


> Is there a way to make Bro run with the ?C flag from broctl?

Adding "redef ignore_checksums=T;" to your site/local.bro then doing `broctl check && broctl install && broctl restart` should be the equivalent.

    Jon


From dmandelb at bbn.com  Wed Mar 20 15:28:27 2013
From: dmandelb at bbn.com (David Mandelberg)
Date: Wed, 20 Mar 2013 18:28:27 -0400
Subject: [Bro] crash with default in record in record
Message-ID: <1363818507.7519.7.camel@titan>

Hi,

When I have a default value in one record that's inside of another
record, I get a crash trying to print that value. I've attached source
code that reliably crashes for me along with a log of what I see when it
crashes. The output I'm expecting is:

[bar=4321, foo=[foo=1234, quux=9876]]
[foo=1234, quux=9876]
9876

Has anybody seen this before? Am I trying to do something that's not
supported?


P.S. What I really want is something like "foo: Foo
&default=[$foo=bar];" so that bar$foo$foo defaults to the same value as
bar$bar. Is that possible?
-------------- next part --------------
type Foo: record {
	foo: count;
	quux: count &default=9876;
};

type Bar: record {
	bar: count;

	foo: Foo &default=[$foo=1234];
};

global bar: Bar = [$bar=4321];
print(bar);
print(bar$foo);
print(bar$foo$quux);
-------------- next part --------------
A non-text attachment was scrubbed...
Name: crash.log
Type: text/x-log
Size: 5097 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130320/167614cb/attachment.bin 

From seth at icir.org  Wed Mar 20 18:34:17 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 20 Mar 2013 21:34:17 -0400
Subject: [Bro] Capturing DHCP events
In-Reply-To: <1202BE242E080642B0CD0AD0A03E8552AC62FD@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <1797_1363809323_r2KJtMfE005654_BAY002-W935C02B117468F9031C870B4EA0@phx.gbl>
	<1202BE242E080642B0CD0AD0A03E8552AC62FD@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <83E9A702-A4EE-4F12-8301-3F2AE2AA5153@icir.org>


On Mar 20, 2013, at 5:14 PM, Vlad Grigorescu <vladg at cmu.edu> wrote:

> Please let me know if you run into any problems with the script. I've been running it in production for about a month, and it works well for me.


Juuuuuust waiting for that merge request?. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From seth at icir.org  Wed Mar 20 20:08:55 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 20 Mar 2013 23:08:55 -0400
Subject: [Bro] crash with default in record in record
In-Reply-To: <1363818507.7519.7.camel@titan>
References: <1363818507.7519.7.camel@titan>
Message-ID: <AF72B358-0104-4CB2-B4A9-85F1960FC399@icir.org>


On Mar 20, 2013, at 6:28 PM, David Mandelberg <dmandelb at bbn.com> wrote:

> Has anybody seen this before? Am I trying to do something that's not
> supported?


I've never seen anyone do that before, but your code should just be rejected at parse time anyway.  Feel free to file a ticket related to this, we can at least discuss your sample script in the tracker.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From seth at icir.org  Wed Mar 20 20:21:27 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 20 Mar 2013 23:21:27 -0400
Subject: [Bro] crash with default in record in record
In-Reply-To: <AF72B358-0104-4CB2-B4A9-85F1960FC399@icir.org>
References: <1363818507.7519.7.camel@titan>
	<AF72B358-0104-4CB2-B4A9-85F1960FC399@icir.org>
Message-ID: <9C400F6A-E122-4345-B3EC-49809CD6BD7C@icir.org>


On Mar 20, 2013, at 11:08 PM, Seth Hall <seth at icir.org> wrote:

> I've never seen anyone do that before, but your code should just be rejected at parse time anyway.

Actually ignore that comment about the code being rejected.  I read it wrong the first time and apparently I didn't finish correcting my response.  The code seems reasonable.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From robin at icir.org  Thu Mar 21 04:34:41 2013
From: robin at icir.org (Robin Sommer)
Date: Thu, 21 Mar 2013 04:34:41 -0700
Subject: [Bro] Capturing DHCP events
In-Reply-To: <83E9A702-A4EE-4F12-8301-3F2AE2AA5153@icir.org>
References: <1797_1363809323_r2KJtMfE005654_BAY002-W935C02B117468F9031C870B4EA0@phx.gbl>
	<1202BE242E080642B0CD0AD0A03E8552AC62FD@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<83E9A702-A4EE-4F12-8301-3F2AE2AA5153@icir.org>
Message-ID: <20130321113441.GG48845@icir.org>

Indeed! :)

Robin

On Wed, Mar 20, 2013 at 21:34 -0400, you wrote:

> 
> On Mar 20, 2013, at 5:14 PM, Vlad Grigorescu <vladg at cmu.edu> wrote:
> 
> > Please let me know if you run into any problems with the script. I've been running it in production for about a month, and it works well for me.
> 
> 
> Juuuuuust waiting for that merge request?. :)
> 
>   .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From lyeph at outlook.com  Thu Mar 21 05:13:41 2013
From: lyeph at outlook.com (lyeph .)
Date: Thu, 21 Mar 2013 08:13:41 -0400
Subject: [Bro] Capturing DHCP events
In-Reply-To: <1202BE242E080642B0CD0AD0A03E8552AC62FD@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <1797_1363809323_r2KJtMfE005654_BAY002-W935C02B117468F9031C870B4EA0@phx.gbl>,
	<1202BE242E080642B0CD0AD0A03E8552AC62FD@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <BAY002-W139CC1310FFB0F3DB3DBF9CB4EB0@phx.gbl>

Vlad,
Thanks! This is very close to what I was looking for. Any plans to include the DHCP Options Host Name in the logs?

> From: vladg at cmu.edu
> To: lyeph at outlook.com
> CC: bro at bro.org
> Subject: Re: [Bro] Capturing DHCP events
> Date: Wed, 20 Mar 2013 21:14:48 +0000
> 
> I have an experimental git branch with DHCP support here: <https://github.com/grigorescu/bro/tree/remotes/origin/topic/vladg/dhcp>.
> 
> In order to use it, you'll need to do the following:
> 
> > git clone --recursive git://git.bro.org/bro
> > cd bro
> > git remote add dhcp_github https://github.com/grigorescu/bro.git
> > git pull -s recursive -X theirs dhcp_github remotes/origin/topic/vladg/dhcp
> 
> Build Bro with the usual ./configure, make, make install
> 
> To enable the DHCP script, add this to local.bro:
> 
> > @load base/protocols/dhcp
> 
> You should then see a dhcp.log. Here's an example from mine:
> 
> > #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	mac	assigned_ip	lease_time	trans_id
> > #types	time	string	addr	port	addr	port	string	addr	interval	count
> > 1363813938.019865	4qCaj285Yr1	128.237.242.XX	68	128.237.224.X	67	58:bd:a3:ff:ee:dd	128.237.242.WWW	240.000000	3420159137
> > 1363813938.020145	XLPtua3OiKg	128.237.237.YY	68	128.237.224.Y	67	90:27:e4:99:aa:cc	128.237.237.ZZ	240.000000	3009200041
> 
> Please let me know if you run into any problems with the script. I've been running it in production for about a month, and it works well for me.
> 
>   --Vlad
> 
> 
> On Mar 20, 2013, at 3:55 PM, lyeph lyeph <lyeph at outlook.com>
>  wrote:
> 
> > Hello,
> > 
> > I'm looking into capturing DHCP events with Bro. I am hoping I am just overlooking a bro script that already handles the collection of DHCP events. The only one I have been able to find so far is at http://svn.icir.org/bro/trunk/bro/policy/dhcp.bro, but it does not appear to work with Bro 2.1. Can anyone point me in a better direction? 
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130321/32343616/attachment.html 

From jswan at sugf.com  Thu Mar 21 11:33:20 2013
From: jswan at sugf.com (Swan, Jay)
Date: Thu, 21 Mar 2013 18:33:20 +0000
Subject: [Bro] Capturing DHCP events
In-Reply-To: <BAY002-W139CC1310FFB0F3DB3DBF9CB4EB0@phx.gbl>
References: <1797_1363809323_r2KJtMfE005654_BAY002-W935C02B117468F9031C870B4EA0@phx.gbl>,
	<1202BE242E080642B0CD0AD0A03E8552AC62FD@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<BAY002-W139CC1310FFB0F3DB3DBF9CB4EB0@phx.gbl>
Message-ID: <20130321183322.7F9C02C4003@rock.ICSI.Berkeley.EDU>

Very nice. I'd also find it very useful to have the hostname included.

From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of lyeph .
Sent: Thursday, March 21, 2013 6:14 AM
To: Vlad Grigorescu
Cc: bro at bro.org
Subject: Re: [Bro] Capturing DHCP events

Vlad,

Thanks! This is very close to what I was looking for. Any plans to include the DHCP Options Host Name in the logs?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130321/66e1bc23/attachment.html 

From tritium.cat at gmail.com  Thu Mar 21 13:32:26 2013
From: tritium.cat at gmail.com (Tritium Cat)
Date: Thu, 21 Mar 2013 13:32:26 -0700
Subject: [Bro] Workers fail to process traffic due to a PF_RING problem
Message-ID: <CAMPgRd5gcvPMtqkiv7_s0EzjiBBf1o=ymcxqBu=UcqByM_4iLA@mail.gmail.com>

Hi all,

I've noticed a problem with the cluster where some workers do not start.  I
can tell this by looking at the stats from broctl using "netstats",
"status", and "ps.bro".

It seems there is a problem with how PF_RING is used or PF_RING itself.
 (Or maybe my setup.)  Has anyone else encountered this problem and started
trying to isolate it ?

Restarting the worker seems to make it "work" again.  This phenomenon seems
to happen almost every time I start the cluster.  Some times, on some
nodes, more than one worker is affected.

If it helps to know I am using PF_RING from SVN 2013-03-19 and have
experienced the issue with all previous versions.  Bro is 2.1-380.

I searched the bug/problem tracker at http://tracker.bro.org/bro without
result.  If this is something not resolved by mailing list and worth
tracking in a ticket I will set it up.


Thanks,

--TC




Examples.  Via netstats, notice worker-1-4 is lame.


[BroControl] > netstats
worker-1-1: 1363895818.282736 recvd=29542788 dropped=4 link=29542788
worker-1-10: 1363895818.482747 recvd=20389244 dropped=1 link=20389244
worker-1-11: 1363895818.682289 recvd=24803977 dropped=1 link=24803977
worker-1-12: 1363895818.882953 recvd=28730644 dropped=1 link=28730644
worker-1-13: 1363895819.082850 recvd=19810612 dropped=0 link=19810612
worker-1-14: 1363895819.290962 recvd=22651710 dropped=0 link=22651710
worker-1-15: 1363895819.490876 recvd=27415776 dropped=0 link=27415776
worker-1-16: 1363895819.694541 recvd=21634742 dropped=0 link=21634742
worker-1-17: 1363895819.895422 recvd=20572973 dropped=0 link=20572973
worker-1-18: 1363895820.095018 recvd=25490613 dropped=2 link=25490613
worker-1-19: 1363895820.298648 recvd=19699362 dropped=0 link=19699362
worker-1-2: 1363895820.499099 recvd=23931030 dropped=1 link=23931030
worker-1-20: 1363895820.699632 recvd=21769411 dropped=0 link=21769411
worker-1-3: 1363895820.899525 recvd=21604270 dropped=1 link=21604270
worker-1-4: 1363895821.102857 recvd=0 dropped=0 link=0
worker-1-5: 1363895821.307124 recvd=22320056 dropped=0 link=22320056
(..cut..)


Find what PID worker-1-4 is using by checking broctl "status".

[BroControl] > status
Name       Type       Host       Status        Pid    Peers  Started
(...cut...)
worker-1-4 worker    10.1.1.1  running       17618  2      21 Mar 12:20:21
(...cut...)



Go check the PF_RING stats for PID 17618


root at bro:/home/bro# cat /proc/net/pf_ring/17618-eth5.9
Bound Device(s)    : eth5
Active             : 1
Breed              : Non-DNA
Sampling Rate      : 1
Capture Direction  : RX+TX
Socket Mode        : RX+TX
Appl. Name         : <unknown>
IP Defragment      : No
BPF Filtering      : Enabled
# Sw Filt. Rules   : 0
# Hw Filt. Rules   : 0
Poll Pkt Watermark : 1
Num Poll Calls     : 16161864
Channel Id Mask    : 0xFFFFFFFF
Cluster Id         : 20
Slot Version       : 15 [5.5.3]
Min Num Slots      : 6966
Bucket Len         : 9600
Slot Len           : 9632 [bucket+header]
Tot Memory         : 67108864
Tot Packets        : 0
Tot Pkt Lost       : 0
Tot Insert         : 0
Tot Read           : 0
Insert Offset      : 0
Remove Offset      : 0
TX: Send Ok        : 0
TX: Send Errors    : 0
Reflect: Fwd Ok    : 0
Reflect: Fwd Errors: 0
Num Free Slots     : 6966



No packets huh.  Must be something with how PF_RING is used or PF_RING
itself.  What does restarting the worker do ?


[BroControl] > restart worker-1-4
stopping ...
stopping worker-1-4 ...
starting ...
starting worker-1-4 ...

[BroControl] > status
Name       Type       Host       Status        Pid    Peers  Started
(...cut...)
worker-1-4 worker     10.1.1.1 running       18854  2      21 Mar 12:58:00
(...cut...)

[BroControl] > netstats
(...cut...)
worker-1-4: 1363896589.166826 recvd=6413632 dropped=112989 link=6413632
(...cut...)



On checking the PF_RING stats again it looks like things are working now.
 There was a brief moment of "dropped packets" during the restart but that
counter has not incremented since.


root at bro:/home/bro# cat /proc/net/pf_ring/18854-eth5.21
Bound Device(s)    : eth5
Active             : 1
Breed              : Non-DNA
Sampling Rate      : 1
Capture Direction  : RX+TX
Socket Mode        : RX+TX
Appl. Name         : <unknown>
IP Defragment      : No
BPF Filtering      : Enabled
# Sw Filt. Rules   : 0
# Hw Filt. Rules   : 0
Poll Pkt Watermark : 1
Num Poll Calls     : 6637605
Channel Id Mask    : 0xFFFFFFFF
Cluster Id         : 20
Slot Version       : 15 [5.5.3]
Min Num Slots      : 6966
Bucket Len         : 9600
Slot Len           : 9632 [bucket+header]
Tot Memory         : 67108864
Tot Packets        : 7711193
Tot Pkt Lost       : 112989
Tot Insert         : 7598204
Tot Read           : 7598197
Insert Offset      : 4454256
Remove Offset      : 4446288
TX: Send Ok        : 0
TX: Send Errors    : 0
Reflect: Fwd Ok    : 0
Reflect: Fwd Errors: 0
Num Free Slots     : 6959
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130321/50c5ec4b/attachment.html 

From tritium.cat at gmail.com  Thu Mar 21 13:36:42 2013
From: tritium.cat at gmail.com (Tritium Cat)
Date: Thu, 21 Mar 2013 13:36:42 -0700
Subject: [Bro] Detecting software components that do strange dns queries
In-Reply-To: <CAGZoo6H9PzdZBSA8hdcmSW2HaTQj1mzh+4Df5RRMN+_wn0pWJQ@mail.gmail.com>
References: <CAEjQA5LTJdDi3CFQwpwvqQZR6omkvCckpVRwxeTeGxUeFPfhsw@mail.gmail.com>
	<CAN4C-D=_tajwviFDGMjF6Q_QiJv1=V4baR77_71-zY0_JOV3hA@mail.gmail.com>
	<CAGZoo6H9PzdZBSA8hdcmSW2HaTQj1mzh+4Df5RRMN+_wn0pWJQ@mail.gmail.com>
Message-ID: <CAMPgRd7N6tMbWqtE8ZewRse8Ks-jL=TiO2x=UNW4DxYs8qXZcg@mail.gmail.com>

Character frequency analysis.



On Wed, Mar 20, 2013 at 5:41 AM, Mike Sconzo <sconzo at visiblerisk.com> wrote:

> Are you asking from a host perspective (now that you've seen this
> traffic on a network, what is causing it on the host) or from a
> network perspective (how do I find suspicious queries like the in
> network traffic)?
>
> -=Mike
>
> On Wed, Mar 20, 2013 at 3:03 AM, Heine Lysemose <lysemose at gmail.com>
> wrote:
> > Hi
> >
> > Maybe this could help you...
> > http://code.google.com/p/security-onion/wiki/DNSAnomalyDetection
> >
> > /Lysemose
> >
> >
> > On Wed, Mar 20, 2013 at 8:25 AM, C. L. Martinez <carlopmart at gmail.com>
> > wrote:
> >>
> >> Hi all,
> >>
> >>  Is it possible to detect what software components do "strange"
> >> queries?? For example, in our network, we detected queries to
> >> "abnormal" domains like these:
> >>
> >>
> >>
> 1363608064.778525|VmUnpNRkiF5|192.168.65.160|2933|10.196.0.67|53|udp|54891|
> gqtpngnqt.com|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
> >>
> >>
> 1363608064.792823|JT4SuPtIQ3k|192.168.65.160|2940|10.196.0.67|53|udp|3431|wvxzfmyw.cc|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
> >>
> >>
> 1363608064.794325|tYWZyjP18fd|192.168.65.160|2941|10.196.0.67|53|udp|15204|
> shlghhw.org|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
> >>
> >>
> 1363608079.436835|TO6u5Zqbx1|192.168.65.160|2962|10.196.0.67|53|udp|50810|
> xqqkwjqdbhh.ws
> |1|C_INTERNET|1|A|0|NOERROR|F|F|T|T|0|149.20.56.32,149.20.56.33,149.20.56.34|
> 6024.000000,6024.000000,6024.000000
> >>
> >> .. and a lot of more.
> >>
> >> Any ideas how to accomplish this??
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> --
> cat ~/.bash_history > documentation.txt
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130321/45788f28/attachment.html 

From carlopmart at gmail.com  Fri Mar 22 00:32:39 2013
From: carlopmart at gmail.com (C. L. Martinez)
Date: Fri, 22 Mar 2013 08:32:39 +0100
Subject: [Bro] Detecting software components that do strange dns queries
In-Reply-To: <CAMPgRd7N6tMbWqtE8ZewRse8Ks-jL=TiO2x=UNW4DxYs8qXZcg@mail.gmail.com>
References: <CAEjQA5LTJdDi3CFQwpwvqQZR6omkvCckpVRwxeTeGxUeFPfhsw@mail.gmail.com>
	<CAN4C-D=_tajwviFDGMjF6Q_QiJv1=V4baR77_71-zY0_JOV3hA@mail.gmail.com>
	<CAGZoo6H9PzdZBSA8hdcmSW2HaTQj1mzh+4Df5RRMN+_wn0pWJQ@mail.gmail.com>
	<CAMPgRd7N6tMbWqtE8ZewRse8Ks-jL=TiO2x=UNW4DxYs8qXZcg@mail.gmail.com>
Message-ID: <CAEjQA5+60Ebd4kLsyQezd=KhiswPAdjSVakBKt-OOZKaO=i4og@mail.gmail.com>

On Thu, Mar 21, 2013 at 9:36 PM, Tritium Cat <tritium.cat at gmail.com> wrote:
> Character frequency analysis.
>
>
>

Do you mean https://www.google.es/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CDAQFjAA&url=http%3A%2F%2Farxiv.org%2Fpdf%2F1004.4358&ei=eQFMUcnUGsamhAfDzYGoAQ&usg=AFQjCNG7i1H_2CSKH5k11Z44zOg6sLAQgA&bvm=bv.44158598,d.ZG4??


From lysemose at gmail.com  Fri Mar 22 00:47:51 2013
From: lysemose at gmail.com (Heine Lysemose)
Date: Fri, 22 Mar 2013 08:47:51 +0100
Subject: [Bro] Detecting software components that do strange dns queries
In-Reply-To: <CAEjQA5+60Ebd4kLsyQezd=KhiswPAdjSVakBKt-OOZKaO=i4og@mail.gmail.com>
References: <CAEjQA5LTJdDi3CFQwpwvqQZR6omkvCckpVRwxeTeGxUeFPfhsw@mail.gmail.com>
	<CAN4C-D=_tajwviFDGMjF6Q_QiJv1=V4baR77_71-zY0_JOV3hA@mail.gmail.com>
	<CAGZoo6H9PzdZBSA8hdcmSW2HaTQj1mzh+4Df5RRMN+_wn0pWJQ@mail.gmail.com>
	<CAMPgRd7N6tMbWqtE8ZewRse8Ks-jL=TiO2x=UNW4DxYs8qXZcg@mail.gmail.com>
	<CAEjQA5+60Ebd4kLsyQezd=KhiswPAdjSVakBKt-OOZKaO=i4og@mail.gmail.com>
Message-ID: <CAN4C-Dm6yFTEJE+vc2gKKicrDaC=16Qv9vt-7yL2Zp+9JX7OEg@mail.gmail.com>

I saw this the other day on Twitter,
https://github.com/sethhall/bro-domain-generation, but that still doesn't
answer your original question.

/Lysemose


On Fri, Mar 22, 2013 at 8:32 AM, C. L. Martinez <carlopmart at gmail.com>wrote:

> On Thu, Mar 21, 2013 at 9:36 PM, Tritium Cat <tritium.cat at gmail.com>
> wrote:
> > Character frequency analysis.
> >
> >
> >
>
> Do you mean
> https://www.google.es/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CDAQFjAA&url=http%3A%2F%2Farxiv.org%2Fpdf%2F1004.4358&ei=eQFMUcnUGsamhAfDzYGoAQ&usg=AFQjCNG7i1H_2CSKH5k11Z44zOg6sLAQgA&bvm=bv.44158598,d.ZG4
> ??
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130322/81fa0ba6/attachment.html 

From dn1nj4 at gmail.com  Fri Mar 22 03:49:29 2013
From: dn1nj4 at gmail.com (Digital Ninja)
Date: Fri, 22 Mar 2013 06:49:29 -0400
Subject: [Bro] Extracting Email Attachments
Message-ID: <CADBniYj-+Mmb2gF0-WbKs+SjCEq0DquWc-wUequfxZ5_qpF0vg@mail.gmail.com>

Hello all,

New bro user here.  I'm trying to understand how to enable email attachment
extraction with bro.  I see in smtp-entities the setting "extract-file"
which by default is False.  What is the right way to enable it and set the
directory where these attachments will reside?

Thanks in advance!
Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130322/a7c0da97/attachment.html 

From vladg at cmu.edu  Fri Mar 22 07:06:43 2013
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Fri, 22 Mar 2013 14:06:43 +0000
Subject: [Bro] Detecting software components that do strange dns queries
In-Reply-To: <8671_1363899017_r2LKoGto010910_CAMPgRd7N6tMbWqtE8ZewRse8Ks-jL=TiO2x=UNW4DxYs8qXZcg@mail.gmail.com>
References: <CAEjQA5LTJdDi3CFQwpwvqQZR6omkvCckpVRwxeTeGxUeFPfhsw@mail.gmail.com>
	<CAN4C-D=_tajwviFDGMjF6Q_QiJv1=V4baR77_71-zY0_JOV3hA@mail.gmail.com>
	<CAGZoo6H9PzdZBSA8hdcmSW2HaTQj1mzh+4Df5RRMN+_wn0pWJQ@mail.gmail.com>
	<8671_1363899017_r2LKoGto010910_CAMPgRd7N6tMbWqtE8ZewRse8Ks-jL=TiO2x=UNW4DxYs8qXZcg@mail.gmail.com>
Message-ID: <1202BE242E080642B0CD0AD0A03E8552ACC0D4@PGH-MSGMB-03.andrew.ad.cmu.edu>

You can do character frequency analysis with a simple Bro script. Look at <http://www.bro.org/documentation-git/scripts/base/strings.bif.html> to see the functions you can use for strings.

I think that this is asking the wrong question, however. I'd be amazed if you could reliably determine "good" domains from "bad" domains based simply on character frequency analysis. Bro can calculate entropy for you: <http://www.bro.org/documentation/scripts/base/bro.bif.html#id-find_entropy>. That being said, I don't think entropy is the right answer either. 

Here are the entropy results (in no particular order) for the 4 domains you listed and for 4 very common domains (google.com, twitter.com, fbcdn.net and amazon.co.uk):

[entropy=2.646439, chi_square=450.8, mean=100.2, monte_carlo_pi=4.0, serial_correlation=0.096875]
[entropy=3.085055, chi_square=400.538462, mean=104.692308, monte_carlo_pi=4.0, serial_correlation=-0.005991]
[entropy=3.095795, chi_square=338.090909, mean=106.727273, monte_carlo_pi=4.0, serial_correlation=0.062381]
[entropy=3.027169, chi_square=384.636364, mean=104.727273, monte_carlo_pi=4.0, serial_correlation=0.011643]
[entropy=3.182006, chi_square=424.857143, mean=105.5, monte_carlo_pi=4.0, serial_correlation=-0.050923]
[entropy=2.947703, chi_square=303.888889, mean=98.0, monte_carlo_pi=4.0, serial_correlation=-0.316796]
[entropy=3.084963, chi_square=372.0, mean=97.666667, monte_carlo_pi=4.0, serial_correlation=-0.248104]
[entropy=2.845351, chi_square=431.181818, mean=102.818182, monte_carlo_pi=4.0, serial_correlation=-0.322755]

I don't know about you, but I can't tell which are good and which are bad. I suspect that DNS names are too short of a sample to provide any meaningful data.

I think you should focus instead on the behavior that you're trying to detect. Looking at your example below, some alerts that'd be more useful might be: 

 - Too many NXDOMAIN queries. 
 - A query that resolves to an ISC sinkhole. 
 - Queries for a domain that no one else queried.
 - Repetitive queries every X seconds with little to no deviation.
 - Queries for a domain that you haven't seen before.

Hope this helps,

  --Vlad

On Mar 21, 2013, at 4:36 PM, Tritium Cat <tritium.cat at gmail.com> wrote:

> Character frequency analysis.
> 
> 
> 
> On Wed, Mar 20, 2013 at 5:41 AM, Mike Sconzo <sconzo at visiblerisk.com> wrote:
> Are you asking from a host perspective (now that you've seen this
> traffic on a network, what is causing it on the host) or from a
> network perspective (how do I find suspicious queries like the in
> network traffic)?
> 
> -=Mike
> 
> On Wed, Mar 20, 2013 at 3:03 AM, Heine Lysemose <lysemose at gmail.com> wrote:
> > Hi
> >
> > Maybe this could help you...
> > http://code.google.com/p/security-onion/wiki/DNSAnomalyDetection
> >
> > /Lysemose
> >
> >
> > On Wed, Mar 20, 2013 at 8:25 AM, C. L. Martinez <carlopmart at gmail.com>
> > wrote:
> >>
> >> Hi all,
> >>
> >>  Is it possible to detect what software components do "strange"
> >> queries?? For example, in our network, we detected queries to
> >> "abnormal" domains like these:
> >>
> >>
> >> 1363608064.778525|VmUnpNRkiF5|192.168.65.160|2933|10.196.0.67|53|udp|54891|gqtpngnqt.com|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
> >>
> >> 1363608064.792823|JT4SuPtIQ3k|192.168.65.160|2940|10.196.0.67|53|udp|3431|wvxzfmyw.cc|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
> >>
> >> 1363608064.794325|tYWZyjP18fd|192.168.65.160|2941|10.196.0.67|53|udp|15204|shlghhw.org|1|C_INTERNET|1|A|-|-|F|F|T|F|0|-|-
> >>
> >> 1363608079.436835|TO6u5Zqbx1|192.168.65.160|2962|10.196.0.67|53|udp|50810|xqqkwjqdbhh.ws|1|C_INTERNET|1|A|0|NOERROR|F|F|T|T|0|149.20.56.32,149.20.56.33,149.20.56.34|6024.000000,6024.000000,6024.000000
> >>
> >> .. and a lot of more.
> >>
> >> Any ideas how to accomplish this??
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> 
> --
> cat ~/.bash_history > documentation.txt
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From init.conf at gmail.com  Fri Mar 22 07:32:09 2013
From: init.conf at gmail.com (Aashish SHARMA)
Date: Fri, 22 Mar 2013 07:32:09 -0700
Subject: [Bro] Extracting Email Attachments
In-Reply-To: <CADBniYj-+Mmb2gF0-WbKs+SjCEq0DquWc-wUequfxZ5_qpF0vg@mail.gmail.com>
References: <CADBniYj-+Mmb2gF0-WbKs+SjCEq0DquWc-wUequfxZ5_qpF0vg@mail.gmail.com>
Message-ID: <22DBDB28-B818-47B8-9BC9-BFB7B18C2E4C@gmail.com>


## define the mime types you want extracted /.*/ means everything

redef SMTP::extract_file_types += /application\/*/; 

## path where extracted attachments need to go: 
redef SMTP::extraction_prefix = "/data/bro/extract/smtp-entity" ;



On Mar 22, 2013, at 3:49 AM, Digital Ninja <dn1nj4 at gmail.com> wrote:

> Hello all,
> 
> New bro user here.  I'm trying to understand how to enable email attachment extraction with bro.  I see in smtp-entities the setting "extract-file" which by default is False.  What is the right way to enable it and set the directory where these attachments will reside?
> 
> Thanks in advance!
> Jason
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From carlopmart at gmail.com  Fri Mar 22 08:06:02 2013
From: carlopmart at gmail.com (C. L. Martinez)
Date: Fri, 22 Mar 2013 15:06:02 +0000
Subject: [Bro] Detecting software components that do strange dns queries
In-Reply-To: <1202BE242E080642B0CD0AD0A03E8552ACC0D4@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <CAEjQA5LTJdDi3CFQwpwvqQZR6omkvCckpVRwxeTeGxUeFPfhsw@mail.gmail.com>
	<CAN4C-D=_tajwviFDGMjF6Q_QiJv1=V4baR77_71-zY0_JOV3hA@mail.gmail.com>
	<CAGZoo6H9PzdZBSA8hdcmSW2HaTQj1mzh+4Df5RRMN+_wn0pWJQ@mail.gmail.com>
	<8671_1363899017_r2LKoGto010910_CAMPgRd7N6tMbWqtE8ZewRse8Ks-jL=TiO2x=UNW4DxYs8qXZcg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E8552ACC0D4@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <CAEjQA5K1T0jYfj4H5Xb8R8NgETKrbvCz9FKx_h-g4YQU6niVcw@mail.gmail.com>

On Fri, Mar 22, 2013 at 2:06 PM, Vlad Grigorescu <vladg at cmu.edu> wrote:
> You can do character frequency analysis with a simple Bro script. Look at <http://www.bro.org/documentation-git/scripts/base/strings.bif.html> to see the functions you can use for strings.
>
> I think that this is asking the wrong question, however. I'd be amazed if you could reliably determine "good" domains from "bad" domains based simply on character frequency analysis. Bro can calculate entropy for you: <http://www.bro.org/documentation/scripts/base/bro.bif.html#id-find_entropy>. That being said, I don't think entropy is the right answer either.
>
> Here are the entropy results (in no particular order) for the 4 domains you listed and for 4 very common domains (google.com, twitter.com, fbcdn.net and amazon.co.uk):
>
> [entropy=2.646439, chi_square=450.8, mean=100.2, monte_carlo_pi=4.0, serial_correlation=0.096875]
> [entropy=3.085055, chi_square=400.538462, mean=104.692308, monte_carlo_pi=4.0, serial_correlation=-0.005991]
> [entropy=3.095795, chi_square=338.090909, mean=106.727273, monte_carlo_pi=4.0, serial_correlation=0.062381]
> [entropy=3.027169, chi_square=384.636364, mean=104.727273, monte_carlo_pi=4.0, serial_correlation=0.011643]
> [entropy=3.182006, chi_square=424.857143, mean=105.5, monte_carlo_pi=4.0, serial_correlation=-0.050923]
> [entropy=2.947703, chi_square=303.888889, mean=98.0, monte_carlo_pi=4.0, serial_correlation=-0.316796]
> [entropy=3.084963, chi_square=372.0, mean=97.666667, monte_carlo_pi=4.0, serial_correlation=-0.248104]
> [entropy=2.845351, chi_square=431.181818, mean=102.818182, monte_carlo_pi=4.0, serial_correlation=-0.322755]
>
> I don't know about you, but I can't tell which are good and which are bad. I suspect that DNS names are too short of a sample to provide any meaningful data.
>
> I think you should focus instead on the behavior that you're trying to detect. Looking at your example below, some alerts that'd be more useful might be:
>
>  - Too many NXDOMAIN queries.
>  - A query that resolves to an ISC sinkhole.
>  - Queries for a domain that no one else queried.
>  - Repetitive queries every X seconds with little to no deviation.
>  - Queries for a domain that you haven't seen before.
>
> Hope this helps,
>
>   --Vlad
>

Many many thanks Vlad for your explanation ... I'll think about it this weekend



From dmandelb at bbn.com  Fri Mar 22 09:52:21 2013
From: dmandelb at bbn.com (David Mandelberg)
Date: Fri, 22 Mar 2013 12:52:21 -0400
Subject: [Bro] set intersection
Message-ID: <1363971141.14456.21.camel@titan>

Hi,

Is there a built-in way to compute the intersection of two sets in a Bro
script, or should I write my own intersection function?



From seth at icir.org  Fri Mar 22 12:07:57 2013
From: seth at icir.org (Seth Hall)
Date: Fri, 22 Mar 2013 15:07:57 -0400
Subject: [Bro] set intersection
In-Reply-To: <1363971141.14456.21.camel@titan>
References: <1363971141.14456.21.camel@titan>
Message-ID: <D859FFD6-5267-4F11-97F7-AAD670B90689@icir.org>


On Mar 22, 2013, at 12:52 PM, David Mandelberg <dmandelb at bbn.com> wrote:

> Is there a built-in way to compute the intersection of two sets in a Bro
> script, or should I write my own intersection function?


There isn't a set intersection operator.  Would be awfully handy. ;)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From seth at icir.org  Fri Mar 22 12:15:34 2013
From: seth at icir.org (Seth Hall)
Date: Fri, 22 Mar 2013 15:15:34 -0400
Subject: [Bro] Bro Exchange Videos
Message-ID: <BC99A809-7EAC-4D08-8D8A-5CC83B109760@icir.org>

I just realized that I forgot to send an email to this list to mention that I *finally* got the videos from last year's Bro Exchange posted.

Here they are:
	http://bro.org/community/exchange2012.html

  .Seth
--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130322/bef48c37/attachment.bin 

From rjenkins at rmjconsulting.net  Fri Mar 22 12:20:18 2013
From: rjenkins at rmjconsulting.net (Ron Jenkins)
Date: Fri, 22 Mar 2013 19:20:18 +0000
Subject: [Bro] Bro Exchange Videos
In-Reply-To: <BC99A809-7EAC-4D08-8D8A-5CC83B109760@icir.org>
References: <BC99A809-7EAC-4D08-8D8A-5CC83B109760@icir.org>
Message-ID: <A3B923A8-806D-4B85-A9AD-9496323801A4@rmjconsulting.net>

Thank you!

Ron Jenkins (SnortCP,VCP 3 / 4,MCNE,CNE6,MCPS,MCNPS,CCNA)
RMJ Consulting, LLC.
"Bringing Companies and Solutions Together"
Owner / Senior Architect
Physical Address
11715 Bricksome Ave STE B-7
Baton Rouge, LA 70816
Mail Address
7575 Jefferson Hwy #103
Baton Rouge, LA 70806
Office. 225-448-5214
Fax. 225-448-5324
Cell. 225-931-1632
Email. rjenkins at rmjconsulting.net
Web. http://www.rmjconsulting.net<http://www.rmjconsulting.net/>

On Mar 22, 2013, at 2:16 PM, "Seth Hall" <seth at icir.org> wrote:

> I just realized that I forgot to send an email to this list to mention that I *finally* got the videos from last year's Bro Exchange posted.
> 
> Here they are:
>    http://bro.org/community/exchange2012.html
> 
>  .Seth
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From jlay at slave-tothe-box.net  Fri Mar 22 12:33:17 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 22 Mar 2013 13:33:17 -0600
Subject: [Bro] Quick question
Message-ID: <8f57bda070321c5e7650421d72cd7999@localhost>

Hey all

So quick question, from the notice log:

1363973380.007453       5L7Bsj0Y8dj     x.x.x.x    36237   
206.169.145.206 80      tcp     HTTP::MD5       x.x.x.x 
88c48daab78eee9f856c8bff2141f09b 
http://r3---sn-ufuxaxjvh-v53e.c.pack.google.com/edgedl/toolbar/t7/data/7.4.3607.2246/GoogleToolbarInstaller_updater_signed.exe?ms=nvh&mv=u&mt=1363972912&ir=1&cms_redirect=yes 
88c48daab78eee9f856c8bff2141f09b x.x.x.x    206.169.145.206 80      -   
    bro     Notice::ACTION_LOG      6       3600.000000     F       -    
   -       -       -       -       -       -       -


What's this telling me?  Usually there's something like Invalid Cert or 
something like that in the notice.log to tell me why it hit, just wasn't 
seeing the reason here.  Thank you.

James


From dn1nj4 at gmail.com  Fri Mar 22 12:34:07 2013
From: dn1nj4 at gmail.com (Digital Ninja)
Date: Fri, 22 Mar 2013 15:34:07 -0400
Subject: [Bro] Extracting Email Attachments
In-Reply-To: <22DBDB28-B818-47B8-9BC9-BFB7B18C2E4C@gmail.com>
References: <CADBniYj-+Mmb2gF0-WbKs+SjCEq0DquWc-wUequfxZ5_qpF0vg@mail.gmail.com>
	<22DBDB28-B818-47B8-9BC9-BFB7B18C2E4C@gmail.com>
Message-ID: <CADBniYitmJbsJDxizCdVyGcH+5Md5oHYUr7QQ4U-S_eGzkgfCg@mail.gmail.com>

Based on Aashish's recommendations, I added the following 4 lines to the
end of my local.bro:

redef SMTP::extract_file_types += /application\/*/;
redef SMTP::extraction_prefix = "/tmp/extracted_";
redef SMTP::extract_file = T;
redef SMTP::calc_md5 = T;

While there are attachments listed in the smtp_entities.log, they have no
MD5 hashes and have not been extracted to /tmp.  What am I missing?



On Fri, Mar 22, 2013 at 10:32 AM, Aashish SHARMA <init.conf at gmail.com>wrote:

>
> ## define the mime types you want extracted /.*/ means everything
>
> redef SMTP::extract_file_types += /application\/*/;
>
> ## path where extracted attachments need to go:
> redef SMTP::extraction_prefix = "/data/bro/extract/smtp-entity" ;
>
>
>
> On Mar 22, 2013, at 3:49 AM, Digital Ninja <dn1nj4 at gmail.com> wrote:
>
> > Hello all,
> >
> > New bro user here.  I'm trying to understand how to enable email
> attachment extraction with bro.  I see in smtp-entities the setting
> "extract-file" which by default is False.  What is the right way to enable
> it and set the directory where these attachments will reside?
> >
> > Thanks in advance!
> > Jason
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130322/4797f1d0/attachment.html 

From bernhard at ICSI.Berkeley.EDU  Fri Mar 22 12:44:44 2013
From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann)
Date: Fri, 22 Mar 2013 12:44:44 -0700
Subject: [Bro] Extracting Email Attachments
In-Reply-To: <CADBniYitmJbsJDxizCdVyGcH+5Md5oHYUr7QQ4U-S_eGzkgfCg@mail.gmail.com>
References: <CADBniYj-+Mmb2gF0-WbKs+SjCEq0DquWc-wUequfxZ5_qpF0vg@mail.gmail.com>
	<22DBDB28-B818-47B8-9BC9-BFB7B18C2E4C@gmail.com>
	<CADBniYitmJbsJDxizCdVyGcH+5Md5oHYUr7QQ4U-S_eGzkgfCg@mail.gmail.com>
Message-ID: <A0F530FF-4914-45EC-A5E8-F1DF34DCB486@icsi.berkeley.edu>

I think it should have been /application\/.*/ instead of /application\/*/ - I think
Aashish made a small typo there. Could you try adding the missing "."? :)

Bernhard

On Mar 22, 2013, at 12:34 PM, Digital Ninja <dn1nj4 at gmail.com> wrote:

> Based on Aashish's recommendations, I added the following 4 lines to the end of my local.bro: 
> 
> redef SMTP::extract_file_types += /application\/*/;
> redef SMTP::extraction_prefix = "/tmp/extracted_";
> redef SMTP::extract_file = T;
> redef SMTP::calc_md5 = T;
> 
> While there are attachments listed in the smtp_entities.log, they have no MD5 hashes and have not been extracted to /tmp.  What am I missing?
> 
> 
> 
> On Fri, Mar 22, 2013 at 10:32 AM, Aashish SHARMA <init.conf at gmail.com> wrote:
> 
> ## define the mime types you want extracted /.*/ means everything
> 
> redef SMTP::extract_file_types += /application\/*/;
> 
> ## path where extracted attachments need to go:
> redef SMTP::extraction_prefix = "/data/bro/extract/smtp-entity" ;
> 
> 
> 
> On Mar 22, 2013, at 3:49 AM, Digital Ninja <dn1nj4 at gmail.com> wrote:
> 
> > Hello all,
> >
> > New bro user here.  I'm trying to understand how to enable email attachment extraction with bro.  I see in smtp-entities the setting "extract-file" which by default is False.  What is the right way to enable it and set the directory where these attachments will reside?
> >
> > Thanks in advance!
> > Jason
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From seth at icir.org  Fri Mar 22 12:56:13 2013
From: seth at icir.org (Seth Hall)
Date: Fri, 22 Mar 2013 15:56:13 -0400
Subject: [Bro] Quick question
In-Reply-To: <8f57bda070321c5e7650421d72cd7999@localhost>
References: <8f57bda070321c5e7650421d72cd7999@localhost>
Message-ID: <EFDAAC61-A1D4-42D5-8E7A-9D7E3A41153D@icir.org>


On Mar 22, 2013, at 3:33 PM, James Lay <jlay at slave-tothe-box.net> wrote:

> 206.169.145.206 80      tcp     HTTP::MD5       x.x.x.x 
> 
> What's this telling me?  Usually there's something like Invalid Cert or 
> something like that in the notice.log to tell me why it hit, just wasn't 
> seeing the reason here.  Thank you.


I *hate* that notice and it will be going away in the 2.2 release.  The notice type is HTTP::MD5 in that line.  It's a hold over from how I implemented file hashing originally back in 2007 or so.  It should never have made it into a Bro release (let alone two!).

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From seth at icir.org  Fri Mar 22 12:59:25 2013
From: seth at icir.org (Seth Hall)
Date: Fri, 22 Mar 2013 15:59:25 -0400
Subject: [Bro] Extracting Email Attachments
In-Reply-To: <CADBniYitmJbsJDxizCdVyGcH+5Md5oHYUr7QQ4U-S_eGzkgfCg@mail.gmail.com>
References: <CADBniYj-+Mmb2gF0-WbKs+SjCEq0DquWc-wUequfxZ5_qpF0vg@mail.gmail.com>
	<22DBDB28-B818-47B8-9BC9-BFB7B18C2E4C@gmail.com>
	<CADBniYitmJbsJDxizCdVyGcH+5Md5oHYUr7QQ4U-S_eGzkgfCg@mail.gmail.com>
Message-ID: <541D2AF6-506A-4BD2-A26F-4CF7B904EF89@icir.org>


On Mar 22, 2013, at 3:34 PM, Digital Ninja <dn1nj4 at gmail.com> wrote:

> redef SMTP::extract_file = T;
> redef SMTP::calc_md5 = T;

These two lines don't make any sense.  Those names aren't globals, they are fields of the SMTP::EntityInfo record type.  You can set them in certain situations but you shouldn't need to since the other lines you had should take care of what you're trying to do (once you have that little error that Bernhard pointed out fixed).

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From dn1nj4 at gmail.com  Fri Mar 22 12:59:27 2013
From: dn1nj4 at gmail.com (Digital Ninja)
Date: Fri, 22 Mar 2013 15:59:27 -0400
Subject: [Bro] Extracting Email Attachments
In-Reply-To: <A0F530FF-4914-45EC-A5E8-F1DF34DCB486@icsi.berkeley.edu>
References: <CADBniYj-+Mmb2gF0-WbKs+SjCEq0DquWc-wUequfxZ5_qpF0vg@mail.gmail.com>
	<22DBDB28-B818-47B8-9BC9-BFB7B18C2E4C@gmail.com>
	<CADBniYitmJbsJDxizCdVyGcH+5Md5oHYUr7QQ4U-S_eGzkgfCg@mail.gmail.com>
	<A0F530FF-4914-45EC-A5E8-F1DF34DCB486@icsi.berkeley.edu>
Message-ID: <CADBniYjyi7rZO4ozG9oEDf9cnNur9bpYEjYEUJn52qqmenUYYA@mail.gmail.com>

I tried both:
redef SMTP::extract_file_types += /application\/.*/;
and
redef SMTP::extract_file_types += /.*/;

But still end up with no attachments in /tmp, nor MD5s in the
smtp_entities.log.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130322/5eb3ed2f/attachment.html 

From jmellander at lbl.gov  Fri Mar 22 13:04:37 2013
From: jmellander at lbl.gov (Jim Mellander)
Date: Fri, 22 Mar 2013 13:04:37 -0700
Subject: [Bro] "Faking" connections and http records
Message-ID: <CADju=b6vuMkwCsFU0Dmwu3xxa2Ba7DnYu-wV6x71ACejhNm81w@mail.gmail.com>

Hi all:

I'm in the process of processing our syslogs for apache logs (which
will allow us visibility into ssl sessions into our webservers), and
am at the point where I am able to import the data into bro using the
input framework.  There's enough data to fill in most of a connection
record, and to fake other stuff.  What would be really cool would be
to create a connection record, and have it go thru the normal
processing, feed the http data in for processing via the standard http
processes, and close down the connection.  This would allow for
standard logging, and standard IDS processes to act upon this info.

Does anyone have suggestions on how to proceed with this?

Thanks in advance,

Jim Mellander
NERSC Cybersecurity
510-486-7204


From dn1nj4 at gmail.com  Fri Mar 22 13:08:53 2013
From: dn1nj4 at gmail.com (Digital Ninja)
Date: Fri, 22 Mar 2013 16:08:53 -0400
Subject: [Bro] Extracting Email Attachments
In-Reply-To: <CADBniYjyi7rZO4ozG9oEDf9cnNur9bpYEjYEUJn52qqmenUYYA@mail.gmail.com>
References: <CADBniYj-+Mmb2gF0-WbKs+SjCEq0DquWc-wUequfxZ5_qpF0vg@mail.gmail.com>
	<22DBDB28-B818-47B8-9BC9-BFB7B18C2E4C@gmail.com>
	<CADBniYitmJbsJDxizCdVyGcH+5Md5oHYUr7QQ4U-S_eGzkgfCg@mail.gmail.com>
	<A0F530FF-4914-45EC-A5E8-F1DF34DCB486@icsi.berkeley.edu>
	<CADBniYjyi7rZO4ozG9oEDf9cnNur9bpYEjYEUJn52qqmenUYYA@mail.gmail.com>
Message-ID: <CADBniYj9uMg1TFPAn_Mp9UfJPUNU6+-XGEOKQr9=ZgnZHTvTqQ@mail.gmail.com>

So apparently I was incorrect in thinking that local.bro was loading
automatically when running bro from the command line.  Including the
local.bro policy successfully extracted the attachments.

What it also told me was that these two lines:
redef SMTP::extract_file = T;
redef SMTP::calc_md5 = T;

Are not valid.  But poking around a little bit in entities.bro I found the
generate_md5 mime-types and redefined that in the local.bro file.

Thanks for the help all!





On Fri, Mar 22, 2013 at 3:59 PM, Digital Ninja <dn1nj4 at gmail.com> wrote:

> I tried both:
> redef SMTP::extract_file_types += /application\/.*/;
> and
> redef SMTP::extract_file_types += /.*/;
>
> But still end up with no attachments in /tmp, nor MD5s in the
> smtp_entities.log.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130322/f745c780/attachment.html 

From init.conf at gmail.com  Fri Mar 22 13:47:54 2013
From: init.conf at gmail.com (Aashish SHARMA)
Date: Fri, 22 Mar 2013 13:47:54 -0700
Subject: [Bro] Quick question
In-Reply-To: <EFDAAC61-A1D4-42D5-8E7A-9D7E3A41153D@icir.org>
References: <8f57bda070321c5e7650421d72cd7999@localhost>
	<EFDAAC61-A1D4-42D5-8E7A-9D7E3A41153D@icir.org>
Message-ID: <FFFA5A57-6CE7-40CE-932D-1746C3E0F3B7@gmail.com>

( A notice for binaries on the network is a subjective decision but I think it was not as bad an idea Seth. )

I think back in the day when md5 sum calculation was new feature in bro, this notice merely tells you the fact that binaries (and other mime types of choice)  traversing through the network and here is its md5

We found this useful in notice log was for post processing because:

1) Searches were much faster in notice log for known bad md5 compared to http.log due to smaller size of notice logs. 
2) Also this provided an easy search string  HTTP::MD5 (or SMTP:MD5) to grep on (just a preference) 
2) Post processing for HTTP::MD5 allowed us to make a white list of known binaries  that we serve from our network and flag if there is a new binary being hosted by us.  This was useful to find things like accidental shares open (eg. C:/ is word readable ) or if bad guys are using a webserver to host malware. 

Off course, now with input framework we can feed bad md5s into bro realtime etc, but still post-processing of some nature in useful and needed.  

Aashish 


On Mar 22, 2013, at 12:56 PM, Seth Hall <seth at icir.org> wrote:

> 
> On Mar 22, 2013, at 3:33 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> 
>> 206.169.145.206 80      tcp     HTTP::MD5       x.x.x.x 
>> 
>> What's this telling me?  Usually there's something like Invalid Cert or 
>> something like that in the notice.log to tell me why it hit, just wasn't 
>> seeing the reason here.  Thank you.
> 
> 
> I *hate* that notice and it will be going away in the 2.2 release.  The notice type is HTTP::MD5 in that line.  It's a hold over from how I implemented file hashing originally back in 2007 or so.  It should never have made it into a Bro release (let alone two!).
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From seth at icir.org  Fri Mar 22 13:54:29 2013
From: seth at icir.org (Seth Hall)
Date: Fri, 22 Mar 2013 16:54:29 -0400
Subject: [Bro] "Faking" connections and http records
In-Reply-To: <CADju=b6vuMkwCsFU0Dmwu3xxa2Ba7DnYu-wV6x71ACejhNm81w@mail.gmail.com>
References: <CADju=b6vuMkwCsFU0Dmwu3xxa2Ba7DnYu-wV6x71ACejhNm81w@mail.gmail.com>
Message-ID: <4B8A0C7E-A77E-4A2E-A11E-520E7537B437@icir.org>


On Mar 22, 2013, at 4:04 PM, Jim Mellander <jmellander at LBL.GOV> wrote:

> Does anyone have suggestions on how to proceed with this?


It wouldn't work very well. :)

Nearly all of the detections rely on the various http_ events.  I would go down a slightly different route with logs than I would with raw traffic.  This is something that I've been talking about for quite a while and I suspect something related to happen in the next year.

I think it's really cool that you're importing logs into Bro!

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From seth at icir.org  Fri Mar 22 13:56:50 2013
From: seth at icir.org (Seth Hall)
Date: Fri, 22 Mar 2013 16:56:50 -0400
Subject: [Bro] Quick question
In-Reply-To: <FFFA5A57-6CE7-40CE-932D-1746C3E0F3B7@gmail.com>
References: <8f57bda070321c5e7650421d72cd7999@localhost>
	<EFDAAC61-A1D4-42D5-8E7A-9D7E3A41153D@icir.org>
	<FFFA5A57-6CE7-40CE-932D-1746C3E0F3B7@gmail.com>
Message-ID: <75D3BD4F-180F-415A-BE9F-3DAB19B7B47B@icir.org>


On Mar 22, 2013, at 4:47 PM, Aashish SHARMA <init.conf at gmail.com> wrote:

> ( A notice for binaries on the network is a subjective decision but I think it was not as bad an idea Seth. )

I'm saying the implementation was bad on my part.  It just shouldn't be a notice. :)

> Off course, now with input framework we can feed bad md5s into bro realtime etc, but still post-processing of some nature in useful and needed.  


The file analysis framework is going to affect all of your points and generally I think it will make all of them nicer and more consistent.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From jmellander at lbl.gov  Fri Mar 22 14:18:29 2013
From: jmellander at lbl.gov (Jim Mellander)
Date: Fri, 22 Mar 2013 14:18:29 -0700
Subject: [Bro] "Faking" connections and http records
In-Reply-To: <4B8A0C7E-A77E-4A2E-A11E-520E7537B437@icir.org>
References: <CADju=b6vuMkwCsFU0Dmwu3xxa2Ba7DnYu-wV6x71ACejhNm81w@mail.gmail.com>
	<4B8A0C7E-A77E-4A2E-A11E-520E7537B437@icir.org>
Message-ID: <CADju=b6QSYpemr6h6NtYFQH65FqSziBsXipXxUAdzcWefJQrHQ@mail.gmail.com>

Well, its unfortunate that we can't feed in data from other sources
and subject it to the same policies that network traffic is subject
to.

In the meantime, I may just write some code that fakes the data into
pcap files that can be read by bro directly.

On Fri, Mar 22, 2013 at 1:54 PM, Seth Hall <seth at icir.org> wrote:
>
> On Mar 22, 2013, at 4:04 PM, Jim Mellander <jmellander at LBL.GOV> wrote:
>
>> Does anyone have suggestions on how to proceed with this?
>
>
> It wouldn't work very well. :)
>
> Nearly all of the detections rely on the various http_ events.  I would go down a slightly different route with logs than I would with raw traffic.  This is something that I've been talking about for quite a while and I suspect something related to happen in the next year.
>
> I think it's really cool that you're importing logs into Bro!
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>



From jlay at slave-tothe-box.net  Fri Mar 22 14:38:20 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 22 Mar 2013 15:38:20 -0600
Subject: [Bro] Quick question
In-Reply-To: <75D3BD4F-180F-415A-BE9F-3DAB19B7B47B@icir.org>
References: <8f57bda070321c5e7650421d72cd7999@localhost>
	<EFDAAC61-A1D4-42D5-8E7A-9D7E3A41153D@icir.org>
	<FFFA5A57-6CE7-40CE-932D-1746C3E0F3B7@gmail.com>
	<75D3BD4F-180F-415A-BE9F-3DAB19B7B47B@icir.org>
Message-ID: <e038db57047c8b30383df661cbd823f3@localhost>

On 2013-03-22 14:56, Seth Hall wrote:
> On Mar 22, 2013, at 4:47 PM, Aashish SHARMA <init.conf at gmail.com> 
> wrote:
>
>> ( A notice for binaries on the network is a subjective decision but 
>> I think it was not as bad an idea Seth. )
>
> I'm saying the implementation was bad on my part.  It just shouldn't
> be a notice. :)
>
>> Off course, now with input framework we can feed bad md5s into bro 
>> realtime etc, but still post-processing of some nature in useful and 
>> needed.
>
>
> The file analysis framework is going to affect all of your points and
> generally I think it will make all of them nicer and more consistent.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/


Thanks for the responses gents..helps me out :)

James


From tritium.cat at gmail.com  Fri Mar 22 18:53:03 2013
From: tritium.cat at gmail.com (Tritium Cat)
Date: Fri, 22 Mar 2013 18:53:03 -0700
Subject: [Bro] Detecting software components that do strange dns queries
In-Reply-To: <CAEjQA5K1T0jYfj4H5Xb8R8NgETKrbvCz9FKx_h-g4YQU6niVcw@mail.gmail.com>
References: <CAEjQA5LTJdDi3CFQwpwvqQZR6omkvCckpVRwxeTeGxUeFPfhsw@mail.gmail.com>
	<CAN4C-D=_tajwviFDGMjF6Q_QiJv1=V4baR77_71-zY0_JOV3hA@mail.gmail.com>
	<CAGZoo6H9PzdZBSA8hdcmSW2HaTQj1mzh+4Df5RRMN+_wn0pWJQ@mail.gmail.com>
	<8671_1363899017_r2LKoGto010910_CAMPgRd7N6tMbWqtE8ZewRse8Ks-jL=TiO2x=UNW4DxYs8qXZcg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E8552ACC0D4@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<CAEjQA5K1T0jYfj4H5Xb8R8NgETKrbvCz9FKx_h-g4YQU6niVcw@mail.gmail.com>
Message-ID: <CAMPgRd6gq7T6ma2V6B5+0MuSkXg14F_QU8O21XeX2RqYC5ufBg@mail.gmail.com>

Yes, thanks for the example and detail.

CFA was the first thing that crossed my mind so I googled for it and found
the Arxiv paper; it sounds promising to me but I can see your point about
the length.

While searching for supporting information I found old Google and Github
projects with some code inspired by the paper.   It appears someone forked
the original project but abandoned it after updating the README file.  :/

Readme:  https://code.google.com/p/dnapy/
Code:       https://github.com/gourryinverse/dnapy




On Fri, Mar 22, 2013 at 8:06 AM, C. L. Martinez <carlopmart at gmail.com>wrote:

> On Fri, Mar 22, 2013 at 2:06 PM, Vlad Grigorescu <vladg at cmu.edu> wrote:
> > You can do character frequency analysis with a simple Bro script. Look
> at <http://www.bro.org/documentation-git/scripts/base/strings.bif.html>
> to see the functions you can use for strings.
> >
> > I think that this is asking the wrong question, however. I'd be amazed
> if you could reliably determine "good" domains from "bad" domains based
> simply on character frequency analysis. Bro can calculate entropy for you: <
> http://www.bro.org/documentation/scripts/base/bro.bif.html#id-find_entropy>.
> That being said, I don't think entropy is the right answer either.
> >
> > Here are the entropy results (in no particular order) for the 4 domains
> you listed and for 4 very common domains (google.com, twitter.com,
> fbcdn.net and amazon.co.uk):
> >
> > [entropy=2.646439, chi_square=450.8, mean=100.2, monte_carlo_pi=4.0,
> serial_correlation=0.096875]
> > [entropy=3.085055, chi_square=400.538462, mean=104.692308,
> monte_carlo_pi=4.0, serial_correlation=-0.005991]
> > [entropy=3.095795, chi_square=338.090909, mean=106.727273,
> monte_carlo_pi=4.0, serial_correlation=0.062381]
> > [entropy=3.027169, chi_square=384.636364, mean=104.727273,
> monte_carlo_pi=4.0, serial_correlation=0.011643]
> > [entropy=3.182006, chi_square=424.857143, mean=105.5,
> monte_carlo_pi=4.0, serial_correlation=-0.050923]
> > [entropy=2.947703, chi_square=303.888889, mean=98.0, monte_carlo_pi=4.0,
> serial_correlation=-0.316796]
> > [entropy=3.084963, chi_square=372.0, mean=97.666667, monte_carlo_pi=4.0,
> serial_correlation=-0.248104]
> > [entropy=2.845351, chi_square=431.181818, mean=102.818182,
> monte_carlo_pi=4.0, serial_correlation=-0.322755]
> >
> > I don't know about you, but I can't tell which are good and which are
> bad. I suspect that DNS names are too short of a sample to provide any
> meaningful data.
> >
> > I think you should focus instead on the behavior that you're trying to
> detect. Looking at your example below, some alerts that'd be more useful
> might be:
> >
> >  - Too many NXDOMAIN queries.
> >  - A query that resolves to an ISC sinkhole.
> >  - Queries for a domain that no one else queried.
> >  - Repetitive queries every X seconds with little to no deviation.
> >  - Queries for a domain that you haven't seen before.
> >
> > Hope this helps,
> >
> >   --Vlad
> >
>
> Many many thanks Vlad for your explanation ... I'll think about it this
> weekend
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130322/6ab97177/attachment.html 

From mbower2 at gmail.com  Mon Mar 25 08:01:07 2013
From: mbower2 at gmail.com (Michael Bower)
Date: Mon, 25 Mar 2013 11:01:07 -0400
Subject: [Bro] Help with searching logs
Message-ID: <CACqtyMmP_75C50VbCzewwn8up=QEvnAVR7XpJOGwVRSL-PEgvQ@mail.gmail.com>

Im still learning, so bare with me.  I ran the following command:

bro-cut id.orig_h orig_bytes < conn.log             \
    | sort                                          \
    | awk '{ if (host != $1) {                      \
if (size != 0)                     \                     print $1,
size;                \                  host=$1;
   \                  size=0                            \
} else                                \                  size += $2
                    \            }
  \            END {                                   \
 if (size != 0)                      \                     print $1,
size                 \                }'
   \
    | sort -rnk 2                                     \
    | head -n 10


This worked well to show me the top 10 hosts (originators).  What Im trying
to do is show the top 10 hosts and the time (ts).  Maybe show
the resp_bytes field too, if that is possible.  Any help would be
greatly appreciated.

Thanks!
-- 

Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130325/31fea21a/attachment.html 

From dmandelb at bbn.com  Mon Mar 25 09:24:17 2013
From: dmandelb at bbn.com (David Mandelberg)
Date: Mon, 25 Mar 2013 12:24:17 -0400
Subject: [Bro] running some code on only one cluster node
Message-ID: <1364228657.14456.69.camel@titan>

Hi,

I'm working on a script that aggregates data into a &synchronized and
&mergeable table, then processes that table on a regular interval. The
code to process that table only needs to be run on one node. What's the
best way to do that?


Would something like this work?

@if (Cluster::local_node_type() in {Cluster::MANAGER, Cluster::NONE})
<code for only one node>
@endif


Or should I do something more like this?

const table_processor = "" &redef;
@if (!Cluster::is_enabled() || Cluster::node == table_processor)
<code for only one node>
@endif



From dmandelb at bbn.com  Mon Mar 25 10:51:03 2013
From: dmandelb at bbn.com (David Mandelberg)
Date: Mon, 25 Mar 2013 13:51:03 -0400
Subject: [Bro] nested &defaults
Message-ID: <1364233863.14456.78.camel@titan>

Hello again,

I have a record with a field like this:

type Histogram: record {
	histogram: vector of count;
};

I'd like histogram field to default to an empty vector and the
individual counts in the vector to default to 0. I.e., I want the below
code to print "42".

global foo: Histogram;
foo$histogram[5] += 42;
print(foo$histogram[5]);

Is that possible to set defaults for both $histogram and $histogram[N]?



From jsiwek at illinois.edu  Mon Mar 25 11:32:54 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 25 Mar 2013 18:32:54 +0000
Subject: [Bro] nested &defaults
In-Reply-To: <1364233863.14456.78.camel@titan>
References: <1364233863.14456.78.camel@titan>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D116F62966@CITESMBX2.ad.uillinois.edu>


On Mar 25, 2013, at 12:51 PM, David Mandelberg <dmandelb at bbn.com> wrote:

> I have a record with a field like this:
> 
> type Histogram: record {
> 	histogram: vector of count;
> };
> 
> I'd like histogram field to default to an empty vector and the
> individual counts in the vector to default to 0. I.e., I want the below
> code to print "42".
> 
> global foo: Histogram;
> foo$histogram[5] += 42;
> print(foo$histogram[5]);
> 
> Is that possible to set defaults for both $histogram and $histogram[N]?


I didn't think vectors supported a &default attribute very well, so I didn't try too hard to make that work, but here's what I came up with using a table indexed by counts and yielding counts (which is pretty much equivalent to a vector of count):

    type HistogramType: table[count] of count;

    function histogram_default(index: count): count
        {
        return 0;
        }

    const new_histogram: HistogramType = table() &default=histogram_default;

    type HistogramRecord: record {
        histogram: HistogramType &default=copy(new_histogram);
        other_stuff: string &optional;
    };

    global foo: HistogramRecord;
    global foo2: HistogramRecord;

    foo$histogram[5] += 42;
    print(foo$histogram[5]);
    foo2$histogram[5] += 13;
    print(foo2$histogram[5]);

The big trick there is that attributes don't currently propagate from types to values, so instead of setting a &default function on the HistogramType, I had to apply it to a constant, empty table and clone that for every new histogram instance.

- Jon


From dmandelb at bbn.com  Mon Mar 25 12:19:08 2013
From: dmandelb at bbn.com (David Mandelberg)
Date: Mon, 25 Mar 2013 15:19:08 -0400
Subject: [Bro] nested &defaults
In-Reply-To: <B3E27C90B7D2DD4CAEF09215A29EB0D116F62966@CITESMBX2.ad.uillinois.edu>
References: <1364233863.14456.78.camel@titan>
	<B3E27C90B7D2DD4CAEF09215A29EB0D116F62966@CITESMBX2.ad.uillinois.edu>
Message-ID: <1364239148.14456.80.camel@titan>

Thanks, that works! One question below, though:

On Mon, 2013-03-25 at 18:32 +0000, Siwek, Jonathan Luke wrote:
>     function histogram_default(index: count): count
>         {
>         return 0;
>         }
> 
>     const new_histogram: HistogramType = table() &default=histogram_default;

Why not just do &default=0 instead of histogram_default? That works for
me.



From jsiwek at illinois.edu  Mon Mar 25 12:39:23 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 25 Mar 2013 19:39:23 +0000
Subject: [Bro] nested &defaults
In-Reply-To: <1364239148.14456.80.camel@titan>
References: <1364233863.14456.78.camel@titan>
	<B3E27C90B7D2DD4CAEF09215A29EB0D116F62966@CITESMBX2.ad.uillinois.edu>
	<1364239148.14456.80.camel@titan>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D116F62B50@CITESMBX2.ad.uillinois.edu>


> Why not just do &default=0 instead of histogram_default? That works for
> me.

I was only playing around to verify what I could get to work.  Since you don't need the default value to be a function of the index, &default=0 is fine.

- Jon


From dmandelb at bbn.com  Mon Mar 25 12:57:43 2013
From: dmandelb at bbn.com (David Mandelberg)
Date: Mon, 25 Mar 2013 15:57:43 -0400
Subject: [Bro] statistical functions
Message-ID: <1364241463.14456.89.camel@titan>

Hi,

Is anybody interested in or planning to add statistical functions from
boost::math to the Bro scripting language?

I'm working on a script that uses the CDF and its complement (SF) for
normal and Poisson distributions to determine if the number of addresses
a host interacts with is anomalous, based on the host's past behavior. I
know it's possible to implement those functions in Bro scripting
language, but it seems like re-inventing the wheel since Boost already
has them.



From seth at icir.org  Wed Mar 27 21:13:09 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 28 Mar 2013 00:13:09 -0400
Subject: [Bro] running some code on only one cluster node
In-Reply-To: <1364228657.14456.69.camel@titan>
References: <1364228657.14456.69.camel@titan>
Message-ID: <2D5C424F-2B85-4F8E-A7AA-FC59D71A4670@icir.org>


On Mar 25, 2013, at 12:24 PM, David Mandelberg <dmandelb at bbn.com> wrote:

> @if (Cluster::local_node_type() in {Cluster::MANAGER, Cluster::NONE})
> <code for only one node>
> @endif


I tend toward this style.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From michael-haney at utulsa.edu  Thu Mar 28 23:53:18 2013
From: michael-haney at utulsa.edu (Michael Haney)
Date: Fri, 29 Mar 2013 01:53:18 -0500
Subject: [Bro] Modbus protocol event handler for Bro
Message-ID: <CAFTpQxgYnhTkww_L+Q1=WZo=J=u_eVhgdu-CUNQu3fQ=30Ltxg@mail.gmail.com>

I'm reviewing this paper and the related code for DNP3:
http://csiir.ornl.gov/csiirw/12/BPAwards/csiirw8Submission7.pdf

But I have a network I'm analyzing that has modbus over tcp and has
implemented things in a somewhat unorthodox way. They've used port
assignments as a means of categorizing subsets of systems, and a bit of
security by obscurity. So nothing is on the standard port 502. It's all
over the place on ranges of ports from 2100 to 9900.

Enter Bro and it's acclaimed ability to recognize protocols not by port
number but by semantics of the payload.

But has anyone done this for modbus yet?  Anyone interested to use it if I
start working on it? (read: volunteer beta tester and guinea pig).

What about other ICS/SCADA protocols?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130329/86e89b38/attachment.html 

From jlay at slave-tothe-box.net  Fri Mar 29 08:17:52 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 29 Mar 2013 09:17:52 -0600
Subject: [Bro] "bro-cut -d | grep" vs. "grep | bro-cut -d"
Message-ID: <fca7ca3b44a22fee8a201f8a1de6b143@localhost>

Topic (sorta) says it.  Example:

[08:49:21 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | grep light | 
bro-cut -d

[08:49:25 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | bro-cut -d | 
grep light
2013-03-28T20:42:09-0600        X8KFdodB5Ie     x.x.x.x    55051   
x.x.x.x    53      udp     43494   www.lighting.com      1       
C_INTERNET      1       A       0       NOERROR F       F       T       
T       0       x.x.x.x    3600.000000
[08:49:50 ids:~/broarchive/03-28-2013$]

I'd like to grep out the content before sending to bro-cut as it takes 
a fraction of the time (as shown above).  I've made sure that no 
colorization is happening.  Any hints on how I can get this to fly?  
Thank you.

James


From vladg at cmu.edu  Fri Mar 29 08:24:34 2013
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Fri, 29 Mar 2013 15:24:34 +0000
Subject: [Bro] "bro-cut -d | grep" vs. "grep | bro-cut -d"
In-Reply-To: <17341_1364570279_r2TFHv2a019759_fca7ca3b44a22fee8a201f8a1de6b143@localhost>
References: <17341_1364570279_r2TFHv2a019759_fca7ca3b44a22fee8a201f8a1de6b143@localhost>
Message-ID: <1202BE242E080642B0CD0AD0A03E8552AE4E26@PGH-MSGMB-03.andrew.ad.cmu.edu>

bro-cut relies on the header fields. You can use something like this:

> grep -E '(^#|light)'

  --Vlad


On Mar 29, 2013, at 11:17 AM, James Lay <jlay at slave-tothe-box.net>
 wrote:

> Topic (sorta) says it.  Example:
> 
> [08:49:21 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | grep light | 
> bro-cut -d
> 
> [08:49:25 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | bro-cut -d | 
> grep light
> 2013-03-28T20:42:09-0600        X8KFdodB5Ie     x.x.x.x    55051   
> x.x.x.x    53      udp     43494   www.lighting.com      1       
> C_INTERNET      1       A       0       NOERROR F       F       T       
> T       0       x.x.x.x    3600.000000
> [08:49:50 ids:~/broarchive/03-28-2013$]
> 
> I'd like to grep out the content before sending to bro-cut as it takes 
> a fraction of the time (as shown above).  I've made sure that no 
> colorization is happening.  Any hints on how I can get this to fly?  
> Thank you.
> 
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From jlay at slave-tothe-box.net  Fri Mar 29 08:48:33 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 29 Mar 2013 09:48:33 -0600
Subject: [Bro] "bro-cut -d | grep" vs. "grep | bro-cut -d"
In-Reply-To: <1202BE242E080642B0CD0AD0A03E8552AE4E26@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <17341_1364570279_r2TFHv2a019759_fca7ca3b44a22fee8a201f8a1de6b143@localhost>
	<1202BE242E080642B0CD0AD0A03E8552AE4E26@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <924f7fe8c8a9d0f47f2acd6095c2900f@localhost>

On 2013-03-29 09:24, Vlad Grigorescu wrote:
> bro-cut relies on the header fields. You can use something like this:
>
>> grep -E '(^#|light)'
>
>   --Vlad
>
>
> On Mar 29, 2013, at 11:17 AM, James Lay <jlay at slave-tothe-box.net>
>  wrote:
>
>> Topic (sorta) says it.  Example:
>>
>> [08:49:21 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | grep light 
>> |
>> bro-cut -d
>>
>> [08:49:25 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | bro-cut -d 
>> |
>> grep light
>> 2013-03-28T20:42:09-0600        X8KFdodB5Ie     x.x.x.x    55051
>> x.x.x.x    53      udp     43494   www.lighting.com      1
>> C_INTERNET      1       A       0       NOERROR F       F       T
>> T       0       x.x.x.x    3600.000000
>> [08:49:50 ids:~/broarchive/03-28-2013$]
>>
>> I'd like to grep out the content before sending to bro-cut as it 
>> takes
>> a fraction of the time (as shown above).  I've made sure that no
>> colorization is happening.  Any hints on how I can get this to fly?
>> Thank you.
>>
>> James

Thanks Vlad that does help.  I guess I should explain my usage as well. 
I see some event, and I go right to current and:

grep x.x.x.x *

and get lot's of good information, but no timestamp.  Interestingly, 
the below works:

cat * | egrep '(^#|x.x.x.x)' | /opt/bin/bro-cut -d

but the below script and execution does not (no results)
#!/bin/bash
#cleanbro
cat * | egrep '(^#|$1)' |  /opt/bin/bro-cut -d

./cleanbro x.x.x.x

I really just want to be able to look through all logs for the current 
day (rotation every 24 hours) and for the past days..I don't really just 
choose one log, I check them all.  I had my suspicions up front that the 
timestamp was going to be an issue.  Devs, any chance you can get an 
option in for human readable timestamps IN logs?  Thanks.

James


From jessebowling at gmail.com  Fri Mar 29 08:50:54 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Fri, 29 Mar 2013 11:50:54 -0400
Subject: [Bro] "bro-cut -d | grep" vs. "grep | bro-cut -d"
In-Reply-To: <fca7ca3b44a22fee8a201f8a1de6b143@localhost>
References: <fca7ca3b44a22fee8a201f8a1de6b143@localhost>
Message-ID: <CAGFhEnmK+V27G_zD07F1hANv+cZABydiYu7_UmoYfZkfQdC4Gg@mail.gmail.com>

Hi James,

I asked a similar question under a subject like "Feature request; up to 50%
done?" and got this answer from Seth, which solves some of the problems I
think you're trying to solve...

Cheers,

Jesse

On Feb 11, 2013, at 4:17 PM, Jesse Bowling <jessebowling at gmail.com> wrote:

> So, I suppose I'm requesting that someone with more gawk chops than
myself give a shot at integrating this into bro-cut

I tend to use these lines in my profile...

alias bro-column="sed \"s/fields.//;s/types.//\" | column -s $'\t' -t"
alias bro-awk='awk -F"  "'
bro-grep() { grep -E "(^#)|$1" $2; }
bro-zgrep() { zgrep -E "(^#)|$1" $2; }

What you're trying to do can then be accomplished like this?

bro-zgrep '10.10.10.10' /usr/local/bro/logs/conn.*.log.gz | bro-cut
id.orig_h,id.resp_h

It *would* be handy to be able to do this through bro-cut though but that
would make bro-cut start to sound like an incorrectly named utility. :)

Have you tried using the ElasticSearch writer and Brownian?

  .Seth


On Fri, Mar 29, 2013 at 11:17 AM, James Lay <jlay at slave-tothe-box.net>wrote:

> Topic (sorta) says it.  Example:
>
> [08:49:21 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | grep light |
> bro-cut -d
>
> [08:49:25 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | bro-cut -d |
> grep light
> 2013-03-28T20:42:09-0600        X8KFdodB5Ie     x.x.x.x    55051
> x.x.x.x    53      udp     43494   www.lighting.com      1
> C_INTERNET      1       A       0       NOERROR F       F       T
> T       0       x.x.x.x    3600.000000
> [08:49:50 ids:~/broarchive/03-28-2013$]
>
> I'd like to grep out the content before sending to bro-cut as it takes
> a fraction of the time (as shown above).  I've made sure that no
> colorization is happening.  Any hints on how I can get this to fly?
> Thank you.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130329/980688e5/attachment.html 

From jlay at slave-tothe-box.net  Fri Mar 29 09:34:46 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Fri, 29 Mar 2013 10:34:46 -0600
Subject: [Bro] "bro-cut -d | grep" vs. "grep | bro-cut -d"
In-Reply-To: <CAGFhEnmK+V27G_zD07F1hANv+cZABydiYu7_UmoYfZkfQdC4Gg@mail.gmail.com>
References: <fca7ca3b44a22fee8a201f8a1de6b143@localhost>
	<CAGFhEnmK+V27G_zD07F1hANv+cZABydiYu7_UmoYfZkfQdC4Gg@mail.gmail.com>
Message-ID: <46aeee6a15da89e1e71bef28a456bc68@localhost>

On 2013-03-29 09:50, Jesse Bowling wrote:
> Hi James,
>
> I asked a similar question under a subject like "Feature request; up
> to 50% done?" and got this answer from Seth, which solves some of the
> problems I think youre trying to solve...
>
> Cheers,
>
> Jesse
>
> On Feb 11, 2013, at 4:17 PM, Jesse Bowling <jessebowling at gmail.com
> [4]> wrote:
>
>  > So, I suppose Im requesting that someone with more gawk chops than
> myself give a shot at integrating this into bro-cut
>
>  I tend to use these lines in my profile...
>
>  alias bro-column="sed "s/fields.//;s/types.//" | column -s $t -t"
>  alias bro-awk=awk -F" ?"
>  bro-grep() { grep -E "(^#)|$1" $2; }
>  bro-zgrep() { zgrep -E "(^#)|$1" $2; }
>
>  What youre trying to do can then be accomplished like this?
>
>  bro-zgrep 10.10.10.10 /usr/local/bro/logs/conn.*.log.gz | bro-cut
> id.orig_h,id.resp_h
>
>  It *would* be handy to be able to do this through bro-cut though but
> that would make bro-cut start to sound like an incorrectly named
> utility. :)
>
>  Have you tried using the ElasticSearch writer and Brownian?
>
>  ? .Seth


Thanks for the repost Jess, that helps :)

James


From jessebowling at gmail.com  Fri Mar 29 09:44:05 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Fri, 29 Mar 2013 12:44:05 -0400
Subject: [Bro] "bro-cut -d | grep" vs. "grep | bro-cut -d"
In-Reply-To: <924f7fe8c8a9d0f47f2acd6095c2900f@localhost>
References: <17341_1364570279_r2TFHv2a019759_fca7ca3b44a22fee8a201f8a1de6b143@localhost>
	<1202BE242E080642B0CD0AD0A03E8552AE4E26@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<924f7fe8c8a9d0f47f2acd6095c2900f@localhost>
Message-ID: <CAGFhEn=qkdo0mJgH-SMeF2tdA90-289Bdyz1ArhnBuzCZEvhNQ@mail.gmail.com>

I've also dropped this in an executable script and piped my output to it
(naturally it assumes the timestamp is the first field):

#!/bin/gawk
{
val=strftime("%Y-%m-%dT%H:%M:%S%z", $1, 0)
$1=val
print $0
}


Cheers,

Jesse

On Fri, Mar 29, 2013 at 11:48 AM, James Lay <jlay at slave-tothe-box.net>wrote:

> On 2013-03-29 09:24, Vlad Grigorescu wrote:
> > bro-cut relies on the header fields. You can use something like this:
> >
> >> grep -E '(^#|light)'
> >
> >   --Vlad
> >
> >
> > On Mar 29, 2013, at 11:17 AM, James Lay <jlay at slave-tothe-box.net>
> >  wrote:
> >
> >> Topic (sorta) says it.  Example:
> >>
> >> [08:49:21 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | grep light
> >> |
> >> bro-cut -d
> >>
> >> [08:49:25 ids:~/broarchive/03-28-2013$] zcat dns.log.gz | bro-cut -d
> >> |
> >> grep light
> >> 2013-03-28T20:42:09-0600        X8KFdodB5Ie     x.x.x.x    55051
> >> x.x.x.x    53      udp     43494   www.lighting.com      1
> >> C_INTERNET      1       A       0       NOERROR F       F       T
> >> T       0       x.x.x.x    3600.000000
> >> [08:49:50 ids:~/broarchive/03-28-2013$]
> >>
> >> I'd like to grep out the content before sending to bro-cut as it
> >> takes
> >> a fraction of the time (as shown above).  I've made sure that no
> >> colorization is happening.  Any hints on how I can get this to fly?
> >> Thank you.
> >>
> >> James
>
> Thanks Vlad that does help.  I guess I should explain my usage as well.
> I see some event, and I go right to current and:
>
> grep x.x.x.x *
>
> and get lot's of good information, but no timestamp.  Interestingly,
> the below works:
>
> cat * | egrep '(^#|x.x.x.x)' | /opt/bin/bro-cut -d
>
> but the below script and execution does not (no results)
> #!/bin/bash
> #cleanbro
> cat * | egrep '(^#|$1)' |  /opt/bin/bro-cut -d
>
> ./cleanbro x.x.x.x
>
> I really just want to be able to look through all logs for the current
> day (rotation every 24 hours) and for the past days..I don't really just
> choose one log, I check them all.  I had my suspicions up front that the
> timestamp was going to be an issue.  Devs, any chance you can get an
> option in for human readable timestamps IN logs?  Thanks.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130329/5c88ca25/attachment.html 

From michael-haney at utulsa.edu  Fri Mar 29 10:49:42 2013
From: michael-haney at utulsa.edu (Michael Haney)
Date: Fri, 29 Mar 2013 12:49:42 -0500
Subject: [Bro] Modbus protocol event handler for Bro
In-Reply-To: <CAKq214mS6tSgUpnunts2v_HJAaOgR7f8qLb-ZTRzEPXC2a-Gfg@mail.gmail.com>
References: <35b4251d91604effa68d54c235e05f9e@CITESHT2.ad.uillinois.edu>
	<CAKq214mS6tSgUpnunts2v_HJAaOgR7f8qLb-ZTRzEPXC2a-Gfg@mail.gmail.com>
Message-ID: <CAFTpQxgN=cP4pWoU7a73bH+2T9vctRe06Rj8j8TybvigKDNVgA@mail.gmail.com>

I appreciate you guys getting back to me, and I can also tell you that
there is other interest in this out there based on off-list responses.

So, yes, Dynamic Protocol Detection (DPD) is the acronym that didn't come
to me last night. I'm relatively new to Bro, and unfortunately only have a
cursory knowledge of the SCADA protocols I need to know about.  But I do
have access to about a terabyte of recorded pcaps of a mix of things on a
production network, but most of it is SCADA related and primarily in
Modbus, RockLink, and a couple other vendor versions of protocols. But as I
said, non-standard ports so port triggering is probably not what I'm
looking for (or we're going to have a list of about 5000 different ports to
trigger on).

As I understand it, there are a couple of methods to use for handling
protocol anomalies and detecting malicious activities. One is to look at
the protocol specs and see if you can identify a packet as valid or invalid
for that protocol.  Invalid packets show signs of fuzzing,
misconfiguration, or malicious activity.  Then there is identifying the
"normal" traffic. Like matching one response to one request. If you get
unsolicited "responses" for example, it may be a replay attack,
misconfiguration, or other malicious activity.

Then the third (final?) step is to look at the actual legitimate traffic
that matches up with the protocol and everything, but determine if it's a
malicious attempt to extract system information or sabotage critical
infrastructure, by an insider for example, or by a malicious outsider who
knows how to talk Modbus but is sending commands that are outside of normal
operation. Lots of work has been done in that area to see if we can
validate normal operation parameters and out-of-norm activities. The
problem is, we can't automate it on this dataset unless we can first
identify an arbitrary packet as being a modbus packet or not.

Have I got that breakdown right? Is this similar to what others are looking
for?

So I'm excited about being a brogrammer and I think this is as good a place
for me to jump in as any. But I'm also very glad to hear more about the
work that has already been done (and I don't have to take on such a task
alone from scratch).  I'd like to offer myself as your beta tester/guinea
pig. But there will be some minor hurdles to overcome with data sharing.
Technically, our research grant has expired (ended in December) and I need
to get permission from the corporate entity to continue to use their data
for our testing, as well as carve out time from work and school duties to
focus on this.  But assuming all that works out, I look forward to helping
out with this development effort.

Regards,
Michael

On Fri, Mar 29, 2013 at 9:08 AM, Hugo <hugolin615 at gmail.com> wrote:

>
>
> On Fri, Mar 29, 2013 at 1:53 AM, Michael Haney <michael-haney at utulsa.edu>wrote:
>
>>  I'm reviewing this paper and the related code for DNP3:
>> http://csiir.ornl.gov/csiirw/12/BPAwards/csiirw8Submission7.pdf
>>
>> But I have a network I'm analyzing that has modbus over tcp and has
>> implemented things in a somewhat unorthodox way. They've used port
>> assignments as a means of categorizing subsets of systems, and a bit of
>> security by obscurity. So nothing is on the standard port 502. It's all
>> over the place on ranges of ports from 2100 to 9900.
>>
>> Enter Bro and it's acclaimed ability to recognize protocols not by port
>> number but by semantics of the payload.
>>
>
> Just FYI, Bro has three ways to activate a new analyzer.
> http://www.bro.org/development/dpd.html
>
> Analyzers can use one of three ways to be fed new connections:
>
>    - Use a preconfigured set of ports, thus triggering on all connections
>    using any of the registered ports.
>    - Use content signatures, thus triggering on all connections that
>    match the relevant signatures.
>    - Hard-code to trigger on all connections, when signatures won?t cut
>    it and the protocol uses arbitrary ports. This should be avoided whenever
>    possible obviously.
>
>
>
>>
>> But has anyone done this for modbus yet?  Anyone interested to use it if
>> I start working on it? (read: volunteer beta tester and guinea pig).
>>
>
> I  believe that Modbus analyzer is already included in Bro master branch
> for a while now. So you can directly use it. You are more than welcome to
> use it.
>
>
>>
>> What about other ICS/SCADA protocols?
>>
>
> I am mainly responsible for the DNP3 analyzer. Due to some legacy issues,
> I spent a comparatively long period to include it in Bro. I actually
> finished almost everything just yesterday (still working on some comments
> to people who want to understand my code). If you want to use it, please
> let me know, I can direct you to download from one of the branch that we
> about to include in Master soon.
>
>
> Best,
>
> Hugo
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130329/a7e76163/attachment.html 

From seth at icir.org  Fri Mar 29 17:51:15 2013
From: seth at icir.org (Seth Hall)
Date: Fri, 29 Mar 2013 20:51:15 -0400
Subject: [Bro] Modbus protocol event handler for Bro
In-Reply-To: <CAFTpQxgYnhTkww_L+Q1=WZo=J=u_eVhgdu-CUNQu3fQ=30Ltxg@mail.gmail.com>
References: <CAFTpQxgYnhTkww_L+Q1=WZo=J=u_eVhgdu-CUNQu3fQ=30Ltxg@mail.gmail.com>
Message-ID: <9F746B5F-9113-494C-874E-CDD312F6AC9A@icir.org>


On Mar 29, 2013, at 2:53 AM, Michael Haney <michael-haney at utulsa.edu> wrote:

> But I have a network I'm analyzing that has modbus over tcp and has implemented things in a somewhat unorthodox way. They've used port assignments as a means of categorizing subsets of systems, and a bit of security by obscurity. So nothing is on the standard port 502. It's all over the place on ranges of ports from 2100 to 9900.

When I was reviewing and preparing the modbus analyzer to be merged I didn't create signatures for DPD because modbus doesn't have a very clear structure to identify.  I'll file a ticket now to come back around before the release and try to make a signature for identifying modbus.

Regardless, you will always be able to define ports that the analyzer is always used on.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/