From seth at icir.org  Wed Aug  1 06:57:52 2012
From: seth at icir.org (Seth Hall)
Date: Wed, 1 Aug 2012 09:57:52 -0400
Subject: [Bro] Bro Exchange weather warning
In-Reply-To: <21DD7C64179C9843B756C6DD491634DB252426DA@Mailbox1.boco.co.boulder.co.us>
References: <21DD7C64179C9843B756C6DD491634DB252426DA@Mailbox1.boco.co.boulder.co.us>
Message-ID: <C0BC92CE-BF5C-4E71-A13E-66B2753F0BE1@icir.org>


On Jul 31, 2012, at 12:36 PM, "Castle, Shane" <scastle at bouldercounty.org> wrote:

> Boulder will not be so extreme but it's possible for it to get quite chilly at night. So be warned!


Thanks for letting us how the local weather looks!

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From robin at icir.org  Wed Aug  1 10:50:17 2012
From: robin at icir.org (Robin Sommer)
Date: Wed, 1 Aug 2012 10:50:17 -0700
Subject: [Bro] Bro 2.1 Public Beta
Message-ID: <20120801175017.GQ45936@icir.org>


We are happy to announce a public beta of Bro 2.1, see the blog
posting for more information:

    http://blog.bro-ids.org/2012/08/bro-21-public-beta.html

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From christopher.p.crawford at gmail.com  Wed Aug  1 12:27:49 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Wed, 1 Aug 2012 15:27:49 -0400
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CAMPgRd5us7tC=tZoaRkG8Q+ZeQQt7Vip-SStR7J1=dYbPLdO6A@mail.gmail.com>
References: <CAMPgRd5us7tC=tZoaRkG8Q+ZeQQt7Vip-SStR7J1=dYbPLdO6A@mail.gmail.com>
Message-ID: <CAG1VgOYrXeHrayWPGcDBF+rmYv1wFkszqG9cg5ymMGRvcWJxMQ@mail.gmail.com>

Have you seen any of my threads from earlier this year?

http://bit.ly/JJQVVf
http://bit.ly/N2l4yT

Your issue sounds similar to what I was experiencing.

Bro 2.0 is routinely uses up all available memory and then crashes for me.

In my case, an early suggestion was that Bro should not be run in a
virtual machine.  I set up a second instance of Bro 2.0 on a FreeBSD
machine (not a VM), though, and got the same results -- routine
crashes.  I read quite a few stack traces from those crashes, and
noticed that there seemed to be an issue (maybe a leak) allocating
memory when Bro attempts to reassemble fragmented traffic.

Can you get a stack trace from any of your crashes?

I have a cron job that restarts everything as soon as it experiences a
crash, so I can get fairly continuous coverage.  Unfortunately, seeing
bro crash so frequently reduces my confidence that it's catching
everything.

If I want to be more confident about the results bro produces, I'll
run it over pcap from tcpdump.

-Chris

On Mon, Jul 30, 2012 at 4:57 PM, Tritium Cat <tritium.cat at gmail.com> wrote:
> Hello,
>
> I'm using the latest development build 2.0-907.
>
> The deployment consists of six servers; one as a manager and the other five
> as nodes.  Each node runs 20 workers and 2 proxies.  The manager is FreeBSD;
> the workers are Linux with PF_RING transparent_mode=2.
>
> After starting bro, the manger continually consumes memory until system
> exhaustion (64GB).  The CPU usage is high as well.
>
> Another problem is over 50% of the workers consume 100% CPU.  This is very
> odd considering the low volume traffic between 400-1000 Mbps per node.
>
> Where do you suggest I start debugging this ?
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From tritium.cat at gmail.com  Wed Aug  1 13:00:59 2012
From: tritium.cat at gmail.com (Tritium Cat)
Date: Wed, 1 Aug 2012 20:00:59 +0000
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CC3DFAC0.5976%vladg@andrew.cmu.edu>
References: <6730_1343778612_q6VNoAio021810_CAMPgRd6BctXTp9BHP96NaEkhuUykeyAq2DtbG=V_ftObHc4erg@mail.gmail.com>
	<CC3DFAC0.5976%vladg@andrew.cmu.edu>
Message-ID: <CAMPgRd68HYui4O4LWBngYdvLMph-r0Y4H0bffugaz1Cm3281QA@mail.gmail.com>

On Wed, Aug 1, 2012 at 1:23 AM, Vlad Grigorescu <vladg at cmu.edu> wrote:

> > I based the design on the very rough suggestion each core will handle
> >60-120 Mbps of traffic.  So 20 workers on a 48 core would be 1.2 Gbps to
> >2.4 Gbps.  I want to say I even recall reading about that threshold in a
> >paper somewhere.
>
> As you mentioned, that's a very rough estimate. There are several factors
> in play here, including:
>
> 1) What you're loading into Bro,
> 2) What your traffic looks like,
> 3) Resource contention,
> 4) Disk I/O,
> 5) Tuning some script-level optimizations.
>
> I think that what you're running into is resource contention - I believe
> that the 80-150 Mbps/core guideline is based on empirical evidence with
> 4-8 workers/system. AFAIK, running more than 12 or so workers/system is
> venturing into not-well-knonwn territory (I know of one setup that start
> experiencing stability and performance issues moving past 12
> workers/system). I also don't think that that guideline is really linear -
> I've analyzed ~1 Gbps with PF_RING and 8 workers (albeit without any
> scripts that do MD5 hashing).
>
> Additionally, once you start talking about so many workers, I think that
> your disk I/O on the manager will be a real issue - that's simply a lot of
> data to write out to disk.
>
>
You might be right.  I had considered disk I/O and ran the manager on a
decent raid-10 array (the elsa server), disk i/o did not appear to be the
problem so much as CPU and memory.  That observation carried over to
development builds with a threaded manager, but at an accelerated rate;
right now I'm watching the disk I/O under threshold and the manager is
consuming 100M of memory per second until memory exhaustion.

My configuration is default with Conn::Log turned off.  I'm still trying to
obtain an idea of the performance baseline before adding additional scripts.

For my current setup it seems to work best if I segment the cluster so I'll
continue with that approach.

--TC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120801/57435107/attachment.html 

From scampbell at lbl.gov  Wed Aug  1 13:59:09 2012
From: scampbell at lbl.gov (Scott Campbell)
Date: Wed, 01 Aug 2012 15:59:09 -0500
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CAMPgRd68HYui4O4LWBngYdvLMph-r0Y4H0bffugaz1Cm3281QA@mail.gmail.com>
References: <6730_1343778612_q6VNoAio021810_CAMPgRd6BctXTp9BHP96NaEkhuUykeyAq2DtbG=V_ftObHc4erg@mail.gmail.com>
	<CC3DFAC0.5976%vladg@andrew.cmu.edu>
	<CAMPgRd68HYui4O4LWBngYdvLMph-r0Y4H0bffugaz1Cm3281QA@mail.gmail.com>
Message-ID: <5019989D.5060902@lbl.gov>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have noticed that there seems to be a large volume of "soft"
information for high performance bro installations.  This is
particularly true with PF_RING/DNA/libzero configuration and use.
Finding nice simple examples are also a bit scarce.

My proposal to address this is as follows: For anybody willing to
document this info - something as simple as a cut and paste of the
PF_RING/ixgbe startup configs and the node.cfg - I will purchase a
beer*.  We will put this info up on the Bro web site for the
edification of those staring into the abyss of 10 (or 100!) G.

That is it.  This project is personal and not funded or supported by
ICSI or anybody else for that matter.  Can you imagine the IRB?

cheers!
scott

* Till I run out of money dedicated to the Bro documentation liquidity
fund.  :-)

On 8/1/12 3:00 PM, Tritium Cat wrote:
> On Wed, Aug 1, 2012 at 1:23 AM, Vlad Grigorescu <vladg at cmu.edu 
> <mailto:vladg at cmu.edu>> wrote:
> 
>> I based the design on the very rough suggestion each core will
>> handle 60-120 Mbps of traffic.  So 20 workers on a 48 core would
>> be 1.2
> Gbps to
>> 2.4 Gbps.  I want to say I even recall reading about that
>> threshold
> in a
>> paper somewhere.
> 
> As you mentioned, that's a very rough estimate. There are several 
> factors in play here, including:
> 
> 1) What you're loading into Bro, 2) What your traffic looks like, 
> 3) Resource contention, 4) Disk I/O, 5) Tuning some script-level
> optimizations.
> 
> I think that what you're running into is resource contention - I
> believe that the 80-150 Mbps/core guideline is based on empirical
> evidence with 4-8 workers/system. AFAIK, running more than 12 or so
> workers/system is venturing into not-well-knonwn territory (I know
> of one setup that start experiencing stability and performance
> issues moving past 12 workers/system). I also don't think that that
> guideline is really linear - I've analyzed ~1 Gbps with PF_RING and
> 8 workers (albeit without any scripts that do MD5 hashing).
> 
> Additionally, once you start talking about so many workers, I think
> that your disk I/O on the manager will be a real issue - that's
> simply a lot of data to write out to disk.
> 
> 
> You might be right.  I had considered disk I/O and ran the manager
> on a decent raid-10 array (the elsa server), disk i/o did not
> appear to be the problem so much as CPU and memory.  That
> observation carried over to development builds with a threaded
> manager, but at an accelerated rate; right now I'm watching the
> disk I/O under threshold and the manager is consuming 100M of
> memory per second until memory exhaustion.
> 
> My configuration is default with Conn::Log turned off.  I'm still
> trying to obtain an idea of the performance baseline before adding
> additional scripts.
> 
> For my current setup it seems to work best if I segment the cluster
> so I'll continue with that approach.
> 
> --TC
> 
> 
> 
> _______________________________________________ Bro mailing list 
> bro at bro-ids.org 
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFQGZidK2Plq8B7ZBwRAm3fAKCeVvu8jsSW3B5qmbcExWOWqA55UQCfaMhJ
dMvW/6xUyqnS2TxWzvjDGoU=
=hRud
-----END PGP SIGNATURE-----


From mcholste at gmail.com  Wed Aug  1 14:09:27 2012
From: mcholste at gmail.com (Martin Holste)
Date: Wed, 1 Aug 2012 16:09:27 -0500
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CAMPgRd68HYui4O4LWBngYdvLMph-r0Y4H0bffugaz1Cm3281QA@mail.gmail.com>
References: <6730_1343778612_q6VNoAio021810_CAMPgRd6BctXTp9BHP96NaEkhuUykeyAq2DtbG=V_ftObHc4erg@mail.gmail.com>
	<CC3DFAC0.5976%vladg@andrew.cmu.edu>
	<CAMPgRd68HYui4O4LWBngYdvLMph-r0Y4H0bffugaz1Cm3281QA@mail.gmail.com>
Message-ID: <CANpnLHgJqK2otX5kk1V19h5t+R1ieT+bnpowUF9OPmX0BzNqQQ@mail.gmail.com>

> You might be right.  I had considered disk I/O and ran the manager on a
> decent raid-10 array (the elsa server), disk i/o did not appear to be the
> problem so much as CPU and memory.  That observation carried over to
> development builds with a threaded manager, but at an accelerated rate;
> right now I'm watching the disk I/O under threshold and the manager is
> consuming 100M of memory per second until memory exhaustion.

How much I/O from the manager have you seen thus far?  I'm not yet
convinced that writing raw text files is the bottleneck.  When you
think about writing raw pcap, it's orders of magnitude more MB/sec to
disk than logging.


From vladg at cmu.edu  Wed Aug  1 14:10:46 2012
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Wed, 1 Aug 2012 21:10:46 +0000
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CC3F11EC.5ADA%vladg@andrew.cmu.edu>
Message-ID: <CC3F1382.5AE7%vladg@andrew.cmu.edu>

On 8/1/12 4:59 PM, "Scott Campbell" <scampbell at lbl.gov> wrote:

> ? something as simple as a cut and paste of the
> PF_RING/ixgbe startup configs and the node.cfg


Perhaps I'm misunderstanding what you're looking for, but is this still
needed with the new load balancing support in broctl?

>From <http://git.bro-ids.org/broctl.git/blob/HEAD:/CHANGES>:

* Improvements to built-in load-balancing support. Instead of adding
  a separate worker entry in node.cfg for each Bro worker process on
  each worker host, it is now possible to just specify the number of
  worker processes on each host. (Daniel Thayer)

  This change adds three new keywords to the node.cfg file (to be
  used with worker entries): lb_procs (specifies number of workers
  on a host), lb_method (specifies what type of load balancing to
  use: pf_ring, myricom, or interfaces), and lb_interfaces (used
  only with "lb_method=interfaces" to specify which interfaces to
  load-balance on).

  Two new broctl plugins (which operate automatically and the user
  doesn't need to be aware of them) are added to set the appropriate
  environment variables when either PF_RING or myricom
  load-balancing is being used.



What might be interesting is to document some use cases on the website.
e.g. a university is looking at 8 Gbps on 5 boxes with 8 workers each,
using Myricom NICs, and a Gigamon Whatever-42 frontend.


  --Vlad





From tritium.cat at gmail.com  Wed Aug  1 14:31:46 2012
From: tritium.cat at gmail.com (Tritium Cat)
Date: Wed, 1 Aug 2012 21:31:46 +0000
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <5019989D.5060902@lbl.gov>
References: <6730_1343778612_q6VNoAio021810_CAMPgRd6BctXTp9BHP96NaEkhuUykeyAq2DtbG=V_ftObHc4erg@mail.gmail.com>
	<CC3DFAC0.5976%vladg@andrew.cmu.edu>
	<CAMPgRd68HYui4O4LWBngYdvLMph-r0Y4H0bffugaz1Cm3281QA@mail.gmail.com>
	<5019989D.5060902@lbl.gov>
Message-ID: <CAMPgRd4NUcKB1C8C_MPGj8bZk+fSLTf+XSeTY_ofz7twVypukw@mail.gmail.com>

On Wed, Aug 1, 2012 at 8:59 PM, Scott Campbell <scampbell at lbl.gov> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I have noticed that there seems to be a large volume of "soft"
> information for high performance bro installations.  This is
> particularly true with PF_RING/DNA/libzero configuration and use.
> Finding nice simple examples are also a bit scarce.
>
> My proposal to address this is as follows: For anybody willing to
> document this info - something as simple as a cut and paste of the
> PF_RING/ixgbe startup configs and the node.cfg - I will purchase a
> beer*.  We will put this info up on the Bro web site for the
> edification of those staring into the abyss of 10 (or 100!) G.
>
> That is it.  This project is personal and not funded or supported by
> ICSI or anybody else for that matter.  Can you imagine the IRB?
>
> cheers!
> scott
>
> * Till I run out of money dedicated to the Bro documentation liquidity
> fund.  :-)
>
>

Here here.  This configures the IXGBE card at boot.


#!/bin/sh
################################
# bro_setup_capture.sh
#
#
# Load Intel Driver - allow_any_sfp patch applied to code
#                       to use cheap Finistar SFPs.  see [1] below.
#
#
#       [1] http://lasdkfjslkdf
#
rmmod ixgbe
modprobe ixgbe allow_any_sfp=1


# Turn on PF_RING
#       Have to recompile and reinstall after upgrading kernel.
#
rmmod pf_ring
modprobe pf_ring transparent_mode=2 enable_tx_capture=0


# Adjust interface features
#
ethtool -K eth5 rx off
ethtool -K eth5 tx off
ethtool -K eth5 sg off
ethtool -K eth5 tso off
#ethtool -K eth5 ufo off
ethtool -K eth5 gso off
ethtool -K eth5 gro off
ethtool -K eth5 lro off
ethtool -K eth5 rxvlan off
ethtool -K eth5 txvlan off
ethtool -K eth5 ntuple on
#
ethtool -s eth5 speed 10000 duplex full
ifconfig eth5 mtu 9600
ifconfig eth5 up

# Start bro, first check config.
#
#/usr/local/3rd-party/bin/broctl check
#/usr/local/3rd-party/bin/broctl start


This sets the capabilities after installing bro.


#!/bin/sh
#######################
# bro_set_capabilities.sh
#
# - After installing bro and setting ownership to bro, the
# capabilities need to be modified to allow packet capture.
#
# Make sure to edit lib/broctl/BroControl/install.py and adjust
# the 'Syncs' array: set 'bindir' and 'libdir' to False to avoid
# overwriting the binaries below.  This also helps when running
# the manager on a different platform (freebsd vs linux).
#
setcap cap_net_raw,cap_net_admin=eip /usr/local/3rd-party/bro/bin/bro
setcap cap_net_raw,cap_net_admin=eip /usr/local/3rd-party/bro/bin/capstats
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120801/ced8fe2e/attachment.html 

From scampbell at lbl.gov  Wed Aug  1 14:33:24 2012
From: scampbell at lbl.gov (Scott Campbell)
Date: Wed, 01 Aug 2012 16:33:24 -0500
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CC3F11EC.5ADA%vladg@andrew.cmu.edu>
References: <CC3F11EC.5ADA%vladg@andrew.cmu.edu>
Message-ID: <5019A0A4.9070807@lbl.gov>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This is not a particularly well thought out venture, but the intention
is to lower the pain associated with the process through example.
With this new functionality, bro itself is is well placed for
clustering, but the systems side - like OS tuning, driver
configuration, traffic distribution etc etc still could be much better
documented.

I completely agree that use cases are probably the best way to go
about conveying this information.  See you there -

cheers,
scott

On 8/1/12 4:08 PM, Vlad Grigorescu wrote:
> On 8/1/12 4:59 PM, "Scott Campbell" <scampbell at lbl.gov> wrote:
> 
>> ? something as simple as a cut and paste of the PF_RING/ixgbe
>> startup configs and the node.cfg
> 
> 
> Perhaps I'm misunderstanding what you're looking for, but is this
> still needed with the new load balancing support in broctl?
> 
> From <http://git.bro-ids.org/broctl.git/blob/HEAD:/CHANGES>:
> 
> * Improvements to built-in load-balancing support. Instead of
> adding a separate worker entry in node.cfg for each Bro worker
> process on each worker host, it is now possible to just specify the
> number of worker processes on each host. (Daniel Thayer)
> 
> This change adds three new keywords to the node.cfg file (to be 
> used with worker entries): lb_procs (specifies number of workers on
> a host), lb_method (specifies what type of load balancing to use:
> pf_ring, myricom, or interfaces), and lb_interfaces (used only with
> "lb_method=interfaces" to specify which interfaces to load-balance
> on).
> 
> Two new broctl plugins (which operate automatically and the user 
> doesn't need to be aware of them) are added to set the appropriate 
> environment variables when either PF_RING or myricom load-balancing
> is being used.
> 
> 
> 
> What might be interesting is to document some use cases on the
> website. e.g. a university is looking at 8 Gbps on 5 boxes with 8
> workers each, using Myricom NICs, and a Gigamon Whatever-42
> frontend.
> 
> 
> --Vlad
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iD8DBQFQGaCkK2Plq8B7ZBwRAhHLAKCwQaieXtoBh61RtuVRJLCJVgwo+QCg5IYM
CaceVLzdD4NG3z9N+B5yUIg=
=g+Qf
-----END PGP SIGNATURE-----


From vladg at cmu.edu  Wed Aug  1 14:34:51 2012
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Wed, 1 Aug 2012 21:34:51 +0000
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <14382_1343855395_q71L9rNU009767_CANpnLHgJqK2otX5kk1V19h5t+R1ieT+bnpowUF9OPmX0BzNqQQ@mail.gmail.com>
Message-ID: <CC3F14E2.5AEE%vladg@andrew.cmu.edu>

This sounds like an interesting experiment. The main difference that I see
is that when writing raw PCAP, you're writing to mostly-contiguous blocks.
With the text logs, you're writing to 20 files, scattered who-knows-how,
and at least 3 or 4 of those files are heavily used. The SATA layer
*should* try to reorder the writes to minimize seeks, but who knows how
efficient that is.

If this actually was the issue, it'd be nice if Bro would detect that
$sourceRate > $sinkRate, and log something to reporter, that there's a
bottleneck in the log writer. That's a pretty daunting task, however.

  --Vlad

On 8/1/12 5:09 PM, "Martin Holste" <mcholste at gmail.com> wrote:

> You might be right.  I had considered disk I/O and ran the manager on a
> decent raid-10 array (the elsa server), disk i/o did not appear to be the
> problem so much as CPU and memory.  That observation carried over to
> development builds with a threaded manager, but at an accelerated rate;
> right now I'm watching the disk I/O under threshold and the manager is
> consuming 100M of memory per second until memory exhaustion.

How much I/O from the manager have you seen thus far?  I'm not yet
convinced that writing raw text files is the bottleneck.  When you
think about writing raw pcap, it's orders of magnitude more MB/sec to
disk than logging.
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From mcholste at gmail.com  Wed Aug  1 14:39:44 2012
From: mcholste at gmail.com (Martin Holste)
Date: Wed, 1 Aug 2012 16:39:44 -0500
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CC3F14E2.5AEE%vladg@andrew.cmu.edu>
References: <14382_1343855395_q71L9rNU009767_CANpnLHgJqK2otX5kk1V19h5t+R1ieT+bnpowUF9OPmX0BzNqQQ@mail.gmail.com>
	<CC3F14E2.5AEE%vladg@andrew.cmu.edu>
Message-ID: <CANpnLHjJ9NGwrnf0AA97ZffSp+WC8LhETOGB9dpqWffgBpmZqg@mail.gmail.com>

It ought to be easy enough to tell if it is indeed disk logging IO
that's the bottleneck.  I would think simply disabling all logs or
sending all logs to /dev/null via symlink or even putting the logging
directory on /dev/shm for a bit would all be ways of seeing if the raw
IO is the problem.

On Wed, Aug 1, 2012 at 4:34 PM, Vlad Grigorescu <vladg at cmu.edu> wrote:
> This sounds like an interesting experiment. The main difference that I see
> is that when writing raw PCAP, you're writing to mostly-contiguous blocks.
> With the text logs, you're writing to 20 files, scattered who-knows-how,
> and at least 3 or 4 of those files are heavily used. The SATA layer
> *should* try to reorder the writes to minimize seeks, but who knows how
> efficient that is.
>
> If this actually was the issue, it'd be nice if Bro would detect that
> $sourceRate > $sinkRate, and log something to reporter, that there's a
> bottleneck in the log writer. That's a pretty daunting task, however.
>
>   --Vlad
>
> On 8/1/12 5:09 PM, "Martin Holste" <mcholste at gmail.com> wrote:
>
>> You might be right.  I had considered disk I/O and ran the manager on a
>> decent raid-10 array (the elsa server), disk i/o did not appear to be the
>> problem so much as CPU and memory.  That observation carried over to
>> development builds with a threaded manager, but at an accelerated rate;
>> right now I'm watching the disk I/O under threshold and the manager is
>> consuming 100M of memory per second until memory exhaustion.
>
> How much I/O from the manager have you seen thus far?  I'm not yet
> convinced that writing raw text files is the bottleneck.  When you
> think about writing raw pcap, it's orders of magnitude more MB/sec to
> disk than logging.
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>


From tritium.cat at gmail.com  Wed Aug  1 14:58:00 2012
From: tritium.cat at gmail.com (Tritium Cat)
Date: Wed, 1 Aug 2012 21:58:00 +0000
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CANpnLHgJqK2otX5kk1V19h5t+R1ieT+bnpowUF9OPmX0BzNqQQ@mail.gmail.com>
References: <6730_1343778612_q6VNoAio021810_CAMPgRd6BctXTp9BHP96NaEkhuUykeyAq2DtbG=V_ftObHc4erg@mail.gmail.com>
	<CC3DFAC0.5976%vladg@andrew.cmu.edu>
	<CAMPgRd68HYui4O4LWBngYdvLMph-r0Y4H0bffugaz1Cm3281QA@mail.gmail.com>
	<CANpnLHgJqK2otX5kk1V19h5t+R1ieT+bnpowUF9OPmX0BzNqQQ@mail.gmail.com>
Message-ID: <CAMPgRd5fHcBzzokZeo61_GXBuTF0W-+=z33=Tm7VaK3yn76NdQ@mail.gmail.com>

On Wed, Aug 1, 2012 at 9:09 PM, Martin Holste <mcholste at gmail.com> wrote:

> > You might be right.  I had considered disk I/O and ran the manager on a
> > decent raid-10 array (the elsa server), disk i/o did not appear to be the
> > problem so much as CPU and memory.  That observation carried over to
> > development builds with a threaded manager, but at an accelerated rate;
> > right now I'm watching the disk I/O under threshold and the manager is
> > consuming 100M of memory per second until memory exhaustion.
>
> How much I/O from the manager have you seen thus far?  I'm not yet
> convinced that writing raw text files is the bottleneck.  When you
> think about writing raw pcap, it's orders of magnitude more MB/sec to
> disk than logging.
>


The highest I've seen so far is 443 MB/s with waves of activity between 80
- 390 MB/s.  That's all within a 3 or 4 minute window of time before all
the memory is near exhaustion.  (45-50G for the manager).

I'm using the following to help profile the system:
  people.freebsd.org/~kris/scaling/Help_my_system_is_slow.pdf

--TC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120801/2acd9024/attachment.html 

From tritium.cat at gmail.com  Wed Aug  1 14:59:12 2012
From: tritium.cat at gmail.com (Tritium Cat)
Date: Wed, 1 Aug 2012 21:59:12 +0000
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CANpnLHjJ9NGwrnf0AA97ZffSp+WC8LhETOGB9dpqWffgBpmZqg@mail.gmail.com>
References: <14382_1343855395_q71L9rNU009767_CANpnLHgJqK2otX5kk1V19h5t+R1ieT+bnpowUF9OPmX0BzNqQQ@mail.gmail.com>
	<CC3F14E2.5AEE%vladg@andrew.cmu.edu>
	<CANpnLHjJ9NGwrnf0AA97ZffSp+WC8LhETOGB9dpqWffgBpmZqg@mail.gmail.com>
Message-ID: <CAMPgRd5M-ZHtFmUorGF5LS95K6bhaC1dhv9w8k2L5DbbfMeJXA@mail.gmail.com>

I was going to try capturing from an interface with little to no traffic.

--TC

On Wed, Aug 1, 2012 at 9:39 PM, Martin Holste <mcholste at gmail.com> wrote:

> It ought to be easy enough to tell if it is indeed disk logging IO
> that's the bottleneck.  I would think simply disabling all logs or
> sending all logs to /dev/null via symlink or even putting the logging
> directory on /dev/shm for a bit would all be ways of seeing if the raw
> IO is the problem.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120801/f9f1b42e/attachment.html 

From tritium.cat at gmail.com  Wed Aug  1 16:48:45 2012
From: tritium.cat at gmail.com (Tritium Cat)
Date: Wed, 1 Aug 2012 23:48:45 +0000
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CAMPgRd5M-ZHtFmUorGF5LS95K6bhaC1dhv9w8k2L5DbbfMeJXA@mail.gmail.com>
References: <14382_1343855395_q71L9rNU009767_CANpnLHgJqK2otX5kk1V19h5t+R1ieT+bnpowUF9OPmX0BzNqQQ@mail.gmail.com>
	<CC3F14E2.5AEE%vladg@andrew.cmu.edu>
	<CANpnLHjJ9NGwrnf0AA97ZffSp+WC8LhETOGB9dpqWffgBpmZqg@mail.gmail.com>
	<CAMPgRd5M-ZHtFmUorGF5LS95K6bhaC1dhv9w8k2L5DbbfMeJXA@mail.gmail.com>
Message-ID: <CAMPgRd5wd=WV_9YR7e0S=mOiAp651CPFOHDBpLMNS+o_rHgG8g@mail.gmail.com>

The process is constantly in a uwait status which I believe indicates a
problem with threading.

  PID USERNAME   THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
95626 bro         24  20    0  7473M  7354M uwait  36   8:46 638.82% bro
95628 bro          1 103    5   994M   772M CPU12   0   1:25 85.35% bro
61097 mysql       24  20    0   929M 12556K uwait  39 184:16  0.00% mysqld


Using ktrace I see some errors regarding umtx_op (userland mutex):

95626 157938 bro      0.003929 RET   _umtx_op -1 errno 60 Operation timed
out


--TC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120801/cafe4109/attachment.html 

From robin at icir.org  Thu Aug  2 08:54:01 2012
From: robin at icir.org (Robin Sommer)
Date: Thu, 2 Aug 2012 08:54:01 -0700
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CAMPgRd5wd=WV_9YR7e0S=mOiAp651CPFOHDBpLMNS+o_rHgG8g@mail.gmail.com>
References: <14382_1343855395_q71L9rNU009767_CANpnLHgJqK2otX5kk1V19h5t+R1ieT+bnpowUF9OPmX0BzNqQQ@mail.gmail.com>
	<CC3F14E2.5AEE%vladg@andrew.cmu.edu>
	<CANpnLHjJ9NGwrnf0AA97ZffSp+WC8LhETOGB9dpqWffgBpmZqg@mail.gmail.com>
	<CAMPgRd5M-ZHtFmUorGF5LS95K6bhaC1dhv9w8k2L5DbbfMeJXA@mail.gmail.com>
	<CAMPgRd5wd=WV_9YR7e0S=mOiAp651CPFOHDBpLMNS+o_rHgG8g@mail.gmail.com>
Message-ID: <20120802155400.GE89373@icir.org>


On Wed, Aug 01, 2012 at 23:48 +0000, you wrote:

> The process is constantly in a uwait status which I believe indicates a
> problem with threading.

How does it look when you enable displaying individual threads in top?
Some threads will be pretty much idle most of them time, while others
should be quite busy. If it really seems to hang somewhere, attaching
a gdb should show where exactly the threads stall (in particular the
main one). But seeing the high aggregate CPU load, I'm not sure that's
what's happening.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From robin at icir.org  Thu Aug  2 08:55:11 2012
From: robin at icir.org (Robin Sommer)
Date: Thu, 2 Aug 2012 08:55:11 -0700
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <5019989D.5060902@lbl.gov>
References: <6730_1343778612_q6VNoAio021810_CAMPgRd6BctXTp9BHP96NaEkhuUykeyAq2DtbG=V_ftObHc4erg@mail.gmail.com>
	<CC3DFAC0.5976%vladg@andrew.cmu.edu>
	<CAMPgRd68HYui4O4LWBngYdvLMph-r0Y4H0bffugaz1Cm3281QA@mail.gmail.com>
	<5019989D.5060902@lbl.gov>
Message-ID: <20120802155511.GF89373@icir.org>


On Wed, Aug 01, 2012 at 15:59 -0500, you wrote:

> beer*.  We will put this info up on the Bro web site for the
> edification of those staring into the abyss of 10 (or 100!) G.

This would be great indeed! And thanks for offering the "funding". :-)

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From tritium.cat at gmail.com  Thu Aug  2 12:33:55 2012
From: tritium.cat at gmail.com (Tritium Cat)
Date: Thu, 2 Aug 2012 19:33:55 +0000
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <20120802155400.GE89373@icir.org>
References: <14382_1343855395_q71L9rNU009767_CANpnLHgJqK2otX5kk1V19h5t+R1ieT+bnpowUF9OPmX0BzNqQQ@mail.gmail.com>
	<CC3F14E2.5AEE%vladg@andrew.cmu.edu>
	<CANpnLHjJ9NGwrnf0AA97ZffSp+WC8LhETOGB9dpqWffgBpmZqg@mail.gmail.com>
	<CAMPgRd5M-ZHtFmUorGF5LS95K6bhaC1dhv9w8k2L5DbbfMeJXA@mail.gmail.com>
	<CAMPgRd5wd=WV_9YR7e0S=mOiAp651CPFOHDBpLMNS+o_rHgG8g@mail.gmail.com>
	<20120802155400.GE89373@icir.org>
Message-ID: <CAMPgRd4L3ehXErp9RWV9ry1KFnEogySj7Z50Pa5nwiuXibHG9A@mail.gmail.com>

On Thu, Aug 2, 2012 at 3:54 PM, Robin Sommer <robin at icir.org> wrote:

>
> On Wed, Aug 01, 2012 at 23:48 +0000, you wrote:
>
> > The process is constantly in a uwait status which I believe indicates a
> > problem with threading.
>
> How does it look when you enable displaying individual threads in top?
> Some threads will be pretty much idle most of them time, while others
> should be quite busy. If it really seems to hang somewhere, attaching
> a gdb should show where exactly the threads stall (in particular the
> main one). But seeing the high aggregate CPU load, I'm not sure that's
> what's happening.
>
>
Considering 2.1-beta...

I discovered that last night; displaying by threads showed exactly what
you've described.

I'm not an expert with threads but I do know they are challenging to
program, not to mention debug.  I'm assuming I/O is not the problem here as
the disk latency is 0.2ms at max.

When running 5 clusters instead of one I still have the problem with the
manager consuming all memory on each server, it just takes longer for it to
happen.  The last configuration I tried was 16 workers on each cluster.  I
wasn't able to tell via top and GDB what was going on, so I guess it's time
to litter the code with debugging blocks and figure it out.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120802/ddeb5bd2/attachment.html 

From seth at icir.org  Thu Aug  2 12:44:45 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 2 Aug 2012 15:44:45 -0400
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CAMPgRd4L3ehXErp9RWV9ry1KFnEogySj7Z50Pa5nwiuXibHG9A@mail.gmail.com>
References: <14382_1343855395_q71L9rNU009767_CANpnLHgJqK2otX5kk1V19h5t+R1ieT+bnpowUF9OPmX0BzNqQQ@mail.gmail.com>
	<CC3F14E2.5AEE%vladg@andrew.cmu.edu>
	<CANpnLHjJ9NGwrnf0AA97ZffSp+WC8LhETOGB9dpqWffgBpmZqg@mail.gmail.com>
	<CAMPgRd5M-ZHtFmUorGF5LS95K6bhaC1dhv9w8k2L5DbbfMeJXA@mail.gmail.com>
	<CAMPgRd5wd=WV_9YR7e0S=mOiAp651CPFOHDBpLMNS+o_rHgG8g@mail.gmail.com>
	<20120802155400.GE89373@icir.org>
	<CAMPgRd4L3ehXErp9RWV9ry1KFnEogySj7Z50Pa5nwiuXibHG9A@mail.gmail.com>
Message-ID: <71355800-0A68-4E56-AF58-05F4BF64E52D@icir.org>


On Aug 2, 2012, at 3:33 PM, Tritium Cat <tritium.cat at gmail.com> wrote:

> When running 5 clusters instead of one I still have the problem with the manager consuming all memory on each server

We may be converging on the source of this problem.  We'll be sure to update again.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From tritium.cat at gmail.com  Thu Aug  2 12:45:45 2012
From: tritium.cat at gmail.com (Tritium Cat)
Date: Thu, 2 Aug 2012 19:45:45 +0000
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CAG1VgOYrXeHrayWPGcDBF+rmYv1wFkszqG9cg5ymMGRvcWJxMQ@mail.gmail.com>
References: <CAMPgRd5us7tC=tZoaRkG8Q+ZeQQt7Vip-SStR7J1=dYbPLdO6A@mail.gmail.com>
	<CAG1VgOYrXeHrayWPGcDBF+rmYv1wFkszqG9cg5ymMGRvcWJxMQ@mail.gmail.com>
Message-ID: <CAMPgRd5buSM19Xk0+aLBM9qioHewZJUOcpnvBOdFXB7cOtiz3Q@mail.gmail.com>

On Wed, Aug 1, 2012 at 7:27 PM, Chris Crawford <
christopher.p.crawford at gmail.com> wrote:

> Have you seen any of my threads from earlier this year?
>
> http://bit.ly/JJQVVf
> http://bit.ly/N2l4yT
>
> Your issue sounds similar to what I was experiencing.
>
> Bro 2.0 is routinely uses up all available memory and then crashes for me.
>
>
Ok so we are lucky to share the same misfortune.  Sounds like the same
problem I've had since using Bro.  Contact me off-list and we'll exchange
notes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120802/0924961e/attachment.html 

From jones at tacc.utexas.edu  Thu Aug  2 12:47:31 2012
From: jones at tacc.utexas.edu (William Jones)
Date: Thu, 2 Aug 2012 19:47:31 +0000
Subject: [Bro] Bro and PF_RING Cluster ID
In-Reply-To: <CABEaz7+OoAarfTAp+YBv=pq3GdS0i0Q6TjyCbd13mUtamkOX1A@mail.gmail.com>
References: <CABEaz7+OoAarfTAp+YBv=pq3GdS0i0Q6TjyCbd13mUtamkOX1A@mail.gmail.com>
Message-ID: <D0BD4A5C03717E40AC104854AF37B8C21DCFEE91@EXDMBX03.austin.utexas.edu>

It is not necessary for PF_RING to use different cluster id per capture interface.

You can increase the the number of works per cluster id by changing CLUSTER_LEN linux/pf_ring.h from 8 to 16 or 32. 

There seems other limitation in the number of works on host in bro whne you gove above 8 works on a hosts.


Bill Jones

-----Original Message-----
From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf Of Robert Rotsted
Sent: Tuesday, July 31, 2012 5:36 PM
To: bro at bro-ids.org
Subject: [Bro] Bro and PF_RING Cluster ID

Hi all,

I'm running a clustered Bro instance with workers capturing traffic on three PF_RING enabled e1000e interfaces.

While looking in /proc/net/pf_ring/ I noticed that all of my Bro workers belong to cluster id 21. Is it possible (or desirable) in Bro to create a PF_RING cluster id per capture interface?

I read that PF_RING allows a maximum of eight workers per cluster id, is this still true?

Best,

Bob
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From tritium.cat at gmail.com  Thu Aug  2 13:13:24 2012
From: tritium.cat at gmail.com (Tritium Cat)
Date: Thu, 2 Aug 2012 20:13:24 +0000
Subject: [Bro] Bro and PF_RING Cluster ID
In-Reply-To: <D0BD4A5C03717E40AC104854AF37B8C21DCFEE91@EXDMBX03.austin.utexas.edu>
References: <CABEaz7+OoAarfTAp+YBv=pq3GdS0i0Q6TjyCbd13mUtamkOX1A@mail.gmail.com>
	<D0BD4A5C03717E40AC104854AF37B8C21DCFEE91@EXDMBX03.austin.utexas.edu>
Message-ID: <CAMPgRd6CprVu-n_KOMNfB41KA_-MafWXuVOpTVMOe9rQ6=QWvw@mail.gmail.com>

The CLUSTER_LEN depends on what version of PF_RING you use.  The latest
development build is set to 32.

You can set the cluster ID in the broctl.conf file but I'm not sure how
that works when more than one PF_RING interface is being used.  If it makes
sense then maybe leveraging the load balancing configuration and having
something like "lb_ring_id" ?



On Thu, Aug 2, 2012 at 7:47 PM, William Jones <jones at tacc.utexas.edu> wrote:

> It is not necessary for PF_RING to use different cluster id per capture
> interface.
>
> You can increase the the number of works per cluster id by changing
> CLUSTER_LEN linux/pf_ring.h from 8 to 16 or 32.
>
> There seems other limitation in the number of works on host in bro whne
> you gove above 8 works on a hosts.
>
>
> Bill Jones
>
> -----Original Message-----
> From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf
> Of Robert Rotsted
> Sent: Tuesday, July 31, 2012 5:36 PM
> To: bro at bro-ids.org
> Subject: [Bro] Bro and PF_RING Cluster ID
>
> Hi all,
>
> I'm running a clustered Bro instance with workers capturing traffic on
> three PF_RING enabled e1000e interfaces.
>
> While looking in /proc/net/pf_ring/ I noticed that all of my Bro workers
> belong to cluster id 21. Is it possible (or desirable) in Bro to create a
> PF_RING cluster id per capture interface?
>
> I read that PF_RING allows a maximum of eight workers per cluster id, is
> this still true?
>
> Best,
>
> Bob
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120802/174bb058/attachment.html 

From robin at icir.org  Fri Aug  3 13:10:05 2012
From: robin at icir.org (Robin Sommer)
Date: Fri, 3 Aug 2012 13:10:05 -0700
Subject: [Bro] Bro 2.1 Public Beta
In-Reply-To: <20120801175017.GQ45936@icir.org>
References: <20120801175017.GQ45936@icir.org>
Message-ID: <20120803201005.GA30850@icir.org>


On Wed, Aug 01, 2012 at 10:50 -0700, I wrote:

>     http://blog.bro-ids.org/2012/08/bro-21-public-beta.html

For the folks with high traffic volumes who are seeing memory and/or
CPU problems with 2.1 beta (in comparision with 2.0), please apply
this patch to src/logging/writers/Ascii.cc:

    http://git.bro-ids.org/bro.git/patch/9829cf9a296b4f1c6614658b80225cd80f2e24ec

>From what we hear, that seems to solve the problem. You can
alternatively just update to git master.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From faarshad at purdue.edu  Sat Aug  4 05:45:24 2012
From: faarshad at purdue.edu (Fahad Ali Arshad)
Date: Sat, 4 Aug 2012 08:45:24 -0400
Subject: [Bro] Regular expression behavior
Message-ID: <2D17D73D-FBD8-4BB0-BE6A-0BD6D8DA91C6@purdue.edu>

Hi,
I am testing some regular expression in bro and Bro2.0 is giving me weird behavior. For example, the following regular expression puts Bro in an infinite loop.

/(merge.*?using\s*?\()|execute\s*?immediate\s*[\"\'`?\x2018\x2019])|(\W+\d*?\s*?having\s*?[^\s\-])|(match\s*?[\w(),\+\-]+\s*?against\s*?\()/

Note that the \x2018 and \x2019 are ASCII codes for LEFT SINGLE QUOTATION MARK and RIGHT SINGLE QUOTATION MARK.

If I plug in the actual single quotation marks then I get a segmentation fault. 

Any idea what is going on?

Best,
Fahad Arshad



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120804/0eb42865/attachment.html 

From ehjulio at gmail.com  Mon Aug  6 06:28:44 2012
From: ehjulio at gmail.com (Ernesto Julio)
Date: Mon, 06 Aug 2012 09:28:44 -0400
Subject: [Bro] count inside a packet, possible?
Message-ID: <1344259724.2247.41.camel@kareem>

Hi,

Wonder if anybody could confirm or not if it is possible to count within
a packet, in Bro. What I mean is: if a single packet contains the string
'6xxx6xxx6xxx6', is it possible to use a counter to determine the number
of characters '6' in the string is 4?

I have been looking at the examples inside Bro where regular expressions
are used, so an option could be '6.+?6.+?6.+?6'. But this might not be
scalable if I need to count for several characters, and in my case, I
don't know the order in which the characters appear in the packet.

I am a Bro newbie, sorry for the question as I understand this might go
against regular practices (as one might prefer to do a fast matching of
a packet). But I am learning and have the question bugging me.

Many thanks,


Ernesto



From Huiping.Song at ultra-3eti.com  Mon Aug  6 12:23:59 2012
From: Huiping.Song at ultra-3eti.com (Huiping Song)
Date: Mon, 6 Aug 2012 19:23:59 +0000
Subject: [Bro] Use ACTION_DROP to Limit HTTP/FTP Access?
Message-ID: <394E2A9510AB1642987C5AADE63C547607583B7E@RockMX01.rock.corp>

We are thinking about limiting HTTP/FTP access by using the "base/frameworks/notice/actions/drop.bro" script:   only allow HTTP file upload and FTP put/mput commands from specified IP addresses.

Is it possible to accomplish this by simply modifying http/ftp scripts from the Bro script packages?  Does anyone have sample scripts for using ACTION_DROP for HTTP or FTP traffic?  I couldn't find out any usages of ACTION_DROP in the installed bro scripts.

Thank you,
Huiping

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120806/bd0f7ba3/attachment.html 

From charles.fair at mac.com  Mon Aug  6 12:49:25 2012
From: charles.fair at mac.com (Charles A. Fair)
Date: Mon, 06 Aug 2012 13:49:25 -0600
Subject: [Bro] Informal pre Bro Exchange get togetherainformal
In-Reply-To: <1344259724.2247.41.camel@kareem>
References: <1344259724.2247.41.camel@kareem>
Message-ID: <AE4CE618-9ACD-4259-9302-7E932A07B05A@mac.com>

Anybody here for the Bro Exchange, staying at the Rodeway Inn?  If so, let's get together for dinner tonight. Please contact me offline at charles.fair at mac.com . 

Charles "Chuck A. Fair 


From charles.fair at mac.com  Mon Aug  6 14:58:43 2012
From: charles.fair at mac.com (Charles A. Fair)
Date: Mon, 06 Aug 2012 15:58:43 -0600
Subject: [Bro] Informal pre Bro Exchange get together (redux version)
Message-ID: <4DBBE825-A9D6-4A5D-8F15-7DDCFA8A7E9A@mac.com>

(Looks like the RBN obviously hijacked my email and sent it out prematurely) 

Anybody here for the Bro Exchange, staying at the Rodeway Inn, Boulder Colorado?  If so, let's get together for dinner/drinks tonight. 

Dark Horse Bar And Grill looks like a great Venue.   We could shoot for 8-10.  Any takers? 

Please contact me direct off the list at:  
- cell: 405-582-0676 
- charles.fair at mac.com 

Charles "Chuck A. Fair 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120806/5822eb78/attachment.html 

From christopher.p.crawford at gmail.com  Mon Aug  6 16:32:02 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Mon, 6 Aug 2012 19:32:02 -0400
Subject: [Bro] Potential Memory Leak and Denial of Service Vulnerability in
	Bro 2
In-Reply-To: <CAG1VgObREeq4YejWU_=7Mv=Kj105jvfvzojdOsYxt_tutjd5Qg@mail.gmail.com>
References: <CAG1VgObREeq4YejWU_=7Mv=Kj105jvfvzojdOsYxt_tutjd5Qg@mail.gmail.com>
Message-ID: <CAG1VgOYSUMO9fP3UcWvmoLX+ua5eMdfHW0XDaHUCjkU8urf7Sg@mail.gmail.com>

I worked up a set of notes to share with Tritium Cat as a result of this thread:

http://mailman.icsi.berkeley.edu/pipermail/bro/2012-July/005676.html

I've decided to share those notes to benefit the community.  My notes
are giant brain dump in an html file that I put together very quickly.
 My apologies for the terrible formatting.  I posted a copy at:
http://ec2-50-19-133-29.compute-1.amazonaws.com/

Please contact me off list if you'd like you're own copy.

-Chris


From christopher.p.crawford at gmail.com  Wed Aug  8 08:20:53 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Wed, 8 Aug 2012 11:20:53 -0400
Subject: [Bro] pcap_next Question
In-Reply-To: <523E76AD-5308-4D8A-9E96-F586755483E3@illinois.edu>
References: <CAG1VgOapQQbSmnOkZOO9dhY54XdaNKXy+Y3+ks6M4Hn_9UXFNQ@mail.gmail.com>
	<BC3C8AB0-B4C0-41A1-A2E7-39F71CE2C664@illinois.edu>
	<CAG1VgOaUqPjXpCTUJexqPwiwDQYr7wmPfSiEjW-z1YTcUi7yrQ@mail.gmail.com>
	<523E76AD-5308-4D8A-9E96-F586755483E3@illinois.edu>
Message-ID: <CAG1VgOa3UnDHHf=P6XYH-YhLb-yaZyA4wBhWRN4=u2bzqkooYQ@mail.gmail.com>

I've been doing some thinking about this thread lately.

Here's the bit of code related to the last post in this thread
(http://mailman.icsi.berkeley.edu/pipermail/bro/2012-May/005564.html)

Reassem.cc, starting at line 23:


       block = new u_char[size];
       if ( ! block )
                reporter->InternalError("out of memory");


Since the new operator is throwing a std::bad_alloc error, that line
should be surrounded with a bit of error handling like in the attached
patch file. (To apply, copy to bro/src and use "patch
--ignore-whitespace < Reassem.cc.patch").

The result would look something like this:

       try
       {
               block = new u_char[size];
       }
       catch(std::bad_alloc& exc)
       {
                reporter->InternalError("out of memory");
       }


That way, if there is no memory available, at least this error can be
handled gracefully.

As it is right now, if new fails to allocate memory, the following if
statement will never execute and the end user will never see the "out
of memory" message in their logs.

-Chris

On Mon, May 21, 2012 at 3:45 PM, Siwek, Jonathan Luke
<jsiwek at illinois.edu> wrote:
>
>> So, either
>> pcap_next is returning an out of bounds pointer, or something happens
>> to data between the point in time when pcap_next returns a values and
>> PktSrc::Process calls net_packet_dispatch.
>
> Could be, but then I'd expect a segfault much earlier, at least before NetSessions::DoNextPacket() when the packet data is accessed.  In the end, your crash doesn't look related to accessing that data at all, it's an unhandled exception thrown from operator new[] (failure to allocate memory).
>
> Another possibility could be that since GDB is working with optimized code (the reason why some arguments are "<value optimized out>"), the "out of bounds" indicators don't necessarily indicate a problem yet.  If you `./configure --enable-debug` then rebuild/reinstall to disable optimizations, you can see if stack traces for future crashes start reporting valid addresses for the pointer.
>
> +Jon
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Reassem.cc.patch
Type: application/octet-stream
Size: 501 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120808/b4c53cb0/attachment.obj 

From christopher.p.crawford at gmail.com  Wed Aug  8 08:23:16 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Wed, 8 Aug 2012 11:23:16 -0400
Subject: [Bro] pcap_next Question
In-Reply-To: <CAG1VgOa3UnDHHf=P6XYH-YhLb-yaZyA4wBhWRN4=u2bzqkooYQ@mail.gmail.com>
References: <CAG1VgOapQQbSmnOkZOO9dhY54XdaNKXy+Y3+ks6M4Hn_9UXFNQ@mail.gmail.com>
	<BC3C8AB0-B4C0-41A1-A2E7-39F71CE2C664@illinois.edu>
	<CAG1VgOaUqPjXpCTUJexqPwiwDQYr7wmPfSiEjW-z1YTcUi7yrQ@mail.gmail.com>
	<523E76AD-5308-4D8A-9E96-F586755483E3@illinois.edu>
	<CAG1VgOa3UnDHHf=P6XYH-YhLb-yaZyA4wBhWRN4=u2bzqkooYQ@mail.gmail.com>
Message-ID: <CAG1VgOYVNXMZ904HYAbE_coB4Ztz2KjHRPjD6hVrZn65Zv57bA@mail.gmail.com>

Looks like the patch file did not post to the list.

This is what it looks like:


--- Reassem.cc  2012-01-11 16:53:40.000000000 -0500
+++ Reassem.cc.new      2012-08-06 22:32:52.000000000 -0400
@@ -20,9 +20,14 @@
        seq = arg_seq;
        upper = seq + size;

-       block = new u_char[size];
-       if ( ! block )
+       try
+       {
+               block = new u_char[size];
+       }
+       catch(std::bad_alloc& exc)
+       {
                reporter->InternalError("out of memory");
+       }

        memcpy((void*) block, (const void*) data, size);

-Chris

On Wed, Aug 8, 2012 at 11:20 AM, Chris Crawford
<christopher.p.crawford at gmail.com> wrote:
> I've been doing some thinking about this thread lately.
>
> Here's the bit of code related to the last post in this thread
> (http://mailman.icsi.berkeley.edu/pipermail/bro/2012-May/005564.html)
>
> Reassem.cc, starting at line 23:
>
>
>        block = new u_char[size];
>        if ( ! block )
>                 reporter->InternalError("out of memory");
>
>
> Since the new operator is throwing a std::bad_alloc error, that line
> should be surrounded with a bit of error handling like in the attached
> patch file. (To apply, copy to bro/src and use "patch
> --ignore-whitespace < Reassem.cc.patch").
>
> The result would look something like this:
>
>        try
>        {
>                block = new u_char[size];
>        }
>        catch(std::bad_alloc& exc)
>        {
>                 reporter->InternalError("out of memory");
>        }
>
>
> That way, if there is no memory available, at least this error can be
> handled gracefully.
>
> As it is right now, if new fails to allocate memory, the following if
> statement will never execute and the end user will never see the "out
> of memory" message in their logs.
>
> -Chris
>
> On Mon, May 21, 2012 at 3:45 PM, Siwek, Jonathan Luke
> <jsiwek at illinois.edu> wrote:
>>
>>> So, either
>>> pcap_next is returning an out of bounds pointer, or something happens
>>> to data between the point in time when pcap_next returns a values and
>>> PktSrc::Process calls net_packet_dispatch.
>>
>> Could be, but then I'd expect a segfault much earlier, at least before NetSessions::DoNextPacket() when the packet data is accessed.  In the end, your crash doesn't look related to accessing that data at all, it's an unhandled exception thrown from operator new[] (failure to allocate memory).
>>
>> Another possibility could be that since GDB is working with optimized code (the reason why some arguments are "<value optimized out>"), the "out of bounds" indicators don't necessarily indicate a problem yet.  If you `./configure --enable-debug` then rebuild/reinstall to disable optimizations, you can see if stack traces for future crashes start reporting valid addresses for the pointer.
>>
>> +Jon



From hammadog at gmail.com  Wed Aug  8 08:38:07 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Wed, 8 Aug 2012 11:38:07 -0400
Subject: [Bro] Some BPF love....
Message-ID: <CANCSO1cXHO7PkaUDvA0_r4SEck39cYMUmMTOC5PXHfd5LUV_rQ@mail.gmail.com>

Sent this off to the SecurityOnion group, but probably should have
sent it here.   Oopsy!

Anyway

Please....I know I must be doing something noobish...but man, I have
tried it 15 ways to Sunday and no love.

editing:  /nsm/bro/spool/policy/site/local.bro

added "redef cmd_line_bpf_filter = "not src host ipaddress";

I want to tweak a tad more based on dst port, but need to at least get
the filter working for the IP.

I then do a check/install/restart

I watch BRO dns.log for the for the IP I added and she shows up.  What
the heck am I missing?

Any help much appreciated.


-- 
Tom O'Brion
Twitter: @tobrion

"Life is too short to spend time with people who suck the happy out of you."


From tritium.cat at gmail.com  Wed Aug  8 08:48:33 2012
From: tritium.cat at gmail.com (Tritium Cat)
Date: Wed, 8 Aug 2012 09:48:33 -0600
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CAMPgRd5buSM19Xk0+aLBM9qioHewZJUOcpnvBOdFXB7cOtiz3Q@mail.gmail.com>
References: <CAMPgRd5us7tC=tZoaRkG8Q+ZeQQt7Vip-SStR7J1=dYbPLdO6A@mail.gmail.com>
	<CAG1VgOYrXeHrayWPGcDBF+rmYv1wFkszqG9cg5ymMGRvcWJxMQ@mail.gmail.com>
	<CAMPgRd5buSM19Xk0+aLBM9qioHewZJUOcpnvBOdFXB7cOtiz3Q@mail.gmail.com>
Message-ID: <CAMPgRd7mu0Pcd9b4tSwM1uyCAWtDt8-Q2EbqWf-XUjgV3AWRuA@mail.gmail.com>

On Thu, Aug 2, 2012 at 1:45 PM, Tritium Cat <tritium.cat at gmail.com> wrote:

> On Wed, Aug 1, 2012 at 7:27 PM, Chris Crawford <
> christopher.p.crawford at gmail.com> wrote:
>
>> Have you seen any of my threads from earlier this year?
>>
>> http://bit.ly/JJQVVf
>> http://bit.ly/N2l4yT
>>
>> Your issue sounds similar to what I was experiencing.
>>
>> Bro 2.0 is routinely uses up all available memory and then crashes for me.
>>
>>
Someone mentioned it's likely due to the traffic on the network; they had a
similar problem that involved certain SSL traffic.  The idea is to disable
features until finding the problem and then devise a workaround.  That's
the plan for now.

--TC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120808/3407150a/attachment.html 

From christopher.p.crawford at gmail.com  Wed Aug  8 09:02:54 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Wed, 8 Aug 2012 12:02:54 -0400
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CAMPgRd7mu0Pcd9b4tSwM1uyCAWtDt8-Q2EbqWf-XUjgV3AWRuA@mail.gmail.com>
References: <CAMPgRd5us7tC=tZoaRkG8Q+ZeQQt7Vip-SStR7J1=dYbPLdO6A@mail.gmail.com>
	<CAG1VgOYrXeHrayWPGcDBF+rmYv1wFkszqG9cg5ymMGRvcWJxMQ@mail.gmail.com>
	<CAMPgRd5buSM19Xk0+aLBM9qioHewZJUOcpnvBOdFXB7cOtiz3Q@mail.gmail.com>
	<CAMPgRd7mu0Pcd9b4tSwM1uyCAWtDt8-Q2EbqWf-XUjgV3AWRuA@mail.gmail.com>
Message-ID: <CAG1VgOa0TRRH8de0vJR0FQ44Cs0cCEgYftDZ5vJZtLr31AVPwg@mail.gmail.com>

I have the following in my local.bro file:

redef SMTP::generate_md5 += /image.*/;
redef HTTP::generate_md5 += /image.*/;
redef SMTP::generate_md5 += /text.*/;
redef HTTP::generate_md5 += /text.*/;
redef SMTP::generate_md5 += /application.*/;
redef HTTP::generate_md5 += /application.*/;
redef SMTP::generate_md5 += /audio.*/;
redef HTTP::generate_md5 += /audio.*/;
redef SMTP::generate_md5 += /video.*/;
redef HTTP::generate_md5 += /video.*/;


Using broctl's top and a little trial and error, I can see that these
lines are the cause of my high CPU usage.  It also causes higher
memory usage as well, but memory usage always climbs and never gets
smaller.  I don't know if these lines are responsible for just higher
memory usage in general, or whether they are also responsible gradual
climb in memory.  It appears that memory gradually climbs even without
these lines, but I haven't had enough time to test that idea.  I
believe that the climbing memory eventually leads to a crash,
typically when Reassem.cc attempts to allocate some new memory and an
unhandled exception is triggered.  The broctl cron command restarts
bro for me.

-Chris

On Wed, Aug 8, 2012 at 11:48 AM, Tritium Cat <tritium.cat at gmail.com> wrote:
> On Thu, Aug 2, 2012 at 1:45 PM, Tritium Cat <tritium.cat at gmail.com> wrote:
>>
>> On Wed, Aug 1, 2012 at 7:27 PM, Chris Crawford
>> <christopher.p.crawford at gmail.com> wrote:
>>>
>>> Have you seen any of my threads from earlier this year?
>>>
>>> http://bit.ly/JJQVVf
>>> http://bit.ly/N2l4yT
>>>
>>> Your issue sounds similar to what I was experiencing.
>>>
>>> Bro 2.0 is routinely uses up all available memory and then crashes for
>>> me.
>>>
>
> Someone mentioned it's likely due to the traffic on the network; they had a
> similar problem that involved certain SSL traffic.  The idea is to disable
> features until finding the problem and then devise a workaround.  That's the
> plan for now.
>
> --TC
>
>


From christopher.p.crawford at gmail.com  Wed Aug  8 10:21:18 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Wed, 8 Aug 2012 13:21:18 -0400
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CAG1VgOa0TRRH8de0vJR0FQ44Cs0cCEgYftDZ5vJZtLr31AVPwg@mail.gmail.com>
References: <CAMPgRd5us7tC=tZoaRkG8Q+ZeQQt7Vip-SStR7J1=dYbPLdO6A@mail.gmail.com>
	<CAG1VgOYrXeHrayWPGcDBF+rmYv1wFkszqG9cg5ymMGRvcWJxMQ@mail.gmail.com>
	<CAMPgRd5buSM19Xk0+aLBM9qioHewZJUOcpnvBOdFXB7cOtiz3Q@mail.gmail.com>
	<CAMPgRd7mu0Pcd9b4tSwM1uyCAWtDt8-Q2EbqWf-XUjgV3AWRuA@mail.gmail.com>
	<CAG1VgOa0TRRH8de0vJR0FQ44Cs0cCEgYftDZ5vJZtLr31AVPwg@mail.gmail.com>
Message-ID: <CAG1VgOahewH-CWjgvpq_v+Gpy7a-Tuo8d+QKihgx3K=Qo_Z2Cg@mail.gmail.com>

I've also noticed something peculiar about the node.cfg file that
causes high CPU usage, independent from generating MD5s.

I was under the impression that I needed to configure node.cfg from the default:

[bro]
type=standalone
host=localhost
interface=eth0

To something that makes more sense for my environment

[bro]
type=standalone
host=1.2.3.4
interface=eth0

For some reason, when I do this, it causes broctl to take a very long
time to return from the status command and the number of peers
reported is ??? and not the expected 0.  Configuring my host to an IP
address also causes CPU to spike to about 100%.

-Chris

On Wed, Aug 8, 2012 at 12:02 PM, Chris Crawford
<christopher.p.crawford at gmail.com> wrote:
> I have the following in my local.bro file:
>
> redef SMTP::generate_md5 += /image.*/;
> redef HTTP::generate_md5 += /image.*/;
> redef SMTP::generate_md5 += /text.*/;
> redef HTTP::generate_md5 += /text.*/;
> redef SMTP::generate_md5 += /application.*/;
> redef HTTP::generate_md5 += /application.*/;
> redef SMTP::generate_md5 += /audio.*/;
> redef HTTP::generate_md5 += /audio.*/;
> redef SMTP::generate_md5 += /video.*/;
> redef HTTP::generate_md5 += /video.*/;
>
>
> Using broctl's top and a little trial and error, I can see that these
> lines are the cause of my high CPU usage.  It also causes higher
> memory usage as well, but memory usage always climbs and never gets
> smaller.  I don't know if these lines are responsible for just higher
> memory usage in general, or whether they are also responsible gradual
> climb in memory.  It appears that memory gradually climbs even without
> these lines, but I haven't had enough time to test that idea.  I
> believe that the climbing memory eventually leads to a crash,
> typically when Reassem.cc attempts to allocate some new memory and an
> unhandled exception is triggered.  The broctl cron command restarts
> bro for me.
>
> -Chris
>
> On Wed, Aug 8, 2012 at 11:48 AM, Tritium Cat <tritium.cat at gmail.com> wrote:
>> On Thu, Aug 2, 2012 at 1:45 PM, Tritium Cat <tritium.cat at gmail.com> wrote:
>>>
>>> On Wed, Aug 1, 2012 at 7:27 PM, Chris Crawford
>>> <christopher.p.crawford at gmail.com> wrote:
>>>>
>>>> Have you seen any of my threads from earlier this year?
>>>>
>>>> http://bit.ly/JJQVVf
>>>> http://bit.ly/N2l4yT
>>>>
>>>> Your issue sounds similar to what I was experiencing.
>>>>
>>>> Bro 2.0 is routinely uses up all available memory and then crashes for
>>>> me.
>>>>
>>
>> Someone mentioned it's likely due to the traffic on the network; they had a
>> similar problem that involved certain SSL traffic.  The idea is to disable
>> features until finding the problem and then devise a workaround.  That's the
>> plan for now.
>>
>> --TC
>>
>>


From vern at icir.org  Wed Aug  8 23:01:47 2012
From: vern at icir.org (Vern Paxson)
Date: Wed, 08 Aug 2012 23:01:47 -0700
Subject: [Bro] pcap_next Question
In-Reply-To: <CAG1VgOa3UnDHHf=P6XYH-YhLb-yaZyA4wBhWRN4=u2bzqkooYQ@mail.gmail.com>
	(Wed, 08 Aug 2012 11:20:53 EDT).
Message-ID: <20120809060147.174E52C4008@rock.ICSI.Berkeley.EDU>

>        block = new u_char[size];
>        if ( ! block )
>                 reporter->InternalError("out of memory");
> 
> 
> Since the new operator is throwing a std::bad_alloc error, that line
> should be surrounded with a bit of error handling like in the attached
> patch file.

I thought 'new' is defined to return a nil pointer upon memory exhaustion.
Is that wrong / dates me horribly?

		Vern


From christopher.p.crawford at gmail.com  Thu Aug  9 03:43:47 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Thu, 9 Aug 2012 06:43:47 -0400
Subject: [Bro] pcap_next Question
In-Reply-To: <20120809060147.174E52C4008@rock.ICSI.Berkeley.EDU>
References: <CAG1VgOa3UnDHHf=P6XYH-YhLb-yaZyA4wBhWRN4=u2bzqkooYQ@mail.gmail.com>
	<20120809060147.174E52C4008@rock.ICSI.Berkeley.EDU>
Message-ID: <CAG1VgOaxbvmMha2zvoVoBfRHh2gGjAFWEeBhvV59rj0FXOPoVQ@mail.gmail.com>

I don't think that is quite right for C++.

In this case, I have a stack trace showing that std::bad_alloc is
thrown, but never caught.

For a less official explanation, here's a stackoverflow post that
addresses the same issue:
http://stackoverflow.com/questions/7277637/new-stdnothrow-vs-new-within-a-try-catch-block

Cplusplus.com has a more official explanation.

Here is a high level view of how new works in the C++ STL:
http://www.cplusplus.com/reference/std/new/

If you take a look at both new and new[], you can see that in either
case "On failure, it throws a bad_alloc exception."

This explains std::nothrow and how to catch the error:
http://www.cplusplus.com/reference/std/new/bad_alloc/

To get the effect you're talking about in C++, you have to explicitly
cast a new operation with std::nothrow:
http://www.cplusplus.com/reference/std/new/nothrow/

Here is another example:
http://www.cprogramming.com/tips/tip/cincrement-new-does-not-return-0-on-failure

I did a bit a of Googling to find some documentation that would say
that the new operator returns a NULL pointer by defaut, versus
returning std::bad_alloc.  The only thing I could really find was a an
bug in Visual C++ that was documented in this KB article:
http://support.microsoft.com/?kbid=167733


-Chris

On Thu, Aug 9, 2012 at 2:01 AM, Vern Paxson <vern at icir.org> wrote:
>>        block = new u_char[size];
>>        if ( ! block )
>>                 reporter->InternalError("out of memory");
>>
>>
>> Since the new operator is throwing a std::bad_alloc error, that line
>> should be surrounded with a bit of error handling like in the attached
>> patch file.
>
> I thought 'new' is defined to return a nil pointer upon memory exhaustion.
> Is that wrong / dates me horribly?
>
>                 Vern


From tyler.schoenke at colorado.edu  Thu Aug  9 07:38:58 2012
From: tyler.schoenke at colorado.edu (Tyler T. Schoenke)
Date: Thu, 09 Aug 2012 08:38:58 -0600
Subject: [Bro] Some BPF love....
In-Reply-To: <CANCSO1cXHO7PkaUDvA0_r4SEck39cYMUmMTOC5PXHfd5LUV_rQ@mail.gmail.com>
References: <CANCSO1cXHO7PkaUDvA0_r4SEck39cYMUmMTOC5PXHfd5LUV_rQ@mail.gmail.com>
Message-ID: <5023CB82.7080408@colorado.edu>

I've only briefly tested SecurityOnion, but in vanilla Bro, you would
add something like this to local.bro.  That file is located under
$BROHOME/share/bro/site.

redef restrict_filters += { ["host exemptions"] = "not (host 4.2.2.2)" };

I don't know SecuritiyOnion's layout, but I don't think you want to add
it under spool.  That is typically where runtime files are created.

Tyler

--
Tyler Schoenke
Network Security Manager
IT Security Office
University of Colorado at Boulder

On 8/8/12 9:38 AM, Tom OBrion wrote:
> Sent this off to the SecurityOnion group, but probably should have
> sent it here.   Oopsy!
> 
> Anyway
> 
> Please....I know I must be doing something noobish...but man, I have
> tried it 15 ways to Sunday and no love.
> 
> editing:  /nsm/bro/spool/policy/site/local.bro
> 
> added "redef cmd_line_bpf_filter = "not src host ipaddress";
> 
> I want to tweak a tad more based on dst port, but need to at least get
> the filter working for the IP.
> 
> I then do a check/install/restart
> 
> I watch BRO dns.log for the for the IP I added and she shows up.  What
> the heck am I missing?
> 
> Any help much appreciated.
> 
> 


From JAzoff at albany.edu  Thu Aug  9 08:15:09 2012
From: JAzoff at albany.edu (Justin Azoff)
Date: Thu, 9 Aug 2012 11:15:09 -0400
Subject: [Bro] Some BPF love....
In-Reply-To: <5023CB82.7080408@colorado.edu>
References: <CANCSO1cXHO7PkaUDvA0_r4SEck39cYMUmMTOC5PXHfd5LUV_rQ@mail.gmail.com>
	<5023CB82.7080408@colorado.edu>
Message-ID: <20120809151509.GO1957@datacomm.albany.edu>

On Thu, Aug 09, 2012 at 08:38:58AM -0600, Tyler T. Schoenke wrote:
> I've only briefly tested SecurityOnion, but in vanilla Bro, you would
> add something like this to local.bro.  That file is located under
> $BROHOME/share/bro/site.
> 
> redef restrict_filters += { ["host exemptions"] = "not (host 4.2.2.2)" };

Might also need

redef PacketFilter::all_packets = F; # don't capture all packets

-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From hammadog at gmail.com  Thu Aug  9 08:26:38 2012
From: hammadog at gmail.com (Tom OBrion)
Date: Thu, 9 Aug 2012 11:26:38 -0400
Subject: [Bro] Some BPF love....
In-Reply-To: <5023CB82.7080408@colorado.edu>
References: <CANCSO1cXHO7PkaUDvA0_r4SEck39cYMUmMTOC5PXHfd5LUV_rQ@mail.gmail.com>
	<5023CB82.7080408@colorado.edu>
Message-ID: <CANCSO1cTPGKzjQuGuWT9gXQGUFsjySg7g9FObutTecQnyDk27g@mail.gmail.com>

Hey Tyler

Thanks, I was updating it in the spool folder based on the DOC I was
reading out on the SO groups site.  I thought it was wierd that I
update in the spool location and not the share location.  Maybe I was
just reading it wrong in the DOC.  I have been known to skin reading
and not completely reading it fully.  :)  Anyway, made the updates in
the location you mentioned and it seems to be working.  I am not using
your syntax though, I am using this:

redef cmd_line_bpf_filter = "not (host x.x.x.x)";

Worked like a champ.  Now I will tweak to include dest port and should
be good to go.  Thanks man.  Got me on the right track!

Tom

On Thu, Aug 9, 2012 at 10:38 AM, Tyler T. Schoenke
<tyler.schoenke at colorado.edu> wrote:
> I've only briefly tested SecurityOnion, but in vanilla Bro, you would
> add something like this to local.bro.  That file is located under
> $BROHOME/share/bro/site.
>
> redef restrict_filters += { ["host exemptions"] = "not (host 4.2.2.2)" };
>
> I don't know SecuritiyOnion's layout, but I don't think you want to add
> it under spool.  That is typically where runtime files are created.
>
> Tyler
>
> --
> Tyler Schoenke
> Network Security Manager
> IT Security Office
> University of Colorado at Boulder
>
> On 8/8/12 9:38 AM, Tom OBrion wrote:
>> Sent this off to the SecurityOnion group, but probably should have
>> sent it here.   Oopsy!
>>
>> Anyway
>>
>> Please....I know I must be doing something noobish...but man, I have
>> tried it 15 ways to Sunday and no love.
>>
>> editing:  /nsm/bro/spool/policy/site/local.bro
>>
>> added "redef cmd_line_bpf_filter = "not src host ipaddress";
>>
>> I want to tweak a tad more based on dst port, but need to at least get
>> the filter working for the IP.
>>
>> I then do a check/install/restart
>>
>> I watch BRO dns.log for the for the IP I added and she shows up.  What
>> the heck am I missing?
>>
>> Any help much appreciated.
>>
>>



-- 
Tom O'Brion
@tobrion

"Life is too short to spend time with people who suck the happy out of you."


From doug.burks at gmail.com  Thu Aug  9 08:33:27 2012
From: doug.burks at gmail.com (Doug Burks)
Date: Thu, 9 Aug 2012 11:33:27 -0400
Subject: [Bro] [security-onion] Re:  Some BPF love....
In-Reply-To: <CANCSO1cTPGKzjQuGuWT9gXQGUFsjySg7g9FObutTecQnyDk27g@mail.gmail.com>
References: <CANCSO1cXHO7PkaUDvA0_r4SEck39cYMUmMTOC5PXHfd5LUV_rQ@mail.gmail.com>
	<5023CB82.7080408@colorado.edu>
	<CANCSO1cTPGKzjQuGuWT9gXQGUFsjySg7g9FObutTecQnyDk27g@mail.gmail.com>
Message-ID: <CAK8kjrAq84AUK=HUiDaPH5D49ryuj2KFmHY87POOC87_enAVBA@mail.gmail.com>

Thanks for finding this documentation bug!  It is now fixed.

If I understand Seth correctly, we won't have to do this anymore in
Bro 2.1 since it will just read our existing bpf.conf.

Thanks,
Doug

On Thu, Aug 9, 2012 at 11:26 AM, Tom OBrion <hammadog at gmail.com> wrote:
> Hey Tyler
>
> Thanks, I was updating it in the spool folder based on the DOC I was
> reading out on the SO groups site.  I thought it was wierd that I
> update in the spool location and not the share location.  Maybe I was
> just reading it wrong in the DOC.  I have been known to skin reading
> and not completely reading it fully.  :)  Anyway, made the updates in
> the location you mentioned and it seems to be working.  I am not using
> your syntax though, I am using this:
>
> redef cmd_line_bpf_filter = "not (host x.x.x.x)";
>
> Worked like a champ.  Now I will tweak to include dest port and should
> be good to go.  Thanks man.  Got me on the right track!
>
> Tom
>
> On Thu, Aug 9, 2012 at 10:38 AM, Tyler T. Schoenke
> <tyler.schoenke at colorado.edu> wrote:
>> I've only briefly tested SecurityOnion, but in vanilla Bro, you would
>> add something like this to local.bro.  That file is located under
>> $BROHOME/share/bro/site.
>>
>> redef restrict_filters += { ["host exemptions"] = "not (host 4.2.2.2)" };
>>
>> I don't know SecuritiyOnion's layout, but I don't think you want to add
>> it under spool.  That is typically where runtime files are created.
>>
>> Tyler
>>
>> --
>> Tyler Schoenke
>> Network Security Manager
>> IT Security Office
>> University of Colorado at Boulder
>>
>> On 8/8/12 9:38 AM, Tom OBrion wrote:
>>> Sent this off to the SecurityOnion group, but probably should have
>>> sent it here.   Oopsy!
>>>
>>> Anyway
>>>
>>> Please....I know I must be doing something noobish...but man, I have
>>> tried it 15 ways to Sunday and no love.
>>>
>>> editing:  /nsm/bro/spool/policy/site/local.bro
>>>
>>> added "redef cmd_line_bpf_filter = "not src host ipaddress";
>>>
>>> I want to tweak a tad more based on dst port, but need to at least get
>>> the filter working for the IP.
>>>
>>> I then do a check/install/restart
>>>
>>> I watch BRO dns.log for the for the IP I added and she shows up.  What
>>> the heck am I missing?
>>>
>>> Any help much appreciated.
>>>
>>>
>
>
>
> --
> Tom O'Brion
> @tobrion
>
> "Life is too short to spend time with people who suck the happy out of you."
>
> --
>
>



-- 
Doug Burks
http://securityonion.blogspot.com


From robin at icir.org  Thu Aug  9 15:23:42 2012
From: robin at icir.org (Robin Sommer)
Date: Thu, 9 Aug 2012 15:23:42 -0700
Subject: [Bro] pcap_next Question
In-Reply-To: <CAG1VgOaxbvmMha2zvoVoBfRHh2gGjAFWEeBhvV59rj0FXOPoVQ@mail.gmail.com>
References: <CAG1VgOa3UnDHHf=P6XYH-YhLb-yaZyA4wBhWRN4=u2bzqkooYQ@mail.gmail.com>
	<20120809060147.174E52C4008@rock.ICSI.Berkeley.EDU>
	<CAG1VgOaxbvmMha2zvoVoBfRHh2gGjAFWEeBhvV59rj0FXOPoVQ@mail.gmail.com>
Message-ID: <20120809222342.GS27086@icir.org>


On Thu, Aug 09, 2012 at 06:43 -0400, Chris Crawford wrote:

> I don't think that is quite right for C++.

Yeah, it throws an exception by default. However, there's a way to
catch a failing new by setting a corresponding handler, see:

    http://www.cplusplus.com/reference/std/new/set_new_handler

The funny thing is that I thought we did that already; and in fact
there's a function bro_new_handler() that reports an out of memory
error and aborts. However, we must have lost the call to the
corresponding set_new_handler() at some point: I don't find
bro_new_handler() used anywhere. So what we should do is add that back
in; I'll do that.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From christopher.p.crawford at gmail.com  Thu Aug  9 15:49:38 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Thu, 9 Aug 2012 18:49:38 -0400
Subject: [Bro] pcap_next Question
In-Reply-To: <20120809222342.GS27086@icir.org>
References: <CAG1VgOa3UnDHHf=P6XYH-YhLb-yaZyA4wBhWRN4=u2bzqkooYQ@mail.gmail.com>
	<20120809060147.174E52C4008@rock.ICSI.Berkeley.EDU>
	<CAG1VgOaxbvmMha2zvoVoBfRHh2gGjAFWEeBhvV59rj0FXOPoVQ@mail.gmail.com>
	<20120809222342.GS27086@icir.org>
Message-ID: <CAG1VgOYifdjQQ0zsxY7AJqNH86nPXAfADaqd6OWjVQSPZR6uPQ@mail.gmail.com>

Oh cool, that seems like a much cleaner approach.

The if-statement that follows still doesn't make sense to me, though:

       if ( ! block )
                reporter->InternalError("out of memory");

If the handler in bro_new_handler() aborts, then this never executes.
If the handler succeeds in making more storage available and returns,
it seems to me that this if block still wouldn't execute because block
would not be null.  If this if-statement did execute, then it seems
like the logic would be off -- you're not out of memory yet, because
the handler just returned with some after a second try.

It seems like a logical place to print an "out of memory" message to
the logs would be just before an abort in the function registered in
bro_new_handler() to handle catching std::bad_alloc errors.

-Chris

On Thu, Aug 9, 2012 at 6:23 PM, Robin Sommer <robin at icir.org> wrote:
>
> On Thu, Aug 09, 2012 at 06:43 -0400, Chris Crawford wrote:
>
>> I don't think that is quite right for C++.
>
> Yeah, it throws an exception by default. However, there's a way to
> catch a failing new by setting a corresponding handler, see:
>
>     http://www.cplusplus.com/reference/std/new/set_new_handler
>
> The funny thing is that I thought we did that already; and in fact
> there's a function bro_new_handler() that reports an out of memory
> error and aborts. However, we must have lost the call to the
> corresponding set_new_handler() at some point: I don't find
> bro_new_handler() used anywhere. So what we should do is add that back
> in; I'll do that.
>
> Robin
>
> --
> Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
> ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From robin at icir.org  Thu Aug  9 16:00:28 2012
From: robin at icir.org (Robin Sommer)
Date: Thu, 9 Aug 2012 16:00:28 -0700
Subject: [Bro] pcap_next Question
In-Reply-To: <CAG1VgOYifdjQQ0zsxY7AJqNH86nPXAfADaqd6OWjVQSPZR6uPQ@mail.gmail.com>
References: <CAG1VgOa3UnDHHf=P6XYH-YhLb-yaZyA4wBhWRN4=u2bzqkooYQ@mail.gmail.com>
	<20120809060147.174E52C4008@rock.ICSI.Berkeley.EDU>
	<CAG1VgOaxbvmMha2zvoVoBfRHh2gGjAFWEeBhvV59rj0FXOPoVQ@mail.gmail.com>
	<20120809222342.GS27086@icir.org>
	<CAG1VgOYifdjQQ0zsxY7AJqNH86nPXAfADaqd6OWjVQSPZR6uPQ@mail.gmail.com>
Message-ID: <20120809230028.GU27086@icir.org>


On Thu, Aug 09, 2012 at 18:49 -0400, Chris Crawford wrote:

> The if-statement that follows still doesn't make sense to me, though:

Yes, agreed.

> It seems like a logical place to print an "out of memory" message to
> the logs would be just before an abort in the function registered in
> bro_new_handler() to handle catching std::bad_alloc errors.

Indeed, that's what should happen once the handler gets installed
(with the caveat that the logging might not work if it needs to
allocate memory itself).

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From vern at icir.org  Thu Aug  9 16:05:46 2012
From: vern at icir.org (Vern Paxson)
Date: Thu, 09 Aug 2012 16:05:46 -0700
Subject: [Bro] pcap_next Question
In-Reply-To: <CAG1VgOYifdjQQ0zsxY7AJqNH86nPXAfADaqd6OWjVQSPZR6uPQ@mail.gmail.com>
	(Thu, 09 Aug 2012 18:49:38 EDT).
Message-ID: <20120809230546.4E31A2C400A@rock.ICSI.Berkeley.EDU>

> The if-statement that follows still doesn't make sense to me, though:

Right, it's vestigial from back when my understanding of new's behavior
was actually correct.  (Now do me a favor and don't explicitly date my
coding skillz from that comment. ;-)

		Vern


From commike at reservoir.com  Thu Aug  9 18:24:24 2012
From: commike at reservoir.com (Alan Commike)
Date: Thu, 09 Aug 2012 18:24:24 -0700
Subject: [Bro] Chimera Language
Message-ID: <502462C8.6030308@reservoir.com>

It was great seeing everyone that was able to attend the Bro workshop.

One tidbit that I wanted to sneak into my talk but didn't get a chance 
to was that Reservoir is a coauthor of a paper with the NSA that was 
presented today at the USENIX Security Conference about Chimera, a 
language and compiler for network traffic cyber security analytics. The 
interesting bits to the Bro community is that the compiler compiles down 
into Bro language. Chimera is a declarative language with semantics 
similar to SQL.

We have a copy of the paper and a preliminary language specification up 
on http://www.chimera-query.org
There's a preliminary compiler based on the specification which we will 
be releasing to the community over the next year.

    ...alan

-- 
Alan Commike
commike at reservoir.com
Reservoir Labs
212.780.0527 x136



From seth at icir.org  Thu Aug  9 19:09:55 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 9 Aug 2012 22:09:55 -0400
Subject: [Bro] Some BPF love....
In-Reply-To: <20120809151509.GO1957@datacomm.albany.edu>
References: <CANCSO1cXHO7PkaUDvA0_r4SEck39cYMUmMTOC5PXHfd5LUV_rQ@mail.gmail.com>
	<5023CB82.7080408@colorado.edu>
	<20120809151509.GO1957@datacomm.albany.edu>
Message-ID: <DB79D8BD-5120-4A81-978E-BC787C11D4B8@icir.org>


On Aug 9, 2012, at 11:15 AM, Justin Azoff <JAzoff at albany.edu> wrote:

> Might also need
> 
> redef PacketFilter::all_packets = F; # don't capture all packets


Thanks for pointing that out!  That bit of poor design is unfortunately still going to remain for 2.1, but it will absolutely be gone for 2.2.  I'll make sure that in the 2.2 release we have good examples for the new way of working with the packet filter framework.  

For anyone making changes to your packet filter now, please keep your changes in one place so that it will be easier to upgrade to 2.2 when that time comes.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Thu Aug  9 19:12:33 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 9 Aug 2012 22:12:33 -0400
Subject: [Bro] [security-onion]  Some BPF love....
In-Reply-To: <CAK8kjrAq84AUK=HUiDaPH5D49ryuj2KFmHY87POOC87_enAVBA@mail.gmail.com>
References: <CANCSO1cXHO7PkaUDvA0_r4SEck39cYMUmMTOC5PXHfd5LUV_rQ@mail.gmail.com>
	<5023CB82.7080408@colorado.edu>
	<CANCSO1cTPGKzjQuGuWT9gXQGUFsjySg7g9FObutTecQnyDk27g@mail.gmail.com>
	<CAK8kjrAq84AUK=HUiDaPH5D49ryuj2KFmHY87POOC87_enAVBA@mail.gmail.com>
Message-ID: <A0EC13D9-2A39-4594-A999-972863876ABC@icir.org>


On Aug 9, 2012, at 11:33 AM, Doug Burks <doug.burks at gmail.com> wrote:

> If I understand Seth correctly, we won't have to do this anymore in
> Bro 2.1 since it will just read our existing bpf.conf.


I'll probably commit a script to a personal repository on github which you can then run on security onion for 2.1.  I don't want to include in the 2.1 release since it will be a little hacky since the rewritten packet filter framework isn't going to be included yet.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From vladg at cmu.edu  Thu Aug  9 22:28:17 2012
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Fri, 10 Aug 2012 05:28:17 +0000
Subject: [Bro] Brownian Demo Site
Message-ID: <1202BE242E080642B0CD0AD0A03E85521A45C8@PGH-MSGMB-03.andrew.ad.cmu.edu>

Hi Everyone,

As some of you may know, I've been working on a web interface for quickly querying your Bro logs called Brownian. For those of you that I got to meet at the Bro Exchange, I just wanted to quickly follow-up and mention that the demo site is now up at: 

http://brownian.bro-ids.org/?time=all

If you weren't able to make it out to the Exchange, the video of my (and everyone else's) presentations will be made available soon.

In either case, if anyone would like to know more, has questions, or feature requests, please feel free to contact me. To find out more about the project itself, please see the GitHub page at: <https://github.com/grigorescu/Brownian>.

Thanks,

  --Vlad


From vladg at cmu.edu  Fri Aug 10 10:22:17 2012
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Fri, 10 Aug 2012 17:22:17 +0000
Subject: [Bro] Announcing an Unofficial Bro Script Repository
Message-ID: <1202BE242E080642B0CD0AD0A03E85521B2100@PGH-MSGMB-03.andrew.ad.cmu.edu>

Hello,

Following up on some ideas from the Exchange, I've created an unofficial git repository for community submitted Bro scripts.

<https://github.com/grigorescu/bro-scripts>

If you have any scripts that you'd like to share with the community, please fork this repo, add your script, and submit a pull request. I know that there are many scripts floating around out there, and I think everyone would be interested to see what others are doing.

Currently, I'll be reviewing all submitted scripts for security vulnerabilities (strictly in regards to not exfiltrating your sensitive Bro data, not introducing new, sensitive data in your logs unless it's clearly documented, that sort of thing).

Obviously, none of these scripts come with any sort of guarantee - however, if you encounter a broken script, you can always fix it and push it back! Please also keep in mind that this is really a temporary solution until a better script repository system is designed.

Feel free to let me know any comments or questions that you may have, or if you can think of a better way to implement this.

Thanks,

  --Vlad





From mcholste at gmail.com  Fri Aug 10 10:37:58 2012
From: mcholste at gmail.com (Martin Holste)
Date: Fri, 10 Aug 2012 12:37:58 -0500
Subject: [Bro] Announcing an Unofficial Bro Script Repository
In-Reply-To: <1202BE242E080642B0CD0AD0A03E85521B2100@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <1202BE242E080642B0CD0AD0A03E85521B2100@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <CANpnLHhcEwq6gXPv4SgMj2-bXxu_Q6OHpe2GNCuoOaX++UupRw@mail.gmail.com>

Nice!  http-exe-bad-attributes.bro alone is worth checking out.  Thanks!

On Fri, Aug 10, 2012 at 12:22 PM, Vlad Grigorescu <vladg at cmu.edu> wrote:
> Hello,
>
> Following up on some ideas from the Exchange, I've created an unofficial git repository for community submitted Bro scripts.
>
> <https://github.com/grigorescu/bro-scripts>
>
> If you have any scripts that you'd like to share with the community, please fork this repo, add your script, and submit a pull request. I know that there are many scripts floating around out there, and I think everyone would be interested to see what others are doing.
>
> Currently, I'll be reviewing all submitted scripts for security vulnerabilities (strictly in regards to not exfiltrating your sensitive Bro data, not introducing new, sensitive data in your logs unless it's clearly documented, that sort of thing).
>
> Obviously, none of these scripts come with any sort of guarantee - however, if you encounter a broken script, you can always fix it and push it back! Please also keep in mind that this is really a temporary solution until a better script repository system is designed.
>
> Feel free to let me know any comments or questions that you may have, or if you can think of a better way to implement this.
>
> Thanks,
>
>   --Vlad
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at icir.org  Fri Aug 10 10:39:23 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 10 Aug 2012 13:39:23 -0400
Subject: [Bro] Announcing an Unofficial Bro Script Repository
In-Reply-To: <1202BE242E080642B0CD0AD0A03E85521B2100@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <1202BE242E080642B0CD0AD0A03E85521B2100@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <77A16810-9D29-44EC-BBD8-1E39693BDACE@icir.org>


On Aug 10, 2012, at 1:22 PM, Vlad Grigorescu <vladg at cmu.edu> wrote:

> Following up on some ideas from the Exchange, I've created an unofficial git repository for community submitted Bro scripts.


Thanks Vlad!  Looking forward to some organization and general cleanup.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From JAzoff at albany.edu  Fri Aug 10 10:54:27 2012
From: JAzoff at albany.edu (Justin Azoff)
Date: Fri, 10 Aug 2012 13:54:27 -0400
Subject: [Bro] Announcing an Unofficial Bro Script Repository
In-Reply-To: <CANpnLHhcEwq6gXPv4SgMj2-bXxu_Q6OHpe2GNCuoOaX++UupRw@mail.gmail.com>
References: <1202BE242E080642B0CD0AD0A03E85521B2100@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<CANpnLHhcEwq6gXPv4SgMj2-bXxu_Q6OHpe2GNCuoOaX++UupRw@mail.gmail.com>
Message-ID: <20120810175427.GW1957@datacomm.albany.edu>

On Fri, Aug 10, 2012 at 12:37:58PM -0500, Martin Holste wrote:
> Nice!  http-exe-bad-attributes.bro alone is worth checking out.  Thanks!

I have a bunch of other ones that could be included(after some cleanups)
here:

https://github.com/justinazoff/bro_scripts/tree/2.0

    active-hosts-metrics.bro
    rogue-access-points.bro
    log-external-dns.bro

are useful.

I want to update log-external-dns to cache the result of lookup_addr and
have another notice type/flag when it doesn't resolve.  After running it
for a while I've found that external dns servers without PTR records are
almost always the really nasty ones.

-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From seth at icir.org  Fri Aug 10 11:26:12 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 10 Aug 2012 14:26:12 -0400
Subject: [Bro] Brownian Demo Site
In-Reply-To: <1202BE242E080642B0CD0AD0A03E85521A45C8@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <1202BE242E080642B0CD0AD0A03E85521A45C8@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <9BA9E7A8-0260-4F85-A68C-F070D368BD79@icir.org>


On Aug 10, 2012, at 1:28 AM, Vlad Grigorescu <vladg at cmu.edu> wrote:

> http://brownian.bro-ids.org/?time=all


I do want to point out some small things about that demo site.  It's hosted on a virtual machine and it seems to be running slower than I would expect.  My experience in other cases with many, many more logs has shown much better performance than that and I still can't explain the slowness except that rendering the logs is taking too long.  There may also be other users hitting the site at the same time as you which could result in even further slow downs.

My recommendation if you are interested in seeing how it really performs would be to try it locally with Vlad's installation instructions:
	https://github.com/grigorescu/Brownian/blob/master/README.md

There are instructions for using the elasticsearch plugin in our beta and git repository master branch here:
	http://git.bro-ids.org/bro.git/blob/HEAD:/doc/logging-elasticsearch.rst
	
In particular, pay attention to the section that talks about this script (for automatically logging to text logs and elastic search at the same time):
	tuning/logs-to-elasticsearch.bro

Have fun and remember that we are declaring the elastic search plugin as "in testing" for the 2.1 release.  Thanks for Brownian Vlad!

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Fri Aug 10 11:46:19 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 10 Aug 2012 14:46:19 -0400
Subject: [Bro] Announcing an Unofficial Bro Script Repository
In-Reply-To: <20120810175427.GW1957@datacomm.albany.edu>
References: <1202BE242E080642B0CD0AD0A03E85521B2100@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<CANpnLHhcEwq6gXPv4SgMj2-bXxu_Q6OHpe2GNCuoOaX++UupRw@mail.gmail.com>
	<20120810175427.GW1957@datacomm.albany.edu>
Message-ID: <03648C4C-614C-4A52-B228-25620E860AC0@icir.org>


On Aug 10, 2012, at 1:54 PM, Justin Azoff <JAzoff at albany.edu> wrote:

>   active-hosts-metrics.bro
>    rogue-access-points.bro
>    log-external-dns.bro


FYI, I think that some of these scripts may require a special branch of bro with extensions to the metrics framework that aren't going to show up until 2.2.  Do any of these scripts you pointed out require that branch Justin?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From JAzoff at albany.edu  Fri Aug 10 12:00:26 2012
From: JAzoff at albany.edu (Justin Azoff)
Date: Fri, 10 Aug 2012 15:00:26 -0400
Subject: [Bro] Announcing an Unofficial Bro Script Repository
In-Reply-To: <03648C4C-614C-4A52-B228-25620E860AC0@icir.org>
References: <1202BE242E080642B0CD0AD0A03E85521B2100@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<CANpnLHhcEwq6gXPv4SgMj2-bXxu_Q6OHpe2GNCuoOaX++UupRw@mail.gmail.com>
	<20120810175427.GW1957@datacomm.albany.edu>
	<03648C4C-614C-4A52-B228-25620E860AC0@icir.org>
Message-ID: <20120810190026.GX1957@datacomm.albany.edu>

On Fri, Aug 10, 2012 at 02:46:19PM -0400, Seth Hall wrote:
> 
> On Aug 10, 2012, at 1:54 PM, Justin Azoff <JAzoff at albany.edu> wrote:
> 
> >   active-hosts-metrics.bro
> >    rogue-access-points.bro
> >    log-external-dns.bro
> 
> 
> FYI, I think that some of these scripts may require a special branch of bro with extensions to the metrics framework that aren't going to show up until 2.2.  Do any of these scripts you pointed out require that branch Justin?
> 
>   .Seth
> 

active-hosts-metrics.bro does, and most of the other metrics scripts in
that repo.

-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From rmkml at yahoo.fr  Fri Aug 10 17:19:36 2012
From: rmkml at yahoo.fr (rmkml)
Date: Sat, 11 Aug 2012 02:19:36 +0200 (CEST)
Subject: [Bro] Emerging Threats signatures on Bro ids ?
Message-ID: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>

Hi,

Anyone interested for supporting / converting Emerging Threats [ET] signatures on Bro IDS ?

- convert on regexp bro format (if threats are easy)

- or better convert to a bro powerful language... (more complex threats)

Not a automatic converter, need (long long) review all signatures for understand threats and use better (bro) converter...

What do you think ?

Im interested if anyone are running futur bro+ET direct feedback... (FP, FN, performance....)

Happy Detect with Bro, Suricata and Snort.
Regards
Rmkml

http://twitter.com/rmkml


From mcholste at gmail.com  Fri Aug 10 15:33:45 2012
From: mcholste at gmail.com (Martin Holste)
Date: Fri, 10 Aug 2012 17:33:45 -0500
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
Message-ID: <CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>

Your best bet would be to try to convert the ET USER_AGENTS signatures
and modify them for inclusion in
https://github.com/grigorescu/bro-scripts/blob/9d59a7a482b068304a2d33a3c9c8dc696c176650/scripts/http-exe-bad-attributes.bro
.  That would be a good start.

On Fri, Aug 10, 2012 at 7:19 PM, rmkml <rmkml at yahoo.fr> wrote:
> Hi,
>
> Anyone interested for supporting / converting Emerging Threats [ET] signatures on Bro IDS ?
>
> - convert on regexp bro format (if threats are easy)
>
> - or better convert to a bro powerful language... (more complex threats)
>
> Not a automatic converter, need (long long) review all signatures for understand threats and use better (bro) converter...
>
> What do you think ?
>
> Im interested if anyone are running futur bro+ET direct feedback... (FP, FN, performance....)
>
> Happy Detect with Bro, Suricata and Snort.
> Regards
> Rmkml
>
> http://twitter.com/rmkml
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From vladg at cmu.edu  Fri Aug 10 16:48:21 2012
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Fri, 10 Aug 2012 23:48:21 +0000
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
Message-ID: <1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>

Hi Rmkml,

As Martin said, that would be a good start, and would provide everyone with some very useful data. Longterm, however, I think that this is a perfect fit for the upcoming intelligence framework. As I understand it, the goal of that framework is to separate the scripting layer from the intelligence layer (so, you have a user-agent analyzer script, which reads good or bad user agents from the intelligence layer. Your script stays nice and clean, and your intelligence can just be presented in a logical way, and be processed by the script into something useful).

Unfortunately, Emerging Threats doesn't present the intelligence in a logical way, and it's preprocessed for Snort. What I'd love to see is ET just provide *data*, and then you have a script to convert it to a format Snort understands, Bro processes it into something it understands, and so on.

tl;dr: I think it'd be very useful to have this data, but I don't think anyone should sink too much time into it until the intel framework comes out.

  --Vlad

On Aug 10, 2012, at 6:33 PM, Martin Holste <mcholste at gmail.com>
 wrote:

> Your best bet would be to try to convert the ET USER_AGENTS signatures
> and modify them for inclusion in
> https://github.com/grigorescu/bro-scripts/blob/9d59a7a482b068304a2d33a3c9c8dc696c176650/scripts/http-exe-bad-attributes.bro
> .  That would be a good start.
> 
> On Fri, Aug 10, 2012 at 7:19 PM, rmkml <rmkml at yahoo.fr> wrote:
>> Hi,
>> 
>> Anyone interested for supporting / converting Emerging Threats [ET] signatures on Bro IDS ?
>> 
>> - convert on regexp bro format (if threats are easy)
>> 
>> - or better convert to a bro powerful language... (more complex threats)
>> 
>> Not a automatic converter, need (long long) review all signatures for understand threats and use better (bro) converter...
>> 
>> What do you think ?
>> 
>> Im interested if anyone are running futur bro+ET direct feedback... (FP, FN, performance....)
>> 
>> Happy Detect with Bro, Suricata and Snort.
>> Regards
>> Rmkml
>> 
>> http://twitter.com/rmkml
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From seth at icir.org  Fri Aug 10 18:18:34 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 10 Aug 2012 21:18:34 -0400
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>


On Aug 10, 2012, at 7:48 PM, Vlad Grigorescu <vladg at cmu.edu> wrote:

> tl;dr: I think it'd be very useful to have this data, but I don't think anyone should sink too much time into it until the intel framework comes out.


You hit that perfectly.  I'm working hard on getting the intelligence framework ready for some people to start testing soon hopefully.  It's in memory tuning now to reduce worker memory usage on clusters.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From vallentin at icir.org  Fri Aug 10 22:16:43 2012
From: vallentin at icir.org (Matthias Vallentin)
Date: Fri, 10 Aug 2012 22:16:43 -0700
Subject: [Bro] Announcing an Unofficial Bro Script Repository
In-Reply-To: <1202BE242E080642B0CD0AD0A03E85521B2100@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <1202BE242E080642B0CD0AD0A03E85521B2100@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <CADTpMNazW0SC0WoYzGm4_ZSJgR-ySOffSe25hRjruEgC=9HM_Q@mail.gmail.com>

> Please also keep in mind that this is really a temporary solution until a better script repository system is designed.

Good to see you're approaching this still actively. We're at a point
where having the BPAN would significantly help bootstrapping community
efforts like yours. Also, there is already a concrete propsal of how
this could look like:

    http://www.bro-ids.org/development/projects/cban.html

Any feedback on this would be highly valuable, so that we can go ahead
and incorporate into the design.

    Matthias

(For completeness, here's my assorted Bro script repository:
https://github.com/mavam/brospects)


From rmkml at yahoo.fr  Sat Aug 11 17:41:49 2012
From: rmkml at yahoo.fr (rmkml)
Date: Sun, 12 Aug 2012 02:41:49 +0200 (CEST)
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
Message-ID: <alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>

Hi,

Ok first alpha release on yesterday update (open-gpl) Emerging Threats signatures :

  http://88.191.140.111/et_bro2_10aug.bro
(contains only 13 signatures)

Im interested if you have comments/feedback/flame/performance/FP/FN please.

Tested on bro v2.0 with:
  bro -C -r test.pcap et_bro2_10aug


Futur work:
I have a small pb on this bro powerful language:
-I have used a global variables (sid2015596...) for http_header because my test on pcap fire four times for each signature.

Regards
Rmkml

http://twitter.com/rmkml


From rmkml at yahoo.fr  Sun Aug 12 16:01:59 2012
From: rmkml at yahoo.fr (rmkml)
Date: Mon, 13 Aug 2012 01:01:59 +0200 (CEST)
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
Message-ID: <alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>

Hi,

ok please found second alpha release update (open-gpl) Emerging Threats signatures :

  http://88.191.140.111/et_bro2_11aug.bro
  (contains only 37 signatures, fixed one bug on previous rule set)

Im always interested if you have comments/feedback/flame/performance/FP/FN please.

Futur work:
1) I have a small pb on this bro powerful language:
   -I have used a global variables (sid2015596...) for http_header because my test on pcap fire four times for each signature.
2) find case insensitive more "simplify" regexp ?
3) adding local_net / external_net...

Regards
Rmkml

http://twitter.com/rmkml


From seth at icir.org  Sun Aug 12 18:52:55 2012
From: seth at icir.org (Seth Hall)
Date: Sun, 12 Aug 2012 21:52:55 -0400
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
Message-ID: <2C64451F-43FD-4135-B6E9-889B569FF5E8@icir.org>


On Aug 12, 2012, at 7:01 PM, rmkml <rmkml at yahoo.fr> wrote:

> Im always interested if you have comments/feedback/flame/performance/FP/FN please.


Have you tried running Bro on live traffic with this script?  I looked through it briefly and it seems like it would severely impact performance.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From rmkml at yahoo.fr  Mon Aug 13 09:38:22 2012
From: rmkml at yahoo.fr (rmkml at yahoo.fr)
Date: Mon, 13 Aug 2012 16:38:22 +0000 (UTC)
Subject: [Bro] RE : Re:  Emerging Threats signatures on Bro ids ?
Message-ID: <428348865.5966.1344875983877.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>

An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120813/c1415cb3/attachment.html 

From jsiwek at illinois.edu  Mon Aug 13 09:54:33 2012
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 13 Aug 2012 16:54:33 +0000
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CAG1VgOahewH-CWjgvpq_v+Gpy7a-Tuo8d+QKihgx3K=Qo_Z2Cg@mail.gmail.com>
References: <CAMPgRd5us7tC=tZoaRkG8Q+ZeQQt7Vip-SStR7J1=dYbPLdO6A@mail.gmail.com>
	<CAG1VgOYrXeHrayWPGcDBF+rmYv1wFkszqG9cg5ymMGRvcWJxMQ@mail.gmail.com>
	<CAMPgRd5buSM19Xk0+aLBM9qioHewZJUOcpnvBOdFXB7cOtiz3Q@mail.gmail.com>
	<CAMPgRd7mu0Pcd9b4tSwM1uyCAWtDt8-Q2EbqWf-XUjgV3AWRuA@mail.gmail.com>
	<CAG1VgOa0TRRH8de0vJR0FQ44Cs0cCEgYftDZ5vJZtLr31AVPwg@mail.gmail.com>
	<CAG1VgOahewH-CWjgvpq_v+Gpy7a-Tuo8d+QKihgx3K=Qo_Z2Cg@mail.gmail.com>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D10FDFF05C@CITESMBX2.ad.uillinois.edu>


> 
> I was under the impression that I needed to configure node.cfg from the default:
> 
> [bro]
> type=standalone
> host=localhost
> interface=eth0
> 
> To something that makes more sense for my environment
> 
> [bro]
> type=standalone
> host=1.2.3.4
> interface=eth0
> 
> For some reason, when I do this, it causes broctl to take a very long
> time to return from the status command and the number of peers
> reported is ??? and not the expected 0.  Configuring my host to an IP
> address also causes CPU to spike to about 100%.

For a standalone config, localhost is probably preferable -- node.cfg is just specifying how nodes can contact/communicate with each other, so going over loopback or at least a private IP space might be what most people aim to do.

That 'peer' column of `broctl status` is obtained from the manager attempting to send a status request event to the Bro worker instance and expecting a status response event back. The failure cases in which "???" are displayed are:

(1) The manager's broccoli connection to the Bro peer fails
(2) The manager times out sending a status request event
(3) The manager times out receiving a status response event

To see which is the case, you could exit all BroControl shells, add "Debug=1" to your etc/broctl.cfg and then try `broctl status` again.  There will be a spool/debug.log in which you should find one of these messages:

broccoli: cannot connect
broccoli: timeout during send
broccoli: timeout during receive

I'm guessing you're just running into (1) because a firewall is now blocking the connection.  In that case, I think the timeout length for connect(2) can be pretty long (~2mins), but not sure if it also generally results in high CPU usage.

For (2) or (3), event status is polled once every second for a maximum of "CommTimeout" seconds (default 10 secs), so probably not CPU intensive, but can end up taking a long time for cluster setups especially since the manager queries node status serially.

    Jon


From seth at icir.org  Mon Aug 13 10:00:05 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 13 Aug 2012 13:00:05 -0400
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <428348865.5966.1344875983877.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>
References: <428348865.5966.1344875983877.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>
Message-ID: <C611DC6B-B6DE-4BF4-8A62-D88D02512989@icir.org>


On Aug 13, 2012, at 12:38 PM, "rmkml at yahoo.fr" <rmkml at yahoo.fr> wrote:

> Anyone tested please?
> What's performance impact? (only 33sigs)

There are a number of potential and definite problems.

 - For each http_request event, you are doing a lot of if & if else statements which *could* impact performance.

 - For each http header you are similarly doing a lot of if statements which will almost certainly cause a performance impact.  Also, you are accessing collected state in the c$http record when you should probably be using the name and value variables directly.  If you want to look through data before things are logged, your best bet is to use the HTTP::log_http logging framework event.

 - Again, lots of if statements for every dns request is probably going to have a severe performance impact.

 - For every single chunk of http entity data, you are running lots of if statements with pattern conditions again.

 - Handling the packet_contents event at all is generally really bad.  The auto-generated documentation even comments on the fact that using that event is not really feasible for any traffic volume:
	http://www.bro-ids.org/documentation/scripts/base/event.bif.html?highlight=packet_contents#id-packet_contents


This is one of the interesting things about Bro.  Due to it primarily being a programming language, you can absolutely do things that will negatively impact performance and break other analysis.  So like any other language you have to constantly be aware of what you are doing and the potential impacts.  We are actively working now to make it possible for you and others to do these detections more easily and with less potential performance impact.  Unfortunately we're still at the very beginning of a newly-found operational security engineering focus so this stuff is taking a bit longer than most people would like (me included!).

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Mon Aug 13 10:02:40 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 13 Aug 2012 13:02:40 -0400
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <428348865.5966.1344875983877.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>
References: <428348865.5966.1344875983877.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>
Message-ID: <0F046BF2-0200-4184-AAC8-53C710C72195@icir.org>


On Aug 13, 2012, at 12:38 PM, rmkml at yahoo.fr wrote:

> This is why I need feedback please.

Oh!  I forgot to include an alternate approach I thought of.  If you are still interested in going down this route, could you start by pulling out malicious software user-agents from the ET signatures?  That's something that would fit well and easily into Bro right now and into the intelligence framework in the future.

What do you think about that?  We can certainly start small with very well defined goals and move from there.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From gc355804 at ohio.edu  Mon Aug 13 11:26:10 2012
From: gc355804 at ohio.edu (Gilbert Clark)
Date: Mon, 13 Aug 2012 14:26:10 -0400
Subject: [Bro] RE : Re:  Emerging Threats signatures on Bro ids ?
In-Reply-To: <428348865.5966.1344875983877.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>
References: <428348865.5966.1344875983877.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>
Message-ID: <502946C2.4050506@ohio.edu>

Hi:

You might try obtaining a few rather large traces and running bro 
against those traces with '-r'.  Record how long it takes to process 
these traces both without the changes you've made and with the changes 
you've made.  The difference in these two times might give you a rough 
idea of how your modifications impact bro's performance (given observed 
traffic similar to that of the analyzed trace).

--Gilbert

On 8/13/2012 12:38 PM, rmkml at yahoo.fr wrote:
> Hi Seth,
> I don't have quick internet access, only a *dsl access.
> This is why I need feedback please.
> Anyone tested please?
> What's performance impact? (only 33sigs)
> Regards
> Rmkml
>
>
>
>
> Seth Hall a ?crit :
>
>
> On Aug 12, 2012, at 7:01 PM, rmkml <rmkml at yahoo.fr> wrote:
>
> > Im always interested if you have 
> comments/feedback/flame/performance/FP/FN please.
>
>
> Have you tried running Bro on live traffic with this script?  I looked 
> through it briefly and it seems like it would severely impact performance.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120813/92a14cff/attachment.html 

From rmkml at yahoo.fr  Mon Aug 13 14:29:32 2012
From: rmkml at yahoo.fr (rmkml)
Date: Mon, 13 Aug 2012 23:29:32 +0200 (CEST)
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <C611DC6B-B6DE-4BF4-8A62-D88D02512989@icir.org>
References: <428348865.5966.1344875983877.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>
	<C611DC6B-B6DE-4BF4-8A62-D88D02512989@icir.org>
Message-ID: <alpine.LFD.2.01.1208132238220.1860@lenovo.localdomain>

Thx you for reply Seth,

ok I have started very small bench on my local network: (wget, one cnx)

-without et_bro2_11aug, download http at --limit-rate=85m, bro cpu around (top) 40%-45%

-all sigs et_bro2_11aug, download http at --limit-rate=85m, bro cpu around (top) 75%-90%

-disabled only "packet_contents" on et_bro2_11aug, download http at --limit-rate=85m, bro cpu around (top) 75%-90%

-disabled only "entity_data" on et_bro2_11aug, download http at --limit-rate=85m, bro cpu around (top) 75%-90%

-disabled only "dns_request" on et_bro2_11aug, download http at --limit-rate=85m, bro cpu around (top) 75%-90%

-disabled only "http_header" on et_bro2_11aug, download http at --limit-rate=85m, bro cpu around (top) 75%-90%

-disabled only "http_request" on et_bro2_11aug, download http at --limit-rate=85m, bro cpu around (top) 75%-90%

well, no special sig penalty.


I have discovered one pb on my case: in ids mode, bro not fire immediatly, after 5mn not fire,  fire only when I kill bro, it's possible to fire immediatly on my rule set please?

Best Regards
Rmkml



On Mon, 13 Aug 2012, Seth Hall wrote:

>
> On Aug 13, 2012, at 12:38 PM, "rmkml at yahoo.fr" <rmkml at yahoo.fr> wrote:
>
>> Anyone tested please?
>> What's performance impact? (only 33sigs)
>
> There are a number of potential and definite problems.
>
> - For each http_request event, you are doing a lot of if & if else statements which *could* impact performance.
>
> - For each http header you are similarly doing a lot of if statements which will almost certainly cause a performance impact.
>   Also, you are accessing collected state in the c$http record when you should probably be using the name and value variables directly.
>   If you want to look through data before things are logged, your best bet is to use the HTTP::log_http logging framework event.
>
> - Again, lots of if statements for every dns request is probably going to have a severe performance impact.
>
> - For every single chunk of http entity data, you are running lots of if statements with pattern conditions again.
> 
> - Handling the packet_contents event at all is generally really bad.  The auto-generated documentation even comments on the fact that using that event is not really feasible for any traffic volume:
> 	http://www.bro-ids.org/documentation/scripts/base/event.bif.html?highlight=packet_contents#id-packet_contents
>
>
> This is one of the interesting things about Bro.  Due to it primarily being a programming language, you can absolutely do things that will negatively impact performance and break other analysis.  So like any other language you have to constantly be aware of what you are doing and the potential impacts.  We are actively working now to make it possible for you and others to do these detections more easily and with less potential performance impact.  Unfortunately we're still at the very beginning of a newly-found operational security engineering focus so this stuff is taking a bit longer than most people would like (me included!).
>
>  .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>


From rmkml at yahoo.fr  Mon Aug 13 14:33:03 2012
From: rmkml at yahoo.fr (rmkml)
Date: Mon, 13 Aug 2012 23:33:03 +0200 (CEST)
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <0F046BF2-0200-4184-AAC8-53C710C72195@icir.org>
References: <428348865.5966.1344875983877.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>
	<0F046BF2-0200-4184-AAC8-53C710C72195@icir.org>
Message-ID: <alpine.LFD.2.01.1208132331580.1860@lenovo.localdomain>

ok Im look on user-agent ET sigs.
Regards
Rmkml


On Mon, 13 Aug 2012, Seth Hall wrote:

>
> On Aug 13, 2012, at 12:38 PM, rmkml at yahoo.fr wrote:
>
>> This is why I need feedback please.
>
> Oh!  I forgot to include an alternate approach I thought of.  If you are still interested in going down this route, could you start by pulling out malicious software user-agents from the ET signatures?
>  That's something that would fit well and easily into Bro right now and into the intelligence framework in the future.
>
> What do you think about that?  We can certainly start small with very well defined goals and move from there.
>
>  .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>


From seth at icir.org  Mon Aug 13 12:36:15 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 13 Aug 2012 15:36:15 -0400
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208132238220.1860@lenovo.localdomain>
References: <428348865.5966.1344875983877.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>
	<C611DC6B-B6DE-4BF4-8A62-D88D02512989@icir.org>
	<alpine.LFD.2.01.1208132238220.1860@lenovo.localdomain>
Message-ID: <1890CF35-42BA-499A-9E5A-D2597A387077@icir.org>


On Aug 13, 2012, at 5:29 PM, rmkml <rmkml at yahoo.fr> wrote:

> ok I have started very small bench on my local network: (wget, one cnx)


You need to do this with a decent sized tracefile (>1GB) of mixed traffic and run Bro with the "time" command to see how long it takes for it to analyze the full file.  I suspect the performance degradation will become much more obvious there.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jsiwek at illinois.edu  Mon Aug 13 13:37:16 2012
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 13 Aug 2012 20:37:16 +0000
Subject: [Bro] Version: 2.0-907 -- Bro manager memory exhaustion
In-Reply-To: <CAG1VgOa0TRRH8de0vJR0FQ44Cs0cCEgYftDZ5vJZtLr31AVPwg@mail.gmail.com>
References: <CAMPgRd5us7tC=tZoaRkG8Q+ZeQQt7Vip-SStR7J1=dYbPLdO6A@mail.gmail.com>
	<CAG1VgOYrXeHrayWPGcDBF+rmYv1wFkszqG9cg5ymMGRvcWJxMQ@mail.gmail.com>
	<CAMPgRd5buSM19Xk0+aLBM9qioHewZJUOcpnvBOdFXB7cOtiz3Q@mail.gmail.com>
	<CAMPgRd7mu0Pcd9b4tSwM1uyCAWtDt8-Q2EbqWf-XUjgV3AWRuA@mail.gmail.com>
	<CAG1VgOa0TRRH8de0vJR0FQ44Cs0cCEgYftDZ5vJZtLr31AVPwg@mail.gmail.com>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D10FDFF64F@CITESMBX2.ad.uillinois.edu>


> I have the following in my local.bro file:
> 
> redef SMTP::generate_md5 += /image.*/;
> redef HTTP::generate_md5 += /image.*/;
...
> 
> Using broctl's top and a little trial and error, I can see that these
> lines are the cause of my high CPU usage.  It also causes higher
> memory usage as well, but memory usage always climbs and never gets
> smaller.  I don't know if these lines are responsible for just higher
> memory usage in general, or whether they are also responsible gradual
> climb in memory.  It appears that memory gradually climbs even without
> these lines, but I haven't had enough time to test that idea. 

In general, the digest BiFs don't look like they leak, but if there is not a md5_hash_finish() for each corresponding md5_hash_init(), that could lead to growth of some internal state over time.  The base scripts all attempt to clean up any md5_hash_init()'s with a corresponding md5_hash_finish(), but I'm not confident all edge cases are covered.

If you have any other local changes, you might see if there's a difference running with them rather than just the vanilla bro scripts -- it can be easy to add something which causes too much state to accumulate over time.  Another quick check is to look for any errors in reporter.log -- currently interpreter exceptions due to scripting errors will not abort bro, but do cause a memory leak.  Otherwise, it might be easiest for you to start looking into using a memory profiling tool (e.g. valgrind, gperftools) to try to locate the problem more definitely.

    Jon


From Huiping.Song at ultra-3eti.com  Mon Aug 13 13:56:55 2012
From: Huiping.Song at ultra-3eti.com (Huiping Song)
Date: Mon, 13 Aug 2012 20:56:55 +0000
Subject: [Bro] Support SNMP and MODBUS/TCP Protocols?
Message-ID: <394E2A9510AB1642987C5AADE63C547607584076@RockMX01.rock.corp>

We like to use Bro to monitor and analyze SNMP and MODBUS/TCP traffic in industrial control networks.  Does the latest version of Bro support SNMP, MODBUS/TCP and any other industrial control protocols?

If not currently supported, what are the typical steps to make bro to support a new protocol?

Thank you,
Huiping

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120813/cdc44ecc/attachment.html 

From rmkml at yahoo.fr  Mon Aug 13 17:13:38 2012
From: rmkml at yahoo.fr (rmkml)
Date: Tue, 14 Aug 2012 02:13:38 +0200 (CEST)
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208132331580.1860@lenovo.localdomain>
References: <428348865.5966.1344875983877.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>
	<0F046BF2-0200-4184-AAC8-53C710C72195@icir.org>
	<alpine.LFD.2.01.1208132331580.1860@lenovo.localdomain>
Message-ID: <alpine.LFD.2.01.1208140209100.1860@lenovo.localdomain>

starting hard works...

question please: it's possible to detect POST and uri (/abc) and argument (arg=test) ?
example:
  POST /abc HTTP/1.0
  ...
  \r\n
  \r\n
  arg=test

not work but like:
("POST"==c$http$method)&&(/\/abc/ in c$http$uri)&&(/arg\=test/ in c$http$body????)

Regards
Rmkml


On Mon, 13 Aug 2012, rmkml wrote:

> ok Im look on user-agent ET sigs.
> Regards
> Rmkml
>
>
> On Mon, 13 Aug 2012, Seth Hall wrote:
>
>> 
>> On Aug 13, 2012, at 12:38 PM, rmkml at yahoo.fr wrote:
>> 
>>> This is why I need feedback please.
>> 
>> Oh!  I forgot to include an alternate approach I thought of.  If you are 
>> still interested in going down this route, could you start by pulling out 
>> malicious software user-agents from the ET signatures?
>>  That's something that would fit well and easily into Bro right now and 
>> into the intelligence framework in the future.
>> 
>> What do you think about that?  We can certainly start small with very well 
>> defined goals and move from there.
>>
>>  .Seth
>> 
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro-ids.org/
>> 
>> 
>


From rmkml at yahoo.fr  Mon Aug 13 17:56:58 2012
From: rmkml at yahoo.fr (rmkml)
Date: Tue, 14 Aug 2012 02:56:58 +0200 (CEST)
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
Message-ID: <alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>

Hi,

ok please found third alpha release update (open-gpl) Emerging Threats signatures :

  http://88.191.140.111/et_bro2_12aug.bro
  (contains only 54 signatures, begin User-Agent sigs)

Im always interested if you have comments/feedback/flame/performance/FP/FN please.
Enable or disable variable in bro script reduce number sigs (et_currents, et_enabled, et_dns, et_trojan...).

Futur work:
1) I have a small pb on this bro powerful language:
   -I have used a global variables (sid2015596...) for http_header because my test on pcap fire four times for each signature.
2) find case insensitive more "simplify" regexp ?
3) adding local_net / external_net...
4) how to match POST http_method with argument ?

Regards
Rmkml

http://twitter.com/rmkml


From robin at icir.org  Mon Aug 13 16:21:16 2012
From: robin at icir.org (Robin Sommer)
Date: Mon, 13 Aug 2012 16:21:16 -0700
Subject: [Bro] Support SNMP and MODBUS/TCP Protocols?
In-Reply-To: <394E2A9510AB1642987C5AADE63C547607584076@RockMX01.rock.corp>
References: <394E2A9510AB1642987C5AADE63C547607584076@RockMX01.rock.corp>
Message-ID: <20120813232116.GT73973@icir.org>


On Mon, Aug 13, 2012 at 20:56 +0000, you wrote:

> We like to use Bro to monitor and analyze SNMP and MODBUS/TCP traffic
> in industrial control networks.  Does the latest version of Bro
> support SNMP, MODBUS/TCP and any other industrial control protocols?

No, not yet. We've a prototype of Modbus support (and DNP3), which
will likely make it into Bro 2.2. Nobody is working on SNMP yet though
as far as I know.

> If not currently supported, what are the typical steps to make bro to
> support a new protocol?

The best way is to use our binpac parser generator, see here for a
skeleton:

    http://www.bro-ids.org/development/binpac-sample-analyzer.html

Also take a look at the existing analyzers in src/*.pac.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From Huiping.Song at ultra-3eti.com  Tue Aug 14 07:04:58 2012
From: Huiping.Song at ultra-3eti.com (Huiping Song)
Date: Tue, 14 Aug 2012 14:04:58 +0000
Subject: [Bro] Support SNMP and MODBUS/TCP Protocols?
In-Reply-To: <20120813232116.GT73973@icir.org>
References: <394E2A9510AB1642987C5AADE63C547607584076@RockMX01.rock.corp>
	<20120813232116.GT73973@icir.org>
Message-ID: <394E2A9510AB1642987C5AADE63C547607584113@RockMX01.rock.corp>

Hi Robin,

Thanks for the updates.  Good to know that there will be a prototype of MODBUS support in Bro 2.2.  Any estimates about the release timeline for Bro 2.2?

Can the prototype of MODBUS support also be customized to work with Bro 2.0 quickly?  We are eager to experiment using Bro to monitor and analyze MODBUS/TCP traffic. :)

Best regards,
Huiping


-----Original Message-----
From: Robin Sommer [mailto:robin at icir.org] 
Sent: Monday, August 13, 2012 7:21 PM
To: Huiping Song
Cc: bro at bro-ids.org
Subject: Re: [Bro] Support SNMP and MODBUS/TCP Protocols?


On Mon, Aug 13, 2012 at 20:56 +0000, you wrote:

> We like to use Bro to monitor and analyze SNMP and MODBUS/TCP traffic 
> in industrial control networks.  Does the latest version of Bro 
> support SNMP, MODBUS/TCP and any other industrial control protocols?

No, not yet. We've a prototype of Modbus support (and DNP3), which will likely make it into Bro 2.2. Nobody is working on SNMP yet though as far as I know.

> If not currently supported, what are the typical steps to make bro to 
> support a new protocol?

The best way is to use our binpac parser generator, see here for a
skeleton:

    http://www.bro-ids.org/development/binpac-sample-analyzer.html

Also take a look at the existing analyzers in src/*.pac.

Robin

--
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org



From hlin33 at illinois.edu  Tue Aug 14 07:37:44 2012
From: hlin33 at illinois.edu (Hui Lin (Hugo) )
Date: Tue, 14 Aug 2012 09:37:44 -0500
Subject: [Bro] Support SNMP and MODBUS/TCP Protocols?
In-Reply-To: <2240a523d6c946b793fca2338dbeefee@CITESHT1.ad.uillinois.edu>
References: <394E2A9510AB1642987C5AADE63C547607584076@RockMX01.rock.corp>
	<20120813232116.GT73973@icir.org>
	<2240a523d6c946b793fca2338dbeefee@CITESHT1.ad.uillinois.edu>
Message-ID: <CAKq214noi0oCNrabfAEfYon0UB7jKzvh5nPVP9mB2HhT7pJMNA@mail.gmail.com>

Hi, Huiping,

We are working on merging the Modbus at this moment. I think merging Modbus
would not take too long as the code size of it is not that big.

Also in case that you want to build your own analyzer in binpac, here is
some sample codes:
http://www.bro-ids.org/development/binpac-sample-analyzer.html

FYI, binpac can easily handle application layer protocol directly over TCP
or UDP. But with complex protocol which includes session layer or
presentation layer, u may need to do some modifications on Bro's code to
integrate binpac code.

Hope this help.

Best,

Hui


On Tue, Aug 14, 2012 at 9:04 AM, Huiping Song
<Huiping.Song at ultra-3eti.com>wrote:

> Hi Robin,
>
> Thanks for the updates.  Good to know that there will be a prototype of
> MODBUS support in Bro 2.2.  Any estimates about the release timeline for
> Bro 2.2?
>
> Can the prototype of MODBUS support also be customized to work with Bro
> 2.0 quickly?  We are eager to experiment using Bro to monitor and analyze
> MODBUS/TCP traffic. :)
>
> Best regards,
> Huiping
>
>
> -----Original Message-----
> From: Robin Sommer [mailto:robin at icir.org]
> Sent: Monday, August 13, 2012 7:21 PM
> To: Huiping Song
> Cc: bro at bro-ids.org
> Subject: Re: [Bro] Support SNMP and MODBUS/TCP Protocols?
>
>
> On Mon, Aug 13, 2012 at 20:56 +0000, you wrote:
>
> > We like to use Bro to monitor and analyze SNMP and MODBUS/TCP traffic
> > in industrial control networks.  Does the latest version of Bro
> > support SNMP, MODBUS/TCP and any other industrial control protocols?
>
> No, not yet. We've a prototype of Modbus support (and DNP3), which will
> likely make it into Bro 2.2. Nobody is working on SNMP yet though as far as
> I know.
>
> > If not currently supported, what are the typical steps to make bro to
> > support a new protocol?
>
> The best way is to use our binpac parser generator, see here for a
> skeleton:
>
>     http://www.bro-ids.org/development/binpac-sample-analyzer.html
>
> Also take a look at the existing analyzers in src/*.pac.
>
> Robin
>
> --
> Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
> ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Hui Lin
PhD Candidate, Research Assistant
Electrical and Computer Engineering Department
University of Illinois at Urbana-Champaign
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120814/4e964c15/attachment.html 

From Huiping.Song at ultra-3eti.com  Tue Aug 14 15:03:43 2012
From: Huiping.Song at ultra-3eti.com (Huiping Song)
Date: Tue, 14 Aug 2012 22:03:43 +0000
Subject: [Bro] Support SNMP and MODBUS/TCP Protocols?
In-Reply-To: <CAKq214noi0oCNrabfAEfYon0UB7jKzvh5nPVP9mB2HhT7pJMNA@mail.gmail.com>
References: <394E2A9510AB1642987C5AADE63C547607584076@RockMX01.rock.corp>
	<20120813232116.GT73973@icir.org>
	<2240a523d6c946b793fca2338dbeefee@CITESHT1.ad.uillinois.edu>
	<CAKq214noi0oCNrabfAEfYon0UB7jKzvh5nPVP9mB2HhT7pJMNA@mail.gmail.com>
Message-ID: <394E2A9510AB1642987C5AADE63C5476075843FD@RockMX01.rock.corp>

Hi Hui,

We look forward to testing the MODBUS and DNP3 analyzers as soon as they are available.

We are also interested in protocols for building automation and control networks, such as BACnet.   Is there anyone currently working (or plan to work) on BACnet protocol analyzer?   We may try to learn/experiment  building a BACnet protocol analyzer using the BinPAC parser generator.  This looks to be a daunting task at the moment.

Thanks for the help.

Huiping


From: Hui Lin (Hugo) [mailto:hlin33 at illinois.edu]
Sent: Tuesday, August 14, 2012 10:38 AM
To: Huiping Song
Cc: Robin Sommer; bro at bro-ids.org
Subject: Re: [Bro] Support SNMP and MODBUS/TCP Protocols?

Hi, Huiping,

We are working on merging the Modbus at this moment. I think merging Modbus would not take too long as the code size of it is not that big.

Also in case that you want to build your own analyzer in binpac, here is some sample codes:
http://www.bro-ids.org/development/binpac-sample-analyzer.html

FYI, binpac can easily handle application layer protocol directly over TCP or UDP. But with complex protocol which includes session layer or presentation layer, u may need to do some modifications on Bro's code to integrate binpac code.

Hope this help.

Best,

Hui


On Tue, Aug 14, 2012 at 9:04 AM, Huiping Song <Huiping.Song at ultra-3eti.com<mailto:Huiping.Song at ultra-3eti.com>> wrote:
Hi Robin,

Thanks for the updates.  Good to know that there will be a prototype of MODBUS support in Bro 2.2.  Any estimates about the release timeline for Bro 2.2?

Can the prototype of MODBUS support also be customized to work with Bro 2.0 quickly?  We are eager to experiment using Bro to monitor and analyze MODBUS/TCP traffic. :)

Best regards,
Huiping


-----Original Message-----
From: Robin Sommer [mailto:robin at icir.org<mailto:robin at icir.org>]
Sent: Monday, August 13, 2012 7:21 PM
To: Huiping Song
Cc: bro at bro-ids.org<mailto:bro at bro-ids.org>
Subject: Re: [Bro] Support SNMP and MODBUS/TCP Protocols?


On Mon, Aug 13, 2012 at 20:56 +0000, you wrote:

> We like to use Bro to monitor and analyze SNMP and MODBUS/TCP traffic
> in industrial control networks.  Does the latest version of Bro
> support SNMP, MODBUS/TCP and any other industrial control protocols?

No, not yet. We've a prototype of Modbus support (and DNP3), which will likely make it into Bro 2.2. Nobody is working on SNMP yet though as far as I know.

> If not currently supported, what are the typical steps to make bro to
> support a new protocol?

The best way is to use our binpac parser generator, see here for a
skeleton:

    http://www.bro-ids.org/development/binpac-sample-analyzer.html

Also take a look at the existing analyzers in src/*.pac.

Robin

--
Robin Sommer * Phone +1 (510) 722-6541<tel:%2B1%20%28510%29%20722-6541> * robin at icir.org<mailto:robin at icir.org>
ICSI/LBNL    * Fax   +1 (510) 666-2956<tel:%2B1%20%28510%29%20666-2956> *   www.icir.org<http://www.icir.org>

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



--
Hui Lin
PhD Candidate, Research Assistant
Electrical and Computer Engineering Department
University of Illinois at Urbana-Champaign

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120814/686660da/attachment.html 

From rmkml at yahoo.fr  Tue Aug 14 17:16:06 2012
From: rmkml at yahoo.fr (rmkml)
Date: Wed, 15 Aug 2012 02:16:06 +0200 (CEST)
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208132238220.1860@lenovo.localdomain>
References: <428348865.5966.1344875983877.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>
	<C611DC6B-B6DE-4BF4-8A62-D88D02512989@icir.org>
	<alpine.LFD.2.01.1208132238220.1860@lenovo.localdomain>
Message-ID: <alpine.LFD.2.01.1208150157270.1970@lenovo.localdomain>

Hi,
ok I have advance my performance penalty, simply disable "packet_contents" and "entity_data", results my performance go to 40%-45%...
(download one file size 1.9Go with wget multiple times)
good news.

Anyone tested partial ET open-gpl on live trafic please ?
Regards
Rmkml


On Mon, 13 Aug 2012, rmkml wrote:

> Thx you for reply Seth,
>
> ok I have started very small bench on my local network: (wget, one cnx)
>
> -without et_bro2_11aug, download http at --limit-rate=85m, bro cpu around 
> (top) 40%-45%
>
> -all sigs et_bro2_11aug, download http at --limit-rate=85m, bro cpu around 
> (top) 75%-90%
>
> -disabled only "packet_contents" on et_bro2_11aug, download http at 
> --limit-rate=85m, bro cpu around (top) 75%-90%
>
> -disabled only "entity_data" on et_bro2_11aug, download http at 
> --limit-rate=85m, bro cpu around (top) 75%-90%
>
> -disabled only "dns_request" on et_bro2_11aug, download http at 
> --limit-rate=85m, bro cpu around (top) 75%-90%
>
> -disabled only "http_header" on et_bro2_11aug, download http at 
> --limit-rate=85m, bro cpu around (top) 75%-90%
>
> -disabled only "http_request" on et_bro2_11aug, download http at 
> --limit-rate=85m, bro cpu around (top) 75%-90%
>
> well, no special sig penalty.
>
>
> I have discovered one pb on my case: in ids mode, bro not fire immediatly, 
> after 5mn not fire,  fire only when I kill bro, it's possible to fire 
> immediatly on my rule set please?
>
> Best Regards
> Rmkml
>
>
>
> On Mon, 13 Aug 2012, Seth Hall wrote:
>
>> 
>> On Aug 13, 2012, at 12:38 PM, "rmkml at yahoo.fr" <rmkml at yahoo.fr> wrote:
>> 
>>> Anyone tested please?
>>> What's performance impact? (only 33sigs)
>> 
>> There are a number of potential and definite problems.
>> 
>> - For each http_request event, you are doing a lot of if & if else 
>> statements which *could* impact performance.
>> 
>> - For each http header you are similarly doing a lot of if statements which 
>> will almost certainly cause a performance impact.
>>   Also, you are accessing collected state in the c$http record when you 
>> should probably be using the name and value variables directly.
>>   If you want to look through data before things are logged, your best bet 
>> is to use the HTTP::log_http logging framework event.
>> 
>> - Again, lots of if statements for every dns request is probably going to 
>> have a severe performance impact.
>> 
>> - For every single chunk of http entity data, you are running lots of if 
>> statements with pattern conditions again.
>> 
>> - Handling the packet_contents event at all is generally really bad.  The 
>> auto-generated documentation even comments on the fact that using that 
>> event is not really feasible for any traffic volume:
>> 	http://www.bro-ids.org/documentation/scripts/base/event.bif.html?highlight=packet_contents#id-packet_contents
>> 
>> 
>> This is one of the interesting things about Bro.  Due to it primarily being 
>> a programming language, you can absolutely do things that will negatively 
>> impact performance and break other analysis.  So like any other language 
>> you have to constantly be aware of what you are doing and the potential 
>> impacts.  We are actively working now to make it possible for you and 
>> others to do these detections more easily and with less potential 
>> performance impact.  Unfortunately we're still at the very beginning of a 
>> newly-found operational security engineering focus so this stuff is taking 
>> a bit longer than most people would like (me included!).
>>
>>  .Seth
>> 
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro-ids.org/
>> 
>> 
>


From rmkml at yahoo.fr  Tue Aug 14 19:05:29 2012
From: rmkml at yahoo.fr (rmkml)
Date: Wed, 15 Aug 2012 04:05:29 +0200 (CEST)
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
	<alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>
Message-ID: <alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>

Hi,

ok please found Four alpha release update (open-gpl) Emerging Threats  signatures :

  http://88.191.140.111/et_bro2_13aug.bro
   -sorry, no new signatures
   -moved print to notice alarm realtime mode
   -disabled by default sig performance penalty with et_performancepenalty variable

Im always interested if you have comments/feedback/flame/performance/FP/FN  please.
Enable or disable variable in bro script reduce number sigs (et_currents, et_enabled, et_dns, et_trojan...).

Futur work:
1) I have a small pb on this bro powerful language:
   -I have used a global variables (sid2015596...) for http_header because my test on pcap fire four times for each signature.
2) find case insensitive more "simplify" regexp ?
3) adding local_net / external_net...
4) how to match POST http_method with argument ?

Regards
Rmkml

http://twitter.com/rmkml


From rmkml at yahoo.fr  Wed Aug 15 09:14:12 2012
From: rmkml at yahoo.fr (rmkml)
Date: Wed, 15 Aug 2012 18:14:12 +0200 (CEST)
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
	<alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>
	<alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>
Message-ID: <alpine.LFD.2.01.1208151803180.1865@lenovo.localdomain>

Hi,
Im continue to update (user-agent actually) converting (open-gpl) Emerging Threats signatures:
  http://88.191.140.111/et_bro2_14aug_pb.bro
It's work.

but when I de-comment/enable these two lines on et_bro2_14aug_pb.bro:
  228: else if ( (/[gG][oO][oO][gG][lL][eE][bB][oO][tT]/ in c$http$user_agent) && sid2015529 && (et_currents || et_useragent) )
  229: NOTICE([$conn=c, $note=EmergingThreats, $msg=fmt("[1:2015529:1] ET CURRENT_EVENTS Googlebot User-Agent Outbound (likely malicious)")]);

bro produce an error:
  bro20 -C -r testbro.pcap et_bro2_14aug_pb
  error in policy/et_bro2_14aug_pb.bro, line 229: memory exhausted, at or near "("

Only for test, continue to enabled lines 228 and 229, but comment/disable previous lines 224 and 225, bro fire on my test...
maybe it's a internal memory related pb on bro ?

Anyone known this pb and how to fix please?
Regards
Rmkml

http://twitter.com/rmkml


From dina at ICSI.Berkeley.EDU  Wed Aug 15 07:28:47 2012
From: dina at ICSI.Berkeley.EDU (Dina Hadziosmanovic)
Date: Wed, 15 Aug 2012 16:28:47 +0200
Subject: [Bro] Support SNMP and MODBUS/TCP Protocols?
In-Reply-To: <394E2A9510AB1642987C5AADE63C5476075843FD@RockMX01.rock.corp>
References: <394E2A9510AB1642987C5AADE63C547607584076@RockMX01.rock.corp>	<20120813232116.GT73973@icir.org>	<2240a523d6c946b793fca2338dbeefee@CITESHT1.ad.uillinois.edu>	<CAKq214noi0oCNrabfAEfYon0UB7jKzvh5nPVP9mB2HhT7pJMNA@mail.gmail.com>
	<394E2A9510AB1642987C5AADE63C5476075843FD@RockMX01.rock.corp>
Message-ID: <000d01cd7af2$4aa40570$dfec1050$@icsi.berkeley.edu>

Hi Huiping,

 

To the best of my(our) knowledge no one is working on BACnet protocol
analyzer nor its in near future plans of the people I know (mainly because
its building automation protocol and not process automation). But we also
might have some data for testing BACnet in near future, so if you manage to
have the analyzer running, we might be able to help with more date for
validation purposes.   

 

Good luck:)

 

__

Dina Hadziosmanovic

 

Distributed and Embedded Security Group, University of Twente, The
Netherlands

Email: dina.hadziosmanovic at utwente.nl

Homepage: http://dies.ewi.utwente.nl/~hadziosmanovicd

Address: Faculty of EEMCS, P.O. Box 217, 7500 AE, Enschede, The Netherlands

Office: Zilvering building, room 3032

Phone: +31 (0)53 489 2542 

 

 

From: Huiping Song [mailto:Huiping.Song at ultra-3eti.com] 
Sent: woensdag 15 augustus 2012 0:04
To: Hui Lin (Hugo) 
Cc: bro at bro-ids.org
Subject: Re: [Bro] Support SNMP and MODBUS/TCP Protocols?

 

Hi Hui,

 

We look forward to testing the MODBUS and DNP3 analyzers as soon as they are
available.

 

We are also interested in protocols for building automation and control
networks, such as BACnet.   Is there anyone currently working (or plan to
work) on BACnet protocol analyzer?   We may try to learn/experiment
building a BACnet protocol analyzer using the BinPAC parser generator.  This
looks to be a daunting task at the moment.

 

Thanks for the help.

 

Huiping

 

 

From: Hui Lin (Hugo) [mailto:hlin33 at illinois.edu] 
Sent: Tuesday, August 14, 2012 10:38 AM
To: Huiping Song
Cc: Robin Sommer; bro at bro-ids.org
Subject: Re: [Bro] Support SNMP and MODBUS/TCP Protocols?

 

Hi, Huiping,

 

We are working on merging the Modbus at this moment. I think merging Modbus
would not take too long as the code size of it is not that big.  

 

Also in case that you want to build your own analyzer in binpac, here is
some sample codes:

http://www.bro-ids.org/development/binpac-sample-analyzer.html 

 

FYI, binpac can easily handle application layer protocol directly over TCP
or UDP. But with complex protocol which includes session layer or
presentation layer, u may need to do some modifications on Bro's code to
integrate binpac code.

 

Hope this help. 

 

Best,

 

Hui 

 

 

On Tue, Aug 14, 2012 at 9:04 AM, Huiping Song <Huiping.Song at ultra-3eti.com>
wrote:

Hi Robin,

Thanks for the updates.  Good to know that there will be a prototype of
MODBUS support in Bro 2.2.  Any estimates about the release timeline for Bro
2.2?

Can the prototype of MODBUS support also be customized to work with Bro 2.0
quickly?  We are eager to experiment using Bro to monitor and analyze
MODBUS/TCP traffic. :)

Best regards,
Huiping



-----Original Message-----
From: Robin Sommer [mailto:robin at icir.org]
Sent: Monday, August 13, 2012 7:21 PM
To: Huiping Song
Cc: bro at bro-ids.org
Subject: Re: [Bro] Support SNMP and MODBUS/TCP Protocols?


On Mon, Aug 13, 2012 at 20:56 +0000, you wrote:

> We like to use Bro to monitor and analyze SNMP and MODBUS/TCP traffic
> in industrial control networks.  Does the latest version of Bro
> support SNMP, MODBUS/TCP and any other industrial control protocols?

No, not yet. We've a prototype of Modbus support (and DNP3), which will
likely make it into Bro 2.2. Nobody is working on SNMP yet though as far as
I know.

> If not currently supported, what are the typical steps to make bro to
> support a new protocol?

The best way is to use our binpac parser generator, see here for a
skeleton:

    http://www.bro-ids.org/development/binpac-sample-analyzer.html

Also take a look at the existing analyzers in src/*.pac.

Robin

--
Robin Sommer * Phone +1 (510) 722-6541 <tel:%2B1%20%28510%29%20722-6541>  *
robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 <tel:%2B1%20%28510%29%20666-2956>  *
www.icir.org

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro





 

-- 
Hui Lin

PhD Candidate, Research Assistant
Electrical and Computer Engineering Department
University of Illinois at Urbana-Champaign

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120815/5ffa186e/attachment.html 

From Alex.Tarter at ultra-3eti.com  Wed Aug 15 14:02:14 2012
From: Alex.Tarter at ultra-3eti.com (Alex Tarter)
Date: Wed, 15 Aug 2012 21:02:14 +0000
Subject: [Bro] HELP!
In-Reply-To: <20120723211607.GI72491@icir.org>
References: <EEA086D8173F2F4E80C71CC221ED77D2A9262A@RockMX01.rock.corp>
	<CAL_2yna4pahFoVFZ5HXRCW_EmENFB9cCtRU48LL2M-96zPRuTQ@mail.gmail.com>
	<20120723211607.GI72491@icir.org>
Message-ID: <EEA086D8173F2F4E80C71CC221ED77D2AF94C0@RockMX01.rock.corp>

Robin,

I was wondering if you could help us out. We've been hitting our head
against the wall trying to get Bro doing what we need and we're running out
of time. We need to get a simple demo done by the end of the month that we
thought would be simple to do in Bro but is fast becoming a nightmare!

I know you guys are busy, but could you assist?

What we want to do is simple:
1. Track the amount of TCP traffic over the course of an hour and log it
2. If the amount of traffic over one hour goes above a certain amount then
raise an alarm - hopefully spawn a process to send an SNMP trap rather than
send an email
3. Record the netflow info of each connection in a log

It's that simple!

We probably sound like idiots, but for some reason we can't work out how to
do it. Anything  you could do to point us on the right direction would be
great.

If we could possibly have a telecom as well, then we'd be ecstatic :)

Much obliged, and I hope your Bro-Exchange went well

Alex
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5352 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120815/31ba739f/attachment.bin 

From rmkml at yahoo.fr  Wed Aug 15 16:17:38 2012
From: rmkml at yahoo.fr (rmkml)
Date: Thu, 16 Aug 2012 01:17:38 +0200 (CEST)
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
	<alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>
	<alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>
Message-ID: <alpine.LFD.2.01.1208160110570.1865@lenovo.localdomain>

Hi,

ok please found Five alpha release update (open-gpl) Emerging Threats  signatures :

  http://88.191.140.111/et_bro2_14aug.bro
   -bypassed "memory exhausted" pb with if loop (w/o else if) at this time
   -72 new signatures (total 124)
   -use different pattern matching regexp or not
   -disabled by default sig performance penalty with et_performancepenalty variable

Im always interested if you have comments/feedback/flame/performance/FP/FN  please.
Enable or disable variable in bro script reduce number sigs (et_useragent, et_malware, et_currents, et_dns, et_trojan...).

Futur work:
1) I have a small pb on this bro powerful language:
   -I have used a global variables (sid2015596...) for http_header because my test on pcap fire four times for each signature.
2) find case insensitive more "simplify" regexp ?
3) adding local_net / external_net...
4) how to match POST http_method with argument ?

Regards
Rmkml

http://twitter.com/rmkml


From sconzo at visiblerisk.com  Wed Aug 15 15:02:15 2012
From: sconzo at visiblerisk.com (Mike Sconzo)
Date: Wed, 15 Aug 2012 17:02:15 -0500
Subject: [Bro] HELP!
In-Reply-To: <EEA086D8173F2F4E80C71CC221ED77D2AF94C0@RockMX01.rock.corp>
References: <EEA086D8173F2F4E80C71CC221ED77D2A9262A@RockMX01.rock.corp>
	<CAL_2yna4pahFoVFZ5HXRCW_EmENFB9cCtRU48LL2M-96zPRuTQ@mail.gmail.com>
	<20120723211607.GI72491@icir.org>
	<EEA086D8173F2F4E80C71CC221ED77D2AF94C0@RockMX01.rock.corp>
Message-ID: <CAGZoo6GPZmXCebs95=ea4BHT0fubku-+gsU0eB7=0iuaim1Evg@mail.gmail.com>

I'm probably way off base here, but since you mention netflow, why not use it?

On Wed, Aug 15, 2012 at 4:02 PM, Alex Tarter <Alex.Tarter at ultra-3eti.com> wrote:
> Robin,
>
> I was wondering if you could help us out. We've been hitting our head
> against the wall trying to get Bro doing what we need and we're running out
> of time. We need to get a simple demo done by the end of the month that we
> thought would be simple to do in Bro but is fast becoming a nightmare!
>
> I know you guys are busy, but could you assist?
>
> What we want to do is simple:
> 1. Track the amount of TCP traffic over the course of an hour and log it
> 2. If the amount of traffic over one hour goes above a certain amount then
> raise an alarm - hopefully spawn a process to send an SNMP trap rather than
> send an email
> 3. Record the netflow info of each connection in a log
>
> It's that simple!
>
> We probably sound like idiots, but for some reason we can't work out how to
> do it. Anything  you could do to point us on the right direction would be
> great.
>
> If we could possibly have a telecom as well, then we'd be ecstatic :)
>
> Much obliged, and I hope your Bro-Exchange went well
>
> Alex
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
cat ~/.bash_history > documentation.txt


From slagell at illinois.edu  Wed Aug 15 15:24:27 2012
From: slagell at illinois.edu (Slagell, Adam J)
Date: Wed, 15 Aug 2012 22:24:27 +0000
Subject: [Bro] HELP!
In-Reply-To: <CAGZoo6GPZmXCebs95=ea4BHT0fubku-+gsU0eB7=0iuaim1Evg@mail.gmail.com>
References: <EEA086D8173F2F4E80C71CC221ED77D2A9262A@RockMX01.rock.corp>
	<CAL_2yna4pahFoVFZ5HXRCW_EmENFB9cCtRU48LL2M-96zPRuTQ@mail.gmail.com>
	<20120723211607.GI72491@icir.org>
	<EEA086D8173F2F4E80C71CC221ED77D2AF94C0@RockMX01.rock.corp>,
	<CAGZoo6GPZmXCebs95=ea4BHT0fubku-+gsU0eB7=0iuaim1Evg@mail.gmail.com>
Message-ID: <3026FA41-9A9C-42EA-9D41-D87BBDE89579@illinois.edu>

Or use the conn.log

Sent from my mobile

On Aug 15, 2012, at 5:03 PM, "Mike Sconzo" <sconzo at visiblerisk.com> wrote:

> I'm probably way off base here, but since you mention netflow, why not use it?
> 
> On Wed, Aug 15, 2012 at 4:02 PM, Alex Tarter <Alex.Tarter at ultra-3eti.com> wrote:
>> Robin,
>> 
>> I was wondering if you could help us out. We've been hitting our head
>> against the wall trying to get Bro doing what we need and we're running out
>> of time. We need to get a simple demo done by the end of the month that we
>> thought would be simple to do in Bro but is fast becoming a nightmare!
>> 
>> I know you guys are busy, but could you assist?
>> 
>> What we want to do is simple:
>> 1. Track the amount of TCP traffic over the course of an hour and log it
>> 2. If the amount of traffic over one hour goes above a certain amount then
>> raise an alarm - hopefully spawn a process to send an SNMP trap rather than
>> send an email
>> 3. Record the netflow info of each connection in a log
>> 
>> It's that simple!
>> 
>> We probably sound like idiots, but for some reason we can't work out how to
>> do it. Anything  you could do to point us on the right direction would be
>> great.
>> 
>> If we could possibly have a telecom as well, then we'd be ecstatic :)
>> 
>> Much obliged, and I hope your Bro-Exchange went well
>> 
>> Alex
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> 
> -- 
> cat ~/.bash_history > documentation.txt
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 



From seth at icir.org  Wed Aug 15 19:36:56 2012
From: seth at icir.org (Seth Hall)
Date: Wed, 15 Aug 2012 22:36:56 -0400
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208160110570.1865@lenovo.localdomain>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
	<alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>
	<alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>
	<alpine.LFD.2.01.1208160110570.1865@lenovo.localdomain>
Message-ID: <10FA4B25-BBCD-49EB-8D21-7E82CB5CF1E6@icir.org>


On Aug 15, 2012, at 7:17 PM, rmkml <rmkml at yahoo.fr> wrote:

> ok please found Five alpha release update (open-gpl) Emerging Threats  signatures :


I think there are some fundamental issues at play here, and integrating the EmergingThreats signatures in this manner is probably not the right way to go.

Some comments on the script:

    - That et_performancepenalty variable won't really help since just handling the packet_content event is what will cause most of the overhead.  

    - Your other variables such as sid2014029, et_trojan, and et_useragent are not actually providing any benefit because you aren't ordering the conditions correctly to allow for short circuiting.

    - You are accessing c$http$user_agent inside of http_header event handlers which will have several problems by itself.  That variable won't be filled out until the user-agent header is seen which means you will be filling the reporter log with a lot of error messages and possibly causing memory leaks because the event handler will be failing due to an attempt to access a null field in the record.  It's also something that could just be checked at a later point (like in http_message_done).


Those three big issues were just from a quick glance through the code.  There are better and more flexible ways of approaching this, but a big part of the problem is the way the intelligence from emerging threats is distributed is not suitable as-is for Bro right now.

The best thing that could come out of our community now is guidance for EmergingThreats on how to provide their data in a way that is less product specific.  The signatures are written for Snort and Suricata; trying to jam them into Bro without thinking hard about the problem is probably going to be a waste of effort unfortunately.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From robin at icir.org  Thu Aug 16 10:46:22 2012
From: robin at icir.org (Robin Sommer)
Date: Thu, 16 Aug 2012 10:46:22 -0700
Subject: [Bro] Support SNMP and MODBUS/TCP Protocols?
In-Reply-To: <394E2A9510AB1642987C5AADE63C547607584113@RockMX01.rock.corp>
References: <394E2A9510AB1642987C5AADE63C547607584076@RockMX01.rock.corp>
	<20120813232116.GT73973@icir.org>
	<394E2A9510AB1642987C5AADE63C547607584113@RockMX01.rock.corp>
Message-ID: <20120816174622.GA61350@icir.org>


On Tue, Aug 14, 2012 at 14:04 +0000, you wrote:

> Thanks for the updates.  Good to know that there will be a prototype
> of MODBUS support in Bro 2.2.  Any estimates about the release
> timeline for Bro 2.2?

The actual release will still take a bit, maybe around the end of the
year. However, I'm hoping to have initial experimental support merged
into git master rather soon when 2.2 development starts (once the 2.1
release it out later this month, if all goes well).

> Can the prototype of MODBUS support also be customized to work with
> Bro 2.0 quickly? We are eager to experiment using Bro to monitor and
> analyze MODBUS/TCP traffic. :)

Once it's in git master, backporting to 2.0 or 2.1 shouldn't be
difficult (if still needed then). If you guys (or anybody else here)
could help us testing, that would be much appreciated.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From seth at icir.org  Thu Aug 16 10:54:57 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 16 Aug 2012 13:54:57 -0400
Subject: [Bro] SSN detection script
Message-ID: <4481F54F-820E-4B85-B7CD-CCCC4642E016@icir.org>

I finally sat down and updated a script I had been promising someone for a long time.  It will detect US Social Security Numbers on the network. 

	https://github.com/sethhall/ssn-exposure

Right now it only runs on 2.1+ (beta release or git master branch) because I integrated the input framework into it.  Make sure you follow the installation and configuration directions closely:
	https://github.com/sethhall/ssn-exposure#readme

Any feedback would be great!

Thanks,
  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jonkman at emergingthreats.net  Thu Aug 16 11:28:09 2012
From: jonkman at emergingthreats.net (Matt Jonkman)
Date: Thu, 16 Aug 2012 14:28:09 -0400
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <10FA4B25-BBCD-49EB-8D21-7E82CB5CF1E6@icir.org>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
	<alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>
	<alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>
	<alpine.LFD.2.01.1208160110570.1865@lenovo.localdomain>
	<10FA4B25-BBCD-49EB-8D21-7E82CB5CF1E6@icir.org>
Message-ID: <CAMHk8WmVFGP3AE_2CBa7OCogv9rZ+R6AcJ_SS70MseHeF2ETsQ@mail.gmail.com>

On Wed, Aug 15, 2012 at 10:36 PM, Seth Hall <seth at icir.org> wrote:
> I think there are some fundamental issues at play here, and integrating the EmergingThreats signatures in this manner is probably not the right way to go.
>
> Some comments on the script:
>
>     - That et_performancepenalty variable won't really help since just handling the packet_content event is what will cause most of the overhead.
>
>     - Your other variables such as sid2014029, et_trojan, and et_useragent are not actually providing any benefit because you aren't ordering the conditions correctly to allow for short circuiting.
>
>     - You are accessing c$http$user_agent inside of http_header event handlers which will have several problems by itself.  That variable won't be filled out until the user-agent header is seen which means you will be filling the reporter log with a lot of error messages and possibly causing memory leaks because the event handler will be failing due to an attempt to access a null field in the record.  It's also something that could just be checked at a later point (like in http_message_done).
>
>
> Those three big issues were just from a quick glance through the code.  There are better and more flexible ways of approaching this, but a big part of the problem is the way the intelligence from emerging threats is distributed is not suitable as-is for Bro right now.
>
> The best thing that could come out of our community now is guidance for EmergingThreats on how to provide their data in a way that is less product specific.  The signatures are written for Snort and Suricata; trying to jam them into Bro without thinking hard about the problem is probably going to be a waste of effort unfortunately.

We're definitely all for that! We already distribute our rules in
forms other than Snort and Suricata for a few customers. We're set up
well on the backend to manage multiple languages.

I have to claim significant ignorance of how the Bro language looks
and is structured. But very eager to dive in and see what we can do.

We tried this years ago as I mentioned, and the impact we had on
performance wasn't good. What is essentially the approach we need to
take to put the same intel into a form Bro can use effectively? Do we
have to not think "one bro sig == one suricata sig"?

Thanks Seth!

Matt


>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From mcholste at gmail.com  Thu Aug 16 11:55:10 2012
From: mcholste at gmail.com (Martin Holste)
Date: Thu, 16 Aug 2012 13:55:10 -0500
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <CAMHk8WmVFGP3AE_2CBa7OCogv9rZ+R6AcJ_SS70MseHeF2ETsQ@mail.gmail.com>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
	<alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>
	<alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>
	<alpine.LFD.2.01.1208160110570.1865@lenovo.localdomain>
	<10FA4B25-BBCD-49EB-8D21-7E82CB5CF1E6@icir.org>
	<CAMHk8WmVFGP3AE_2CBa7OCogv9rZ+R6AcJ_SS70MseHeF2ETsQ@mail.gmail.com>
Message-ID: <CANpnLHhYFkF5Xq1EF2iC5hY_d+HmVzD3UsCCsuXaHD3FYEGz7Q@mail.gmail.com>

So, here's the intel feed I'd want:
{
  host:<some bad hostname pattern, e.g. 'example.com'>
  uri: <some bad URI pattern, e.g. 'in.cgi'>
  uri_params: <array of URI parameters that constitutes "badness",
e.g. [ 'id', 'os', 'affid' ]
  headers: <hash of header content of badness, e.g. { 'user-agent': 'Presto' },
  etc...
}

As you can probably see, yara would be a great fit for something like this.

On Thu, Aug 16, 2012 at 1:28 PM, Matt Jonkman
<jonkman at emergingthreats.net> wrote:
> On Wed, Aug 15, 2012 at 10:36 PM, Seth Hall <seth at icir.org> wrote:
>> I think there are some fundamental issues at play here, and integrating the EmergingThreats signatures in this manner is probably not the right way to go.
>>
>> Some comments on the script:
>>
>>     - That et_performancepenalty variable won't really help since just handling the packet_content event is what will cause most of the overhead.
>>
>>     - Your other variables such as sid2014029, et_trojan, and et_useragent are not actually providing any benefit because you aren't ordering the conditions correctly to allow for short circuiting.
>>
>>     - You are accessing c$http$user_agent inside of http_header event handlers which will have several problems by itself.  That variable won't be filled out until the user-agent header is seen which means you will be filling the reporter log with a lot of error messages and possibly causing memory leaks because the event handler will be failing due to an attempt to access a null field in the record.  It's also something that could just be checked at a later point (like in http_message_done).
>>
>>
>> Those three big issues were just from a quick glance through the code.  There are better and more flexible ways of approaching this, but a big part of the problem is the way the intelligence from emerging threats is distributed is not suitable as-is for Bro right now.
>>
>> The best thing that could come out of our community now is guidance for EmergingThreats on how to provide their data in a way that is less product specific.  The signatures are written for Snort and Suricata; trying to jam them into Bro without thinking hard about the problem is probably going to be a waste of effort unfortunately.
>
> We're definitely all for that! We already distribute our rules in
> forms other than Snort and Suricata for a few customers. We're set up
> well on the backend to manage multiple languages.
>
> I have to claim significant ignorance of how the Bro language looks
> and is structured. But very eager to dive in and see what we can do.
>
> We tried this years ago as I mentioned, and the impact we had on
> performance wasn't good. What is essentially the approach we need to
> take to put the same intel into a form Bro can use effectively? Do we
> have to not think "one bro sig == one suricata sig"?
>
> Thanks Seth!
>
> Matt
>
>
>>
>>   .Seth
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro-ids.org/
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at icir.org  Thu Aug 16 12:09:45 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 16 Aug 2012 15:09:45 -0400
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <CAMHk8WmVFGP3AE_2CBa7OCogv9rZ+R6AcJ_SS70MseHeF2ETsQ@mail.gmail.com>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
	<alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>
	<alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>
	<alpine.LFD.2.01.1208160110570.1865@lenovo.localdomain>
	<10FA4B25-BBCD-49EB-8D21-7E82CB5CF1E6@icir.org>
	<CAMHk8WmVFGP3AE_2CBa7OCogv9rZ+R6AcJ_SS70MseHeF2ETsQ@mail.gmail.com>
Message-ID: <19BD89A9-6108-424A-ACCC-060A2A441C00@icir.org>


On Aug 16, 2012, at 2:28 PM, Matt Jonkman <jonkman at emergingthreats.net> wrote:

> We tried this years ago as I mentioned, and the impact we had on
> performance wasn't good. What is essentially the approach we need to
> take to put the same intel into a form Bro can use effectively? Do we
> have to not think "one bro sig == one suricata sig"?


It's even a bit further than that I'm afraid.  The problem is that in the case of many of your rules you have some intelligence in them, but it's encoded with the implicit assumption that you are just scanning a byte stream (in most cases at least).  

Since I work best in very concrete term, I'll give some examples of signatures and how they could be reapplied into general intelligence that we could more easily consume?

alert tcp $EXTERNAL_NET any -> $SMTP_SERVERS 25 (msg:"ET CURRENT_EVENTS DHL Spam Inbound"; flow:established,to_server; content:"name=|22|DHL"; nocase; content:".zip|22|"; within:68; nocase; pcre:"/name=\x22DHL(\s|_|\-)?[a-z0-9\-_\.\s]{0,63}\.zip\x22/i"; reference:url,doc.emergingthreats.net/2010148; classtype:trojan-activity; sid:2010148; rev:12;)

What that rule is really doing is looking for file names matching the regular expression?
	/^DHL(\s|_|\-)?[a-z0-9\-_\.\s]{0,63}\.zip$/

The first version of the intelligence framework in Bro won't support regular expressions unfortunately, but hopefully it will eventually.  The data would be included into Bro like this (this is made up right now, just to get the idea across):

	[$pattern=/^DHL(\s|_|\-)?[a-z0-9\-_\.\s]{0,63}\.zip$/, $subtype=Intel::FILENAME, $expected_in=Intel::EMAIL]

If you had a full filename to match it might look like this?

	[$str="DHL.zip", $subtype=Intel::FILENAME, $expected_in=Intel::EMAIL]

By feeding in intelligence this way we can suddenly reuse that information to start doing these matches in other protocols and in ways that you didn't originally expect.


Another example:

#alert http $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET CURRENT_EVENTS TROJAN Likely TDSS Download (197.exe)"; flow:established,to_server; content:"GET "; depth:4; nocase; uricontent:"/codec/197.exe"; nocase; reference:url,doc.emergingthreats.net/2010056; classtype:trojan-activity; sid:2010056; rev:3;)

This would be:
	[$glob="*/codec/197.exe", $subtype=Intel::URL, $expected_in=Intel::URL]
	


I will say there are plenty of examples in your set now that we don't yet have a great answer for, but we're considering how to make those work as well.

  .Seth




--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Thu Aug 16 12:20:10 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 16 Aug 2012 15:20:10 -0400
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <CANpnLHhYFkF5Xq1EF2iC5hY_d+HmVzD3UsCCsuXaHD3FYEGz7Q@mail.gmail.com>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
	<alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>
	<alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>
	<alpine.LFD.2.01.1208160110570.1865@lenovo.localdomain>
	<10FA4B25-BBCD-49EB-8D21-7E82CB5CF1E6@icir.org>
	<CAMHk8WmVFGP3AE_2CBa7OCogv9rZ+R6AcJ_SS70MseHeF2ETsQ@mail.gmail.com>
	<CANpnLHhYFkF5Xq1EF2iC5hY_d+HmVzD3UsCCsuXaHD3FYEGz7Q@mail.gmail.com>
Message-ID: <57C079C4-0447-43EF-B0F6-3B3F5F083F05@icir.org>


On Aug 16, 2012, at 2:55 PM, Martin Holste <mcholste at gmail.com> wrote:

> So, here's the intel feed I'd want:
> {
>  host:<some bad hostname pattern, e.g. 'example.com'>
>  uri: <some bad URI pattern, e.g. 'in.cgi'>
>  uri_params: <array of URI parameters that constitutes "badness",
> e.g. [ 'id', 'os', 'affid' ]
>  headers: <hash of header content of badness, e.g. { 'user-agent': 'Presto' },
>  etc...
> }
> 
> As you can probably see, yara would be a great fit for something like this.


Haha.  That's actually fairly nice and similar to how Bro's existing signature language works already (we have a number of special keywords besides "payload"), but the problem that I see is where you know a file name that you know is being used by malicious actors and you'd like to watch for that filename anywhere.  You're really looking to just insert the filename as intelligence and watch everywhere that file names are found.

You do make a good point though that reapplying our signature language to intelligence correlation might be a good idea.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From mcholste at gmail.com  Thu Aug 16 12:30:26 2012
From: mcholste at gmail.com (Martin Holste)
Date: Thu, 16 Aug 2012 14:30:26 -0500
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <57C079C4-0447-43EF-B0F6-3B3F5F083F05@icir.org>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
	<alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>
	<alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>
	<alpine.LFD.2.01.1208160110570.1865@lenovo.localdomain>
	<10FA4B25-BBCD-49EB-8D21-7E82CB5CF1E6@icir.org>
	<CAMHk8WmVFGP3AE_2CBa7OCogv9rZ+R6AcJ_SS70MseHeF2ETsQ@mail.gmail.com>
	<CANpnLHhYFkF5Xq1EF2iC5hY_d+HmVzD3UsCCsuXaHD3FYEGz7Q@mail.gmail.com>
	<57C079C4-0447-43EF-B0F6-3B3F5F083F05@icir.org>
Message-ID: <CANpnLHg6peHe2cOc-2EjZpE8Jdubo9u_XahR397XNU5EDgtbQg@mail.gmail.com>

If you start with the intel that filename x=bad, then you could have
an internal Bro generation that extends the filename to URI.  So, some
of the URI information would be specific to URI patterns etc., but
some would be autogenerated at runtime by Bro:

pseudo-spec:
{
  type: FILE,
  filename:foo
}

Synthesizes this additional check:

{
  type: HTTP,
  uri: foo
}

On Thu, Aug 16, 2012 at 2:20 PM, Seth Hall <seth at icir.org> wrote:
>
> On Aug 16, 2012, at 2:55 PM, Martin Holste <mcholste at gmail.com> wrote:
>
>> So, here's the intel feed I'd want:
>> {
>>  host:<some bad hostname pattern, e.g. 'example.com'>
>>  uri: <some bad URI pattern, e.g. 'in.cgi'>
>>  uri_params: <array of URI parameters that constitutes "badness",
>> e.g. [ 'id', 'os', 'affid' ]
>>  headers: <hash of header content of badness, e.g. { 'user-agent': 'Presto' },
>>  etc...
>> }
>>
>> As you can probably see, yara would be a great fit for something like this.
>
>
> Haha.  That's actually fairly nice and similar to how Bro's existing signature language works already (we have a number of special keywords besides "payload"), but the problem that I see is where you know a file name that you know is being used by malicious actors and you'd like to watch for that filename anywhere.  You're really looking to just insert the filename as intelligence and watch everywhere that file names are found.
>
> You do make a good point though that reapplying our signature language to intelligence correlation might be a good idea.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>



From rmkml at yahoo.fr  Thu Aug 16 15:42:42 2012
From: rmkml at yahoo.fr (rmkml)
Date: Fri, 17 Aug 2012 00:42:42 +0200 (CEST)
Subject: [Bro] bro signature http-request double encoded cause FN ?
Message-ID: <alpine.LFD.2.01.1208170033390.1844@lenovo.localdomain>

Hi,
ok it's long time I don't worked on bro signature "language",
but Im back today and Im start few tests:

0) web test without encoding : /abc
OK, detected by  http-request /.*\/abc.*/

1) http utf8 simple encoded like /ab%63
OK, detected by  http-request /.*\/abc.*/

2) http utf8 double encoded like /ab%2563
NOT detected by  http-request /.*\/abc.*/

Anyone confirm this ?
Maybe need switch a variable, where ?

Regards
Rmkml

http://twitter.com/rmkml


From christopher.p.crawford at gmail.com  Thu Aug 16 14:32:28 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Thu, 16 Aug 2012 17:32:28 -0400
Subject: [Bro] Converting a Bro Script to A New Stream
Message-ID: <CAG1VgOab88X87hCP01PXtT3n7hJkhnY=Mf7kRa_YfNV5+V3Vaw@mail.gmail.com>

I have a short bro script that I wrote that hooks the DNS log
(http://www.bro-ids.org/documentation/logging.html#hooking-into-the-logging).
 Each time a DNS::log_dns event fires, if a specific IP is in
rec$answers, the script prints out rec$ts, rec$uid, rec$id$orig_h, and
rec$query.

I want the entries from the script to go to their own log, though.  I
am struggling to figure out how to make that work.  Based on the
documentation for logging, it looks like I'd need to define a new
Stream to create a new log file.
(http://www.bro-ids.org/documentation/logging.html#adding-streams)

Here's my original script that hooks the DNS logs:

event DNS::log_dns(rec: DNS::Info)
    {
        for( i in rec$answers )
                if( "1.2.3.4" in rec$answers[i] )
                        print fmt("%s %s %s %s", rec$ts, rec$uid,
rec$id$orig_h, rec$query);
    }


To make this go to its own log, I tried this:



module Foo;

export {
    # Create an ID for the our new stream. By convention, this is
    # called "LOG".
    redef enum Log::ID += { LOG };

    # Define the fields. By convention, the type is called "Info".
    type Info: record {
        ts:                      time      &log;
        uid:                     string    &log;
        orig_ip:                 string    &log;
        query:                   string    &log;
    };

    # Define a hook event. By convention, this is called
    # "log_<stream>".
    global log_foo: event(rec: Info);

}

redef record rec += {
        foo: Info &optional;
};

# This event should be handled at a higher priority so that when
# users modify your stream later and they do it at priority 0,
# their code runs after this.
event bro_init() &priority=5
    {
    # Create the stream. This also adds a default filter automatically.
    Log::create_stream(Foo::LOG, [$columns=Info]);
    }

event DNS::log_dns(rec: DNS::Info)
    {
        for( i in rec$answers )
                if( "1.2.3.4" in rec$answers[i] )
                        local rec: Foo::Info = [ $ts=rec$ts,
$uid=rec$uid, $orig_h=rec$id$orig_h, $query=rec$query];
                        rec$foo = rec;
                        Log::write(Foo::LOG, rec);
    }

I get the errors:

error in ./test.bro, line 22: unknown identifier (Foo::rec)
error in ./test.bro, line 35 and ./test.bro, line 39: already defined (Foo::rec)


I am new to writing Bro scripts.  Any pointers on what I'm doing wrong?


-Chris


From rmkml at yahoo.fr  Fri Aug 17 16:18:03 2012
From: rmkml at yahoo.fr (rmkml)
Date: Sat, 18 Aug 2012 01:18:03 +0200 (CEST)
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208160110570.1865@lenovo.localdomain>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
	<alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>
	<alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>
	<alpine.LFD.2.01.1208160110570.1865@lenovo.localdomain>
Message-ID: <alpine.LFD.2.01.1208180041280.1860@lenovo.localdomain>

Hi,

ok please found Six alpha release update (open-gpl) Emerging Threats 
signatures, I have switched to bro signature language:
  http://88.191.140.111/et_bro2_16aug.sig

You can start bro like this:
  bro -i eth0 -s et_bro2_16aug.sig

Features:
-previously used bro powerful language, now use bro signature language!
-update to last Emerging Threats 16 Aug 2012
-contains only 111 sig at this time, work in progress
-bro signature language use regular expression (like juniper/onesecure) need rewrite signature
-remember bro tcp reassembly only first 1k for performance reason, check dpd_buffer_size

Im always interested if you have comments/feedback/flame/performance/FP/FN please.

Futur work:
1) use Dynamic Port Detection (not static port)
1) use local_net / external_net
2) split signature per category files
3) find case insensitive more "simplify" regexp ?
4) create on new dns parser like dns-request on bro signature language

More information on Bro Signature Language:
  http://bro-ids.org/documentation/signatures.html

Regards
Rmkml

http://twitter.com/rmkml


From stephane.chazelas at gmail.com  Sat Aug 18 14:51:39 2012
From: stephane.chazelas at gmail.com (Stephane Chazelas)
Date: Sat, 18 Aug 2012 22:51:39 +0100
Subject: [Bro] skype analyzer?
Message-ID: <20120818215139.GB5889@chaz.gmail.com>

Hiya,

I can see many references on the web of a skype analyzer for
bro, but can't find it anywhere. I'd just want to be able to
tell in the conn.log whether a TCP 443 connection is skype or
not. It should be easy based on that faked TLS ServerHello with
fixed "random" pattern that skype seems to consistently be
sending, but before I reinvent the wheel, can anybody please
point me to the existing implementations?

Thanks,
Stephane



From rmkml at yahoo.fr  Sat Aug 18 17:35:11 2012
From: rmkml at yahoo.fr (rmkml)
Date: Sun, 19 Aug 2012 02:35:11 +0200 (CEST)
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208180041280.1860@lenovo.localdomain>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
	<alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>
	<alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>
	<alpine.LFD.2.01.1208160110570.1865@lenovo.localdomain>
	<alpine.LFD.2.01.1208180041280.1860@lenovo.localdomain>
Message-ID: <alpine.LFD.2.01.1208190230020.2661@lenovo.localdomain>

Hi,
ok please new release update (open-gpl) Emerging Threats signatures to Bro Signature langage:
  http://88.191.140.111/et_bro2_17aug.sig

You can start bro like this:
  bro -i eth0 -s et_bro2_17aug.sig

Features:
-update to last Emerging Threats 17 Aug 2012
-contains only 269 sig at this time, work in progress
-bro signature language use regular expression (like juniper/onesecure) need  rewrite ET signature
-remember bro tcp reassembly only first 1k for performance reason, check dpd_buffer_size

Im always interested if you have comments/feedback/flame/performance/FP/FN please.

Futur work:
1) use Dynamic Port Detection (not static port)
2) use local_net / external_net
3) split signatures per category files
4) find case insensitive more "simplify" regexp ?
5) create on new dns parser like dns-request on bro signature language

More information on Bro Signature Language:
http://bro-ids.org/documentation/signatures.html

Regards
Rmkml

http://twitter.com/rmkml


From christopher.p.crawford at gmail.com  Mon Aug 20 06:43:15 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Mon, 20 Aug 2012 09:43:15 -0400
Subject: [Bro] Converting a Bro Script to A New Stream
In-Reply-To: <CAG1VgOab88X87hCP01PXtT3n7hJkhnY=Mf7kRa_YfNV5+V3Vaw@mail.gmail.com>
References: <CAG1VgOab88X87hCP01PXtT3n7hJkhnY=Mf7kRa_YfNV5+V3Vaw@mail.gmail.com>
Message-ID: <CAG1VgOZvx0Z5zn-OXARH_Jh6NiEOq_uOmjGDiWhHqnQuVCO1GQ@mail.gmail.com>

I was able to make some progress.  This script does what I want, but
there are still some problems.

module Foo;

export {
    redef enum Log::ID += { LOG };

    type Info: record {
        ts:                      time      &log;
        uid:                     string    &log;
        orig_ip:                 addr      &log;
        query:                   string    &log;
    };

}

event bro_init() &priority=5
    {
    Log::create_stream(Foo::LOG, [$columns=Info]);
    }

event DNS::log_dns(rec: DNS::Info)
    {
        if(rec$qtype_name == "A")
                if( |rec$answers| > 0 )
                        for( i in rec$answers )
                                if( "1.2.3.4" in rec$answers[i] )
                                        Log::write(Foo::LOG, [
$ts=rec$ts, $uid=rec$uid, $orig_ip=rec$id$orig_h, $query=rec$query]);
    }

I get the log I want, with the data I expect.

The problem is, I get thousands of entries in reporter.log that say:

"Reporter::ERROR field value missing [Foo::rec$qtype_name]"

And I get hundreds of entries that say:

"Reporter::ERROR field value missing [Foo::rec$answers]"

I think I'm getting these error messages because log_dns fires, but
rec is empty or doesn't exist.  How can I check my theory?  If that is
truly the case, is there a better way to write the initial logic in
event DNS::log_dns(rec: DNS::Info) that would break out of that
function if rec does not exist?  I thought that checking for DNS
answers and then checking to make sure that there is a positive number
of answers would help to eliminate the errors I see in reporter.log,
but it doesn't.

-Chris

On Thu, Aug 16, 2012 at 5:32 PM, Chris Crawford
<christopher.p.crawford at gmail.com> wrote:
> I have a short bro script that I wrote that hooks the DNS log
> (http://www.bro-ids.org/documentation/logging.html#hooking-into-the-logging).
>  Each time a DNS::log_dns event fires, if a specific IP is in
> rec$answers, the script prints out rec$ts, rec$uid, rec$id$orig_h, and
> rec$query.
>
> I want the entries from the script to go to their own log, though.  I
> am struggling to figure out how to make that work.  Based on the
> documentation for logging, it looks like I'd need to define a new
> Stream to create a new log file.
> (http://www.bro-ids.org/documentation/logging.html#adding-streams)
>
> Here's my original script that hooks the DNS logs:
>
> event DNS::log_dns(rec: DNS::Info)
>     {
>         for( i in rec$answers )
>                 if( "1.2.3.4" in rec$answers[i] )
>                         print fmt("%s %s %s %s", rec$ts, rec$uid,
> rec$id$orig_h, rec$query);
>     }
>
>
> To make this go to its own log, I tried this:
>
>
>
> module Foo;
>
> export {
>     # Create an ID for the our new stream. By convention, this is
>     # called "LOG".
>     redef enum Log::ID += { LOG };
>
>     # Define the fields. By convention, the type is called "Info".
>     type Info: record {
>         ts:                      time      &log;
>         uid:                     string    &log;
>         orig_ip:                 string    &log;
>         query:                   string    &log;
>     };
>
>     # Define a hook event. By convention, this is called
>     # "log_<stream>".
>     global log_foo: event(rec: Info);
>
> }
>
> redef record rec += {
>         foo: Info &optional;
> };
>
> # This event should be handled at a higher priority so that when
> # users modify your stream later and they do it at priority 0,
> # their code runs after this.
> event bro_init() &priority=5
>     {
>     # Create the stream. This also adds a default filter automatically.
>     Log::create_stream(Foo::LOG, [$columns=Info]);
>     }
>
> event DNS::log_dns(rec: DNS::Info)
>     {
>         for( i in rec$answers )
>                 if( "1.2.3.4" in rec$answers[i] )
>                         local rec: Foo::Info = [ $ts=rec$ts,
> $uid=rec$uid, $orig_h=rec$id$orig_h, $query=rec$query];
>                         rec$foo = rec;
>                         Log::write(Foo::LOG, rec);
>     }
>
> I get the errors:
>
> error in ./test.bro, line 22: unknown identifier (Foo::rec)
> error in ./test.bro, line 35 and ./test.bro, line 39: already defined (Foo::rec)
>
>
> I am new to writing Bro scripts.  Any pointers on what I'm doing wrong?
>
>
> -Chris


From christopher.p.crawford at gmail.com  Mon Aug 20 06:48:11 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Mon, 20 Aug 2012 09:48:11 -0400
Subject: [Bro] www-old.bro-ids.org
Message-ID: <CAG1VgOZYtxCb=J2miBKJCPgNZPTNPYL6gGxXnkZLhpaHGL3L9g@mail.gmail.com>

I've been using
http://www-old.bro-ids.org/wiki/index.php/Reference_Manual lately. It
appears to be a better reference for learning how to write Bro scripts
than http://www.bro-ids.org/documentation/index.html .  Is
http://www-old.bro-ids.org/wiki/index.php/Reference_Manual still the
recommended place to learn the Bro scripting language?

-Chris


From JAzoff at albany.edu  Mon Aug 20 07:07:00 2012
From: JAzoff at albany.edu (Justin Azoff)
Date: Mon, 20 Aug 2012 10:07:00 -0400
Subject: [Bro] Converting a Bro Script to A New Stream
In-Reply-To: <CAG1VgOab88X87hCP01PXtT3n7hJkhnY=Mf7kRa_YfNV5+V3Vaw@mail.gmail.com>
References: <CAG1VgOab88X87hCP01PXtT3n7hJkhnY=Mf7kRa_YfNV5+V3Vaw@mail.gmail.com>
Message-ID: <20120820140700.GF1957@datacomm.albany.edu>

On Thu, Aug 16, 2012 at 05:32:28PM -0400, Chris Crawford wrote:
> I have a short bro script that I wrote that hooks the DNS log
> (http://www.bro-ids.org/documentation/logging.html#hooking-into-the-logging).
>  Each time a DNS::log_dns event fires, if a specific IP is in
> rec$answers, the script prints out rec$ts, rec$uid, rec$id$orig_h, and
> rec$query.
> 
> I want the entries from the script to go to their own log, though.  I
> am struggling to figure out how to make that work.  Based on the
> documentation for logging, it looks like I'd need to define a new
> Stream to create a new log file.
> (http://www.bro-ids.org/documentation/logging.html#adding-streams)

This is really simple.. see some of the examples here: 

http://blog.bro-ids.org/2012/02/filtering-logs-with-bro.html

You want to use Log:add_filter with a pred function that does the
filtering. like this example:

 Log::add_filter(HTTP::LOG, [$name = "http-executables",
                             $path = "http_exe",
                             $pred(rec: HTTP::Info) = { return rec?$mime_type && rec$mime_type == "application/x-dosexec"; }]);

-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From seth at icir.org  Mon Aug 20 07:42:01 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 20 Aug 2012 10:42:01 -0400
Subject: [Bro] Converting a Bro Script to A New Stream
In-Reply-To: <CAG1VgOab88X87hCP01PXtT3n7hJkhnY=Mf7kRa_YfNV5+V3Vaw@mail.gmail.com>
References: <CAG1VgOab88X87hCP01PXtT3n7hJkhnY=Mf7kRa_YfNV5+V3Vaw@mail.gmail.com>
Message-ID: <A69CFAEE-A138-46B6-89F5-7705FA714ECD@icir.org>


On Aug 16, 2012, at 5:32 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:

> redef record rec += {
>        foo: Info &optional;
> };

> error in ./test.bro, line 22: unknown identifier (Foo::rec)
> error in ./test.bro, line 35 and ./test.bro, line 39: already defined (Foo::rec)

I don't think you need that little chunk of code I left above.  We do that in many base scripts as a way of hiding protocol specific information within the connection record.  There is no existing record type named "rec" though and it doesn't look like you need to hide this information anywhere since you are deriving all of your log directly from data in the DNS::log_dns event.

There is a better way to do this though and it was something we specifically considered in the logging framework.  Here's a log filter you can run that will give you the log you want?

event bro_init()
	{
	local filter: Log::Filter = [
		$name="only-1.2.3.4", 
		$path="foo", 
		$pred(rec: DNS::Info) = {
			if ( rec?$qtype_name && rec?$answers &&
			     rec$qtype_name == "A" )
				{
				for ( i in rec$answers )
					if ( "1.2.3.4" in rec$answers[i] )
						return T;
				}
			return F;
		},
		$include=set("ts", "uid", "id.orig_h", "query")];
	Log::add_filter(DNS::LOG, filter);
	}

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Mon Aug 20 07:44:08 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 20 Aug 2012 10:44:08 -0400
Subject: [Bro] www-old.bro-ids.org
In-Reply-To: <CAG1VgOZYtxCb=J2miBKJCPgNZPTNPYL6gGxXnkZLhpaHGL3L9g@mail.gmail.com>
References: <CAG1VgOZYtxCb=J2miBKJCPgNZPTNPYL6gGxXnkZLhpaHGL3L9g@mail.gmail.com>
Message-ID: <134D1CB7-9758-445F-B843-61B1D09B3F00@icir.org>


On Aug 20, 2012, at 9:48 AM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:

> Is
> http://www-old.bro-ids.org/wiki/index.php/Reference_Manual still the
> recommended place to learn the Bro scripting language?


Unfortunately if you are learning the language in deeper detail that is probably the best documentation around right now.  The only thing to keep in mind is that the newer script structure and new idioms to script development aren't documented there.  Once we have better documentation we may likely retire that old site permanently.

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From rmkml at yahoo.fr  Mon Aug 20 07:43:18 2012
From: rmkml at yahoo.fr (rmkml at yahoo.fr)
Date: Mon, 20 Aug 2012 14:43:18 +0000 (UTC)
Subject: [Bro] RE : bro signature http-request double encoded cause FN ?
Message-ID: <1511173204.11384.1345473877860.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>

An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120820/0d827fd5/attachment.html 

From vladg at cmu.edu  Mon Aug 20 08:16:24 2012
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Mon, 20 Aug 2012 15:16:24 +0000
Subject: [Bro] RE : bro signature http-request double encoded cause FN ?
In-Reply-To: <22760_1345474348_q7KEqQNh008742_1511173204.11384.1345473877860.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>
References: <22760_1345474348_q7KEqQNh008742_1511173204.11384.1345473877860.JavaMail.seven@ap14.p2.fra.samsungsocialhub.net>
Message-ID: <1202BE242E080642B0CD0AD0A03E8552254C2F@PGH-MSGMB-03.andrew.ad.cmu.edu>

Hi rmkml,

First off, let me just thank you for all the work you've been doing recently. I think a lot of people are interested in integrating additional intelligence sources (like Emerging Threats) into Bro.

However, I'm concerned that a lot of your work seems to be based on just passing content through a bunch of regular expressions. A few others have also expressed concern with this approach. As a result, I think most people here are wary to try your scripts on their clusters. Even a 10% slowdown translates to one or two extra 16-core machines that would need to be added to the cluster in some places. Apart from that, at least to me, this approach goes against a lot of the Bro philosophy. If people just wanted basic signature-matching, they'd use one of the many much more simplistic tools out there. With Bro, this is really viewed as intelligence instead of an amalgamation of signatures.

Personally, I think you'd get much more interest if you could just create a text-file with known bad user-agents from the Emerging Threats sigs. I think that's a good place to start, and once that's in place, we can help you figure out the best way to extend that to domain names, URIs, filenames, etc.

Just my 2 cents on why all the time you've been investing into this isn't getting the interest and response one would expect.

--
Vlad Grigorescu
Senior Information Security Engineer
Carnegie Mellon University

On Aug 20, 2012, at 10:43 AM, "rmkml at yahoo.fr" <rmkml at yahoo.fr>
 wrote:

> Hi,
> Nobody interested please ?
> Regards
> Rmkml
> 
> 
> 
> rmkml a ?crit :
> 
> Hi,
> ok it's long time I don't worked on bro signature "language",
> but Im back today and Im start few tests:
> 
> 0) web test without encoding : /abc
> OK, detected by  http-request /.*\/abc.*/
> 
> 1) http utf8 simple encoded like /ab%63
> OK, detected by  http-request /.*\/abc.*/
> 
> 2) http utf8 double encoded like /ab%2563
> NOT detected by  http-request /.*\/abc.*/
> 
> Anyone confirm this ?
> Maybe need switch a variable, where ?
> 
> Regards
> Rmkml
> 
> http://twitter.com/rmkml
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From rmkml at yahoo.fr  Mon Aug 20 15:22:10 2012
From: rmkml at yahoo.fr (rmkml)
Date: Tue, 21 Aug 2012 00:22:10 +0200 (CEST)
Subject: [Bro] Requesting help for adding dns-request on Bro signature
 langage please
Message-ID: <alpine.LFD.2.01.1208210012110.1844@lenovo.localdomain>

Hi,

Im try to add new "dns-request" keyword on Bro Signature langage, but it's not fire.
Anyone help please? (Im not a C++/developper)

ok Im test on bro v2.0 on linux plateform.

Modified two files, first is src/Rule.h :

@@ -37,7 +37,7 @@

         enum PatternType {
                 PAYLOAD, HTTP_REQUEST, HTTP_REQUEST_BODY, HTTP_REQUEST_HEADER,
-               HTTP_REPLY_BODY, HTTP_REPLY_HEADER, FTP, FINGER, TYPES,
+               HTTP_REPLY_BODY, HTTP_REPLY_HEADER, DNS_REQUEST, FTP, FINGER, TYPES,
         };

         bool Active()   { return active; }



Second change are src/DNS.cc :

@@ -1093,6 +1093,8 @@
         if ( buf_n < msg_size )
                 // Haven't filled up the message buffer yet, no more to do.
                 return;
+               Conn()->Match(Rule::DNS_REQUEST, (const u_char*) msg_buf,
+                       msg_size, true, true, 1, true);

         ForwardPacket(msg_size, msg_buf, orig, -1, 0, 0);



With this new signature:
  dns-request /.*g.*/


Regards
Rmkml

http://twitter.com/rmkml


From seth at icir.org  Mon Aug 20 14:12:41 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 20 Aug 2012 17:12:41 -0400
Subject: [Bro] SSN detection script
In-Reply-To: <4481F54F-820E-4B85-B7CD-CCCC4642E016@icir.org>
References: <4481F54F-820E-4B85-B7CD-CCCC4642E016@icir.org>
Message-ID: <05278E98-54B7-4A0C-9216-C44DC936377E@icir.org>


On Aug 16, 2012, at 1:54 PM, Seth Hall <seth at icir.org> wrote:

> https://github.com/sethhall/ssn-exposure


I just added a small configuration option for this script to enable redaction on the ssn_exposure.log since users were having PII logs created for them by positive detections.

It can be enabled with:
	redef SsnExposure::redact_logs = T;

I did another little fix to remove SSNs from notices too (they weren't supposed to be there in the first place!).

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Mon Aug 20 17:47:13 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 20 Aug 2012 20:47:13 -0400
Subject: [Bro] Emerging Threats signatures on Bro ids ?
In-Reply-To: <alpine.LFD.2.01.1208190230020.2661@lenovo.localdomain>
References: <alpine.LFD.2.01.1208110204250.1841@lenovo.localdomain>
	<4681_1344638489_q7AMfSw8028745_CANpnLHjcSv53cA9O1+4P0juiJ=YnqkSE1i1TKsRv3dFTGr59kg@mail.gmail.com>
	<1202BE242E080642B0CD0AD0A03E85521B6F8C@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<8FAA42DF-F8F5-4ADB-92EC-49B4D3AE4FB8@icir.org>
	<alpine.LFD.2.01.1208120230430.1872@lenovo.localdomain>
	<alpine.LFD.2.01.1208130048010.1901@lenovo.localdomain>
	<alpine.LFD.2.01.1208140248060.1860@lenovo.localdomain>
	<alpine.LFD.2.01.1208150402140.1970@lenovo.localdomain>
	<alpine.LFD.2.01.1208160110570.1865@lenovo.localdomain>
	<alpine.LFD.2.01.1208180041280.1860@lenovo.localdomain>
	<alpine.LFD.2.01.1208190230020.2661@lenovo.localdomain>
Message-ID: <9AC8CB83-C895-49F8-BF79-43B31EA22D38@icir.org>


On Aug 18, 2012, at 8:35 PM, rmkml <rmkml at yahoo.fr> wrote:

> ok please new release update (open-gpl) Emerging Threats signatures to Bro Signature langage:
>  http://88.191.140.111/et_bro2_17aug.sig


Could you please stop announcing every update you do to these signatures on our mailing list?  You are free to start your own mailing list and announce these there for all who are interested.

The signatures you are writing aren't actually something that can be deployed on most users Bro installations due to performance issues and hold very little interest for much of the community due to that.  Additionally, the signatures already exist and are tested for other platforms.  Anyone running your signatures would have to trust that you converted them into Bro scripts and signatures correctly.

Thanks,
  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From stephane.chazelas at gmail.com  Wed Aug 22 13:44:24 2012
From: stephane.chazelas at gmail.com (Stephane Chazelas)
Date: Wed, 22 Aug 2012 21:44:24 +0100
Subject: [Bro] setting a connection "service" in a signature
Message-ID: <20120822204423.GB9894@chaz.gmail.com>

Hiya,

I thought I'd share a way to mark the fake HTTPS connections
done by skype as such in conn.log. We've been seeing connections
to various IP addresses around the world sending hundreds of
megabytes of data and wanted to make sure it wasn't any
information leak. Most of the time, it is skype traffic but we
wanted a way to automatically determine it was the case.

Here is a simple way. It just uses the "service" flag of a bro
"connection" to mark the fact it is skype traffic.

It detects skype traffic by looking at the fake SSL
"ServerHello" that skype responders send. (basically, they send
a fixed "random data" with a date in 2004 where a normal SSL
server would send the current date and a truly random data, I
suspect it is designed that way to help recognise skype traffic
easily).

I've got in my local.bro:

function mark_conn_as_skype(state: signature_state): bool
        {
        add state$conn$service["skype"];
        return T;
        }
redef signature_files += "skype-detect.sig";

(change to "return F" to avoid the alarm in notice.log)

And in skype-detect.sig

signature skype_fake_https {
  ip-proto == tcp
  tcp-state established,responder
  event "Skype fake HTTPS connection"
  src-port == 443
  payload /\x16\x03\x01\x00\x4a\x02\x00\x00\x46\x03\x01\x40\x1b\xe4\x86\x02\xad\xe0\x29\xe1\x77\x74\xe5\x44\xb9\xc9\x9c\xb4\x31\x31\x5e\x02\xdd\x77\x9d\x15\x4a\x96\x09\xba\x5d\xa8\x70/
  eval mark_conn_as_skype
}

Then you'll see "skype" in the "service" column for those
connections and need worry less when you see 200MB of data being
sent to Ukraine or any country you usually don't do business
with.

-- 
Stephane



From stephane.chazelas at gmail.com  Thu Aug 23 03:11:30 2012
From: stephane.chazelas at gmail.com (Stephane Chazelas)
Date: Thu, 23 Aug 2012 11:11:30 +0100
Subject: [Bro] setting a connection "service" in a signature
In-Reply-To: <20120822204423.GB9894@chaz.gmail.com>
References: <20120822204423.GB9894@chaz.gmail.com>
Message-ID: <20120823101130.GA5744@chaz.gmail.com>

2012-08-22 21:44:24 +0100, Stephane Chazelas:
[...]
> Here is a simple way. It just uses the "service" flag of a bro
> "connection" to mark the fact it is skype traffic.
[...]

Oh well, sorry, I spoke too soon. That makes bro crash in

#1  0x081d5ee8 in BroFunc::Call(ValPList*, Frame*) const ()
#2  0x0824e677 in RuleConditionEval::DoMatch(Rule*, RuleEndpointState*, unsigned char const*, int) ()
#3  0x0824f2b4 in RuleMatcher::EvalRuleConditions(Rule*, RuleEndpointState*, unsigned char const*, int, bool) ()
#4  0x08250adc in RuleMatcher::Match(RuleEndpointState*, Rule::PatternType, unsigned char const*, int, bool, bool, bool) ()
#5  0x08234eac in PIA_TCP::DeliverStream(int, unsigned char const*, bool) ()
#6  0x0814966f in Analyzer::NextStream(int, unsigned char const*, bool) ()
#7  0x08149d2d in Analyzer::ForwardStream(int, unsigned char const*, bool) ()
#8  0x08283bef in TCP_Reassembler::DeliverBlock(int, int, unsigned char const*) ()
#9  0x08283f2e in TCP_Reassembler::BlockInserted(DataBlock*) ()
#10 0x08282b0f in TCP_Reassembler::DataSent(double, int, int, unsigned char const*, bool) ()
#11 0x082823c2 in TCP_Endpoint::DataSent(double, int, int, int, unsigned char const*, IP_Hdr const*, tcphdr const*) ()
#12 0x08281873 in TCP_Analyzer::DeliverPacket(int, unsigned char const*, bool, int, IP_Hdr const*, int) ()
#13 0x08149821 in Analyzer::NextPacket(int, unsigned char const*, bool, int, IP_Hdr const*, int) ()
#14 0x08163131 in Connection::NextPacket(double, int, IP_Hdr const*, int, int, unsigned char const*&, int&, int&, pcap_pkthdr const*, unsigned char const*, int) ()
#15 0x082698dd in NetSessions::DoNextPacket(double, pcap_pkthdr const*, IP_Hdr const*, unsigned char const*, int) ()
#16 0x08269eae in NetSessions::NextPacket(double, pcap_pkthdr const*, unsigned char const*, int, PacketSortElement*) ()
#17 0x08222d2f in net_packet_dispatch(double, pcap_pkthdr const*, unsigned char const*, int, PktSrc*, PacketSortElement*) ()
#18 0x08223013 in net_packet_arrival(double, pcap_pkthdr const*, unsigned char const*, int, PktSrc*) ()
#19 0x0823288b in PktSrc::Process() ()
#20 0x082230a3 in net_run() ()
#21 0x0814423a in main ()

with frames above that varying:

$ for f (**/core(m-1)) gdb -core $f =bro --batch -ex bt
[New Thread 29511]
Core was generated by `/usr/local/bin/bro -i eth1 -U .status -p broctl -p broctl-live -p standalone -p'.
Program terminated with signal 6, Aborted.
#0  0xb777b430 in __kernel_vsyscall ()
#0  0xb777b430 in __kernel_vsyscall ()
#1  0xb72d2651 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2  0xb72d5a82 in *__GI_abort () at abort.c:92
#3  0x08209367 in Reporter::InternalError(char const*, ...) ()
#4  0x0822bdb2 in bad_ref(int) ()
#5  0x081d6142 in BroFunc::Call(ValPList*, Frame*) const ()
#6  0x0824e677 in RuleConditionEval::DoMatch(Rule*, RuleEndpointState*, unsigned char const*, int) ()
#7  0x0824f2b4 in RuleMatcher::EvalRuleConditions(Rule*, RuleEndpointState*, unsigned char const*, int, bool) ()
#8  0x08250adc in RuleMatcher::Match(RuleEndpointState*, Rule::PatternType, unsigned char const*, int, bool, bool, bool) ()
#9  0x08234eac in PIA_TCP::DeliverStream(int, unsigned char const*, bool) ()
#10 0x0814966f in Analyzer::NextStream(int, unsigned char const*, bool) ()
#11 0x08149d2d in Analyzer::ForwardStream(int, unsigned char const*, bool) ()
#12 0x08283bef in TCP_Reassembler::DeliverBlock(int, int, unsigned char const*) ()
#13 0x08283f2e in TCP_Reassembler::BlockInserted(DataBlock*) ()
#14 0x08282b0f in TCP_Reassembler::DataSent(double, int, int, unsigned char const*, bool) ()
#15 0x082823c2 in TCP_Endpoint::DataSent(double, int, int, int, unsigned char const*, IP_Hdr const*, tcphdr const*) ()
#16 0x08281873 in TCP_Analyzer::DeliverPacket(int, unsigned char const*, bool, int, IP_Hdr const*, int) ()
#17 0x08149821 in Analyzer::NextPacket(int, unsigned char const*, bool, int, IP_Hdr const*, int) ()
#18 0x08163131 in Connection::NextPacket(double, int, IP_Hdr const*, int, int, unsigned char const*&, int&, int&, pcap_pkthdr const*, unsigned char const*, int) ()
#19 0x082698dd in NetSessions::DoNextPacket(double, pcap_pkthdr const*, IP_Hdr const*, unsigned char const*, int) ()
#20 0x08269eae in NetSessions::NextPacket(double, pcap_pkthdr const*, unsigned char const*, int, PacketSortElement*) ()
#21 0x08222d2f in net_packet_dispatch(double, pcap_pkthdr const*, unsigned char const*, int, PktSrc*, PacketSortElement*) ()
#22 0x08223013 in net_packet_arrival(double, pcap_pkthdr const*, unsigned char const*, int, PktSrc*) ()
#23 0x0823288b in PktSrc::Process() ()
#24 0x082230a3 in net_run() ()
#25 0x0814423a in main ()
[New Thread 10653]
Core was generated by `/usr/local/bin/bro -i eth1 -U .status -p broctl -p broctl-live -p standalone -p'.
Program terminated with signal 11, Segmentation fault.
#0  0x081d5ec3 in BroFunc::Call(ValPList*, Frame*) const ()
#0  0x081d5ec3 in BroFunc::Call(ValPList*, Frame*) const ()
#1  0x0824e677 in RuleConditionEval::DoMatch(Rule*, RuleEndpointState*, unsigned char const*, int) ()
#2  0x0824f2b4 in RuleMatcher::EvalRuleConditions(Rule*, RuleEndpointState*, unsigned char const*, int, bool) ()
#3  0x08250adc in RuleMatcher::Match(RuleEndpointState*, Rule::PatternType, unsigned char const*, int, bool, bool, bool) ()
#4  0x08234eac in PIA_TCP::DeliverStream(int, unsigned char const*, bool) ()
#5  0x0814966f in Analyzer::NextStream(int, unsigned char const*, bool) ()
#6  0x08149d2d in Analyzer::ForwardStream(int, unsigned char const*, bool) ()
#7  0x08283bef in TCP_Reassembler::DeliverBlock(int, int, unsigned char const*) ()
#8  0x08283f2e in TCP_Reassembler::BlockInserted(DataBlock*) ()
#9  0x08282b0f in TCP_Reassembler::DataSent(double, int, int, unsigned char const*, bool) ()
#10 0x082823c2 in TCP_Endpoint::DataSent(double, int, int, int, unsigned char const*, IP_Hdr const*, tcphdr const*) ()
#11 0x08281873 in TCP_Analyzer::DeliverPacket(int, unsigned char const*, bool, int, IP_Hdr const*, int) ()
#12 0x08149821 in Analyzer::NextPacket(int, unsigned char const*, bool, int, IP_Hdr const*, int) ()
#13 0x08163131 in Connection::NextPacket(double, int, IP_Hdr const*, int, int, unsigned char const*&, int&, int&, pcap_pkthdr const*, unsigned char const*, int) ()
#14 0x082698dd in NetSessions::DoNextPacket(double, pcap_pkthdr const*, IP_Hdr const*, unsigned char const*, int) ()
#15 0x08269eae in NetSessions::NextPacket(double, pcap_pkthdr const*, unsigned char const*, int, PacketSortElement*) ()
#16 0x08222d2f in net_packet_dispatch(double, pcap_pkthdr const*, unsigned char const*, int, PktSrc*, PacketSortElement*) ()
#17 0x08223013 in net_packet_arrival(double, pcap_pkthdr const*, unsigned char const*, int, PktSrc*) ()
#18 0x0823288b in PktSrc::Process() ()
#19 0x082230a3 in net_run() ()
#20 0x0814423a in main ()
[New Thread 21546]
Core was generated by `/usr/local/bin/bro -i eth1 -U .status -p broctl -p broctl-live -p standalone -p'.
Program terminated with signal 11, Segmentation fault.
#0  0xb73cc300 in tmpnam_buffer () from /lib/tls/i686/cmov/libc.so.6
#0  0xb73cc300 in tmpnam_buffer () from /lib/tls/i686/cmov/libc.so.6
#1  0x081d5ee8 in BroFunc::Call(ValPList*, Frame*) const ()
#2  0x0824e677 in RuleConditionEval::DoMatch(Rule*, RuleEndpointState*, unsigned char const*, int) ()
#3  0x0824f2b4 in RuleMatcher::EvalRuleConditions(Rule*, RuleEndpointState*, unsigned char const*, int, bool) ()
#4  0x08250adc in RuleMatcher::Match(RuleEndpointState*, Rule::PatternType, unsigned char const*, int, bool, bool, bool) ()
#5  0x08234eac in PIA_TCP::DeliverStream(int, unsigned char const*, bool) ()
#6  0x0814966f in Analyzer::NextStream(int, unsigned char const*, bool) ()
#7  0x08149d2d in Analyzer::ForwardStream(int, unsigned char const*, bool) ()
#8  0x08283bef in TCP_Reassembler::DeliverBlock(int, int, unsigned char const*) ()
#9  0x08283f2e in TCP_Reassembler::BlockInserted(DataBlock*) ()
#10 0x08282b0f in TCP_Reassembler::DataSent(double, int, int, unsigned char const*, bool) ()
#11 0x082823c2 in TCP_Endpoint::DataSent(double, int, int, int, unsigned char const*, IP_Hdr const*, tcphdr const*) ()
#12 0x08281873 in TCP_Analyzer::DeliverPacket(int, unsigned char const*, bool, int, IP_Hdr const*, int) ()
#13 0x08149821 in Analyzer::NextPacket(int, unsigned char const*, bool, int, IP_Hdr const*, int) ()
#14 0x08163131 in Connection::NextPacket(double, int, IP_Hdr const*, int, int, unsigned char const*&, int&, int&, pcap_pkthdr const*, unsigned char const*, int) ()
#15 0x082698dd in NetSessions::DoNextPacket(double, pcap_pkthdr const*, IP_Hdr const*, unsigned char const*, int) ()
#16 0x08269eae in NetSessions::NextPacket(double, pcap_pkthdr const*, unsigned char const*, int, PacketSortElement*) ()
#17 0x08222d2f in net_packet_dispatch(double, pcap_pkthdr const*, unsigned char const*, int, PktSrc*, PacketSortElement*) ()
#18 0x08223013 in net_packet_arrival(double, pcap_pkthdr const*, unsigned char const*, int, PktSrc*) ()
#19 0x0823288b in PktSrc::Process() ()
#20 0x082230a3 in net_run() ()
#21 0x0814423a in main ()
[New Thread 27005]
Core was generated by `/usr/local/bin/bro -i eth1 -U .status -p broctl -p broctl-live -p standalone -p'.
Program terminated with signal 6, Aborted.
#0  0xb7754430 in __kernel_vsyscall ()
#0  0xb7754430 in __kernel_vsyscall ()
#1  0xb72ab651 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2  0xb72aea82 in *__GI_abort () at abort.c:92
#3  0xb72e206d in __libc_message (do_abort=2, fmt=0xb73b6f78 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
#4  0xb72ec161 in malloc_printerr (action=<value optimized out>, str=0x6 <Address 0x6 out of bounds>, ptr=0xa5f9e50) at malloc.c:6266
#5  0xb72ef2e0 in _int_malloc (av=<value optimized out>, bytes=<value optimized out>) at malloc.c:4308
#6  0xb72f0b6c in *__GI___libc_malloc (bytes=5) at malloc.c:3660
#7  0xb74ddc07 in operator new(unsigned int) () from /usr/lib/libstdc++.so.6
#8  0xb74ddd3d in operator new[](unsigned int) () from /usr/lib/libstdc++.so.6
#9  0x081f7437 in HashKey::CopyKey(void const*, int) const ()
#10 0x081f7496 in HashKey::TakeKey() ()
#11 0x0829ebf3 in TableVal::Assign(Val*, HashKey*, Val*, Opcode) ()
#12 0x0829f28e in TableVal::Assign(Val*, Val*, Opcode) ()
#13 0x081c1041 in IndexExpr::Add(Frame*) ()
#14 0x08273191 in AddStmt::Exec(Frame*, stmt_flow_type&) const ()
#15 0x08273d4f in StmtList::Exec(Frame*, stmt_flow_type&) const ()
#16 0x081d5f69 in BroFunc::Call(ValPList*, Frame*) const ()
#17 0x0824e677 in RuleConditionEval::DoMatch(Rule*, RuleEndpointState*, unsigned char const*, int) ()
#18 0x0824f2b4 in RuleMatcher::EvalRuleConditions(Rule*, RuleEndpointState*, unsigned char const*, int, bool) ()
#19 0x08250adc in RuleMatcher::Match(RuleEndpointState*, Rule::PatternType, unsigned char const*, int, bool, bool, bool) ()
#20 0x08234eac in PIA_TCP::DeliverStream(int, unsigned char const*, bool) ()
#21 0x0814966f in Analyzer::NextStream(int, unsigned char const*, bool) ()
#22 0x08149d2d in Analyzer::ForwardStream(int, unsigned char const*, bool) ()
#23 0x08283bef in TCP_Reassembler::DeliverBlock(int, int, unsigned char const*) ()
#24 0x08283f2e in TCP_Reassembler::BlockInserted(DataBlock*) ()
#25 0x08282b0f in TCP_Reassembler::DataSent(double, int, int, unsigned char const*, bool) ()
#26 0x082823c2 in TCP_Endpoint::DataSent(double, int, int, int, unsigned char const*, IP_Hdr const*, tcphdr const*) ()
#27 0x08281873 in TCP_Analyzer::DeliverPacket(int, unsigned char const*, bool, int, IP_Hdr const*, int) ()
#28 0x08149821 in Analyzer::NextPacket(int, unsigned char const*, bool, int, IP_Hdr const*, int) ()
#29 0x08163131 in Connection::NextPacket(double, int, IP_Hdr const*, int, int, unsigned char const*&, int&, int&, pcap_pkthdr const*, unsigned char const*, int) ()
#30 0x082698dd in NetSessions::DoNextPacket(double, pcap_pkthdr const*, IP_Hdr const*, unsigned char const*, int) ()
#31 0x08269eae in NetSessions::NextPacket(double, pcap_pkthdr const*, unsigned char const*, int, PacketSortElement*) ()
#32 0x08222d2f in net_packet_dispatch(double, pcap_pkthdr const*, unsigned char const*, int, PktSrc*, PacketSortElement*) ()
#33 0x08223013 in net_packet_arrival(double, pcap_pkthdr const*, unsigned char const*, int, PktSrc*) ()
#34 0x0823288b in PktSrc::Process() ()
#35 0x082230a3 in net_run() ()
#36 0x0814423a in main ()
[New Thread 28453]
Core was generated by `/usr/local/bin/bro -i eth1 -U .status -p broctl -p broctl-live -p standalone -p'.
Program terminated with signal 6, Aborted.
#0  0xb76e4430 in __kernel_vsyscall ()
#0  0xb76e4430 in __kernel_vsyscall ()
#1  0xb723b651 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2  0xb723ea82 in *__GI_abort () at abort.c:92
#3  0x08209367 in Reporter::InternalError(char const*, ...) ()
#4  0x0822bdb2 in bad_ref(int) ()
#5  0x081d6142 in BroFunc::Call(ValPList*, Frame*) const ()
#6  0x0824e677 in RuleConditionEval::DoMatch(Rule*, RuleEndpointState*, unsigned char const*, int) ()
#7  0x0824f2b4 in RuleMatcher::EvalRuleConditions(Rule*, RuleEndpointState*, unsigned char const*, int, bool) ()
#8  0x08250adc in RuleMatcher::Match(RuleEndpointState*, Rule::PatternType, unsigned char const*, int, bool, bool, bool) ()
#9  0x08234eac in PIA_TCP::DeliverStream(int, unsigned char const*, bool) ()
#10 0x0814966f in Analyzer::NextStream(int, unsigned char const*, bool) ()
#11 0x08149d2d in Analyzer::ForwardStream(int, unsigned char const*, bool) ()
#12 0x08283bef in TCP_Reassembler::DeliverBlock(int, int, unsigned char const*) ()
#13 0x08283f2e in TCP_Reassembler::BlockInserted(DataBlock*) ()
#14 0x08282b0f in TCP_Reassembler::DataSent(double, int, int, unsigned char const*, bool) ()
#15 0x082823c2 in TCP_Endpoint::DataSent(double, int, int, int, unsigned char const*, IP_Hdr const*, tcphdr const*) ()
#16 0x08281873 in TCP_Analyzer::DeliverPacket(int, unsigned char const*, bool, int, IP_Hdr const*, int) ()
#17 0x08149821 in Analyzer::NextPacket(int, unsigned char const*, bool, int, IP_Hdr const*, int) ()
#18 0x08163131 in Connection::NextPacket(double, int, IP_Hdr const*, int, int, unsigned char const*&, int&, int&, pcap_pkthdr const*, unsigned char const*, int) ()
#19 0x082698dd in NetSessions::DoNextPacket(double, pcap_pkthdr const*, IP_Hdr const*, unsigned char const*, int) ()
#20 0x08269eae in NetSessions::NextPacket(double, pcap_pkthdr const*, unsigned char const*, int, PacketSortElement*) ()
#21 0x08222d2f in net_packet_dispatch(double, pcap_pkthdr const*, unsigned char const*, int, PktSrc*, PacketSortElement*) ()
#22 0x08223013 in net_packet_arrival(double, pcap_pkthdr const*, unsigned char const*, int, PktSrc*) ()
#23 0x0823288b in PktSrc::Process() ()
#24 0x082230a3 in net_run() ()
#25 0x0814423a in main ()


stderr.log had once:

*** glibc detected *** /usr/local/bin/bro: malloc(): smallbin double linked list corrupted: 0x0a5f9e50 ***

I suppose  it doesn't like me adding a service in that context.

Anybody got a better idea on how to make it work?

(that's the bro 2.0 in securityonion)

-- 
Stephane




From stephane.chazelas at gmail.com  Thu Aug 23 05:00:29 2012
From: stephane.chazelas at gmail.com (Stephane Chazelas)
Date: Thu, 23 Aug 2012 13:00:29 +0100
Subject: [Bro] setting a connection "service" in a signature
In-Reply-To: <20120823101130.GA5744@chaz.gmail.com>
References: <20120822204423.GB9894@chaz.gmail.com>
	<20120823101130.GA5744@chaz.gmail.com>
Message-ID: <20120823120029.GB5744@chaz.gmail.com>

2012-08-23 11:11:30 +0100, Stephane Chazelas:
> 2012-08-22 21:44:24 +0100, Stephane Chazelas:
> [...]
> > Here is a simple way. It just uses the "service" flag of a bro
> > "connection" to mark the fact it is skype traffic.
> [...]
> 
> Oh well, sorry, I spoke too soon. That makes bro crash.

If I change it to:

function mark_conn_as_skype(state: signature_state): bool
        {
	# use a temp var to prevent bro from crashing
        local srv = state$conn$service;
        add srv["skype"];
        return T;
        }


Then, it longer crashes and seems to work fine.

-- 
Stephane



From diwakar.dinkar at gmail.com  Thu Aug 23 05:13:01 2012
From: diwakar.dinkar at gmail.com (Diwakar Dinkar)
Date: Thu, 23 Aug 2012 17:43:01 +0530
Subject: [Bro] I am not getting the report on my email id
Message-ID: <CANX=KDNovXgZnu=AsYkStcho4BDpQHN9gHUMZaBDqqE_DUuQyw@mail.gmail.com>

hello sir,
As i set my email id on $BROHOME/etc/bro.cfg
log files are created but  I am not getting the report on my id. so  just
unable to do the analysis part of the report. I am also unable to know  how
can I understand the particular attack by seeing the report. Is there any
link which can describe "how to detect attack??"

Thanking you
D.K.Dinkar
Research Fellow
IIT Patna
India
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120823/f02199b5/attachment.html 

From seth at icir.org  Thu Aug 23 06:56:17 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 23 Aug 2012 09:56:17 -0400
Subject: [Bro] setting a connection "service" in a signature
In-Reply-To: <20120822204423.GB9894@chaz.gmail.com>
References: <20120822204423.GB9894@chaz.gmail.com>
Message-ID: <A411F734-EBF9-42FB-8D6F-B771F91217E5@icir.org>


On Aug 22, 2012, at 4:44 PM, Stephane Chazelas <stephane.chazelas at gmail.com> wrote:

> It detects skype traffic by looking at the fake SSL
> "ServerHello" that skype responders send. (basically, they send
> a fixed "random data" with a date in 2004 where a normal SSL
> server would send the current date and a truly random data, I
> suspect it is designed that way to help recognise skype traffic
> easily).

Cool technique!  Thanks for sharing.  Do these connections show up in ssl.log or generate the ssl_server_hello event?  It would probably be better to detect them through the SSL analyzer if possible.

> function mark_conn_as_skype(state: signature_state): bool
>        {
>        add state$conn$service["skype"];
>        return T;
>        }
> redef signature_files += "skype-detect.sig";


I have a couple of comments here?

The prototype for your function should be: 
	function cond(state: signature_state, data: string): bool;

You are missing the data variable which could be partly what's contributing to the crash you are seeing.  I'll try and look into this more closely soon to see what exactly we need to fix (something here needs fixed, I'm just not sure what it is yet).  

Additionally, beginning with 2.0 you can use the @load-sigs directive which gives you relative path loading so you can distribute your protocol detection script as a "module" and it doesn't matter where people put it on the BROPATH the skype-detect.sig signatures would still be find-able.  The following example will look for the signature file in the same directory of the script that called @load-sigs.

	@load-sigs ./skype-detect.sig

One other comment, we do have a Skype analyzer that is currently not enabled in the script-land.  That should detect and log Skype connections as well (the base scripts need to be written for it still).

Again, thanks for sending that in!  Definitely a cool trick.  Do you think you could package it up in a git repository like I've been doing with my recent scripts?  The ssn-exposure script even has an example of @load-sigs

	https://github.com/sethhall/ssn-exposure
	https://github.com/sethhall/relog

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Thu Aug 23 06:58:40 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 23 Aug 2012 09:58:40 -0400
Subject: [Bro] I am not getting the report on my email id
In-Reply-To: <CANX=KDNovXgZnu=AsYkStcho4BDpQHN9gHUMZaBDqqE_DUuQyw@mail.gmail.com>
References: <CANX=KDNovXgZnu=AsYkStcho4BDpQHN9gHUMZaBDqqE_DUuQyw@mail.gmail.com>
Message-ID: <6121C8CF-F82E-4FFD-9D8D-B65E707D5CEF@icir.org>


On Aug 23, 2012, at 8:13 AM, Diwakar Dinkar <diwakar.dinkar at gmail.com> wrote:

> log files are created but  I am not getting the report on my id.


Does sending email through the command line "sendmail" program work on your computer?  That's how Bro sends email and it needs to be configured correctly.

> Is there any link which can describe "how to detect attack??"

What in particular are you looking for?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From stephane.chazelas at gmail.com  Thu Aug 23 07:25:59 2012
From: stephane.chazelas at gmail.com (Stephane Chazelas)
Date: Thu, 23 Aug 2012 15:25:59 +0100
Subject: [Bro] setting a connection "service" in a signature
In-Reply-To: <A411F734-EBF9-42FB-8D6F-B771F91217E5@icir.org>
References: <20120822204423.GB9894@chaz.gmail.com>
	<A411F734-EBF9-42FB-8D6F-B771F91217E5@icir.org>
Message-ID: <20120823142559.GC5744@chaz.gmail.com>

2012-08-23 09:56:17 -0400, Seth Hall:
> 
> On Aug 22, 2012, at 4:44 PM, Stephane Chazelas <stephane.chazelas at gmail.com> wrote:
> 
> > It detects skype traffic by looking at the fake SSL
> > "ServerHello" that skype responders send. (basically, they send
> > a fixed "random data" with a date in 2004 where a normal SSL
> > server would send the current date and a truly random data, I
> > suspect it is designed that way to help recognise skype traffic
> > easily).
> 
> Cool technique!  Thanks for sharing.  Do these connections
> show up in ssl.log or generate the ssl_server_hello event?  It
> would probably be better to detect them through the SSL
> analyzer if possible.

Hi Seth, thanks.

It's not real SSL, though some of them do show up occasionally
as "ssl" and some of them cause some errors. The packets past
the ServerHello, are not SSL packets.

> > function mark_conn_as_skype(state: signature_state): bool
> >        {
> >        add state$conn$service["skype"];
> >        return T;
> >        }
> > redef signature_files += "skype-detect.sig";
> 
> 
> I have a couple of comments here?
> 
> The prototype for your function should be: 
> 	function cond(state: signature_state, data: string): bool;
> 
> You are missing the data variable which could be partly what's
> contributing to the crash you are seeing.

I copy-pasted from ssl-worm.bro in securityonion which BTW has comments like:
        "# FIXME: Bro segfaults without the tmp variable"
which made me try and use a tmp variable as well.

After adding the ",data: string" and reverting to add
state$conn$service, it seems not to crash, so it's probably what
the problem was.


[...]
> Additionally, beginning with 2.0 you can use the @load-sigs
> directive which gives you relative path loading so you can
> distribute your protocol detection script as a "module" and it
> doesn't matter where people put it on the BROPATH the
> skype-detect.sig signatures would still be find-able.  The
> following example will look for the signature file in the same
> directory of the script that called @load-sigs.
> 
> 	@load-sigs ./skype-detect.sig

It didn't like it:
   error in /usr/local/share/bro/site/local.bro, line 90: unrecognized character - @
   error in /usr/local/share/bro/site/local.bro, line 90: unknown identifier load, at or near "load"

I can't see any mention of "load-sigs" in the source. Are you
sure it's not in a newer version.

> One other comment, we do have a Skype analyzer that is
> currently not enabled in the script-land.  That should detect
> and log Skype connections as well (the base scripts need to be
> written for it still).

I enquired about that on the list a few days ago, as I wasn't
able to find it. Someone kindly sent me a version that was
designed for an older version of bro, and goes far beyond what I
need (identify those port 443 connections).

> Again, thanks for sending that in!  Definitely a cool trick.
> Do you think you could package it up in a git repository like
> I've been doing with my recent scripts?  The ssn-exposure
> script even has an example of @load-sigs
> 
> 	https://github.com/sethhall/ssn-exposure
> 	https://github.com/sethhall/relog
[...]

I'll have a look.

-- 
Stephane


From seth at icir.org  Thu Aug 23 07:38:59 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 23 Aug 2012 10:38:59 -0400
Subject: [Bro] setting a connection "service" in a signature
In-Reply-To: <20120823142559.GC5744@chaz.gmail.com>
References: <20120822204423.GB9894@chaz.gmail.com>
	<A411F734-EBF9-42FB-8D6F-B771F91217E5@icir.org>
	<20120823142559.GC5744@chaz.gmail.com>
Message-ID: <06A44C1E-0344-4F92-A9E3-7C79AFC076D4@icir.org>


On Aug 23, 2012, at 10:25 AM, Stephane Chazelas <stephane.chazelas at gmail.com> wrote:

> I copy-pasted from ssl-worm.bro in securityonion which BTW has comments like:
>        "# FIXME: Bro segfaults without the tmp variable"
> which made me try and use a tmp variable as well.

Ah, securityonion has a problem right now because they had installed 1.5 and it wasn't installed as a package so they couldn't delete the older scripts.  2.0 was installed as a package over top of it.  ssl-worm.bro is an older script that shouldn't even be there.

> After adding the ",data: string" and reverting to add
> state$conn$service, it seems not to crash, so it's probably what
> the problem was.

Yep, apparently we need to have that as a syntax error if a signature eval function doesn't have the proper syntax.  Robin, Jon, any idea of if that would be possible?

> I can't see any mention of "load-sigs" in the source. Are you
> sure it's not in a newer version.

Hah, oops.  Sorry about that.  It was added long enough ago that I thought it was in 2.0 apparently it's going to be in 2.1 though.  It will work *very* soon. :)

> I enquired about that on the list a few days ago, as I wasn't
> able to find it. Someone kindly sent me a version that was
> designed for an older version of bro, and goes far beyond what I
> need (identify those port 443 connections).

Sorry about not responding to that.  I was meaning to get back to it but I obviously didn't.  I'm actually glad everything worked out like it did though and you wrote your new script.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From stephane.chazelas at gmail.com  Thu Aug 23 07:48:45 2012
From: stephane.chazelas at gmail.com (Stephane Chazelas)
Date: Thu, 23 Aug 2012 15:48:45 +0100
Subject: [Bro] reverse DNS based on bro's forward DNS query log
Message-ID: <20120823144845.GD5744@chaz.gmail.com>

Hiya,

while I'm at sharing bro related stuff, I've got a setup here
that parses bro's DNS log in real time and updates the database
of a local DNS server (powerdns with mysql backend) so as to
provide with  more  useful PTR records.

$ tail -1 dns.log
1345732627.030897       jUJU3ZwGOv4     x.x.x.x      54866   x.x.x.x   53      udp     44687   static.ak.facebook.com  1       C_INTERNET      1       A       0       NOERROR F     F
        F       T       T       0       static.ak.facebook.com.edgesuite.net,a749.dsw4.akamai.net,84.53.132.80,84.53.132.88     3364.000000,348.000000,15.000000,15.000000

$ dig -x 84.53.132.88 +short
static.ak.facebook.com.C-EU.120823T143707.

For many "cloud" IP addresses, it won't necessarily be useful,
but in many case, it gives more useful information than the
real PTR record.

Above, it tells us that 84.53.132.88 was last the result of a
query for an A record for static.ak.facebook.com (at 14:37:07),
and it gives you the result of the geoiplookup.

It can be generalised for other things (like for internal IP
addresses, I use other sources of information to feed the
powerdns database).

If anybody is interested, I can post the intructions on how to
set it up along with the small perl script that parses the bro
dns logs here or on github.

-- 
Stephane



From seth at icir.org  Thu Aug 23 08:15:23 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 23 Aug 2012 11:15:23 -0400
Subject: [Bro] reverse DNS based on bro's forward DNS query log
In-Reply-To: <20120823144845.GD5744@chaz.gmail.com>
References: <20120823144845.GD5744@chaz.gmail.com>
Message-ID: <575F9FF7-456F-45F6-A6FB-525C5E20EFE3@icir.org>


On Aug 23, 2012, at 10:48 AM, Stephane Chazelas <stephane.chazelas at gmail.com> wrote:

> $ tail -1 dns.log
> 1345732627.030897       jUJU3ZwGOv4     x.x.x.x      54866   x.x.x.x   53      udp     44687   static.ak.facebook.com  1       C_INTERNET      1       A       0       NOERROR F     F
>        F       T       T       0       static.ak.facebook.com.edgesuite.net,a749.dsw4.akamai.net,84.53.132.80,84.53.132.88     3364.000000,348.000000,15.000000,15.000000
> 
> $ dig -x 84.53.132.88 +short
> static.ak.facebook.com.C-EU.120823T143707.

That's cool!  Definitely send along anything you can.  I'm sure that quite a few people will be interested in this (I am).  

In 2.2 we should have some database logging framework writer plugins so we might be able to remove your script eventually and have Bro send these logs directly to the database.

Yet another cool Bro thing!  You're on a roll today.

FYI, the mailing list address is bro at bro-ids.org now.  The old lbl.gov address was deprecated a while ago.

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jsiwek at illinois.edu  Thu Aug 23 10:04:24 2012
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Thu, 23 Aug 2012 17:04:24 +0000
Subject: [Bro] setting a connection "service" in a signature
In-Reply-To: <06A44C1E-0344-4F92-A9E3-7C79AFC076D4@icir.org>
References: <20120822204423.GB9894@chaz.gmail.com>
	<A411F734-EBF9-42FB-8D6F-B771F91217E5@icir.org>
	<20120823142559.GC5744@chaz.gmail.com>
	<06A44C1E-0344-4F92-A9E3-7C79AFC076D4@icir.org>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D116E869EA@CITESMBX2.ad.uillinois.edu>


> 
> Yep, apparently we need to have that as a syntax error if a signature eval function doesn't have the proper syntax.  Robin, Jon, any idea of if that would be possible?

Yes, fixed on fastpath branch now.

    Jon


From seth at icir.org  Thu Aug 23 10:06:16 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 23 Aug 2012 13:06:16 -0400
Subject: [Bro] setting a connection "service" in a signature
In-Reply-To: <B3E27C90B7D2DD4CAEF09215A29EB0D116E869EA@CITESMBX2.ad.uillinois.edu>
References: <20120822204423.GB9894@chaz.gmail.com>
	<A411F734-EBF9-42FB-8D6F-B771F91217E5@icir.org>
	<20120823142559.GC5744@chaz.gmail.com>
	<06A44C1E-0344-4F92-A9E3-7C79AFC076D4@icir.org>
	<B3E27C90B7D2DD4CAEF09215A29EB0D116E869EA@CITESMBX2.ad.uillinois.edu>
Message-ID: <92A32FB0-3F6B-4CD8-8F80-6BDD10EB724A@icir.org>


On Aug 23, 2012, at 1:04 PM, "Siwek, Jonathan Luke" <jsiwek at illinois.edu> wrote:

>> Yep, apparently we need to have that as a syntax error if a signature eval function doesn't have the proper syntax.  Robin, Jon, any idea of if that would be possible?
> 
> Yes, fixed on fastpath branch now.


Wha!?  Awesome!  I didn't expect to see that fixed yet.  As always, thanks Jon!

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Thu Aug 23 13:55:49 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 23 Aug 2012 16:55:49 -0400
Subject: [Bro] pagerduty.com?
Message-ID: <E8C54DDC-0067-460A-BFC8-97A5D368EADB@icir.org>

Does anyone here use pagerduty.com?  I was playing around and I think I could integrate with it pretty easily, but someone actually using it and benefitting from a script might change whether or not that script actually gets written. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/


From stephane.chazelas at gmail.com  Fri Aug 24 05:33:41 2012
From: stephane.chazelas at gmail.com (Stephane Chazelas)
Date: Fri, 24 Aug 2012 13:33:41 +0100
Subject: [Bro] setting a connection "service" in a signature
In-Reply-To: <A411F734-EBF9-42FB-8D6F-B771F91217E5@icir.org>
References: <20120822204423.GB9894@chaz.gmail.com>
	<A411F734-EBF9-42FB-8D6F-B771F91217E5@icir.org>
Message-ID: <20120824123341.GA9908@chaz.gmail.com>

2012-08-23 09:56:17 -0400, Seth Hall:
[...]
> Again, thanks for sending that in!  Definitely a cool trick.
> Do you think you could package it up in a git repository like
> I've been doing with my recent scripts?  The ssn-exposure
> script even has an example of @load-sigs
> 
> 	https://github.com/sethhall/ssn-exposure
> 	https://github.com/sethhall/relog
[...]

Here you go:
https://github.com/stephane-chazelas/bro-skype-fake-https-detect

unfortunately, I couldn't test it. The bro I compiled from the
git head doesn't detect TCP connections properly (all marked as
OTH even after I disable NIC offloads), and I don't have any
time to look at it in any more detail.

-- 
Stephane


From seth at icir.org  Fri Aug 24 05:47:39 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 24 Aug 2012 08:47:39 -0400
Subject: [Bro] setting a connection "service" in a signature
In-Reply-To: <20120824123341.GA9908@chaz.gmail.com>
References: <20120822204423.GB9894@chaz.gmail.com>
	<A411F734-EBF9-42FB-8D6F-B771F91217E5@icir.org>
	<20120824123341.GA9908@chaz.gmail.com>
Message-ID: <122ED161-9183-4435-ADA7-FF3C44BE95E5@icir.org>


On Aug 24, 2012, at 8:33 AM, Stephane Chazelas <stephane.chazelas at gmail.com> wrote:

> Here you go:
> https://github.com/stephane-chazelas/bro-skype-fake-https-detect

Cool, thanks!  

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From stephane.chazelas at gmail.com  Fri Aug 24 07:41:40 2012
From: stephane.chazelas at gmail.com (Stephane Chazelas)
Date: Fri, 24 Aug 2012 15:41:40 +0100
Subject: [Bro] reverse DNS based on bro's forward DNS query log
In-Reply-To: <575F9FF7-456F-45F6-A6FB-525C5E20EFE3@icir.org>
References: <20120823144845.GD5744@chaz.gmail.com>
	<575F9FF7-456F-45F6-A6FB-525C5E20EFE3@icir.org>
Message-ID: <20120824144140.GB9908@chaz.gmail.com>

2012-08-23 11:15:23 -0400, Seth Hall:
> 
> On Aug 23, 2012, at 10:48 AM, Stephane Chazelas <stephane.chazelas at gmail.com> wrote:
> 
> > $ tail -1 dns.log
> > 1345732627.030897       jUJU3ZwGOv4     x.x.x.x      54866   x.x.x.x   53      udp     44687   static.ak.facebook.com  1       C_INTERNET      1       A       0       NOERROR F     F
> >        F       T       T       0       static.ak.facebook.com.edgesuite.net,a749.dsw4.akamai.net,84.53.132.80,84.53.132.88     3364.000000,348.000000,15.000000,15.000000
> > 
> > $ dig -x 84.53.132.88 +short
> > static.ak.facebook.com.C-EU.120823T143707.
> 
> That's cool!  Definitely send along anything you can.  I'm sure that quite a few people will be interested in this (I am).  
[...]

Here you go:
https://github.com/stephane-chazelas/bro-pdns-forward-dns
please test and tell me what you think.

-- 
Stephane


From sconzo at visiblerisk.com  Fri Aug 24 10:06:22 2012
From: sconzo at visiblerisk.com (Mike Sconzo)
Date: Fri, 24 Aug 2012 12:06:22 -0500
Subject: [Bro] RESTful shim for Time Machine
Message-ID: <CAGZoo6HurdhUA4zJYmbbrg-DRm4P1v-FGXGW0e4ZABTbGMf-JQ@mail.gmail.com>

Has anybody created or thought of creating a REST shim for Time
Machine? I was going to create something pretty basic, but enough to
send HTTP commands to pull back pcaps from Time Machine w/o having to
SSH to the box. If nobody has done it, that's ok I'll be sure to post
mine when I'm finished with it.

-=Mike

-- 
cat ~/.bash_history > documentation.txt


From JAzoff at albany.edu  Fri Aug 24 10:18:57 2012
From: JAzoff at albany.edu (Justin Azoff)
Date: Fri, 24 Aug 2012 13:18:57 -0400
Subject: [Bro] RESTful shim for Time Machine
In-Reply-To: <CAGZoo6HurdhUA4zJYmbbrg-DRm4P1v-FGXGW0e4ZABTbGMf-JQ@mail.gmail.com>
References: <CAGZoo6HurdhUA4zJYmbbrg-DRm4P1v-FGXGW0e4ZABTbGMf-JQ@mail.gmail.com>
Message-ID: <20120824171857.GS1957@datacomm.albany.edu>

On Fri, Aug 24, 2012 at 12:06:22PM -0500, Mike Sconzo wrote:
> Has anybody created or thought of creating a REST shim for Time
> Machine? I was going to create something pretty basic, but enough to
> send HTTP commands to pull back pcaps from Time Machine w/o having to
> SSH to the box. If nobody has done it, that's ok I'll be sure to post
> mine when I'm finished with it.
> 
> -=Mike

I have such a thing.. I thought it was on github but apparently not.
It's so tiny so I'm attaching it now.  I can push it to github, but I
need to audit the historical changes first to make sure I didn't have
anything sensitive in there at some point.

-- 
-- Justin Azoff
-- Network Security & Performance Analyst
-------------- next part --------------
A non-text attachment was scrubbed...
Name: time-machine-server-0.6.tar.gz
Type: application/octet-stream
Size: 2392 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120824/f185c5ab/attachment.obj 

From sconzo at visiblerisk.com  Fri Aug 24 11:50:50 2012
From: sconzo at visiblerisk.com (Mike Sconzo)
Date: Fri, 24 Aug 2012 13:50:50 -0500
Subject: [Bro] RESTful shim for Time Machine
In-Reply-To: <20120824171857.GS1957@datacomm.albany.edu>
References: <CAGZoo6HurdhUA4zJYmbbrg-DRm4P1v-FGXGW0e4ZABTbGMf-JQ@mail.gmail.com>
	<20120824171857.GS1957@datacomm.albany.edu>
Message-ID: <CAGZoo6GdqSvXz3RnhbfcW99Nmg0T+rS7yOoHaBYnSFknCJ2aQg@mail.gmail.com>

Awesome and thanks! I'll play around with this.


On Fri, Aug 24, 2012 at 12:18 PM, Justin Azoff <JAzoff at albany.edu> wrote:
> On Fri, Aug 24, 2012 at 12:06:22PM -0500, Mike Sconzo wrote:
>> Has anybody created or thought of creating a REST shim for Time
>> Machine? I was going to create something pretty basic, but enough to
>> send HTTP commands to pull back pcaps from Time Machine w/o having to
>> SSH to the box. If nobody has done it, that's ok I'll be sure to post
>> mine when I'm finished with it.
>>
>> -=Mike
>
> I have such a thing.. I thought it was on github but apparently not.
> It's so tiny so I'm attaching it now.  I can push it to github, but I
> need to audit the historical changes first to make sure I didn't have
> anything sensitive in there at some point.
>
> --
> -- Justin Azoff
> -- Network Security & Performance Analyst



-- 
cat ~/.bash_history > documentation.txt


From christopher.p.crawford at gmail.com  Mon Aug 27 14:48:02 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Mon, 27 Aug 2012 17:48:02 -0400
Subject: [Bro] Converting a Bro Script to A New Stream
In-Reply-To: <A69CFAEE-A138-46B6-89F5-7705FA714ECD@icir.org>
References: <CAG1VgOab88X87hCP01PXtT3n7hJkhnY=Mf7kRa_YfNV5+V3Vaw@mail.gmail.com>
	<A69CFAEE-A138-46B6-89F5-7705FA714ECD@icir.org>
Message-ID: <CAG1VgOb2=Rrw9PgSedKJqraiht0+w-jwCJdFeT=bRM7PV_a3ww@mail.gmail.com>

Thanks, Seth.  This works great, and it gives me better insight into
how to write my own bro scripts.

Now, I feel like my follow up question has an obvious answer, but I'm
just not seeing it --

Let's say I want to also email the alert, in addition to logging it.
I've added a notice statement, an attempted to redefine
Notice::mail_dest and Notice::policy as outlined in the bro docs:
http://www.bro-ids.org/documentation/notice.html

When I run the script, though, I don't receive an email.  I know that
bro's email is working, because I am receiving hourly reports.  What
am I missing?

Attached the script, but if it doesn't post to the list, I'll follow
up this post with the code.

-Chris

On Mon, Aug 20, 2012 at 10:42 AM, Seth Hall <seth at icir.org> wrote:
>
> On Aug 16, 2012, at 5:32 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:
>
>> redef record rec += {
>>        foo: Info &optional;
>> };
>
>> error in ./test.bro, line 22: unknown identifier (Foo::rec)
>> error in ./test.bro, line 35 and ./test.bro, line 39: already defined (Foo::rec)
>
> I don't think you need that little chunk of code I left above.  We do that in many base scripts as a way of hiding protocol specific information within the connection record.  There is no existing record type named "rec" though and it doesn't look like you need to hide this information anywhere since you are deriving all of your log directly from data in the DNS::log_dns event.
>
> There is a better way to do this though and it was something we specifically considered in the logging framework.  Here's a log filter you can run that will give you the log you want?
>
> event bro_init()
>         {
>         local filter: Log::Filter = [
>                 $name="only-1.2.3.4",
>                 $path="foo",
>                 $pred(rec: DNS::Info) = {
>                         if ( rec?$qtype_name && rec?$answers &&
>                              rec$qtype_name == "A" )
>                                 {
>                                 for ( i in rec$answers )
>                                         if ( "1.2.3.4" in rec$answers[i] )
>                                                 return T;
>                                 }
>                         return F;
>                 },
>                 $include=set("ts", "uid", "id.orig_h", "query")];
>         Log::add_filter(DNS::LOG, filter);
>         }
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
-------------- next part --------------
export {
        redef Notice::mail_dest = "email_address at inet.com";

        redef enum Notice::Type += {
                Foo,
        };

        redef Notice::emailed_types += {
                Foo,
        };
}



redef Notice::policy += {
  [$pred(n: Notice::Info) = {
     return n$note == Foo;
   },
   $action = Notice::ACTION_EMAIL]
  };

event bro_init()
        {
        local filter: Log::Filter = [
                $name="poison_hits",
                $path="poison_hits",
                $pred(rec: DNS::Info) = {
                        if ( rec?$qtype_name && rec?$answers && rec$qtype_name == "A" )
                                {
                                for ( i in rec$answers )
                                        if ( "1.2.3.4" in rec$answers[i] )
                                                {
                                                NOTICE([$note=Foo, $msg="Foo detected."]);
                                                return T;
                                                }
                                }
                        return F;
                },
                $include=set("ts", "uid", "id.orig_h", "query")];
        Log::add_filter(DNS::LOG, filter);
        }

From christopher.p.crawford at gmail.com  Mon Aug 27 14:49:54 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Mon, 27 Aug 2012 17:49:54 -0400
Subject: [Bro] Converting a Bro Script to A New Stream
In-Reply-To: <CAG1VgOb2=Rrw9PgSedKJqraiht0+w-jwCJdFeT=bRM7PV_a3ww@mail.gmail.com>
References: <CAG1VgOab88X87hCP01PXtT3n7hJkhnY=Mf7kRa_YfNV5+V3Vaw@mail.gmail.com>
	<A69CFAEE-A138-46B6-89F5-7705FA714ECD@icir.org>
	<CAG1VgOb2=Rrw9PgSedKJqraiht0+w-jwCJdFeT=bRM7PV_a3ww@mail.gmail.com>
Message-ID: <CAG1VgObifWpo6B_fvspSBhHhmPNBO98C6-c6XkeZYu=eg36gTQ@mail.gmail.com>

Looks like the code didn't post.

For the benefit of the mailing list, this is what the script looks like:

export {
        redef Notice::mail_dest = "email_address at inet.com";

        redef enum Notice::Type += {
                Foo,
        };

        redef Notice::emailed_types += {
                Foo,
        };
}



redef Notice::policy += {
  [$pred(n: Notice::Info) = {
     return n$note == Foo;
   },
   $action = Notice::ACTION_EMAIL]
  };

event bro_init()
        {
        local filter: Log::Filter = [
                $name="poison_hits",
                $path="poison_hits",
                $pred(rec: DNS::Info) = {
                        if ( rec?$qtype_name && rec?$answers &&
rec$qtype_name == "A" )
                                {
                                for ( i in rec$answers )
                                        if ( "1.2.3.4" in rec$answers[i] )
                                                {
                                                NOTICE([$note=Foo,
$msg="Foo detected."]);
                                                return T;
                                                }
                                }
                        return F;
                },
                $include=set("ts", "uid", "id.orig_h", "query")];
        Log::add_filter(DNS::LOG, filter);
        }


On Mon, Aug 27, 2012 at 5:48 PM, Chris Crawford
<christopher.p.crawford at gmail.com> wrote:
> Thanks, Seth.  This works great, and it gives me better insight into
> how to write my own bro scripts.
>
> Now, I feel like my follow up question has an obvious answer, but I'm
> just not seeing it --
>
> Let's say I want to also email the alert, in addition to logging it.
> I've added a notice statement, an attempted to redefine
> Notice::mail_dest and Notice::policy as outlined in the bro docs:
> http://www.bro-ids.org/documentation/notice.html
>
> When I run the script, though, I don't receive an email.  I know that
> bro's email is working, because I am receiving hourly reports.  What
> am I missing?
>
> Attached the script, but if it doesn't post to the list, I'll follow
> up this post with the code.
>
> -Chris
>
> On Mon, Aug 20, 2012 at 10:42 AM, Seth Hall <seth at icir.org> wrote:
>>
>> On Aug 16, 2012, at 5:32 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:
>>
>>> redef record rec += {
>>>        foo: Info &optional;
>>> };
>>
>>> error in ./test.bro, line 22: unknown identifier (Foo::rec)
>>> error in ./test.bro, line 35 and ./test.bro, line 39: already defined (Foo::rec)
>>
>> I don't think you need that little chunk of code I left above.  We do that in many base scripts as a way of hiding protocol specific information within the connection record.  There is no existing record type named "rec" though and it doesn't look like you need to hide this information anywhere since you are deriving all of your log directly from data in the DNS::log_dns event.
>>
>> There is a better way to do this though and it was something we specifically considered in the logging framework.  Here's a log filter you can run that will give you the log you want?
>>
>> event bro_init()
>>         {
>>         local filter: Log::Filter = [
>>                 $name="only-1.2.3.4",
>>                 $path="foo",
>>                 $pred(rec: DNS::Info) = {
>>                         if ( rec?$qtype_name && rec?$answers &&
>>                              rec$qtype_name == "A" )
>>                                 {
>>                                 for ( i in rec$answers )
>>                                         if ( "1.2.3.4" in rec$answers[i] )
>>                                                 return T;
>>                                 }
>>                         return F;
>>                 },
>>                 $include=set("ts", "uid", "id.orig_h", "query")];
>>         Log::add_filter(DNS::LOG, filter);
>>         }
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro-ids.org/
>>



From Corey.Roach at utah.edu  Mon Aug 27 15:41:48 2012
From: Corey.Roach at utah.edu (Corey Roach  (ISO))
Date: Mon, 27 Aug 2012 22:41:48 +0000
Subject: [Bro] BPF packet filter syntax
Message-ID: <AE802E5B2E279644BDF56F44B9EE6FCC2D48D95D@X-MB6.xds.umail.utah.edu>

Before I go dive into source I thought I'd throw a quick question to the group.

Can you use the entire BPF syntax (things other than just "host") when building a Bro filter?

For example, I've got something like this in my local.bro:

redef PacketFilter::all_packets = F;
redef capture_filters = [[ "all"] = "ip or not ip"];
redef restrict_filters += [ ["not-one-host"] = "not host 10.10.1.1"];
redef restrict_filters += [ ["not-two-hosts"] = "not host 10.20.1.1 and not host 10.30.1.1"];
redef restrict_filters += [ ["not-one-net"] = "not net 10.40.1.192/26"];
redef restrict_filters += [ ["not-two-nets"] = "not net 10.50.1.0/20 and not net 10.60.1.0/22"];

But it seems that the "10.50.1.0/20" network is still leaking in traffic?

Ultimately I'd like to eliminate the traffic at my upstream device, but in the mean time, does anyone see something I'm doing obviously wrong?

Thanks,
Corey




From seth at icir.org  Mon Aug 27 19:39:17 2012
From: seth at icir.org (Seth Hall)
Date: Mon, 27 Aug 2012 22:39:17 -0400
Subject: [Bro] BPF packet filter syntax
In-Reply-To: <AE802E5B2E279644BDF56F44B9EE6FCC2D48D95D@X-MB6.xds.umail.utah.edu>
References: <AE802E5B2E279644BDF56F44B9EE6FCC2D48D95D@X-MB6.xds.umail.utah.edu>
Message-ID: <36B60826-EE96-4A23-A222-BBBDE3ACF714@icir.org>


On Aug 27, 2012, at 6:41 PM, Corey Roach (ISO) <Corey.Roach at utah.edu> wrote:

> redef restrict_filters += [ ["not-two-nets"] = "not net 10.50.1.0/20 and not net 10.60.1.0/22"];

I'm surprised that Bro is starting up for you.  When I try running with these lines I get a message about a bad filter.  The line that I left above doesn't work as a valid BPF filter, there are network bits beyond the netmask which BPF doesn't seem to like.

I think the CIDRs you meant to use are:
	10.50.0.0/20
	10.60.0.0/11

So use this instead:
	redef restrict_filters += { ["not-two-nets"] = "not net 10.50.0.0/20 and not net 10.60.0.0/22" };


> Ultimately I'd like to eliminate the traffic at my upstream device, but in the mean time, does anyone see something I'm doing obviously wrong?


Once again, I'd like to apologize to everyone for not getting the rewritten packet filter framework into 2.1.  This will be so much easier when that's finally included (the worst part is that it's already done!).

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From sconzo at visiblerisk.com  Mon Aug 27 21:14:24 2012
From: sconzo at visiblerisk.com (Mike Sconzo)
Date: Mon, 27 Aug 2012 23:14:24 -0500
Subject: [Bro] Converting a Bro Script to A New Stream
In-Reply-To: <A69CFAEE-A138-46B6-89F5-7705FA714ECD@icir.org>
References: <CAG1VgOab88X87hCP01PXtT3n7hJkhnY=Mf7kRa_YfNV5+V3Vaw@mail.gmail.com>
	<A69CFAEE-A138-46B6-89F5-7705FA714ECD@icir.org>
Message-ID: <CAGZoo6ECqnPNC5xmVstmqf+g--cyrhbkn=byzUFOxZ3-rEBbJw@mail.gmail.com>

I've got a general question around this idea. When is it best to
create a log filter vs. creating a log file with just the information
you want in it vs. adding an additional field to a existing log?

I'm curious becase I would have approached this the same way (an
output log for the data I was interested in), but you mentioned that
for this case the filter was the better way to do this.

Any insight would be appreciated.

Thanks!

On Mon, Aug 20, 2012 at 9:42 AM, Seth Hall <seth at icir.org> wrote:
>
> On Aug 16, 2012, at 5:32 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:
>
>> redef record rec += {
>>        foo: Info &optional;
>> };
>
>> error in ./test.bro, line 22: unknown identifier (Foo::rec)
>> error in ./test.bro, line 35 and ./test.bro, line 39: already defined (Foo::rec)
>
> I don't think you need that little chunk of code I left above.  We do that in many base scripts as a way of hiding protocol specific information within the connection record.  There is no existing record type named "rec" though and it doesn't look like you need to hide this information anywhere since you are deriving all of your log directly from data in the DNS::log_dns event.
>
> There is a better way to do this though and it was something we specifically considered in the logging framework.  Here's a log filter you can run that will give you the log you want?
>
> event bro_init()
>         {
>         local filter: Log::Filter = [
>                 $name="only-1.2.3.4",
>                 $path="foo",
>                 $pred(rec: DNS::Info) = {
>                         if ( rec?$qtype_name && rec?$answers &&
>                              rec$qtype_name == "A" )
>                                 {
>                                 for ( i in rec$answers )
>                                         if ( "1.2.3.4" in rec$answers[i] )
>                                                 return T;
>                                 }
>                         return F;
>                 },
>                 $include=set("ts", "uid", "id.orig_h", "query")];
>         Log::add_filter(DNS::LOG, filter);
>         }
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
cat ~/.bash_history > documentation.txt



From Corey.Roach at utah.edu  Mon Aug 27 22:24:33 2012
From: Corey.Roach at utah.edu (Corey Roach  (ISO))
Date: Tue, 28 Aug 2012 05:24:33 +0000
Subject: [Bro] BPF packet filter syntax
In-Reply-To: <36B60826-EE96-4A23-A222-BBBDE3ACF714@icir.org>
References: <AE802E5B2E279644BDF56F44B9EE6FCC2D48D95D@X-MB6.xds.umail.utah.edu>
	<36B60826-EE96-4A23-A222-BBBDE3ACF714@icir.org>
Message-ID: <AE802E5B2E279644BDF56F44B9EE6FCC2D48E7D3@X-MB6.xds.umail.utah.edu>

Actually, I fudged the addresses in an attempt to simplify the question, but I neglected to check the cidr boundaries, so I guess it didn't make it simpler after all. :)


More specifically, the actual line in question is:

redef restrict_filters += [ ["not-flux"] = "not net 155.98.32.0/20 and not net 155.98.60.0/22"];

And I'm getting the notice:

SSH::Password_Guessing  Threshold crossed by metric_index(host=137.132.209.209) 30/30

from "137.132.209.209" pounding on the IP "155.98.35.7".

So I was thinking the filter should have dropped the traffic before the SSH connection got process or ever reached the detect-bruteforcing.bro script and not incremented the counter or kicked off the notice.


Thanks for answering my question in the general that I should be able to use "net" as well as "host", but can you see any reason that should not be working?

 - Corey


On Aug 27, 2012, at 8:39 PM, Seth Hall <seth at icir.org> wrote:

> 
> On Aug 27, 2012, at 6:41 PM, Corey Roach (ISO) <Corey.Roach at utah.edu> wrote:
> 
>> redef restrict_filters += [ ["not-two-nets"] = "not net 10.50.1.0/20 and not net 10.60.1.0/22"];
> 
> I'm surprised that Bro is starting up for you.  When I try running with these lines I get a message about a bad filter.  The line that I left above doesn't work as a valid BPF filter, there are network bits beyond the netmask which BPF doesn't seem to like.
> 
> I think the CIDRs you meant to use are:
> 	10.50.0.0/20
> 	10.60.0.0/11
> 
> So use this instead:
> 	redef restrict_filters += { ["not-two-nets"] = "not net 10.50.0.0/20 and not net 10.60.0.0/22" };
> 
> 
>> Ultimately I'd like to eliminate the traffic at my upstream device, but in the mean time, does anyone see something I'm doing obviously wrong?
> 
> 
> Once again, I'd like to apologize to everyone for not getting the rewritten packet filter framework into 2.1.  This will be so much easier when that's finally included (the worst part is that it's already done!).
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
> 




From seth at icir.org  Mon Aug 27 22:34:59 2012
From: seth at icir.org (Seth Hall)
Date: Tue, 28 Aug 2012 01:34:59 -0400
Subject: [Bro] BPF packet filter syntax
In-Reply-To: <AE802E5B2E279644BDF56F44B9EE6FCC2D48E7D3@X-MB6.xds.umail.utah.edu>
References: <AE802E5B2E279644BDF56F44B9EE6FCC2D48D95D@X-MB6.xds.umail.utah.edu>
	<36B60826-EE96-4A23-A222-BBBDE3ACF714@icir.org>
	<AE802E5B2E279644BDF56F44B9EE6FCC2D48E7D3@X-MB6.xds.umail.utah.edu>
Message-ID: <4459C289-FCA3-4452-B377-B77935BC2E6B@icir.org>


On Aug 28, 2012, at 1:24 AM, "Corey Roach  (ISO)" <Corey.Roach at utah.edu> wrote:

> Actually, I fudged the addresses in an attempt to simplify the question, but I neglected to check the cidr boundaries, so I guess it didn't make it simpler after all. :)

Hah!  No problem.

> redef restrict_filters += [ ["not-flux"] = "not net 155.98.32.0/20 and not net 155.98.60.0/22"];

One funny thing is that I was *really* surprised that this syntax works.  Normally I would surround the table values with curly braces like this?

redef restrict_filters += { ["not-flux"] = "not net 155.98.32.0/20 and not net 155.98.60.0/22" };

At some point we're really going to have to tighten up the language's use of curly braces and square brackets, there is still a lot of inconsistency floating around.

> SSH::Password_Guessing  Threshold crossed by metric_index(host=137.132.209.209) 30/30
> 
> from "137.132.209.209" pounding on the IP "155.98.35.7".

Are you seeing that address (155.98.35.7) in your conn.log?  I assume you are, but the metric based detection doesn't inherently indicate that.

> So I was thinking the filter should have dropped the traffic before the SSH connection got process or ever reached the detect-bruteforcing.bro script and not incremented the counter or kicked off the notice.

Yeah, I would think so too.

Could you send me the output from the packet_filter.log?  Feel free to send it off list if it has anything in it that you'd rather not broadcast publicly.

Thanks,
  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Mon Aug 27 22:41:10 2012
From: seth at icir.org (Seth Hall)
Date: Tue, 28 Aug 2012 01:41:10 -0400
Subject: [Bro] Converting a Bro Script to A New Stream
In-Reply-To: <CAGZoo6ECqnPNC5xmVstmqf+g--cyrhbkn=byzUFOxZ3-rEBbJw@mail.gmail.com>
References: <CAG1VgOab88X87hCP01PXtT3n7hJkhnY=Mf7kRa_YfNV5+V3Vaw@mail.gmail.com>
	<A69CFAEE-A138-46B6-89F5-7705FA714ECD@icir.org>
	<CAGZoo6ECqnPNC5xmVstmqf+g--cyrhbkn=byzUFOxZ3-rEBbJw@mail.gmail.com>
Message-ID: <2072986A-5E15-4EAF-B688-4AAC097A2E1F@icir.org>


On Aug 28, 2012, at 12:14 AM, Mike Sconzo <sconzo at visiblerisk.com> wrote:

> I've got a general question around this idea. When is it best to
> create a log filter vs. creating a log file with just the information
> you want in it vs. adding an additional field to a existing log?


In this case, the log being created was a strict subset of the existing log.  I definitely wouldn't bother creating a completely new log just to restrict the output columns since we have that supported as a feature of the logging framework.

I guess in most cases where you are just going to end up outputting all of the same fields from an existing log I wouldn't bother creating a new log stream.  There are some cases where it just comes down to a style decision if you are adding extra fields but you are also including several fields from an existing log whether or not you create a new log stream.

A good rule of thumb is if your first thought is to handle a log stream event (i.e. DNS::log_dns, HTTP::log_http, etc) to fill out your log file, you should probably use a filter.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From Corey.Roach at utah.edu  Tue Aug 28 06:50:08 2012
From: Corey.Roach at utah.edu (Corey Roach  (ISO))
Date: Tue, 28 Aug 2012 13:50:08 +0000
Subject: [Bro] BPF packet filter syntax
In-Reply-To: <4459C289-FCA3-4452-B377-B77935BC2E6B@icir.org>
References: <AE802E5B2E279644BDF56F44B9EE6FCC2D48D95D@X-MB6.xds.umail.utah.edu>
	<36B60826-EE96-4A23-A222-BBBDE3ACF714@icir.org>
	<AE802E5B2E279644BDF56F44B9EE6FCC2D48E7D3@X-MB6.xds.umail.utah.edu>
	<4459C289-FCA3-4452-B377-B77935BC2E6B@icir.org>
Message-ID: <AE802E5B2E279644BDF56F44B9EE6FCC2D48EF00@X-MB6.xds.umail.utah.edu>

On Aug 27, 2012, at 11:34 PM, Seth Hall <seth at icir.org> wrote:

> One funny thing is that I was *really* surprised that this syntax works.  Normally I would surround the table values with curly braces like this?

> At some point we're really going to have to tighten up the language's use of curly braces and square brackets, there is still a lot of inconsistency floating around.

Ha! That'll teach me to copy-and-paste from the list without paying attention to syntax...

I fixed the square-brackets to curly-brackets, but it did not see to fix the problem. I'll dig around and send you some logs offline as soon as I get a chance.

Thanks!




From christopher.p.crawford at gmail.com  Tue Aug 28 08:55:32 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Tue, 28 Aug 2012 11:55:32 -0400
Subject: [Bro] Converting a Bro Script to A New Stream
In-Reply-To: <CAG1VgObifWpo6B_fvspSBhHhmPNBO98C6-c6XkeZYu=eg36gTQ@mail.gmail.com>
References: <CAG1VgOab88X87hCP01PXtT3n7hJkhnY=Mf7kRa_YfNV5+V3Vaw@mail.gmail.com>
	<A69CFAEE-A138-46B6-89F5-7705FA714ECD@icir.org>
	<CAG1VgOb2=Rrw9PgSedKJqraiht0+w-jwCJdFeT=bRM7PV_a3ww@mail.gmail.com>
	<CAG1VgObifWpo6B_fvspSBhHhmPNBO98C6-c6XkeZYu=eg36gTQ@mail.gmail.com>
Message-ID: <CAG1VgOZgXA9Gi1c4mFUfw40SStmAMnfY4VrPrvPdS62D9idNgQ@mail.gmail.com>

One additional note.  Foo is showing up in my notice.log:

#separator \x09
#set_separator	,
#empty_field	(empty)
#unset_field	-
#path	notice
#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	note	msg	sub	src	dst	p	n	peer_descr	actions	policy_items	suppress_for	dropped	remote_location.country_code	remote_location.region	remote_location.city	remote_location.latitude	remote_location.longitude	metric_index.host	metric_index.str	metric_index.network
#types	time	string	addr	port	addr	port	enum	enum	string	string	addr	addr	port	count	string	table[enum]	table[count]	interval	bool	string	string	string	double	double	addr	string	subnet
1345028672.101773	-	-	-	-	-	-	Foo	Foo
detected.	-	-	-	-	-	bro	Notice::ACTION_LOG,Notice::ACTION_EMAIL	7,6	3600.000000	F	-	-	-	-	-	-	-	-

So, the notice framework seems to be doing something.

-Chris


On Mon, Aug 27, 2012 at 5:49 PM, Chris Crawford
<christopher.p.crawford at gmail.com> wrote:
> Looks like the code didn't post.
>
> For the benefit of the mailing list, this is what the script looks like:
>
> export {
>         redef Notice::mail_dest = "email_address at inet.com";
>
>         redef enum Notice::Type += {
>                 Foo,
>         };
>
>         redef Notice::emailed_types += {
>                 Foo,
>         };
> }
>
>
>
> redef Notice::policy += {
>   [$pred(n: Notice::Info) = {
>      return n$note == Foo;
>    },
>    $action = Notice::ACTION_EMAIL]
>   };
>
> event bro_init()
>         {
>         local filter: Log::Filter = [
>                 $name="poison_hits",
>                 $path="poison_hits",
>                 $pred(rec: DNS::Info) = {
>                         if ( rec?$qtype_name && rec?$answers &&
> rec$qtype_name == "A" )
>                                 {
>                                 for ( i in rec$answers )
>                                         if ( "1.2.3.4" in rec$answers[i] )
>                                                 {
>                                                 NOTICE([$note=Foo,
> $msg="Foo detected."]);
>                                                 return T;
>                                                 }
>                                 }
>                         return F;
>                 },
>                 $include=set("ts", "uid", "id.orig_h", "query")];
>         Log::add_filter(DNS::LOG, filter);
>         }
>
>
> On Mon, Aug 27, 2012 at 5:48 PM, Chris Crawford
> <christopher.p.crawford at gmail.com> wrote:
>> Thanks, Seth.  This works great, and it gives me better insight into
>> how to write my own bro scripts.
>>
>> Now, I feel like my follow up question has an obvious answer, but I'm
>> just not seeing it --
>>
>> Let's say I want to also email the alert, in addition to logging it.
>> I've added a notice statement, an attempted to redefine
>> Notice::mail_dest and Notice::policy as outlined in the bro docs:
>> http://www.bro-ids.org/documentation/notice.html
>>
>> When I run the script, though, I don't receive an email.  I know that
>> bro's email is working, because I am receiving hourly reports.  What
>> am I missing?
>>
>> Attached the script, but if it doesn't post to the list, I'll follow
>> up this post with the code.
>>
>> -Chris
>>
>> On Mon, Aug 20, 2012 at 10:42 AM, Seth Hall <seth at icir.org> wrote:
>>>
>>> On Aug 16, 2012, at 5:32 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:
>>>
>>>> redef record rec += {
>>>>        foo: Info &optional;
>>>> };
>>>
>>>> error in ./test.bro, line 22: unknown identifier (Foo::rec)
>>>> error in ./test.bro, line 35 and ./test.bro, line 39: already defined (Foo::rec)
>>>
>>> I don't think you need that little chunk of code I left above.  We do that in many base scripts as a way of hiding protocol specific information within the connection record.  There is no existing record type named "rec" though and it doesn't look like you need to hide this information anywhere since you are deriving all of your log directly from data in the DNS::log_dns event.
>>>
>>> There is a better way to do this though and it was something we specifically considered in the logging framework.  Here's a log filter you can run that will give you the log you want?
>>>
>>> event bro_init()
>>>         {
>>>         local filter: Log::Filter = [
>>>                 $name="only-1.2.3.4",
>>>                 $path="foo",
>>>                 $pred(rec: DNS::Info) = {
>>>                         if ( rec?$qtype_name && rec?$answers &&
>>>                              rec$qtype_name == "A" )
>>>                                 {
>>>                                 for ( i in rec$answers )
>>>                                         if ( "1.2.3.4" in rec$answers[i] )
>>>                                                 return T;
>>>                                 }
>>>                         return F;
>>>                 },
>>>                 $include=set("ts", "uid", "id.orig_h", "query")];
>>>         Log::add_filter(DNS::LOG, filter);
>>>         }
>>>
>>> --
>>> Seth Hall
>>> International Computer Science Institute
>>> (Bro) because everyone has a network
>>> http://www.bro-ids.org/
>>>



From christopher.p.crawford at gmail.com  Tue Aug 28 09:00:19 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Tue, 28 Aug 2012 12:00:19 -0400
Subject: [Bro] Converting a Bro Script to A New Stream
In-Reply-To: <CAG1VgOZgXA9Gi1c4mFUfw40SStmAMnfY4VrPrvPdS62D9idNgQ@mail.gmail.com>
References: <CAG1VgOab88X87hCP01PXtT3n7hJkhnY=Mf7kRa_YfNV5+V3Vaw@mail.gmail.com>
	<A69CFAEE-A138-46B6-89F5-7705FA714ECD@icir.org>
	<CAG1VgOb2=Rrw9PgSedKJqraiht0+w-jwCJdFeT=bRM7PV_a3ww@mail.gmail.com>
	<CAG1VgObifWpo6B_fvspSBhHhmPNBO98C6-c6XkeZYu=eg36gTQ@mail.gmail.com>
	<CAG1VgOZgXA9Gi1c4mFUfw40SStmAMnfY4VrPrvPdS62D9idNgQ@mail.gmail.com>
Message-ID: <CAG1VgOYOH0y3sW=x=C1CqP8ZHJOU4rRH0SGVWnVL9J7NBZbUUA@mail.gmail.com>

Foo shows up notice_policy.log too:

#separator \x09
#set_separator	,
#empty_field	(empty)
#unset_field	-
#path	notice_policy
#fields	position	priority	action	pred	halt	suppress_for
#types	count	count	enum	func	bool	interval
0	10	Notice::ACTION_ADD_GEODATA	anonymous-function\x0a{ \x0areturn
((Notice::n$note in Notice::lookup_location_types));\x0a}	F	-
1	9	Notice::ACTION_NO_SUPPRESS	anonymous-function\x0a{ \x0areturn
((Notice::n$note in Notice::not_suppressed_types));\x0a}	F	-
2	9	Notice::ACTION_NONE	anonymous-function\x0a{ \x0areturn
((Notice::n$note in Notice::ignored_types));\x0a}	T	-
3	8	Notice::ACTION_NONE	anonymous-function\x0a{ \x0aif (Notice::n$note
in Notice::type_suppression_intervals) \x0a\x09{
\x0a\x09Notice::n$suppress_for =
Notice::type_suppression_intervals[Notice::n$note];\x0a\x09return
(T);\x0a\x09}\x0a\x0areturn (F);\x0a}	F	-
4	8	Notice::ACTION_ALARM	anonymous-function\x0a{ \x0areturn
((Notice::n$note in Notice::alarmed_types));\x0a}	F	-
5	8	Notice::ACTION_EMAIL	anonymous-function\x0a{ \x0areturn
((Notice::n$note in Notice::emailed_types));\x0a}	F	-
6	5	Notice::ACTION_EMAIL	anonymous-function\x0a{ \x0areturn (n$note ==
Foo);\x0a}	F	-
7	0	Notice::ACTION_LOG	-	F	-


On Tue, Aug 28, 2012 at 11:55 AM, Chris Crawford
<christopher.p.crawford at gmail.com> wrote:
> One additional note.  Foo is showing up in my notice.log:
>
> #separator \x09
> #set_separator  ,
> #empty_field    (empty)
> #unset_field    -
> #path   notice
> #fields ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       proto   note    msg     sub     src     dst     p       n       peer_descr      actions policy_items    suppress_for    dropped remote_location.country_code    remote_location.region  remote_location.city    remote_location.latitude        remote_location.longitude       metric_index.host       metric_index.str        metric_index.network
> #types  time    string  addr    port    addr    port    enum    enum    string  string  addr    addr    port    count   string  table[enum]     table[count]    interval        bool    string  string  string  double  double  addr    string  subnet
> 1345028672.101773       -       -       -       -       -       -       Foo     Foo
> detected.       -       -       -       -       -       bro     Notice::ACTION_LOG,Notice::ACTION_EMAIL 7,6     3600.000000     F       -       -       -       -       -       -       -       -
>
> So, the notice framework seems to be doing something.
>
> -Chris
>
>
> On Mon, Aug 27, 2012 at 5:49 PM, Chris Crawford
> <christopher.p.crawford at gmail.com> wrote:
>> Looks like the code didn't post.
>>
>> For the benefit of the mailing list, this is what the script looks like:
>>
>> export {
>>         redef Notice::mail_dest = "email_address at inet.com";
>>
>>         redef enum Notice::Type += {
>>                 Foo,
>>         };
>>
>>         redef Notice::emailed_types += {
>>                 Foo,
>>         };
>> }
>>
>>
>>
>> redef Notice::policy += {
>>   [$pred(n: Notice::Info) = {
>>      return n$note == Foo;
>>    },
>>    $action = Notice::ACTION_EMAIL]
>>   };
>>
>> event bro_init()
>>         {
>>         local filter: Log::Filter = [
>>                 $name="poison_hits",
>>                 $path="poison_hits",
>>                 $pred(rec: DNS::Info) = {
>>                         if ( rec?$qtype_name && rec?$answers &&
>> rec$qtype_name == "A" )
>>                                 {
>>                                 for ( i in rec$answers )
>>                                         if ( "1.2.3.4" in rec$answers[i] )
>>                                                 {
>>                                                 NOTICE([$note=Foo,
>> $msg="Foo detected."]);
>>                                                 return T;
>>                                                 }
>>                                 }
>>                         return F;
>>                 },
>>                 $include=set("ts", "uid", "id.orig_h", "query")];
>>         Log::add_filter(DNS::LOG, filter);
>>         }
>>
>>
>> On Mon, Aug 27, 2012 at 5:48 PM, Chris Crawford
>> <christopher.p.crawford at gmail.com> wrote:
>>> Thanks, Seth.  This works great, and it gives me better insight into
>>> how to write my own bro scripts.
>>>
>>> Now, I feel like my follow up question has an obvious answer, but I'm
>>> just not seeing it --
>>>
>>> Let's say I want to also email the alert, in addition to logging it.
>>> I've added a notice statement, an attempted to redefine
>>> Notice::mail_dest and Notice::policy as outlined in the bro docs:
>>> http://www.bro-ids.org/documentation/notice.html
>>>
>>> When I run the script, though, I don't receive an email.  I know that
>>> bro's email is working, because I am receiving hourly reports.  What
>>> am I missing?
>>>
>>> Attached the script, but if it doesn't post to the list, I'll follow
>>> up this post with the code.
>>>
>>> -Chris
>>>
>>> On Mon, Aug 20, 2012 at 10:42 AM, Seth Hall <seth at icir.org> wrote:
>>>>
>>>> On Aug 16, 2012, at 5:32 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:
>>>>
>>>>> redef record rec += {
>>>>>        foo: Info &optional;
>>>>> };
>>>>
>>>>> error in ./test.bro, line 22: unknown identifier (Foo::rec)
>>>>> error in ./test.bro, line 35 and ./test.bro, line 39: already defined (Foo::rec)
>>>>
>>>> I don't think you need that little chunk of code I left above.  We do that in many base scripts as a way of hiding protocol specific information within the connection record.  There is no existing record type named "rec" though and it doesn't look like you need to hide this information anywhere since you are deriving all of your log directly from data in the DNS::log_dns event.
>>>>
>>>> There is a better way to do this though and it was something we specifically considered in the logging framework.  Here's a log filter you can run that will give you the log you want?
>>>>
>>>> event bro_init()
>>>>         {
>>>>         local filter: Log::Filter = [
>>>>                 $name="only-1.2.3.4",
>>>>                 $path="foo",
>>>>                 $pred(rec: DNS::Info) = {
>>>>                         if ( rec?$qtype_name && rec?$answers &&
>>>>                              rec$qtype_name == "A" )
>>>>                                 {
>>>>                                 for ( i in rec$answers )
>>>>                                         if ( "1.2.3.4" in rec$answers[i] )
>>>>                                                 return T;
>>>>                                 }
>>>>                         return F;
>>>>                 },
>>>>                 $include=set("ts", "uid", "id.orig_h", "query")];
>>>>         Log::add_filter(DNS::LOG, filter);
>>>>         }
>>>>
>>>> --
>>>> Seth Hall
>>>> International Computer Science Institute
>>>> (Bro) because everyone has a network
>>>> http://www.bro-ids.org/
>>>>



From christopher.p.crawford at gmail.com  Tue Aug 28 13:22:49 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Tue, 28 Aug 2012 16:22:49 -0400
Subject: [Bro] Debugging Bro Scripts Where action = Notice::ACTION_EMAIL
Message-ID: <CAG1VgObsmSqRu1cw0rqRgNTPAwCmjcHAXKZjei+0XyLwHg2UFw@mail.gmail.com>

I spent quite a bit of time and effort trying to figure out.  Dropping
a note out to the community to hopefully help the next guy.

Over in this thread
http://mailman.icsi.berkeley.edu/pipermail/bro/2012-August/005811.html

I couldn't figure out why this script
http://mailman.icsi.berkeley.edu/pipermail/bro/2012-August/005812.html

would not send an email alert via the Notice framework.

I was testing the script on a small pcap file. I thought that
debugging approach would enable me to quickly, easily, and reliably
check to see if my new bro script was working as intended.

Here's the problem with that development/debugging approach.  The
first few lines in the function email_notice_to (found in
frameworks/notice/main.bro specifically) check to see if you are
reading traffic from a trace file, and then silently disable email
alerting if you are.  This turned out to be very frustrating to debug.

To confirm that my script was working as expected, I had to change the
following lines in frameworks/notice/main.bro:

function email_notice_to(n: Notice::Info, dest: string, extend: bool)
        {
       if ( reading_traces() || dest == "" )
               return;

to the following:

function email_notice_to(n: Notice::Info, dest: string, extend: bool)
        {
#       if ( reading_traces() || dest == "" )
#               return;

If you plan to test a new script where you expect it to send an email
via the Notice framework, I recommend that you send traffic that ought
to should trigger an email alert over the wire.

That's not a viable option for me, so commenting out the lines above
is a better approach.

Would also recommend that either the bro documentation make note of
this "feature" or that the resulting notice.log print a message to
indicate that email alerting was disabled because it isn't reading
traffic from a live network capture.

-Chris


From christopher.p.crawford at gmail.com  Tue Aug 28 15:12:20 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Tue, 28 Aug 2012 18:12:20 -0400
Subject: [Bro] Debugging Bro Scripts Where action = Notice::ACTION_EMAIL
In-Reply-To: <CAG1VgObsmSqRu1cw0rqRgNTPAwCmjcHAXKZjei+0XyLwHg2UFw@mail.gmail.com>
References: <CAG1VgObsmSqRu1cw0rqRgNTPAwCmjcHAXKZjei+0XyLwHg2UFw@mail.gmail.com>
Message-ID: <CAG1VgOa09dJ3GJTKhA0FpdRpDLXJP=jobNFHGK2egV0twjpVMg@mail.gmail.com>

One more note:

A line like

redef Notice::mail_dest = "alert at email.com";

in a custom Bro script doesn't appear to override the value specified
by the MailTo variable set in etc/broctl.cfg .

It appears that either your email alerts from the notice framework are
sent to the value specified by MailTo, or they don't get sent out at
all.

-Chris

On Tue, Aug 28, 2012 at 4:22 PM, Chris Crawford
<christopher.p.crawford at gmail.com> wrote:
> I spent quite a bit of time and effort trying to figure out.  Dropping
> a note out to the community to hopefully help the next guy.
>
> Over in this thread
> http://mailman.icsi.berkeley.edu/pipermail/bro/2012-August/005811.html
>
> I couldn't figure out why this script
> http://mailman.icsi.berkeley.edu/pipermail/bro/2012-August/005812.html
>
> would not send an email alert via the Notice framework.
>
> I was testing the script on a small pcap file. I thought that
> debugging approach would enable me to quickly, easily, and reliably
> check to see if my new bro script was working as intended.
>
> Here's the problem with that development/debugging approach.  The
> first few lines in the function email_notice_to (found in
> frameworks/notice/main.bro specifically) check to see if you are
> reading traffic from a trace file, and then silently disable email
> alerting if you are.  This turned out to be very frustrating to debug.
>
> To confirm that my script was working as expected, I had to change the
> following lines in frameworks/notice/main.bro:
>
> function email_notice_to(n: Notice::Info, dest: string, extend: bool)
>         {
>        if ( reading_traces() || dest == "" )
>                return;
>
> to the following:
>
> function email_notice_to(n: Notice::Info, dest: string, extend: bool)
>         {
> #       if ( reading_traces() || dest == "" )
> #               return;
>
> If you plan to test a new script where you expect it to send an email
> via the Notice framework, I recommend that you send traffic that ought
> to should trigger an email alert over the wire.
>
> That's not a viable option for me, so commenting out the lines above
> is a better approach.
>
> Would also recommend that either the bro documentation make note of
> this "feature" or that the resulting notice.log print a message to
> indicate that email alerting was disabled because it isn't reading
> traffic from a live network capture.
>
> -Chris


From Corey.Roach at utah.edu  Wed Aug 29 09:57:43 2012
From: Corey.Roach at utah.edu (Corey Roach  (ISO))
Date: Wed, 29 Aug 2012 16:57:43 +0000
Subject: [Bro] BPF packet filter syntax
In-Reply-To: <AE802E5B2E279644BDF56F44B9EE6FCC2D48EF00@X-MB6.xds.umail.utah.edu>
References: <AE802E5B2E279644BDF56F44B9EE6FCC2D48D95D@X-MB6.xds.umail.utah.edu>
	<36B60826-EE96-4A23-A222-BBBDE3ACF714@icir.org>
	<AE802E5B2E279644BDF56F44B9EE6FCC2D48E7D3@X-MB6.xds.umail.utah.edu>
	<4459C289-FCA3-4452-B377-B77935BC2E6B@icir.org>
	<AE802E5B2E279644BDF56F44B9EE6FCC2D48EF00@X-MB6.xds.umail.utah.edu>
Message-ID: <AE802E5B2E279644BDF56F44B9EE6FCC2D4923AF@X-MB6.xds.umail.utah.edu>

Hey Gang,

I still don't have this working properly, but I think I'm making progress and I've got it down to a repeatable test.

For testing I installed the latest pfring SVN and Bro v2.1-rc3 on an Ubuntu Server 12.04.1 VMware Fusion VM.

The only change I made to the plain-vanilla install is to add the following lines to the bottom of the local.bro:

redef PacketFilter::all_packets = F;
redef capture_filters = { ["all"] = "ip or not ip" };
redef restrict_filters += { ["not-one-host"] = "not host 10.10.10.1" };
redef restrict_filters += { ["not-one-net"] = "not net 10.10.20.0/24" };

I start it up, and the filter shows up properly in the packet_filter.log

I then change the node.cfg from stand-alone mode to a single box cluster (manager, proxy and worker all on the same box) and start it up again and nothing shows up in the packet_filter.log.

So, it appears to possibly be a stand-alone vs cluster issue.

Has any successfully applied a packet filter to a clustered environment? Did you have to make any other tweaks to get it to work?

- Corey




From seth at icir.org  Wed Aug 29 10:06:48 2012
From: seth at icir.org (Seth Hall)
Date: Wed, 29 Aug 2012 13:06:48 -0400
Subject: [Bro] Debugging Bro Scripts Where action = Notice::ACTION_EMAIL
In-Reply-To: <CAG1VgOa09dJ3GJTKhA0FpdRpDLXJP=jobNFHGK2egV0twjpVMg@mail.gmail.com>
References: <CAG1VgObsmSqRu1cw0rqRgNTPAwCmjcHAXKZjei+0XyLwHg2UFw@mail.gmail.com>
	<CAG1VgOa09dJ3GJTKhA0FpdRpDLXJP=jobNFHGK2egV0twjpVMg@mail.gmail.com>
Message-ID: <F51C520C-6C8E-4987-B56B-CCBCAD917E41@icir.org>


On Aug 28, 2012, at 6:12 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:

> redef Notice::mail_dest = "alert at email.com";
> 
> in a custom Bro script doesn't appear to override the value specified
> by the MailTo variable set in etc/broctl.cfg .

Yes, this is an unfortunate side effect to automatically changing settings through BroControl.  I believe that the documentation has been updated for 2.1 to clarify which variables are affected.  I still don't think we are completely sure the direction this will go long term, but it is definitely unclear at the moment.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From tyler.schoenke at colorado.edu  Wed Aug 29 10:08:19 2012
From: tyler.schoenke at colorado.edu (Tyler T. Schoenke)
Date: Wed, 29 Aug 2012 11:08:19 -0600
Subject: [Bro] BPF packet filter syntax
In-Reply-To: <AE802E5B2E279644BDF56F44B9EE6FCC2D4923AF@X-MB6.xds.umail.utah.edu>
References: <AE802E5B2E279644BDF56F44B9EE6FCC2D48D95D@X-MB6.xds.umail.utah.edu>
	<36B60826-EE96-4A23-A222-BBBDE3ACF714@icir.org>
	<AE802E5B2E279644BDF56F44B9EE6FCC2D48E7D3@X-MB6.xds.umail.utah.edu>
	<4459C289-FCA3-4452-B377-B77935BC2E6B@icir.org>
	<AE802E5B2E279644BDF56F44B9EE6FCC2D48EF00@X-MB6.xds.umail.utah.edu>
	<AE802E5B2E279644BDF56F44B9EE6FCC2D4923AF@X-MB6.xds.umail.utah.edu>
Message-ID: <503E4C83.2020405@colorado.edu>

You can run broctl print capture_filters or broctl print
restrict_filters to see which filters are being loaded by the cluster.
 I never thought to check my packet_filter* log files, but looked and
they are empty even though the filters are running.

Tyler

--
Tyler Schoenke
Network Security Manager
IT Security Office
University of Colorado at Boulder

On 8/29/12 10:57 AM, Corey Roach (ISO) wrote:
> Hey Gang,
> 
> I still don't have this working properly, but I think I'm making progress and I've got it down to a repeatable test.
> 
> For testing I installed the latest pfring SVN and Bro v2.1-rc3 on an Ubuntu Server 12.04.1 VMware Fusion VM.
> 
> The only change I made to the plain-vanilla install is to add the following lines to the bottom of the local.bro:
> 
> redef PacketFilter::all_packets = F;
> redef capture_filters = { ["all"] = "ip or not ip" };
> redef restrict_filters += { ["not-one-host"] = "not host 10.10.10.1" };
> redef restrict_filters += { ["not-one-net"] = "not net 10.10.20.0/24" };
> 
> I start it up, and the filter shows up properly in the packet_filter.log
> 
> I then change the node.cfg from stand-alone mode to a single box cluster (manager, proxy and worker all on the same box) and start it up again and nothing shows up in the packet_filter.log.
> 
> So, it appears to possibly be a stand-alone vs cluster issue.
> 
> Has any successfully applied a packet filter to a clustered environment? Did you have to make any other tweaks to get it to work?
> 
> - Corey
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 


From seth at icir.org  Wed Aug 29 10:10:19 2012
From: seth at icir.org (Seth Hall)
Date: Wed, 29 Aug 2012 13:10:19 -0400
Subject: [Bro] Debugging Bro Scripts Where action = Notice::ACTION_EMAIL
In-Reply-To: <CAG1VgObsmSqRu1cw0rqRgNTPAwCmjcHAXKZjei+0XyLwHg2UFw@mail.gmail.com>
References: <CAG1VgObsmSqRu1cw0rqRgNTPAwCmjcHAXKZjei+0XyLwHg2UFw@mail.gmail.com>
Message-ID: <798161CB-55E6-42EB-9CC0-3CD5DD20D3BB@icir.org>


On Aug 28, 2012, at 4:22 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:

> If you plan to test a new script where you expect it to send an email
> via the Notice framework, I recommend that you send traffic that ought
> to should trigger an email alert over the wire.

Why are you looking to send an email while reading a tracefile?  The same notice will be in the notice.log.  

I do agree that we should output a reporter message if someone tries to send an email while reading a tracefile though, we just can't sneak that feature into 2.1 but I'll file a ticket for it.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From Liam.Randall at gigaco.com  Wed Aug 29 10:58:49 2012
From: Liam.Randall at gigaco.com (Liam Randall)
Date: Wed, 29 Aug 2012 13:58:49 -0400
Subject: [Bro] Debugging Bro Scripts Where action = Notice::ACTION_EMAIL
In-Reply-To: <798161CB-55E6-42EB-9CC0-3CD5DD20D3BB@icir.org>
References: <CAG1VgObsmSqRu1cw0rqRgNTPAwCmjcHAXKZjei+0XyLwHg2UFw@mail.gmail.com>
	<798161CB-55E6-42EB-9CC0-3CD5DD20D3BB@icir.org>
Message-ID: <2697B86A359F6C43BF6FD44F8403D71741E3CB@giga-dc001.GigaCo.local>

Seth, 

Might be a useful feature if someone were to integrate bro as a Cuckoo
box plugin or in a sandnet of some sort.  Just a thought.

Liam

-----Original Message-----
From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf
Of Seth Hall
Sent: Wednesday, August 29, 2012 1:10 PM
To: Chris Crawford
Cc: bro at bro-ids.org
Subject: Re: [Bro] Debugging Bro Scripts Where action =
Notice::ACTION_EMAIL


On Aug 28, 2012, at 4:22 PM, Chris Crawford
<christopher.p.crawford at gmail.com> wrote:

> If you plan to test a new script where you expect it to send an email 
> via the Notice framework, I recommend that you send traffic that ought

> to should trigger an email alert over the wire.

Why are you looking to send an email while reading a tracefile?  The
same notice will be in the notice.log.  

I do agree that we should output a reporter message if someone tries to
send an email while reading a tracefile though, we just can't sneak that
feature into 2.1 but I'll file a ticket for it.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/


_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at icir.org  Wed Aug 29 11:09:29 2012
From: seth at icir.org (Seth Hall)
Date: Wed, 29 Aug 2012 14:09:29 -0400
Subject: [Bro] Debugging Bro Scripts Where action = Notice::ACTION_EMAIL
In-Reply-To: <2697B86A359F6C43BF6FD44F8403D71741E3CB@giga-dc001.GigaCo.local>
References: <CAG1VgObsmSqRu1cw0rqRgNTPAwCmjcHAXKZjei+0XyLwHg2UFw@mail.gmail.com>
	<798161CB-55E6-42EB-9CC0-3CD5DD20D3BB@icir.org>
	<2697B86A359F6C43BF6FD44F8403D71741E3CB@giga-dc001.GigaCo.local>
Message-ID: <007CCC72-C3CB-4E4B-AE0B-38148A86ADF3@icir.org>


On Aug 29, 2012, at 1:58 PM, Liam Randall <Liam.Randall at gigaco.com> wrote:

> Might be a useful feature if someone were to integrate bro as a Cuckoo
> box plugin or in a sandnet of some sort.  Just a thought.


Good point.  I'll think about this some more.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/



From Corey.Roach at utah.edu  Wed Aug 29 12:19:36 2012
From: Corey.Roach at utah.edu (Corey Roach  (ISO))
Date: Wed, 29 Aug 2012 19:19:36 +0000
Subject: [Bro] BPF packet filter syntax
In-Reply-To: <503E4C83.2020405@colorado.edu>
References: <AE802E5B2E279644BDF56F44B9EE6FCC2D48D95D@X-MB6.xds.umail.utah.edu>
	<36B60826-EE96-4A23-A222-BBBDE3ACF714@icir.org>
	<AE802E5B2E279644BDF56F44B9EE6FCC2D48E7D3@X-MB6.xds.umail.utah.edu>
	<4459C289-FCA3-4452-B377-B77935BC2E6B@icir.org>
	<AE802E5B2E279644BDF56F44B9EE6FCC2D48EF00@X-MB6.xds.umail.utah.edu>
	<AE802E5B2E279644BDF56F44B9EE6FCC2D4923AF@X-MB6.xds.umail.utah.edu>
	<503E4C83.2020405@colorado.edu>
Message-ID: <AE802E5B2E279644BDF56F44B9EE6FCC2D493C3E@X-MB6.xds.umail.utah.edu>

Nice! Thanks Tyler. Looks like they are there. I wonder why they don't get logged.

If I'm reading the packet-filter framework right, they see to get logged to indicate they were applied correctly. I'll have to do some functionality testing to make sure they are getting applied as well as being held in the variables, but if they are working without logging that is good enough for me at the moment.

- Corey



On Aug 29, 2012, at 11:08 AM, Tyler T. Schoenke <tyler.schoenke at colorado.edu> wrote:

> You can run broctl print capture_filters or broctl print
> restrict_filters to see which filters are being loaded by the cluster.
> I never thought to check my packet_filter* log files, but looked and
> they are empty even though the filters are running.
> 
> Tyler
> 
> --
> Tyler Schoenke
> Network Security Manager
> IT Security Office
> University of Colorado at Boulder




From christopher.p.crawford at gmail.com  Wed Aug 29 13:11:11 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Wed, 29 Aug 2012 16:11:11 -0400
Subject: [Bro] Debugging Bro Scripts Where action = Notice::ACTION_EMAIL
In-Reply-To: <F51C520C-6C8E-4987-B56B-CCBCAD917E41@icir.org>
References: <CAG1VgObsmSqRu1cw0rqRgNTPAwCmjcHAXKZjei+0XyLwHg2UFw@mail.gmail.com>
	<CAG1VgOa09dJ3GJTKhA0FpdRpDLXJP=jobNFHGK2egV0twjpVMg@mail.gmail.com>
	<F51C520C-6C8E-4987-B56B-CCBCAD917E41@icir.org>
Message-ID: <CAG1VgObeUZiO3L99s03QJ1hfbSp=+OpNcwDgkbW+v2ZLPf_aAw@mail.gmail.com>

Not sure how much of a pain this idea would be to implement, but it
would be great if all email went to the address specified by MailTo,
unless you specifically override that value in a custom bro script.
That way a bro script could have something similar to a local MailTo
variable and all notices sent out from that custom script would be
sent to the new email address.

-Chris

On Wed, Aug 29, 2012 at 1:06 PM, Seth Hall <seth at icir.org> wrote:
>
> On Aug 28, 2012, at 6:12 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:
>
>> redef Notice::mail_dest = "alert at email.com";
>>
>> in a custom Bro script doesn't appear to override the value specified
>> by the MailTo variable set in etc/broctl.cfg .
>
> Yes, this is an unfortunate side effect to automatically changing settings through BroControl.  I believe that the documentation has been updated for 2.1 to clarify which variables are affected.  I still don't think we are completely sure the direction this will go long term, but it is definitely unclear at the moment.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>



From christopher.p.crawford at gmail.com  Wed Aug 29 13:14:56 2012
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Wed, 29 Aug 2012 16:14:56 -0400
Subject: [Bro] Debugging Bro Scripts Where action = Notice::ACTION_EMAIL
In-Reply-To: <798161CB-55E6-42EB-9CC0-3CD5DD20D3BB@icir.org>
References: <CAG1VgObsmSqRu1cw0rqRgNTPAwCmjcHAXKZjei+0XyLwHg2UFw@mail.gmail.com>
	<798161CB-55E6-42EB-9CC0-3CD5DD20D3BB@icir.org>
Message-ID: <CAG1VgOY6167Vuq7ieEzd2pagtf8p9iQYg-b=a=HAf+RPfqvK_A@mail.gmail.com>

I wasn't confident that an email would fire from my bro script, so I
wanted to test it with a tracefile first.  I knew that the tracefile
contained the network conditions I was interested in, so this seemed
like a reasonable way to verify my script.  The fact that I didn't see
any email while I was testing the script led me to believe that either
I was doing something wrong, or that part of the notice framework was
broken.  Either way, I wouldn't have had much confidence that my
script was doing what I wanted it to do, so what would be the point in
using it in a live capture scenario?

-Chris

On Wed, Aug 29, 2012 at 1:10 PM, Seth Hall <seth at icir.org> wrote:
>
> On Aug 28, 2012, at 4:22 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:
>
>> If you plan to test a new script where you expect it to send an email
>> via the Notice framework, I recommend that you send traffic that ought
>> to should trigger an email alert over the wire.
>
> Why are you looking to send an email while reading a tracefile?  The same notice will be in the notice.log.
>
> I do agree that we should output a reporter message if someone tries to send an email while reading a tracefile though, we just can't sneak that feature into 2.1 but I'll file a ticket for it.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>


From robin at icir.org  Wed Aug 29 18:17:52 2012
From: robin at icir.org (Robin Sommer)
Date: Wed, 29 Aug 2012 18:17:52 -0700
Subject: [Bro] Bro 2.1 Release
Message-ID: <20120830011752.GD31398@icir.org>

We are very excited to release Bro 2.1. See the blog for more
information at

    http://blog.bro-ids.org/2012/08/bro-21-release.html

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From tritium.cat at gmail.com  Thu Aug 30 14:46:07 2012
From: tritium.cat at gmail.com (Tritium Cat)
Date: Thu, 30 Aug 2012 21:46:07 +0000
Subject: [Bro] Troubleshooting crashes
Message-ID: <CAMPgRd4KK0DF4+YHRNiEGOxi=jhV8ddbvQb5MgCjo6Oyums=Fg@mail.gmail.com>

Hello all,

What's the best way to disable Bro in a systematic way to isolate crashes ?

I disabled all the protocols except SSH and a few default scripts that
utilize it.  After ~12 hours I returned to find many of the worker nodes
had crashed.  I forgot to look at the diag for the crashed workers before
stopping the cluster.

Suggestions greatly appreciated.

Thanks,

-TC


local.bro configuration
=======================================
bro at bc : [9:20pm] : bro : grep -v "^#" site/local.bro |grep "[a-z]"
@load misc/loaded-scripts
@load tuning/defaults
@load protocols/ssh/software
@load protocols/ssh/geo-data
@load protocols/ssh/detect-bruteforcing
@load protocols/ssh/interesting-hostnames


base/init-default.bro configuration
=======================================
bro at bc : [9:20pm] : bro : grep -v "^#" base/init-default.bro  | grep "[a-z]"
@load base/utils/site
@load base/utils/addrs
@load base/utils/conn-ids
@load base/utils/directions-and-hosts
@load base/utils/files
@load base/utils/numbers
@load base/utils/paths
@load base/utils/patterns
@load base/utils/strings
@load base/utils/thresholds
@load base/frameworks/notice
@load base/frameworks/dpd
@load base/frameworks/signatures
@load base/frameworks/packet-filter
@load base/frameworks/software
@load base/frameworks/communication
@load base/frameworks/control
@load base/frameworks/cluster
@load base/frameworks/metrics
@load base/frameworks/intel
@load base/frameworks/reporter
@load base/frameworks/tunnels
@load base/protocols/ssh




Here's the PF_RING info:
=======================================
PF_RING Version     : 5.4.6 ($Revision: 5658$)
Ring slots          : 4096
Slot version        : 14
Capture TX          : No [RX only]
IP Defragment       : No
Socket Mode         : Standard
Transparent mode    : No (mode 2)
Total rings         : 10
Total plugins       : 0



Here's a top from one of the physical servers running workers; all five
servers have a similar profile.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

15423 bro       20   0 1059m 957m  69m R   95  1.5   2:21.79 bro

15429 bro       20   0 1041m 939m  69m R   80  1.5   1:49.08 bro

15426 bro       20   0 1041m 939m  69m S   72  1.5   1:56.46 bro

15422 bro       20   0 1041m 939m  69m R   70  1.5   1:58.51 bro

15424 bro       20   0 1041m 939m  69m S   70  1.5   1:51.85 bro

15427 bro       20   0 1041m 939m  69m R   64  1.5   1:59.86 bro

15420 bro       20   0 1041m 939m  69m S   62  1.5   1:52.02 bro

15421 bro       20   0 1041m 939m  69m R   60  1.5   1:43.13 bro

15425 bro       20   0 1041m 939m  69m R   58  1.5   1:39.75 bro

15428 bro       20   0 1041m 939m  69m S   56  1.5   1:39.05 bro

15430 bro       25   5  186m  76m  64m S   19  0.1   0:28.42 bro

15437 bro       25   5  186m  76m  64m S   18  0.1   0:28.08 bro

15431 bro       25   5  186m  76m  64m S   14  0.1   0:27.35 bro

15432 bro       25   5  186m  76m  64m S   14  0.1   0:26.63 bro

15433 bro       25   5  186m  76m  64m S   14  0.1   0:26.72 bro

15436 bro       25   5  186m  76m  64m S   14  0.1   0:23.36 bro

15434 bro       25   5  186m  76m  64m S   12  0.1   0:25.94 bro

15438 bro       25   5  186m  76m  64m S   12  0.1   0:22.98 bro

15435 bro       25   5  186m  76m  64m S   10  0.1   0:20.59 bro

15439 bro       25   5  186m  76m  64m S   10  0.1   0:20.54 bro


Here's a snapshot from the server running the manager, no problem here...

  PID USERNAME   THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
62903 bro          1  28    5   192M 20616K select  4   0:23  0.29% bro
62937 bro          1  28    5   188M 19916K select  0   0:24  0.20% bro


Results in logs so far:
=======================================
-rw-r--r--  1 bro  bro   1409892 Aug 30 21:27 communication.log
-rw-r--r--  1 bro  bro   5611960 Aug 30 21:27 dpd.log
-rw-r--r--  1 bro  bro      6844 Aug 30 21:12 loaded_scripts.log
-rw-r--r--  1 bro  bro    214670 Aug 30 21:27 notice.log
-rw-r--r--  1 bro  bro      1101 Aug 30 21:12 notice_policy.log
-rw-r--r--  1 bro  bro       187 Aug 30 21:12 packet_filter.log
-rw-r--r--  1 bro  bro     10323 Aug 30 21:15 reporter.log
-rw-r--r--  1 bro  bro    115243 Aug 30 21:27 software.log
-rw-r--r--  1 bro  bro   3640436 Aug 30 21:27 ssh.log
-rw-r--r--  1 bro  bro         0 Aug 30 21:12 stderr.log
-rw-r--r--  1 bro  bro        29 Aug 30 21:12 stdout.log
-rw-r--r--  1 bro  bro   5575087 Aug 30 21:27 tunnel.log
-rw-r--r--  1 bro  bro  52846117 Aug 30 21:27 weird.log

+ communication.log
   -- nothing but info on workers

+ dpd.log
   -- UDP 53 Teredo payload length messages

+ notice.log
   -- SSH messages and errors on workers.

1346362754.085749       -       -       -       -       -       -
PacketFilter::Dropped_Packets   876266 packets dropped after filtering,
876266 received -       -       -       -       -worker-3-1
Notice::ACTION_LOG      6       3600.000000     F       -       -       -
    -       -       -       -       -
1346362752.730723       -       -       -       -       -       -
PacketFilter::Dropped_Packets   387600 packets dropped after filtering,
1248522 received, 860922 on link        -       --
        -       -       worker-1-3      Notice::ACTION_LOG      6
3600.000000     F       -       -       -       -       -       -       -
    -
1346362755.198122       -       -       -       -       -       -
PacketFilter::Dropped_Packets   958912 packets dropped after filtering,
958912 received -       -       -       -       -worker-2-6
Notice::ACTION_LOG      6       3600.000000     F       -       -       -
    -       -       -       -       -

worker 3-1 is PID 15278

>>> 10.1.1.1
   (+) bro      15278 15189 59.4  1.6 1143708 1105120 ?     R 14:12:52
00:16:29 bro
   (+) bro      15279 15188 40.5  1.6 1106704 1067676 ?     S 14:12:52
00:11:15 bro



Look at PF_RING for this PID:
====================================
user at bro_server:~$ cat /proc/net/pf_ring/15278-eth5.24
Bound Device(s)    : eth5
Active             : 1
Breed              : Non-DNA
Sampling Rate      : 1
Capture Direction  : RX+TX
Socket Mode        : RX+TX
Appl. Name         : <unknown>
IP Defragment      : No
BPF Filtering      : Enabled
# Sw Filt. Rules   : 0
# Hw Filt. Rules   : 0
Poll Pkt Watermark : 1
Num Poll Calls     : 13786320
Channel Id         : -1
Cluster Id         : 0
Slot Version       : 14 [5.4.6]
Min Num Slots      : 8159
Bucket Len         : 8192
Slot Len           : 8224 [bucket+header]
Tot Memory         : 67108864
Tot Packets        : 169050905
Tot Pkt Lost       : 0
Tot Insert         : 169050905
Tot Read           : 169050895
Insert Offset      : 24725033
Remove Offset      : 24713556
TX: Send Ok        : 0
TX: Send Errors    : 0
Reflect: Fwd Ok    : 0
Reflect: Fwd Errors: 0
Num Free Slots     : 8150
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20120830/c599a813/attachment.html 

From seth at icir.org  Thu Aug 30 18:18:56 2012
From: seth at icir.org (Seth Hall)
Date: Thu, 30 Aug 2012 21:18:56 -0400
Subject: [Bro] Troubleshooting crashes
In-Reply-To: <CAMPgRd4KK0DF4+YHRNiEGOxi=jhV8ddbvQb5MgCjo6Oyums=Fg@mail.gmail.com>
References: <CAMPgRd4KK0DF4+YHRNiEGOxi=jhV8ddbvQb5MgCjo6Oyums=Fg@mail.gmail.com>
Message-ID: <C108E70D-BF4F-4275-B298-17060206015F@icir.org>


On Aug 30, 2012, at 5:46 PM, Tritium Cat <tritium.cat at gmail.com> wrote:

> What's the best way to disable Bro in a systematic way to isolate crashes ?

Sending us the diag output from broctl is best since it will include a back trace.

> I disabled all the protocols except SSH and a few default scripts that utilize it.

How were you disabling protocols? (nevermind, i see the answer later and i'll comment there)

>  After ~12 hours I returned to find many of the worker nodes had crashed.  I forgot to look at the diag for the crashed workers before stopping the cluster.

Do you have the cron command setup correctly?  The workers should have been restart automatically after they crashed and a diagnostic email sent to you.
	Mentioned in this section: http://bro-ids.org/documentation/quickstart.html#a-minimal-starting-configuration

> base/init-default.bro configuration 
> =======================================
> bro at bc : [9:20pm] : bro : grep -v "^#" base/init-default.bro  | grep "[a-z]"

I see that you were modifying scripts in the base directory to disable analyzers and I just wanted to point out that we don't support directly making changes to scripts there at all and it's possible that you could get into trouble if you try to update or reinstall with changes in that area.

That said, nice job figuring that out since we don't provide a lot of support in that area right now. :)  I'm hoping to introduce an analyzer framework which will provide an API for doing that with the 2.2 release.

> Total rings         : 10

How many CPU cores do you have?

> -rw-r--r--  1 bro  bro     10323 Aug 30 21:15 reporter.log
> -rw-r--r--  1 bro  bro  52846117 Aug 30 21:27 weird.log

I'm curious about what's in reporter.log, normally that shouldn't have too much in it.  Also, that's an astonishingly large weird.log.  Is there anything that stands out in those two?

Could you show me your node.cfg configuration too?

Oh, and one last thing, have you made sure to disable all of special NIC features?
	http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Fri Aug 31 06:29:48 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 31 Aug 2012 09:29:48 -0400
Subject: [Bro] Debugging Bro Scripts Where action = Notice::ACTION_EMAIL
In-Reply-To: <CAG1VgOY6167Vuq7ieEzd2pagtf8p9iQYg-b=a=HAf+RPfqvK_A@mail.gmail.com>
References: <CAG1VgObsmSqRu1cw0rqRgNTPAwCmjcHAXKZjei+0XyLwHg2UFw@mail.gmail.com>
	<798161CB-55E6-42EB-9CC0-3CD5DD20D3BB@icir.org>
	<CAG1VgOY6167Vuq7ieEzd2pagtf8p9iQYg-b=a=HAf+RPfqvK_A@mail.gmail.com>
Message-ID: <38F9F536-B8B4-4ABC-B438-B471A14DEA24@icir.org>


On Aug 29, 2012, at 4:14 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:

> Either way, I wouldn't have had much confidence that my
> script was doing what I wanted it to do, so what would be the point in
> using it in a live capture scenario?


You will still see the Notice::ACTION_EMAIL action attached to your notice in notice.log.  

I do agree though that we need to revisit the decision of how email is handled.  You may be right that the correct decision is to get rid of those two lines like you did.  We just need to think about it a bit more.

Thanks!
  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Fri Aug 31 06:33:39 2012
From: seth at icir.org (Seth Hall)
Date: Fri, 31 Aug 2012 09:33:39 -0400
Subject: [Bro] BPF packet filter syntax
In-Reply-To: <AE802E5B2E279644BDF56F44B9EE6FCC2D4923AF@X-MB6.xds.umail.utah.edu>
References: <AE802E5B2E279644BDF56F44B9EE6FCC2D48D95D@X-MB6.xds.umail.utah.edu>
	<36B60826-EE96-4A23-A222-BBBDE3ACF714@icir.org>
	<AE802E5B2E279644BDF56F44B9EE6FCC2D48E7D3@X-MB6.xds.umail.utah.edu>
	<4459C289-FCA3-4452-B377-B77935BC2E6B@icir.org>
	<AE802E5B2E279644BDF56F44B9EE6FCC2D48EF00@X-MB6.xds.umail.utah.edu>
	<AE802E5B2E279644BDF56F44B9EE6FCC2D4923AF@X-MB6.xds.umail.utah.edu>
Message-ID: <4BACEB77-2134-4060-BE9B-D4F1BBB1D8F3@icir.org>


On Aug 29, 2012, at 12:57 PM, Corey Roach (ISO) <Corey.Roach at utah.edu> wrote:

> So, it appears to possibly be a stand-alone vs cluster issue.
> 
> Has any successfully applied a packet filter to a clustered environment? Did you have to make any other tweaks to get it to work?


After thinking about this for a couple of days, I'm starting to wonder if there is some problem with when this log write is happening.  It's possible that the remote communication has not yet begun by the time the filter is being applied so the remote logging isn't happening right.

I'll try and carve out some time soon to look into it more closely though since it sounds like you were saying that the filter wasn't being applied at all since you were seeing traffic in your conn.log that you wouldn't expect to see.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/