From jsiwek at illinois.edu  Tue Oct  1 07:37:04 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Tue, 1 Oct 2013 14:37:04 +0000
Subject: [Bro] Trouble building Bro 2.2 on ARMv6 (Raspberry Pi / Wheezy)
In-Reply-To: <CAEZw2bwkxSP8AM7_uPRKEqnm0e2omreFqkdcP=C+S-XRybMx4g@mail.gmail.com>
References: <20130930121747.GE57016@icir.org>
	<6d228d26-d14f-4973-97d9-2d33bdc3189f@me.com>
	<CAEZw2bwkxSP8AM7_uPRKEqnm0e2omreFqkdcP=C+S-XRybMx4g@mail.gmail.com>
Message-ID: <34F95A48-D8BA-4814-AFD4-2D57603CF583@illinois.edu>


On Sep 30, 2013, at 10:15 PM, anthony kasza <anthony.kasza at gmail.com> wrote:

> Fresh install here, too. Command output below.
> 
> root at raspberrypi:/opt/bro_src# ./build/src/bifcl -s src/probabilistic/top-k.bif
> usage: bifcl [-p] *.bif
> root at raspberrypi:/opt/bro_src# find ./ -iname top-k*
> ./src/probabilistic/top-k.bif
> 
> I checked the dependency versions from the RPi's repos and they are equal to or newer than a vanilla Debian 6 system (which Bro builds fine on).

Can you try the patch below?

- Jon

diff --git a/src/builtin-func.l b/src/builtin-func.l
index c70b99b..b032fd1 100644
--- a/src/builtin-func.l
+++ b/src/builtin-func.l
@@ -275,7 +275,7 @@ void finish_alternative_mode()
 
 int main(int argc, char* argv[])
        {
-       char opt;
+       int opt;
 
        while ( (opt = getopt(argc, argv, "p:s")) != -1 )
                {


From n.siow at wustl.edu  Tue Oct  1 07:42:39 2013
From: n.siow at wustl.edu (Nicholas Siow)
Date: Tue, 1 Oct 2013 09:42:39 -0500
Subject: [Bro] More connection info in Software::Info?
Message-ID: <CALcsdGttYGeDyS_XW16CBb1c6GYc0qRy+wVF0aesfNgnDzMvcA@mail.gmail.com>

Hey, had a quick question about the connection information in the
Software::Info structure.

>From what I can see in the source code / manual there seems to be only one
side of the connection represented ( only "host" and "host_p" ).

For what we are trying to do, we want the full connection to be logged.
Just how HTTP for example records the originating / responding host / port.
Since the Software::found function seems to take a connection as a
parameter, would it be possible to pull c$id$orig_h, c$id$orig_p,
c$id$resp_h, and c$id$resp_p fields out and log all of them? Or is there
some limitation that prevents those fields from being accessed / logged?

Thanks,
N. Siow
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131001/1936e2df/attachment.html 

From seth at icir.org  Tue Oct  1 10:33:30 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 1 Oct 2013 13:33:30 -0400
Subject: [Bro] Trouble building Bro 2.2 on ARMv6 (Raspberry Pi / Wheezy)
In-Reply-To: <34F95A48-D8BA-4814-AFD4-2D57603CF583@illinois.edu>
References: <20130930121747.GE57016@icir.org>
	<6d228d26-d14f-4973-97d9-2d33bdc3189f@me.com>
	<CAEZw2bwkxSP8AM7_uPRKEqnm0e2omreFqkdcP=C+S-XRybMx4g@mail.gmail.com>
	<34F95A48-D8BA-4814-AFD4-2D57603CF583@illinois.edu>
Message-ID: <83F5E8FA-B273-4BAD-B0EC-211909F4A92F@icir.org>


On Oct 1, 2013, at 10:37 AM, "Siwek, Jonathan Luke" <jsiwek at illinois.edu> wrote:

> Can you try the patch below?


That got past the problem for me.  If it ever finished building I'll run the test suite.
 
 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131001/69ce9ad8/attachment.bin 

From thedestiny.labs at gmail.com  Wed Oct  2 08:50:39 2013
From: thedestiny.labs at gmail.com (destiny labs)
Date: Wed, 2 Oct 2013 12:50:39 -0300
Subject: [Bro] Error openBSD 5.3
Message-ID: <CAFiW5gr-JVoCf_QBDnMgvsMMsMiLddp4-r4ucM2Nj+3bgqNu5w@mail.gmail.com>

Any one help me, please??

-- Performing Test res_mkquery_works_bind
-- Performing Test res_mkquery_works_bind - Success
-- Could NOT find BIND (missing:  BIND_LIBRARY)
-- libmagic version: 4.24
-- Found LibMagic: /usr/local/lib/libmagic.so.3.0
-- Found ZLIB: /usr/lib/libz.so.4.1 (found version "1.2.3")
CMake Error at aux/binpac/CMakeLists.txt:17 (message):
   Could not find prerequisite package 'BIND'
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131002/d1a3c20e/attachment.html 

From jsiwek at illinois.edu  Wed Oct  2 10:03:17 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Wed, 2 Oct 2013 17:03:17 +0000
Subject: [Bro] Error openBSD 5.3
In-Reply-To: <CAFiW5gr-JVoCf_QBDnMgvsMMsMiLddp4-r4ucM2Nj+3bgqNu5w@mail.gmail.com>
References: <CAFiW5gr-JVoCf_QBDnMgvsMMsMiLddp4-r4ucM2Nj+3bgqNu5w@mail.gmail.com>
Message-ID: <57702898-5880-4353-8289-C8760BE50848@illinois.edu>

>    Could not find prerequisite package 'BIND'

You'll need to first get/compile/install the libbind from isc.org.  See [1] for previous discussion.

- Jon

[1] http://mailman.icsi.berkeley.edu/pipermail/bro/2013-August/006745.html


From jamesfhook at gmail.com  Wed Oct  2 10:08:00 2013
From: jamesfhook at gmail.com (James Hook)
Date: Wed, 2 Oct 2013 18:08:00 +0100
Subject: [Bro] Interprocess Communication from BroScript
Message-ID: <CAC+_1Sbe8X913FpCyGTrrxAFhpd7UBLmAn2VF6M2iMifKfuemw@mail.gmail.com>

Hi all,

I have some C code that analyses data (inc. URLs, domains) that I want to
invoke from Bro to extract additional data from network data. I envisage
this data being an extra column in the resultant bro log files.

I'm fairly sure I can do this with awk retrospectively but wanted to ask
the list whether it was possible using some of the IPC commands in the
language. Reading the specifications for these functions there appears to
be no way to invoke the commands and receive non-trivial output.

The commands i'm looking at are
system,system_env - returns the return code from the command (limited to
being an integer) - i thought about returning the result into an
environment variable, but that would require invoked command to be
recompiled
piped_exec - only returns true/false on success failure

Has anybody else done anything similar?
Are there anyways to load modules, call c functions (or functions in
scripting languages) from the bro language?

Thanks in advance

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131002/b3cf5891/attachment.html 

From jamesfhook at gmail.com  Wed Oct  2 10:28:05 2013
From: jamesfhook at gmail.com (James Hook)
Date: Wed, 2 Oct 2013 18:28:05 +0100
Subject: [Bro] Interprocess Communication from BroScript
Message-ID: <CAC+_1SZ6juYw3_Gs7CBN1734apBKeMYnWKYscXXv2CBDp=cGtQ@mail.gmail.com>

Hi all,

I have some C code that analyses data (inc. URLs, domains) that I want to
invoke from Bro to extract additional data from network data. I envisage
this data being an extra column in the resultant bro log files.

I'm fairly sure I can do this with awk retrospectively but wanted to ask
the list whether it was possible using some of the IPC commands in the
language. Reading the specifications for these functions there appears to
be no way to invoke the commands and receive non-trivial output.

The commands i'm looking at are
system,system_env - returns the return code from the command (limited to
being an integer) - i thought about returning the result into an
environment variable, but that would require invoked command to be
recompiled
piped_exec - only returns true/false on success failure

Has anybody else done anything similar?
Are there anyways to load modules, call c functions (or functions in
scripting languages) from the bro language?

Thanks in advance

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131002/fe25d87b/attachment.html 

From jmellander at lbl.gov  Wed Oct  2 10:48:24 2013
From: jmellander at lbl.gov (Jim Mellander)
Date: Wed, 2 Oct 2013 10:48:24 -0700
Subject: [Bro] Interprocess Communication from BroScript
In-Reply-To: <CAC+_1Sbe8X913FpCyGTrrxAFhpd7UBLmAn2VF6M2iMifKfuemw@mail.gmail.com>
References: <CAC+_1Sbe8X913FpCyGTrrxAFhpd7UBLmAn2VF6M2iMifKfuemw@mail.gmail.com>
Message-ID: <CADju=b5UTcL_0EQSviOPcmN0U1umxQG2UhwEoZm3cPY9dopKKg@mail.gmail.com>

Hi James:

Sounds like you could use the Broccoli library to perform the task you are
envisioning: http://www.bro.org/download/README.broccoli.html

Jim Mellander
NERSC Cybersecurity


On Wed, Oct 2, 2013 at 10:08 AM, James Hook <jamesfhook at gmail.com> wrote:

> Hi all,
>
> I have some C code that analyses data (inc. URLs, domains) that I want to
> invoke from Bro to extract additional data from network data. I envisage
> this data being an extra column in the resultant bro log files.
>
> I'm fairly sure I can do this with awk retrospectively but wanted to ask
> the list whether it was possible using some of the IPC commands in the
> language. Reading the specifications for these functions there appears to
> be no way to invoke the commands and receive non-trivial output.
>
> The commands i'm looking at are
> system,system_env - returns the return code from the command (limited to
> being an integer) - i thought about returning the result into an
> environment variable, but that would require invoked command to be
> recompiled
> piped_exec - only returns true/false on success failure
>
> Has anybody else done anything similar?
> Are there anyways to load modules, call c functions (or functions in
> scripting languages) from the bro language?
>
> Thanks in advance
>
> James
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131002/9ef87323/attachment.html 

From seth at icir.org  Wed Oct  2 11:59:57 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 2 Oct 2013 14:59:57 -0400
Subject: [Bro] Interprocess Communication from BroScript
In-Reply-To: <CAC+_1Sbe8X913FpCyGTrrxAFhpd7UBLmAn2VF6M2iMifKfuemw@mail.gmail.com>
References: <CAC+_1Sbe8X913FpCyGTrrxAFhpd7UBLmAn2VF6M2iMifKfuemw@mail.gmail.com>
Message-ID: <0F576C12-0A7A-49FC-84C0-E56C85BD7298@icir.org>


On Oct 2, 2013, at 1:08 PM, James Hook <jamesfhook at gmail.com> wrote:

> Has anybody else done anything similar?
> Are there anyways to load modules, call c functions (or functions in scripting languages) from the bro language?

Bro 2.2 beta has an exec module.

----script------
redef exit_only_after_terminate = T;

@load base/utils/exec
event bro_init()
	{
	when ( local result = Exec::run([$cmd="date"]) )
		{
		print result;
		}
	timeout 5secs
		{
		print "Hm, it didn't return.";
		}
	}
-----end script-----

$ bro -b ./test-exec.bro 
[exit_code=0, signal_exit=F, stdout=[Wed Oct  2 14:59:02 EDT 2013], stderr=<uninitialized>, files=<uninitialized>]


 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131002/3309b62e/attachment.bin 

From jamesfhook at gmail.com  Thu Oct  3 08:03:01 2013
From: jamesfhook at gmail.com (James Hook)
Date: Thu, 3 Oct 2013 16:03:01 +0100
Subject: [Bro] Interprocess Communication from BroScript
In-Reply-To: <0F576C12-0A7A-49FC-84C0-E56C85BD7298@icir.org>
References: <CAC+_1Sbe8X913FpCyGTrrxAFhpd7UBLmAn2VF6M2iMifKfuemw@mail.gmail.com>
	<0F576C12-0A7A-49FC-84C0-E56C85BD7298@icir.org>
Message-ID: <CAC+_1SYf7ksUiPiCO4ks6ukV1MT9OM0-7qG+0PL9m2ACdOnO-Q@mail.gmail.com>

Thanks Seth, I think this is exactly what I need, so I'm going to install
2.2. Beta now!
James


On 2 October 2013 19:59, Seth Hall <seth at icir.org> wrote:

>
> On Oct 2, 2013, at 1:08 PM, James Hook <jamesfhook at gmail.com> wrote:
>
> > Has anybody else done anything similar?
> > Are there anyways to load modules, call c functions (or functions in
> scripting languages) from the bro language?
>
> Bro 2.2 beta has an exec module.
>
> ----script------
> redef exit_only_after_terminate = T;
>
> @load base/utils/exec
> event bro_init()
>         {
>         when ( local result = Exec::run([$cmd="date"]) )
>                 {
>                 print result;
>                 }
>         timeout 5secs
>                 {
>                 print "Hm, it didn't return.";
>                 }
>         }
> -----end script-----
>
> $ bro -b ./test-exec.bro
> [exit_code=0, signal_exit=F, stdout=[Wed Oct  2 14:59:02 EDT 2013],
> stderr=<uninitialized>, files=<uninitialized>]
>
>
>  .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131003/cf5a54b5/attachment.html 

From jmp242 at cornell.edu  Thu Oct  3 12:24:02 2013
From: jmp242 at cornell.edu (James M. Pulver)
Date: Thu, 3 Oct 2013 19:24:02 +0000
Subject: [Bro] Sizing suggestions for Bro 2.2?
Message-ID: <3C0ED4FA0BB9854C91F5B7CF74DF2EA31E969AA8@CH1PRD0411MB431.namprd04.prod.outlook.com>

I have a network with about 3500 network devices spread across multiple subnets, some private and not routed off our internal network. I have mostly 1Gb right now, with some limited SAN networks at 10Gb. I'm guessing I'll want to be able to handle a network core of 10Gb though as we expect to be running that soon for certain links between buildings and out to experimental stations. Our campus backbone link will also be 10Gb soon.

I think currently I am most interested in network traffic from our internal untrusted subnets and VPN subnet to our public and trusted internal subnets.

What additional information should I provide, and what sort of hardware should I expect to run Bro on Scientific Linux 6.4?

--
James Pulver
CLASSE Computer Group
Cornell University






From edthoma at sandia.gov  Thu Oct  3 12:53:07 2013
From: edthoma at sandia.gov (Thomas, Eric D)
Date: Thu, 3 Oct 2013 19:53:07 +0000
Subject: [Bro] [EXTERNAL] Re: Log::add_filter with mime_type or filename
 predicate
In-Reply-To: <B3EA66F8-FF95-4220-B52E-7DAC5CD6DA36@icir.org>
Message-ID: <CE73182D.22591%edthoma@sandia.gov>

I'm still working through this, still encountering issues. The return
statement you provided makes sense and is what I need, but bro is giving
me an error when I use it:

...: not an index type (application/x-dosexec in rec$resp_mime_types)

Does the 'in' operator work with a string and a vector type?
-- 
Eric Thomas

edthoma at sandia.gov




On 9/30/13 1:40 PM, "Seth Hall" <seth at icir.org> wrote:

>
>On Sep 30, 2013, at 3:08 PM, "Thomas, Eric D" <edthoma at sandia.gov> wrote:
>
>> # This line was in the predicate function, but it no longer works
>> # return rec?$mime_type && rec$mime_type == "application/x-dosexec"; },
>
>
>return rec?$resp_mime_types && "application/x-dosexec" in
>rec$resp_mime_types;
>
>  .Seth
>
>--
>Seth Hall
>International Computer Science Institute
>(Bro) because everyone has a network
>http://www.bro.org/
>




From anthony.kasza at gmail.com  Thu Oct  3 19:36:07 2013
From: anthony.kasza at gmail.com (anthony kasza)
Date: Thu, 3 Oct 2013 19:36:07 -0700
Subject: [Bro] Trouble building Bro 2.2 on ARMv6 (Raspberry Pi / Wheezy)
In-Reply-To: <83F5E8FA-B273-4BAD-B0EC-211909F4A92F@icir.org>
References: <20130930121747.GE57016@icir.org>
	<6d228d26-d14f-4973-97d9-2d33bdc3189f@me.com>
	<CAEZw2bwkxSP8AM7_uPRKEqnm0e2omreFqkdcP=C+S-XRybMx4g@mail.gmail.com>
	<34F95A48-D8BA-4814-AFD4-2D57603CF583@illinois.edu>
	<83F5E8FA-B273-4BAD-B0EC-211909F4A92F@icir.org>
Message-ID: <CAEZw2by6Lcd5MpMj-HFa7ZS59V-hV7QkdExc1HAOvaTM_qv1OQ@mail.gmail.com>

Thanks Jon, the patch worked for me too. If anyone is interested, running
make test on the RPi results in the following failures.

[#1] bifs.bloomfilter ... failed
[#1] doc.sphinx.using_bro ... failed
[#1] istate.broccoli ... failed
[#1] istate.broccoli-ipv6 ... failed
[#1] istate.broccoli-ssl ... failed
[#1] istate.pybroccoli ... failed
[#1] scripts.base.frameworks.cluster.start-it-up ... failed
[#1] scripts.base.frameworks.control.configuration_update ... failed
[#1] scripts.base.frameworks.control.id_value ... failed
[#1] scripts.base.frameworks.control.shutdown ... failed
[#1] scripts.base.frameworks.file-analysis.input.basic ... failed
[#1] scripts.base.frameworks.input.event ... failed
[#1] scripts.base.frameworks.input.optional ... failed
[#1] scripts.base.frameworks.input.predicate ... failed
[#1] scripts.base.frameworks.input.raw.basic ... failed
[#1] scripts.base.frameworks.input.reread ... failed
[#1] scripts.base.frameworks.input.sqlite.basic ... failed
[#1] scripts.base.frameworks.input.sqlite.error ... failed
[#1] scripts.base.frameworks.input.sqlite.port ... failed
[#1] scripts.base.frameworks.input.sqlite.types ... failed
[#1] scripts.base.frameworks.input.tableevent ... failed
[#1] scripts.base.frameworks.intel.cluster-transparency ... failed
[#1] scripts.base.frameworks.intel.input-and-match ... failed
[#1] scripts.base.frameworks.intel.read-file-dist-cluster ... failed
[#1] scripts.base.frameworks.logging.remote-types ... failed
[#1] scripts.base.frameworks.logging.sqlite.error ... failed
[#1] scripts.base.frameworks.logging.sqlite.types ... failed
[#1] scripts.base.frameworks.logging.sqlite.wikipedia ... failed
[#1] scripts.base.frameworks.notice.suppression-cluster ... failed
[#1] scripts.base.frameworks.sumstats.basic-cluster ... failed
[#1] scripts.base.frameworks.sumstats.cluster-intermediate-update ... failed
[#1] scripts.base.frameworks.sumstats.on-demand-cluster ... failed
[#1] scripts.base.frameworks.sumstats.sample-cluster ... failed
[#1] scripts.base.frameworks.sumstats.topk-cluster ... failed
[#1] scripts.base.protocols.http.http-methods ... failed


On Tue, Oct 1, 2013 at 10:33 AM, Seth Hall <seth at icir.org> wrote:

>
> On Oct 1, 2013, at 10:37 AM, "Siwek, Jonathan Luke" <jsiwek at illinois.edu>
> wrote:
>
> > Can you try the patch below?
>
>
> That got past the problem for me.  If it ever finished building I'll run
> the test suite.
>
>  .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131003/b7ad13fa/attachment.html 

From jdness at mac.com  Thu Oct  3 19:38:50 2013
From: jdness at mac.com (Jonathan Ness)
Date: Thu, 03 Oct 2013 19:38:50 -0700
Subject: [Bro] Trouble building Bro 2.2 on ARMv6 (Raspberry Pi / Wheezy)
In-Reply-To: <CAEZw2by6Lcd5MpMj-HFa7ZS59V-hV7QkdExc1HAOvaTM_qv1OQ@mail.gmail.com>
References: <20130930121747.GE57016@icir.org>
	<6d228d26-d14f-4973-97d9-2d33bdc3189f@me.com>
	<CAEZw2bwkxSP8AM7_uPRKEqnm0e2omreFqkdcP=C+S-XRybMx4g@mail.gmail.com>
	<34F95A48-D8BA-4814-AFD4-2D57603CF583@illinois.edu>
	<83F5E8FA-B273-4BAD-B0EC-211909F4A92F@icir.org>
	<CAEZw2by6Lcd5MpMj-HFa7ZS59V-hV7QkdExc1HAOvaTM_qv1OQ@mail.gmail.com>
Message-ID: <6EECC971-2314-4312-96A2-6219168C2253@mac.com>

Patch worked for me as well.  Thanks.

> On Oct 3, 2013, at 7:36 PM, anthony kasza <anthony.kasza at gmail.com> wrote:
> 
> Thanks Jon, the patch worked for me too. If anyone is interested, running make test on the RPi results in the following failures. 
> 
> [#1] bifs.bloomfilter ... failed
> [#1] doc.sphinx.using_bro ... failed
> [#1] istate.broccoli ... failed
> [#1] istate.broccoli-ipv6 ... failed
> [#1] istate.broccoli-ssl ... failed
> [#1] istate.pybroccoli ... failed
> [#1] scripts.base.frameworks.cluster.start-it-up ... failed
> [#1] scripts.base.frameworks.control.configuration_update ... failed
> [#1] scripts.base.frameworks.control.id_value ... failed
> [#1] scripts.base.frameworks.control.shutdown ... failed
> [#1] scripts.base.frameworks.file-analysis.input.basic ... failed
> [#1] scripts.base.frameworks.input.event ... failed
> [#1] scripts.base.frameworks.input.optional ... failed
> [#1] scripts.base.frameworks.input.predicate ... failed
> [#1] scripts.base.frameworks.input.raw.basic ... failed
> [#1] scripts.base.frameworks.input.reread ... failed
> [#1] scripts.base.frameworks.input.sqlite.basic ... failed
> [#1] scripts.base.frameworks.input.sqlite.error ... failed
> [#1] scripts.base.frameworks.input.sqlite.port ... failed
> [#1] scripts.base.frameworks.input.sqlite.types ... failed
> [#1] scripts.base.frameworks.input.tableevent ... failed
> [#1] scripts.base.frameworks.intel.cluster-transparency ... failed
> [#1] scripts.base.frameworks.intel.input-and-match ... failed
> [#1] scripts.base.frameworks.intel.read-file-dist-cluster ... failed
> [#1] scripts.base.frameworks.logging.remote-types ... failed
> [#1] scripts.base.frameworks.logging.sqlite.error ... failed
> [#1] scripts.base.frameworks.logging.sqlite.types ... failed
> [#1] scripts.base.frameworks.logging.sqlite.wikipedia ... failed
> [#1] scripts.base.frameworks.notice.suppression-cluster ... failed
> [#1] scripts.base.frameworks.sumstats.basic-cluster ... failed
> [#1] scripts.base.frameworks.sumstats.cluster-intermediate-update ... failed
> [#1] scripts.base.frameworks.sumstats.on-demand-cluster ... failed
> [#1] scripts.base.frameworks.sumstats.sample-cluster ... failed
> [#1] scripts.base.frameworks.sumstats.topk-cluster ... failed
> [#1] scripts.base.protocols.http.http-methods ... failed
> 
> 
>> On Tue, Oct 1, 2013 at 10:33 AM, Seth Hall <seth at icir.org> wrote:
>> 
>> On Oct 1, 2013, at 10:37 AM, "Siwek, Jonathan Luke" <jsiwek at illinois.edu> wrote:
>> 
>> > Can you try the patch below?
>> 
>> 
>> That got past the problem for me.  If it ever finished building I'll run the test suite.
>> 
>>  .Seth
>> 
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131003/aa098325/attachment.html 

From seth at icir.org  Thu Oct  3 22:17:37 2013
From: seth at icir.org (Seth Hall)
Date: Fri, 4 Oct 2013 01:17:37 -0400
Subject: [Bro] Trouble building Bro 2.2 on ARMv6 (Raspberry Pi / Wheezy)
In-Reply-To: <CAEZw2by6Lcd5MpMj-HFa7ZS59V-hV7QkdExc1HAOvaTM_qv1OQ@mail.gmail.com>
References: <20130930121747.GE57016@icir.org>
	<6d228d26-d14f-4973-97d9-2d33bdc3189f@me.com>
	<CAEZw2bwkxSP8AM7_uPRKEqnm0e2omreFqkdcP=C+S-XRybMx4g@mail.gmail.com>
	<34F95A48-D8BA-4814-AFD4-2D57603CF583@illinois.edu>
	<83F5E8FA-B273-4BAD-B0EC-211909F4A92F@icir.org>
	<CAEZw2by6Lcd5MpMj-HFa7ZS59V-hV7QkdExc1HAOvaTM_qv1OQ@mail.gmail.com>
Message-ID: <1C2ADC62-F45E-417A-A846-A86C3C2773BA@icir.org>


On Oct 3, 2013, at 10:36 PM, anthony kasza <anthony.kasza at gmail.com> wrote:

> Thanks Jon, the patch worked for me too. If anyone is interested, running make test on the RPi results in the following failures. 
> 
> [#1] bifs.bloomfilter ? failed

I believe all of these test failures are from the Pi running out of memory. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131004/6e9e1cc3/attachment.bin 

From seth at icir.org  Thu Oct  3 22:33:06 2013
From: seth at icir.org (Seth Hall)
Date: Fri, 4 Oct 2013 01:33:06 -0400
Subject: [Bro] [EXTERNAL] Log::add_filter with mime_type or filename
	predicate
In-Reply-To: <CE73182D.22591%edthoma@sandia.gov>
References: <CE73182D.22591%edthoma@sandia.gov>
Message-ID: <33C57462-5170-47A1-90D7-D73E12230FC0@icir.org>


On Oct 3, 2013, at 3:53 PM, "Thomas, Eric D" <edthoma at sandia.gov> wrote:

> ...: not an index type (application/x-dosexec in rec$resp_mime_types)
> 
> Does the 'in' operator work with a string and a vector type?

Arg!  I forgot that was a vector, I was thinking it was a set.  It's little things like this that are pretty annoying to eventually find out (that I didn't consider some situation).

I don't really like this solution but it should work if you put it in your predicate...

if ( rec?$resp_mime_types ) 
	{
	for ( i in rec$resp_mime_types )
		{
		if ( "application/x-dosexec" == rec$resp_mime_types[i] )
			return T;
		}
	}
return F;


  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131004/d51537b1/attachment.bin 

From dnthayer at illinois.edu  Fri Oct  4 08:37:47 2013
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Fri, 4 Oct 2013 10:37:47 -0500
Subject: [Bro] Trouble building Bro 2.2 on ARMv6 (Raspberry Pi / Wheezy)
In-Reply-To: <CAEZw2by6Lcd5MpMj-HFa7ZS59V-hV7QkdExc1HAOvaTM_qv1OQ@mail.gmail.com>
References: <20130930121747.GE57016@icir.org>
	<6d228d26-d14f-4973-97d9-2d33bdc3189f@me.com>
	<CAEZw2bwkxSP8AM7_uPRKEqnm0e2omreFqkdcP=C+S-XRybMx4g@mail.gmail.com>
	<34F95A48-D8BA-4814-AFD4-2D57603CF583@illinois.edu>
	<83F5E8FA-B273-4BAD-B0EC-211909F4A92F@icir.org>
	<CAEZw2by6Lcd5MpMj-HFa7ZS59V-hV7QkdExc1HAOvaTM_qv1OQ@mail.gmail.com>
Message-ID: <524EE0CB.7080909@illinois.edu>

What happens if you try to re-run only the failed tests?  (using
the "-r" option to btest)

   cd bro/testing/btest
   ../../aux/btest/btest -r


On 10/03/2013 09:36 PM, anthony kasza wrote:
> Thanks Jon, the patch worked for me too. If anyone is interested,
> running make test on the RPi results in the following failures.
>
> [#1] bifs.bloomfilter ... failed
> [#1] doc.sphinx.using_bro ... failed
> [#1] istate.broccoli ... failed
> [#1] istate.broccoli-ipv6 ... failed
> [#1] istate.broccoli-ssl ... failed
> [#1] istate.pybroccoli ... failed
> [#1] scripts.base.frameworks.cluster.start-it-up ... failed
> [#1] scripts.base.frameworks.control.configuration_update ... failed
> [#1] scripts.base.frameworks.control.id_value ... failed
> [#1] scripts.base.frameworks.control.shutdown ... failed
> [#1] scripts.base.frameworks.file-analysis.input.basic ... failed
> [#1] scripts.base.frameworks.input.event ... failed
> [#1] scripts.base.frameworks.input.optional ... failed
> [#1] scripts.base.frameworks.input.predicate ... failed
> [#1] scripts.base.frameworks.input.raw.basic ... failed
> [#1] scripts.base.frameworks.input.reread ... failed
> [#1] scripts.base.frameworks.input.sqlite.basic ... failed
> [#1] scripts.base.frameworks.input.sqlite.error ... failed
> [#1] scripts.base.frameworks.input.sqlite.port ... failed
> [#1] scripts.base.frameworks.input.sqlite.types ... failed
> [#1] scripts.base.frameworks.input.tableevent ... failed
> [#1] scripts.base.frameworks.intel.cluster-transparency ... failed
> [#1] scripts.base.frameworks.intel.input-and-match ... failed
> [#1] scripts.base.frameworks.intel.read-file-dist-cluster ... failed
> [#1] scripts.base.frameworks.logging.remote-types ... failed
> [#1] scripts.base.frameworks.logging.sqlite.error ... failed
> [#1] scripts.base.frameworks.logging.sqlite.types ... failed
> [#1] scripts.base.frameworks.logging.sqlite.wikipedia ... failed
> [#1] scripts.base.frameworks.notice.suppression-cluster ... failed
> [#1] scripts.base.frameworks.sumstats.basic-cluster ... failed
> [#1] scripts.base.frameworks.sumstats.cluster-intermediate-update ... failed
> [#1] scripts.base.frameworks.sumstats.on-demand-cluster ... failed
> [#1] scripts.base.frameworks.sumstats.sample-cluster ... failed
> [#1] scripts.base.frameworks.sumstats.topk-cluster ... failed
> [#1] scripts.base.protocols.http.http-methods ... failed
>
>
> On Tue, Oct 1, 2013 at 10:33 AM, Seth Hall <seth at icir.org
> <mailto:seth at icir.org>> wrote:
>
>
>     On Oct 1, 2013, at 10:37 AM, "Siwek, Jonathan Luke"
>     <jsiwek at illinois.edu <mailto:jsiwek at illinois.edu>> wrote:
>
>      > Can you try the patch below?
>
>
>     That got past the problem for me.  If it ever finished building I'll
>     run the test suite.
>
>       .Seth
>
>     --
>     Seth Hall
>     International Computer Science Institute
>     (Bro) because everyone has a network
>     http://www.bro.org/
>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



From jswan at sugf.com  Fri Oct  4 09:54:05 2013
From: jswan at sugf.com (Swan, Jay)
Date: Fri, 4 Oct 2013 16:54:05 +0000
Subject: [Bro] Bro vs NetFlow
Message-ID: <20131004165407.4244C2C4002@rock.ICSI.Berkeley.EDU>

I've been running Bro 2.1 as part of Security Onion on our Internet edge for quite a while now, and I've found that at least for security-related tasks I now almost always use Bro instead of NetFlow, in situations where I would have previously used NetFlow alone.

We are currently evaluating new NetFlow products, and I started wondering: do folks using Bro also use NetFlow as part of day-to-day operations, or are the NetFlow users generally on a separate network team that's not using Bro at all? Has anyone completely replaced NetFlow with Bro?

I'm aware that Bro has a NetFlow v5 analyzer, but I don't know much about it. Is anyone using that extensively? If so, how?

However, one of the reasons we're looking at a new NetFlow product is for compatibility with various proprietary IPFIX export data, such as Cisco's various exports based off their NBAR2 feature set. I remember Seth mentioning at FloCon that writing an IPFIX analyzer for Bro would be insanely complex, so I don't have hopes of that happening anytime soon.

Jay Swan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131004/5b892cb3/attachment.html 

From oguzyarimtepe at gmail.com  Fri Oct  4 11:18:59 2013
From: oguzyarimtepe at gmail.com (=?UTF-8?B?T8SfdXogWWFyxLFtdGVwZQ==?=)
Date: Fri, 4 Oct 2013 21:18:59 +0300
Subject: [Bro] another kdd cup question
Message-ID: <CAKtWOTTPtRW-VP3WinFQos0=R8O9h9dYTMQOdPhsM=Snx+wAog@mail.gmail.com>

I was investigating to create KDD Cup 99 attributes on a live traffic. I
encountered with some papers telling that they reproduce the same attribute
values by using Bro-IDS. I am not sure whether all the values can be
gathered from a live traffic, so i am asking whether it is possible to
calculate the below attributes from a live GBit traffic.

    Num.

Name

Type

Description

1

duration

integer

duration of the connection

2

   protocol_type

nominal

protocol type of the connection: TCP, UDP and ICMP

3

   service

nominal

http, ftp, smtp, telnet... and other (if not much used service)

4


  flag

nominal

connection status. The possible status are this: SF, S0, S1, S2, S3,OTH,
REJ, RSTO, RSTOS0, SH, RSTRH, SHR

5


  src_bytes

integer

bytes sent in one connection

6

   dst_bytes

integer

bytes received in one connection

7


  land

binary

if source and destination IP addresses and port numbers are equal then this
variable takes value 1 else 0

8

   wrong_fragment

integer

sum of bad checksum packets in a connection

9


  urgent

integer

sum of urgent packets in a connections. Urgent packets are packet with the
urgent bit activated

Here i am not sure about the wrong_fragment and urgent packet number part.
Will be great if someone enlightens me.

    Num.

Name

Type

Description

10

hot


  integer


  sum of hot actions in a connection such as: entering a systetory,
creating programs and executing programs


   11

   num_failed_logins


  integer


  number of incorrect logins in a connection


   12

   logged_in


  integer


  if the login is correct then 1 else 0


   13


  num_compromised


  integer


  sum of times appearance ?not found? error in a connection


   14


  root_shell


  integer


  if the root gets the shell then 1 else 0


   15

   su_attempted


  integer


  if the su command has been used then 1 else 0


   16


  num_root


  integer


  sum of operations performed as root in a connection


   17

   num_file_creations


  integer


  sum of file creations in a connection


   18


  num_shells


  integer


  number of logins of normal users


   19

num_access_files


  integer


  sum of operations in control files in a connection


   20

num_outbound_cmds


  integer


  sum of outbound commands in a ftp session


   21

is_hot_login


  integer


  if the user is accessing as root or adm


   22

is_guest_login


  integer


  if the user is accessing as guest, anonymous or visitor




It seems these attributes require payload analysis. I am not sure whether
Bro is able to detect some of them by default rules or whether i will need
to write some custom ones.


    Num.

Name

Type

Description

23

count

  integer

sum of connections to the same destination IP address

24

   srv_count

  integer

sum of connections to the same destination port number

25

   serror_rate

  real

the percentage of connections that have activated the flag (4) s0, s1, s2

or s3, among the connections aggregated in count (23)

26


  srv_serror_rate

  real

the percentage of connections that have activated the flag (4) s0, s1, s2

or s3, among the connections aggregated in srv_count (24)

27


  rerror_rate

  real

the percentage of connections that have activated the flag (4) REJ,

among the connections aggregated in count (23)

28

   srv_error_rate

  real

the percentage of connections that have activated the flag (4) REJ,

among the connections aggregated in srv_count (24)

29


  same_srv_rate

  real

the percentage of connections that were to the same service, among

the connections aggregated in count (23)

30

   diff_srv_rate

  real

the percentage of connections that were to different services, among

the connections aggregated in count (23)

31


  srv_diff_host_rate

  real

the percentage of connections that were to different destination ma-

chines among the connections aggregated in srv_count (24)

These are totally ambiguous to me. I think i will need extra issue to
handle som results. But whether to wait some people to guide me first.


So if bro-ids is enough to calculate above attributes from a live traffic
somehow, whether either saving some attributes to DB and then reprocessing
them or any guidance will be appreciated. What i am trying is to recreate
these attributes for a real traffic and test my algorithm with the up to
date dataset.

-- 
O?uz Yar?mtepe
http://about.me/oguzy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131004/23aabff3/attachment.html 

From jmellander at lbl.gov  Fri Oct  4 12:57:52 2013
From: jmellander at lbl.gov (Jim Mellander)
Date: Fri, 4 Oct 2013 12:57:52 -0700
Subject: [Bro] another kdd cup question
In-Reply-To: <CAKtWOTTPtRW-VP3WinFQos0=R8O9h9dYTMQOdPhsM=Snx+wAog@mail.gmail.com>
References: <CAKtWOTTPtRW-VP3WinFQos0=R8O9h9dYTMQOdPhsM=Snx+wAog@mail.gmail.com>
Message-ID: <CADju=b7RjOxdbJ_48CeaoiaH9qUuYU9+0Hejz0ivTVhenG+2sA@mail.gmail.com>

Hi:

A number of your items (specifically # 10-22) appear to require inspection
inside interactive sessions, which (unless the connection is cleartext), is
not accessible to a network level monitor.  Lack of inspection into
sessions, and the security benefits gained as a result are major benefits
of modern session tools, of which the standard is ssh.

If you have access to the systems you wish to monitor, you can install
Instrumented SSHd, which will send a clear-text stream of the session to a
bro monitor for inspection. See: https://code.google.com/p/auditing-sshd/

Some of the information you want might also be logged via syslog, such as
authentication events.

Hope this helps.

Jim Mellander
NERSC Cybersecurity


On Fri, Oct 4, 2013 at 11:18 AM, O?uz Yar?mtepe <oguzyarimtepe at gmail.com>wrote:

> I was investigating to create KDD Cup 99 attributes on a live traffic. I
> encountered with some papers telling that they reproduce the same attribute
> values by using Bro-IDS. I am not sure whether all the values can be
> gathered from a live traffic, so i am asking whether it is possible to
> calculate the below attributes from a live GBit traffic.
>
>     Num.
>
> Name
>
> Type
>
> Description
>
> 1
>
> duration
>
> integer
>
> duration of the connection
>
> 2
>
>    protocol_type
>
> nominal
>
> protocol type of the connection: TCP, UDP and ICMP
>
> 3
>
>    service
>
> nominal
>
> http, ftp, smtp, telnet... and other (if not much used service)
>
> 4
>
>
>   flag
>
> nominal
>
> connection status. The possible status are this: SF, S0, S1, S2, S3,OTH,
> REJ, RSTO, RSTOS0, SH, RSTRH, SHR
>
> 5
>
>
>   src_bytes
>
> integer
>
> bytes sent in one connection
>
> 6
>
>    dst_bytes
>
> integer
>
> bytes received in one connection
>
> 7
>
>
>   land
>
> binary
>
> if source and destination IP addresses and port numbers are equal then
> this variable takes value 1 else 0
>
> 8
>
>    wrong_fragment
>
> integer
>
> sum of bad checksum packets in a connection
>
> 9
>
>
>   urgent
>
> integer
>
> sum of urgent packets in a connections. Urgent packets are packet with the
> urgent bit activated
>
> Here i am not sure about the wrong_fragment and urgent packet number part.
> Will be great if someone enlightens me.
>
>     Num.
>
> Name
>
> Type
>
> Description
>
> 10
>
> hot
>
>
>   integer
>
>
>   sum of hot actions in a connection such as: entering a systetory,
> creating programs and executing programs
>
>
>    11
>
>    num_failed_logins
>
>
>   integer
>
>
>   number of incorrect logins in a connection
>
>
>    12
>
>    logged_in
>
>
>   integer
>
>
>   if the login is correct then 1 else 0
>
>
>    13
>
>
>   num_compromised
>
>
>   integer
>
>
>   sum of times appearance ?not found? error in a connection
>
>
>    14
>
>
>   root_shell
>
>
>   integer
>
>
>   if the root gets the shell then 1 else 0
>
>
>    15
>
>    su_attempted
>
>
>   integer
>
>
>   if the su command has been used then 1 else 0
>
>
>    16
>
>
>   num_root
>
>
>   integer
>
>
>   sum of operations performed as root in a connection
>
>
>    17
>
>    num_file_creations
>
>
>   integer
>
>
>   sum of file creations in a connection
>
>
>    18
>
>
>   num_shells
>
>
>   integer
>
>
>   number of logins of normal users
>
>
>    19
>
> num_access_files
>
>
>   integer
>
>
>   sum of operations in control files in a connection
>
>
>    20
>
> num_outbound_cmds
>
>
>   integer
>
>
>   sum of outbound commands in a ftp session
>
>
>    21
>
> is_hot_login
>
>
>   integer
>
>
>   if the user is accessing as root or adm
>
>
>    22
>
> is_guest_login
>
>
>   integer
>
>
>   if the user is accessing as guest, anonymous or visitor
>
>
>
>
> It seems these attributes require payload analysis. I am not sure whether
> Bro is able to detect some of them by default rules or whether i will need
> to write some custom ones.
>
>
>     Num.
>
> Name
>
> Type
>
> Description
>
> 23
>
> count
>
>   integer
>
> sum of connections to the same destination IP address
>
> 24
>
>    srv_count
>
>   integer
>
> sum of connections to the same destination port number
>
> 25
>
>    serror_rate
>
>   real
>
> the percentage of connections that have activated the flag (4) s0, s1, s2
>
> or s3, among the connections aggregated in count (23)
>
> 26
>
>
>   srv_serror_rate
>
>   real
>
> the percentage of connections that have activated the flag (4) s0, s1, s2
>
> or s3, among the connections aggregated in srv_count (24)
>
> 27
>
>
>   rerror_rate
>
>   real
>
> the percentage of connections that have activated the flag (4) REJ,
>
> among the connections aggregated in count (23)
>
> 28
>
>    srv_error_rate
>
>   real
>
> the percentage of connections that have activated the flag (4) REJ,
>
> among the connections aggregated in srv_count (24)
>
> 29
>
>
>   same_srv_rate
>
>   real
>
> the percentage of connections that were to the same service, among
>
> the connections aggregated in count (23)
>
> 30
>
>    diff_srv_rate
>
>   real
>
> the percentage of connections that were to different services, among
>
> the connections aggregated in count (23)
>
> 31
>
>
>   srv_diff_host_rate
>
>   real
>
> the percentage of connections that were to different destination ma-
>
> chines among the connections aggregated in srv_count (24)
>
> These are totally ambiguous to me. I think i will need extra issue to
> handle som results. But whether to wait some people to guide me first.
>
>
> So if bro-ids is enough to calculate above attributes from a live traffic
> somehow, whether either saving some attributes to DB and then reprocessing
> them or any guidance will be appreciated. What i am trying is to recreate
> these attributes for a real traffic and test my algorithm with the up to
> date dataset.
>
> --
> O?uz Yar?mtepe
> http://about.me/oguzy
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131004/84c3f5cc/attachment.html 

From anthony.kasza at gmail.com  Fri Oct  4 18:52:47 2013
From: anthony.kasza at gmail.com (anthony kasza)
Date: Fri, 4 Oct 2013 18:52:47 -0700
Subject: [Bro] Trouble building Bro 2.2 on ARMv6 (Raspberry Pi / Wheezy)
In-Reply-To: <524EE0CB.7080909@illinois.edu>
References: <20130930121747.GE57016@icir.org>
	<6d228d26-d14f-4973-97d9-2d33bdc3189f@me.com>
	<CAEZw2bwkxSP8AM7_uPRKEqnm0e2omreFqkdcP=C+S-XRybMx4g@mail.gmail.com>
	<34F95A48-D8BA-4814-AFD4-2D57603CF583@illinois.edu>
	<83F5E8FA-B273-4BAD-B0EC-211909F4A92F@icir.org>
	<CAEZw2by6Lcd5MpMj-HFa7ZS59V-hV7QkdExc1HAOvaTM_qv1OQ@mail.gmail.com>
	<524EE0CB.7080909@illinois.edu>
Message-ID: <CAEZw2bwhj3BvVCM4jNLS-HKGE8VLMfOV+V1E3wzwCqqjh16FYw@mail.gmail.com>

Dan,

The same failures occur.
root at raspberrypi:/opt/bro_src/testing/btest# ../../aux/btest/btest -r
[  0%] bifs.bloomfilter ... failed
[  2%] doc.sphinx.using_bro ... failed
[  5%] istate.broccoli ... failed
[  8%] istate.broccoli-ipv6 ... failed
[ 11%] istate.broccoli-ssl ... failed
[ 14%] istate.pybroccoli ... failed
[ 17%] scripts.base.frameworks.cluster.start-it-up ... failed
[ 20%] scripts.base.frameworks.control.configuration_update ... failed
[ 22%] scripts.base.frameworks.control.id_value ... failed
[ 25%] scripts.base.frameworks.control.shutdown ... failed
[ 28%] scripts.base.frameworks.file-analysis.input.basic ... failed
[ 31%] scripts.base.frameworks.input.event ... failed
[ 34%] scripts.base.frameworks.input.optional ... failed
[ 37%] scripts.base.frameworks.input.predicate ... failed
[ 40%] scripts.base.frameworks.input.raw.basic ... failed
[ 45%] scripts.base.frameworks.input.sqlite.basic ... failed
[ 48%] scripts.base.frameworks.input.sqlite.error ... failed
[ 51%] scripts.base.frameworks.input.sqlite.port ... failed
[ 54%] scripts.base.frameworks.input.sqlite.types ... failed
[ 57%] scripts.base.frameworks.input.tableevent ... failed
[ 60%] scripts.base.frameworks.intel.cluster-transparency ... failed
[ 62%] scripts.base.frameworks.intel.input-and-match ... failed
[ 65%] scripts.base.frameworks.intel.read-file-dist-cluster ... failed
[ 68%] scripts.base.frameworks.logging.remote-types ... failed
[ 71%] scripts.base.frameworks.logging.sqlite.error ... failed
[ 74%] scripts.base.frameworks.logging.sqlite.types ... failed
[ 77%] scripts.base.frameworks.logging.sqlite.wikipedia ... failed
[ 80%] scripts.base.frameworks.notice.suppression-cluster ... failed
[ 82%] scripts.base.frameworks.sumstats.basic-cluster ... failed
[ 85%] scripts.base.frameworks.sumstats.cluster-intermediate-update ...
failed
[ 88%] scripts.base.frameworks.sumstats.on-demand-cluster ... failed
[ 91%] scripts.base.frameworks.sumstats.sample-cluster ... failed
[ 94%] scripts.base.frameworks.sumstats.topk-cluster ... failed
[ 97%] scripts.base.protocols.http.http-methods ... failed
34 of 35 tests failed



On Fri, Oct 4, 2013 at 8:37 AM, Daniel Thayer <dnthayer at illinois.edu> wrote:

> What happens if you try to re-run only the failed tests?  (using
> the "-r" option to btest)
>
>   cd bro/testing/btest
>   ../../aux/btest/btest -r
>
>
>
> On 10/03/2013 09:36 PM, anthony kasza wrote:
>
>> Thanks Jon, the patch worked for me too. If anyone is interested,
>> running make test on the RPi results in the following failures.
>>
>> [#1] bifs.bloomfilter ... failed
>> [#1] doc.sphinx.using_bro ... failed
>> [#1] istate.broccoli ... failed
>> [#1] istate.broccoli-ipv6 ... failed
>> [#1] istate.broccoli-ssl ... failed
>> [#1] istate.pybroccoli ... failed
>> [#1] scripts.base.frameworks.**cluster.start-it-up ... failed
>> [#1] scripts.base.frameworks.**control.configuration_update ... failed
>> [#1] scripts.base.frameworks.**control.id_value ... failed
>> [#1] scripts.base.frameworks.**control.shutdown ... failed
>> [#1] scripts.base.frameworks.file-**analysis.input.basic ... failed
>> [#1] scripts.base.frameworks.input.**event ... failed
>> [#1] scripts.base.frameworks.input.**optional ... failed
>> [#1] scripts.base.frameworks.input.**predicate ... failed
>> [#1] scripts.base.frameworks.input.**raw.basic ... failed
>> [#1] scripts.base.frameworks.input.**reread ... failed
>> [#1] scripts.base.frameworks.input.**sqlite.basic ... failed
>> [#1] scripts.base.frameworks.input.**sqlite.error ... failed
>> [#1] scripts.base.frameworks.input.**sqlite.port ... failed
>> [#1] scripts.base.frameworks.input.**sqlite.types ... failed
>> [#1] scripts.base.frameworks.input.**tableevent ... failed
>> [#1] scripts.base.frameworks.intel.**cluster-transparency ... failed
>> [#1] scripts.base.frameworks.intel.**input-and-match ... failed
>> [#1] scripts.base.frameworks.intel.**read-file-dist-cluster ... failed
>> [#1] scripts.base.frameworks.**logging.remote-types ... failed
>> [#1] scripts.base.frameworks.**logging.sqlite.error ... failed
>> [#1] scripts.base.frameworks.**logging.sqlite.types ... failed
>> [#1] scripts.base.frameworks.**logging.sqlite.wikipedia ... failed
>> [#1] scripts.base.frameworks.**notice.suppression-cluster ... failed
>> [#1] scripts.base.frameworks.**sumstats.basic-cluster ... failed
>> [#1] scripts.base.frameworks.**sumstats.cluster-intermediate-**update
>> ... failed
>> [#1] scripts.base.frameworks.**sumstats.on-demand-cluster ... failed
>> [#1] scripts.base.frameworks.**sumstats.sample-cluster ... failed
>> [#1] scripts.base.frameworks.**sumstats.topk-cluster ... failed
>> [#1] scripts.base.protocols.http.**http-methods ... failed
>>
>>
>> On Tue, Oct 1, 2013 at 10:33 AM, Seth Hall <seth at icir.org
>> <mailto:seth at icir.org>> wrote:
>>
>>
>>     On Oct 1, 2013, at 10:37 AM, "Siwek, Jonathan Luke"
>>     <jsiwek at illinois.edu <mailto:jsiwek at illinois.edu>> wrote:
>>
>>      > Can you try the patch below?
>>
>>
>>     That got past the problem for me.  If it ever finished building I'll
>>     run the test suite.
>>
>>       .Seth
>>
>>     --
>>     Seth Hall
>>     International Computer Science Institute
>>     (Bro) because everyone has a network
>>     http://www.bro.org/
>>
>>
>>
>>
>> ______________________________**_________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.**EDU/mailman/listinfo/bro<http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131004/abfe5f55/attachment.html 

From seth at icir.org  Fri Oct  4 19:16:43 2013
From: seth at icir.org (Seth Hall)
Date: Fri, 4 Oct 2013 22:16:43 -0400
Subject: [Bro] Trouble building Bro 2.2 on ARMv6 (Raspberry Pi / Wheezy)
In-Reply-To: <CAEZw2bwhj3BvVCM4jNLS-HKGE8VLMfOV+V1E3wzwCqqjh16FYw@mail.gmail.com>
References: <20130930121747.GE57016@icir.org>
	<6d228d26-d14f-4973-97d9-2d33bdc3189f@me.com>
	<CAEZw2bwkxSP8AM7_uPRKEqnm0e2omreFqkdcP=C+S-XRybMx4g@mail.gmail.com>
	<34F95A48-D8BA-4814-AFD4-2D57603CF583@illinois.edu>
	<83F5E8FA-B273-4BAD-B0EC-211909F4A92F@icir.org>
	<CAEZw2by6Lcd5MpMj-HFa7ZS59V-hV7QkdExc1HAOvaTM_qv1OQ@mail.gmail.com>
	<524EE0CB.7080909@illinois.edu>
	<CAEZw2bwhj3BvVCM4jNLS-HKGE8VLMfOV+V1E3wzwCqqjh16FYw@mail.gmail.com>
Message-ID: <F7959501-7BF9-4882-9B89-548E145A916D@icir.org>


On Oct 4, 2013, at 9:52 PM, anthony kasza <anthony.kasza at gmail.com> wrote:

> The same failures occur.

I'm almost certain that these are just memory exhaustion. (I think you meant to run that with -d too instead of -r).

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131004/0e6df937/attachment.bin 

From dnthayer at illinois.edu  Fri Oct  4 20:59:56 2013
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Fri, 4 Oct 2013 22:59:56 -0500
Subject: [Bro] Trouble building Bro 2.2 on ARMv6 (Raspberry Pi / Wheezy)
In-Reply-To: <CAEZw2bwhj3BvVCM4jNLS-HKGE8VLMfOV+V1E3wzwCqqjh16FYw@mail.gmail.com>
References: <20130930121747.GE57016@icir.org>
	<6d228d26-d14f-4973-97d9-2d33bdc3189f@me.com>
	<CAEZw2bwkxSP8AM7_uPRKEqnm0e2omreFqkdcP=C+S-XRybMx4g@mail.gmail.com>
	<34F95A48-D8BA-4814-AFD4-2D57603CF583@illinois.edu>
	<83F5E8FA-B273-4BAD-B0EC-211909F4A92F@icir.org>
	<CAEZw2by6Lcd5MpMj-HFa7ZS59V-hV7QkdExc1HAOvaTM_qv1OQ@mail.gmail.com>
	<524EE0CB.7080909@illinois.edu>
	<CAEZw2bwhj3BvVCM4jNLS-HKGE8VLMfOV+V1E3wzwCqqjh16FYw@mail.gmail.com>
Message-ID: <524F8EBC.8090804@illinois.edu>

Have you tried looking in the diag.log file?
That file contains diagnostic output for all failed tests,
and is automatically created when using "make test" (if
running btest manually, then you need to use the "-f"
option to create that file).


On 10/04/2013 08:52 PM, anthony kasza wrote:
> Dan,
>
> The same failures occur.
> root at raspberrypi:/opt/bro_src/testing/btest# ../../aux/btest/btest -r
> [  0%] bifs.bloomfilter ... failed
> [  2%] doc.sphinx.using_bro ... failed
> [  5%] istate.broccoli ... failed
> [  8%] istate.broccoli-ipv6 ... failed
> [ 11%] istate.broccoli-ssl ... failed
> [ 14%] istate.pybroccoli ... failed
> [ 17%] scripts.base.frameworks.cluster.start-it-up ... failed
> [ 20%] scripts.base.frameworks.control.configuration_update ... failed
> [ 22%] scripts.base.frameworks.control.id_value ... failed
> [ 25%] scripts.base.frameworks.control.shutdown ... failed
> [ 28%] scripts.base.frameworks.file-analysis.input.basic ... failed
> [ 31%] scripts.base.frameworks.input.event ... failed
> [ 34%] scripts.base.frameworks.input.optional ... failed
> [ 37%] scripts.base.frameworks.input.predicate ... failed
> [ 40%] scripts.base.frameworks.input.raw.basic ... failed
> [ 45%] scripts.base.frameworks.input.sqlite.basic ... failed
> [ 48%] scripts.base.frameworks.input.sqlite.error ... failed
> [ 51%] scripts.base.frameworks.input.sqlite.port ... failed
> [ 54%] scripts.base.frameworks.input.sqlite.types ... failed
> [ 57%] scripts.base.frameworks.input.tableevent ... failed
> [ 60%] scripts.base.frameworks.intel.cluster-transparency ... failed
> [ 62%] scripts.base.frameworks.intel.input-and-match ... failed
> [ 65%] scripts.base.frameworks.intel.read-file-dist-cluster ... failed
> [ 68%] scripts.base.frameworks.logging.remote-types ... failed
> [ 71%] scripts.base.frameworks.logging.sqlite.error ... failed
> [ 74%] scripts.base.frameworks.logging.sqlite.types ... failed
> [ 77%] scripts.base.frameworks.logging.sqlite.wikipedia ... failed
> [ 80%] scripts.base.frameworks.notice.suppression-cluster ... failed
> [ 82%] scripts.base.frameworks.sumstats.basic-cluster ... failed
> [ 85%] scripts.base.frameworks.sumstats.cluster-intermediate-update ...
> failed
> [ 88%] scripts.base.frameworks.sumstats.on-demand-cluster ... failed
> [ 91%] scripts.base.frameworks.sumstats.sample-cluster ... failed
> [ 94%] scripts.base.frameworks.sumstats.topk-cluster ... failed
> [ 97%] scripts.base.protocols.http.http-methods ... failed
> 34 of 35 tests failed
>
>
>
> On Fri, Oct 4, 2013 at 8:37 AM, Daniel Thayer <dnthayer at illinois.edu
> <mailto:dnthayer at illinois.edu>> wrote:
>
>     What happens if you try to re-run only the failed tests?  (using
>     the "-r" option to btest)
>
>        cd bro/testing/btest
>        ../../aux/btest/btest -r
>
>
>
>     On 10/03/2013 09:36 PM, anthony kasza wrote:
>
>         Thanks Jon, the patch worked for me too. If anyone is interested,
>         running make test on the RPi results in the following failures.
>
>         [#1] bifs.bloomfilter ... failed
>         [#1] doc.sphinx.using_bro ... failed
>         [#1] istate.broccoli ... failed
>         [#1] istate.broccoli-ipv6 ... failed
>         [#1] istate.broccoli-ssl ... failed
>         [#1] istate.pybroccoli ... failed
>         [#1] scripts.base.frameworks.__cluster.start-it-up ... failed
>         [#1] scripts.base.frameworks.__control.configuration_update ...
>         failed
>         [#1] scripts.base.frameworks.__control.id_value ... failed
>         [#1] scripts.base.frameworks.__control.shutdown ... failed
>         [#1] scripts.base.frameworks.file-__analysis.input.basic ... failed
>         [#1] scripts.base.frameworks.input.__event ... failed
>         [#1] scripts.base.frameworks.input.__optional ... failed
>         [#1] scripts.base.frameworks.input.__predicate ... failed
>         [#1] scripts.base.frameworks.input.__raw.basic ... failed
>         [#1] scripts.base.frameworks.input.__reread ... failed
>         [#1] scripts.base.frameworks.input.__sqlite.basic ... failed
>         [#1] scripts.base.frameworks.input.__sqlite.error ... failed
>         [#1] scripts.base.frameworks.input.__sqlite.port ... failed
>         [#1] scripts.base.frameworks.input.__sqlite.types ... failed
>         [#1] scripts.base.frameworks.input.__tableevent ... failed
>         [#1] scripts.base.frameworks.intel.__cluster-transparency ... failed
>         [#1] scripts.base.frameworks.intel.__input-and-match ... failed
>         [#1] scripts.base.frameworks.intel.__read-file-dist-cluster ...
>         failed
>         [#1] scripts.base.frameworks.__logging.remote-types ... failed
>         [#1] scripts.base.frameworks.__logging.sqlite.error ... failed
>         [#1] scripts.base.frameworks.__logging.sqlite.types ... failed
>         [#1] scripts.base.frameworks.__logging.sqlite.wikipedia ... failed
>         [#1] scripts.base.frameworks.__notice.suppression-cluster ... failed
>         [#1] scripts.base.frameworks.__sumstats.basic-cluster ... failed
>         [#1]
>         scripts.base.frameworks.__sumstats.cluster-intermediate-__update
>         ... failed
>         [#1] scripts.base.frameworks.__sumstats.on-demand-cluster ... failed
>         [#1] scripts.base.frameworks.__sumstats.sample-cluster ... failed
>         [#1] scripts.base.frameworks.__sumstats.topk-cluster ... failed
>         [#1] scripts.base.protocols.http.__http-methods ... failed
>
>
>         On Tue, Oct 1, 2013 at 10:33 AM, Seth Hall <seth at icir.org
>         <mailto:seth at icir.org>
>         <mailto:seth at icir.org <mailto:seth at icir.org>>> wrote:
>
>
>              On Oct 1, 2013, at 10:37 AM, "Siwek, Jonathan Luke"
>              <jsiwek at illinois.edu <mailto:jsiwek at illinois.edu>
>         <mailto:jsiwek at illinois.edu <mailto:jsiwek at illinois.edu>>> wrote:
>
>               > Can you try the patch below?
>
>
>              That got past the problem for me.  If it ever finished
>         building I'll
>              run the test suite.
>
>                .Seth
>
>              --
>              Seth Hall
>              International Computer Science Institute
>              (Bro) because everyone has a network
>         http://www.bro.org/
>
>
>
>
>         _________________________________________________
>         Bro mailing list
>         bro at bro-ids.org <mailto:bro at bro-ids.org>
>         http://mailman.ICSI.Berkeley.__EDU/mailman/listinfo/bro
>         <http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro>
>
>
>



From oguzyarimtepe at gmail.com  Sat Oct  5 00:38:54 2013
From: oguzyarimtepe at gmail.com (=?UTF-8?B?T8SfdXogWWFyxLFtdGVwZQ==?=)
Date: Sat, 5 Oct 2013 10:38:54 +0300
Subject: [Bro] another kdd cup question
In-Reply-To: <CADju=b7RjOxdbJ_48CeaoiaH9qUuYU9+0Hejz0ivTVhenG+2sA@mail.gmail.com>
References: <CAKtWOTTPtRW-VP3WinFQos0=R8O9h9dYTMQOdPhsM=Snx+wAog@mail.gmail.com>
	<CADju=b7RjOxdbJ_48CeaoiaH9qUuYU9+0Hejz0ivTVhenG+2sA@mail.gmail.com>
Message-ID: <CAKtWOTQv6KM8BGheKP1jOZPCwUdLekE7Kp0XUgMd+G6xxrkkkw@mail.gmail.com>

On Fri, Oct 4, 2013 at 10:57 PM, Jim Mellander <jmellander at lbl.gov> wrote:

> Hi:
>
>
Hi,



> A number of your items (specifically # 10-22) appear to require inspection
> inside interactive sessions, which (unless the connection is cleartext), is
> not accessible to a network level monitor.  Lack of inspection into
> sessions, and the security benefits gained as a result are major benefits
> of modern session tools, of which the standard is ssh.
>
> If you have access to the systems you wish to monitor, you can install
> Instrumented SSHd, which will send a clear-text stream of the session to a
> bro monitor for inspection. See: https://code.google.com/p/auditing-sshd/
>
> Some of the information you want might also be logged via syslog, such as
> authentication events.
>
>
Very informative thank you. How about sum of bad checksum packets in a
connection and sum of urgent packets in a connections? Does Bro display the
packet based info as well or should i write some custom handlers for it?




-- 
O?uz Yar?mtepe
http://about.me/oguzy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131005/80103a99/attachment.html 

From Alec.Waters at dataline.co.uk  Sat Oct  5 08:54:54 2013
From: Alec.Waters at dataline.co.uk (Alec Waters)
Date: Sat, 5 Oct 2013 15:54:54 +0000
Subject: [Bro] Bro vs NetFlow
In-Reply-To: <20131004165407.4244C2C4002@rock.ICSI.Berkeley.EDU>
References: <20131004165407.4244C2C4002@rock.ICSI.Berkeley.EDU>
Message-ID: <8350146BADDCE04480B969B36967473D04008D4E@ZEUS.olympus.dataline.co.uk>

Hi Jay,

We use both NetFlow and Bro.

Each has its pros and cons of course, but from my point of view NetFlow is a "truer" picture of flow activity because it's a 100% layer3 export - things like malformed TCP streams sometimes won't get logged by apps that take a layer4-or-higher view of the world. It's also unidirectional - apps that that mung the A->B traffic together with the B->A traffic occasionally suffer from ambiguity of client/server identity, which makes report writing hard. Finally, NetFlow can be configured to prematurely export long-lived flows meaning that you can see how many bytes were transferred at different points in the flow, rather than just knowing that X bytes were transferred in total.

In my experience, having a two or more tools doing the "same" job in different ways isn't the galactic waste it may appear to be :)

alec
--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131005/009ceb2a/attachment.html 

From romig.1 at osu.edu  Sat Oct  5 15:00:33 2013
From: romig.1 at osu.edu (Romig, Steven)
Date: Sat, 5 Oct 2013 22:00:33 +0000
Subject: [Bro] Bro vs NetFlow
In-Reply-To: <8350146BADDCE04480B969B36967473D04008D4E@ZEUS.olympus.dataline.co.uk>
References: <20131004165407.4244C2C4002@rock.ICSI.Berkeley.EDU>,
	<8350146BADDCE04480B969B36967473D04008D4E@ZEUS.olympus.dataline.co.uk>
Message-ID: <B54A5749-788E-41E9-845D-9A2FE19F4780@osu.edu>

We use both, they're very complementary.  Detailed info from Bro, less so with netflow.  We collect netflow from our core routers and border fibers (using argus and indexing them live into Splunk), bro is just border and a few key places internally.  We're using snort also - why settle for less info when you can have more? :-)

--- Steve



From vmuthu at ucdavis.edu  Sat Oct  5 16:20:40 2013
From: vmuthu at ucdavis.edu (Vishak Muthukumar)
Date: Sat, 5 Oct 2013 16:20:40 -0700
Subject: [Bro] Bro - Broccoli Data Type Mismatch issue
Message-ID: <CAHOKaoQZaa1=_2iJ2utW=XMZG_J9Cx2kcf_Ce4uk0YfDxnK-nQ@mail.gmail.com>

Hi,

I am developing Bro scripts for reading modbus packets.
I send the packet details to broccoli as events.
When I send an event with parameters of type "addr" and "vector of
count" from the bro script to broccoli, the broccoli doesn't receive the
parameters correctly.
The code --

Bro script --


global pong: event(reg: vector of count, ipaddr: addr);

event modbus_read_holding_registers_response(c: connection, headers:
ModbusHeaders, registers: ModbusRegisters)
{

 event pong(ModbusRegisters,c$id$resp_h);

#ModbusRegisters - type vector of count
#c$id$resp_h type-addr

}



Broccoli Script --

static void bro_pong(BroConn *conn,void *data,BroRecord *rec,BroAddr
*address)
{

        int type = BRO_TYPE_COUNT;
        uint64 *register;
        if (! (seq = bro_record_get_nth_val(rec, 0, &type)))
        {
                printf("Error getting sequence count from event, got type
%i\n", type);
                return;
        }
        printf("pong event from seq=%"PRIu64"",*register);


        printf("\nAddress:");
        for(i=3;i>=0;i--)
        {
                printf("%"PRIu32".",address->addr[i]);
        }
        printf("\n");

}

Thanks in advance!

Vishak Muthukumar

Graduate Student

University of California, Davis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131005/a742afda/attachment.html 

From seth at icir.org  Mon Oct  7 05:35:21 2013
From: seth at icir.org (Seth Hall)
Date: Mon, 7 Oct 2013 08:35:21 -0400
Subject: [Bro] Bro - Broccoli Data Type Mismatch issue
In-Reply-To: <CAHOKaoQZaa1=_2iJ2utW=XMZG_J9Cx2kcf_Ce4uk0YfDxnK-nQ@mail.gmail.com>
References: <CAHOKaoQZaa1=_2iJ2utW=XMZG_J9Cx2kcf_Ce4uk0YfDxnK-nQ@mail.gmail.com>
Message-ID: <9E733DEE-4700-4A24-A2C4-B4C0E225B3FA@icir.org>


On Oct 5, 2013, at 7:20 PM, Vishak Muthukumar <vmuthu at ucdavis.edu> wrote:

> event modbus_read_holding_registers_response(c: connection, headers: ModbusHeaders, registers: ModbusRegisters)
> {
> 
>  event pong(ModbusRegisters,c$id$resp_h);

You are passing the type, you need to pass the value.

event pong(headers, c$id$resp_h);

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131007/727783bd/attachment.bin 

From seth at icir.org  Mon Oct  7 13:48:29 2013
From: seth at icir.org (Seth Hall)
Date: Mon, 7 Oct 2013 16:48:29 -0400
Subject: [Bro] Bro vs NetFlow
In-Reply-To: <8350146BADDCE04480B969B36967473D04008D4E@ZEUS.olympus.dataline.co.uk>
References: <20131004165407.4244C2C4002@rock.ICSI.Berkeley.EDU>
	<8350146BADDCE04480B969B36967473D04008D4E@ZEUS.olympus.dataline.co.uk>
Message-ID: <F191533B-205E-4102-AC62-B3B9331189FE@icir.org>


On Oct 5, 2013, at 11:54 AM, Alec Waters <Alec.Waters at dataline.co.uk> wrote:

> Each has its pros and cons of course, but from my point of view NetFlow is a ?truer? picture of flow activity because it?s a 100% layer3 export ? things like malformed TCP streams sometimes won?t get logged by apps that take a layer4-or-higher view of the world.

That's not exactly true.  Netflow is unidirectional layer-4 (except for newer bidirectional ipfix extensions).

> It?s also unidirectional ? apps that that mung the A->B traffic together with the B->A traffic occasionally suffer from ambiguity of client/server identity, which makes report writing hard.

I've thought about this for a long time and I think the conclusion that I reached is that all you are doing in that model is placing the analysis and understanding of the connections onto analysts.  When I did a lot of Netflow analysis I was constantly working to figure out the flows that matched in each direction.  The only difference in Bro is that it's doing that combining for you which would seem to make sense to me since a lot of people aren't aware of the crazy complexity you can have in trying to interpret netflow.  

Granted, Bro (and all other packet sniffers) will be wrong sometimes but you will have typically have confusing netflow in those cases of failure too.

> Finally, NetFlow can be configured to prematurely export long-lived flows meaning that you can see how many bytes were transferred at different points in the flow, rather than just knowing that X bytes were transferred in total.

You could do this in Bro too.  It would be pretty easy to write a Bro script to do a netflow-style log for connections every X minutes.  I just chose to avoid this model in Bro by default because of the number of times broken flows tripped me up in the past.

>  In my experience, having a two or more tools doing the ?same? job in different ways isn?t the galactic waste it may appear to be J

After all of my curmudgeonly grumbling, I totally agree with this. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131007/9796da6b/attachment.bin 

From seth at icir.org  Mon Oct  7 13:52:56 2013
From: seth at icir.org (Seth Hall)
Date: Mon, 7 Oct 2013 16:52:56 -0400
Subject: [Bro] Article mentioning Bro
Message-ID: <6F469748-D050-478A-A109-E777F214FCC0@icir.org>

The author obviously used the 2.2 beta in this one since it's using the file analysis framework.

http://blog.rootshell.be/2013/10/02/what-do-attendees-during-a-security-conference/

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131007/e03b5c80/attachment.bin 

From rjenkins at rmjconsulting.net  Mon Oct  7 15:47:57 2013
From: rjenkins at rmjconsulting.net (Ron Jenkins)
Date: Mon, 7 Oct 2013 22:47:57 +0000
Subject: [Bro] Article mentioning Bro
In-Reply-To: <6F469748-D050-478A-A109-E777F214FCC0@icir.org>
References: <6F469748-D050-478A-A109-E777F214FCC0@icir.org>
Message-ID: <AFC97506-FBE8-411E-8E07-832F3E3FA2FF@rmjconsulting.net>

Nice

Ron Jenkins (SnortCP,VCP 3 / 4,MCNE,MCPS,MCNPS,CCNA)
RMJ Consulting, LLC.
"Bringing Companies and Solutions Together"
Owner / Senior Architect
11715 Bricksome Ave STE B-7
Baton Rouge, LA. 70816
Toll. 855-448-5214
Direct. 225-448-5214
Fax. 225-448-5324
Cell. 225-931-1632
Email. rjenkins at rmjconsulting.net
Web. http://www.rmjconsulting.net<http://www.rmjconsulting.net/>
http://www.linkedin.com/in/ronmjenkins

> On Oct 7, 2013, at 3:54 PM, "Seth Hall" <seth at icir.org> wrote:
> 
> The author obviously used the 2.2 beta in this one since it's using the file analysis framework.
> 
> http://blog.rootshell.be/2013/10/02/what-do-attendees-during-a-security-conference/
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From jswan at sugf.com  Tue Oct  8 06:26:23 2013
From: jswan at sugf.com (Swan, Jay)
Date: Tue, 8 Oct 2013 13:26:23 +0000
Subject: [Bro] Bro vs NetFlow
In-Reply-To: <F191533B-205E-4102-AC62-B3B9331189FE@icir.org>
References: <20131004165407.4244C2C4002@rock.ICSI.Berkeley.EDU>
	<8350146BADDCE04480B969B36967473D04008D4E@ZEUS.olympus.dataline.co.uk>
	<F191533B-205E-4102-AC62-B3B9331189FE@icir.org>
Message-ID: <20131008132624.B8EEA2C4034@rock.ICSI.Berkeley.EDU>

I probably screwed up by titling the thread Bro *versus* NetFlow... I was mainly curious if anyone had managed to do away with NetFlow analysis through pervasive use of Bro. I didn't think that would likely be the case.

The best thing about our existing commercial NetFlow product is that it integrates into our existing commercial NMS system, so all the sysadmins and help desk people can see it if they want to. It has a lot of problems, however, with stuff like directionality, flow deduplication, and data summarization (it drops flow details to save space that are valuable for forensics).

I don't think I could get all the performance-related data I want just from Bro, at least at my current level of expertise. Forensically, I think Bro has almost everything I want, but putting sensors at every tiny location we have is logistically infeasible, but we already have IPFIX-exporting Cisco routers at all those locations. Hence, we're looking for a new (probably commercial) NetFlow product.

Jay

-----Original Message-----
From: Seth Hall [mailto:seth at icir.org] 
Sent: Monday, October 07, 2013 2:48 PM
To: Alec Waters
Cc: Swan, Jay; bro at bro.org
Subject: Re: [Bro] Bro vs NetFlow


On Oct 5, 2013, at 11:54 AM, Alec Waters <Alec.Waters at dataline.co.uk> wrote:

> Each has its pros and cons of course, but from my point of view NetFlow is a "truer" picture of flow activity because it's a 100% layer3 export - things like malformed TCP streams sometimes won't get logged by apps that take a layer4-or-higher view of the world.

That's not exactly true.  Netflow is unidirectional layer-4 (except for newer bidirectional ipfix extensions).

> It's also unidirectional - apps that that mung the A->B traffic together with the B->A traffic occasionally suffer from ambiguity of client/server identity, which makes report writing hard.

I've thought about this for a long time and I think the conclusion that I reached is that all you are doing in that model is placing the analysis and understanding of the connections onto analysts.  When I did a lot of Netflow analysis I was constantly working to figure out the flows that matched in each direction.  The only difference in Bro is that it's doing that combining for you which would seem to make sense to me since a lot of people aren't aware of the crazy complexity you can have in trying to interpret netflow.  

Granted, Bro (and all other packet sniffers) will be wrong sometimes but you will have typically have confusing netflow in those cases of failure too.

> Finally, NetFlow can be configured to prematurely export long-lived flows meaning that you can see how many bytes were transferred at different points in the flow, rather than just knowing that X bytes were transferred in total.

You could do this in Bro too.  It would be pretty easy to write a Bro script to do a netflow-style log for connections every X minutes.  I just chose to avoid this model in Bro by default because of the number of times broken flows tripped me up in the past.

>  In my experience, having a two or more tools doing the "same" job in different ways isn't the galactic waste it may appear to be J

After all of my curmudgeonly grumbling, I totally agree with this. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/




From slagell at illinois.edu  Tue Oct  8 06:36:44 2013
From: slagell at illinois.edu (Slagell, Adam J)
Date: Tue, 8 Oct 2013 13:36:44 +0000
Subject: [Bro] Bro vs NetFlow
In-Reply-To: <20131008132624.B8EEA2C4034@rock.ICSI.Berkeley.EDU>
References: <20131004165407.4244C2C4002@rock.ICSI.Berkeley.EDU>
	<8350146BADDCE04480B969B36967473D04008D4E@ZEUS.olympus.dataline.co.uk>
	<F191533B-205E-4102-AC62-B3B9331189FE@icir.org>
	<20131008132624.B8EEA2C4034@rock.ICSI.Berkeley.EDU>
Message-ID: <748C41D2-9208-4E19-835E-B7B039B9E92E@illinois.edu>

Even at the NCSA, with our strong use of Bro, we record and utilize netflows. Part of it is because we have tools and infrastructure in place to quickly search NetFlows. Part of it is because all our routers give us NetFlows, giving us some views of internal traffic that Bro sensors will never see.

It really isn't an either-or case for us.

On Oct 8, 2013, at 8:26 AM, "Swan, Jay" <jswan at sugf.com>
 wrote:

> I probably screwed up by titling the thread Bro *versus* NetFlow... I was mainly curious if anyone had managed to do away with NetFlow analysis through pervasive use of Bro. I didn't think that would likely be the case.

------

Adam J. Slagell
Chief Information Security Officer
Sr. Research Scientist
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info

"Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." 












From seth at icir.org  Tue Oct  8 06:40:36 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 8 Oct 2013 09:40:36 -0400
Subject: [Bro] Bro vs NetFlow
In-Reply-To: <20131008132624.B36442C400F@rock.ICSI.Berkeley.EDU>
References: <20131004165407.4244C2C4002@rock.ICSI.Berkeley.EDU>
	<8350146BADDCE04480B969B36967473D04008D4E@ZEUS.olympus.dataline.co.uk>
	<F191533B-205E-4102-AC62-B3B9331189FE@icir.org>
	<20131008132624.B36442C400F@rock.ICSI.Berkeley.EDU>
Message-ID: <737311EF-28C7-4D8D-BD41-2220446BA850@icir.org>


On Oct 8, 2013, at 9:26 AM, "Swan, Jay" <jswan at sugf.com> wrote:

> I probably screwed up by titling the thread Bro *versus* NetFlow... I was mainly curious if anyone had managed to do away with NetFlow analysis through pervasive use of Bro. I didn't think that would likely be the case.

I've been meaning to write some scripts for Bro to start doing netflow logging and analysis for a long time but I still haven't done it.

> I don't think I could get all the performance-related data I want just from Bro, at least at my current level of expertise. Forensically, I think Bro has almost everything I want, but putting sensors at every tiny location we have is logistically infeasible

Yep, I think this is the major win with netflow.  It's pervasive deployment certainly has a lot of benefits.

> Hence, we're looking for a new (probably commercial) NetFlow product.


Yeah, Bro is definitely not in a position right now to fill that need for you.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131008/1da74725/attachment.bin 

From vladg at cmu.edu  Tue Oct  8 07:08:28 2013
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Tue, 8 Oct 2013 14:08:28 +0000
Subject: [Bro] Bro vs NetFlow
In-Reply-To: <6328_1381238806_r98DQif5006590_20131008132624.B8EEA2C4034@rock.ICSI.Berkeley.EDU>
References: <20131004165407.4244C2C4002@rock.ICSI.Berkeley.EDU>
	<8350146BADDCE04480B969B36967473D04008D4E@ZEUS.olympus.dataline.co.uk>
	<F191533B-205E-4102-AC62-B3B9331189FE@icir.org>
	<6328_1381238806_r98DQif5006590_20131008132624.B8EEA2C4034@rock.ICSI.Berkeley.EDU>
Message-ID: <1202BE242E080642B0CD0AD0A03E8552E71C83@PGH-MSGMB-03.andrew.ad.cmu.edu>


On Oct 8, 2013, at 9:26 AM, Swan, Jay <jswan at sugf.com> wrote:

> I was mainly curious if anyone had managed to do away with NetFlow analysis through pervasive use of Bro. I didn't think that would likely be the case.

Carnegie Mellon has. We used Netflow and Argus previously, but have replaced them with Bro. We do plan to deploy Time Machine[1] as well. While this isn't duplicating tools, having full PCAPs available complements Bro well.

  --Vlad Grigorescu
    Information Security Office
    Carnegie Mellon University


[1] - <http://www.bro.org/community/time-machine.html>





From brendan-dalpe at utulsa.edu  Tue Oct  8 14:16:11 2013
From: brendan-dalpe at utulsa.edu (Brendan Dalpe)
Date: Tue, 8 Oct 2013 16:16:11 -0500
Subject: [Bro] Duplicate log entries for events
Message-ID: <CAPkA_r20yvTYjz67TvR0o91mqO6oFfajCf57Pc8zvaxYdpeFxg@mail.gmail.com>

I'm running into an issue where I'm seeing multiple entries in my logs for
a single event which is duplicated 4 times. The entries are all identical
except for the UID. I'm running 4 worker processes on the server monitoring
one interface.

node.cfg:

[manager]
type=manager
host=10.1.26.22

[proxy]
type=proxy
host=10.1.26.22

[bro-eth0]
type=worker
host=10.1.26.22
interface=eth0
lb_method=pf_ring
lb_procs=4


Any thoughts?

Thanks in advance,

Brendan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131008/02221f84/attachment.html 

From seth at icir.org  Wed Oct  9 09:35:00 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 9 Oct 2013 12:35:00 -0400
Subject: [Bro] Duplicate log entries for events
In-Reply-To: <CAPkA_r20yvTYjz67TvR0o91mqO6oFfajCf57Pc8zvaxYdpeFxg@mail.gmail.com>
References: <CAPkA_r20yvTYjz67TvR0o91mqO6oFfajCf57Pc8zvaxYdpeFxg@mail.gmail.com>
Message-ID: <7ADBCBF9-F2D9-4C60-BDCA-01EA819787FC@icir.org>


On Oct 8, 2013, at 5:16 PM, Brendan Dalpe <brendan-dalpe at utulsa.edu> wrote:

> [bro-eth0]
> type=worker
> host=10.1.26.22
> interface=eth0
> lb_method=pf_ring
> lb_procs=4
> 
> 
> Any thoughts?

It sounds like something isn't installed correctly.  Did you successfully build Bro against the pf_ring libpcap wrapper?  Your traffic isn't load balancing and each worker is getting the full stream.

Maybe you could show us your configure command?  You can see exactly what you did if you go to your source and look at build/config.status

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131009/2a615b19/attachment.bin 

From seth at icir.org  Wed Oct  9 09:50:14 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 9 Oct 2013 12:50:14 -0400
Subject: [Bro] Bro - Broccoli Data Type Mismatch issue
In-Reply-To: <CAHOKaoTRLJTnt0Ro4g6AR3tE49YAtLDXH3f34SMUdmvxdJVzgw@mail.gmail.com>
References: <CAHOKaoQZaa1=_2iJ2utW=XMZG_J9Cx2kcf_Ce4uk0YfDxnK-nQ@mail.gmail.com>
	<9E733DEE-4700-4A24-A2C4-B4C0E225B3FA@icir.org>
	<CAHOKaoTRLJTnt0Ro4g6AR3tE49YAtLDXH3f34SMUdmvxdJVzgw@mail.gmail.com>
Message-ID: <67C2E8A4-2066-46A1-8AC1-37A2AA07DB7D@icir.org>


On Oct 7, 2013, at 1:45 PM, Vishak Muthukumar <vishakm92 at gmail.com> wrote:

> global pong: event(reg: vector of count, ipaddr: addr);
<snip>
> static void bro_pong(BroConn *conn,void *data,BroRecord *rec,BroAddr *address)

I'm not completely sure that Vectors are supported as values in broccoli.  From my quick glance it looks like they're an unsupported but you can't just treat them like a record like you are doing.  There may be other problems, but I noticed that immediately.  You also aren't showing us any code where you are subscribing to events with broccoli so I can't even be sure you are doing that correctly.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131009/56ae4bb1/attachment.bin 

From alexwis at gmail.com  Wed Oct  9 12:50:26 2013
From: alexwis at gmail.com (Alex Waher)
Date: Wed, 9 Oct 2013 12:50:26 -0700
Subject: [Bro] Duplicate log entries for events
In-Reply-To: <7ADBCBF9-F2D9-4C60-BDCA-01EA819787FC@icir.org>
References: <CAPkA_r20yvTYjz67TvR0o91mqO6oFfajCf57Pc8zvaxYdpeFxg@mail.gmail.com>
	<7ADBCBF9-F2D9-4C60-BDCA-01EA819787FC@icir.org>
Message-ID: <CAGURB2czmqc6B814CLtKam1+76opFiqwC34bj5sVk5Z5vTrdFw@mail.gmail.com>

Make sure `lsmod` shows that the pf_ring module is loaded. If its not
loaded, `modprobe pf_ring`
Or verify the eth0 interface is running with pf_ring by checking
/proc/net/pf_ring/dev/eth0/info


On Wed, Oct 9, 2013 at 9:35 AM, Seth Hall <seth at icir.org> wrote:

>
> On Oct 8, 2013, at 5:16 PM, Brendan Dalpe <brendan-dalpe at utulsa.edu>
> wrote:
>
> > [bro-eth0]
> > type=worker
> > host=10.1.26.22
> > interface=eth0
> > lb_method=pf_ring
> > lb_procs=4
> >
> >
> > Any thoughts?
>
> It sounds like something isn't installed correctly.  Did you successfully
> build Bro against the pf_ring libpcap wrapper?  Your traffic isn't load
> balancing and each worker is getting the full stream.
>
> Maybe you could show us your configure command?  You can see exactly what
> you did if you go to your source and look at build/config.status
>
>   .Seth
>
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131009/42ee248e/attachment.html 

From jamesfhook at gmail.com  Wed Oct  9 13:31:24 2013
From: jamesfhook at gmail.com (James Hook)
Date: Wed, 9 Oct 2013 21:31:24 +0100
Subject: [Bro] Implementing broping.c in broscript
Message-ID: <CAC+_1SZgypEb+9DGZPR0_CoypOo=zZmfX5HZUYyZbVPtYDBy+Q@mail.gmail.com>

Hi,

I have loads of questions about communicating between bro instances. I have
been "playing" with some of the examples from the the Broccoli 1.93
download. I'm wondering if it's possible to implement the broping.c example
entirely within broscript.

I have successfully built and tested the broping example, which comprises
of C code connecting to a broscript listening on TCP port 47758. I have
been looking for examples of broscripts connecting to broscripts (or
listening broccoli applications)

I have tried adding to the connection::nodes table

redef Communication::nodes += {
["broping"] = [$host = 127.0.0.1, $events = /pong|ping/, $connect=T, $ssl=F]
};

as well as manually creating a connection in bro_init().  I haven't been
able to send the "ping" events from the bro script to broping.bro.

I guess i'm asking is whether it is possible (or even whether it is a good
idea) to implement broping.c in broscript.

Thanks in advance for any help

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131009/e3eb1001/attachment.html 

From seth at icir.org  Wed Oct  9 13:55:17 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 9 Oct 2013 16:55:17 -0400
Subject: [Bro] Implementing broping.c in broscript
In-Reply-To: <CAC+_1SZgypEb+9DGZPR0_CoypOo=zZmfX5HZUYyZbVPtYDBy+Q@mail.gmail.com>
References: <CAC+_1SZgypEb+9DGZPR0_CoypOo=zZmfX5HZUYyZbVPtYDBy+Q@mail.gmail.com>
Message-ID: <3BBB351C-4AB9-4A30-996E-D48A01E6F5E0@icir.org>


On Oct 9, 2013, at 4:31 PM, James Hook <jamesfhook at gmail.com> wrote:

> I guess i'm asking is whether it is possible (or even whether it is a good idea) to implement broping.c in broscript.

You can definitely do that.  It would just be sending events between multiple Bro processes which is very regularly done in some of the frameworks we ship.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131009/bc384f09/attachment.bin 

From jamesfhook at gmail.com  Thu Oct 10 02:11:13 2013
From: jamesfhook at gmail.com (James Hook)
Date: Thu, 10 Oct 2013 10:11:13 +0100
Subject: [Bro] Implementing broping.c in broscript
In-Reply-To: <3BBB351C-4AB9-4A30-996E-D48A01E6F5E0@icir.org>
References: <CAC+_1SZgypEb+9DGZPR0_CoypOo=zZmfX5HZUYyZbVPtYDBy+Q@mail.gmail.com>
	<3BBB351C-4AB9-4A30-996E-D48A01E6F5E0@icir.org>
Message-ID: <CAC+_1Sbfm3JmP4XvJyAyxRb4rPcTEWN4tR9Bq8UGktPb6ctVrA@mail.gmail.com>

Thanks Seth, i knew it had to be possible.
I'm struggling to get it working. Do you have any examples in the
frameworks where this is done, or can you pick out anything missing from
what i've already tried.
Thanks in advance for any help you can give with this.
James

=================
broping_clt.bro (starting using "bro broping_clt.bro" after "bro
broping.bro" has been started in another terminal)
"broping.bro" is as in the broccoli 1.93 source.
i am wanting this script to send a ping event to "broping" and for it to
handle a pong event coming back (as broping.c does)
=================

global pong: event(src_time: time, dst_time: time, seq: count);
global ping: event(src_time: time, seq: count);

redef Communication::nodes += {
    ["broping2"] = [$host = 127.0.0.1, $p = 47758/tcp, $events = /pong/,
$connect=T, $ssl=F]
};

event bro_init()
        {
        #local w : count = connect(
        #    127.0.0.1, "",    47758/tcp,    "",    5secs,     F    );
        #request_remote_events(w,/pong/);

        event ping(current_time(), 1);
        }


event pong(src_time: time, dst_time: time, seq: count)
        {
        print "pong!!!";
        }






On 9 October 2013 21:55, Seth Hall <seth at icir.org> wrote:

>
> On Oct 9, 2013, at 4:31 PM, James Hook <jamesfhook at gmail.com> wrote:
>
> > I guess i'm asking is whether it is possible (or even whether it is a
> good idea) to implement broping.c in broscript.
>
> You can definitely do that.  It would just be sending events between
> multiple Bro processes which is very regularly done in some of the
> frameworks we ship.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131010/0c36b886/attachment.html 

From jason.trost at gmail.com  Thu Oct 10 03:50:21 2013
From: jason.trost at gmail.com (Jason Trost)
Date: Thu, 10 Oct 2013 06:50:21 -0400
Subject: [Bro] Correlate DNS request/response with TCP/UDP connections
Message-ID: <CAPsZEyTLLRXgHsxGSpwYM5gf+y-+O7sbMYU+Smo4tgwDk2VQ4w@mail.gmail.com>

Is there a good way to correlate DNS requests/responses with subsequent
TCP/UDP connections using Bro (in realtime)?  With how my tap is configured
I can see the client's DNS request/response and all their traffic.  I want
to be able to combine their DNS request (if there is one) with the
corresponding TCP/UDP following it.  For my use case I need this to be done
in realtime (not later by simply doing a JOIN).  So, I am interested in a
single log entry with DNS request/response AND connection info.

It seems like this should be possible by basically doing the following:

    dns_response.dst_ip == conn.src_ip AND
    conn.dst_ip == dns_response.answer_ip AND
    (conn.timestamp - dns_response.timestamp) < THRESHOLD

Has anyone done this?  Any guidance would be greatly appreciated.

Thanks,

--Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131010/4e7aadd5/attachment.html 

From seth at icir.org  Thu Oct 10 05:16:18 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 10 Oct 2013 08:16:18 -0400
Subject: [Bro] Implementing broping.c in broscript
In-Reply-To: <CAC+_1Sbfm3JmP4XvJyAyxRb4rPcTEWN4tR9Bq8UGktPb6ctVrA@mail.gmail.com>
References: <CAC+_1SZgypEb+9DGZPR0_CoypOo=zZmfX5HZUYyZbVPtYDBy+Q@mail.gmail.com>
	<3BBB351C-4AB9-4A30-996E-D48A01E6F5E0@icir.org>
	<CAC+_1Sbfm3JmP4XvJyAyxRb4rPcTEWN4tR9Bq8UGktPb6ctVrA@mail.gmail.com>
Message-ID: <9326941E-00F7-4BC9-8440-FF1F83D7F2C4@icir.org>


On Oct 10, 2013, at 5:11 AM, James Hook <jamesfhook at gmail.com> wrote:

> redef Communication::nodes += {
>     ["broping2"] = [$host = 127.0.0.1, $p = 47758/tcp, $events = /pong/, $connect=T, $ssl=F]

You need to subscribe to the ping event, not the pong event.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131010/ab9a84f9/attachment.bin 

From seth at icir.org  Thu Oct 10 05:33:23 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 10 Oct 2013 08:33:23 -0400
Subject: [Bro] Correlate DNS request/response with TCP/UDP connections
In-Reply-To: <CAPsZEyTLLRXgHsxGSpwYM5gf+y-+O7sbMYU+Smo4tgwDk2VQ4w@mail.gmail.com>
References: <CAPsZEyTLLRXgHsxGSpwYM5gf+y-+O7sbMYU+Smo4tgwDk2VQ4w@mail.gmail.com>
Message-ID: <36E85758-6E72-4CF7-9604-A1BDC32C533C@icir.org>


On Oct 10, 2013, at 6:50 AM, Jason Trost <jason.trost at gmail.com> wrote:

>    dns_response.dst_ip == conn.src_ip AND 
>     conn.dst_ip == dns_response.answer_ip AND 
>     (conn.timestamp - dns_response.timestamp) < THRESHOLD
> 
> Has anyone done this?  Any guidance would be greatly appreciated.


Are you running a cluster?  This type of problem is one of the hardest to solve on distributed analysis.  

Otherwise if you are running a single node then it should be fairly easy.  I *think* you would essentially want to create a 2-tuple set with a short timeout.

global watch_for_connections: set[addr, addr] = {} &create_timeout=2secs;

You would fill out that set in one or more DNS event handlers and then check to see if any connections are being made in a connection_established handler.

Again though, if you are running a cluster this is a really hard problem.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131010/4e40f208/attachment.bin 

From jamesfhook at gmail.com  Thu Oct 10 09:59:05 2013
From: jamesfhook at gmail.com (James Hook)
Date: Thu, 10 Oct 2013 17:59:05 +0100
Subject: [Bro] Implementing broping.c in broscript
In-Reply-To: <9326941E-00F7-4BC9-8440-FF1F83D7F2C4@icir.org>
References: <CAC+_1SZgypEb+9DGZPR0_CoypOo=zZmfX5HZUYyZbVPtYDBy+Q@mail.gmail.com>
	<3BBB351C-4AB9-4A30-996E-D48A01E6F5E0@icir.org>
	<CAC+_1Sbfm3JmP4XvJyAyxRb4rPcTEWN4tR9Bq8UGktPb6ctVrA@mail.gmail.com>
	<9326941E-00F7-4BC9-8440-FF1F83D7F2C4@icir.org>
Message-ID: <CAC+_1Sb21PE7OyOOASghv3q6+9p+rOqjs==p=+BjX0HYSP3mqw@mail.gmail.com>

Hi Seth,
Sorry to keep pestering you, i'm still having trouble getting this to work.
I have wireshark'd it and can see the connection being made, but when
comparing it with a wireshark of the compiled c code, i'm missing the
initial ping event going across the connection.

The  C code:
- uses bro_conn_new_str to get a new connection handle
- uses bro_event_registry_add to add the "pong" event to this connection
- uses bro_conn_connect to establish the connection

Would i be right in thinking that all you need to do in bro script is add
the following entry to the Communication::nodes table?

redef Communication::nodes += {
     ["broping2"] = [$host = 127.0.0.1, $p = 47758/tcp, $events = /pong/,
$connect=T, $ssl=F]
};

I have tried both ping and pong (and both) in this entry without any luck -
the C code sends ping events down to the bro instance and is listening for
pong events coming back.

The c code sends the ping event using "bro_event_send( connection, event);"
on line 442 of broping.c.

Again, sorry for my lack of understanding here - hope you can help :-)

James




On 10 October 2013 13:16, Seth Hall <seth at icir.org> wrote:

>
> On Oct 10, 2013, at 5:11 AM, James Hook <jamesfhook at gmail.com> wrote:
>
> > redef Communication::nodes += {
> >     ["broping2"] = [$host = 127.0.0.1, $p = 47758/tcp, $events = /pong/,
> $connect=T, $ssl=F]
>
> You need to subscribe to the ping event, not the pong event.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131010/5c54a103/attachment.html 

From vmuthu at ucdavis.edu  Thu Oct 10 10:20:02 2013
From: vmuthu at ucdavis.edu (Vishak Muthukumar)
Date: Thu, 10 Oct 2013 10:20:02 -0700
Subject: [Bro] Bro - Broccoli Data Type Mismatch issue
In-Reply-To: <CAHOKaoTQwgJx4Qq0DugvpYMQiYaZkp9RKFEYzBqSRw8Qbdxd=A@mail.gmail.com>
References: <CAHOKaoQZaa1=_2iJ2utW=XMZG_J9Cx2kcf_Ce4uk0YfDxnK-nQ@mail.gmail.com>
	<9E733DEE-4700-4A24-A2C4-B4C0E225B3FA@icir.org>
	<CAHOKaoTRLJTnt0Ro4g6AR3tE49YAtLDXH3f34SMUdmvxdJVzgw@mail.gmail.com>
	<67C2E8A4-2066-46A1-8AC1-37A2AA07DB7D@icir.org>
	<CAHOKaoTQwgJx4Qq0DugvpYMQiYaZkp9RKFEYzBqSRw8Qbdxd=A@mail.gmail.com>
Message-ID: <CAHOKaoSWX-gs0DZ5qV752h5cbHVHtk410PGr5FKk=EF06WJOjg@mail.gmail.com>

The issue is solved,
I needed to do typecasting to receive the correct IP address -

The broccoli script is --

static void bro_pong(BroConn *conn,void *data,BroAddr *address)
{

 char addr[INET6_ADDRSTRLEN];
 inet_ntop(AF_INET, address->addr + 3, addr, INET6_ADDRSTRLEN);
 printf("Received bro_address%s\n", addr);
}

Thanks for your help.

Vishak


On Wed, Oct 9, 2013 at 11:11 AM, Vishak Muthukumar <vishakm92 at gmail.com>wrote:

> I understand that the datatype vector is not supported now. But I can't
> even send IP address from Bro to broccoli.
>
> This is the brocooli script --
>
> static void bro_pong(void *data,BroConn *conn,BroAddr *address)
> {
>
> int i;
>         printf("\nAddress:");
>         for(i=3;i>=0;i--)
>         {
>                 printf("%"PRIu32"",address->addr[i]);
>         }
>         printf("\n");
>
>
> }
>
>
> int main(int argc, char **argv)
> {
>         BroConn *bc;
>         extern char *optarg;
>         extern int optind;
>         char hostname[512];
>         int fd = -1;
>
>         bro_init(NULL);
>
>         bro_debug_calltrace = 0;
>         bro_debug_messages  = 0;
>
>
>         snprintf(hostname, 512, "%s:%s", host_str, port_str);
>
>         if (! (bc = bro_conn_new_str(hostname, BRO_CFLAG_RECONNECT |
> BRO_CFLAG_ALWAYS_QUEUE)))
>         {
>
>                 printf("Could not get Bro connection handle.\n");
>                 exit(-1);
>         }
>
>
>
>         void *data;
>         data=malloc(300);
>
>         bro_event_registry_add(bc, "pong", (BroEventFunc) bro_pong,data);
>         bro_event_registry_request (bc);
>
>
>         if (! bro_conn_connect(bc))
>         {
>                 printf("Could not connect to Bro at %s:%s.\n", host_str,
> port_str);
>                 exit(-1);
>         }
>
>         for ( ; ; )
>         {
>                 BroEvent *ev;
>
>                 bro_conn_process_input(bc);
>
> #ifdef __MINGW32__
>                 sleep(1000);
> #else
>                 sleep(1);
> #endif
>         }
>
>         /* Disconnect from Bro and release state. */
>         bro_conn_delete(bc);
>         return 0;
> }
>
> The correseponding BRO script --
>
> module State;
> @load base/protocols/modbus
>
>
> export {
> # Create an ID for our new stream. By convention, this is
> # called "LOG".
>         redef enum Log::ID += { LOG };
> }
>
> # Let's make sure we use the same port no matter whether we use encryption
> or not:
> redef Communication::listen_port = 47758/tcp;
>
> # Redef this to global flag: count=0;T if you want to use SSL.
> redef Communication::listen_ssl = F;
>
> global dstg:addr;
> global pong: event(address: addr);
>
> redef Communication::nodes += {
>         ["broping"] = [$host = 127.0.0.1, $events = /ping/, $connect=F,
> $ssl=F]
> };
>
> event modbus_read_holding_registers_response(c: connection, headers:
> ModbusHeaders, registers: ModbusRegisters)
> {
>         dstg=c$id$resp_h;
>         print "IP-",dstg;
>         event pong(dstg);
> }
>
> In the Bro script I receive the IP address displayed in the console.
> But in the broccoli, I see junk values.
>
> Hope I made myself clear this time.
>
> Thanks
>
> Vishak
>
>
> On Wed, Oct 9, 2013 at 9:50 AM, Seth Hall <seth at icir.org> wrote:
>
>>
>> On Oct 7, 2013, at 1:45 PM, Vishak Muthukumar <vishakm92 at gmail.com>
>> wrote:
>>
>> > global pong: event(reg: vector of count, ipaddr: addr);
>> <snip>
>> > static void bro_pong(BroConn *conn,void *data,BroRecord *rec,BroAddr
>> *address)
>>
>> I'm not completely sure that Vectors are supported as values in broccoli.
>>  From my quick glance it looks like they're an unsupported but you can't
>> just treat them like a record like you are doing.  There may be other
>> problems, but I noticed that immediately.  You also aren't showing us any
>> code where you are subscribing to events with broccoli so I can't even be
>> sure you are doing that correctly.
>>
>>   .Seth
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>>
>>
>
>
> --
>
> Vishak Muthukumar
>
> Graduate Student
>
> University of California, Davis
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131010/52fa0427/attachment.html 

From himself at louruppert.com  Thu Oct 10 10:35:52 2013
From: himself at louruppert.com (Lou RUPPERT)
Date: Thu, 10 Oct 2013 13:35:52 -0400
Subject: [Bro] Cluster input framework in 2.2 beta
Message-ID: <5256E578.60903@louruppert.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey,

My bro_tor scripts didn't seem to survive the transition to 2.2 beta.
My guess is that the worker nodes are limited in which files they are
permitted to access somehow. Here's the error message that causes the
worker processes to crash:

1381424606.274441 fatal error in
/var/spool/bro/installed-scripts-do-not-touch/site/bro_tor/././check.bro,
lines 15-16: Failed to read tor hostlist from
/usr/local/bro/site/tor_servers.txt

Any ideas what I'm doing wrong or how to fix it? It seems a crash is a
bit harsh of a reaction to something like that.

The code is here:https://github.com/lruppert/bro-tor


- -- 
I prefer encrypted email.  Get my key here:
http://www.louruppert.com/keys/115DCF62.asc
PGP Fingerprint: 3261 B9F9 9363 D512 56F8  12DD 127F 4D6A 115D CF62
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Icedove - http://www.enigmail.net/

iEYEARECAAYFAlJW5XIACgkQEn9NahFdz2JraQCg2VbjTOzsmKdvJ5dbb8Tyh6yn
DQEAniJZ4Vyax4ZMwGQQm8WEEbYVd+CY
=ysDZ
-----END PGP SIGNATURE-----


From seth at icir.org  Thu Oct 10 11:05:13 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 10 Oct 2013 14:05:13 -0400
Subject: [Bro] Implementing broping.c in broscript
In-Reply-To: <CAC+_1Sb21PE7OyOOASghv3q6+9p+rOqjs==p=+BjX0HYSP3mqw@mail.gmail.com>
References: <CAC+_1SZgypEb+9DGZPR0_CoypOo=zZmfX5HZUYyZbVPtYDBy+Q@mail.gmail.com>
	<3BBB351C-4AB9-4A30-996E-D48A01E6F5E0@icir.org>
	<CAC+_1Sbfm3JmP4XvJyAyxRb4rPcTEWN4tR9Bq8UGktPb6ctVrA@mail.gmail.com>
	<9326941E-00F7-4BC9-8440-FF1F83D7F2C4@icir.org>
	<CAC+_1Sb21PE7OyOOASghv3q6+9p+rOqjs==p=+BjX0HYSP3mqw@mail.gmail.com>
Message-ID: <7BCB0116-22B3-44A8-A033-6232E359F698@icir.org>


On Oct 10, 2013, at 12:59 PM, James Hook <jamesfhook at gmail.com> wrote:

> redef Communication::nodes += {
>      ["broping2"] = [$host = 127.0.0.1, $p = 47758/tcp, $events = /pong/, $connect=T, $ssl=F]

It sounds like you want your Bro node to subscribe to ping events since that's what your broccoli application is sending.  If you have it listening for pong, the C application might send the ping event, but Bro won't listen to it since it's only listening for pong events.

You could even make Bro listen for all events like this (not recommended for anything other than testing)?

>      ["broping2"] = [$host = 127.0.0.1, $p = 47758/tcp, $events = /.*/, $connect=T, $ssl=F]


  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131010/87c48377/attachment.bin 

From seth at icir.org  Thu Oct 10 11:38:24 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 10 Oct 2013 14:38:24 -0400
Subject: [Bro] Cluster input framework in 2.2 beta
In-Reply-To: <5256E578.60903@louruppert.com>
References: <5256E578.60903@louruppert.com>
Message-ID: <E597A907-D7DC-469B-9AF2-3FDD6AF37D5A@icir.org>


On Oct 10, 2013, at 1:35 PM, Lou RUPPERT <himself at louruppert.com> wrote:

> Any ideas what I'm doing wrong or how to fix it? It seems a crash is a
> bit harsh of a reaction to something like that.

You requested it to shutdown with the call to Reporter::fatal. :)

I sent you a pull request:
	https://github.com/lruppert/bro-tor/pull/1

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131010/7210112d/attachment.bin 

From jamesfhook at gmail.com  Thu Oct 10 14:01:28 2013
From: jamesfhook at gmail.com (James Hook)
Date: Thu, 10 Oct 2013 22:01:28 +0100
Subject: [Bro] Implementing broping.c in broscript
In-Reply-To: <7BCB0116-22B3-44A8-A033-6232E359F698@icir.org>
References: <CAC+_1SZgypEb+9DGZPR0_CoypOo=zZmfX5HZUYyZbVPtYDBy+Q@mail.gmail.com>
	<3BBB351C-4AB9-4A30-996E-D48A01E6F5E0@icir.org>
	<CAC+_1Sbfm3JmP4XvJyAyxRb4rPcTEWN4tR9Bq8UGktPb6ctVrA@mail.gmail.com>
	<9326941E-00F7-4BC9-8440-FF1F83D7F2C4@icir.org>
	<CAC+_1Sb21PE7OyOOASghv3q6+9p+rOqjs==p=+BjX0HYSP3mqw@mail.gmail.com>
	<7BCB0116-22B3-44A8-A033-6232E359F698@icir.org>
Message-ID: <CAC+_1Sbe6NfWuxG_Gv15RC+wvnPbus-y6as1PersjHCDVUpqcg@mail.gmail.com>

Hi Seth,

I think I have got to the bottom of why I was getting the events going down
the connection. I think there is one (maybe two) reasons:

1) the event_peer class (not sure about this?)
in the broping.bro script I dumped the event_peer record in the
remote_event_registered handler.
from the C program it shows as "class=<uninitialized>"
from the bro script it comes through as "class="
I'm not sure whether this has any bearing on it, or whether "" and
"<uninitialized>" are treated as equals. Using bro_conn_set_class() from

2) firing events in "bro_init"
I rewrote bro_init() like this

event bro_init()
{
  event ping(current_time(),1);
  schedule 1 secs
     {
         ping(current_time(),1)
     };
}

i'm only seeing the event get fired once (the scheduled event). Could it be
that the event firing plumbing isn't fully initialised before bro_init is
called?

Anyway, it seems to be working now. Thanks for your help
James



On 10 October 2013 19:05, Seth Hall <seth at icir.org> wrote:

>
> On Oct 10, 2013, at 12:59 PM, James Hook <jamesfhook at gmail.com> wrote:
>
> > redef Communication::nodes += {
> >      ["broping2"] = [$host = 127.0.0.1, $p = 47758/tcp, $events =
> /pong/, $connect=T, $ssl=F]
>
> It sounds like you want your Bro node to subscribe to ping events since
> that's what your broccoli application is sending.  If you have it listening
> for pong, the C application might send the ping event, but Bro won't listen
> to it since it's only listening for pong events.
>
> You could even make Bro listen for all events like this (not recommended
> for anything other than testing)?
>
> >      ["broping2"] = [$host = 127.0.0.1, $p = 47758/tcp, $events = /.*/,
> $connect=T, $ssl=F]
>
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131010/b6012b66/attachment.html 

From anthony.kasza at gmail.com  Thu Oct 10 21:09:13 2013
From: anthony.kasza at gmail.com (anthony kasza)
Date: Thu, 10 Oct 2013 21:09:13 -0700
Subject: [Bro] Correlate DNS request/response with TCP/UDP connections
In-Reply-To: <36E85758-6E72-4CF7-9604-A1BDC32C533C@icir.org>
References: <CAPsZEyTLLRXgHsxGSpwYM5gf+y-+O7sbMYU+Smo4tgwDk2VQ4w@mail.gmail.com>
	<36E85758-6E72-4CF7-9604-A1BDC32C533C@icir.org>
Message-ID: <CAEZw2byoEhQtRTfBrXtFw=ubDzd9yhdRc7PQYEamOqD8Ory+qQ@mail.gmail.com>

I think Seth meant &create_expire. But yeah, this is totally possible if
you're running a single node/standalone mode. You could also use something
like

type DNSqr record {
query: string;
asking_host: addr;
};
dns_query_cache: table[addr] of DNSqr &create_expire=2secs;

where the dns_query_cache is indexed by answer addresses (the a value in
dns_A_reply events).

-AK


On Thu, Oct 10, 2013 at 5:33 AM, Seth Hall <seth at icir.org> wrote:

>
> On Oct 10, 2013, at 6:50 AM, Jason Trost <jason.trost at gmail.com> wrote:
>
> >    dns_response.dst_ip == conn.src_ip AND
> >     conn.dst_ip == dns_response.answer_ip AND
> >     (conn.timestamp - dns_response.timestamp) < THRESHOLD
> >
> > Has anyone done this?  Any guidance would be greatly appreciated.
>
>
> Are you running a cluster?  This type of problem is one of the hardest to
> solve on distributed analysis.
>
> Otherwise if you are running a single node then it should be fairly easy.
>  I *think* you would essentially want to create a 2-tuple set with a short
> timeout.
>
> global watch_for_connections: set[addr, addr] = {} &create_timeout=2secs;
>
> You would fill out that set in one or more DNS event handlers and then
> check to see if any connections are being made in a connection_established
> handler.
>
> Again though, if you are running a cluster this is a really hard problem.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



On Thu, Oct 10, 2013 at 5:33 AM, Seth Hall <seth at icir.org> wrote:

>
> On Oct 10, 2013, at 6:50 AM, Jason Trost <jason.trost at gmail.com> wrote:
>
> >    dns_response.dst_ip == conn.src_ip AND
> >     conn.dst_ip == dns_response.answer_ip AND
> >     (conn.timestamp - dns_response.timestamp) < THRESHOLD
> >
> > Has anyone done this?  Any guidance would be greatly appreciated.
>
>
> Are you running a cluster?  This type of problem is one of the hardest to
> solve on distributed analysis.
>
> Otherwise if you are running a single node then it should be fairly easy.
>  I *think* you would essentially want to create a 2-tuple set with a short
> timeout.
>
> global watch_for_connections: set[addr, addr] = {} &create_timeout=2secs;
>
> You would fill out that set in one or more DNS event handlers and then
> check to see if any connections are being made in a connection_established
> handler.
>
> Again though, if you are running a cluster this is a really hard problem.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131010/c970cfbe/attachment.html 

From michal at rsbac.org  Mon Oct 14 04:03:18 2013
From: michal at rsbac.org (Michal Purzynski)
Date: Mon, 14 Oct 2013 13:03:18 +0200
Subject: [Bro] Yet Another Conference - like no other :)
Message-ID: <525BCF76.6090908@rsbac.org>

Yes. I'm from Mozilla. Now you know :)

Slides from my recent talk about NSM @ Mozilla at YaC 2013 are here, a 
full video will hopefully follow.

http://tech.yandex.ru/events/yac/2013/talks/1131/

Happy to answer questions, share experience, etc. Note it's an emerging 
project and I expect new servers to be shipped this week, so far using 
whatever-I-could-find in terms of CPU, but the rest matches.


From jlay at slave-tothe-box.net  Mon Oct 14 05:09:15 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Mon, 14 Oct 2013 06:09:15 -0600
Subject: [Bro] Yet Another Conference - like no other :)
In-Reply-To: <525BCF76.6090908@rsbac.org>
References: <525BCF76.6090908@rsbac.org>
Message-ID: <CAA9F949-C4F4-4058-8990-2479B8101030@slave-tothe-box.net>


On Oct 14, 2013, at 5:03 AM, Michal Purzynski <michal at rsbac.org> wrote:

> Yes. I'm from Mozilla. Now you know :)
> 
> Slides from my recent talk about NSM @ Mozilla at YaC 2013 are here, a 
> full video will hopefully follow.
> 
> http://tech.yandex.ru/events/yac/2013/talks/1131/
> 
> Happy to answer questions, share experience, etc. Note it's an emerging 
> project and I expect new servers to be shipped this week, so far using 
> whatever-I-could-find in terms of CPU, but the rest matches.

This was a good read this morning?thank you.

James

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131014/6e291a50/attachment.bin 

From brendan-dalpe at utulsa.edu  Mon Oct 14 06:08:45 2013
From: brendan-dalpe at utulsa.edu (Brendan Dalpe)
Date: Mon, 14 Oct 2013 08:08:45 -0500
Subject: [Bro] Duplicate log entries for events
Message-ID: <CAPkA_r3YeXSNL7Oyt4z1X136TaV6vrtsU8PJ75bRGu79VJA5HQ@mail.gmail.com>

Seth and Alex,

I was able to resolve the issue by installing the Linux headers for the
version of the kernel we are running and then recompiling the pf_ring
module. We had updated the box, but it appears that the kernel module
didn't compile correctly.

Thanks,

Brendan


On Wed, Oct 9, 2013 at 2:50 PM, Alex Waher <alexwis at gmail.com> wrote:

> Make sure `lsmod` shows that the pf_ring module is loaded. If its not
> loaded, `modprobe pf_ring`
> Or verify the eth0 interface is running with pf_ring by checking
> /proc/net/pf_ring/dev/eth0/info
>
>
> On Wed, Oct 9, 2013 at 9:35 AM, Seth Hall <seth at icir.org> wrote:
>
>>
>> On Oct 8, 2013, at 5:16 PM, Brendan Dalpe <brendan-dalpe at utulsa.edu>
>> wrote:
>>
>> > [bro-eth0]
>> > type=worker
>> > host=10.1.26.22
>> > interface=eth0
>> > lb_method=pf_ring
>> > lb_procs=4
>> >
>> >
>> > Any thoughts?
>>
>> It sounds like something isn't installed correctly.  Did you successfully
>> build Bro against the pf_ring libpcap wrapper?  Your traffic isn't load
>> balancing and each worker is getting the full stream.
>>
>> Maybe you could show us your configure command?  You can see exactly what
>> you did if you go to your source and look at build/config.status
>>
>>   .Seth
>>
>>
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>


-- 
Brendan Dalpe
brendan-dalpe at utulsa.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131014/af7c2670/attachment.html 

From jlay at slave-tothe-box.net  Tue Oct 15 14:48:33 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 15 Oct 2013 15:48:33 -0600
Subject: [Bro] Bro 2.2 beta and logstash
Message-ID: <b42777ef5716407723dfea48db122f72@localhost>

Hey all,

Just starting into this...wondering if anyone has done any legwork to 
get this into logstash at all?  I've got it inputing in now, but none of 
the fields are set yet.  I see where with 2.2.beta I can enable elastic 
search:

@load tuning/logs-to-elasticsearch

but that doesn't seem to work with the embedded option of logstash.  
Any hints getting this to be useful would help...thank you.

James


From jlay at slave-tothe-box.net  Wed Oct 16 12:41:24 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Wed, 16 Oct 2013 13:41:24 -0600
Subject: [Bro] Bro 2.2 beta and logstash
In-Reply-To: <525EE7E5.3060806@g-clef.net>
References: <b42777ef5716407723dfea48db122f72@localhost>
	<525EE7E5.3060806@g-clef.net>
Message-ID: <4ac1e70051a556a50723328021160f75@localhost>

Hey thanks a bunch for this Aaron...this totally helps me out.

James

On 2013-10-16 13:24, Aaron Gee-Clough wrote:
> I had it working for the dns.log for bro 2.1 (I'm not running 2.2
> yet), but I ended up bailing on logstash for other reasons (logstash
> would stop sending logs to ElasticSearch, for reasons I could never
> figure out...it never logged anything during the pauses). I was also
> parsing the bro logs after they were sent over to another server via
> syslog, which added another timestamp, so I wasn't trying to parse 
> the
> bro timestamp in grok. That timestamp shouldn't be hard to parse,
> though, the bro timestamp is the first  %{NUMBER} in the grok pattern
> below. (and, obviously, you'll need to change the date parsing also 
> if
> you use the bro timestamp.)
>
> In any case, here's the logstash config I used to get it going:
>
>
> input {
>     file {
>         type    => "brodns"
>         path => ["<insert path to dns.log file here>"]
>         sincedb_path => ["<insert path that the logstash user can
> write to here>"]
>     }
> }
>
> filter {
>     grok {
>         type => "brodns"
>         match => ["message",
> 
> "^%{SYSLOGTIMESTAMP:syslog_timestamp}%{SPACE}%{NUMBER}%{SPACE}%{WORD:conn_id}%{SPACE}%{IP:source_ip}%{SPACE}%{NUMBER:source_port}%{SPACE}%{IP:dest_ip}%{SPACE}%{NUMBER:dest_port}%{SPACE}%{WORD:proto}%{SPACE}%{NUMBER}%{SPACE}%{NOTSPACE:query}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE:query_class}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE:query_type}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE:response_code}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE:answers}%{SPACE}%{NOTSPACE:ttls}"]
>     }
>     date {
>         match => [ "syslog_timestamp","MMM  d HH:mm:ss", "MMM dd 
> HH:mm:ss"  ]
>         remove_field => ['syslog_timestamp']
>     }
> }
>
> output {
>     elasticsearch_http{
>         host => "<insert ES Server here>"
>         flush_size => 5000
>     }
> }
>
> aaron
>
> On 10/15/2013 05:48 PM, James Lay wrote:
>> Hey all,
>>
>> Just starting into this...wondering if anyone has done any legwork 
>> to
>> get this into logstash at all?  I've got it inputing in now, but 
>> none of
>> the fields are set yet.  I see where with 2.2.beta I can enable 
>> elastic
>> search:
>>
>> @load tuning/logs-to-elasticsearch
>>
>> but that doesn't seem to work with the embedded option of logstash.
>> Any hints getting this to be useful would help...thank you.
>>
>> James
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From r.fulton at auckland.ac.nz  Wed Oct 16 18:49:33 2013
From: r.fulton at auckland.ac.nz (Russell Fulton)
Date: Thu, 17 Oct 2013 14:49:33 +1300
Subject: [Bro] is there a standars config option for pruning historical logs?
Message-ID: <CB7EEB13-2477-4F5A-8A74-44144F44E931@auckland.ac.nz>

Yes, I know it is trivial but if there is a built in way to do it I would rather use it!  That way when someone changes the log directory between release it keeps working ;)

I am putting much of the data into elsa so there isn't much point in keeping it hanging around in bro-logs for long.

Russell


From init.conf at gmail.com  Thu Oct 17 11:06:25 2013
From: init.conf at gmail.com (Aashish Sharma)
Date: Thu, 17 Oct 2013 11:06:25 -0700
Subject: [Bro] is there a standars config option for pruning historical
	logs?
In-Reply-To: <CB7EEB13-2477-4F5A-8A74-44144F44E931@auckland.ac.nz>
References: <CB7EEB13-2477-4F5A-8A74-44144F44E931@auckland.ac.nz>
Message-ID: <B1C38FD5-7635-4805-B408-C4B645A28EB3@gmail.com>

Hello Russel:

Here is a conf setting in your broctl.cfg  :

# Expiration interval for log files in LogDir. Files older than this many days
# will be deleted upon running "broctl cron".
LogExpireInterval = 30

# Lower threshold for space available on the disk that holds SpoolDir. If less
# space is available, BroControl starts sending out warning emails.
MinDiskSpace = 5

-- 

Hope this helps, 

Aashish 

On Oct 16, 2013, at 6:49 PM, Russell Fulton <r.fulton at auckland.ac.nz> wrote:

> Yes, I know it is trivial but if there is a built in way to do it I would rather use it!  That way when someone changes the log directory between release it keeps working ;)
> 
> I am putting much of the data into elsa so there isn't much point in keeping it hanging around in bro-logs for long.
> 
> Russell
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From r.fulton at auckland.ac.nz  Thu Oct 17 15:45:22 2013
From: r.fulton at auckland.ac.nz (Russell Fulton)
Date: Fri, 18 Oct 2013 11:45:22 +1300
Subject: [Bro] is there a standars config option for pruning historical
	logs?
In-Reply-To: <B1C38FD5-7635-4805-B408-C4B645A28EB3@gmail.com>
References: <CB7EEB13-2477-4F5A-8A74-44144F44E931@auckland.ac.nz>
	<B1C38FD5-7635-4805-B408-C4B645A28EB3@gmail.com>
Message-ID: <BDFF91C7-3A3B-4F5F-B9AA-EF281E094884@auckland.ac.nz>

Thanks!  I had seen that.  In my case bro shares a partition with several other disk hungry apps (streamdb and elsa) so I need something with more control then free disk space.

Either number of days or an absolute quota.  Absolute quota is safest.

I probably add a cron job to monitor disk usage and prune files from the log directory as needed.

Any interest in me adding this to broctl's cron job?

KeepDays = <digits><units>
LogSizeLimit = <digits><units>

R


On 18/10/2013, at 7:06 AM, Aashish Sharma <init.conf at gmail.com> wrote:

> Hello Russel:
> 
> Here is a conf setting in your broctl.cfg  :
> 
> # Expiration interval for log files in LogDir. Files older than this many days
> # will be deleted upon running "broctl cron".
> LogExpireInterval = 30
> 
> # Lower threshold for space available on the disk that holds SpoolDir. If less
> # space is available, BroControl starts sending out warning emails.
> MinDiskSpace = 5
> 
> -- 
> 
> Hope this helps, 
> 
> Aashish 
> 
> On Oct 16, 2013, at 6:49 PM, Russell Fulton <r.fulton at auckland.ac.nz> wrote:
> 
>> Yes, I know it is trivial but if there is a built in way to do it I would rather use it!  That way when someone changes the log directory between release it keeps working ;)
>> 
>> I am putting much of the data into elsa so there isn't much point in keeping it hanging around in bro-logs for long.
>> 
>> Russell
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 




From r.fulton at auckland.ac.nz  Thu Oct 17 16:08:04 2013
From: r.fulton at auckland.ac.nz (Russell Fulton)
Date: Fri, 18 Oct 2013 12:08:04 +1300
Subject: [Bro] is there a standars config option for pruning historical
	logs?
In-Reply-To: <76403A1F-0EC9-44AD-A76C-47248219CEC0@gmail.com>
References: <CB7EEB13-2477-4F5A-8A74-44144F44E931@auckland.ac.nz>
	<B1C38FD5-7635-4805-B408-C4B645A28EB3@gmail.com>
	<76403A1F-0EC9-44AD-A76C-47248219CEC0@gmail.com>
Message-ID: <14F40896-BFD7-4F74-817F-9C54CC0540FC@auckland.ac.nz>


On 18/10/2013, at 12:02 PM, Aashish Sharma <init.conf at gmail.com>
 wrote:

> Russell : 
> 
> Well in broctl.conf: 
> 
> LogExpireInterval = X ;  means how many days (X)  of logs do you want to keep on the disk. 
> 
> Isn't this what you want ? 

Right!!  The 'interval' confused me.  I thought it was the interval between checking!

I thought it odd that some time based expiry was not supported out of the box!

Russell




From michal at rsbac.org  Fri Oct 18 03:52:33 2013
From: michal at rsbac.org (Michal Purzynski)
Date: Fri, 18 Oct 2013 12:52:33 +0200
Subject: [Bro] Yet Another Conference - like no other :)
In-Reply-To: <CAA9F949-C4F4-4058-8990-2479B8101030@slave-tothe-box.net>
References: <525BCF76.6090908@rsbac.org>
	<CAA9F949-C4F4-4058-8990-2479B8101030@slave-tothe-box.net>
Message-ID: <526112F1.6020901@rsbac.org>

On 10/14/13 2:09 PM, James Lay wrote:
> On Oct 14, 2013, at 5:03 AM, Michal Purzynski <michal at rsbac.org> wrote:
>
>> Yes. I'm from Mozilla. Now you know :)
>>
>> Slides from my recent talk about NSM @ Mozilla at YaC 2013 are here, a
>> full video will hopefully follow.
>>
>> http://tech.yandex.ru/events/yac/2013/talks/1131/
>>
>> Happy to answer questions, share experience, etc. Note it's an emerging
>> project and I expect new servers to be shipped this week, so far using
>> whatever-I-could-find in terms of CPU, but the rest matches.
>
http://tech.yandex.ru/events/yac/2013/talks/1131/

Video is there.

1. I'm not a native speaker. That's why I'm talking slower ;)
2. There's quite a bit of people in the audience but the camera mostly 
gets what's in 2-3 front rows which were reserved
3. Being is the last in a day, presenting at 19:30 is hard

Hope it's providing some useful background on our implementation of Bro. 
Feedback welcome :)


From kristoffer.bjork at gmail.com  Fri Oct 18 10:38:15 2013
From: kristoffer.bjork at gmail.com (=?ISO-8859-1?Q?Kristoffer_Bj=F6rk?=)
Date: Fri, 18 Oct 2013 19:38:15 +0200
Subject: [Bro] Yet Another Conference - like no other :)
In-Reply-To: <526112F1.6020901@rsbac.org>
References: <525BCF76.6090908@rsbac.org>
	<CAA9F949-C4F4-4058-8990-2479B8101030@slave-tothe-box.net>
	<526112F1.6020901@rsbac.org>
Message-ID: <CAEGuVoKdSgxtRND4iZjXEFTNc7jiRv2AsV7wwKu4Ehv-ZsQfaQ@mail.gmail.com>

Great presentation!
Do you use security onion for the bro & snort clusters or you installed it
on vanilla linux/bsd boxes?

//Kristoffer


On Fri, Oct 18, 2013 at 12:52 PM, Michal Purzynski <michal at rsbac.org> wrote:

> On 10/14/13 2:09 PM, James Lay wrote:
> > On Oct 14, 2013, at 5:03 AM, Michal Purzynski <michal at rsbac.org> wrote:
> >
> >> Yes. I'm from Mozilla. Now you know :)
> >>
> >> Slides from my recent talk about NSM @ Mozilla at YaC 2013 are here, a
> >> full video will hopefully follow.
> >>
> >> http://tech.yandex.ru/events/yac/2013/talks/1131/
> >>
> >> Happy to answer questions, share experience, etc. Note it's an emerging
> >> project and I expect new servers to be shipped this week, so far using
> >> whatever-I-could-find in terms of CPU, but the rest matches.
> >
> http://tech.yandex.ru/events/yac/2013/talks/1131/
>
> Video is there.
>
> 1. I'm not a native speaker. That's why I'm talking slower ;)
> 2. There's quite a bit of people in the audience but the camera mostly
> gets what's in 2-3 front rows which were reserved
> 3. Being is the last in a day, presenting at 19:30 is hard
>
> Hope it's providing some useful background on our implementation of Bro.
> Feedback welcome :)
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131018/dad39409/attachment.html 

From michal at rsbac.org  Sat Oct 19 04:19:19 2013
From: michal at rsbac.org (Michal Purzynski)
Date: Sat, 19 Oct 2013 13:19:19 +0200
Subject: [Bro] Yet Another Conference - like no other :)
In-Reply-To: <CAEGuVoKdSgxtRND4iZjXEFTNc7jiRv2AsV7wwKu4Ehv-ZsQfaQ@mail.gmail.com>
References: <525BCF76.6090908@rsbac.org>	<CAA9F949-C4F4-4058-8990-2479B8101030@slave-tothe-box.net>	<526112F1.6020901@rsbac.org>
	<CAEGuVoKdSgxtRND4iZjXEFTNc7jiRv2AsV7wwKu4Ehv-ZsQfaQ@mail.gmail.com>
Message-ID: <52626AB7.7070608@rsbac.org>

On 10/18/13 7:38 PM, Kristoffer Bj?rk wrote:
> Great presentation!
> Do you use security onion for the bro & snort clusters or you 
> installed it on vanilla linux/bsd boxes?
>
It's all Security Onion, tuned to our needs. That's the power of SO - 
it's so flexible you can enable/disable/change parts of it without 
impacting the rest. I can't imagine doing all the integration that SO 
does, myself. Technically doable, but -ENOTIME :)
> //Kristoffer
>
>
> On Fri, Oct 18, 2013 at 12:52 PM, Michal Purzynski <michal at rsbac.org 
> <mailto:michal at rsbac.org>> wrote:
>
>     On 10/14/13 2:09 PM, James Lay wrote:
>     > On Oct 14, 2013, at 5:03 AM, Michal Purzynski <michal at rsbac.org
>     <mailto:michal at rsbac.org>> wrote:
>     >
>     >> Yes. I'm from Mozilla. Now you know :)
>     >>
>     >> Slides from my recent talk about NSM @ Mozilla at YaC 2013 are
>     here, a
>     >> full video will hopefully follow.
>     >>
>     >> http://tech.yandex.ru/events/yac/2013/talks/1131/
>     >>
>     >> Happy to answer questions, share experience, etc. Note it's an
>     emerging
>     >> project and I expect new servers to be shipped this week, so
>     far using
>     >> whatever-I-could-find in terms of CPU, but the rest matches.
>     >
>     http://tech.yandex.ru/events/yac/2013/talks/1131/
>
>     Video is there.
>
>     1. I'm not a native speaker. That's why I'm talking slower ;)
>     2. There's quite a bit of people in the audience but the camera mostly
>     gets what's in 2-3 front rows which were reserved
>     3. Being is the last in a day, presenting at 19:30 is hard
>
>     Hope it's providing some useful background on our implementation
>     of Bro.
>     Feedback welcome :)
>     _______________________________________________
>     Bro mailing list
>     bro at bro-ids.org <mailto:bro at bro-ids.org>
>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131019/af925848/attachment.html 

From jamesfhook at gmail.com  Sat Oct 19 16:35:33 2013
From: jamesfhook at gmail.com (James Hook)
Date: Sun, 20 Oct 2013 00:35:33 +0100
Subject: [Bro] deleting objects passed to BiF event handlers
Message-ID: <CAC+_1SYfPWu3KfArPk6knd2gYEimY_ueEJ6PKH49-wE+vOwnfw@mail.gmail.com>

Hi all,

I've just been working through the example of creating built in functions
http://www.bro.org/development/howtos/bif-doc/example.html

I have a quick question about what the design pattern should be for
deleting objects created and passed to the firing event handler. For
example, in the code below, is deleting the msg pointer likely to cause
problems to QueueEvent?

Or does the StringVal class do something clever behind the scenes to stop
memory leaks?

Many thanks in advance

James

------

msg = new StringVal("Hello, World!");

if (bif_test_event)
            {

                val_list *vl = new val_list;
                vl->append(msg);

                mgr.QueueEvent(bif_test_event, vl);
            }

delete msg;
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131020/6f7f5196/attachment.html 

From lysemose at gmail.com  Mon Oct 21 05:15:03 2013
From: lysemose at gmail.com (Heine Lysemose)
Date: Mon, 21 Oct 2013 14:15:03 +0200
Subject: [Bro] Yet Another Conference - like no other :)
In-Reply-To: <52626AB7.7070608@rsbac.org>
References: <525BCF76.6090908@rsbac.org>
	<CAA9F949-C4F4-4058-8990-2479B8101030@slave-tothe-box.net>
	<526112F1.6020901@rsbac.org>
	<CAEGuVoKdSgxtRND4iZjXEFTNc7jiRv2AsV7wwKu4Ehv-ZsQfaQ@mail.gmail.com>
	<52626AB7.7070608@rsbac.org>
Message-ID: <CAN4C-D=9xEx3ty-VoJji1LZo5wG7ZWHoqiVnfsytf2FiU3PDwg@mail.gmail.com>

Hi Michal

Nice presentation!!
Cool to see some real world experiences... Especially the part "what's
working" and "what's not working".

Regards,
Lysemose


On Sat, Oct 19, 2013 at 1:19 PM, Michal Purzynski <michal at rsbac.org> wrote:

>  On 10/18/13 7:38 PM, Kristoffer Bj?rk wrote:
>
> Great presentation!
> Do you use security onion for the bro & snort clusters or you installed it
> on vanilla linux/bsd boxes?
>
>   It's all Security Onion, tuned to our needs. That's the power of SO -
> it's so flexible you can enable/disable/change parts of it without
> impacting the rest. I can't imagine doing all the integration that SO does,
> myself. Technically doable, but -ENOTIME :)
>
>  //Kristoffer
>
>
> On Fri, Oct 18, 2013 at 12:52 PM, Michal Purzynski <michal at rsbac.org>wrote:
>
>>  On 10/14/13 2:09 PM, James Lay wrote:
>> > On Oct 14, 2013, at 5:03 AM, Michal Purzynski <michal at rsbac.org> wrote:
>> >
>> >> Yes. I'm from Mozilla. Now you know :)
>> >>
>> >> Slides from my recent talk about NSM @ Mozilla at YaC 2013 are here, a
>> >> full video will hopefully follow.
>> >>
>> >> http://tech.yandex.ru/events/yac/2013/talks/1131/
>> >>
>> >> Happy to answer questions, share experience, etc. Note it's an emerging
>> >> project and I expect new servers to be shipped this week, so far using
>> >> whatever-I-could-find in terms of CPU, but the rest matches.
>> >
>>  http://tech.yandex.ru/events/yac/2013/talks/1131/
>>
>> Video is there.
>>
>> 1. I'm not a native speaker. That's why I'm talking slower ;)
>> 2. There's quite a bit of people in the audience but the camera mostly
>> gets what's in 2-3 front rows which were reserved
>> 3. Being is the last in a day, presenting at 19:30 is hard
>>
>> Hope it's providing some useful background on our implementation of Bro.
>> Feedback welcome :)
>>  _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131021/3c6f4eb4/attachment.html 

From eric at nixwizard.net  Mon Oct 21 06:32:42 2013
From: eric at nixwizard.net (Eric G)
Date: Mon, 21 Oct 2013 09:32:42 -0400
Subject: [Bro] Yet Another Conference - like no other :)
In-Reply-To: <525BCF76.6090908@rsbac.org>
References: <525BCF76.6090908@rsbac.org>
Message-ID: <CAFzrbiHdiFKkdppRnCcdgwQPGH-pTiOgypa5niqX9XtC3ALs2A@mail.gmail.com>

On Oct 14, 2013 6:19 AM, "Michal Purzynski" <michal at rsbac.org> wrote:
>
> Yes. I'm from Mozilla. Now you know :)
>
> Slides from my recent talk about NSM @ Mozilla at YaC 2013 are here, a
> full video will hopefully follow.
>
> http://tech.yandex.ru/events/yac/2013/talks/1131/
>

Nice presentation, it confirms a few things I was suspecting :-)

I see you are logging to elasticsearch from Bro... have you taken a look at
Moloch for full packet capture? It's not included in Security Onion (yet?)
but we have played with it at work and we're now budgeting for Moloch
boxes. Moloch just recently added support for pfring as well, and from the
mailing list I saw someone posting that they were using pfring with
success. It does a really good job of indexing packet captures and has some
protocol decoders built in... I've found I don't even need to pull a pcap
out of it half the time because I get a clear picture from Moloch's web
interface

https://github.com/aol/moloch is their Github site

Just a thought

--
Eric
http://www.linkedin.com/in/ericgearhart
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131021/2e17a182/attachment.html 

From jswan at sugf.com  Mon Oct 21 07:20:58 2013
From: jswan at sugf.com (Swan, Jay)
Date: Mon, 21 Oct 2013 14:20:58 +0000
Subject: [Bro] Yet Another Conference - like no other :)
In-Reply-To: <52626AB7.7070608@rsbac.org>
References: <525BCF76.6090908@rsbac.org>
	<CAA9F949-C4F4-4058-8990-2479B8101030@slave-tothe-box.net>
	<526112F1.6020901@rsbac.org>
	<CAEGuVoKdSgxtRND4iZjXEFTNc7jiRv2AsV7wwKu4Ehv-ZsQfaQ@mail.gmail.com>
	<52626AB7.7070608@rsbac.org>
Message-ID: <20131021142101.0C3D92C4014@rock.ICSI.Berkeley.EDU>

You mentioned that you're using Bro 2.2, though. Is that on a separate cluster or are you building 2.2 into a Security Onion install? If the latter, how do you manage that process? Seems like it would be complex.

From: bro-bounces at bro.org [mailto:bro-bounces at bro.org] On Behalf Of Michal Purzynski
Sent: Saturday, October 19, 2013 5:19 AM
To: Kristoffer Bj?rk
Cc: bro at bro.org
Subject: Re: [Bro] Yet Another Conference - like no other :)

On 10/18/13 7:38 PM, Kristoffer Bj?rk wrote:
Great presentation!
Do you use security onion for the bro & snort clusters or you installed it on vanilla linux/bsd boxes?

It's all Security Onion, tuned to our needs. That's the power of SO - it's so flexible you can enable/disable/change parts of it without impacting the rest. I can't imagine doing all the integration that SO does, myself. Technically doable, but -ENOTIME :)
//Kristoffer

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131021/ab1a245d/attachment.html 

From jsiwek at illinois.edu  Mon Oct 21 07:50:49 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 21 Oct 2013 14:50:49 +0000
Subject: [Bro] deleting objects passed to BiF event handlers
In-Reply-To: <CAC+_1SYfPWu3KfArPk6knd2gYEimY_ueEJ6PKH49-wE+vOwnfw@mail.gmail.com>
References: <CAC+_1SYfPWu3KfArPk6knd2gYEimY_ueEJ6PKH49-wE+vOwnfw@mail.gmail.com>
Message-ID: <CF88DF81-2DEC-4397-B802-BBA5719FD8DF@illinois.edu>


On Oct 19, 2013, at 6:35 PM, James Hook <jamesfhook at gmail.com> wrote:

> I've just been working through the example of creating built in functions http://www.bro.org/development/howtos/bif-doc/example.html
> 
> I have a quick question about what the design pattern should be for deleting objects created and passed to the firing event handler. For example, in the code below, is deleting the msg pointer likely to cause problems to QueueEvent?

Yes (it will probably cause a double-free when there's a bif_test_event handler).  EventMgr::QueueEvent takes ownership of the val_list and a single reference count to any elements in it.

> Or does the StringVal class do something clever behind the scenes to stop memory leaks?

No.  I'll fix the example code in the docs to better demonstrate correct memory management -- which probably the best way is to just not allocate a Val or val_list unless it's known it's going to get passed to EventMgr::QueueEvent.

- Jon


From seth at icir.org  Mon Oct 21 07:57:16 2013
From: seth at icir.org (Seth Hall)
Date: Mon, 21 Oct 2013 10:57:16 -0400
Subject: [Bro] Yet Another Conference - like no other :)
In-Reply-To: <CAFzrbiHdiFKkdppRnCcdgwQPGH-pTiOgypa5niqX9XtC3ALs2A@mail.gmail.com>
References: <525BCF76.6090908@rsbac.org>
	<CAFzrbiHdiFKkdppRnCcdgwQPGH-pTiOgypa5niqX9XtC3ALs2A@mail.gmail.com>
Message-ID: <B3DF9348-1F7C-4501-868E-8B5C5E0082EB@icir.org>


On Oct 21, 2013, at 9:32 AM, Eric G <eric at nixwizard.net> wrote:

> It does a really good job of indexing packet captures and has some protocol decoders built in... I've found I don't even need to pull a pcap out of it half the time because I get a clear picture from Moloch's web interface

Just curious, why are you advocating for Moloch on the Bro mailing list?  Is it only because of the interface?  

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131021/d2a9e89a/attachment.bin 

From michal at rsbac.org  Mon Oct 21 07:59:45 2013
From: michal at rsbac.org (Michal Purzynski)
Date: Mon, 21 Oct 2013 16:59:45 +0200
Subject: [Bro] Yet Another Conference - like no other :)
In-Reply-To: <20131021142101.0C3D92C4014@rock.ICSI.Berkeley.EDU>
References: <525BCF76.6090908@rsbac.org>	<CAA9F949-C4F4-4058-8990-2479B8101030@slave-tothe-box.net>	<526112F1.6020901@rsbac.org>	<CAEGuVoKdSgxtRND4iZjXEFTNc7jiRv2AsV7wwKu4Ehv-ZsQfaQ@mail.gmail.com>	<52626AB7.7070608@rsbac.org>
	<20131021142101.0C3D92C4014@rock.ICSI.Berkeley.EDU>
Message-ID: <52654161.3090100@rsbac.org>

On 10/21/13 4:20 PM, Swan, Jay wrote:
>
> You mentioned that you're using Bro 2.2, though. Is that on a separate 
> cluster or are you building 2.2 into a Security Onion install? If the 
> latter, how do you manage that process? Seems like it would be complex.
>
>

So far yes, it's a manual installation of Bro 2.2 beta on top of SO but 
in such fast moving project SO is, I'm sure the new Bro will be 
integrated as soon as it's released. For now it's just a mv /opt/bro 
/opt/bro.dist and installing a new /opt/bro :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131021/a49570ff/attachment.html 

From JAzoff at albany.edu  Mon Oct 21 08:31:23 2013
From: JAzoff at albany.edu (Justin Azoff)
Date: Mon, 21 Oct 2013 11:31:23 -0400
Subject: [Bro] is there a standars config option for pruning historical
 logs?
In-Reply-To: <CB7EEB13-2477-4F5A-8A74-44144F44E931@auckland.ac.nz>
References: <CB7EEB13-2477-4F5A-8A74-44144F44E931@auckland.ac.nz>
Message-ID: <20131021153123.GI13133@datacomm.albany.edu>

On Thu, Oct 17, 2013 at 02:49:33PM +1300, Russell Fulton wrote:
> Yes, I know it is trivial but if there is a built in way to do it I would rather use it!  That way when someone changes the log directory between release it keeps working ;)
> 
> I am putting much of the data into elsa so there isn't much point in keeping it hanging around in bro-logs for long.
> 

In addition to the options you found another option is to use cron jobs
to have per log file retention periods.  I do something like this:

    @daily root find /usr/local/bro/logs/ -name 'http*' -mtime +6 | xargs rm -f
    @daily root find /usr/local/bro/logs/ -name 'ftp.*' -mtime +6 | xargs rm -f

to only keep http and ftp logs for a week.  Once the 'big' log files are
removed a days logs are < 100M instead of 10G

-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From michal at rsbac.org  Mon Oct 21 12:06:35 2013
From: michal at rsbac.org (Michal Purzynski)
Date: Mon, 21 Oct 2013 21:06:35 +0200
Subject: [Bro] Yet Another Conference - like no other :)
In-Reply-To: <CAFzrbiHdiFKkdppRnCcdgwQPGH-pTiOgypa5niqX9XtC3ALs2A@mail.gmail.com>
References: <525BCF76.6090908@rsbac.org>
	<CAFzrbiHdiFKkdppRnCcdgwQPGH-pTiOgypa5niqX9XtC3ALs2A@mail.gmail.com>
Message-ID: <52657B3B.6040406@rsbac.org>


> >
> > Slides from my recent talk about NSM @ Mozilla at YaC 2013 are here, a
> > full video will hopefully follow.
> >
> > http://tech.yandex.ru/events/yac/2013/talks/1131/
> >
>
> Nice presentation, it confirms a few things I was suspecting :-)
>
> I see you are logging to elasticsearch from Bro... have you taken a 
> look at Moloch for full packet capture? It's not included in Security 
> Onion (yet?) but we have played with it at work and we're now 
> budgeting for Moloch boxes. Moloch just recently added support for 
> pfring as well, and from the mailing list I saw someone posting that 
> they were using pfring with success. It does a really good job of 
> indexing packet captures and has some protocol decoders built in... 
> I've found I don't even need to pull a pcap out of it half the time 
> because I get a clear picture from Moloch's web interface
>
> https://github.com/aol/moloch is their Github site
>
>
Replacing netsniff-ng with anything else is possible here, but I don't 
feel like I need it - SO has a great integration between pcap agent, 
ELSA and Bro. I can go to ELSA, find the flow I need and request a 
transcript - simple and very effective.

As for the metadata and data about my flows, content, protocol decoders, 
scripting - I would not change Bro for a 1024 kg of pure gold, if that's 
what you are asking :)


From jamesfhook at gmail.com  Tue Oct 22 00:24:53 2013
From: jamesfhook at gmail.com (James Hook)
Date: Tue, 22 Oct 2013 08:24:53 +0100
Subject: [Bro] deleting objects passed to BiF event handlers
In-Reply-To: <CF88DF81-2DEC-4397-B802-BBA5719FD8DF@illinois.edu>
References: <CAC+_1SYfPWu3KfArPk6knd2gYEimY_ueEJ6PKH49-wE+vOwnfw@mail.gmail.com>
	<CF88DF81-2DEC-4397-B802-BBA5719FD8DF@illinois.edu>
Message-ID: <CAC+_1Sb17+75maKVwnVR=YOA0_2G90SwG8vUPx7Ct1ZxsZuLNg@mail.gmail.com>

Thanks Jon, that makes a sense.

-------
P.S.: the example code was pretty good for getting up and running very
quickly and demonstrating a lot of the marshaling of bro data types. There
were a couple of bits I had to change for Bro2.2Beta.
- BroTypePtr:: is now just BroType
- bro.init is now init-bare.bro
Very seldom I get these things to work so quickly though :-)




On 21 October 2013 15:50, Siwek, Jonathan Luke <jsiwek at illinois.edu> wrote:

>
> On Oct 19, 2013, at 6:35 PM, James Hook <jamesfhook at gmail.com> wrote:
>
> > I've just been working through the example of creating built in
> functions http://www.bro.org/development/howtos/bif-doc/example.html
> >
> > I have a quick question about what the design pattern should be for
> deleting objects created and passed to the firing event handler. For
> example, in the code below, is deleting the msg pointer likely to cause
> problems to QueueEvent?
>
> Yes (it will probably cause a double-free when there's a bif_test_event
> handler).  EventMgr::QueueEvent takes ownership of the val_list and a
> single reference count to any elements in it.
>
> > Or does the StringVal class do something clever behind the scenes to
> stop memory leaks?
>
> No.  I'll fix the example code in the docs to better demonstrate correct
> memory management -- which probably the best way is to just not allocate a
> Val or val_list unless it's known it's going to get passed to
> EventMgr::QueueEvent.
>
> - Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131022/a243bb4c/attachment.html 

From jlay at slave-tothe-box.net  Tue Oct 22 09:57:11 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 22 Oct 2013 10:57:11 -0600
Subject: [Bro] Frontend
Message-ID: <099bee944ab3559ae5e3c5cadd6212fd@localhost>

Hey all!

So...I'm looking for SOMETHING that will allow me to parse and 
aggregate bro, snort, and firewall logs.  I've looked at logstash, but 
the latest version seems poorly documented...everything that I wanted to 
try took ages to figure out.  Anyone have anything that will accomplish 
something like this?  Thanks all.

James


From doug.burks at gmail.com  Tue Oct 22 10:14:04 2013
From: doug.burks at gmail.com (Doug Burks)
Date: Tue, 22 Oct 2013 13:14:04 -0400
Subject: [Bro] Frontend
In-Reply-To: <099bee944ab3559ae5e3c5cadd6212fd@localhost>
References: <099bee944ab3559ae5e3c5cadd6212fd@localhost>
Message-ID: <CAK8kjrB0KmY--9aiT6p6CNedPM1FC9XpR-oqbYs6rzYRX967=w@mail.gmail.com>

Hi James,

Have you considered ELSA?
https://code.google.com/p/enterprise-log-search-and-archive/wiki/Documentation

Also see:
https://www.youtube.com/watch?v=33HZyIxbg6c&list=PLMN5wm-C5YjyieO63g8LbaiWTSJRj0DBe



On Tue, Oct 22, 2013 at 12:57 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> Hey all!
>
> So...I'm looking for SOMETHING that will allow me to parse and
> aggregate bro, snort, and firewall logs.  I've looked at logstash, but
> the latest version seems poorly documented...everything that I wanted to
> try took ages to figure out.  Anyone have anything that will accomplish
> something like this?  Thanks all.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



-- 
Doug Burks
http://securityonion.blogspot.com


From jthoel at gmail.com  Tue Oct 22 10:16:17 2013
From: jthoel at gmail.com (Jeremy Hoel)
Date: Tue, 22 Oct 2013 17:16:17 +0000
Subject: [Bro] Frontend
In-Reply-To: <099bee944ab3559ae5e3c5cadd6212fd@localhost>
References: <099bee944ab3559ae5e3c5cadd6212fd@localhost>
Message-ID: <CAH_p-VNhhAkAnq_KpDWZxNDMOyzMLa+msyvsoO-pEFXYm+LCGA@mail.gmail.com>

splunk  -)

And Security onion uses Elsa to do the same thing.  Elsa is just a
moving target (changes a lot) but Martin is doing an awesome job
fixing things, adding features and helping people out.

Elsa has the benefit of being free with no limits, vs splunk 500mb/day

On Tue, Oct 22, 2013 at 4:57 PM, James Lay <jlay at slave-tothe-box.net> wrote:
> Hey all!
>
> So...I'm looking for SOMETHING that will allow me to parse and
> aggregate bro, snort, and firewall logs.  I've looked at logstash, but
> the latest version seems poorly documented...everything that I wanted to
> try took ages to figure out.  Anyone have anything that will accomplish
> something like this?  Thanks all.
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


From jlay at slave-tothe-box.net  Tue Oct 22 10:17:04 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 22 Oct 2013 11:17:04 -0600
Subject: [Bro] Frontend
In-Reply-To: <CAK8kjrB0KmY--9aiT6p6CNedPM1FC9XpR-oqbYs6rzYRX967=w@mail.gmail.com>
References: <099bee944ab3559ae5e3c5cadd6212fd@localhost>
	<CAK8kjrB0KmY--9aiT6p6CNedPM1FC9XpR-oqbYs6rzYRX967=w@mail.gmail.com>
Message-ID: <23726e3152a5f27c8cd450f19966b260@localhost>

On 2013-10-22 11:14, Doug Burks wrote:
> Hi James,
>
> Have you considered ELSA?
> 
> https://code.google.com/p/enterprise-log-search-and-archive/wiki/Documentation
>
> Also see:
> 
> https://www.youtube.com/watch?v=33HZyIxbg6c&list=PLMN5wm-C5YjyieO63g8LbaiWTSJRj0DBe
>
>
>

Thanks for the ELSA recommendations all...giving it a look now.

James


From eric at nixwizard.net  Tue Oct 22 10:26:41 2013
From: eric at nixwizard.net (Eric G)
Date: Tue, 22 Oct 2013 13:26:41 -0400
Subject: [Bro] Frontend
In-Reply-To: <099bee944ab3559ae5e3c5cadd6212fd@localhost>
References: <099bee944ab3559ae5e3c5cadd6212fd@localhost>
Message-ID: <CAFzrbiHN+OhNChYy-PX0N9MUBqhnAiearp3dXAiuTr9W6T1ztg@mail.gmail.com>

On Oct 22, 2013 12:06 PM, "James Lay" <jlay at slave-tothe-box.net> wrote:
>
> Hey all!
>
> So...I'm looking for SOMETHING that will allow me to parse and
> aggregate bro, snort, and firewall logs.

Splunk on the commercial side, ELSA on the free side would be my
suggestions without hearing more details about your environment or needs

On the free side you're going to spend time setting them up and getting
stuff configured... That's the price of the open source log aggregation
stuff out there...

--
Eric
http://www.linkedin.com/in/ericgearhart
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131022/5db2fc4b/attachment.html 

From jlay at slave-tothe-box.net  Tue Oct 22 10:29:11 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Tue, 22 Oct 2013 11:29:11 -0600
Subject: [Bro] Frontend
In-Reply-To: <CAFzrbiHN+OhNChYy-PX0N9MUBqhnAiearp3dXAiuTr9W6T1ztg@mail.gmail.com>
References: <099bee944ab3559ae5e3c5cadd6212fd@localhost>
	<CAFzrbiHN+OhNChYy-PX0N9MUBqhnAiearp3dXAiuTr9W6T1ztg@mail.gmail.com>
Message-ID: <7808f1cfd53c8a876d556de4a56ace76@localhost>

On 2013-10-22 11:26, Eric G wrote:
> On Oct 22, 2013 12:06 PM, "James Lay" <jlay at slave-tothe-box.net [1]>
> wrote:
>  >
>  > Hey all!
>  >
>  > So...Im looking for SOMETHING that will allow me to parse and
>  > aggregate bro, snort, and firewall logs.
>
> Splunk on the commercial side, ELSA on the free side would be my
> suggestions without hearing more details about your environment or
> needs
>
> On the free side youre going to spend time setting them up and 
> getting
> stuff configured... Thats the price of the open source log 
> aggregation
> stuff out there...
>
> --
>  Eric
>  http://www.linkedin.com/in/ericgearhart [2]

Thanks Eric...something that lifts my spirits:

Plugins

ELSA ships with several plugins:

     Windows logs from Eventlog-to-Syslog
     Snort/Suricata logs
     Bro logs
     Url logs from httpry_logger

So THAT helps...I won't have to reinvent anything.  Documentation looks 
pretty tasty as well, so let's hope it's not too much of a hassle.  I'll 
report my success/failures here.

James


From eric at nixwizard.net  Tue Oct 22 10:35:54 2013
From: eric at nixwizard.net (Eric G)
Date: Tue, 22 Oct 2013 13:35:54 -0400
Subject: [Bro] Frontend
In-Reply-To: <7808f1cfd53c8a876d556de4a56ace76@localhost>
References: <099bee944ab3559ae5e3c5cadd6212fd@localhost>
	<CAFzrbiHN+OhNChYy-PX0N9MUBqhnAiearp3dXAiuTr9W6T1ztg@mail.gmail.com>
	<7808f1cfd53c8a876d556de4a56ace76@localhost>
Message-ID: <CAFzrbiEBEd7ps578qFHB2Dpa8-tkDt=SMwQ0m7A+7f_zrTohYg@mail.gmail.com>

On Oct 22, 2013 12:29 PM, "James Lay" <jlay at slave-tothe-box.net> wrote:
>
> On 2013-10-22 11:26, Eric G wrote:
>>
>> On Oct 22, 2013 12:06 PM, "James Lay" <jlay at slave-tothe-box.net [1]>
>> wrote:
>>  >
>>  > Hey all!
>>  >
>>  > So...Im looking for SOMETHING that will allow me to parse and
>>
>>  > aggregate bro, snort, and firewall logs.
>>
>> Splunk on the commercial side, ELSA on the free side would be my
>> suggestions without hearing more details about your environment or
>> needs
>>
>> On the free side youre going to spend time setting them up and getting
>> stuff configured... Thats the price of the open source log aggregation
>>
>> stuff out there...
>>
>> --
>>  Eric
>>  http://www.linkedin.com/in/ericgearhart [2]
>
>
> Thanks Eric...something that lifts my spirits:
>
> Plugins
>
> ELSA ships with several plugins:
>
>     Windows logs from Eventlog-to-Syslog
>     Snort/Suricata logs
>     Bro logs
>     Url logs from httpry_logger
>
> So THAT helps...I won't have to reinvent anything.  Documentation looks
pretty tasty as well, so let's hope it's not too much of a hassle.  I'll
report my success/failures here.
>

I think there are parsers for a couple of firewall vendors too. I would be
open to helping ya get a parser written if there isn't one for whatever
firewall solution you're using. ELSA's a really neat project, so it'd be
cool to help it out

--
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131022/05380525/attachment.html 

From eric at nixwizard.net  Tue Oct 22 10:39:46 2013
From: eric at nixwizard.net (Eric G)
Date: Tue, 22 Oct 2013 13:39:46 -0400
Subject: [Bro] Frontend
In-Reply-To: <7808f1cfd53c8a876d556de4a56ace76@localhost>
References: <099bee944ab3559ae5e3c5cadd6212fd@localhost>
	<CAFzrbiHN+OhNChYy-PX0N9MUBqhnAiearp3dXAiuTr9W6T1ztg@mail.gmail.com>
	<7808f1cfd53c8a876d556de4a56ace76@localhost>
Message-ID: <CAFzrbiFjYQrKHX43r-QOXRp2gvYOSYkG_3tvUgZ2LTkVBFV6AA@mail.gmail.com>

On Oct 22, 2013 12:29 PM, "James Lay" <jlay at slave-tothe-box.net> wrote:
>
> On 2013-10-22 11:26, Eric G wrote:
>>
>> On Oct 22, 2013 12:06 PM, "James Lay" <jlay at slave-tothe-box.net [1]>
>> wrote:
>>  >
>>  > Hey all!
>>  >
>>  > So...Im looking for SOMETHING that will allow me to parse and
>>
>>  > aggregate bro, snort, and firewall logs.
>>
>> Splunk on the commercial side, ELSA on the free side would be my
>> suggestions without hearing more details about your environment or
>> needs
>>
>> On the free side youre going to spend time setting them up and getting
>> stuff configured... Thats the price of the open source log aggregation
>>
>> stuff out there...
>>
>> --
>>  Eric
>>  http://www.linkedin.com/in/ericgearhart [2]
>
>
> Thanks Eric...something that lifts my spirits:
>
> Plugins
>
> ELSA ships with several plugins:
>
>     Windows logs from Eventlog-to-Syslog
>     Snort/Suricata logs
>     Bro logs
>     Url logs from httpry_logger
>
> So THAT helps...I won't have to reinvent anything.  Documentation looks
pretty tasty as well, so let's hope it's not too much of a hassle.  I'll
report my success/failures here.
>

Yup looks like plugins for a few different vendors have been written:
" By popular demand, I've added a number of new parsers to the ELSA
repertoire to support parsing fields from the following devices:
 - Fortinet (URL, traffic)
 - Checkpoint
 - Palo Alto (URL, traffic)
 - Barracuda (scan, receive, send)
 - OSSEC Windows logs (automatically appears as class Windows)"

(from http://ossectools.blogspot.com/2012/02/new-elsa-log-parsers.html)

--
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131022/b32cbb25/attachment.html 

From robin at icir.org  Wed Oct 23 09:53:34 2013
From: robin at icir.org (Robin Sommer)
Date: Wed, 23 Oct 2013 09:53:34 -0700
Subject: [Bro] Open positions
Message-ID: <20131023165334.GZ49505@icir.org>

We have open positions at ICSI for Bro research and engineering work,
see here for more information:

    http://www.bro.org/jobs.html

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 *     robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 * www.icir.org/robin


From jlay at slave-tothe-box.net  Thu Oct 24 08:10:00 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 24 Oct 2013 09:10:00 -0600
Subject: [Bro] Changing separator
Message-ID: <00610fcd390f7ef7a725bdacfb1e9973@localhost>

So...I'm almost certain that there was a way to change from the tab 
character, to comma (I thought), but for the life of me I can't seem to 
find it.  Been working with logstash and currently messages come through 
as:

1382627138.211512\tCQ74U23HZlcab0LNnh\t192.168.1.3\t64079\t224.0.0.1\t8612\tudp\t-\t-\t-\t-\tS0\tT\t0\tD\t1\t44\t0\t0\t(empty)

Which is kind of painful for matching.  Any quick pointers on how to do 
this?  Thank you.

James


From bernhard at ICSI.Berkeley.EDU  Thu Oct 24 08:15:20 2013
From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann)
Date: Thu, 24 Oct 2013 08:15:20 -0700
Subject: [Bro] Changing separator
In-Reply-To: <00610fcd390f7ef7a725bdacfb1e9973@localhost>
References: <00610fcd390f7ef7a725bdacfb1e9973@localhost>
Message-ID: <B8AC9A9E-900B-4E6E-A24B-5DA0F60C6481@icsi.berkeley.edu>

Hi,

you can redefine Log::separator. See http://www.bro.org/sphinx-git/scripts/base/frameworks/logging/main.html

Bernhard

On Oct 24, 2013, at 8:10 AM, James Lay <jlay at slave-tothe-box.net> wrote:

> So...I'm almost certain that there was a way to change from the tab 
> character, to comma (I thought), but for the life of me I can't seem to 
> find it.  Been working with logstash and currently messages come through 
> as:
> 
> 1382627138.211512\tCQ74U23HZlcab0LNnh\t192.168.1.3\t64079\t224.0.0.1\t8612\tudp\t-\t-\t-\t-\tS0\tT\t0\tD\t1\t44\t0\t0\t(empty)
> 
> Which is kind of painful for matching.  Any quick pointers on how to do 
> this?  Thank you.
> 
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From jsiwek at illinois.edu  Thu Oct 24 08:25:41 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Thu, 24 Oct 2013 15:25:41 +0000
Subject: [Bro] Changing separator
In-Reply-To: <00610fcd390f7ef7a725bdacfb1e9973@localhost>
References: <00610fcd390f7ef7a725bdacfb1e9973@localhost>
Message-ID: <A6651D6F-1848-4F1E-96CA-1DB6B2321CBF@illinois.edu>


On Oct 24, 2013, at 10:10 AM, James Lay <jlay at slave-tothe-box.net> wrote:

> So...I'm almost certain that there was a way to change from the tab 
> character, to comma (I thought), but for the life of me I can't seem to 
> find it.  

http://bro.org/sphinx-git/scripts/base/frameworks/logging/writers/ascii.html#id-LogAscii::separator

Note that commas are already used for LogAscii::set_separator, but you can change that, too.

- Jon



From jlay at slave-tothe-box.net  Thu Oct 24 08:41:49 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 24 Oct 2013 09:41:49 -0600
Subject: [Bro] Changing separator
In-Reply-To: <00610fcd390f7ef7a725bdacfb1e9973@localhost>
References: <00610fcd390f7ef7a725bdacfb1e9973@localhost>
Message-ID: <aeb01f8fb52a12b5a23968f695b3effe@localhost>

On 2013-10-24 09:10, James Lay wrote:
> So...I'm almost certain that there was a way to change from the tab
> character, to comma (I thought), but for the life of me I can't seem 
> to
> find it.  Been working with logstash and currently messages come 
> through
> as:
>
> 
> 1382627138.211512\tCQ74U23HZlcab0LNnh\t192.168.1.3\t64079\t224.0.0.1\t8612\tudp\t-\t-\t-\t-\tS0\tT\t0\tD\t1\t44\t0\t0\t(empty)
>
> Which is kind of painful for matching.  Any quick pointers on how to 
> do
> this?  Thank you.
>
> James

Thanks for the responses gents...every little bit helps.

James



From jonschipp at gmail.com  Thu Oct 24 09:33:24 2013
From: jonschipp at gmail.com (Jon Schipp)
Date: Thu, 24 Oct 2013 11:33:24 -0500
Subject: [Bro] Yet Another Conference - like no other :)
In-Reply-To: <52657B3B.6040406@rsbac.org>
References: <525BCF76.6090908@rsbac.org>
	<CAFzrbiHdiFKkdppRnCcdgwQPGH-pTiOgypa5niqX9XtC3ALs2A@mail.gmail.com>
	<52657B3B.6040406@rsbac.org>
Message-ID: <CAB15j_BQ1gBAfYdFynsN5SbRzT80-uXTMhYKmoT=PnJmK-Be7A@mail.gmail.com>

Thanks for sharing Michal.


On Mon, Oct 21, 2013 at 2:06 PM, Michal Purzynski <michal at rsbac.org> wrote:

>
> > >
> > > Slides from my recent talk about NSM @ Mozilla at YaC 2013 are here, a
> > > full video will hopefully follow.
> > >
> > > http://tech.yandex.ru/events/yac/2013/talks/1131/
> > >
> >
> > Nice presentation, it confirms a few things I was suspecting :-)
> >
> > I see you are logging to elasticsearch from Bro... have you taken a
> > look at Moloch for full packet capture? It's not included in Security
> > Onion (yet?) but we have played with it at work and we're now
> > budgeting for Moloch boxes. Moloch just recently added support for
> > pfring as well, and from the mailing list I saw someone posting that
> > they were using pfring with success. It does a really good job of
> > indexing packet captures and has some protocol decoders built in...
> > I've found I don't even need to pull a pcap out of it half the time
> > because I get a clear picture from Moloch's web interface
> >
> > https://github.com/aol/moloch is their Github site
> >
> >
> Replacing netsniff-ng with anything else is possible here, but I don't
> feel like I need it - SO has a great integration between pcap agent,
> ELSA and Bro. I can go to ELSA, find the flow I need and request a
> transcript - simple and very effective.
>
> As for the metadata and data about my flows, content, protocol decoders,
> scripting - I would not change Bro for a 1024 kg of pure gold, if that's
> what you are asking :)
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Jon Schipp,
jonschipp.com, sickbits.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131024/359e8f03/attachment.html 

From jlay at slave-tothe-box.net  Thu Oct 24 14:48:10 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 24 Oct 2013 15:48:10 -0600
Subject: [Bro] Bro-IDS and Logstash....a start
Message-ID: <9052ad62fc8a6270e62c2ae96e4b0dc5@localhost>

And away we go!  Snag logstash 
(https://download.elasticsearch.org/logstash/logstash/logstash-1.2.1-flatjar.jar), 
make a dir and put it in there, create a file (logstash.conf) and add:

input {
         file {
                 path => "/usr/local/bro/spool/bro/conn.log"
         }
}

filter {
         grok {
                 match => [ "message", 
"%{BASE10NUM:unixtime}\t%{WORD:uid}\t%{IP:src_ip}\t%{BASE10NUM:src_port}\t%{IP:dst_ip}\t%{BASE10NUM:dst_port}\t%{WORD:proto}\t%{USERNAME:service}\t%{USERNAME:sec_dur}\t%{USERNAME:orig_bytes}\t%{USERNAME:resp_bytes}\t%{WORD:conn_state}\t%{WORD:local_orig}\t%{INT:missed_bytes}\t%{WORD:history}\t%{USERNAME:orig_packets}\t%{INT:orig_ip_bytes}\t%{INT:resp_packets}\t%{INT:resp_bytes}\t%{DATA:tun_parent}" 
]
         }
         geoip {
                 source => "src_ip"
                 target => "src_geoip"
                 fields => [ "ip", "country_code2", "country_name", 
"latitude", "longitude" ]
                 add_field => [ "coordinates", 
"%{[src_geoip][longitude]},%{[src_geoip][latitude]}" ]
                 add_field => [ "srccountry", 
"%{[src_geoip][country_code2]}"]
         }
         geoip {
                 source => "dst_ip"
                 target => "dst_geoip"
                 fields => [ "ip", "country_code2", "country_name", 
"latitude", "longitude" ]
                 add_field => [ "coordinates", 
"%{[dst_geoip][longitude]},%{[dst_geoip][latitude]}" ]
                 add_field => [ "dstcountry", 
"%{[dst_geoip][country_code2]}"]
         }

}
output {
         elasticsearch { embedded => true }
}

start with:
sudo java -jar logstash-1.2.2-flatjar.jar agent -f logstash.conf -- web

If you're local on net point your firefox to yourmachine:9292, I needed 
to tunnel 9200, 9300, and 9301 to get it to work remotely.  That's all I 
got currently..more to come I hope.  Enjoy!

James


From jeff.geiger at gmail.com  Fri Oct 25 13:23:01 2013
From: jeff.geiger at gmail.com (Jeff Geiger)
Date: Fri, 25 Oct 2013 15:23:01 -0500
Subject: [Bro] Bro-IDS and Logstash....a start
In-Reply-To: <9052ad62fc8a6270e62c2ae96e4b0dc5@localhost>
References: <9052ad62fc8a6270e62c2ae96e4b0dc5@localhost>
Message-ID: <CAMgL1RtDk6L8vwWM=CKUfMQQzf+UxY02wPO7XyQ-+XPkq4EgGg@mail.gmail.com>

James,  Check this out for a quick way to get grok filters and a stub
config for logstash:  http://brostash.herokuapp.com/


On Thu, Oct 24, 2013 at 4:48 PM, James Lay <jlay at slave-tothe-box.net> wrote:

> And away we go!  Snag logstash
> (
> https://download.elasticsearch.org/logstash/logstash/logstash-1.2.1-flatjar.jar
> ),
> make a dir and put it in there, create a file (logstash.conf) and add:
>
> input {
>          file {
>                  path => "/usr/local/bro/spool/bro/conn.log"
>          }
> }
>
> filter {
>          grok {
>                  match => [ "message",
>
> "%{BASE10NUM:unixtime}\t%{WORD:uid}\t%{IP:src_ip}\t%{BASE10NUM:src_port}\t%{IP:dst_ip}\t%{BASE10NUM:dst_port}\t%{WORD:proto}\t%{USERNAME:service}\t%{USERNAME:sec_dur}\t%{USERNAME:orig_bytes}\t%{USERNAME:resp_bytes}\t%{WORD:conn_state}\t%{WORD:local_orig}\t%{INT:missed_bytes}\t%{WORD:history}\t%{USERNAME:orig_packets}\t%{INT:orig_ip_bytes}\t%{INT:resp_packets}\t%{INT:resp_bytes}\t%{DATA:tun_parent}"
> ]
>          }
>          geoip {
>                  source => "src_ip"
>                  target => "src_geoip"
>                  fields => [ "ip", "country_code2", "country_name",
> "latitude", "longitude" ]
>                  add_field => [ "coordinates",
> "%{[src_geoip][longitude]},%{[src_geoip][latitude]}" ]
>                  add_field => [ "srccountry",
> "%{[src_geoip][country_code2]}"]
>          }
>          geoip {
>                  source => "dst_ip"
>                  target => "dst_geoip"
>                  fields => [ "ip", "country_code2", "country_name",
> "latitude", "longitude" ]
>                  add_field => [ "coordinates",
> "%{[dst_geoip][longitude]},%{[dst_geoip][latitude]}" ]
>                  add_field => [ "dstcountry",
> "%{[dst_geoip][country_code2]}"]
>          }
>
> }
> output {
>          elasticsearch { embedded => true }
> }
>
> start with:
> sudo java -jar logstash-1.2.2-flatjar.jar agent -f logstash.conf -- web
>
> If you're local on net point your firefox to yourmachine:9292, I needed
> to tunnel 9200, 9300, and 9301 to get it to work remotely.  That's all I
> got currently..more to come I hope.  Enjoy!
>
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131025/3801e5d1/attachment.html 

From jp.bourget at gmail.com  Sat Oct 26 13:26:01 2013
From: jp.bourget at gmail.com (JP Bourget)
Date: Sat, 26 Oct 2013 15:26:01 -0500
Subject: [Bro] Compiling git master on Mavericks (OSX 10.9)
Message-ID: <CAGb0Yipozp-8U5vO6szeeEPzrvYoNyWTa-pCendhaugMT=x0og@mail.gmail.com>

Hey there -

So I'm trying to get git master to compile on Mac OSX 10.9

What I've done so far:

update XCode to 5.0.1
update XCode command line tools: xcode-select --install

./configure --prefix=/usr/local/bro (works fine)
make install - fails with this:

Building C object
aux/broccoli/bindings/broccoli-ruby/CMakeFiles/broccoli_ext.dir/ext/broccoli_ext/broccoli_internRUBY_wrap.c.o

*/tmp/bro/build/aux/broccoli/bindings/broccoli-ruby/ext/broccoli_ext/broccoli_internRUBY_wrap.c:1851:10:
**fatal error: **'rubyio.h' file not found*

#include "rubyio.h"



Any suggestions?

-- 
JP
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131026/99a55400/attachment.html 

From michal at rsbac.org  Mon Oct 28 08:53:50 2013
From: michal at rsbac.org (Michal Purzynski)
Date: Mon, 28 Oct 2013 16:53:50 +0100
Subject: [Bro] Bro-IDS and Logstash....a start
In-Reply-To: <CAMgL1RtDk6L8vwWM=CKUfMQQzf+UxY02wPO7XyQ-+XPkq4EgGg@mail.gmail.com>
References: <9052ad62fc8a6270e62c2ae96e4b0dc5@localhost>
	<CAMgL1RtDk6L8vwWM=CKUfMQQzf+UxY02wPO7XyQ-+XPkq4EgGg@mail.gmail.com>
Message-ID: <526E888E.3090700@rsbac.org>

On 10/25/13, 10:23 PM, Jeff Geiger wrote:
> James,  Check this out for a quick way to get grok filters and a stub 
> config for logstash: http://brostash.herokuapp.com/
>
Hm, do you have a source for this web application? It would be a pity to 
have it disappear some day as it's pretty much awesome.



From jsiwek at illinois.edu  Mon Oct 28 09:17:25 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 28 Oct 2013 16:17:25 +0000
Subject: [Bro] Compiling git master on Mavericks (OSX 10.9)
In-Reply-To: <CAGb0Yipozp-8U5vO6szeeEPzrvYoNyWTa-pCendhaugMT=x0og@mail.gmail.com>
References: <CAGb0Yipozp-8U5vO6szeeEPzrvYoNyWTa-pCendhaugMT=x0og@mail.gmail.com>
Message-ID: <832958D4-16FB-4C54-BCF4-48090EE59C2D@illinois.edu>


On Oct 26, 2013, at 3:26 PM, JP Bourget <jp.bourget at gmail.com> wrote:

> So I'm trying to get git master to compile on Mac OSX 10.9

> /tmp/bro/build/aux/broccoli/bindings/broccoli-ruby/ext/broccoli_ext/broccoli_internRUBY_wrap.c:1851:10: fatal error: 'rubyio.h' file not found

A workaround for if you don?t need to use the ruby bindings for broccoli (you?d know if you did):

    ./configure ?disable-ruby

I don?t have a fix/workaround at the moment for actually getting the ruby bindings to compile on Mavericks (at least w/ MacPorts? swig-ruby).

- Jon


From jp.bourget at gmail.com  Mon Oct 28 12:06:07 2013
From: jp.bourget at gmail.com (JP Bourget)
Date: Mon, 28 Oct 2013 14:06:07 -0500
Subject: [Bro] Compiling git master on Mavericks (OSX 10.9)
In-Reply-To: <832958D4-16FB-4C54-BCF4-48090EE59C2D@illinois.edu>
References: <CAGb0Yipozp-8U5vO6szeeEPzrvYoNyWTa-pCendhaugMT=x0og@mail.gmail.com>
	<832958D4-16FB-4C54-BCF4-48090EE59C2D@illinois.edu>
Message-ID: <CAGb0Yir_XiD=9RhU_9wKnHTg4sAdE1hQ+wFWnUSwrOOq00gRfg@mail.gmail.com>

That worked like a charm, thanks John!


On Mon, Oct 28, 2013 at 11:17 AM, Siwek, Jonathan Luke
<jsiwek at illinois.edu>wrote:

>
> On Oct 26, 2013, at 3:26 PM, JP Bourget <jp.bourget at gmail.com> wrote:
>
> > So I'm trying to get git master to compile on Mac OSX 10.9
>
> >
> /tmp/bro/build/aux/broccoli/bindings/broccoli-ruby/ext/broccoli_ext/broccoli_internRUBY_wrap.c:1851:10:
> fatal error: 'rubyio.h' file not found
>
> A workaround for if you don?t need to use the ruby bindings for broccoli
> (you?d know if you did):
>
>     ./configure ?disable-ruby
>
> I don?t have a fix/workaround at the moment for actually getting the ruby
> bindings to compile on Mavericks (at least w/ MacPorts? swig-ruby).
>
> - Jon




-- 
JP
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131028/e971fbdf/attachment.html 

From ben.bt.wood at gmail.com  Mon Oct 28 15:02:25 2013
From: ben.bt.wood at gmail.com (Benjamin Wood)
Date: Mon, 28 Oct 2013 18:02:25 -0400
Subject: [Bro] Compiling Bro with Endace Dag Support
Message-ID: <CA+i_NU_oGFDn-eWb1n4XCvMU=GVFf--0SLQqjhhWwfNfZtOGhQ@mail.gmail.com>

All,

Is this functionality still supported?

I believe I've been able to get bro to use libpcap compiled with dag
libraries, but is there native support?

I remember reading somewhere that I could run "./configure --with-dag" to
compile bro with dag support.

If not I'll just recompile libpcap with the dag libraries, and then
recompile bro with the new libpcap.

Thanks,
Ben Wood
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131028/3613e707/attachment.html 

From mike.patterson at uwaterloo.ca  Mon Oct 28 15:50:17 2013
From: mike.patterson at uwaterloo.ca (Mike Patterson)
Date: Mon, 28 Oct 2013 18:50:17 -0400
Subject: [Bro] Compiling Bro with Endace Dag Support
In-Reply-To: <CA+i_NU_oGFDn-eWb1n4XCvMU=GVFf--0SLQqjhhWwfNfZtOGhQ@mail.gmail.com>
References: <CA+i_NU_oGFDn-eWb1n4XCvMU=GVFf--0SLQqjhhWwfNfZtOGhQ@mail.gmail.com>
Message-ID: <E06F7058-B70C-4201-BE60-B1957626FA73@uwaterloo.ca>

Can?t speak for what?s actually supported, but I?ve not been disappointed with using libpcap with DAG, bro vs that libpcap.

That said, use pcap 1.3.0. 1.4.0 had some issues, can?t recall their precise nature but ?they don?t work at all? sticks in the mind. :)

Mike

On Oct 28, 2013, at 6:02 PM, Benjamin Wood <ben.bt.wood at gmail.com> wrote:

> All,
> 
> Is this functionality still supported?
> 
> I believe I've been able to get bro to use libpcap compiled with dag libraries, but is there native support?
> 
> I remember reading somewhere that I could run "./configure --with-dag" to compile bro with dag support.
> 
> If not I'll just recompile libpcap with the dag libraries, and then recompile bro with the new libpcap.
> 
> Thanks,
> Ben Wood
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From hckim at narusec.com  Mon Oct 28 19:21:56 2013
From: hckim at narusec.com (=?UTF-8?B?6rmA7Z2s7LKg?=)
Date: Tue, 29 Oct 2013 11:21:56 +0900
Subject: [Bro] Bro-IDS and Logstash....a start (James Lay)
Message-ID: <CAHs13Ai80+G40T8L=fDuY=NqJmrCROJBK3MiqCijLZjw6QteMA@mail.gmail.com>

Hi
I am having trouble  with logstash

I followed  your direction  but when I run
java -jar logstash-1.2.2-flatjar.jar agent -f logstash.conf -- web

I got error message

The error reported is:
  pattern %{BASE10NUM:unixtime} not defined

here is some related link  <https://logstash.jira.com/browse/LOGSTASH-1316>

is there a way to run with web?


On Sat, Oct 26, 2013 at 4:00 AM, <bro-request at bro.org> wrote:

> Send Bro mailing list submissions to
>         bro at bro.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> or, via email, send a message with subject or body 'help' to
>         bro-request at bro.org
>
> You can reach the person managing the list at
>         bro-owner at bro.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Bro digest..."
>
>
> Today's Topics:
>
>    1. Bro-IDS and Logstash....a start (James Lay)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 24 Oct 2013 15:48:10 -0600
> From: James Lay <jlay at slave-tothe-box.net>
> Subject: [Bro] Bro-IDS and Logstash....a start
> To: Bro <bro at bro.org>
> Message-ID: <9052ad62fc8a6270e62c2ae96e4b0dc5 at localhost>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> And away we go!  Snag logstash
> (
> https://download.elasticsearch.org/logstash/logstash/logstash-1.2.1-flatjar.jar
> ),
> make a dir and put it in there, create a file (logstash.conf) and add:
>
> input {
>          file {
>                  path => "/usr/local/bro/spool/bro/conn.log"
>          }
> }
>
> filter {
>          grok {
>                  match => [ "message",
>
> "%{BASE10NUM:unixtime}\t%{WORD:uid}\t%{IP:src_ip}\t%{BASE10NUM:src_port}\t%{IP:dst_ip}\t%{BASE10NUM:dst_port}\t%{WORD:proto}\t%{USERNAME:service}\t%{USERNAME:sec_dur}\t%{USERNAME:orig_bytes}\t%{USERNAME:resp_bytes}\t%{WORD:conn_state}\t%{WORD:local_orig}\t%{INT:missed_bytes}\t%{WORD:history}\t%{USERNAME:orig_packets}\t%{INT:orig_ip_bytes}\t%{INT:resp_packets}\t%{INT:resp_bytes}\t%{DATA:tun_parent}"
> ]
>          }
>          geoip {
>                  source => "src_ip"
>                  target => "src_geoip"
>                  fields => [ "ip", "country_code2", "country_name",
> "latitude", "longitude" ]
>                  add_field => [ "coordinates",
> "%{[src_geoip][longitude]},%{[src_geoip][latitude]}" ]
>                  add_field => [ "srccountry",
> "%{[src_geoip][country_code2]}"]
>          }
>          geoip {
>                  source => "dst_ip"
>                  target => "dst_geoip"
>                  fields => [ "ip", "country_code2", "country_name",
> "latitude", "longitude" ]
>                  add_field => [ "coordinates",
> "%{[dst_geoip][longitude]},%{[dst_geoip][latitude]}" ]
>                  add_field => [ "dstcountry",
> "%{[dst_geoip][country_code2]}"]
>          }
>
> }
> output {
>          elasticsearch { embedded => true }
> }
>
> start with:
> sudo java -jar logstash-1.2.2-flatjar.jar agent -f logstash.conf -- web
>
> If you're local on net point your firefox to yourmachine:9292, I needed
> to tunnel 9200, 9300, and 9301 to get it to work remotely.  That's all I
> got currently..more to come I hope.  Enjoy!
>
> James
>
>
> ------------------------------
>
> _______________________________________________
> Bro mailing list
> Bro at bro.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> End of Bro Digest, Vol 90, Issue 29
> ***********************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131029/988753a4/attachment.html 

From jeff.geiger at gmail.com  Mon Oct 28 19:35:43 2013
From: jeff.geiger at gmail.com (Jeff Geiger)
Date: Mon, 28 Oct 2013 21:35:43 -0500
Subject: [Bro] Bro-IDS and Logstash....a start (James Lay)
In-Reply-To: <CAHs13Ai80+G40T8L=fDuY=NqJmrCROJBK3MiqCijLZjw6QteMA@mail.gmail.com>
References: <CAHs13Ai80+G40T8L=fDuY=NqJmrCROJBK3MiqCijLZjw6QteMA@mail.gmail.com>
Message-ID: <5534832284967263016@unknownmsgid>

Sounds like you are missing the base patterns.

See:
https://github.com/logstash/logstash/blob/master/patterns/grok-patterns


- Jeff

On Oct 28, 2013, at 21:30, "???" <hckim at narusec.com> wrote:

Hi
I am having trouble  with logstash

I followed  your direction  but when I run
java -jar logstash-1.2.2-flatjar.jar agent -f logstash.conf -- web

I got error message

The error reported is:
  pattern %{BASE10NUM:unixtime} not defined

here is some related link  <https://logstash.jira.com/browse/LOGSTASH-1316>

is there a way to run with web?


On Sat, Oct 26, 2013 at 4:00 AM, <bro-request at bro.org> wrote:

> Send Bro mailing list submissions to
>         bro at bro.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> or, via email, send a message with subject or body 'help' to
>         bro-request at bro.org
>
> You can reach the person managing the list at
>         bro-owner at bro.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Bro digest..."
>
>
> Today's Topics:
>
>    1. Bro-IDS and Logstash....a start (James Lay)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 24 Oct 2013 15:48:10 -0600
> From: James Lay <jlay at slave-tothe-box.net>
> Subject: [Bro] Bro-IDS and Logstash....a start
> To: Bro <bro at bro.org>
> Message-ID: <9052ad62fc8a6270e62c2ae96e4b0dc5 at localhost>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> And away we go!  Snag logstash
> (
> https://download.elasticsearch.org/logstash/logstash/logstash-1.2.1-flatjar.jar
> ),
> make a dir and put it in there, create a file (logstash.conf) and add:
>
> input {
>          file {
>                  path => "/usr/local/bro/spool/bro/conn.log"
>          }
> }
>
> filter {
>          grok {
>                  match => [ "message",
>
> "%{BASE10NUM:unixtime}\t%{WORD:uid}\t%{IP:src_ip}\t%{BASE10NUM:src_port}\t%{IP:dst_ip}\t%{BASE10NUM:dst_port}\t%{WORD:proto}\t%{USERNAME:service}\t%{USERNAME:sec_dur}\t%{USERNAME:orig_bytes}\t%{USERNAME:resp_bytes}\t%{WORD:conn_state}\t%{WORD:local_orig}\t%{INT:missed_bytes}\t%{WORD:history}\t%{USERNAME:orig_packets}\t%{INT:orig_ip_bytes}\t%{INT:resp_packets}\t%{INT:resp_bytes}\t%{DATA:tun_parent}"
> ]
>          }
>          geoip {
>                  source => "src_ip"
>                  target => "src_geoip"
>                  fields => [ "ip", "country_code2", "country_name",
> "latitude", "longitude" ]
>                  add_field => [ "coordinates",
> "%{[src_geoip][longitude]},%{[src_geoip][latitude]}" ]
>                  add_field => [ "srccountry",
> "%{[src_geoip][country_code2]}"]
>          }
>          geoip {
>                  source => "dst_ip"
>                  target => "dst_geoip"
>                  fields => [ "ip", "country_code2", "country_name",
> "latitude", "longitude" ]
>                  add_field => [ "coordinates",
> "%{[dst_geoip][longitude]},%{[dst_geoip][latitude]}" ]
>                  add_field => [ "dstcountry",
> "%{[dst_geoip][country_code2]}"]
>          }
>
> }
> output {
>          elasticsearch { embedded => true }
> }
>
> start with:
> sudo java -jar logstash-1.2.2-flatjar.jar agent -f logstash.conf -- web
>
> If you're local on net point your firefox to yourmachine:9292, I needed
> to tunnel 9200, 9300, and 9301 to get it to work remotely.  That's all I
> got currently..more to come I hope.  Enjoy!
>
> James
>
>
> ------------------------------
>
> _______________________________________________
> Bro mailing list
> Bro at bro.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> End of Bro Digest, Vol 90, Issue 29
> ***********************************
>

_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131028/de0f9476/attachment.html 

From jlay at slave-tothe-box.net  Mon Oct 28 19:42:24 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Mon, 28 Oct 2013 20:42:24 -0600
Subject: [Bro] Bro-IDS and Logstash....a start (James Lay)
In-Reply-To: <CAHs13Ai80+G40T8L=fDuY=NqJmrCROJBK3MiqCijLZjw6QteMA@mail.gmail.com>
References: <CAHs13Ai80+G40T8L=fDuY=NqJmrCROJBK3MiqCijLZjw6QteMA@mail.gmail.com>
Message-ID: <413FA484-4C7D-4592-A1D7-9C79A5668908@slave-tothe-box.net>

Run it again with sudo?.you?ll most likely get another error, then, run it one more time?3rd time it will work.  Why I have no idea :)

James

On Oct 28, 2013, at 8:21 PM, ??? <hckim at narusec.com> wrote:

> Hi
> I am having trouble  with logstash
> 
> I followed  your direction  but when I run
> java -jar logstash-1.2.2-flatjar.jar agent -f logstash.conf -- web
> 
> I got error message
> 
> The error reported is: 
>   pattern %{BASE10NUM:unixtime} not defined
> 
> here is some related link 
> 
> is there a way to run with web?
> 
> 
> On Sat, Oct 26, 2013 at 4:00 AM, <bro-request at bro.org> wrote:
> Send Bro mailing list submissions to
>         bro at bro.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> or, via email, send a message with subject or body 'help' to
>         bro-request at bro.org
> 
> You can reach the person managing the list at
>         bro-owner at bro.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Bro digest..."
> 
> 
> Today's Topics:
> 
>    1. Bro-IDS and Logstash....a start (James Lay)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Thu, 24 Oct 2013 15:48:10 -0600
> From: James Lay <jlay at slave-tothe-box.net>
> Subject: [Bro] Bro-IDS and Logstash....a start
> To: Bro <bro at bro.org>
> Message-ID: <9052ad62fc8a6270e62c2ae96e4b0dc5 at localhost>
> Content-Type: text/plain; charset=UTF-8; format=flowed
> 
> And away we go!  Snag logstash
> (https://download.elasticsearch.org/logstash/logstash/logstash-1.2.1-flatjar.jar),
> make a dir and put it in there, create a file (logstash.conf) and add:
> 
> input {
>          file {
>                  path => "/usr/local/bro/spool/bro/conn.log"
>          }
> }
> 
> filter {
>          grok {
>                  match => [ "message",
> "%{BASE10NUM:unixtime}\t%{WORD:uid}\t%{IP:src_ip}\t%{BASE10NUM:src_port}\t%{IP:dst_ip}\t%{BASE10NUM:dst_port}\t%{WORD:proto}\t%{USERNAME:service}\t%{USERNAME:sec_dur}\t%{USERNAME:orig_bytes}\t%{USERNAME:resp_bytes}\t%{WORD:conn_state}\t%{WORD:local_orig}\t%{INT:missed_bytes}\t%{WORD:history}\t%{USERNAME:orig_packets}\t%{INT:orig_ip_bytes}\t%{INT:resp_packets}\t%{INT:resp_bytes}\t%{DATA:tun_parent}"
> ]
>          }
>          geoip {
>                  source => "src_ip"
>                  target => "src_geoip"
>                  fields => [ "ip", "country_code2", "country_name",
> "latitude", "longitude" ]
>                  add_field => [ "coordinates",
> "%{[src_geoip][longitude]},%{[src_geoip][latitude]}" ]
>                  add_field => [ "srccountry",
> "%{[src_geoip][country_code2]}"]
>          }
>          geoip {
>                  source => "dst_ip"
>                  target => "dst_geoip"
>                  fields => [ "ip", "country_code2", "country_name",
> "latitude", "longitude" ]
>                  add_field => [ "coordinates",
> "%{[dst_geoip][longitude]},%{[dst_geoip][latitude]}" ]
>                  add_field => [ "dstcountry",
> "%{[dst_geoip][country_code2]}"]
>          }
> 
> }
> output {
>          elasticsearch { embedded => true }
> }
> 
> start with:
> sudo java -jar logstash-1.2.2-flatjar.jar agent -f logstash.conf -- web
> 
> If you're local on net point your firefox to yourmachine:9292, I needed
> to tunnel 9200, 9300, and 9301 to get it to work remotely.  That's all I
> got currently..more to come I hope.  Enjoy!
> 
> James
> 
> 
> ------------------------------
> 
> _______________________________________________
> Bro mailing list
> Bro at bro.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> End of Bro Digest, Vol 90, Issue 29
> ***********************************
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131028/da313785/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131028/da313785/attachment.bin 

From jeff.geiger at gmail.com  Mon Oct 28 22:45:21 2013
From: jeff.geiger at gmail.com (Jeff Geiger)
Date: Tue, 29 Oct 2013 00:45:21 -0500
Subject: [Bro] Bro-IDS and Logstash....a start
In-Reply-To: <526E888E.3090700@rsbac.org>
References: <9052ad62fc8a6270e62c2ae96e4b0dc5@localhost>
	<CAMgL1RtDk6L8vwWM=CKUfMQQzf+UxY02wPO7XyQ-+XPkq4EgGg@mail.gmail.com>
	<526E888E.3090700@rsbac.org>
Message-ID: <CAMgL1Rt8+jtRG0_YdroGXQrmPacMTvcMc8MByfd8=cut-m8H5w@mail.gmail.com>

Michal,

https://github.com/jeffgeiger/brostash

It runs well on Docker as well.

~Jeff Geiger

On Mon, Oct 28, 2013 at 10:53 AM, Michal Purzynski <michal at rsbac.org> wrote:

> On 10/25/13, 10:23 PM, Jeff Geiger wrote:
> > James,  Check this out for a quick way to get grok filters and a stub
> > config for logstash: http://brostash.herokuapp.com/
> >
> Hm, do you have a source for this web application? It would be a pity to
> have it disappear some day as it's pretty much awesome.
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131029/4bdeff2d/attachment.html 

From hiren.panchasara at gmail.com  Mon Oct 28 23:28:12 2013
From: hiren.panchasara at gmail.com (hiren panchasara)
Date: Mon, 28 Oct 2013 23:28:12 -0700
Subject: [Bro] Cluster setup
Message-ID: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>

Hi All,

Here is what I am trying to achieve: Incoming traffic on Host-A should
be sent to worker Host-B (and to more workers in future).

Here is how my config looks like in node.cfg

Manager: Host-A
Proxy:     Host-A
Worker1: Host-B (which is 10.73.149.31)

I have bro installed on all machines. Now, I start broctl on Host-A:

-bash-4.2$ sudo broctl
Password:

Welcome to BroControl 1.1

Type "help" for help.

[BroControl] > install
removing old policies in
/usr/local/spool/installed-scripts-do-not-touch/site ... done.
removing old policies in
/usr/local/spool/installed-scripts-do-not-touch/auto ... done.
creating policy directories ... done.
installing site policies ... done.
generating cluster-layout.bro ... done.
generating local-networks.bro ... done.
generating broctl-config.bro ... done.
updating nodes ... warning: host 10.73.149.31 is not alive
done.
[BroControl] > install
removing old policies in
/usr/local/spool/installed-scripts-do-not-touch/site ... done.
removing old policies in
/usr/local/spool/installed-scripts-do-not-touch/auto ... done.
creating policy directories ... done.
installing site policies ... done.
generating cluster-layout.bro ... done.
generating local-networks.bro ... done.
generating broctl-config.bro ... done.
updating nodes ... done.
[BroControl] > start
starting manager ...
starting proxy-1 ...
starting worker-1 ...
cannot create working directory for worker-1   <<-- not sure why I get
this message.
[BroControl] >

Do I need to do anything on Worker-1?? Do I need to put it in some special
mode?

Any help/pointers would be appreciated.

Cheers,
Hiren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131028/78294374/attachment.html 

From dnthayer at illinois.edu  Mon Oct 28 23:50:51 2013
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Tue, 29 Oct 2013 01:50:51 -0500
Subject: [Bro] Cluster setup
In-Reply-To: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
Message-ID: <526F5ACB.4080404@illinois.edu>

Did you verify that you can ssh from host-A to host-B without
having to type a password?

Next, on host-B, verify that the partition where /usr/local/spool
is located is not mounted read-only and that there is some free
disk space (broctl is trying to create a directory
in /usr/local/spool on host-B).



On 10/29/2013 01:28 AM, hiren panchasara wrote:
>
> Hi All,
>
> Here is what I am trying to achieve: Incoming traffic on Host-A should
> be sent to worker Host-B (and to more workers in future).
>
> Here is how my config looks like in node.cfg
>
> Manager: Host-A
> Proxy:     Host-A
> Worker1: Host-B (which is 10.73.149.31)
>
> I have bro installed on all machines. Now, I start broctl on Host-A:
>
> -bash-4.2$ sudo broctl
> Password:
>
> Welcome to BroControl 1.1
>
> Type "help" for help.
>
> [BroControl] > install
> removing old policies in
> /usr/local/spool/installed-scripts-do-not-touch/site ... done.
> removing old policies in
> /usr/local/spool/installed-scripts-do-not-touch/auto ... done.
> creating policy directories ... done.
> installing site policies ... done.
> generating cluster-layout.bro ... done.
> generating local-networks.bro ... done.
> generating broctl-config.bro ... done.
> updating nodes ... warning: host 10.73.149.31 is not alive
> done.
> [BroControl] > install
> removing old policies in
> /usr/local/spool/installed-scripts-do-not-touch/site ... done.
> removing old policies in
> /usr/local/spool/installed-scripts-do-not-touch/auto ... done.
> creating policy directories ... done.
> installing site policies ... done.
> generating cluster-layout.bro ... done.
> generating local-networks.bro ... done.
> generating broctl-config.bro ... done.
> updating nodes ... done.
> [BroControl] > start
> starting manager ...
> starting proxy-1 ...
> starting worker-1 ...
> cannot create working directory for worker-1   <<-- not sure why I get
> this message.
> [BroControl] >
>
> Do I need to do anything on Worker-1?? Do I need to put it in some
> special mode?
>
> Any help/pointers would be appreciated.
>
> Cheers,
> Hiren
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



From robin at icir.org  Tue Oct 29 06:33:30 2013
From: robin at icir.org (Robin Sommer)
Date: Tue, 29 Oct 2013 06:33:30 -0700
Subject: [Bro] Compiling Bro with Endace Dag Support
In-Reply-To: <CA+i_NU_oGFDn-eWb1n4XCvMU=GVFf--0SLQqjhhWwfNfZtOGhQ@mail.gmail.com>
References: <CA+i_NU_oGFDn-eWb1n4XCvMU=GVFf--0SLQqjhhWwfNfZtOGhQ@mail.gmail.com>
Message-ID: <20131029133330.GF72433@icir.org>



On Mon, Oct 28, 2013 at 18:02 -0400, you wrote:

> Is this functionality still supported?

The native DAG support was removed a while ago but I believe the pcap
wrapper should work fine.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 *     robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 * www.icir.org/robin


From hiren.panchasara at gmail.com  Tue Oct 29 10:09:07 2013
From: hiren.panchasara at gmail.com (hiren panchasara)
Date: Tue, 29 Oct 2013 10:09:07 -0700
Subject: [Bro] Cluster setup
In-Reply-To: <526F5ACB.4080404@illinois.edu>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
	<526F5ACB.4080404@illinois.edu>
Message-ID: <CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>

On Mon, Oct 28, 2013 at 11:50 PM, Daniel Thayer <dnthayer at illinois.edu> wrote:
> Did you verify that you can ssh from host-A to host-B without
> having to type a password?
Just set that up.
>
> Next, on host-B, verify that the partition where /usr/local/spool
> is located is not mounted read-only and that there is some free
> disk space (broctl is trying to create a directory
> in /usr/local/spool on host-B).
Checked this too.

Still,

[BroControl] > install
removing old policies in
/usr/local/spool/installed-scripts-do-not-touch/site ... done.
removing old policies in
/usr/local/spool/installed-scripts-do-not-touch/auto ... done.
creating policy directories ... done.
installing site policies ... done.
generating cluster-layout.bro ... done.
generating local-networks.bro ... done.
generating broctl-config.bro ... done.
updating nodes ... warning: host 10.73.149.31 is not alive
done.
[BroControl] >

What  does that mean? I still cannot get worker-1 to work properly.

in "top" (inside broctl) also, worker-1 is shown <not running>

Do I need to setup anything on worker-1?

Cheers,
Hiren


>
>
>
>
> On 10/29/2013 01:28 AM, hiren panchasara wrote:
>>
>>
>> Hi All,
>>
>> Here is what I am trying to achieve: Incoming traffic on Host-A should
>> be sent to worker Host-B (and to more workers in future).
>>
>> Here is how my config looks like in node.cfg
>>
>> Manager: Host-A
>> Proxy:     Host-A
>> Worker1: Host-B (which is 10.73.149.31)
>>
>> I have bro installed on all machines. Now, I start broctl on Host-A:
>>
>> -bash-4.2$ sudo broctl
>> Password:
>>
>> Welcome to BroControl 1.1
>>
>> Type "help" for help.
>>
>> [BroControl] > install
>> removing old policies in
>> /usr/local/spool/installed-scripts-do-not-touch/site ... done.
>> removing old policies in
>> /usr/local/spool/installed-scripts-do-not-touch/auto ... done.
>> creating policy directories ... done.
>> installing site policies ... done.
>> generating cluster-layout.bro ... done.
>> generating local-networks.bro ... done.
>> generating broctl-config.bro ... done.
>> updating nodes ... warning: host 10.73.149.31 is not alive
>> done.
>> [BroControl] > install
>> removing old policies in
>> /usr/local/spool/installed-scripts-do-not-touch/site ... done.
>> removing old policies in
>> /usr/local/spool/installed-scripts-do-not-touch/auto ... done.
>> creating policy directories ... done.
>> installing site policies ... done.
>> generating cluster-layout.bro ... done.
>> generating local-networks.bro ... done.
>> generating broctl-config.bro ... done.
>> updating nodes ... done.
>> [BroControl] > start
>> starting manager ...
>> starting proxy-1 ...
>> starting worker-1 ...
>> cannot create working directory for worker-1   <<-- not sure why I get
>> this message.
>> [BroControl] >
>>
>> Do I need to do anything on Worker-1?? Do I need to put it in some
>> special mode?
>>
>> Any help/pointers would be appreciated.
>>
>> Cheers,
>> Hiren
>>
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>


From hiren.panchasara at gmail.com  Tue Oct 29 10:50:43 2013
From: hiren.panchasara at gmail.com (hiren panchasara)
Date: Tue, 29 Oct 2013 10:50:43 -0700
Subject: [Bro] Cluster setup
In-Reply-To: <CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
	<526F5ACB.4080404@illinois.edu>
	<CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>
Message-ID: <CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>

On Tue, Oct 29, 2013 at 10:09 AM, hiren panchasara
<hiren.panchasara at gmail.com> wrote:
> On Mon, Oct 28, 2013 at 11:50 PM, Daniel Thayer <dnthayer at illinois.edu> wrote:
>> Did you verify that you can ssh from host-A to host-B without
>> having to type a password?
> Just set that up.

Ah, I realized that I had to do this as "root" because broctl is run as root :-)

Set that up and now:

[BroControl] > install
removing old policies in
/usr/local/spool/installed-scripts-do-not-touch/site ... done.
removing old policies in
/usr/local/spool/installed-scripts-do-not-touch/auto ... done.
creating policy directories ... done.
installing site policies ... done.
generating cluster-layout.bro ... done.
generating local-networks.bro ... done.
generating broctl-config.bro ... done.
updating nodes ... warning: host 10.73.149.31 is not alive
done.
[BroControl] > check
manager is ok.
proxy-1 is ok.
worker-1 is ok.
[BroControl] > start
starting manager ...
starting proxy-1 ...
starting worker-1 ...
worker-1 terminated immediately after starting; check output with "diag"

[BroControl] > diag worker-1
[worker-1]

==== No reporter.log

==== stderr.log
error in /usr/local/share/bro/base/frameworks/cluster/__load__.bro,
line 16: can't open cluster-layout

==== stdout.log
unlimited
536870912
unlimited

==== .cmdline
-i bce1 -U .status -p broctl -p broctl-live -p local -p worker-1
local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto

==== .env_vars
PATH=/usr/local/bin:/usr/local/share/broctl/scripts:/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/y/bin:/root/bin
BROPATH=/usr/local/spool/installed-scripts-do-not-touch/site::/usr/local/spool/installed-scripts-do-not-touch/auto:/usr/local/share/bro:/usr/local/share/bro/policy:/usr/local/share/bro/site
CLUSTER_NODE=worker-1

==== .status
TERMINATED [atexit]

==== No prof.log

==== No packet_filter.log

==== No loaded_scripts.log
[BroControl] >


Trying to determine what is causing this.

cheers,
Hiren


From hiren.panchasara at gmail.com  Tue Oct 29 11:04:31 2013
From: hiren.panchasara at gmail.com (hiren panchasara)
Date: Tue, 29 Oct 2013 11:04:31 -0700
Subject: [Bro] Cluster setup
In-Reply-To: <CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
	<526F5ACB.4080404@illinois.edu>
	<CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>
	<CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>
Message-ID: <CALCpEUGKzcx6vD7fMsYmpZe1AR8ej+zz-cgUJJ0yjMWO4-gEXg@mail.gmail.com>

On Tue, Oct 29, 2013 at 10:50 AM, hiren panchasara
<hiren.panchasara at gmail.com> wrote:
> On Tue, Oct 29, 2013 at 10:09 AM, hiren panchasara
> <hiren.panchasara at gmail.com> wrote:
>> On Mon, Oct 28, 2013 at 11:50 PM, Daniel Thayer <dnthayer at illinois.edu> wrote:
>>> Did you verify that you can ssh from host-A to host-B without
>>> having to type a password?
>> Just set that up.
>
> Ah, I realized that I had to do this as "root" because broctl is run as root :-)
>
> Set that up and now:
>
> [BroControl] > install
> removing old policies in
> /usr/local/spool/installed-scripts-do-not-touch/site ... done.
> removing old policies in
> /usr/local/spool/installed-scripts-do-not-touch/auto ... done.
> creating policy directories ... done.
> installing site policies ... done.
> generating cluster-layout.bro ... done.
> generating local-networks.bro ... done.
> generating broctl-config.bro ... done.
> updating nodes ... warning: host 10.73.149.31 is not alive
> done.
> [BroControl] > check
> manager is ok.
> proxy-1 is ok.
> worker-1 is ok.
> [BroControl] > start
> starting manager ...
> starting proxy-1 ...
> starting worker-1 ...
> worker-1 terminated immediately after starting; check output with "diag"
>
> [BroControl] > diag worker-1
> [worker-1]
>
> ==== No reporter.log
>
> ==== stderr.log
> error in /usr/local/share/bro/base/frameworks/cluster/__load__.bro,
> line 16: can't open cluster-layout
>
> ==== stdout.log
> unlimited
> 536870912
> unlimited
>
> ==== .cmdline
> -i bce1 -U .status -p broctl -p broctl-live -p local -p worker-1
> local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
>
> ==== .env_vars
> PATH=/usr/local/bin:/usr/local/share/broctl/scripts:/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/y/bin:/root/bin
> BROPATH=/usr/local/spool/installed-scripts-do-not-touch/site::/usr/local/spool/installed-scripts-do-not-touch/auto:/usr/local/share/bro:/usr/local/share/bro/policy:/usr/local/share/bro/site
> CLUSTER_NODE=worker-1
>
> ==== .status
> TERMINATED [atexit]
>
> ==== No prof.log
>
> ==== No packet_filter.log
>
> ==== No loaded_scripts.log
> [BroControl] >
>
>
> Trying to determine what is causing this.

I am not able to find/understand what is causing this problem.

Anyone with some clue?

Thanks in advance,
Hiren


From dnthayer at illinois.edu  Tue Oct 29 11:01:28 2013
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Tue, 29 Oct 2013 13:01:28 -0500
Subject: [Bro] Cluster setup
In-Reply-To: <CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
	<526F5ACB.4080404@illinois.edu>
	<CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>
	<CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>
Message-ID: <526FF7F8.3020704@illinois.edu>

Exit from broctl, then verify that you can ping the
worker machine from the manager machine.  If that works,
then do another "broctl install" and make sure you
don't see any error or warning messages.


On 10/29/2013 12:50 PM, hiren panchasara wrote:
> On Tue, Oct 29, 2013 at 10:09 AM, hiren panchasara
> <hiren.panchasara at gmail.com> wrote:
>> On Mon, Oct 28, 2013 at 11:50 PM, Daniel Thayer <dnthayer at illinois.edu> wrote:
>>> Did you verify that you can ssh from host-A to host-B without
>>> having to type a password?
>> Just set that up.
>
> Ah, I realized that I had to do this as "root" because broctl is run as root :-)
>
> Set that up and now:
>
> [BroControl] > install
> removing old policies in
> /usr/local/spool/installed-scripts-do-not-touch/site ... done.
> removing old policies in
> /usr/local/spool/installed-scripts-do-not-touch/auto ... done.
> creating policy directories ... done.
> installing site policies ... done.
> generating cluster-layout.bro ... done.
> generating local-networks.bro ... done.
> generating broctl-config.bro ... done.
> updating nodes ... warning: host 10.73.149.31 is not alive
> done.
> [BroControl] > check
> manager is ok.
> proxy-1 is ok.
> worker-1 is ok.
> [BroControl] > start
> starting manager ...
> starting proxy-1 ...
> starting worker-1 ...
> worker-1 terminated immediately after starting; check output with "diag"
>
> [BroControl] > diag worker-1
> [worker-1]
>
> ==== No reporter.log
>
> ==== stderr.log
> error in /usr/local/share/bro/base/frameworks/cluster/__load__.bro,
> line 16: can't open cluster-layout
>
> ==== stdout.log
> unlimited
> 536870912
> unlimited
>
> ==== .cmdline
> -i bce1 -U .status -p broctl -p broctl-live -p local -p worker-1
> local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
>
> ==== .env_vars
> PATH=/usr/local/bin:/usr/local/share/broctl/scripts:/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/y/bin:/root/bin
> BROPATH=/usr/local/spool/installed-scripts-do-not-touch/site::/usr/local/spool/installed-scripts-do-not-touch/auto:/usr/local/share/bro:/usr/local/share/bro/policy:/usr/local/share/bro/site
> CLUSTER_NODE=worker-1
>
> ==== .status
> TERMINATED [atexit]
>
> ==== No prof.log
>
> ==== No packet_filter.log
>
> ==== No loaded_scripts.log
> [BroControl] >
>
>
> Trying to determine what is causing this.
>
> cheers,
> Hiren
>



From hiren.panchasara at gmail.com  Tue Oct 29 11:16:42 2013
From: hiren.panchasara at gmail.com (hiren panchasara)
Date: Tue, 29 Oct 2013 11:16:42 -0700
Subject: [Bro] Cluster setup
In-Reply-To: <CALCpEUGKzcx6vD7fMsYmpZe1AR8ej+zz-cgUJJ0yjMWO4-gEXg@mail.gmail.com>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
	<526F5ACB.4080404@illinois.edu>
	<CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>
	<CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>
	<CALCpEUGKzcx6vD7fMsYmpZe1AR8ej+zz-cgUJJ0yjMWO4-gEXg@mail.gmail.com>
Message-ID: <CALCpEUFA80g87+ThEDKTTS2=md7L0b2Z3wGY3ZYa1qP_41S3rg@mail.gmail.com>

On Tue, Oct 29, 2013 at 11:04 AM, hiren panchasara
<hiren.panchasara at gmail.com> wrote:
> On Tue, Oct 29, 2013 at 10:50 AM, hiren panchasara
> <hiren.panchasara at gmail.com> wrote:
>> On Tue, Oct 29, 2013 at 10:09 AM, hiren panchasara
>> <hiren.panchasara at gmail.com> wrote:
>>> On Mon, Oct 28, 2013 at 11:50 PM, Daniel Thayer <dnthayer at illinois.edu> wrote:
>>>> Did you verify that you can ssh from host-A to host-B without
>>>> having to type a password?
>>> Just set that up.
>>
>> Ah, I realized that I had to do this as "root" because broctl is run as root :-)
>>
>> Set that up and now:
>>
>> [BroControl] > install
>> removing old policies in
>> /usr/local/spool/installed-scripts-do-not-touch/site ... done.
>> removing old policies in
>> /usr/local/spool/installed-scripts-do-not-touch/auto ... done.
>> creating policy directories ... done.
>> installing site policies ... done.
>> generating cluster-layout.bro ... done.
>> generating local-networks.bro ... done.
>> generating broctl-config.bro ... done.
>> updating nodes ... warning: host 10.73.149.31 is not alive
>> done.
>> [BroControl] > check
>> manager is ok.
>> proxy-1 is ok.
>> worker-1 is ok.
>> [BroControl] > start
>> starting manager ...
>> starting proxy-1 ...
>> starting worker-1 ...
>> worker-1 terminated immediately after starting; check output with "diag"
>>
>> [BroControl] > diag worker-1
>> [worker-1]
>>
>> ==== No reporter.log
>>
>> ==== stderr.log
>> error in /usr/local/share/bro/base/frameworks/cluster/__load__.bro,
>> line 16: can't open cluster-layout
>>
>> ==== stdout.log
>> unlimited
>> 536870912
>> unlimited
>>
>> ==== .cmdline
>> -i bce1 -U .status -p broctl -p broctl-live -p local -p worker-1
>> local.bro broctl base/frameworks/cluster local-worker.bro broctl/auto
>>
>> ==== .env_vars
>> PATH=/usr/local/bin:/usr/local/share/broctl/scripts:/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/y/bin:/root/bin
>> BROPATH=/usr/local/spool/installed-scripts-do-not-touch/site::/usr/local/spool/installed-scripts-do-not-touch/auto:/usr/local/share/bro:/usr/local/share/bro/policy:/usr/local/share/bro/site
>> CLUSTER_NODE=worker-1
>>
>> ==== .status
>> TERMINATED [atexit]
>>
>> ==== No prof.log
>>
>> ==== No packet_filter.log
>>
>> ==== No loaded_scripts.log
>> [BroControl] >
>>
>>
>> Trying to determine what is causing this.
>
> I am not able to find/understand what is causing this problem.

Alright, So its looking for a file: cluster-layout.bro

I could see that on manager node at:
/usr/local/spool/installed-scripts-do-not-touch/auto/
But it was not available on same location in worker-1 node. (please
let me know if there is a better way to this)

I scp'ed that file there and then
[BroControl] > install
[BroControl] > start

worked.

top also shows all the nodes: manager, proxy-1 and worker-1 active.

Now is time for actual traffic.

Cheers,
Hiren


From JAzoff at albany.edu  Tue Oct 29 11:21:42 2013
From: JAzoff at albany.edu (Justin Azoff)
Date: Tue, 29 Oct 2013 14:21:42 -0400
Subject: [Bro] Cluster setup
In-Reply-To: <CALCpEUGKzcx6vD7fMsYmpZe1AR8ej+zz-cgUJJ0yjMWO4-gEXg@mail.gmail.com>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
	<526F5ACB.4080404@illinois.edu>
	<CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>
	<CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>
	<CALCpEUGKzcx6vD7fMsYmpZe1AR8ej+zz-cgUJJ0yjMWO4-gEXg@mail.gmail.com>
Message-ID: <20131029182142.GC4436@datacomm.albany.edu>

On Tue, Oct 29, 2013 at 11:04:31AM -0700, hiren panchasara wrote:
> > updating nodes ... warning: host 10.73.149.31 is not alive

This is from:

    def isAlive(host):
        ...
        (success, output) = runLocalCmd(os.path.join(config.Config.scriptsdir, "is-alive") + " " + util.scopeAddr(host))
        ...
        if not success and not config.Config.cron == "1":
            util.warn("host %s is not alive" % host)

which just runs the is-alive script which runs

    ping -c 1 -W 1 host

so, if that is failing you're running a restrictive iptables policy or have disabled icmp?

-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From ben.bt.wood at gmail.com  Tue Oct 29 14:51:59 2013
From: ben.bt.wood at gmail.com (Benjamin Wood)
Date: Tue, 29 Oct 2013 17:51:59 -0400
Subject: [Bro] Compiling Bro with Endace Dag Support
In-Reply-To: <20131029133330.GF72433@icir.org>
References: <CA+i_NU_oGFDn-eWb1n4XCvMU=GVFf--0SLQqjhhWwfNfZtOGhQ@mail.gmail.com>
	<20131029133330.GF72433@icir.org>
Message-ID: <CA+i_NU_GD8=HXkwj7zcuGPvjp785P86s3vCdxDb57Xt+-Qv0bA@mail.gmail.com>

Thanks, I eventually got it to compile and run correctly.

Turns out ./configure was targeting the system libpcap that was installed
before the dag drivers.
-- Found PCAP: /usr/lib64/libpcap.so
instead of the "post-dag" compiled one in /usr/local/lib

Even when I tried "./configure --with-pcap=/usr/local/lib" it still favored
the system libpcap. I don't know if this is by design.

My eventual solution was to remove the system libpcap. Once it was gone,
./configure for bro found the one in /usr/local/lib.
-- Found PCAP: /usr/local/lib/libpcap.so

Cheers,

Benjamin Wood


On Tue, Oct 29, 2013 at 9:33 AM, Robin Sommer <robin at icir.org> wrote:

>
>
> On Mon, Oct 28, 2013 at 18:02 -0400, you wrote:
>
> > Is this functionality still supported?
>
> The native DAG support was removed a while ago but I believe the pcap
> wrapper should work fine.
>
> Robin
>
> --
> Robin Sommer * Phone +1 (510) 722-6541 *     robin at icir.org
> ICSI/LBNL    * Fax   +1 (510) 666-2956 * www.icir.org/robin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131029/c5602799/attachment.html 

From knrd at rogers.com  Tue Oct 29 15:36:17 2013
From: knrd at rogers.com (Konrad Weglowski)
Date: Tue, 29 Oct 2013 18:36:17 -0400
Subject: [Bro] BRO conn.log - connection flow direction wrong - non standard
	telnet port connection
Message-ID: <003301ced4f7$48c406a0$da4c13e0$@com>

Hello,

 

I am pretty new user of BRO and use it as a part of the Security Onion
distributions. I currently came across a problem which I was hoping one of
you might be able to help with.

 

When looking at some telnet connections on a non-standard TCP port I noticed
that some data flows are reported in the wrong direction. When checked the
conn.log files, all the entries in question had the same characteristics
below:

 

1. They would only appear in the archive (gzip) conn.*.log.zip files - not
the current conn.log file.

2. Entries would always be at the beginning of the zipped conn.*.log.zip
file 

3. Conn_State field would say RSTR 4. History field would be DaFr (on most
of them)

 

Below are some examples, as you can see the file name reflects from/to
date/time, and the characteristics of the entries in question where flow
direction is reversed are below:

 

zcat conn.16:27:17-17:00:00.log.gz | bro-cut -d ts proto conn_state history
| grep RSTR

2013-10-25T16:27:12+0000        tcp     RSTR    DaFr

2013-10-25T16:27:12+0000        tcp     RSTR    DaFr

2013-10-25T16:27:12+0000        tcp     RSTR    DaFr

<snip>

 

zcat conn.18:36:28-19:00:00.log.gz | bro-cut -d ts proto conn_state history
| grep RSTR

2013-10-25T18:36:23+0000        tcp     RSTR    DaFr

2013-10-25T18:36:23+0000        tcp     RSTR    DaFr

2013-10-25T18:36:23+0000        tcp     RSTR    DaFr

<snip>

 

zcat conn.18:36:28-19:00:00.log.gz | bro-cut -d ts proto conn_state history
| grep RSTR

2013-10-25T18:36:23+0000        tcp     RSTR    DaFr

2013-10-25T18:36:23+0000        tcp     RSTR    DaFr

2013-10-25T18:36:23+0000        tcp     RSTR    DaFr

<snip>

 

It almost seems that when conn.log file is being divided up and zipped this
happens.

 

Just to give some context, we have a script running which telnets to
multiple devices and polls certain variables and exits on a non-standard
telnet ports.

 

Thanks,

 

Konrad

 

  _____  

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2014.0.4158 / Virus Database: 3615/6790 - Release Date: 10/29/13

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131029/f3d56d86/attachment.html 

From seth at icir.org  Wed Oct 30 07:10:10 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 30 Oct 2013 10:10:10 -0400
Subject: [Bro] Compiling Bro with Endace Dag Support
In-Reply-To: <CA+i_NU_GD8=HXkwj7zcuGPvjp785P86s3vCdxDb57Xt+-Qv0bA@mail.gmail.com>
References: <CA+i_NU_oGFDn-eWb1n4XCvMU=GVFf--0SLQqjhhWwfNfZtOGhQ@mail.gmail.com>
	<20131029133330.GF72433@icir.org>
	<CA+i_NU_GD8=HXkwj7zcuGPvjp785P86s3vCdxDb57Xt+-Qv0bA@mail.gmail.com>
Message-ID: <CB91812C-80CA-4206-A870-63F1D5F5CB33@icir.org>


On Oct 29, 2013, at 5:51 PM, Benjamin Wood <ben.bt.wood at gmail.com> wrote:

> Even when I tried "./configure --with-pcap=/usr/local/lib" it still favored the system libpcap. I don't know if this is by design.

You should use --with-pcap=/usr/local

The configure script needs to find the header and the library so you give the installation prefix.

> My eventual solution was to remove the system libpcap. Once it was gone, ./configure for bro found the one in /usr/local/lib.
> -- Found PCAP: /usr/local/lib/libpcap.so

This worked because /usr/local/ is a secondary search path.

  .Setb

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131030/3ac37209/attachment.bin 

From seth at icir.org  Wed Oct 30 07:20:49 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 30 Oct 2013 10:20:49 -0400
Subject: [Bro] BRO conn.log - connection flow direction wrong - non
	standard telnet port connection
In-Reply-To: <003301ced4f7$48c406a0$da4c13e0$@com>
References: <003301ced4f7$48c406a0$da4c13e0$@com>
Message-ID: <4F534409-ECE8-4B1E-B87E-1C8A326A632E@icir.org>


On Oct 29, 2013, at 6:36 PM, Konrad Weglowski <knrd at rogers.com> wrote:

> Just to give some context, we have a script running which telnets to multiple devices and polls certain variables and exits on a non-standard telnet ports.

Are you dropping a lot of packets?  It looks like Bro isn't seeing the beginning of these connections (syn packets) which makes it nearly impossible to determine the direction without guessing.  Bro's current strategy for "fixing" reversed connections like this is by consulting the likely_server_ports variable but since it sounds like you are using non-standard ports it's unlikely that this would work.

I think the big question we need to answer is why you aren't seeing the SYN packets.  Check for PacketFilter::Dropped_Packets notices in your notice.log and add "@load misc/capture-loss" to your local.bro script so that you will have a capture_loss.log which will give a holistic measurement of packet loss.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131030/715811da/attachment.bin 

From seth at icir.org  Wed Oct 30 07:24:10 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 30 Oct 2013 10:24:10 -0400
Subject: [Bro] Cluster setup
In-Reply-To: <20131029182142.GC4436@datacomm.albany.edu>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
	<526F5ACB.4080404@illinois.edu>
	<CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>
	<CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>
	<CALCpEUGKzcx6vD7fMsYmpZe1AR8ej+zz-cgUJJ0yjMWO4-gEXg@mail.gmail.com>
	<20131029182142.GC4436@datacomm.albany.edu>
Message-ID: <576FD060-B258-4877-B5F0-91B4FDE39602@icir.org>


On Oct 29, 2013, at 2:21 PM, Justin Azoff <JAzoff at albany.edu> wrote:

> so, if that is failing you're running a restrictive iptables policy or have disabled icmp?


Thanks for pointing that out, I've seen that on quite a few hosts (CentOS) where the default iptables setup interferes with BroControl.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131030/cdfe269b/attachment.bin 

From mattchess50 at gmail.com  Wed Oct 30 08:33:31 2013
From: mattchess50 at gmail.com (Matt Stucky)
Date: Wed, 30 Oct 2013 10:33:31 -0500
Subject: [Bro] pf_ring on RHEL/CENTOS 6?
Message-ID: <CAMM5JYoALoCUhXUP8UiwYHZzp45FU7GcHEtHTpycpU44-P_ntA@mail.gmail.com>

I've set up a Bro 2.1 instance with a network tap, but keep getting notice
log entries of "PacketFilter::Dropped_Packets".  I'm assuming this is
because Bro is single threaded and it needs more workers to keep up with
the traffic, so I'm trying to implement pf_ring to distribute the traffic
across multiple workers.  I've installed the pf_ring RPM package from ntop (
http://www.nmon.net/packages/rpm/x86_64/PF_RING/) and that gets the kernel
module loaded but seems to be lacking something still - probably linking
libpcap to pf_ring?  That's what I'm not sure about.  After installing
pf_ring from the RPM package and configuring Bro for multiple workers it
starts up ok but is still dropping packets (all of the workers, per the
notice log) and pf_ring doesn't appear to be used:

# cat /proc/net/pf_ring/info
PF_RING Version          : 5.6.2 ($Revision: 6910$)
Total rings              : 0

Standard (non DNA) Options
Ring slots               : 4096
Slot version             : 15
Capture TX               : No [RX only]
IP Defragment            : No
Socket Mode              : Standard
Transparent mode         : Yes [mode 0]
Total plugins            : 0
Cluster Fragment Queue   : 0
Cluster Fragment Discard : 0

Has anyone had any success with clustered Bro with pf_ring on RHEL/CENTOS,
and did you have to compile it from source and re-compile libpcap?  I'd
prefer to stick with the RPM packages since it tends to make updating less
problematic.  I installed Bro 2.1 as an RPM package as well.

Thanks,
Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131030/98cb595f/attachment.html 

From gary at doit.wisc.edu  Wed Oct 30 09:43:52 2013
From: gary at doit.wisc.edu (Gary Faulkner)
Date: Wed, 30 Oct 2013 11:43:52 -0500
Subject: [Bro] Broctl pf_ring_DNA support / Bro at 100G
Message-ID: <52713748.1020908@doit.wisc.edu>

Hello,

We recently lit up a 100G link and are attempting to tackle migrating 
our IDS and monitoring infrastructure from 10G to 100G capabilities. We 
have an existing set of servers that we are are using to evaluate SNORT, 
Suricata and Bro on with a 100G Gigamon upstream. For purposes of a Bro 
proof of concept I have two of the following Dell 720s to start from:

Dell 720XD
64 G RAM (1600 MHz RDIMMS)
30TB (usable) RAID 6 7.2K RPM SAS 6Gbps
2 146GB 15K RPM SAS 6Gbps
2 Intel Xeon E5-2670 2.60GHz, 20M Cache, 8.0GT/s QPI, Turbo, 8C
3 Intel X520 DP 10Gb DA/SFP+

I'm starting from build 2.2-beta-114 and looking at using it and PF_RING 
with the DNA drivers for the Intel cards for now as some of the other 
popular cards are "complicated" for us to get approval to purchase. I 
haven't found much info on running Bro this way other than issue ID 845 
<https://bro-tracker.atlassian.net/browse/BIT-845> and even that only 
suggests that there is a Bro Control plugin in the works for this, but 
that it may not be fully tested yet. Has anyone tried the plugin yet or 
have any experience configuring Bro and PF_RING/DNA to work together?

Regards,

-- 
Gary Faulkner
UW Madison
Office of Campus Information Security
608-262-8591

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131030/65337af4/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6257 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131030/65337af4/attachment.bin 

From dnthayer at illinois.edu  Wed Oct 30 10:40:13 2013
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Wed, 30 Oct 2013 12:40:13 -0500
Subject: [Bro] Broctl pf_ring_DNA support / Bro at 100G
In-Reply-To: <52713748.1020908@doit.wisc.edu>
References: <52713748.1020908@doit.wisc.edu>
Message-ID: <5271447D.3060909@illinois.edu>

On 10/30/2013 11:43 AM, Gary Faulkner wrote:
> Hello,
>
> We recently lit up a 100G link and are attempting to tackle migrating
> our IDS and monitoring infrastructure from 10G to 100G capabilities. We
> have an existing set of servers that we are are using to evaluate SNORT,
> Suricata and Bro on with a 100G Gigamon upstream. For purposes of a Bro
> proof of concept I have two of the following Dell 720s to start from:
>
> Dell 720XD
> 64 G RAM (1600 MHz RDIMMS)
> 30TB (usable) RAID 6 7.2K RPM SAS 6Gbps
> 2 146GB 15K RPM SAS 6Gbps
> 2 Intel Xeon E5-2670 2.60GHz, 20M Cache, 8.0GT/s QPI, Turbo, 8C
> 3 Intel X520 DP 10Gb DA/SFP+
>
> I'm starting from build 2.2-beta-114 and looking at using it and PF_RING
> with the DNA drivers for the Intel cards for now as some of the other
> popular cards are "complicated" for us to get approval to purchase. I
> haven't found much info on running Bro this way other than issue ID 845
> <https://bro-tracker.atlassian.net/browse/BIT-845> and even that only
> suggests that there is a Bro Control plugin in the works for this, but
> that it may not be fully tested yet. Has anyone tried the plugin yet or
> have any experience configuring Bro and PF_RING/DNA to work together?
>
> Regards,
>
> --
> Gary Faulkner
> UW Madison
> Office of Campus Information Security
> 608-262-8591

If you want to test the PF_RING/DNA plugin, then you'll need to use
the BroControl in the branch "topic/dnthayer/ticket845" (in the broctl
git repo), but I'm not sure if anyone has successfully used it yet.



From connar.rosebraugh at egov.com  Wed Oct 30 12:49:54 2013
From: connar.rosebraugh at egov.com (Rosebraugh, Connar)
Date: Wed, 30 Oct 2013 19:49:54 +0000
Subject: [Bro] Question regarding an error
Message-ID: <088ED11BA811374BACE1259396F485E3DA4608@VADC-MBX02.ad.cdc.nicusa.com>

Hello all,

I am new to Bro and to the concept of a mailing-list-type forum. If this is not the best place for me to post my questions, please let me know what is so that I don't waste anyone's time.

I am trying to write a script that utilizes the new_connection event. The code is as follows:

@load base/protocols/conn

event new_connection(c: connection)
{
               print "hello"; #simplified definition for the purposes of email
}

When I run this with "bro test.bro", I get the following error: error in ./test.bro, line 30: unknown identifier c, at or near "c"

I pulled this syntax straight from the "Scripting in Bro" examples page. Is there something else that I need to load? I am running the release version of Bro (2.1)

Thanks,
Connar Rosebraugh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131030/7a7f21be/attachment.html 

From JAzoff at albany.edu  Wed Oct 30 13:07:15 2013
From: JAzoff at albany.edu (Justin Azoff)
Date: Wed, 30 Oct 2013 16:07:15 -0400
Subject: [Bro] Question regarding an error
In-Reply-To: <088ED11BA811374BACE1259396F485E3DA4608@VADC-MBX02.ad.cdc.nicusa.com>
References: <088ED11BA811374BACE1259396F485E3DA4608@VADC-MBX02.ad.cdc.nicusa.com>
Message-ID: <20131030200715.GD4436@datacomm.albany.edu>

On Wed, Oct 30, 2013 at 07:49:54PM +0000, Rosebraugh, Connar wrote:
> When I run this with ?bro test.bro?, I get the following error: error in ./
> test.bro, line 30: unknown identifier c, at or near "c"
> 

line 30?  What exactly is in test.bro? What is line 30? The 3 lines you
posted work fine by themselves.


-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From JAzoff at albany.edu  Wed Oct 30 13:31:37 2013
From: JAzoff at albany.edu (Justin Azoff)
Date: Wed, 30 Oct 2013 16:31:37 -0400
Subject: [Bro] Question regarding an error
In-Reply-To: <088ED11BA811374BACE1259396F485E3DA484D@VADC-MBX02.ad.cdc.nicusa.com>
References: <088ED11BA811374BACE1259396F485E3DA4608@VADC-MBX02.ad.cdc.nicusa.com>
	<20131030200715.GD4436@datacomm.albany.edu>
	<088ED11BA811374BACE1259396F485E3DA484D@VADC-MBX02.ad.cdc.nicusa.com>
Message-ID: <20131030203137.GE4436@datacomm.albany.edu>

On Wed, Oct 30, 2013 at 08:08:58PM +0000, Rosebraugh, Connar wrote:
> type ConnDelta: record{
>     #time connection was last seen.
>     #consider updating to a set, and taking the stdev of the times
>     lastSeen: time;
>     delta: interval;
> };
> 
> function update_time(t: time, c: ConnDelta): interval
> {
>     c$delta = c$lastSeen - t;
>     c$lastSeen = t;
> }
> 
> #event bro_init()
> #{
> #    local t1 = current_time();
> #    local t2 = network_time();
> #    local t3: interval = t1 - t2;
> #
> #    print t1;
> #    print t2;
> #    print t3;
> #}
> 
> local connection_deltas: table[addr, addr, port] of ConnDelta;

This needs to be 'global' not 'local'.  Possibly a parser bug that it
doesn't return a more helpful error message since it clearly gets
confused.

> @load base/protocols/conn
> 
> #event new_connection(c: connection)
> event new_connection(c: connection)
> {
>     update_time(c$start_time, connection_deltas[c$id$orig_h, c$id$resp_h, c$id$resp_p]);
> }

The next problem you will run into is that you need to set &default for
connection_deltas that returns a default ConnDelta object for the first
time that this tuple is seen.

> event bro_done()
> {
>     #print connection_deltas;
> }
> 



-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From hiren.panchasara at gmail.com  Wed Oct 30 16:43:24 2013
From: hiren.panchasara at gmail.com (hiren panchasara)
Date: Wed, 30 Oct 2013 16:43:24 -0700
Subject: [Bro] Cluster setup
In-Reply-To: <576FD060-B258-4877-B5F0-91B4FDE39602@icir.org>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
	<526F5ACB.4080404@illinois.edu>
	<CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>
	<CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>
	<CALCpEUGKzcx6vD7fMsYmpZe1AR8ej+zz-cgUJJ0yjMWO4-gEXg@mail.gmail.com>
	<20131029182142.GC4436@datacomm.albany.edu>
	<576FD060-B258-4877-B5F0-91B4FDE39602@icir.org>
Message-ID: <CALCpEUHEquaQVjr1iG4JBG71phhJBR75=HdOu0Mawr9AQkqtsQ@mail.gmail.com>

Thanks for all the hints and help guys.

I have a question about parallelism. If I am consuming live traffic on
a 10G intel nic with 8 queues in it on my manager node, can I setup
bro to send data to 8 separate workers such that, queue:1 traffic goes
to a worker:1 and so on...?

Is that possible? Or I am thinking it wrong?

Cheers,
Hiren


From anthony.kasza at gmail.com  Wed Oct 30 21:10:11 2013
From: anthony.kasza at gmail.com (anthony kasza)
Date: Wed, 30 Oct 2013 21:10:11 -0700
Subject: [Bro] Dump Connections to File
Message-ID: <CAEZw2bxF8dcfefnf1DgkNvkE1N+QBbtzij4PguQmem-Uoq=Cyw@mail.gmail.com>

Does anyone know if it is possible to dump all the packets associated
with a specific connection to a file? It would be nice to be able to
check during connection_state_remove if a connection was interesting
or not and write it to disk.

-AK


From seth at icir.org  Thu Oct 31 06:03:37 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 31 Oct 2013 09:03:37 -0400
Subject: [Bro] Cluster setup
In-Reply-To: <CALCpEUHEquaQVjr1iG4JBG71phhJBR75=HdOu0Mawr9AQkqtsQ@mail.gmail.com>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
	<526F5ACB.4080404@illinois.edu>
	<CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>
	<CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>
	<CALCpEUGKzcx6vD7fMsYmpZe1AR8ej+zz-cgUJJ0yjMWO4-gEXg@mail.gmail.com>
	<20131029182142.GC4436@datacomm.albany.edu>
	<576FD060-B258-4877-B5F0-91B4FDE39602@icir.org>
	<CALCpEUHEquaQVjr1iG4JBG71phhJBR75=HdOu0Mawr9AQkqtsQ@mail.gmail.com>
Message-ID: <FFAF558E-2D73-4B37-8F53-5D3C0DC7079D@icir.org>


On Oct 30, 2013, at 7:43 PM, hiren panchasara <hiren.panchasara at gmail.com> wrote:

> I have a question about parallelism. If I am consuming live traffic on
> a 10G intel nic with 8 queues in it on my manager node

Yes, in that case you would just put your workers and your manager on the same physical host.

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131031/cd10cf52/attachment.bin 

From hiren.panchasara at gmail.com  Thu Oct 31 09:33:50 2013
From: hiren.panchasara at gmail.com (hiren panchasara)
Date: Thu, 31 Oct 2013 09:33:50 -0700
Subject: [Bro] Cluster setup
In-Reply-To: <FFAF558E-2D73-4B37-8F53-5D3C0DC7079D@icir.org>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
	<526F5ACB.4080404@illinois.edu>
	<CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>
	<CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>
	<CALCpEUGKzcx6vD7fMsYmpZe1AR8ej+zz-cgUJJ0yjMWO4-gEXg@mail.gmail.com>
	<20131029182142.GC4436@datacomm.albany.edu>
	<576FD060-B258-4877-B5F0-91B4FDE39602@icir.org>
	<CALCpEUHEquaQVjr1iG4JBG71phhJBR75=HdOu0Mawr9AQkqtsQ@mail.gmail.com>
	<FFAF558E-2D73-4B37-8F53-5D3C0DC7079D@icir.org>
Message-ID: <CALCpEUGjsSVruy-Zt8msRXyfZ0T4Pmn42NBbiGDUpK3=n9oH4w@mail.gmail.com>

On Oct 31, 2013 6:03 AM, "Seth Hall" <seth at icir.org> wrote:
>
>
> On Oct 30, 2013, at 7:43 PM, hiren panchasara <hiren.panchasara at gmail.com>
wrote:
>
> > I have a question about parallelism. If I am consuming live traffic on
> > a 10G intel nic with 8 queues in it on my manager node
>
> Yes, in that case you would just put your workers and your manager on the
same physical host.

But then would one host be able to keep up doing everything?

I somehow cannot direct a queue's traffic to a specific worker?

Thanks a lot for your help,
Hiren
>
>  .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131031/06a9b3bf/attachment.html 

From seth at icir.org  Thu Oct 31 09:51:03 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 31 Oct 2013 12:51:03 -0400
Subject: [Bro] Cluster setup
In-Reply-To: <CALCpEUGjsSVruy-Zt8msRXyfZ0T4Pmn42NBbiGDUpK3=n9oH4w@mail.gmail.com>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
	<526F5ACB.4080404@illinois.edu>
	<CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>
	<CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>
	<CALCpEUGKzcx6vD7fMsYmpZe1AR8ej+zz-cgUJJ0yjMWO4-gEXg@mail.gmail.com>
	<20131029182142.GC4436@datacomm.albany.edu>
	<576FD060-B258-4877-B5F0-91B4FDE39602@icir.org>
	<CALCpEUHEquaQVjr1iG4JBG71phhJBR75=HdOu0Mawr9AQkqtsQ@mail.gmail.com>
	<FFAF558E-2D73-4B37-8F53-5D3C0DC7079D@icir.org>
	<CALCpEUGjsSVruy-Zt8msRXyfZ0T4Pmn42NBbiGDUpK3=n9oH4w@mail.gmail.com>
Message-ID: <AD185128-BC02-4F4F-B2E7-7581A6726EED@icir.org>


On Oct 31, 2013, at 12:33 PM, hiren panchasara <hiren.panchasara at gmail.com> wrote:

> I somehow cannot direct a queue's traffic to a specific worker?

You already said you had 8 queues on your NIC though?  I guess I assumed you had something like PF_Ring configured to split your traffic to multiple processes.

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131031/94a4ed0d/attachment.bin 

From n.siow at wustl.edu  Thu Oct 31 10:58:49 2013
From: n.siow at wustl.edu (Nicholas Siow)
Date: Thu, 31 Oct 2013 12:58:49 -0500
Subject: [Bro] Bro install on new machine - SSH logging broken
Message-ID: <CALcsdGvNat=XDzTvxpoaQR_myT5Cwyj494Yt5OQxtzwp6CbSVw@mail.gmail.com>

Hello all,

We recently did a fresh install of Bro 2.1 on a new machine as per the
quick start guide. This machine has been watching traffic for about a week
now and all of the logs seem to be fine except for the SSH logs, which have
the following problems.

1) These logs are not adding geo-location information. The MaxMind
databases were installed and put in the proper location, and a quick bro
script that called the lookup_location() function seems to be working fine
in retrieving this information. However, none of this information is
logged, even for heuristically successful connections.

2) About half of the entries in the SSH log have a status of
"undetermined". This is not something we saw before on our older machine,
where every entry was listed as either a 'success' or 'failure' in the
status column.

3) The "resp_size" field of *every* entry is 0. Once again, this is not
something that we have seen before.

I should also mention that we have an older machine watching the exactly
same network as this one (though with a smaller network card) and that one
seems to be picking up on SSH traffic fine. Any idea what's going on here?


Thank you,
N. Siow
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131031/1b2a4b97/attachment.html 

From jlay at slave-tothe-box.net  Thu Oct 31 11:25:37 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 31 Oct 2013 12:25:37 -0600
Subject: [Bro] Bro install on new machine - SSH logging broken
In-Reply-To: <CALcsdGvNat=XDzTvxpoaQR_myT5Cwyj494Yt5OQxtzwp6CbSVw@mail.gmail.com>
References: <CALcsdGvNat=XDzTvxpoaQR_myT5Cwyj494Yt5OQxtzwp6CbSVw@mail.gmail.com>
Message-ID: <d1af5ac7ed313d45e6e2c976b171ab28@localhost>

On 2013-10-31 11:58, Nicholas Siow wrote:
> Hello all,
>
> We recently did a fresh install of Bro 2.1 on a new machine as per 
> the
> quick start guide. This machine has been watching traffic for about a
> week now and all of the logs seem to be fine except for the SSH logs,
> which have the following problems.
>
> 1) These logs are not adding geo-location information. The MaxMind
> databases were installed and put in the proper location, and a quick
> bro script that called the lookup_location() function seems to be
> working fine in retrieving this information. However, none of this
> information is logged, even for heuristically successful connections.
>
> 2) About half of the entries in the SSH log have a status of
> "undetermined". This is not something we saw before on our older
> machine, where every entry was listed as either a success or failure
> in the status column.
>
> 3) The "resp_size" field of EVERY?entry is 0. Once again, this is not
> something that we have seen before.
>
> I should also mention that we have an older machine watching the
> exactly same network as this one (though with a smaller network card)
> and that one seems to be picking up on SSH traffic fine. Any idea
> whats going on here?
>
> Thank you,
> N. Siow

Run without checksums and see if you notice a difference:

broctl.cfg
<snip>
broargs = --no-checksums

James


From hiren.panchasara at gmail.com  Thu Oct 31 12:20:35 2013
From: hiren.panchasara at gmail.com (hiren panchasara)
Date: Thu, 31 Oct 2013 12:20:35 -0700
Subject: [Bro] Cluster setup
In-Reply-To: <AD185128-BC02-4F4F-B2E7-7581A6726EED@icir.org>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
	<526F5ACB.4080404@illinois.edu>
	<CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>
	<CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>
	<CALCpEUGKzcx6vD7fMsYmpZe1AR8ej+zz-cgUJJ0yjMWO4-gEXg@mail.gmail.com>
	<20131029182142.GC4436@datacomm.albany.edu>
	<576FD060-B258-4877-B5F0-91B4FDE39602@icir.org>
	<CALCpEUHEquaQVjr1iG4JBG71phhJBR75=HdOu0Mawr9AQkqtsQ@mail.gmail.com>
	<FFAF558E-2D73-4B37-8F53-5D3C0DC7079D@icir.org>
	<CALCpEUGjsSVruy-Zt8msRXyfZ0T4Pmn42NBbiGDUpK3=n9oH4w@mail.gmail.com>
	<AD185128-BC02-4F4F-B2E7-7581A6726EED@icir.org>
Message-ID: <CALCpEUGTU7yoOj4UUMOOt5yQSNeZMRpw8WLdSpnEu3o=QgVtUA@mail.gmail.com>

On Thu, Oct 31, 2013 at 9:51 AM, Seth Hall <seth at icir.org> wrote:
>
> On Oct 31, 2013, at 12:33 PM, hiren panchasara <hiren.panchasara at gmail.com> wrote:
>
>> I somehow cannot direct a queue's traffic to a specific worker?
>
> You already said you had 8 queues on your NIC though?  I guess I assumed you had something like PF_Ring configured to split your traffic to multiple processes.

Right. So (afaik) in FreeBSD we do not have PF_RING like functionality
where there is an PF_RING application sdk and applications can choose
which queue it wants to listen to. Intel NIC (that I am using)
definitely can distribute traffic in 8 queues it has but question for
me is, how do I distribute it to the application/workers.

I also have a larger question here though.

How do people usually do something that I want to do: Have a box tap
continuous traffic on a 10G card and let workers parse/interpret it
(via bro).

Do they have PF_RING setup which blindly ports queue:1 traffic to
worker:1 and bro (using PF_RING's sdk) will do the parsing?

Simple aim here is load-distribution.

Thanks a lot of tolerating my (possibly) stupid questions :-)

Cheers,
Hiren


From seth at icir.org  Thu Oct 31 17:51:50 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 31 Oct 2013 20:51:50 -0400
Subject: [Bro] Cluster setup
In-Reply-To: <CALCpEUGTU7yoOj4UUMOOt5yQSNeZMRpw8WLdSpnEu3o=QgVtUA@mail.gmail.com>
References: <CALCpEUFP0+iasa_kxjP-OsSpfiS9M7U10WmshDhYmkoxU3b2_A@mail.gmail.com>
	<526F5ACB.4080404@illinois.edu>
	<CALCpEUHJF3kDa59UH=btczYXZmeWSN8vQM3S1QA1n8_gM_vNSA@mail.gmail.com>
	<CALCpEUHWyGm18RwfQPXaDjG3mveAhdkN-2Le6GQVBFT0DMW00w@mail.gmail.com>
	<CALCpEUGKzcx6vD7fMsYmpZe1AR8ej+zz-cgUJJ0yjMWO4-gEXg@mail.gmail.com>
	<20131029182142.GC4436@datacomm.albany.edu>
	<576FD060-B258-4877-B5F0-91B4FDE39602@icir.org>
	<CALCpEUHEquaQVjr1iG4JBG71phhJBR75=HdOu0Mawr9AQkqtsQ@mail.gmail.com>
	<FFAF558E-2D73-4B37-8F53-5D3C0DC7079D@icir.org>
	<CALCpEUGjsSVruy-Zt8msRXyfZ0T4Pmn42NBbiGDUpK3=n9oH4w@mail.gmail.com>
	<AD185128-BC02-4F4F-B2E7-7581A6726EED@icir.org>
	<CALCpEUGTU7yoOj4UUMOOt5yQSNeZMRpw8WLdSpnEu3o=QgVtUA@mail.gmail.com>
Message-ID: <7E6CC23C-25D6-4DF0-BCE1-F213B51D9198@icir.org>


On Oct 31, 2013, at 3:20 PM, hiren panchasara <hiren.panchasara at gmail.com> wrote:

> Right. So (afaik) in FreeBSD we do not have PF_RING like functionality
> where there is an PF_RING application sdk and applications can choose

Ah, generally right now people are only doing load balancing on FreeBSD with Myricom NICs and the Myricom Sniffer driver.

> which queue it wants to listen to. Intel NIC (that I am using)
> definitely can distribute traffic in 8 queues it has but question for
> me is, how do I distribute it to the application/workers.

In FreeBSD at the moment you don't.  It's possible that if you have netmap enabled you might be able to use that in some fashion, but generally those FlowDirector based queues on the high end Intel NICs aren't actually exposed in userland.  If you are talking about RSS (receive side scaling), then that's insufficient unless you have RX and TX RSS (I'm a little confused about this, but I read something recently that seemed to indicate this might be a thing on some NICs) because both directions of each connection need to go to each process.

> Do they have PF_RING setup which blindly ports queue:1 traffic to
> worker:1 and bro (using PF_RING's sdk) will do the parsing?

Typically people run PF_Ring in mode 0 which is actually not exposing hardware load balanced traffic.  It's collecting all of the traffic and load balancing it in the core.

I don't have much of a suggestion right now for FreeBSD beyond Myricom though.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131031/a66f19b7/attachment.bin 

From jlay at slave-tothe-box.net  Thu Oct 31 18:15:32 2013
From: jlay at slave-tothe-box.net (James Lay)
Date: Thu, 31 Oct 2013 19:15:32 -0600
Subject: [Bro] Bro install on new machine - SSH logging broken
In-Reply-To: <d1af5ac7ed313d45e6e2c976b171ab28@localhost>
References: <CALcsdGvNat=XDzTvxpoaQR_myT5Cwyj494Yt5OQxtzwp6CbSVw@mail.gmail.com>
	<d1af5ac7ed313d45e6e2c976b171ab28@localhost>
Message-ID: <53562AE0-EE71-4E4A-A0B1-72B46D976F1E@slave-tothe-box.net>

For what it?s worth, I only see 0 response on mine?but I don?t see the other two symptoms.

James

On Oct 31, 2013, at 12:25 PM, James Lay <jlay at slave-tothe-box.net> wrote:

> On 2013-10-31 11:58, Nicholas Siow wrote:
>> Hello all,
>> 
>> We recently did a fresh install of Bro 2.1 on a new machine as per 
>> the
>> quick start guide. This machine has been watching traffic for about a
>> week now and all of the logs seem to be fine except for the SSH logs,
>> which have the following problems.
>> 
>> 1) These logs are not adding geo-location information. The MaxMind
>> databases were installed and put in the proper location, and a quick
>> bro script that called the lookup_location() function seems to be
>> working fine in retrieving this information. However, none of this
>> information is logged, even for heuristically successful connections.
>> 
>> 2) About half of the entries in the SSH log have a status of
>> "undetermined". This is not something we saw before on our older
>> machine, where every entry was listed as either a success or failure
>> in the status column.
>> 
>> 3) The "resp_size" field of EVERY entry is 0. Once again, this is not
>> something that we have seen before.
>> 
>> I should also mention that we have an older machine watching the
>> exactly same network as this one (though with a smaller network card)
>> and that one seems to be picking up on SSH traffic fine. Any idea
>> whats going on here?
>> 
>> Thank you,
>> N. Siow
> 
> Run without checksums and see if you notice a difference:
> 
> broctl.cfg
> <snip>
> broargs = --no-checksums
> 
> James
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20131031/d4577392/attachment.html