From dmandelb at bbn.com  Fri Feb  1 16:04:10 2013
From: dmandelb at bbn.com (David Mandelberg)
Date: Fri, 01 Feb 2013 19:04:10 -0500
Subject: [Bro] connection_status_update for inactive flows
Message-ID: <1359763450.18172.5.camel@titan>

Hi,

Does the event connection_status_update fire periodically for inactive
flows that haven't timed out yet, or just for flows that were active
since the last connection_status_update event?



From robin at icir.org  Sat Feb  2 12:27:42 2013
From: robin at icir.org (Robin Sommer)
Date: Sat, 2 Feb 2013 12:27:42 -0800
Subject: [Bro] connection_status_update for inactive flows
In-Reply-To: <1359763450.18172.5.camel@titan>
References: <1359763450.18172.5.camel@titan>
Message-ID: <20130202202742.GF17347@icir.org>



On Fri, Feb 01, 2013 at 19:04 -0500, you wrote:

> Does the event connection_status_update fire periodically for inactive
> flows that haven't timed out yet, or just for flows that were active
> since the last connection_status_update event?

The former, activity doesn't matter.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From dmandelb at bbn.com  Mon Feb  4 11:02:49 2013
From: dmandelb at bbn.com (David Mandelberg)
Date: Mon, 04 Feb 2013 14:02:49 -0500
Subject: [Bro] connection_status_update for inactive flows
In-Reply-To: <20130202202742.GF17347@icir.org>
References: <1359763450.18172.5.camel@titan> <20130202202742.GF17347@icir.org>
Message-ID: <1360004569.18172.12.camel@titan>

On Sat, 2013-02-02 at 12:27 -0800, Robin Sommer wrote:
> 
> On Fri, Feb 01, 2013 at 19:04 -0500, you wrote:
> 
> > Does the event connection_status_update fire periodically for inactive
> > flows that haven't timed out yet, or just for flows that were active
> > since the last connection_status_update event?
> 
> The former, activity doesn't matter.

Thanks!

One more question: What's the best way for a script to handle
connection_status_update_interval? Would it offend site administrators
or other script authors for my script to redef it to a value that works
for that script? Should I just add a comment saying something like "Site
administrators should redef connection_status_update_interval to an
appropriate value, given these considerations about what values are
appropriate?"



From seth at icir.org  Mon Feb  4 12:03:34 2013
From: seth at icir.org (Seth Hall)
Date: Mon, 4 Feb 2013 15:03:34 -0500
Subject: [Bro] connection_status_update for inactive flows
In-Reply-To: <1360004569.18172.12.camel@titan>
References: <1359763450.18172.5.camel@titan> <20130202202742.GF17347@icir.org>
	<1360004569.18172.12.camel@titan>
Message-ID: <7E18EFB0-3EBE-45C2-8000-352A396D68CA@icir.org>


On Feb 4, 2013, at 2:02 PM, David Mandelberg <dmandelb at bbn.com> wrote:

> One more question: What's the best way for a script to handle
> connection_status_update_interval? Would it offend site administrators
> or other script authors for my script to redef it to a value that works
> for that script?

I would approach the problem differently, I *really* don't like the connection_status_update event because of the global change as you've noticed.  Are you familiar with scheduled events?  

You could implement your script like this?

module MyModule;

export {
	## The period of delay for all established connections
	## before rechecking them for whatever I'm checking them for.
	const checkup_interval = 5sec;
}

event MyModule::regular_check(c: connection)
	{
	# Do your check you would have previously done in connection_status_update
	print c$uid;

	# Reschedule this event.
	schedule checkup_interval { MyModule::regular_check(c) };
	}

event connection_established(c: connection)
	{
	# Schedule the event that does the check.
	schedule checkup_interval { MyModule::regular_check(c) };
	}


Does that work for what you're trying to do?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jsiwek at illinois.edu  Mon Feb  4 12:32:16 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 4 Feb 2013 20:32:16 +0000
Subject: [Bro] connection_status_update for inactive flows
In-Reply-To: <7E18EFB0-3EBE-45C2-8000-352A396D68CA@icir.org>
References: <1359763450.18172.5.camel@titan>
	<20130202202742.GF17347@icir.org> <1360004569.18172.12.camel@titan>
	<7E18EFB0-3EBE-45C2-8000-352A396D68CA@icir.org>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D116F1D3F2@CITESMBX2.ad.uillinois.edu>


> You could implement your script like this?
> 
> module MyModule;
> 
> export {
> 	## The period of delay for all established connections
> 	## before rechecking them for whatever I'm checking them for.
> 	const checkup_interval = 5sec;
> }
> 
> event MyModule::regular_check(c: connection)
> 	{
> 	# Do your check you would have previously done in connection_status_update
> 	print c$uid;
> 
> 	# Reschedule this event.
> 	schedule checkup_interval { MyModule::regular_check(c) };
> 	}
> 
> event connection_established(c: connection)
> 	{
> 	# Schedule the event that does the check.
> 	schedule checkup_interval { MyModule::regular_check(c) };
> 	}

There's also a general form of connection polling provided by ConnPolling::watch() in scripts/base/protocols/conn/polling.bro, which allows the "checkup" interval to vary between connections and/or over time.

scripts/base/protocols/ftp/gridftp.bro has an example usage.

    Jon


From dmandelb at bbn.com  Mon Feb  4 12:46:31 2013
From: dmandelb at bbn.com (David Mandelberg)
Date: Mon, 04 Feb 2013 15:46:31 -0500
Subject: [Bro] connection_status_update for inactive flows
In-Reply-To: <7E18EFB0-3EBE-45C2-8000-352A396D68CA@icir.org>
References: <1359763450.18172.5.camel@titan>
	<20130202202742.GF17347@icir.org> <1360004569.18172.12.camel@titan>
	<7E18EFB0-3EBE-45C2-8000-352A396D68CA@icir.org>
Message-ID: <1360010791.18172.16.camel@titan>

On Mon, 2013-02-04 at 15:03 -0500, Seth Hall wrote:
> I would approach the problem differently, I *really* don't like the connection_status_update event because of the global change as you've noticed.  Are you familiar with scheduled events?  
> 
> You could implement your script like this?
[..]
> Does that work for what you're trying to do?

With a bit of modification, I think so. Would the connection object be
updated with new data (duration, sizes, etc.) each time
MyModule::regular_check is called?




From seth at icir.org  Mon Feb  4 12:56:47 2013
From: seth at icir.org (Seth Hall)
Date: Mon, 4 Feb 2013 15:56:47 -0500
Subject: [Bro] connection_status_update for inactive flows
In-Reply-To: <1360010791.18172.16.camel@titan>
References: <1359763450.18172.5.camel@titan> <20130202202742.GF17347@icir.org>
	<1360004569.18172.12.camel@titan>
	<7E18EFB0-3EBE-45C2-8000-352A396D68CA@icir.org>
	<1360010791.18172.16.camel@titan>
Message-ID: <EE4B8A34-3B26-4530-B6CE-EDB9DA8E29EF@icir.org>


On Feb 4, 2013, at 3:46 PM, David Mandelberg <dmandelb at bbn.com> wrote:

> With a bit of modification, I think so. Would the connection object be
> updated with new data (duration, sizes, etc.) each time
> MyModule::regular_check is called?


Yes.  The same goes for the approach Jon mentioned.  I forgot that he had abstracted that notion even further. :)

You're much less likely to step on someone's toes by making a script work this way at least and internally it's basically doing the same thing but you get more script-land flexibility with the approaches Jon and I mentioned.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jsiwek at illinois.edu  Mon Feb  4 13:12:46 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 4 Feb 2013 21:12:46 +0000
Subject: [Bro] connection_status_update for inactive flows
In-Reply-To: <1360010791.18172.16.camel@titan>
References: <1359763450.18172.5.camel@titan>
	<20130202202742.GF17347@icir.org> <1360004569.18172.12.camel@titan>
	<7E18EFB0-3EBE-45C2-8000-352A396D68CA@icir.org>
	<1360010791.18172.16.camel@titan>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D116F1D566@CITESMBX2.ad.uillinois.edu>


> With a bit of modification, I think so. Would the connection object be
> updated with new data (duration, sizes, etc.) each time
> MyModule::regular_check is called?


I don't think they are necessarily.  E.g. if no other events are raised internally for the connection between the time the event is scheduled and the time when the event handler body actually executes, the connection record fields may not be updated.

One way to guarantee/force updated values is to check if the connection is still around with `connection_exists(c$id)` and then call `lookup_connection(c$id)` if it is.  `ConnPolling::watch()` automates that.

    Jon


From james.swaro at gmail.com  Wed Feb  6 13:34:28 2013
From: james.swaro at gmail.com (James Swaro)
Date: Wed, 6 Feb 2013 16:34:28 -0500
Subject: [Bro] p0f v3 signature definitions
Message-ID: <CAAG4iq=97pQgiBzzhkni+b=JQTG+GvEGZ74v1r=Y6p0-39zJOw@mail.gmail.com>

Quick question about OS fingerprinting:

Will the OS fingerprinting code in bro be updated to use the new
fingerprint definitions given in the latest version of p0f(3.06b)?

Thanks,

-- 
James Swaro*
*
Internetworking Research Group
Ohio University
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130206/e1fd1a2b/attachment.html 

From seth at icir.org  Wed Feb  6 14:01:47 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 6 Feb 2013 17:01:47 -0500
Subject: [Bro] p0f v3 signature definitions
In-Reply-To: <CAAG4iq=97pQgiBzzhkni+b=JQTG+GvEGZ74v1r=Y6p0-39zJOw@mail.gmail.com>
References: <CAAG4iq=97pQgiBzzhkni+b=JQTG+GvEGZ74v1r=Y6p0-39zJOw@mail.gmail.com>
Message-ID: <7570B46C-B41D-4B86-82C3-9EAC6B914400@icir.org>


On Feb 6, 2013, at 4:34 PM, James Swaro <james.swaro at gmail.com> wrote:

> Quick question about OS fingerprinting:
> 
> Will the OS fingerprinting code in bro be updated to use the new fingerprint definitions given in the latest version of p0f(3.06b)? 

It depends on what you mean by that. :)

I tend to upgrade the signatures when there are new releases, but we only support the original SYN packet mechanism (and not the newer SYN/ACK mechanism) so not all of the signatures will do anything directly.  We do certainly accept patches if you feel up for updating the p0f code!

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From vladg at cmu.edu  Wed Feb  6 14:12:01 2013
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Wed, 6 Feb 2013 22:12:01 +0000
Subject: [Bro] p0f v3 signature definitions
In-Reply-To: <12157_1360188541_r16M8xNX028230_7570B46C-B41D-4B86-82C3-9EAC6B914400@icir.org>
References: <CAAG4iq=97pQgiBzzhkni+b=JQTG+GvEGZ74v1r=Y6p0-39zJOw@mail.gmail.com>
	<12157_1360188541_r16M8xNX028230_7570B46C-B41D-4B86-82C3-9EAC6B914400@icir.org>
Message-ID: <1202BE242E080642B0CD0AD0A03E855296F8A7@PGH-MSGMB-03.andrew.ad.cmu.edu>

I tried dropping the v3 sigs into Bro's existing p0f mechanism, and it was *really* unhappy - I believe it would just quickly segfault. I even tried only importing the SYN-only sigs. I don't think the new format is backwards compatible with the old format, and would need some work to support.

  --Vlad


On Feb 6, 2013, at 5:01 PM, Seth Hall <seth at icir.org>
 wrote:

> 
> On Feb 6, 2013, at 4:34 PM, James Swaro <james.swaro at gmail.com> wrote:
> 
>> Quick question about OS fingerprinting:
>> 
>> Will the OS fingerprinting code in bro be updated to use the new fingerprint definitions given in the latest version of p0f(3.06b)? 
> 
> It depends on what you mean by that. :)
> 
> I tend to upgrade the signatures when there are new releases, but we only support the original SYN packet mechanism (and not the newer SYN/ACK mechanism) so not all of the signatures will do anything directly.  We do certainly accept patches if you feel up for updating the p0f code!
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From dmandelb at bbn.com  Thu Feb  7 11:00:00 2013
From: dmandelb at bbn.com (David Mandelberg)
Date: Thu, 07 Feb 2013 14:00:00 -0500
Subject: [Bro] missing void type
Message-ID: <1360263600.8228.112.camel@titan>

Hi,

I wrote a script using functions that return void. The signatures are:

function check_unexpected_consume_produce(c: connection): void
function check_long_running_flow(c: connection): void


When I try to test the script, I get:

error in ./bro-scripts/bbn-flow-analyzer.bro, line 61: identifier not defined: void
error in ./bro-scripts/bbn-flow-analyzer.bro, line 84: identifier not defined: void
warning in ./bro-scripts/bbn-flow-analyzer.bro, line 116: expression value ignored (BBNFlowAnalyzer::check_unexpected_consume_produce(BBNFlowAnalyzer::c))
warning in ./bro-scripts/bbn-flow-analyzer.bro, line 117: expression value ignored (BBNFlowAnalyzer::check_long_running_flow(BBNFlowAnalyzer::c))


According to
http://www.bro-ids.org/documentation/scripts/builtins.html#type-void
void is a valid type. Am I missing something? I was originally planning
to write those functions as events instead, but if I understand
correctly, event execution can be delayed from when the event is
triggered. That would cause trouble for those functions, since they're
called from event connection_state_remove and they expect the connection
to still be valid.



From seth at icir.org  Thu Feb  7 11:26:58 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 7 Feb 2013 14:26:58 -0500
Subject: [Bro] missing void type
In-Reply-To: <1360263600.8228.112.camel@titan>
References: <1360263600.8228.112.camel@titan>
Message-ID: <42A3AC73-694C-41B7-A7C8-CBD38111E99D@icir.org>


On Feb 7, 2013, at 2:00 PM, David Mandelberg <dmandelb at bbn.com> wrote:

> According to
> http://www.bro-ids.org/documentation/scripts/builtins.html#type-void
> void is a valid type. Am I missing something?

I noticed this the other day in our documentation.  I believe that void is only an internal type and isn't available in the script land as you discovered.

Just change your functions to this:
  function check_unexpected_consume_produce(c: connection)
  function check_long_running_flow(c: connection)

Problem solved. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From dmandelb at bbn.com  Thu Feb  7 11:51:24 2013
From: dmandelb at bbn.com (David Mandelberg)
Date: Thu, 07 Feb 2013 14:51:24 -0500
Subject: [Bro] missing void type
In-Reply-To: <42A3AC73-694C-41B7-A7C8-CBD38111E99D@icir.org>
References: <1360263600.8228.112.camel@titan>
	<42A3AC73-694C-41B7-A7C8-CBD38111E99D@icir.org>
Message-ID: <1360266684.8228.113.camel@titan>

On Thu, 2013-02-07 at 14:26 -0500, Seth Hall wrote:
> On Feb 7, 2013, at 2:00 PM, David Mandelberg <dmandelb at bbn.com> wrote:
> 
> > According to
> > http://www.bro-ids.org/documentation/scripts/builtins.html#type-void
> > void is a valid type. Am I missing something?
> 
> I noticed this the other day in our documentation.  I believe that void is only an internal type and isn't available in the script land as you discovered.
> 
> Just change your functions to this:
>   function check_unexpected_consume_produce(c: connection)
>   function check_long_running_flow(c: connection)
> 
> Problem solved. :)

Thanks!



From keqhe at cs.wisc.edu  Thu Feb  7 14:11:06 2013
From: keqhe at cs.wisc.edu (keqhe at cs.wisc.edu)
Date: Thu, 7 Feb 2013 16:11:06 -0600
Subject: [Bro] TimeStamp of Bro output
Message-ID: <8e72c88f566b2fbeeeca7cce294c51a0.squirrel@webmail.cs.wisc.edu>

HI Everyone,

We observe that the flows'timestamps in Bro log file are not strcitly in
time order. Also we note that for the same flow, the timestamp in conn.log
and the timestamp in http.log are not the same. Does anyone notice the
problem before and have ideas on this?  Thanks!



From vladg at cmu.edu  Thu Feb  7 14:31:20 2013
From: vladg at cmu.edu (Vlad Grigorescu)
Date: Thu, 7 Feb 2013 22:31:20 +0000
Subject: [Bro] TimeStamp of Bro output
In-Reply-To: <8269_1360275515_r17MIYgb030335_8e72c88f566b2fbeeeca7cce294c51a0.squirrel@webmail.cs.wisc.edu>
References: <8269_1360275515_r17MIYgb030335_8e72c88f566b2fbeeeca7cce294c51a0.squirrel@webmail.cs.wisc.edu>
Message-ID: <1202BE242E080642B0CD0AD0A03E85529763EC@PGH-MSGMB-03.andrew.ad.cmu.edu>

Hi,

I believe what you're seeing is a result of how those timestamps are defined.

In conn.log[1]: "This is the time of the first packet."
In http.log[2]: "Timestamp for when the request happened."

The conn record doesn't get written until the connection closes (or times out). It happens during the connection_state_remove[3] event. By handling it at connection close, you get duration, byte/packet counts, etc.

Also, the times for when the first packet was seen, and when the actual HTTP request was seen can be slightly off.

Does this line up with what you're seeing?

  --Vlad

[1] - <http://www.bro-ids.org/documentation/scripts/base/protocols/conn/main.html#type-Conn::Info>
[2] - <http://www.bro-ids.org/documentation/scripts/base/protocols/http/main.html#type-HTTP::Info>
[3] - <http://www.bro-ids.org/documentation/scripts/base/event.bif.html#id-connection_state_remove>

On Feb 7, 2013, at 5:11 PM, <keqhe at cs.wisc.edu>
 wrote:

> HI Everyone,
> 
> We observe that the flows'timestamps in Bro log file are not strcitly in
> time order. Also we note that for the same flow, the timestamp in conn.log
> and the timestamp in http.log are not the same. Does anyone notice the
> problem before and have ideas on this?  Thanks!
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From robin at icir.org  Thu Feb  7 14:56:53 2013
From: robin at icir.org (Robin Sommer)
Date: Thu, 7 Feb 2013 14:56:53 -0800
Subject: [Bro] TimeStamp of Bro output
In-Reply-To: <1202BE242E080642B0CD0AD0A03E85529763EC@PGH-MSGMB-03.andrew.ad.cmu.edu>
References: <8269_1360275515_r17MIYgb030335_8e72c88f566b2fbeeeca7cce294c51a0.squirrel@webmail.cs.wisc.edu>
	<1202BE242E080642B0CD0AD0A03E85529763EC@PGH-MSGMB-03.andrew.ad.cmu.edu>
Message-ID: <20130207225653.GA51519@icir.org>

Correct, and in particular log lines are explicitly not sorted by
time.

Robin

On Thu, Feb 07, 2013 at 22:31 +0000, Vlad Grigorescu wrote:

> Hi,
> 
> I believe what you're seeing is a result of how those timestamps are defined.
> 
> In conn.log[1]: "This is the time of the first packet."
> In http.log[2]: "Timestamp for when the request happened."
> 
> The conn record doesn't get written until the connection closes (or times out). It happens during the connection_state_remove[3] event. By handling it at connection close, you get duration, byte/packet counts, etc.
> 
> Also, the times for when the first packet was seen, and when the actual HTTP request was seen can be slightly off.
> 
> Does this line up with what you're seeing?
> 
>   --Vlad
> 
> [1] - <http://www.bro-ids.org/documentation/scripts/base/protocols/conn/main.html#type-Conn::Info>
> [2] - <http://www.bro-ids.org/documentation/scripts/base/protocols/http/main.html#type-HTTP::Info>
> [3] - <http://www.bro-ids.org/documentation/scripts/base/event.bif.html#id-connection_state_remove>
> 
> On Feb 7, 2013, at 5:11 PM, <keqhe at cs.wisc.edu>
>  wrote:
> 
> > HI Everyone,
> > 
> > We observe that the flows'timestamps in Bro log file are not strcitly in
> > time order. Also we note that for the same flow, the timestamp in conn.log
> > and the timestamp in http.log are not the same. Does anyone notice the
> > problem before and have ideas on this?  Thanks!
> > 
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro


-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From keqhe at cs.wisc.edu  Fri Feb  8 15:00:48 2013
From: keqhe at cs.wisc.edu (keqhe at cs.wisc.edu)
Date: Fri, 8 Feb 2013 17:00:48 -0600
Subject: [Bro] TimeStamp of Bro output
In-Reply-To: <20130207225653.GA51519@icir.org>
References: <8269_1360275515_r17MIYgb030335_8e72c88f566b2fbeeeca7cce294c51a0.squirrel@webmail.cs.wisc.edu>
	<1202BE242E080642B0CD0AD0A03E85529763EC@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<20130207225653.GA51519@icir.org>
Message-ID: <1ef10b357f1cdb9bb447b26c9d82490f.squirrel@webmail.cs.wisc.edu>

Hi Robin and Vlad,

according to the bro documentation,
http://www.bro-ids.org/documentation/scripts/base/protocols/conn/main.html

there is a 'uid' field in conn.log that is a unique flow identifier. Can
we use uid to identify the same flow in conn.log and http.log/ssl.log?
Timestamp is not suitable for flow identification.

Thanks!
> Correct, and in particular log lines are explicitly not sorted by
> time.
>
> Robin
>
> On Thu, Feb 07, 2013 at 22:31 +0000, Vlad Grigorescu wrote:
>
>> Hi,
>>
>> I believe what you're seeing is a result of how those timestamps are
>> defined.
>>
>> In conn.log[1]: "This is the time of the first packet."
>> In http.log[2]: "Timestamp for when the request happened."
>>
>> The conn record doesn't get written until the connection closes (or
>> times out). It happens during the connection_state_remove[3] event. By
>> handling it at connection close, you get duration, byte/packet counts,
>> etc.
>>
>> Also, the times for when the first packet was seen, and when the actual
>> HTTP request was seen can be slightly off.
>>
>> Does this line up with what you're seeing?
>>
>>   --Vlad
>>
>> [1] -
>> <http://www.bro-ids.org/documentation/scripts/base/protocols/conn/main.html#type-Conn::Info>
>> [2] -
>> <http://www.bro-ids.org/documentation/scripts/base/protocols/http/main.html#type-HTTP::Info>
>> [3] -
>> <http://www.bro-ids.org/documentation/scripts/base/event.bif.html#id-connection_state_remove>
>>
>> On Feb 7, 2013, at 5:11 PM, <keqhe at cs.wisc.edu>
>>  wrote:
>>
>> > HI Everyone,
>> >
>> > We observe that the flows'timestamps in Bro log file are not strcitly
>> in
>> > time order. Also we note that for the same flow, the timestamp in
>> conn.log
>> > and the timestamp in http.log are not the same. Does anyone notice the
>> > problem before and have ideas on this?  Thanks!
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> --
> Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
> ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org
>




From keqhe at cs.wisc.edu  Fri Feb  8 15:04:57 2013
From: keqhe at cs.wisc.edu (keqhe at cs.wisc.edu)
Date: Fri, 8 Feb 2013 17:04:57 -0600
Subject: [Bro] TimeStamp of Bro output
In-Reply-To: <1ef10b357f1cdb9bb447b26c9d82490f.squirrel@webmail.cs.wisc.edu>
References: <8269_1360275515_r17MIYgb030335_8e72c88f566b2fbeeeca7cce294c51a0.squirrel@webmail.cs.wisc.edu>
	<1202BE242E080642B0CD0AD0A03E85529763EC@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<20130207225653.GA51519@icir.org>
	<1ef10b357f1cdb9bb447b26c9d82490f.squirrel@webmail.cs.wisc.edu>
Message-ID: <4612b538eba77de7eb4389801b5b4fe0.squirrel@webmail.cs.wisc.edu>

> Hi Robin and Vlad,
>
> according to the bro documentation,
> http://www.bro-ids.org/documentation/scripts/base/protocols/conn/main.html
>
> there is a 'uid' field in conn.log that is a unique flow identifier. Can
> we use uid to identify the same flow in conn.log and http.log/ssl.log?
> Timestamp is not suitable for flow identification.
The important info we want to know is that---there are more than
50,000,000 flows in the trace files. SO we are not sure whether uid filed
is really UNIQUE.
>
> Thanks!
>> Correct, and in particular log lines are explicitly not sorted by
>> time.
>>
>> Robin
>>
>> On Thu, Feb 07, 2013 at 22:31 +0000, Vlad Grigorescu wrote:
>>
>>> Hi,
>>>
>>> I believe what you're seeing is a result of how those timestamps are
>>> defined.
>>>
>>> In conn.log[1]: "This is the time of the first packet."
>>> In http.log[2]: "Timestamp for when the request happened."
>>>
>>> The conn record doesn't get written until the connection closes (or
>>> times out). It happens during the connection_state_remove[3] event. By
>>> handling it at connection close, you get duration, byte/packet counts,
>>> etc.
>>>
>>> Also, the times for when the first packet was seen, and when the actual
>>> HTTP request was seen can be slightly off.
>>>
>>> Does this line up with what you're seeing?
>>>
>>>   --Vlad
>>>
>>> [1] -
>>> <http://www.bro-ids.org/documentation/scripts/base/protocols/conn/main.html#type-Conn::Info>
>>> [2] -
>>> <http://www.bro-ids.org/documentation/scripts/base/protocols/http/main.html#type-HTTP::Info>
>>> [3] -
>>> <http://www.bro-ids.org/documentation/scripts/base/event.bif.html#id-connection_state_remove>
>>>
>>> On Feb 7, 2013, at 5:11 PM, <keqhe at cs.wisc.edu>
>>>  wrote:
>>>
>>> > HI Everyone,
>>> >
>>> > We observe that the flows'timestamps in Bro log file are not strcitly
>>> in
>>> > time order. Also we note that for the same flow, the timestamp in
>>> conn.log
>>> > and the timestamp in http.log are not the same. Does anyone notice
>>> the
>>> > problem before and have ideas on this?  Thanks!
>>> >
>>> > _______________________________________________
>>> > Bro mailing list
>>> > bro at bro-ids.org
>>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>
>>>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>> --
>> Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
>> ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org
>>
>
>




From robin at icir.org  Fri Feb  8 15:11:31 2013
From: robin at icir.org (Robin Sommer)
Date: Fri, 8 Feb 2013 15:11:31 -0800
Subject: [Bro] TimeStamp of Bro output
In-Reply-To: <4612b538eba77de7eb4389801b5b4fe0.squirrel@webmail.cs.wisc.edu>
References: <8269_1360275515_r17MIYgb030335_8e72c88f566b2fbeeeca7cce294c51a0.squirrel@webmail.cs.wisc.edu>
	<1202BE242E080642B0CD0AD0A03E85529763EC@PGH-MSGMB-03.andrew.ad.cmu.edu>
	<20130207225653.GA51519@icir.org>
	<1ef10b357f1cdb9bb447b26c9d82490f.squirrel@webmail.cs.wisc.edu>
	<4612b538eba77de7eb4389801b5b4fe0.squirrel@webmail.cs.wisc.edu>
Message-ID: <20130208231131.GC74757@icir.org>



On Fri, Feb 08, 2013 at 17:04 -0600, keqhe at cs.wisc.edu wrote:

> The important info we want to know is that---there are more than
> 50,000,000 flows in the trace files. SO we are not sure whether uid filed
> is really UNIQUE.

Indeed, that's the idea behind it. It's unique and identifies flows
across all logs (and even across Bro runs).

Internally it's a hash value so there's a tiny chance for a collision,
but it's a 64-bit value space so you should be fine.

Robin

-- 
Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 *   www.icir.org


From tray at 21ct.com  Mon Feb 11 12:14:45 2013
From: tray at 21ct.com (Tim Ray)
Date: Mon, 11 Feb 2013 14:14:45 -0600
Subject: [Bro] impossibly large packets
Message-ID: <CD3EAD55.495E%tray@21ct.com>

Does Bro have any way to handle corrupt packets that appear to be impossibly large? When we get those in our setup, it hangs. Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130211/092c81b6/attachment.html 

From seth at icir.org  Mon Feb 11 12:32:02 2013
From: seth at icir.org (Seth Hall)
Date: Mon, 11 Feb 2013 15:32:02 -0500
Subject: [Bro] impossibly large packets
In-Reply-To: <CD3EAD55.495E%tray@21ct.com>
References: <CD3EAD55.495E%tray@21ct.com>
Message-ID: <214C3A2C-0C91-4DFC-A7A2-29191B62E12C@icir.org>


On Feb 11, 2013, at 3:14 PM, Tim Ray <tray at 21ct.com> wrote:

> Does Bro have any way to handle corrupt packets that appear to be impossibly large? When we get those in our setup, it hangs. Thanks.

You're going to have to define "impossibly large".  Could you also describe more what you mean when you say it hangs too?

Just a pre-guess though? Do you have any NIC features enabled for extended packet handling?  
	http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From tray at 21ct.com  Mon Feb 11 12:41:09 2013
From: tray at 21ct.com (Tim Ray)
Date: Mon, 11 Feb 2013 14:41:09 -0600
Subject: [Bro] impossibly large packets
In-Reply-To: <214C3A2C-0C91-4DFC-A7A2-29191B62E12C@icir.org>
Message-ID: <CD3EB2D7.496C%tray@21ct.com>

Our current best guess is 1,766,926,155 bytes. That's clearly far above
the jumbo limit, or any other limit I can think of. When we try to open
that packet in Wireshark, it's corrupt, which I believe to be true.

How does Bro handle such a case? Does it understand that such a thing is
corrupt?

On 2/11/13 2:32 PM, "Seth Hall" <seth at icir.org> wrote:

>
>On Feb 11, 2013, at 3:14 PM, Tim Ray <tray at 21ct.com> wrote:
>
>> Does Bro have any way to handle corrupt packets that appear to be
>>impossibly large? When we get those in our setup, it hangs. Thanks.
>
>You're going to have to define "impossibly large".  Could you also
>describe more what you mean when you say it hangs too?
>?
>Just a pre-guess though? Do you have any NIC features enabled for
>extended packet handling?
>	http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not
>-full.html
>
>  .Seth
>
>
>--
>Seth Hall
>International Computer Science Institute
>(Bro) because everyone has a network
>http://www.bro-ids.org/
>




From seth at icir.org  Mon Feb 11 12:44:41 2013
From: seth at icir.org (Seth Hall)
Date: Mon, 11 Feb 2013 15:44:41 -0500
Subject: [Bro] impossibly large packets
In-Reply-To: <CD3EB2D7.496C%tray@21ct.com>
References: <CD3EB2D7.496C%tray@21ct.com>
Message-ID: <8B8B8AA6-7081-45E6-810C-008FB5BC1687@icir.org>


On Feb 11, 2013, at 3:41 PM, Tim Ray <tray at 21ct.com> wrote:

> How does Bro handle such a case? Does it understand that such a thing is
> corrupt?


How was this packet acquired?  It sounds like you have a corrupted packet capture.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jessebowling at gmail.com  Mon Feb 11 12:57:26 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Mon, 11 Feb 2013 15:57:26 -0500
Subject: [Bro] LogExpireInterval not respected?
Message-ID: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>

Hi,

In my /usr/local/bro/etc/broctl.cfg I've specified:

LogExpireInterval = 14

Additionally in /etc/cron.d/bro I've specified:

0-59/5 * * * * /usr/local/bro/bin/broctl cron

However I've found that I have more daily directories present that 14
days...What configuration options should I be checking to troubleshoot this
problem?

Thanks,

Jesse

-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130211/bb789a5a/attachment.html 

From jessebowling at gmail.com  Mon Feb 11 13:17:07 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Mon, 11 Feb 2013 16:17:07 -0500
Subject: [Bro] Feature Request, up to 50% done?
Message-ID: <CAGFhEnkhev+Zzk_9gJPytaU4EHyyZwZjru2uCyR1nNDb0cASXg@mail.gmail.com>

Hi,

When I need to search my bro log files, I find myself using 'zfgrep' quite
a bit...Out of the box this works fine, however I lose the ability to
translate epoch time to 'human readable' time. The examples I've seen
suggest "zcat $file | bro-cut -d" however this means that every line must
be run through awk, which greatly increases search times. for an example
search of a one hour http file, my 'bcut' method runs in 6.9 seconds, while
using bro-cut properly(? "zcat file | bro-cut -d | fgrep string") takes 5
minutes, 1.2 seconds. Doing my zfgrep before the bro-cut causes only blank
lines to be printed (I assume because there are no description fields to be
read).

The workaround I've been implementing is to:

# alias bcut
alias bcut='awk -f /usr/local/bin/epoch_to_human.gawk'

# cat /usr/local/bin/epoch_to_human.gawk
#!/bin/gawk
{
val=strftime("%Y-%m-%dT%H:%M:%S%z", $1, 0)
$1=val
print $0
}

# zfgrep '10.10.10.10' /usr/local/bro/logs/some_log.txt.gz | bcut

This works well enough. Tt would be nice if there were a switch to bro-cut
that would implement this functionality, however I'm unsure of how to
integrate it myself (most of my awk programs are one line throwaways). The
key would be make it clear that you cannot specify field selections with
bro-cut, that this would only attempt to translate the first field into a
"human readable" format. Some error checking is likely in order as well...

So, I suppose I'm requesting that someone with more gawk chops than myself
give a shot at integrating this into bro-cut, or give me some pointers and
I might be able to stumble through creating a patch for submission...Or,
perhaps I'm approaching this problem in the wrong way, and could use a
pointer on a better way to go about it (aside from "ship the logs elsewhere
that indexes searches"; we'll deal with that scenario later) :)

Cheers,

Jesse

-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130211/a5280d95/attachment.html 

From tyler.schoenke at colorado.edu  Mon Feb 11 13:34:38 2013
From: tyler.schoenke at colorado.edu (Tyler T. Schoenke)
Date: Mon, 11 Feb 2013 14:34:38 -0700
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
Message-ID: <511963EE.4000704@colorado.edu>

Have you run broctl install && broctl check?    I always forget to do
that after modifying LogExpireInterval.

Tyler

--
Tyler Schoenke
Network Security Program Manager
IT Security Office
University of Colorado at Boulder


On 2/11/13 1:57 PM, Jesse Bowling wrote:
> Hi,
>
> In my /usr/local/bro/etc/broctl.cfg I've specified:
>
> LogExpireInterval = 14
>
> Additionally in /etc/cron.d/bro I've specified:
>
> 0-59/5 * * * * /usr/local/bro/bin/broctl cron
>
> However I've found that I have more daily directories present that 14
> days...What configuration options should I be checking to troubleshoot
> this problem?
>
> Thanks,
>
> Jesse
>
> -- 
> Jesse Bowling
>


From jessebowling at gmail.com  Mon Feb 11 20:38:44 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Mon, 11 Feb 2013 23:38:44 -0500
Subject: [Bro] Enforce hex encoding in log output
Message-ID: <CAGFhEn=nKziaR8HT8PwodtwYnJKUDY40WbCVHjfU7kRiAvqOjg@mail.gmail.com>

So I wanted some Bro to capture the contents of HTTP POST's, and found in
the archives that Seth had already written such a thing:

module HTTP;

export {
        ## The number of bytes that will be included in the http
        ## log from the client body.
        const post_body_limit = 1024;

        redef record Info += {
                post_body: string &log &optional;
        };

}

event http_entity_data(c: connection, is_orig: bool, length: count, data:
string)
        {
        if ( is_orig )
                {
                if ( ! c$http?$post_body )
                        c$http$post_body = sub_bytes(data, 0,
post_body_limit);
                else if ( |c$http$post_body| < post_body_limit )
                        c$http$post_body = string_cat(c$http$post_body,
sub_bytes(data, 0, post_body_limit-|c$http$post_body|));
                }
        }


So now my question is: in the output of the data, can we ensure that ALL
data is hex encoded, even if it's part of the ASCII character set? I need
to put this data into a feed, and not being able to count on a delimiter is
problematic...

Thanks,

Jesse

-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130211/45b833d2/attachment.html 

From jessebowling at gmail.com  Mon Feb 11 20:40:47 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Mon, 11 Feb 2013 23:40:47 -0500
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <511963EE.4000704@colorado.edu>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
Message-ID: <CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>

Hi Tyler,

Thanks for the response.

Yes, I have ensured that these have been run...I've also tried just running
'broctl cron' manually, but I get no output and it never seems to quit (or
at least, outlasts my patience)...Any other hints?

Cheers,

Jesse

On Mon, Feb 11, 2013 at 4:34 PM, Tyler T. Schoenke <
tyler.schoenke at colorado.edu> wrote:

> Have you run broctl install && broctl check?    I always forget to do
> that after modifying LogExpireInterval.
>
> Tyler
>
> --
> Tyler Schoenke
> Network Security Program Manager
> IT Security Office
> University of Colorado at Boulder
>
>
> On 2/11/13 1:57 PM, Jesse Bowling wrote:
> > Hi,
> >
> > In my /usr/local/bro/etc/broctl.cfg I've specified:
> >
> > LogExpireInterval = 14
> >
> > Additionally in /etc/cron.d/bro I've specified:
> >
> > 0-59/5 * * * * /usr/local/bro/bin/broctl cron
> >
> > However I've found that I have more daily directories present that 14
> > days...What configuration options should I be checking to troubleshoot
> > this problem?
> >
> > Thanks,
> >
> > Jesse
> >
> > --
> > Jesse Bowling
> >
>



-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130211/2139fdef/attachment.html 

From seth at icir.org  Mon Feb 11 21:06:31 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 12 Feb 2013 00:06:31 -0500
Subject: [Bro] Enforce hex encoding in log output
In-Reply-To: <CAGFhEn=nKziaR8HT8PwodtwYnJKUDY40WbCVHjfU7kRiAvqOjg@mail.gmail.com>
References: <CAGFhEn=nKziaR8HT8PwodtwYnJKUDY40WbCVHjfU7kRiAvqOjg@mail.gmail.com>
Message-ID: <1808C227-6F67-439B-8284-28C63F0664DD@icir.org>


On Feb 11, 2013, at 11:38 PM, Jesse Bowling <jessebowling at gmail.com> wrote:

> So now my question is: in the output of the data, can we ensure that ALL data is hex encoded, even if it's part of the ASCII character set? I need to put this data into a feed, and not being able to count on a delimiter is problematic...


Just do this?


                if ( ! c$http?$post_body )
                        c$http$post_body = bytestring_to_hexstr(sub_bytes(data, 0, post_body_limit));
                else if ( |c$http$post_body| < post_body_limit )
                        c$http$post_body = string_cat(c$http$post_body, bytestring_to_hexstr(sub_bytes(data, 0, post_body_limit-|c$http$post_body|)));

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From tyler.schoenke at colorado.edu  Tue Feb 12 06:55:34 2013
From: tyler.schoenke at colorado.edu (Tyler T. Schoenke)
Date: Tue, 12 Feb 2013 07:55:34 -0700
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
Message-ID: <511A57E6.5020203@colorado.edu>

broctl cron typically doesn't give output.  If it is hanging, you should
check for other instances of broctl cron running and kill them.   They
will sometimes log jam.  I haven't figured out why that happens.  

Tyler

--
Tyler Schoenke
Network Security Program Manager
IT Security Office
University of Colorado at Boulder


On 2/11/13 9:40 PM, Jesse Bowling wrote:
> Hi Tyler,
>
> Thanks for the response.
>
> Yes, I have ensured that these have been run...I've also tried just
> running 'broctl cron' manually, but I get no output and it never seems
> to quit (or at least, outlasts my patience)...Any other hints?
>
> Cheers,
>
> Jesse
>
> On Mon, Feb 11, 2013 at 4:34 PM, Tyler T. Schoenke
> <tyler.schoenke at colorado.edu <mailto:tyler.schoenke at colorado.edu>> wrote:
>
>     Have you run broctl install && broctl check?    I always forget to do
>     that after modifying LogExpireInterval.
>
>     Tyler
>
>     --
>     Tyler Schoenke
>     Network Security Program Manager
>     IT Security Office
>     University of Colorado at Boulder
>
>
>     On 2/11/13 1:57 PM, Jesse Bowling wrote:
>     > Hi,
>     >
>     > In my /usr/local/bro/etc/broctl.cfg I've specified:
>     >
>     > LogExpireInterval = 14
>     >
>     > Additionally in /etc/cron.d/bro I've specified:
>     >
>     > 0-59/5 * * * * /usr/local/bro/bin/broctl cron
>     >
>     > However I've found that I have more daily directories present
>     that 14
>     > days...What configuration options should I be checking to
>     troubleshoot
>     > this problem?
>     >
>     > Thanks,
>     >
>     > Jesse
>     >
>     > --
>     > Jesse Bowling
>     >
>
>
>
>
> -- 
> Jesse Bowling
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130212/59a8e92c/attachment.html 

From jones at tacc.utexas.edu  Tue Feb 12 08:24:09 2013
From: jones at tacc.utexas.edu (William Jones)
Date: Tue, 12 Feb 2013 16:24:09 +0000
Subject: [Bro] impossibly large packets
In-Reply-To: <CD3EAD55.495E%tray@21ct.com>
References: <CD3EAD55.495E%tray@21ct.com>
Message-ID: <D0BD4A5C03717E40AC104854AF37B8C225CEF2D8@EXMBX01.austin.utexas.edu>

Most network adapters have LRO on by default.   This can translate to large packets on bro input.   If you running bro on linux you see this behavior.



From: bro-bounces at bro-ids.org [mailto:bro-bounces at bro-ids.org] On Behalf Of Tim Ray
Sent: Monday, February 11, 2013 2:15 PM
To: bro at bro-ids.org
Subject: [Bro] impossibly large packets

Does Bro have any way to handle corrupt packets that appear to be impossibly large? When we get those in our setup, it hangs. Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130212/2f4a1f0d/attachment.html 

From jessebowling at gmail.com  Tue Feb 12 09:18:51 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Tue, 12 Feb 2013 12:18:51 -0500
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <511A57E6.5020203@colorado.edu>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
Message-ID: <CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>

Thanks again Tyler.

Unfortunately, that was not the case here (although I have observed the
same previously). Disabling the cron job and running it manually with
strace, we end up hanging after a few minutes:

read(8, "W", 1)                         = 1
read(8, "r", 1)                         = 1
read(8, "i", 1)                         = 1
read(8, "t", 1)                         = 1
read(8, "e", 1)                         = 1
read(8, " ", 1)                         = 1
read(8, "p", 1)                         = 1
read(8, "a", 1)                         = 1
read(8, "c", 1)                         = 1
read(8, "k", 1)                         = 1
read(8, "e", 1)                         = 1
read(8, "t", 1)                         = 1
read(8, "s", 1)                         = 1
read(8, " ", 1)                         = 1
read(8, "t", 1)                         = 1
read(8, "o", 1)                         = 1
read(8, " ", 1)                         = 1
read(8, "f", 1)                         = 1
read(8, "i", 1)                         = 1
read(8, "l", 1)                         = 1
read(8, "e", 1)                         = 1
read(8, "\n", 1)                        = 1
wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
read(8, "\n", 1)                        = 1
wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
read(8, "~", 1)                         = 1
read(8, "~", 1)                         = 1
read(8, "~", 1)                         = 1
read(8, "\n", 1)                        = 1
wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
read(32, "0", 1)                        = 1
read(32, "\n", 1)                       = 1
wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
read(32,

Any other hints?

Cheers,

Jesse


On Tue, Feb 12, 2013 at 9:55 AM, Tyler T. Schoenke <
tyler.schoenke at colorado.edu> wrote:

>  broctl cron typically doesn't give output.  If it is hanging, you should
> check for other instances of broctl cron running and kill them.   They will
> sometimes log jam.  I haven't figured out why that happens.
>
> Tyler
>
> --
> Tyler Schoenke
> Network Security Program Manager
> IT Security Office
> University of Colorado at Boulder
>
>
> On 2/11/13 9:40 PM, Jesse Bowling wrote:
>
> Hi Tyler,
>
> Thanks for the response.
>
> Yes, I have ensured that these have been run...I've also tried just
> running 'broctl cron' manually, but I get no output and it never seems to
> quit (or at least, outlasts my patience)...Any other hints?
>
> Cheers,
>
> Jesse
>
> On Mon, Feb 11, 2013 at 4:34 PM, Tyler T. Schoenke <
> tyler.schoenke at colorado.edu> wrote:
>
>> Have you run broctl install && broctl check?    I always forget to do
>> that after modifying LogExpireInterval.
>>
>> Tyler
>>
>> --
>> Tyler Schoenke
>> Network Security Program Manager
>> IT Security Office
>> University of Colorado at Boulder
>>
>>
>> On 2/11/13 1:57 PM, Jesse Bowling wrote:
>> > Hi,
>> >
>> > In my /usr/local/bro/etc/broctl.cfg I've specified:
>> >
>> > LogExpireInterval = 14
>> >
>> > Additionally in /etc/cron.d/bro I've specified:
>> >
>> > 0-59/5 * * * * /usr/local/bro/bin/broctl cron
>> >
>> > However I've found that I have more daily directories present that 14
>> > days...What configuration options should I be checking to troubleshoot
>> > this problem?
>> >
>> > Thanks,
>> >
>> > Jesse
>> >
>> > --
>> > Jesse Bowling
>> >
>>
>
>
>
> --
> Jesse Bowling
>
>


-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130212/1aea0404/attachment.html 

From tray at 21ct.com  Tue Feb 12 09:27:48 2013
From: tray at 21ct.com (Tim Ray)
Date: Tue, 12 Feb 2013 11:27:48 -0600
Subject: [Bro] impossibly large packets
In-Reply-To: <D0BD4A5C03717E40AC104854AF37B8C225CEF2D8@EXMBX01.austin.utexas.edu>
Message-ID: <CD3FD784.5271%tray@21ct.com>

What about VM ware? Does that also have an LRO setting in the virtual NIC? Or are there additional issues with VM ware and BRO and packets like this?

From: William Jones <jones at tacc.utexas.edu<mailto:jones at tacc.utexas.edu>>
Date: Tuesday, February 12, 2013 10:24 AM
To: Tim Ray <tray at 21ct.com<mailto:tray at 21ct.com>>, "bro at bro-ids.org<mailto:bro at bro-ids.org>" <bro at bro-ids.org<mailto:bro at bro-ids.org>>
Subject: RE: impossibly large packets

Most network adapters have LRO on by default.   This can translate to large packets on bro input.   If you running bro on linux you see this behavior.



From: bro-bounces at bro-ids.org<mailto:bro-bounces at bro-ids.org> [mailto:bro-bounces at bro-ids.org] On Behalf Of Tim Ray
Sent: Monday, February 11, 2013 2:15 PM
To: bro at bro-ids.org<mailto:bro at bro-ids.org>
Subject: [Bro] impossibly large packets

Does Bro have any way to handle corrupt packets that appear to be impossibly large? When we get those in our setup, it hangs. Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130212/aaa018cf/attachment.html 

From seth at icir.org  Tue Feb 12 10:05:03 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 12 Feb 2013 13:05:03 -0500
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
	<CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
Message-ID: <D9E6200C-CD94-4037-AA47-446237522269@icir.org>


On Feb 12, 2013, at 12:18 PM, Jesse Bowling <jessebowling at gmail.com> wrote:

> Unfortunately, that was not the case here (although I have observed the same previously). Disabling the cron job and running it manually with strace, we end up hanging after a few minutes:

I've typically seen this when the stats.log file grows too large.  Try deleting or moving <prefix>/spool/stats.log and then do a full install/restart in broctl.  

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From paul.halliday at gmail.com  Tue Feb 12 10:15:08 2013
From: paul.halliday at gmail.com (Paul Halliday)
Date: Tue, 12 Feb 2013 14:15:08 -0400
Subject: [Bro] Problems with broctl
Message-ID: <CAJfn-RHbncgdfKSVX2g4cdqxpDsBnmLpvj_SdF=zazQQgNc4qA@mail.gmail.com>

I have enabled cron from brocontrol and have this line in roots crontab:

0-59/5 *       *       *       *       /usr/local/bro/bin/broctl cron
> /dev/null 2>&1

On a reboot, bro never starts but I do get a bunch of these (5 minutes apart):

root    79339   0.0  0.3  91252  13940 ??  Is    2:10PM   0:00.08
/usr/local/bin/python /usr/local/bro/bin/broctl

if I start broctl from here and try to do anything I get 'cannot get
lock'. A killall python and I can do a start however the
"/usr/local/bin/python /usr/local/bro/bin/broctl"
processes keep on coming.

Am I missing something?

Thanks.

-- 
Paul Halliday
http://www.pintumbler.org/


From jessebowling at gmail.com  Tue Feb 12 10:38:18 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Tue, 12 Feb 2013 13:38:18 -0500
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <D9E6200C-CD94-4037-AA47-446237522269@icir.org>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
	<CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
	<D9E6200C-CD94-4037-AA47-446237522269@icir.org>
Message-ID: <CAGFhEnmhH8TTYTtMQ2-s5QrsPwAo3DGK5vRhRBWUC05muGJ90w@mail.gmail.com>

I'll try that, but my stats files are 45 - 107 lines long....Is that too
large?

Cheers,

Jesse

On Tue, Feb 12, 2013 at 1:05 PM, Seth Hall <seth at icir.org> wrote:

>
> On Feb 12, 2013, at 12:18 PM, Jesse Bowling <jessebowling at gmail.com>
> wrote:
>
> > Unfortunately, that was not the case here (although I have observed the
> same previously). Disabling the cron job and running it manually with
> strace, we end up hanging after a few minutes:
>
> I've typically seen this when the stats.log file grows too large.  Try
> deleting or moving <prefix>/spool/stats.log and then do a full
> install/restart in broctl.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>


-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130212/f2214d44/attachment.html 

From seth at icir.org  Tue Feb 12 12:30:56 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 12 Feb 2013 15:30:56 -0500
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <CAGFhEnmhH8TTYTtMQ2-s5QrsPwAo3DGK5vRhRBWUC05muGJ90w@mail.gmail.com>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
	<CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
	<D9E6200C-CD94-4037-AA47-446237522269@icir.org>
	<CAGFhEnmhH8TTYTtMQ2-s5QrsPwAo3DGK5vRhRBWUC05muGJ90w@mail.gmail.com>
Message-ID: <A466A7D0-0BEA-4AAA-AB0A-B47BEC46D4FD@icir.org>


On Feb 12, 2013, at 1:38 PM, Jesse Bowling <jessebowling at gmail.com> wrote:

> I'll try that, but my stats files are 45 - 107 lines long....Is that too large?

Oh, probably not.  Maybe there is some other file I'm trying to think of...  It's been a while since I've seen this problem, I used to see it all the time when the boxes I ran were under high load though.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Tue Feb 12 12:43:53 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 12 Feb 2013 15:43:53 -0500
Subject: [Bro] Feature Request, up to 50% done?
In-Reply-To: <CAGFhEnkhev+Zzk_9gJPytaU4EHyyZwZjru2uCyR1nNDb0cASXg@mail.gmail.com>
References: <CAGFhEnkhev+Zzk_9gJPytaU4EHyyZwZjru2uCyR1nNDb0cASXg@mail.gmail.com>
Message-ID: <004A5B0F-8B74-4B4B-9F09-E9BE38A10531@icir.org>


On Feb 11, 2013, at 4:17 PM, Jesse Bowling <jessebowling at gmail.com> wrote:

> So, I suppose I'm requesting that someone with more gawk chops than myself give a shot at integrating this into bro-cut

I tend to use these lines in my profile...

alias bro-column="sed \"s/fields.//;s/types.//\" | column -s $'\t' -t"
alias bro-awk='awk -F"  "'
bro-grep() { grep -E "(^#)|$1" $2; }
bro-zgrep() { zgrep -E "(^#)|$1" $2; }

What you're trying to do can then be accomplished like this?

bro-zgrep '10.10.10.10' /usr/local/bro/logs/conn.*.log.gz | bro-cut id.orig_h,id.resp_h

It *would* be handy to be able to do this through bro-cut though but that would make bro-cut start to sound like an incorrectly named utility. :)

Have you tried using the ElasticSearch writer and Brownian?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From dnthayer at illinois.edu  Tue Feb 12 14:27:18 2013
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Tue, 12 Feb 2013 16:27:18 -0600
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
	<CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
Message-ID: <511AC1C6.9000501@illinois.edu>

Have you tried looking at the <install_prefix>/spool/debug.log file?
If that file doesn't exist, then uncomment the "Debug = 1" line
in your broctl.cfg file.



On 02/12/2013 11:18 AM, Jesse Bowling wrote:
> Thanks again Tyler.
>
> Unfortunately, that was not the case here (although I have observed the
> same previously). Disabling the cron job and running it manually with
> strace, we end up hanging after a few minutes:
>
> read(8, "W", 1)                         = 1
> read(8, "r", 1)                         = 1
> read(8, "i", 1)                         = 1
> read(8, "t", 1)                         = 1
> read(8, "e", 1)                         = 1
> read(8, " ", 1)                         = 1
> read(8, "p", 1)                         = 1
> read(8, "a", 1)                         = 1
> read(8, "c", 1)                         = 1
> read(8, "k", 1)                         = 1
> read(8, "e", 1)                         = 1
> read(8, "t", 1)                         = 1
> read(8, "s", 1)                         = 1
> read(8, " ", 1)                         = 1
> read(8, "t", 1)                         = 1
> read(8, "o", 1)                         = 1
> read(8, " ", 1)                         = 1
> read(8, "f", 1)                         = 1
> read(8, "i", 1)                         = 1
> read(8, "l", 1)                         = 1
> read(8, "e", 1)                         = 1
> read(8, "\n", 1)                        = 1
> wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
> read(8, "\n", 1)                        = 1
> wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
> read(8, "~", 1)                         = 1
> read(8, "~", 1)                         = 1
> read(8, "~", 1)                         = 1
> read(8, "\n", 1)                        = 1
> wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
> read(32, "0", 1)                        = 1
> read(32, "\n", 1)                       = 1
> wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
> read(32,
>
> Any other hints?
>
> Cheers,
>
> Jesse
>
>
> On Tue, Feb 12, 2013 at 9:55 AM, Tyler T. Schoenke
> <tyler.schoenke at colorado.edu <mailto:tyler.schoenke at colorado.edu>> wrote:
>
>     broctl cron typically doesn't give output.  If it is hanging, you
>     should check for other instances of broctl cron running and kill
>     them.   They will sometimes log jam.  I haven't figured out why that
>     happens.
>
>     Tyler
>
>     --
>     Tyler Schoenke
>     Network Security Program Manager
>     IT Security Office
>     University of Colorado at Boulder
>
>
>     On 2/11/13 9:40 PM, Jesse Bowling wrote:
>>     Hi Tyler,
>>
>>     Thanks for the response.
>>
>>     Yes, I have ensured that these have been run...I've also tried
>>     just running 'broctl cron' manually, but I get no output and it
>>     never seems to quit (or at least, outlasts my patience)...Any
>>     other hints?
>>
>>     Cheers,
>>
>>     Jesse
>>
>>     On Mon, Feb 11, 2013 at 4:34 PM, Tyler T. Schoenke
>>     <tyler.schoenke at colorado.edu <mailto:tyler.schoenke at colorado.edu>>
>>     wrote:
>>
>>         Have you run broctl install && broctl check?    I always
>>         forget to do
>>         that after modifying LogExpireInterval.
>>
>>         Tyler
>>
>>         --
>>         Tyler Schoenke
>>         Network Security Program Manager
>>         IT Security Office
>>         University of Colorado at Boulder
>>
>>
>>         On 2/11/13 1:57 PM, Jesse Bowling wrote:
>>         > Hi,
>>         >
>>         > In my /usr/local/bro/etc/broctl.cfg I've specified:
>>         >
>>         > LogExpireInterval = 14
>>         >
>>         > Additionally in /etc/cron.d/bro I've specified:
>>         >
>>         > 0-59/5 * * * * /usr/local/bro/bin/broctl cron
>>         >
>>         > However I've found that I have more daily directories
>>         present that 14
>>         > days...What configuration options should I be checking to
>>         troubleshoot
>>         > this problem?
>>         >
>>         > Thanks,
>>         >
>>         > Jesse
>>         >
>>         > --
>>         > Jesse Bowling
>>         >
>>
>>
>>
>>
>>     --
>>     Jesse Bowling
>>
>
>
>
> --
> Jesse Bowling
>
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



From jessebowling at gmail.com  Wed Feb 13 07:33:34 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Wed, 13 Feb 2013 10:33:34 -0500
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <511AC1C6.9000501@illinois.edu>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
	<CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
	<511AC1C6.9000501@illinois.edu>
Message-ID: <CAGFhEnkbgQ1XSZ1kFp+bZZCsVktAKusWdgciPNy5VNsi646v-A@mail.gmail.com>

Turned debug on, install && check'ed everything, ensured 'broctl cron' is
in cron, ensured two jobs not running, but still my logs don't expire...

More hints anyone?

Cheers,

Jesse

On Tue, Feb 12, 2013 at 5:27 PM, Daniel Thayer <dnthayer at illinois.edu>wrote:

> Have you tried looking at the <install_prefix>/spool/debug.**log file?
> If that file doesn't exist, then uncomment the "Debug = 1" line
> in your broctl.cfg file.
>
>
>
>
> On 02/12/2013 11:18 AM, Jesse Bowling wrote:
>
>> Thanks again Tyler.
>>
>> Unfortunately, that was not the case here (although I have observed the
>> same previously). Disabling the cron job and running it manually with
>> strace, we end up hanging after a few minutes:
>>
>> read(8, "W", 1)                         = 1
>> read(8, "r", 1)                         = 1
>> read(8, "i", 1)                         = 1
>> read(8, "t", 1)                         = 1
>> read(8, "e", 1)                         = 1
>> read(8, " ", 1)                         = 1
>> read(8, "p", 1)                         = 1
>> read(8, "a", 1)                         = 1
>> read(8, "c", 1)                         = 1
>> read(8, "k", 1)                         = 1
>> read(8, "e", 1)                         = 1
>> read(8, "t", 1)                         = 1
>> read(8, "s", 1)                         = 1
>> read(8, " ", 1)                         = 1
>> read(8, "t", 1)                         = 1
>> read(8, "o", 1)                         = 1
>> read(8, " ", 1)                         = 1
>> read(8, "f", 1)                         = 1
>> read(8, "i", 1)                         = 1
>> read(8, "l", 1)                         = 1
>> read(8, "e", 1)                         = 1
>> read(8, "\n", 1)                        = 1
>> wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>> read(8, "\n", 1)                        = 1
>> wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>> read(8, "~", 1)                         = 1
>> read(8, "~", 1)                         = 1
>> read(8, "~", 1)                         = 1
>> read(8, "\n", 1)                        = 1
>> wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>> read(32, "0", 1)                        = 1
>> read(32, "\n", 1)                       = 1
>> wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>> read(32,
>>
>> Any other hints?
>>
>> Cheers,
>>
>> Jesse
>>
>>
>> On Tue, Feb 12, 2013 at 9:55 AM, Tyler T. Schoenke
>> <tyler.schoenke at colorado.edu <mailto:tyler.schoenke@**colorado.edu<tyler.schoenke at colorado.edu>>>
>> wrote:
>>
>>     broctl cron typically doesn't give output.  If it is hanging, you
>>     should check for other instances of broctl cron running and kill
>>     them.   They will sometimes log jam.  I haven't figured out why that
>>     happens.
>>
>>     Tyler
>>
>>     --
>>     Tyler Schoenke
>>     Network Security Program Manager
>>     IT Security Office
>>     University of Colorado at Boulder
>>
>>
>>     On 2/11/13 9:40 PM, Jesse Bowling wrote:
>>
>>>     Hi Tyler,
>>>
>>>     Thanks for the response.
>>>
>>>     Yes, I have ensured that these have been run...I've also tried
>>>     just running 'broctl cron' manually, but I get no output and it
>>>     never seems to quit (or at least, outlasts my patience)...Any
>>>     other hints?
>>>
>>>     Cheers,
>>>
>>>     Jesse
>>>
>>>     On Mon, Feb 11, 2013 at 4:34 PM, Tyler T. Schoenke
>>>     <tyler.schoenke at colorado.edu <mailto:tyler.schoenke@**colorado.edu<tyler.schoenke at colorado.edu>
>>> >>
>>>
>>>     wrote:
>>>
>>>         Have you run broctl install && broctl check?    I always
>>>         forget to do
>>>         that after modifying LogExpireInterval.
>>>
>>>         Tyler
>>>
>>>         --
>>>         Tyler Schoenke
>>>         Network Security Program Manager
>>>         IT Security Office
>>>         University of Colorado at Boulder
>>>
>>>
>>>         On 2/11/13 1:57 PM, Jesse Bowling wrote:
>>>         > Hi,
>>>         >
>>>         > In my /usr/local/bro/etc/broctl.cfg I've specified:
>>>         >
>>>         > LogExpireInterval = 14
>>>         >
>>>         > Additionally in /etc/cron.d/bro I've specified:
>>>         >
>>>         > 0-59/5 * * * * /usr/local/bro/bin/broctl cron
>>>         >
>>>         > However I've found that I have more daily directories
>>>         present that 14
>>>         > days...What configuration options should I be checking to
>>>         troubleshoot
>>>         > this problem?
>>>         >
>>>         > Thanks,
>>>         >
>>>         > Jesse
>>>         >
>>>         > --
>>>         > Jesse Bowling
>>>         >
>>>
>>>
>>>
>>>
>>>     --
>>>     Jesse Bowling
>>>
>>>
>>
>>
>> --
>> Jesse Bowling
>>
>>
>>
>> ______________________________**_________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.**EDU/mailman/listinfo/bro<http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro>
>>
>>
>


-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130213/86d2b343/attachment.html 

From dnthayer at illinois.edu  Wed Feb 13 08:55:14 2013
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Wed, 13 Feb 2013 10:55:14 -0600
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <CAGFhEnkbgQ1XSZ1kFp+bZZCsVktAKusWdgciPNy5VNsi646v-A@mail.gmail.com>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
	<CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
	<511AC1C6.9000501@illinois.edu>
	<CAGFhEnkbgQ1XSZ1kFp+bZZCsVktAKusWdgciPNy5VNsi646v-A@mail.gmail.com>
Message-ID: <511BC572.2050402@illinois.edu>

When "broctl cron" runs, it will output a line (in debug.log)
that indicates it is starting, followed by a lot of output
showing its progress, and finally a line indicating it
is done. It looks something like this:

12 Feb 14:29:30 [command]  cron

<...lots of output here...>

12 Feb 14:30:37 [main]     cron done

If "broctl cron" is hanging, then it might be
useful to check the debug.log to see how far it
gets before it hangs.



On 02/13/2013 09:33 AM, Jesse Bowling wrote:
> Turned debug on, install && check'ed everything, ensured 'broctl cron'
> is in cron, ensured two jobs not running, but still my logs don't expire...
>
> More hints anyone?
>
> Cheers,
>
> Jesse
>
> On Tue, Feb 12, 2013 at 5:27 PM, Daniel Thayer <dnthayer at illinois.edu
> <mailto:dnthayer at illinois.edu>> wrote:
>
>     Have you tried looking at the <install_prefix>/spool/debug.__log file?
>     If that file doesn't exist, then uncomment the "Debug = 1" line
>     in your broctl.cfg file.
>
>
>
>
>     On 02/12/2013 11:18 AM, Jesse Bowling wrote:
>
>         Thanks again Tyler.
>
>         Unfortunately, that was not the case here (although I have
>         observed the
>         same previously). Disabling the cron job and running it manually
>         with
>         strace, we end up hanging after a few minutes:
>
>         read(8, "W", 1)                         = 1
>         read(8, "r", 1)                         = 1
>         read(8, "i", 1)                         = 1
>         read(8, "t", 1)                         = 1
>         read(8, "e", 1)                         = 1
>         read(8, " ", 1)                         = 1
>         read(8, "p", 1)                         = 1
>         read(8, "a", 1)                         = 1
>         read(8, "c", 1)                         = 1
>         read(8, "k", 1)                         = 1
>         read(8, "e", 1)                         = 1
>         read(8, "t", 1)                         = 1
>         read(8, "s", 1)                         = 1
>         read(8, " ", 1)                         = 1
>         read(8, "t", 1)                         = 1
>         read(8, "o", 1)                         = 1
>         read(8, " ", 1)                         = 1
>         read(8, "f", 1)                         = 1
>         read(8, "i", 1)                         = 1
>         read(8, "l", 1)                         = 1
>         read(8, "e", 1)                         = 1
>         read(8, "\n", 1)                        = 1
>         wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>         read(8, "\n", 1)                        = 1
>         wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>         read(8, "~", 1)                         = 1
>         read(8, "~", 1)                         = 1
>         read(8, "~", 1)                         = 1
>         read(8, "\n", 1)                        = 1
>         wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>         read(32, "0", 1)                        = 1
>         read(32, "\n", 1)                       = 1
>         wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>         read(32,
>
>         Any other hints?
>
>         Cheers,
>
>         Jesse
>
>
>         On Tue, Feb 12, 2013 at 9:55 AM, Tyler T. Schoenke
>         <tyler.schoenke at colorado.edu
>         <mailto:tyler.schoenke at colorado.edu>
>         <mailto:tyler.schoenke at __colorado.edu
>         <mailto:tyler.schoenke at colorado.edu>>> wrote:
>
>              broctl cron typically doesn't give output.  If it is
>         hanging, you
>              should check for other instances of broctl cron running and
>         kill
>              them.   They will sometimes log jam.  I haven't figured out
>         why that
>              happens.
>
>              Tyler
>
>              --
>              Tyler Schoenke
>              Network Security Program Manager
>              IT Security Office
>              University of Colorado at Boulder
>
>
>              On 2/11/13 9:40 PM, Jesse Bowling wrote:
>
>                  Hi Tyler,
>
>                  Thanks for the response.
>
>                  Yes, I have ensured that these have been run...I've
>             also tried
>                  just running 'broctl cron' manually, but I get no
>             output and it
>                  never seems to quit (or at least, outlasts my
>             patience)...Any
>                  other hints?
>
>                  Cheers,
>
>                  Jesse
>
>                  On Mon, Feb 11, 2013 at 4:34 PM, Tyler T. Schoenke
>                  <tyler.schoenke at colorado.edu
>             <mailto:tyler.schoenke at colorado.edu>
>             <mailto:tyler.schoenke at __colorado.edu
>             <mailto:tyler.schoenke at colorado.edu>>>
>
>                  wrote:
>
>                      Have you run broctl install && broctl check?    I
>             always
>                      forget to do
>                      that after modifying LogExpireInterval.
>
>                      Tyler
>
>                      --
>                      Tyler Schoenke
>                      Network Security Program Manager
>                      IT Security Office
>                      University of Colorado at Boulder
>
>
>                      On 2/11/13 1:57 PM, Jesse Bowling wrote:
>                      > Hi,
>                      >
>                      > In my /usr/local/bro/etc/broctl.cfg I've specified:
>                      >
>                      > LogExpireInterval = 14
>                      >
>                      > Additionally in /etc/cron.d/bro I've specified:
>                      >
>                      > 0-59/5 * * * * /usr/local/bro/bin/broctl cron
>                      >
>                      > However I've found that I have more daily directories
>                      present that 14
>                      > days...What configuration options should I be
>             checking to
>                      troubleshoot
>                      > this problem?
>                      >
>                      > Thanks,
>                      >
>                      > Jesse
>                      >
>                      > --
>                      > Jesse Bowling
>                      >
>
>
>
>
>                  --
>                  Jesse Bowling
>
>
>
>
>         --
>         Jesse Bowling
>
>
>
>         _________________________________________________
>         Bro mailing list
>         bro at bro-ids.org <mailto:bro at bro-ids.org>
>         http://mailman.ICSI.Berkeley.__EDU/mailman/listinfo/bro
>         <http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro>
>
>
>
>
>
> --
> Jesse Bowling
>



From jessebowling at gmail.com  Wed Feb 13 09:30:06 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Wed, 13 Feb 2013 12:30:06 -0500
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <CAGFhEnkbgQ1XSZ1kFp+bZZCsVktAKusWdgciPNy5VNsi646v-A@mail.gmail.com>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
	<CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
	<511AC1C6.9000501@illinois.edu>
	<CAGFhEnkbgQ1XSZ1kFp+bZZCsVktAKusWdgciPNy5VNsi646v-A@mail.gmail.com>
Message-ID: <CAGFhEn=9Zx=p1TVVK7On1KNGwzVPVh0oTvkZHxf7oNMt9zAntg@mail.gmail.com>

So my solution is a two-part one...The easy part first: the cron job wasn't
running.

In the document (http://www.bro-ids.org/documentation/quickstart.html) I
used for setup, it was suggested to put the following into crontab:

0-59/5 * * * * /usr/local/bro/bin/broctl cron

However since I was putting this under /etc/cron.d/bro (on a RHEL 6 box),
what I actually need is:


0-59/5 * * * * root /usr/local/bro/bin/broctl cron


If you don't specify the user, it won't run (or complain).

Derp.

So once that was fixed, I ran into a second issue: after the first instance
ran, subsequent runs would not be able to get the lock (generating a
"cannot get lock" error email). Looking at the issue further, it appears
that there are is a capstats instance running at 99%. Killing that job
allowed the initial cron to finish, and my directories were deleted as
expected.

root     21847 21661  0 11:35 ?        00:00:00 bash
/usr/local/bro/share/broctl/scripts/helpers/run-cmd
/usr/local/bro/bin/capstats -i p2p1;p2p2;p2p3;p2p4 -I 5 -n 1
root     21848 21847 99 11:35 ?        00:11:29 /usr/local/bro/bin/capstats
-i p2p1

Seeing that the bash script
/usr/local/bro/share/broctl/scripts/helpers/run-cmd is the dead simple:

echo 0     # Fix me. Should be real return code.
eval $@ 2>&1
echo ~~~

I can surmise the problem: Because my interface specification requires the
use of ';', bash is breaking the command up before it should and capstats
doesn't know it should quit...The format I'm using (p2p1;p2p2;p2p3;p2p4) is
making use of PF_RING to listen to all these interfaces simultaneously. For
snort I have to quote it to prevent it being broken up and I suspected
something similar is required here as well.

Sure enough, patching control.py to quote the interfaces allows the job to
finish and the stats to get written, with one error message in the stats
file: "1360775469.46 proxy-1 error error proxy-1: unexpected capstats
output: capstats [Options] -i interface".

So while this apparently fixes my issue:
--- control.py  2013-02-13 12:08:00.514656601 -0500
+++ control_mod.py      2013-02-13 12:09:38.382663593 -0500
@@ -808,7 +808,7 @@
     for (addr, interface) in hosts.keys():
         node = hosts[addr, interface]

-        capstats = [config.Config.capstatspath, "-i", interface, "-I",
str(interval), "-n", "1"]
+        capstats = [config.Config.capstatspath, "-i", '"' + interface +
'"', "-I", str(interval), "-n", "1"]

 # Unfinished feature: only consider a particular MAC. Works here for
capstats
 # but Bro config is not adapted currently so we disable it for now.

I cannot speak to how this might affect others, the system in general, or
where else this issue might crop up. I suspect that anywhere that involves
bash + interface names is likely to suffer unexpected results due to this
PF_RING style invocation...

I plan to let this run for now, perhaps others might comment on the
advisability of this?

Cheers,

Jesse


On Wed, Feb 13, 2013 at 10:33 AM, Jesse Bowling <jessebowling at gmail.com>wrote:

> Turned debug on, install && check'ed everything, ensured 'broctl cron' is
> in cron, ensured two jobs not running, but still my logs don't expire...
>
> More hints anyone?
>
> Cheers,
>
> Jesse
>
>
> On Tue, Feb 12, 2013 at 5:27 PM, Daniel Thayer <dnthayer at illinois.edu>wrote:
>
>> Have you tried looking at the <install_prefix>/spool/debug.**log file?
>> If that file doesn't exist, then uncomment the "Debug = 1" line
>> in your broctl.cfg file.
>>
>>
>>
>>
>> On 02/12/2013 11:18 AM, Jesse Bowling wrote:
>>
>>> Thanks again Tyler.
>>>
>>> Unfortunately, that was not the case here (although I have observed the
>>> same previously). Disabling the cron job and running it manually with
>>> strace, we end up hanging after a few minutes:
>>>
>>> read(8, "W", 1)                         = 1
>>> read(8, "r", 1)                         = 1
>>> read(8, "i", 1)                         = 1
>>> read(8, "t", 1)                         = 1
>>> read(8, "e", 1)                         = 1
>>> read(8, " ", 1)                         = 1
>>> read(8, "p", 1)                         = 1
>>> read(8, "a", 1)                         = 1
>>> read(8, "c", 1)                         = 1
>>> read(8, "k", 1)                         = 1
>>> read(8, "e", 1)                         = 1
>>> read(8, "t", 1)                         = 1
>>> read(8, "s", 1)                         = 1
>>> read(8, " ", 1)                         = 1
>>> read(8, "t", 1)                         = 1
>>> read(8, "o", 1)                         = 1
>>> read(8, " ", 1)                         = 1
>>> read(8, "f", 1)                         = 1
>>> read(8, "i", 1)                         = 1
>>> read(8, "l", 1)                         = 1
>>> read(8, "e", 1)                         = 1
>>> read(8, "\n", 1)                        = 1
>>> wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>>> read(8, "\n", 1)                        = 1
>>> wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>>> read(8, "~", 1)                         = 1
>>> read(8, "~", 1)                         = 1
>>> read(8, "~", 1)                         = 1
>>> read(8, "\n", 1)                        = 1
>>> wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>>> read(32, "0", 1)                        = 1
>>> read(32, "\n", 1)                       = 1
>>> wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>>> read(32,
>>>
>>> Any other hints?
>>>
>>> Cheers,
>>>
>>> Jesse
>>>
>>>
>>> On Tue, Feb 12, 2013 at 9:55 AM, Tyler T. Schoenke
>>> <tyler.schoenke at colorado.edu <mailto:tyler.schoenke@**colorado.edu<tyler.schoenke at colorado.edu>>>
>>> wrote:
>>>
>>>     broctl cron typically doesn't give output.  If it is hanging, you
>>>     should check for other instances of broctl cron running and kill
>>>     them.   They will sometimes log jam.  I haven't figured out why that
>>>     happens.
>>>
>>>     Tyler
>>>
>>>     --
>>>     Tyler Schoenke
>>>     Network Security Program Manager
>>>     IT Security Office
>>>     University of Colorado at Boulder
>>>
>>>
>>>     On 2/11/13 9:40 PM, Jesse Bowling wrote:
>>>
>>>>     Hi Tyler,
>>>>
>>>>     Thanks for the response.
>>>>
>>>>     Yes, I have ensured that these have been run...I've also tried
>>>>     just running 'broctl cron' manually, but I get no output and it
>>>>     never seems to quit (or at least, outlasts my patience)...Any
>>>>     other hints?
>>>>
>>>>     Cheers,
>>>>
>>>>     Jesse
>>>>
>>>>     On Mon, Feb 11, 2013 at 4:34 PM, Tyler T. Schoenke
>>>>     <tyler.schoenke at colorado.edu <mailto:tyler.schoenke@**colorado.edu<tyler.schoenke at colorado.edu>
>>>> >>
>>>>
>>>>     wrote:
>>>>
>>>>         Have you run broctl install && broctl check?    I always
>>>>         forget to do
>>>>         that after modifying LogExpireInterval.
>>>>
>>>>         Tyler
>>>>
>>>>         --
>>>>         Tyler Schoenke
>>>>         Network Security Program Manager
>>>>         IT Security Office
>>>>         University of Colorado at Boulder
>>>>
>>>>
>>>>         On 2/11/13 1:57 PM, Jesse Bowling wrote:
>>>>         > Hi,
>>>>         >
>>>>         > In my /usr/local/bro/etc/broctl.cfg I've specified:
>>>>         >
>>>>         > LogExpireInterval = 14
>>>>         >
>>>>         > Additionally in /etc/cron.d/bro I've specified:
>>>>         >
>>>>         > 0-59/5 * * * * /usr/local/bro/bin/broctl cron
>>>>         >
>>>>         > However I've found that I have more daily directories
>>>>         present that 14
>>>>         > days...What configuration options should I be checking to
>>>>         troubleshoot
>>>>         > this problem?
>>>>         >
>>>>         > Thanks,
>>>>         >
>>>>         > Jesse
>>>>         >
>>>>         > --
>>>>         > Jesse Bowling
>>>>         >
>>>>
>>>>
>>>>
>>>>
>>>>     --
>>>>     Jesse Bowling
>>>>
>>>>
>>>
>>>
>>> --
>>> Jesse Bowling
>>>
>>>
>>>
>>> ______________________________**_________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.**EDU/mailman/listinfo/bro<http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro>
>>>
>>>
>>
>
>
> --
> Jesse Bowling
>
>


-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130213/c7155e19/attachment.html 

From seth at icir.org  Wed Feb 13 09:46:05 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 13 Feb 2013 12:46:05 -0500
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <CAGFhEn=9Zx=p1TVVK7On1KNGwzVPVh0oTvkZHxf7oNMt9zAntg@mail.gmail.com>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
	<CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
	<511AC1C6.9000501@illinois.edu>
	<CAGFhEnkbgQ1XSZ1kFp+bZZCsVktAKusWdgciPNy5VNsi646v-A@mail.gmail.com>
	<CAGFhEn=9Zx=p1TVVK7On1KNGwzVPVh0oTvkZHxf7oNMt9zAntg@mail.gmail.com>
Message-ID: <8AE1AABA-7BE3-47A7-9F35-92FCAEABA09F@icir.org>


On Feb 13, 2013, at 12:30 PM, Jesse Bowling <jessebowling at gmail.com> wrote:

> I can surmise the problem: Because my interface specification requires the use of ';', bash is breaking the command up before it should and capstats doesn't know it should quit...The format I'm using (p2p1;p2p2;p2p3;p2p4) is making use of PF_RING to listen to all these interfaces simultaneously. For snort I have to quote it to prevent it being broken up and I suspected something similar is required here as well.

Woah!  PF_RING lets you sniff multiple interfaces that way?  If you give that same value to tcpdump (while using the pf_ring libpcap wrapper) does it work there too?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jessebowling at gmail.com  Wed Feb 13 09:55:40 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Wed, 13 Feb 2013 12:55:40 -0500
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <8AE1AABA-7BE3-47A7-9F35-92FCAEABA09F@icir.org>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
	<CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
	<511AC1C6.9000501@illinois.edu>
	<CAGFhEnkbgQ1XSZ1kFp+bZZCsVktAKusWdgciPNy5VNsi646v-A@mail.gmail.com>
	<CAGFhEn=9Zx=p1TVVK7On1KNGwzVPVh0oTvkZHxf7oNMt9zAntg@mail.gmail.com>
	<8AE1AABA-7BE3-47A7-9F35-92FCAEABA09F@icir.org>
Message-ID: <CAGFhEnmVxVBp4fdbXMf_TW6ov6amQ6mG+O9hOddTiOSaQe7YPw@mail.gmail.com>

On Wed, Feb 13, 2013 at 12:46 PM, Seth Hall <seth at icir.org> wrote:

>
> On Feb 13, 2013, at 12:30 PM, Jesse Bowling <jessebowling at gmail.com>
> wrote:
>
> > I can surmise the problem: Because my interface specification requires
> the use of ';', bash is breaking the command up before it should and
> capstats doesn't know it should quit...The format I'm using
> (p2p1;p2p2;p2p3;p2p4) is making use of PF_RING to listen to all these
> interfaces simultaneously. For snort I have to quote it to prevent it being
> broken up and I suspected something similar is required here as well.
>
> Woah!  PF_RING lets you sniff multiple interfaces that way?  If you give
> that same value to tcpdump (while using the pf_ring libpcap wrapper) does
> it work there too?
>
>   .Seth
>
>
That is my understanding. Anything built against PF_RING's libpcap can use
the notation...However, now that I've put it out on the internet and it's
not apparently common knowledge, I'm doubting myself... ;)

As a reference, straight from (one of) the horses mouths:
http://lists.ntop.org/pipermail/ntop-misc/2012-August/003128.html

Cheers,

Jesse


> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>


-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130213/f94ee551/attachment.html 

From seth at icir.org  Wed Feb 13 10:42:50 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 13 Feb 2013 13:42:50 -0500
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <CAGFhEnmVxVBp4fdbXMf_TW6ov6amQ6mG+O9hOddTiOSaQe7YPw@mail.gmail.com>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
	<CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
	<511AC1C6.9000501@illinois.edu>
	<CAGFhEnkbgQ1XSZ1kFp+bZZCsVktAKusWdgciPNy5VNsi646v-A@mail.gmail.com>
	<CAGFhEn=9Zx=p1TVVK7On1KNGwzVPVh0oTvkZHxf7oNMt9zAntg@mail.gmail.com>
	<8AE1AABA-7BE3-47A7-9F35-92FCAEABA09F@icir.org>
	<CAGFhEnmVxVBp4fdbXMf_TW6ov6amQ6mG+O9hOddTiOSaQe7YPw@mail.gmail.com>
Message-ID: <93A68287-4B9A-47AE-A654-134C4489F0D0@icir.org>


On Feb 13, 2013, at 12:55 PM, Jesse Bowling <jessebowling at gmail.com> wrote:

> That is my understanding. Anything built against PF_RING's libpcap can use the notation...However, now that I've put it out on the internet and it's not apparently common knowledge, I'm doubting myself... ;)
> 
> As a reference, straight from (one of) the horses mouths: http://lists.ntop.org/pipermail/ntop-misc/2012-August/003128.html

And you are using the pf_ring aware drivers?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jessebowling at gmail.com  Wed Feb 13 10:49:41 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Wed, 13 Feb 2013 13:49:41 -0500
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <93A68287-4B9A-47AE-A654-134C4489F0D0@icir.org>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
	<CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
	<511AC1C6.9000501@illinois.edu>
	<CAGFhEnkbgQ1XSZ1kFp+bZZCsVktAKusWdgciPNy5VNsi646v-A@mail.gmail.com>
	<CAGFhEn=9Zx=p1TVVK7On1KNGwzVPVh0oTvkZHxf7oNMt9zAntg@mail.gmail.com>
	<8AE1AABA-7BE3-47A7-9F35-92FCAEABA09F@icir.org>
	<CAGFhEnmVxVBp4fdbXMf_TW6ov6amQ6mG+O9hOddTiOSaQe7YPw@mail.gmail.com>
	<93A68287-4B9A-47AE-A654-134C4489F0D0@icir.org>
Message-ID: <CAGFhEnnMB46uPbz9VyeEeCjiKVGpo-H0j4q8rsMrMWA4aVak1g@mail.gmail.com>

Yes, but not the DNA drivers...

On Wed, Feb 13, 2013 at 1:42 PM, Seth Hall <seth at icir.org> wrote:

>
> On Feb 13, 2013, at 12:55 PM, Jesse Bowling <jessebowling at gmail.com>
> wrote:
>
> > That is my understanding. Anything built against PF_RING's libpcap can
> use the notation...However, now that I've put it out on the internet and
> it's not apparently common knowledge, I'm doubting myself... ;)
> >
> > As a reference, straight from (one of) the horses mouths:
> http://lists.ntop.org/pipermail/ntop-misc/2012-August/003128.html
>
> And you are using the pf_ring aware drivers?
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>


-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130213/1ecb245c/attachment.html 

From dnthayer at illinois.edu  Wed Feb 13 15:25:49 2013
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Wed, 13 Feb 2013 17:25:49 -0600
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <CAGFhEnmVxVBp4fdbXMf_TW6ov6amQ6mG+O9hOddTiOSaQe7YPw@mail.gmail.com>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
	<CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
	<511AC1C6.9000501@illinois.edu>
	<CAGFhEnkbgQ1XSZ1kFp+bZZCsVktAKusWdgciPNy5VNsi646v-A@mail.gmail.com>
	<CAGFhEn=9Zx=p1TVVK7On1KNGwzVPVh0oTvkZHxf7oNMt9zAntg@mail.gmail.com>
	<8AE1AABA-7BE3-47A7-9F35-92FCAEABA09F@icir.org>
	<CAGFhEnmVxVBp4fdbXMf_TW6ov6amQ6mG+O9hOddTiOSaQe7YPw@mail.gmail.com>
Message-ID: <511C20FD.2000705@illinois.edu>

On 02/13/2013 11:55 AM, Jesse Bowling wrote:
>
>
> On Wed, Feb 13, 2013 at 12:46 PM, Seth Hall <seth at icir.org
> <mailto:seth at icir.org>> wrote:
>
>
>     On Feb 13, 2013, at 12:30 PM, Jesse Bowling <jessebowling at gmail.com
>     <mailto:jessebowling at gmail.com>> wrote:
>
>      > I can surmise the problem: Because my interface specification
>     requires the use of ';', bash is breaking the command up before it
>     should and capstats doesn't know it should quit...The format I'm
>     using (p2p1;p2p2;p2p3;p2p4) is making use of PF_RING to listen to
>     all these interfaces simultaneously. For snort I have to quote it to
>     prevent it being broken up and I suspected something similar is
>     required here as well.
>
>     Woah!  PF_RING lets you sniff multiple interfaces that way?  If you
>     give that same value to tcpdump (while using the pf_ring libpcap
>     wrapper) does it work there too?
>
>        .Seth
>
>
> That is my understanding. Anything built against PF_RING's libpcap can
> use the notation...However, now that I've put it out on the internet and
> it's not apparently common knowledge, I'm doubting myself... ;)
>
> As a reference, straight from (one of) the horses mouths:
> http://lists.ntop.org/pipermail/ntop-misc/2012-August/003128.html
>
> Cheers,
>
> Jesse


I'm curious how you're getting things working with semicolons in the
interface name.  Do you have a line like this in your node.cfg:

interface=p2p1;p2p2;p2p3;p2p4




From jessebowling at gmail.com  Wed Feb 13 17:19:21 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Wed, 13 Feb 2013 20:19:21 -0500
Subject: [Bro] LogExpireInterval not respected?
In-Reply-To: <511C20FD.2000705@illinois.edu>
References: <CAGFhEn=i=1NVipWnqLW+u55qOOTc2suF995sM2UOjddYUDcDXw@mail.gmail.com>
	<511963EE.4000704@colorado.edu>
	<CAGFhEnmnO2D3Yh+zU8f6SXrju3Effh=CZQ6oK6xT_vJaxmE5Xg@mail.gmail.com>
	<511A57E6.5020203@colorado.edu>
	<CAGFhEn=8fVNvv+xsAzcHWQoCaTRDJmi-tJ94A0dAgTgRUYC6WA@mail.gmail.com>
	<511AC1C6.9000501@illinois.edu>
	<CAGFhEnkbgQ1XSZ1kFp+bZZCsVktAKusWdgciPNy5VNsi646v-A@mail.gmail.com>
	<CAGFhEn=9Zx=p1TVVK7On1KNGwzVPVh0oTvkZHxf7oNMt9zAntg@mail.gmail.com>
	<8AE1AABA-7BE3-47A7-9F35-92FCAEABA09F@icir.org>
	<CAGFhEnmVxVBp4fdbXMf_TW6ov6amQ6mG+O9hOddTiOSaQe7YPw@mail.gmail.com>
	<511C20FD.2000705@illinois.edu>
Message-ID: <CAGFhEn=oLxsi+pMe7Hc-pGAz8eomqn2HosuqYk3aqgC_7Fvk1Q@mail.gmail.com>

I seem to remember trying to quote it at first, but then found that this
worked:

interface=p2p1\;p2p2\;p2p3\;p2p4

Cheers,

Jesse

On Wed, Feb 13, 2013 at 6:25 PM, Daniel Thayer <dnthayer at illinois.edu>wrote:

> On 02/13/2013 11:55 AM, Jesse Bowling wrote:
>
>>
>>
>> On Wed, Feb 13, 2013 at 12:46 PM, Seth Hall <seth at icir.org
>> <mailto:seth at icir.org>> wrote:
>>
>>
>>     On Feb 13, 2013, at 12:30 PM, Jesse Bowling <jessebowling at gmail.com
>>     <mailto:jessebowling at gmail.com**>> wrote:
>>
>>      > I can surmise the problem: Because my interface specification
>>     requires the use of ';', bash is breaking the command up before it
>>     should and capstats doesn't know it should quit...The format I'm
>>     using (p2p1;p2p2;p2p3;p2p4) is making use of PF_RING to listen to
>>     all these interfaces simultaneously. For snort I have to quote it to
>>     prevent it being broken up and I suspected something similar is
>>     required here as well.
>>
>>     Woah!  PF_RING lets you sniff multiple interfaces that way?  If you
>>     give that same value to tcpdump (while using the pf_ring libpcap
>>     wrapper) does it work there too?
>>
>>        .Seth
>>
>>
>> That is my understanding. Anything built against PF_RING's libpcap can
>> use the notation...However, now that I've put it out on the internet and
>> it's not apparently common knowledge, I'm doubting myself... ;)
>>
>> As a reference, straight from (one of) the horses mouths:
>> http://lists.ntop.org/**pipermail/ntop-misc/2012-**August/003128.html<http://lists.ntop.org/pipermail/ntop-misc/2012-August/003128.html>
>>
>> Cheers,
>>
>> Jesse
>>
>
>
> I'm curious how you're getting things working with semicolons in the
> interface name.  Do you have a line like this in your node.cfg:
>
> interface=p2p1;p2p2;p2p3;p2p4
>
>
>


-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130213/b2a48ca7/attachment.html 

From seth at icir.org  Thu Feb 14 05:34:27 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 14 Feb 2013 08:34:27 -0500
Subject: [Bro] Running external command line programs
Message-ID: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>

In case anyone is interested in playing with this *very* early, a couple of days ago I wrote a wrapper for the input framework to execute command line programs and get the result back into Bro in a non-blocking manner.  It makes stdout, stderr, and the exit code available once your command is done executing.

The script:
	https://github.com/sethhall/bro-junk-drawer/blob/master/exec.bro

An example use of the script:
	https://github.com/sethhall/bro-junk-drawer/blob/master/exec-test.bro

I'd appreciate feedback if anyone tries it, thanks!

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130214/c272d60a/attachment.bin 

From carlopmart at gmail.com  Sun Feb 17 04:06:04 2013
From: carlopmart at gmail.com (carlopmart)
Date: Sun, 17 Feb 2013 12:06:04 +0000
Subject: [Bro] Logrotating and forward bro-id logs
Message-ID: <5120C7AC.9060001@gmail.com>

Hi all,

  I have a problem with logrotation under bro-ids. I use a rsyslog to 
forward all bro-ids logs to a third syslog central server. Problem 
appears when, every hour, broctl cron job rotates all bro logs. Rsyslog 
fails queueing bro-ids logs if third syslog central server is down for 
example.

  Is it possible to rotate bro logs using logrotate program?? If not, is 
it possible to launch a rsyslog restart when broctl cron rotates logs??

Thanks.

-- 
CL Martinez
carlopmart {at} gmail {d0t} com


From seth at icir.org  Tue Feb 19 07:11:29 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 19 Feb 2013 10:11:29 -0500
Subject: [Bro] Running external command line programs
In-Reply-To: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
References: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
Message-ID: <01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>


On Feb 14, 2013, at 8:34 AM, Seth Hall <seth at icir.org> wrote:

> I'd appreciate feedback if anyone tries it, thanks!


I thought I should mention that I did some more updates to make this work better and the current commit that is in my github repository is broken.  We're going to be fixing a bug in Bro and likely including this functionality in Bro 2.2.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130219/014a8eea/attachment.bin 

From seth at icir.org  Tue Feb 19 07:38:31 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 19 Feb 2013 10:38:31 -0500
Subject: [Bro] Running external command line programs
In-Reply-To: <01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>
References: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
	<01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>
Message-ID: <F271949E-7182-483F-8D0D-F70C7CA8A792@icir.org>


On Feb 19, 2013, at 10:11 AM, Seth Hall <seth at icir.org> wrote:

> I thought I should mention that I did some more updates to make this work better and the current commit that is in my github repository is broken.  We're going to be fixing a bug in Bro and likely including this functionality in Bro 2.2.


I just got a question asking about getting a working version.  You can checkout a commit after you clone the repository that does work like this..

	git checkout edf424

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130219/decc01b8/attachment.bin 

From Liam.Randall at gigaco.com  Tue Feb 19 21:09:17 2013
From: Liam.Randall at gigaco.com (Liam Randall)
Date: Wed, 20 Feb 2013 00:09:17 -0500
Subject: [Bro] Shmoocon 2013 Bro IDS and Bro Network Programming Talk
Message-ID: <2697B86A359F6C43BF6FD44F8403D7175A16DA@giga-dc001.GigaCo.local>

Hey guys,

 

This group doesn't seem to mix with the hacker culture as frequently,
however I did a presentation at Shmoocon in DC this weekend.

 

I really tried to do a good job representing a lot of the things I have
seen in the Bro Community over the last few years.  Please feel free to
give me any comments or sugestions.

 

http://www.youtube.com/watch?v=7DCPuHdCbpw

 

So far the response has been overwhelmingly positive; I really hope to
see a large number of interested parties- hackers, companies, and
administrators start to look into Bro as the result of this talk.

 

Thanks,

 

Liam Randall

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130220/7950cf92/attachment.html 

From jessebowling at gmail.com  Wed Feb 20 11:11:48 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Wed, 20 Feb 2013 14:11:48 -0500
Subject: [Bro] Question about data format of ssl.log files
Message-ID: <CAGFhEnkDKXEhCgJZT0Q0cmsF-ekHZTOihXL8scdgZ_GbEosRxw@mail.gmail.com>

Hi,

So quite a few infosec folks are looking at Mandiant's APT1 report, myself
included...When I saw that they included some information on SSL certs in
use I thought "Oh, I'll bet I can check my Bro logs for that!".
Unfortunately, I don't see a way to correlate the info from these reports
with my Bro logs (which is pretty vanilla).

So I suppose my question(s) is/are:

*Has anyone else seen a reliable way to correlate the report data with Bro
logs?
*How might I change my Bro logs so that if I were given this info in the
future I could reliably correlate it?

I'm fairly ignorant about how much of an X509 cert one can see on the wire;
serial number seemed promising but is only "required" to be unique per CA,
Signature Algorithm seems promising, as does Public Key Modulus...

Any suggestions/thoughts from the group?

Cheers,

Jesse

http://intelreport.mandiant.com/
http://intelreport.mandiant.com/Mandiant_APT1_Report.pdf
http://intelreport.mandiant.com/Mandiant_APT1_Report_Appendix.zip

-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130220/a534f7b4/attachment.html 

From sconzo at visiblerisk.com  Wed Feb 20 12:18:57 2013
From: sconzo at visiblerisk.com (Mike Sconzo)
Date: Wed, 20 Feb 2013 14:18:57 -0600
Subject: [Bro] Question about data format of ssl.log files
In-Reply-To: <CAGFhEnkDKXEhCgJZT0Q0cmsF-ekHZTOihXL8scdgZ_GbEosRxw@mail.gmail.com>
References: <CAGFhEnkDKXEhCgJZT0Q0cmsF-ekHZTOihXL8scdgZ_GbEosRxw@mail.gmail.com>
Message-ID: <CAGZoo6HHtWUKuGDsKXdKtLkR+u2GQFgdAJwTkfNUro9+gUAQ2A@mail.gmail.com>

There are a couple different things you can look for.

The serial number works pretty well in a lot of cases (I use this 99%
of the time w/o issue). The subject and issuer are useful for finding
odd SSL certs to begin with. A lof of their subjects and issuers are
pretty trash looking. If you were really paranoid you could combine
the 2 for a more accurate match. I'd also say having a 5 year validity
isn't too normal, but I don't have any hard data to back this up (just
from what I remember from doing analysis). Not to mention when subject
= issuer is also a give away for self signed, and while not
immediately malicious it tends to raise my eyebrow (for example, a
self signed mail.aol.com or mail.yahoo.com cert?). For the default bro
logs I'd look at servername, subject, not valid before and not valid
after; that should give you a reasonable starting place.

As an aside this might be one of my favorites:

C=US, ST=Washington, L=Anytown, O=ACLU, OU=A@@hole,
CN=NoName/emailAddress=iamnot at home.com

Just by looking at subjects and/or issuers that should stand out
because that is not normal for legit network traffic. Sorry for the
tangent, but personally I'd be less worried about the specifics in the
report and more about the chances that it's something not in this
report on your network <FUD alert! :) >

-=Mike



On Wed, Feb 20, 2013 at 1:11 PM, Jesse Bowling <jessebowling at gmail.com> wrote:
> Hi,
>
> So quite a few infosec folks are looking at Mandiant's APT1 report, myself
> included...When I saw that they included some information on SSL certs in
> use I thought "Oh, I'll bet I can check my Bro logs for that!".
> Unfortunately, I don't see a way to correlate the info from these reports
> with my Bro logs (which is pretty vanilla).
>
> So I suppose my question(s) is/are:
>
> *Has anyone else seen a reliable way to correlate the report data with Bro
> logs?
> *How might I change my Bro logs so that if I were given this info in the
> future I could reliably correlate it?
>
> I'm fairly ignorant about how much of an X509 cert one can see on the wire;
> serial number seemed promising but is only "required" to be unique per CA,
> Signature Algorithm seems promising, as does Public Key Modulus...
>
> Any suggestions/thoughts from the group?
>
> Cheers,
>
> Jesse
>
> http://intelreport.mandiant.com/
> http://intelreport.mandiant.com/Mandiant_APT1_Report.pdf
> http://intelreport.mandiant.com/Mandiant_APT1_Report_Appendix.zip
>
> --
> Jesse Bowling
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



--
cat ~/.bash_history > documentation.txt


From jessebowling at gmail.com  Wed Feb 20 12:45:02 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Wed, 20 Feb 2013 15:45:02 -0500
Subject: [Bro] Question about data format of ssl.log files
In-Reply-To: <CAGZoo6HHtWUKuGDsKXdKtLkR+u2GQFgdAJwTkfNUro9+gUAQ2A@mail.gmail.com>
References: <CAGFhEnkDKXEhCgJZT0Q0cmsF-ekHZTOihXL8scdgZ_GbEosRxw@mail.gmail.com>
	<CAGZoo6HHtWUKuGDsKXdKtLkR+u2GQFgdAJwTkfNUro9+gUAQ2A@mail.gmail.com>
Message-ID: <CAGFhEnn2feJAu9G2MLS_uyDjxCBr-fKJ6Q9mv08m5RDYySqqRA@mail.gmail.com>

Thank you Mike!

Using:

awk -F '\t' '($11 == $12) && ($11 != "-") {print $0}' ./ssl.log

has brought me immediate joy in that I found that the primary web site for
our auditors is using a self-signed cert; let us hope that is not a
practice they fault us for. :P

On a serious note though, I do not see serial number in the default ssl.log
file; can someone share the incantation to have it added?

Thanks in advance,

Jesse

On Wed, Feb 20, 2013 at 3:18 PM, Mike Sconzo <sconzo at visiblerisk.com> wrote:

> There are a couple different things you can look for.
>
> The serial number works pretty well in a lot of cases (I use this 99%
> of the time w/o issue). The subject and issuer are useful for finding
> odd SSL certs to begin with. A lof of their subjects and issuers are
> pretty trash looking. If you were really paranoid you could combine
> the 2 for a more accurate match. I'd also say having a 5 year validity
> isn't too normal, but I don't have any hard data to back this up (just
> from what I remember from doing analysis). Not to mention when subject
> = issuer is also a give away for self signed, and while not
> immediately malicious it tends to raise my eyebrow (for example, a
> self signed mail.aol.com or mail.yahoo.com cert?). For the default bro
> logs I'd look at servername, subject, not valid before and not valid
> after; that should give you a reasonable starting place.
>
> As an aside this might be one of my favorites:
>
> C=US, ST=Washington, L=Anytown, O=ACLU, OU=A@@hole,
> CN=NoName/emailAddress=iamnot at home.com
>
> Just by looking at subjects and/or issuers that should stand out
> because that is not normal for legit network traffic. Sorry for the
> tangent, but personally I'd be less worried about the specifics in the
> report and more about the chances that it's something not in this
> report on your network <FUD alert! :) >
>
> -=Mike
>
>
>
> On Wed, Feb 20, 2013 at 1:11 PM, Jesse Bowling <jessebowling at gmail.com>
> wrote:
> > Hi,
> >
> > So quite a few infosec folks are looking at Mandiant's APT1 report,
> myself
> > included...When I saw that they included some information on SSL certs in
> > use I thought "Oh, I'll bet I can check my Bro logs for that!".
> > Unfortunately, I don't see a way to correlate the info from these reports
> > with my Bro logs (which is pretty vanilla).
> >
> > So I suppose my question(s) is/are:
> >
> > *Has anyone else seen a reliable way to correlate the report data with
> Bro
> > logs?
> > *How might I change my Bro logs so that if I were given this info in the
> > future I could reliably correlate it?
> >
> > I'm fairly ignorant about how much of an X509 cert one can see on the
> wire;
> > serial number seemed promising but is only "required" to be unique per
> CA,
> > Signature Algorithm seems promising, as does Public Key Modulus...
> >
> > Any suggestions/thoughts from the group?
> >
> > Cheers,
> >
> > Jesse
> >
> > http://intelreport.mandiant.com/
> > http://intelreport.mandiant.com/Mandiant_APT1_Report.pdf
> > http://intelreport.mandiant.com/Mandiant_APT1_Report_Appendix.zip
> >
> > --
> > Jesse Bowling
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
> --
> cat ~/.bash_history > documentation.txt
>



-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130220/faacee2b/attachment.html 

From jessebowling at gmail.com  Wed Feb 20 13:03:21 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Wed, 20 Feb 2013 16:03:21 -0500
Subject: [Bro] Question about data format of ssl.log files
In-Reply-To: <CAGURB2d5rHp=OACaoWM4Hwf0O7yVzSc1VLB5M-zYC4=1+cNOSg@mail.gmail.com>
References: <CAGFhEnkDKXEhCgJZT0Q0cmsF-ekHZTOihXL8scdgZ_GbEosRxw@mail.gmail.com>
	<CAGZoo6HHtWUKuGDsKXdKtLkR+u2GQFgdAJwTkfNUro9+gUAQ2A@mail.gmail.com>
	<CAGURB2d5rHp=OACaoWM4Hwf0O7yVzSc1VLB5M-zYC4=1+cNOSg@mail.gmail.com>
Message-ID: <CAGFhEn=0U8hYQg1dwCYDm_M7wf-YrdMVGTs1LNC4EtSTjaivrQ@mail.gmail.com>

Thank you Alex; this was the conclusion I came to as well...I was hoping I
missed something in my ignorance of certs...

Now, to harass everyone I can find associated with Mandiant in the hopes of
getting those md5's... :)

Cheers,

Jesse

On Wed, Feb 20, 2013 at 3:57 PM, Alex Waher <alexwis at gmail.com> wrote:

> The most consistent/least-false-positive/sustainable way would be to use
> the public certificate's fingerprint, which bro calculates into an md5 and
> logs within ssl.log as 'cert_hash'. Unfortunately Mandiant's report does
> give us the cert hashes-- there might be intelligence communities out there
> that will soon release an appendix supplement with these fingerprints.
>
> Bro is not by default logging the cert serial number (btw, serial is
> controlled by whom issues the cert), so you'll have to dig through the
> issuer and subject strings to find a match based on the info in the APT1
> report.
>
> -Alex
>
>
> On Wed, Feb 20, 2013 at 12:18 PM, Mike Sconzo <sconzo at visiblerisk.com>wrote:
>
>> There are a couple different things you can look for.
>>
>> The serial number works pretty well in a lot of cases (I use this 99%
>> of the time w/o issue). The subject and issuer are useful for finding
>> odd SSL certs to begin with. A lof of their subjects and issuers are
>> pretty trash looking. If you were really paranoid you could combine
>> the 2 for a more accurate match. I'd also say having a 5 year validity
>> isn't too normal, but I don't have any hard data to back this up (just
>> from what I remember from doing analysis). Not to mention when subject
>> = issuer is also a give away for self signed, and while not
>> immediately malicious it tends to raise my eyebrow (for example, a
>> self signed mail.aol.com or mail.yahoo.com cert?). For the default bro
>> logs I'd look at servername, subject, not valid before and not valid
>> after; that should give you a reasonable starting place.
>>
>> As an aside this might be one of my favorites:
>>
>> C=US, ST=Washington, L=Anytown, O=ACLU, OU=A@@hole,
>> CN=NoName/emailAddress=iamnot at home.com
>>
>> Just by looking at subjects and/or issuers that should stand out
>> because that is not normal for legit network traffic. Sorry for the
>> tangent, but personally I'd be less worried about the specifics in the
>> report and more about the chances that it's something not in this
>> report on your network <FUD alert! :) >
>>
>> -=Mike
>>
>>
>>
>> On Wed, Feb 20, 2013 at 1:11 PM, Jesse Bowling <jessebowling at gmail.com>
>> wrote:
>> > Hi,
>> >
>> > So quite a few infosec folks are looking at Mandiant's APT1 report,
>> myself
>> > included...When I saw that they included some information on SSL certs
>> in
>> > use I thought "Oh, I'll bet I can check my Bro logs for that!".
>> > Unfortunately, I don't see a way to correlate the info from these
>> reports
>> > with my Bro logs (which is pretty vanilla).
>> >
>> > So I suppose my question(s) is/are:
>> >
>> > *Has anyone else seen a reliable way to correlate the report data with
>> Bro
>> > logs?
>> > *How might I change my Bro logs so that if I were given this info in the
>> > future I could reliably correlate it?
>> >
>> > I'm fairly ignorant about how much of an X509 cert one can see on the
>> wire;
>> > serial number seemed promising but is only "required" to be unique per
>> CA,
>> > Signature Algorithm seems promising, as does Public Key Modulus...
>> >
>> > Any suggestions/thoughts from the group?
>> >
>> > Cheers,
>> >
>> > Jesse
>> >
>> > http://intelreport.mandiant.com/
>> > http://intelreport.mandiant.com/Mandiant_APT1_Report.pdf
>> > http://intelreport.mandiant.com/Mandiant_APT1_Report_Appendix.zip
>> >
>> > --
>> > Jesse Bowling
>> >
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>>
>> --
>> cat ~/.bash_history > documentation.txt
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>
>


-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130220/2531eb1c/attachment.html 

From tray at 21ct.com  Wed Feb 20 14:01:36 2013
From: tray at 21ct.com (Tim Ray)
Date: Wed, 20 Feb 2013 16:01:36 -0600
Subject: [Bro] Question about data format of ssl.log files
In-Reply-To: <CAGZoo6HHtWUKuGDsKXdKtLkR+u2GQFgdAJwTkfNUro9+gUAQ2A@mail.gmail.com>
Message-ID: <CD4AA222.551A%tray@21ct.com>

So, the APT1 report has the certs in text format. Does Bro use that? Or is
it all in DER?

On 2/20/13 2:18 PM, "Mike Sconzo" <sconzo at visiblerisk.com> wrote:

>There are a couple different things you can look for.
>
>The serial number works pretty well in a lot of cases (I use this 99%
>of the time w/o issue). The subject and issuer are useful for finding
>odd SSL certs to begin with. A lof of their subjects and issuers are
>pretty trash looking. If you were really paranoid you could combine
>the 2 for a more accurate match. I'd also say having a 5 year validity
>isn't too normal, but I don't have any hard data to back this up (just
>from what I remember from doing analysis). Not to mention when subject
>= issuer is also a give away for self signed, and while not
>immediately malicious it tends to raise my eyebrow (for example, a
>self signed mail.aol.com or mail.yahoo.com cert?). For the default bro
>logs I'd look at servername, subject, not valid before and not valid
>after; that should give you a reasonable starting place.
>
>As an aside this might be one of my favorites:
>
>C=US, ST=Washington, L=Anytown, O=ACLU, OU=A@@hole,
>CN=NoName/emailAddress=iamnot at home.com
>
>Just by looking at subjects and/or issuers that should stand out
>because that is not normal for legit network traffic. Sorry for the
>tangent, but personally I'd be less worried about the specifics in the
>report and more about the chances that it's something not in this
>report on your network <FUD alert! :) >
>
>-=Mike
>
>
>
>On Wed, Feb 20, 2013 at 1:11 PM, Jesse Bowling <jessebowling at gmail.com>
>wrote:
>> Hi,
>>
>> So quite a few infosec folks are looking at Mandiant's APT1 report,
>>myself
>> included...When I saw that they included some information on SSL certs
>>in
>> use I thought "Oh, I'll bet I can check my Bro logs for that!".
>> Unfortunately, I don't see a way to correlate the info from these
>>reports
>> with my Bro logs (which is pretty vanilla).
>>
>> So I suppose my question(s) is/are:
>>
>> *Has anyone else seen a reliable way to correlate the report data with
>>Bro
>> logs?
>> *How might I change my Bro logs so that if I were given this info in the
>> future I could reliably correlate it?
>>
>> I'm fairly ignorant about how much of an X509 cert one can see on the
>>wire;
>> serial number seemed promising but is only "required" to be unique per
>>CA,
>> Signature Algorithm seems promising, as does Public Key Modulus...
>>
>> Any suggestions/thoughts from the group?
>>
>> Cheers,
>>
>> Jesse
>>
>> http://intelreport.mandiant.com/
>> http://intelreport.mandiant.com/Mandiant_APT1_Report.pdf
>> http://intelreport.mandiant.com/Mandiant_APT1_Report_Appendix.zip
>>
>> --
>> Jesse Bowling
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
>
>--
>cat ~/.bash_history > documentation.txt
>_______________________________________________
>Bro mailing list
>bro at bro-ids.org
>http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From seth at icir.org  Wed Feb 20 18:55:28 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 20 Feb 2013 21:55:28 -0500
Subject: [Bro] Question about data format of ssl.log files
In-Reply-To: <CD4AA222.551A%tray@21ct.com>
References: <CD4AA222.551A%tray@21ct.com>
Message-ID: <E6E438E2-C71D-45DA-B84D-E1E94095E0E3@icir.org>


On Feb 20, 2013, at 5:01 PM, Tim Ray <tray at 21ct.com> wrote:

> So, the APT1 report has the certs in text format. Does Bro use that? Or is
> it all in DER?


Mandiant didn't actually distribute the certificates with their report they only included some information about the certs and the certs SHA1 hashes.  Unfortunately for a historical reason our SSL scripts by default only log the MD5 hash of the cert.  We can either get Mandiant to release MD5's for the certs or your can start logging SHA1's going forward.

Here's a script to add SHA-1 hashes for certs to your log (this is a very slight modification to the script we ship with 2.1)?

@load base/protocols/ssl
module SSL;
export {
	redef record Info += {
		## SHA1 hash of the DER encoded server certificate.
		sha1: string &log &optional;
	};
}
event x509_certificate(c: connection, is_orig: bool, cert: X509, chain_idx: count, chain_len: count, der_cert: string) &priority=4
	{
	# We aren't tracking client certificates yet and we are also only tracking
	# the primary cert.  Watch that this came from an SSL analyzed session too.
	if ( is_orig || chain_idx != 0 || ! c?$ssl ) 
		return;

	c$ssl$sha1 = sha1_hash(der_cert);
	}

Have fun.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From tray at 21ct.com  Thu Feb 21 08:52:59 2013
From: tray at 21ct.com (Tim Ray)
Date: Thu, 21 Feb 2013 10:52:59 -0600
Subject: [Bro] Question about data format of ssl.log files
In-Reply-To: <E1AA1BDA-CC9F-4886-8BF2-1E1A3BDF5AC2@icir.org>
Message-ID: <CD4BACDF.559E%tray@21ct.com>

Yeah, that's true. Can we convert their public keys using the openSSL
commands? I gave it a try but got an error early.

On 2/21/13 10:07 AM, "Seth Hall" <seth at icir.org> wrote:

>
>On Feb 20, 2013, at 9:55 PM, Seth Hall <seth at icir.org> wrote:
>
>> On Feb 20, 2013, at 5:01 PM, Tim Ray <tray at 21ct.com> wrote:
>> 
>>> So, the APT1 report has the certs in text format. Does Bro use that?
>>>Or is
>>> it all in DER?
>
>
>And I just realized there is a problem now that I look at the data.
>Mandiant didn't distribute hashes for any of the certificates. :(
>
>  .Seth
>
>--
>Seth Hall
>International Computer Science Institute
>(Bro) because everyone has a network
>http://www.bro-ids.org/
>




From seth at icir.org  Thu Feb 21 08:56:39 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 21 Feb 2013 11:56:39 -0500
Subject: [Bro] Question about data format of ssl.log files
In-Reply-To: <CD4BACDF.559E%tray@21ct.com>
References: <CD4BACDF.559E%tray@21ct.com>
Message-ID: <F20EE481-7A7E-4215-8C1B-E2C831557C11@icir.org>


On Feb 21, 2013, at 11:52 AM, Tim Ray <tray at 21ct.com> wrote:

> Yeah, that's true. Can we convert their public keys using the openSSL
> commands? I gave it a try but got an error early.


They didn't distribute the full certificate.  It's only the output from the openssl command being run on the cert.

 .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Thu Feb 21 08:07:34 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 21 Feb 2013 11:07:34 -0500
Subject: [Bro] Question about data format of ssl.log files
In-Reply-To: <E6E438E2-C71D-45DA-B84D-E1E94095E0E3@icir.org>
References: <CD4AA222.551A%tray@21ct.com>
	<E6E438E2-C71D-45DA-B84D-E1E94095E0E3@icir.org>
Message-ID: <E1AA1BDA-CC9F-4886-8BF2-1E1A3BDF5AC2@icir.org>


On Feb 20, 2013, at 9:55 PM, Seth Hall <seth at icir.org> wrote:

> On Feb 20, 2013, at 5:01 PM, Tim Ray <tray at 21ct.com> wrote:
> 
>> So, the APT1 report has the certs in text format. Does Bro use that? Or is
>> it all in DER?


And I just realized there is a problem now that I look at the data.  Mandiant didn't distribute hashes for any of the certificates. :(

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From bernhard at ICSI.Berkeley.EDU  Thu Feb 21 09:14:39 2013
From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann)
Date: Thu, 21 Feb 2013 09:14:39 -0800
Subject: [Bro] Question about data format of ssl.log files
In-Reply-To: <CD4BACDF.559E%tray@21ct.com>
References: <CD4BACDF.559E%tray@21ct.com>
Message-ID: <CDC36224-D2BC-449F-9B78-A589AC40A583@icsi.berkeley.edu>

Hi,

sadly, the mendicant command only contains the human-readable
output of the certificate information and not the actual certificate.

There is no way to convert that back into the actual certificate
and also no way to get the certificate hashes.

Bernhard

On Feb 21, 2013, at 8:52 AM, Tim Ray <tray at 21ct.com> wrote:

> Yeah, that's true. Can we convert their public keys using the openSSL
> commands? I gave it a try but got an error early.
> 
> On 2/21/13 10:07 AM, "Seth Hall" <seth at icir.org> wrote:
> 
>> 
>> On Feb 20, 2013, at 9:55 PM, Seth Hall <seth at icir.org> wrote:
>> 
>>> On Feb 20, 2013, at 5:01 PM, Tim Ray <tray at 21ct.com> wrote:
>>> 
>>>> So, the APT1 report has the certs in text format. Does Bro use that?
>>>> Or is
>>>> it all in DER?
>> 
>> 
>> And I just realized there is a problem now that I look at the data.
>> Mandiant didn't distribute hashes for any of the certificates. :(
>> 
>> .Seth
>> 
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro-ids.org/
>> 
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



From seth at icir.org  Thu Feb 21 11:57:33 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 21 Feb 2013 14:57:33 -0500
Subject: [Bro] APT1 script
Message-ID: <ED9C23CD-53EA-4737-9241-5744F4004D70@icir.org>

I just finished encapsulating portions of the APT1 data into a fairly naive (but workable) Bro script module.  I'd appreciate any feedback for people that try it!

https://github.com/sethhall/bro-apt1

Not bad for an hour of effort. :P

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130221/d31c8d45/attachment.bin 

From christopher.p.crawford at gmail.com  Thu Feb 21 12:16:46 2013
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Thu, 21 Feb 2013 15:16:46 -0500
Subject: [Bro] Running external command line programs
In-Reply-To: <F271949E-7182-483F-8D0D-F70C7CA8A792@icir.org>
References: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
	<01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>
	<F271949E-7182-483F-8D0D-F70C7CA8A792@icir.org>
Message-ID: <CAG1VgOYKfcgm8yG=niX9HEHfVvtj7hyDtqET+uEhvwumXRsy_Q@mail.gmail.com>

Hey Seth,

Having this type of functionality would be awesome!  It would "unlock" bro
to the point where we would only be limited by our imaginations with what
we could make bro do.

I know you mentioned that the current stuff is broken on github, but I gave
it a try anyways (I modified the command in exec-test.bro to the date
command):

$ bro -r test.pcap exec-test.bro
entering the async whatever
yay!
{
[/tmp/bro-exec-4N1gxc3hF32] = [Thu Feb 21 2013]
}
bro: bro-2.1/src/Trigger.cc:227: bool Trigger::Eval(): Assertion
`frame->GetCall()' failed.
Aborted

So close, and yet so far.

I'm assuming that this is the bug that you mentioned Bro 2.2 will fix.
When is Bro 2.2 expected to be released?

-Chris


On Tue, Feb 19, 2013 at 10:38 AM, Seth Hall <seth at icir.org> wrote:

>
> On Feb 19, 2013, at 10:11 AM, Seth Hall <seth at icir.org> wrote:
>
> > I thought I should mention that I did some more updates to make this
> work better and the current commit that is in my github repository is
> broken.  We're going to be fixing a bug in Bro and likely including this
> functionality in Bro 2.2.
>
>
> I just got a question asking about getting a working version.  You can
> checkout a commit after you clone the repository that does work like this..
>
>         git checkout edf424
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130221/459fc0d7/attachment.html 

From seth at icir.org  Thu Feb 21 12:30:45 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 21 Feb 2013 15:30:45 -0500
Subject: [Bro] Running external command line programs
In-Reply-To: <CAG1VgOYKfcgm8yG=niX9HEHfVvtj7hyDtqET+uEhvwumXRsy_Q@mail.gmail.com>
References: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
	<01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>
	<F271949E-7182-483F-8D0D-F70C7CA8A792@icir.org>
	<CAG1VgOYKfcgm8yG=niX9HEHfVvtj7hyDtqET+uEhvwumXRsy_Q@mail.gmail.com>
Message-ID: <F2034F72-F77D-4C50-B158-226E0C3FAAB9@icir.org>


On Feb 21, 2013, at 3:16 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:

> Hey Seth,
> 
> Having this type of functionality would be awesome!  It would "unlock" bro to the point where we would only be limited by our imaginations with what we could make bro do.
> 
> I know you mentioned that the current stuff is broken on github, but I gave it a try anyways (I modified the command in exec-test.bro to the date command):
> 
> $ bro -r test.pcap exec-test.bro
> entering the async whatever
> yay!
> {
> [/tmp/bro-exec-4N1gxc3hF32] = [Thu Feb 21 2013]
> }
> bro: bro-2.1/src/Trigger.cc:227: bool Trigger::Eval(): Assertion `frame->GetCall()' failed.
> Aborted
> 
> So close, and yet so far.

Yep, that's the bug.  Try checking out the other commit that I suggested.  That should make it work.

Additionally I already have a full module named ActiveHTTP wrapped around it (about 100 lines of code) that uses the curl command line client internally (yes, hacky) but present a very nice and clean API to users.  You currently get the body of the response, the response code, the response message, and all of the headers the server returned.

This sort of opens the door to all kinds of crazy stuff though. Someone (you know who you are!) already mentioned the idea of doing an NMAP wrapper so that people could start NMAP scans and get results back into Bro really easily.

> I'm assuming that this is the bug that you mentioned Bro 2.2 will fix.  When is Bro 2.2 expected to be released?

We aren't quite sure yet, we're furiously working on several big features now.  We're 

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From seth at icir.org  Thu Feb 21 12:50:45 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 21 Feb 2013 15:50:45 -0500
Subject: [Bro] Running external command line programs
In-Reply-To: <F2034F72-F77D-4C50-B158-226E0C3FAAB9@icir.org>
References: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
	<01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>
	<F271949E-7182-483F-8D0D-F70C7CA8A792@icir.org>
	<CAG1VgOYKfcgm8yG=niX9HEHfVvtj7hyDtqET+uEhvwumXRsy_Q@mail.gmail.com>
	<F2034F72-F77D-4C50-B158-226E0C3FAAB9@icir.org>
Message-ID: <B0707FA1-F079-440F-A716-C2FAFD56D670@icir.org>


On Feb 21, 2013, at 3:30 PM, Seth Hall <seth at icir.org> wrote:

>> I'm assuming that this is the bug that you mentioned Bro 2.2 will fix.  When is Bro 2.2 expected to be released?
> 
> We aren't quite sure yet, we're furiously working on several big features now.  We're 


Shit, didn't finish that thought.  Nothing ground breaking was going to follow. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From christopher.p.crawford at gmail.com  Thu Feb 21 13:33:09 2013
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Thu, 21 Feb 2013 16:33:09 -0500
Subject: [Bro] Running external command line programs
In-Reply-To: <F2034F72-F77D-4C50-B158-226E0C3FAAB9@icir.org>
References: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
	<01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>
	<F271949E-7182-483F-8D0D-F70C7CA8A792@icir.org>
	<CAG1VgOYKfcgm8yG=niX9HEHfVvtj7hyDtqET+uEhvwumXRsy_Q@mail.gmail.com>
	<F2034F72-F77D-4C50-B158-226E0C3FAAB9@icir.org>
Message-ID: <CAG1VgObpUGw_=y=4ELv_=Je9=p0h3sm_bPm5NKj0MuJcbvOYVg@mail.gmail.com>

Hey Seth,

I checked out that commit, and it fixes the error, but there is also no
output when I run it.

I modified the new version with some print statements to trace what's going
on:

# exec-test.bro
@load ./exec

event bro_init()
        {
        print "hello";
        Exec::run("date", function(r: Exec::Result) {
                print "test";
                if ( ! r?$stdout )
                        {
                        print "nothing?!?";
                        return;
                        }

                for ( i in r$stdout )
                        {
                        print r$stdout[i];
                        print r$stdout;
                        }
                });
}


# exec.bro
function run(cmd: string, cb: function(r: Result))
        {
        print "run";
        local tmpfile = "/tmp/bro-exec-" + unique_id("");
        system(fmt("touch %s_done", tmpfile));
        system(fmt("touch %s_stdout", tmpfile));
        system(fmt("touch %s_stderr", tmpfile));
        # Sleep for 1 sec before writing to the done file to avoid race
conditions
        # This makes sure that all of the data is read from
        system(fmt("%s 2>>%s_stderr 1>> %s_stdout; echo \"exit_code:${?}\"
> %s_done; sleep 1; echo \"done\" >> %s_done", cmd, tmpfile, tmpfile,
tmpfile, tmpfile));

        results[tmpfile] = [];
        callbacks[tmpfile] = cb;

        schedule 1msec { start_watching_files(tmpfile) };
        print "run finished";
        print cmd;
        print results;
}

When I run that, I get the following output:
$ bro -r test.pcap exec-test.bro
hello
run
run finished
date
{
[/tmp/bro-exec-CRmEOhHjsgk] = [exit_code=0, stdout=<uninitialized>,
stderr=<uninitialized>]
}

Something is happening, though:
$ ls /tmp/
bro-exec-Uvha209VjSk_done  bro-exec-Uvha209VjSk_stderr
bro-exec-Uvha209VjSk_stdout
$ cat /tmp/bro-exec-Uvha209VjSk_stdout
Thu Feb 21 2013

But, that data never makes it to the output in the bro script.

I'm curious why "test" never gets printed.


On Thu, Feb 21, 2013 at 3:30 PM, Seth Hall <seth at icir.org> wrote:

>
> On Feb 21, 2013, at 3:16 PM, Chris Crawford <
> christopher.p.crawford at gmail.com> wrote:
>
> > Hey Seth,
> >
> > Having this type of functionality would be awesome!  It would "unlock"
> bro to the point where we would only be limited by our imaginations with
> what we could make bro do.
> >
> > I know you mentioned that the current stuff is broken on github, but I
> gave it a try anyways (I modified the command in exec-test.bro to the date
> command):
> >
> > $ bro -r test.pcap exec-test.bro
> > entering the async whatever
> > yay!
> > {
> > [/tmp/bro-exec-4N1gxc3hF32] = [Thu Feb 21 2013]
> > }
> > bro: bro-2.1/src/Trigger.cc:227: bool Trigger::Eval(): Assertion
> `frame->GetCall()' failed.
> > Aborted
> >
> > So close, and yet so far.
>
> Yep, that's the bug.  Try checking out the other commit that I suggested.
>  That should make it work.
>
> Additionally I already have a full module named ActiveHTTP wrapped around
> it (about 100 lines of code) that uses the curl command line client
> internally (yes, hacky) but present a very nice and clean API to users.
>  You currently get the body of the response, the response code, the
> response message, and all of the headers the server returned.
>
> This sort of opens the door to all kinds of crazy stuff though. Someone
> (you know who you are!) already mentioned the idea of doing an NMAP wrapper
> so that people could start NMAP scans and get results back into Bro really
> easily.
>
> > I'm assuming that this is the bug that you mentioned Bro 2.2 will fix.
>  When is Bro 2.2 expected to be released?
>
> We aren't quite sure yet, we're furiously working on several big features
> now.  We're
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130221/91c814be/attachment.html 

From seth at icir.org  Thu Feb 21 13:41:07 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 21 Feb 2013 16:41:07 -0500
Subject: [Bro] Running external command line programs
In-Reply-To: <CAG1VgObpUGw_=y=4ELv_=Je9=p0h3sm_bPm5NKj0MuJcbvOYVg@mail.gmail.com>
References: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
	<01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>
	<F271949E-7182-483F-8D0D-F70C7CA8A792@icir.org>
	<CAG1VgOYKfcgm8yG=niX9HEHfVvtj7hyDtqET+uEhvwumXRsy_Q@mail.gmail.com>
	<F2034F72-F77D-4C50-B158-226E0C3FAAB9@icir.org>
	<CAG1VgObpUGw_=y=4ELv_=Je9=p0h3sm_bPm5NKj0MuJcbvOYVg@mail.gmail.com>
Message-ID: <2845E3FB-C320-457A-8AAA-D8D10D844521@icir.org>


On Feb 21, 2013, at 4:33 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:

> But, that data never makes it to the output in the bro script.
> 
> I'm curious why "test" never gets printed.

Bro's shutting down before it gets a chance to. :)

When you run Bro, load the frameworks/communication/listen script.  That will cause Bro not to shut down right after starting up and will give your script a chance to run.

  .Seht


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From christopher.p.crawford at gmail.com  Thu Feb 21 13:47:53 2013
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Thu, 21 Feb 2013 16:47:53 -0500
Subject: [Bro] Running external command line programs
In-Reply-To: <2845E3FB-C320-457A-8AAA-D8D10D844521@icir.org>
References: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
	<01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>
	<F271949E-7182-483F-8D0D-F70C7CA8A792@icir.org>
	<CAG1VgOYKfcgm8yG=niX9HEHfVvtj7hyDtqET+uEhvwumXRsy_Q@mail.gmail.com>
	<F2034F72-F77D-4C50-B158-226E0C3FAAB9@icir.org>
	<CAG1VgObpUGw_=y=4ELv_=Je9=p0h3sm_bPm5NKj0MuJcbvOYVg@mail.gmail.com>
	<2845E3FB-C320-457A-8AAA-D8D10D844521@icir.org>
Message-ID: <CAG1VgOa-5YnF3AwmvPovUq+ESKPetugZjughXUftMQkkoCGXAw@mail.gmail.com>

Hmm...I tried it two ways, with no luck:

$ bro -r test.pcap
/usr/local/bro-2.1/share/bro/policy/frameworks/communication/listen.bro
exec-test.bro
hello
run
run finished
date
{
[/tmp/bro-exec-DdEgoyU0zwf] = [exit_code=0, stdout=<uninitialized>,
stderr=<uninitialized>]
}

and

$ cat exec-test.bro
@load ./exec
@load frameworks/communication/listen

event bro_init()
        {
        print "hello";
        Exec::run("date", function(r: Exec::Result) {
                print "test";
                if ( ! r?$stdout )
                        {
                        print "nothing?!?";
                        return;
                        }

                for ( i in r$stdout )
                        {
                        print r$stdout[i];
                        print r$stdout;
                        }
                });
        }

$ bro -r test.pcap exec-test.bro
hello
run
run finished
date
{
[/tmp/bro-exec-f6eBToBcMd6] = [exit_code=0, stdout=<uninitialized>,
stderr=<uninitialized>]
}

Is there another way to load the listen script?

On Thu, Feb 21, 2013 at 4:41 PM, Seth Hall <seth at icir.org> wrote:

>
> On Feb 21, 2013, at 4:33 PM, Chris Crawford <
> christopher.p.crawford at gmail.com> wrote:
>
> > But, that data never makes it to the output in the bro script.
> >
> > I'm curious why "test" never gets printed.
>
> Bro's shutting down before it gets a chance to. :)
>
> When you run Bro, load the frameworks/communication/listen script.  That
> will cause Bro not to shut down right after starting up and will give your
> script a chance to run.
>
>   .Seht
>
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130221/6adcc5b9/attachment.html 

From seth at icir.org  Thu Feb 21 13:54:35 2013
From: seth at icir.org (Seth Hall)
Date: Thu, 21 Feb 2013 16:54:35 -0500
Subject: [Bro] Running external command line programs
In-Reply-To: <CAG1VgOa-5YnF3AwmvPovUq+ESKPetugZjughXUftMQkkoCGXAw@mail.gmail.com>
References: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
	<01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>
	<F271949E-7182-483F-8D0D-F70C7CA8A792@icir.org>
	<CAG1VgOYKfcgm8yG=niX9HEHfVvtj7hyDtqET+uEhvwumXRsy_Q@mail.gmail.com>
	<F2034F72-F77D-4C50-B158-226E0C3FAAB9@icir.org>
	<CAG1VgObpUGw_=y=4ELv_=Je9=p0h3sm_bPm5NKj0MuJcbvOYVg@mail.gmail.com>
	<2845E3FB-C320-457A-8AAA-D8D10D844521@icir.org>
	<CAG1VgOa-5YnF3AwmvPovUq+ESKPetugZjughXUftMQkkoCGXAw@mail.gmail.com>
Message-ID: <AE5AE4D0-B4E2-4757-8818-7CD944573DC7@icir.org>


On Feb 21, 2013, at 4:47 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:

> Is there another way to load the listen script?

Oh, I think it's because you're reading a packet capture.  When reading packets from a file you can't enable the communication framework.

Just try taking out the -r argument.  You should be able to just put frameworks/communication/listen on the command line too instead of the full file path.  Also, Bro won't terminate until you kill it.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From sanders.geoffrey at gmail.com  Thu Feb 21 19:08:18 2013
From: sanders.geoffrey at gmail.com (Geoff Sanders)
Date: Thu, 21 Feb 2013 22:08:18 -0500
Subject: [Bro] Bro Build on OpenBSD 5.2 amd64
Message-ID: <CAOY4LqS=bS-O0z=mFSye4jKjYvLaWYMUtBNP1LbUonQw5tPQ=Q@mail.gmail.com>

Attempting to build a port on OpenBSD 5.2 amd64 and have gotten to a
specific build error.

Was wondering if anyone might have insight into how to resolve/correct
the issue?

FWIW, although the install docs mention a requirement for BIND8, it
appears that the configure script accepts libbind 6.0 as equivalent.

log is attached. any help would be appreciated.

Thank you.

-- 
 -- Geoff
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build.log
Type: application/octet-stream
Size: 34217 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130221/89bb9a10/attachment.obj 

From luoshoufu at gmail.com  Thu Feb 21 20:21:08 2013
From: luoshoufu at gmail.com (Shoufu Luo)
Date: Thu, 21 Feb 2013 23:21:08 -0500
Subject: [Bro] Bro Build on OpenBSD 5.2 amd64
In-Reply-To: <CAOY4LqS=bS-O0z=mFSye4jKjYvLaWYMUtBNP1LbUonQw5tPQ=Q@mail.gmail.com>
References: <CAOY4LqS=bS-O0z=mFSye4jKjYvLaWYMUtBNP1LbUonQw5tPQ=Q@mail.gmail.com>
Message-ID: <9DC35613-D739-4724-8AAF-3D2926ECD640@gmail.com>

If you are in bro-dev mail list, you probably received one email which provides the solution (by Jon) for the problems shown in your log. 
Quick answer is add a line #include <sys/types.h> before #include <sys/socket.h> for bro_inet_ntop.h and threading/SerialTypes.h

-Shoufu-


On Feb 21, 2013, at 10:08 PM, Geoff Sanders wrote:

Attempting to build a port on OpenBSD 5.2 amd64 and have gotten to a
specific build error.

Was wondering if anyone might have insight into how to resolve/correct
the issue?

FWIW, although the install docs mention a requirement for BIND8, it
appears that the configure script accepts libbind 6.0 as equivalent.

log is attached. any help would be appreciated.

Thank you.

-- 
-- Geoff
<build.log>_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From hammadog at gmail.com  Fri Feb 22 07:26:10 2013
From: hammadog at gmail.com (Tom OBrion)
Date: Fri, 22 Feb 2013 10:26:10 -0500
Subject: [Bro] APT1 script
In-Reply-To: <ED9C23CD-53EA-4737-9241-5744F4004D70@icir.org>
References: <ED9C23CD-53EA-4737-9241-5744F4004D70@icir.org>
Message-ID: <CANCSO1fb4GL8MLPDR5-C4njm4B4XBWLO2DF-Ckuwc0NWoRtk4Q@mail.gmail.com>

Tested and she works wicked good!

Thanks Seth

On Thu, Feb 21, 2013 at 2:57 PM, Seth Hall <seth at icir.org> wrote:

> I just finished encapsulating portions of the APT1 data into a fairly
> naive (but workable) Bro script module.  I'd appreciate any feedback for
> people that try it!
>
> https://github.com/sethhall/bro-apt1
>
> Not bad for an hour of effort. :P
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Tom O'Brion
Twitter: @tobrion
Skype: TomOBrion

"Life is too short to spend time with people who suck the happy out of you."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130222/ce91657b/attachment.html 

From seth at icir.org  Fri Feb 22 07:33:56 2013
From: seth at icir.org (Seth Hall)
Date: Fri, 22 Feb 2013 10:33:56 -0500
Subject: [Bro] APT1 script
In-Reply-To: <CANCSO1fb4GL8MLPDR5-C4njm4B4XBWLO2DF-Ckuwc0NWoRtk4Q@mail.gmail.com>
References: <ED9C23CD-53EA-4737-9241-5744F4004D70@icir.org>
	<CANCSO1fb4GL8MLPDR5-C4njm4B4XBWLO2DF-Ckuwc0NWoRtk4Q@mail.gmail.com>
Message-ID: <CA8FB862-A39A-418D-9DA3-F30FFDFB0207@icir.org>


On Feb 22, 2013, at 10:26 AM, Tom OBrion <hammadog at gmail.com> wrote:

> Tested and she works wicked good!

Great!  Hopefully you aren't finding anything. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/



From hammadog at gmail.com  Fri Feb 22 07:41:50 2013
From: hammadog at gmail.com (Tom OBrion)
Date: Fri, 22 Feb 2013 10:41:50 -0500
Subject: [Bro] APT1 script
In-Reply-To: <CA8FB862-A39A-418D-9DA3-F30FFDFB0207@icir.org>
References: <ED9C23CD-53EA-4737-9241-5744F4004D70@icir.org>
	<CANCSO1fb4GL8MLPDR5-C4njm4B4XBWLO2DF-Ckuwc0NWoRtk4Q@mail.gmail.com>
	<CA8FB862-A39A-418D-9DA3-F30FFDFB0207@icir.org>
Message-ID: <CANCSO1dcb8KKr5feJq-ZTUQwPQSQzr3xW6oitksxmohSAafU3g@mail.gmail.com>

Only this Seth:

[image: Inline image 1]

:)


On Fri, Feb 22, 2013 at 10:33 AM, Seth Hall <seth at icir.org> wrote:

>
> On Feb 22, 2013, at 10:26 AM, Tom OBrion <hammadog at gmail.com> wrote:
>
> > Tested and she works wicked good!
>
> Great!  Hopefully you aren't finding anything. :)
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>


-- 
Tom O'Brion
Twitter: @tobrion
Skype: TomOBrion

"Life is too short to spend time with people who suck the happy out of you."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130222/0b56f7d0/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 18881 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130222/0b56f7d0/attachment.bin 

From christopher.p.crawford at gmail.com  Fri Feb 22 08:59:09 2013
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Fri, 22 Feb 2013 11:59:09 -0500
Subject: [Bro] Running external command line programs
In-Reply-To: <AE5AE4D0-B4E2-4757-8818-7CD944573DC7@icir.org>
References: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
	<01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>
	<F271949E-7182-483F-8D0D-F70C7CA8A792@icir.org>
	<CAG1VgOYKfcgm8yG=niX9HEHfVvtj7hyDtqET+uEhvwumXRsy_Q@mail.gmail.com>
	<F2034F72-F77D-4C50-B158-226E0C3FAAB9@icir.org>
	<CAG1VgObpUGw_=y=4ELv_=Je9=p0h3sm_bPm5NKj0MuJcbvOYVg@mail.gmail.com>
	<2845E3FB-C320-457A-8AAA-D8D10D844521@icir.org>
	<CAG1VgOa-5YnF3AwmvPovUq+ESKPetugZjughXUftMQkkoCGXAw@mail.gmail.com>
	<AE5AE4D0-B4E2-4757-8818-7CD944573DC7@icir.org>
Message-ID: <CAG1VgObqsX+QODXA3sCSW7TFvRsw3idLKtwY24-aLeEURRHoNQ@mail.gmail.com>

Cool -- that did the trick.  This is really good stuff.

I decided to try using this against other bro events, besides just
bro_init():

$ cat exec-test.bro
@load ./exec

event dns_message(c: connection, is_orig: bool, msg: dns_msg, len: count)
        {
        Exec::run("./hello", function(r: Exec::Result)
                {
                if ( ! r?$stdout )
                        {
                        print "nothing?!?";
                        return;
                        }

                for ( i in r$stdout )
                        {
                        print r$stdout[i];
                        }
                });
        }

./hello is just a hello world program.

So, when I test that, I see the following.

$ bro -r test.pcap exec-test.bro
Hello World
Hello World
Hello World
nothing?!?
ERROR: 1361476370.202590 no such index (Exec::results[Exec::name])
(././exec.bro, line 25)
[...]


This got me wondering -- why would exec-test.bro ever have a case where (!
r?$stdout) is true, when I have a program that absolutely returns output
every time it's run? (And then print out "nothing?!?")

For convenience:
# exec.bro
 21 event Exec::stdout_line(description: Input::EventDescription, tpe:
Input::Event, s: string)
 22         {
 23         local name = sub(description$name, /_[^_]*$/, "");
 24
 25         if ( ! results[name]?$stdout )
 26                 results[name]$stdout = vector(s);
 27         else
 28                 results[name]$stdout[|results[name]$stdout|] = s;
 29         }


It's also worth noting that exec.bro really trashes the /tmp directory at
this point.

# exec.bro
 94 function run(cmd: string, cb: function(r: Result))
 95         {
 96         local tmpfile = "/tmp/bro-exec-" + unique_id("");
 97         system(fmt("touch %s_done", tmpfile));
 98         system(fmt("touch %s_stdout", tmpfile));
 99         system(fmt("touch %s_stderr", tmpfile));
100         # Sleep for 1 sec before writing to the done file to avoid race
conditions
101         # This makes sure that all of the data is read from
102         system(fmt("%s 2>>%s_stderr 1>> %s_stdout; echo
\"exit_code:${?}\" > %s_done; sleep 1; echo \"done\" >> %s_done", cmd, tmpf
103
104         results[tmpfile] = [];
105         callbacks[tmpfile] = cb;
106
107         schedule 1msec { start_watching_files(tmpfile) };
108         }

That chunk of code probably needs something like this:

system(fmt("rm %s_done", tmpfile));
system(fmt("rm %s_stdout", tmpfile));
system(fmt("rm %s_stderr", tmpfile));

I'm just not sure where it should go.

-Chris

On Thu, Feb 21, 2013 at 4:54 PM, Seth Hall <seth at icir.org> wrote:

>
> On Feb 21, 2013, at 4:47 PM, Chris Crawford <
> christopher.p.crawford at gmail.com> wrote:
>
> > Is there another way to load the listen script?
>
> Oh, I think it's because you're reading a packet capture.  When reading
> packets from a file you can't enable the communication framework.
>
> Just try taking out the -r argument.  You should be able to just put
> frameworks/communication/listen on the command line too instead of the full
> file path.  Also, Bro won't terminate until you kill it.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130222/9a226230/attachment.html 

From seth at icir.org  Fri Feb 22 09:06:54 2013
From: seth at icir.org (Seth Hall)
Date: Fri, 22 Feb 2013 12:06:54 -0500
Subject: [Bro] Running external command line programs
In-Reply-To: <CAG1VgObqsX+QODXA3sCSW7TFvRsw3idLKtwY24-aLeEURRHoNQ@mail.gmail.com>
References: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
	<01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>
	<F271949E-7182-483F-8D0D-F70C7CA8A792@icir.org>
	<CAG1VgOYKfcgm8yG=niX9HEHfVvtj7hyDtqET+uEhvwumXRsy_Q@mail.gmail.com>
	<F2034F72-F77D-4C50-B158-226E0C3FAAB9@icir.org>
	<CAG1VgObpUGw_=y=4ELv_=Je9=p0h3sm_bPm5NKj0MuJcbvOYVg@mail.gmail.com>
	<2845E3FB-C320-457A-8AAA-D8D10D844521@icir.org>
	<CAG1VgOa-5YnF3AwmvPovUq+ESKPetugZjughXUftMQkkoCGXAw@mail.gmail.com>
	<AE5AE4D0-B4E2-4757-8818-7CD944573DC7@icir.org>
	<CAG1VgObqsX+QODXA3sCSW7TFvRsw3idLKtwY24-aLeEURRHoNQ@mail.gmail.com>
Message-ID: <9A62751B-A09A-494A-88A1-53E5803940FD@icir.org>


On Feb 22, 2013, at 11:59 AM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:

> This got me wondering -- why would exec-test.bro ever have a case where (! r?$stdout) is true, when I have a program that absolutely returns output every time it's run? (And then print out "nothing?!?") 

You don't have to do that check if you know your script will have something on stdout.  I may even make stdout an empty vector by default (as opposed to null).  I can see that argument making sense.

> system(fmt("rm %s_done", tmpfile));
> system(fmt("rm %s_stdout", tmpfile));
> system(fmt("rm %s_stderr", tmpfile));
> 
> I'm just not sure where it should go.

You're running old code. :)  I may not have finished taking care of that yet in the version you're running.  Hopefully this will be in Bro's master branch soon but using when statements and just generally being much nicer.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jessebowling at gmail.com  Sun Feb 24 19:08:32 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Sun, 24 Feb 2013 22:08:32 -0500
Subject: [Bro] Writing a Bro script to make an API call?
Message-ID: <CAGFhEnmB0R4hvQFc2p_kN1OLNUag2bs7GzRqxR+vGXLuAQ-Siw@mail.gmail.com>

Similar to how Bro implements the detect-MHR script, I'd like to do a
lookup against a REST API for hashes on executables...I can do it easily
enough in python but...How can I do it in Bro?

I copied the detect-MHR as a template, but immedietly ran into the
questions of "how do I make an http request with Bro?" and "Will that
request now end up in my http.logs?" and "Does Bro have native abilities to
deal with JSON objects in a reasonable way?" and "What happens if I'm
getting two lines in my response: a csv style line and a JSON "object"?"...

Obviously I have a lot to learn, and would appreciate any resourses I could
be point to for doing so... :)

Cheers,

Jesse

-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130224/4aaf19a5/attachment.html 

From jessebowling at gmail.com  Mon Feb 25 12:45:19 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Mon, 25 Feb 2013 15:45:19 -0500
Subject: [Bro] redef LogElasticSearch variables
Message-ID: <CAGFhEn=AEUZwBLaRn1NtW7S8QOOhh5y75SUJqrGZMuPjNADJGA@mail.gmail.com>

Let me preface this with "I have no idea what I'm doing".

I want to test out Bro's native elasticsearch writer...I found that there
appear to be two files for this module:

bro/base/frameworks/logging/writers/elasticsearch.bro
bro/policy/tuning/logs-to-elasticsearch.bro

Both of them specify that the module is called "LogElasticSearch"...Is that
a problem? At any rate...

I want to specify an ElasticSearch server that is not local. I didn't see
any documentation on this, but saw that elasticsearch.bro has variables
like "server_host". Seems like this would be the thing to change...So, I
tried:

@load tuning/logs-to-elasticsearch
redef LogElasticSearch::server_host = "10.10.10.10"

It appears that broctl does not like this invocation. Specifically it
chokes and says:

   error in
/usr/local/bro/share/bro/policy/frameworks/communication/listen.bro, line
6: syntax error, at or near "module"

Which is weird...If I put additional redef's:

@load tuning/logs-to-elasticsearch
redef LogElasticSearch::server_host = "10.9.12.26"
redef LogElasticSearch::server_port= 9200

I then get:

   error in /usr/local/bro/share/bro/site/local.bro, line 113: syntax
error, at or near "redef"

(line 113 is the last redef of server_port).

So...What am I doing wrong and how do I configure this plugin to point to
another host? Is that book on brogramming out yet? :P

Cheers,

Jesse

-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130225/0ac04022/attachment.html 

From JAzoff at albany.edu  Mon Feb 25 12:50:28 2013
From: JAzoff at albany.edu (Justin Azoff)
Date: Mon, 25 Feb 2013 15:50:28 -0500
Subject: [Bro] redef LogElasticSearch variables
In-Reply-To: <CAGFhEn=AEUZwBLaRn1NtW7S8QOOhh5y75SUJqrGZMuPjNADJGA@mail.gmail.com>
References: <CAGFhEn=AEUZwBLaRn1NtW7S8QOOhh5y75SUJqrGZMuPjNADJGA@mail.gmail.com>
Message-ID: <20130225205028.GS9433@datacomm.albany.edu>

On Mon, Feb 25, 2013 at 03:45:19PM -0500, Jesse Bowling wrote:
> @load tuning/logs-to-elasticsearch
> redef LogElasticSearch::server_host = "10.10.10.10"

The redef line is missing a semicolon.  The error when you leave off a
semicolon can be confusing since you get the error after that line.

-- 
-- Justin Azoff
-- Network Security & Performance Analyst


From seth at icir.org  Mon Feb 25 12:56:25 2013
From: seth at icir.org (Seth Hall)
Date: Mon, 25 Feb 2013 15:56:25 -0500
Subject: [Bro] redef LogElasticSearch variables
In-Reply-To: <CAGFhEn=AEUZwBLaRn1NtW7S8QOOhh5y75SUJqrGZMuPjNADJGA@mail.gmail.com>
References: <CAGFhEn=AEUZwBLaRn1NtW7S8QOOhh5y75SUJqrGZMuPjNADJGA@mail.gmail.com>
Message-ID: <4FC6CA61-A6D7-44B4-9B1F-73ABAF14CB1A@icir.org>


On Feb 25, 2013, at 3:45 PM, Jesse Bowling <jessebowling at gmail.com> wrote:

> bro/base/frameworks/logging/writers/elasticsearch.bro

This is the script level support for the elasticsearch writer.

> bro/policy/tuning/logs-to-elasticsearch.bro

This is a utility script to help you send your logs to ElasticSearch.  It has some tuning options so you can choose if you only want to send certain logs.

> Both of them specify that the module is called "LogElasticSearch"...Is that a problem? At any rate?

Defining a module only sets that to your current namespace.  You can define it multiple times.

I believe Justin answered the rest of your question. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jessebowling at gmail.com  Mon Feb 25 12:57:14 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Mon, 25 Feb 2013 15:57:14 -0500
Subject: [Bro] redef LogElasticSearch variables
In-Reply-To: <4FC6CA61-A6D7-44B4-9B1F-73ABAF14CB1A@icir.org>
References: <CAGFhEn=AEUZwBLaRn1NtW7S8QOOhh5y75SUJqrGZMuPjNADJGA@mail.gmail.com>
	<4FC6CA61-A6D7-44B4-9B1F-73ABAF14CB1A@icir.org>
Message-ID: <CAGFhEn=kFZFsyw_-xWiFNkg1kbMkb=mZvrs_JL8rNbjc6SLfuw@mail.gmail.com>

Ah yes, syntax... :)

Thank you all, that checks out ok.

Cheers,

Jesse

On Mon, Feb 25, 2013 at 3:56 PM, Seth Hall <seth at icir.org> wrote:

>
> On Feb 25, 2013, at 3:45 PM, Jesse Bowling <jessebowling at gmail.com> wrote:
>
> > bro/base/frameworks/logging/writers/elasticsearch.bro
>
> This is the script level support for the elasticsearch writer.
>
> > bro/policy/tuning/logs-to-elasticsearch.bro
>
> This is a utility script to help you send your logs to ElasticSearch.  It
> has some tuning options so you can choose if you only want to send certain
> logs.
>
> > Both of them specify that the module is called "LogElasticSearch"...Is
> that a problem? At any rate?
>
> Defining a module only sets that to your current namespace.  You can
> define it multiple times.
>
> I believe Justin answered the rest of your question. :)
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>


-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130225/9271a11e/attachment.html 

From seth at icir.org  Mon Feb 25 12:57:38 2013
From: seth at icir.org (Seth Hall)
Date: Mon, 25 Feb 2013 15:57:38 -0500
Subject: [Bro] Writing a Bro script to make an API call?
In-Reply-To: <CAGFhEnmB0R4hvQFc2p_kN1OLNUag2bs7GzRqxR+vGXLuAQ-Siw@mail.gmail.com>
References: <CAGFhEnmB0R4hvQFc2p_kN1OLNUag2bs7GzRqxR+vGXLuAQ-Siw@mail.gmail.com>
Message-ID: <3025B5AA-3A2E-4B96-A126-BAD7DD8F73F8@icir.org>


On Feb 24, 2013, at 10:08 PM, Jesse Bowling <jessebowling at gmail.com> wrote:

> Similar to how Bro implements the detect-MHR script, I'd like to do a lookup against a REST API for hashes on executables...I can do it easily enough in python but...How can I do it in Bro?

No, not yet.  I'm hoping that for 2.2 we can get some form of active HTTP into Bro.  I have something implemented in my junk drawer repository already, but it needs a bug fix that hasn't been merged into master yet.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jessebowling at gmail.com  Mon Feb 25 13:01:44 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Mon, 25 Feb 2013 16:01:44 -0500
Subject: [Bro] Writing a Bro script to make an API call?
In-Reply-To: <3025B5AA-3A2E-4B96-A126-BAD7DD8F73F8@icir.org>
References: <CAGFhEnmB0R4hvQFc2p_kN1OLNUag2bs7GzRqxR+vGXLuAQ-Siw@mail.gmail.com>
	<3025B5AA-3A2E-4B96-A126-BAD7DD8F73F8@icir.org>
Message-ID: <CAGFhEnmOJc5dXyT-HD_xpDswhQ1M__Goum=GprGvrJLS-kchwg@mail.gmail.com>

Ah, ok...Well, sounds like it's time for me to try out that external
command script you've mentioned... :)

Cheers,

Jesse

On Mon, Feb 25, 2013 at 3:57 PM, Seth Hall <seth at icir.org> wrote:

>
> On Feb 24, 2013, at 10:08 PM, Jesse Bowling <jessebowling at gmail.com>
> wrote:
>
> > Similar to how Bro implements the detect-MHR script, I'd like to do a
> lookup against a REST API for hashes on executables...I can do it easily
> enough in python but...How can I do it in Bro?
>
> No, not yet.  I'm hoping that for 2.2 we can get some form of active HTTP
> into Bro.  I have something implemented in my junk drawer repository
> already, but it needs a bug fix that hasn't been merged into master yet.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>


-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130225/0cb96d9b/attachment.html 

From christopher.p.crawford at gmail.com  Mon Feb 25 13:18:38 2013
From: christopher.p.crawford at gmail.com (Chris Crawford)
Date: Mon, 25 Feb 2013 16:18:38 -0500
Subject: [Bro] Running external command line programs
In-Reply-To: <9A62751B-A09A-494A-88A1-53E5803940FD@icir.org>
References: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
	<01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>
	<F271949E-7182-483F-8D0D-F70C7CA8A792@icir.org>
	<CAG1VgOYKfcgm8yG=niX9HEHfVvtj7hyDtqET+uEhvwumXRsy_Q@mail.gmail.com>
	<F2034F72-F77D-4C50-B158-226E0C3FAAB9@icir.org>
	<CAG1VgObpUGw_=y=4ELv_=Je9=p0h3sm_bPm5NKj0MuJcbvOYVg@mail.gmail.com>
	<2845E3FB-C320-457A-8AAA-D8D10D844521@icir.org>
	<CAG1VgOa-5YnF3AwmvPovUq+ESKPetugZjughXUftMQkkoCGXAw@mail.gmail.com>
	<AE5AE4D0-B4E2-4757-8818-7CD944573DC7@icir.org>
	<CAG1VgObqsX+QODXA3sCSW7TFvRsw3idLKtwY24-aLeEURRHoNQ@mail.gmail.com>
	<9A62751B-A09A-494A-88A1-53E5803940FD@icir.org>
Message-ID: <CAG1VgOao+7=nbedXhPTV8vUvS2ysJSHj45ewLA3v5F7HJiaL9g@mail.gmail.com>

Hey Seth,

Cool.  I pulled the most recent code from github.   It cleans up /tmp, the
code looks cleaner -- with more when statements -- but it also causes bro
to consume 100% CPU and the only way I can kill bro is by doing a Ctrl-Z
and then a kill -9.  I never get output from the command I add to
Exec::run, so I added some print statements to try to trace where things go
off the tracks:


# exec-test.bro
  1 @load ./exec
  2
  3
  4 event bro_init()
  5         {
  6         print "hello";
  7         when ( local result = Exec::run([$cmd="ls /"]) )
  8                 {
  9                 print "it ran?!?";
 10                 if ( result?$stdout )
 11                         print result$stdout;
 12                 if ( result?$files )
 13                         print result$files;
 14                 }
 15         }

# exec.bro
140 function run(cmd: Command): Result
141         {
142         print "hi";

Then, when I run the following:
$ bro
/usr/local/bro-2.1/share/bro/policy/frameworks/communication/listen.bro
exec-test.bro
hello

So, it looks like bro gets hung up before it can get into Exec::run.

-Chris

On Fri, Feb 22, 2013 at 12:06 PM, Seth Hall <seth at icir.org> wrote:

>
> On Feb 22, 2013, at 11:59 AM, Chris Crawford <
> christopher.p.crawford at gmail.com> wrote:
>
> > This got me wondering -- why would exec-test.bro ever have a case where
> (! r?$stdout) is true, when I have a program that absolutely returns output
> every time it's run? (And then print out "nothing?!?")
>
> You don't have to do that check if you know your script will have
> something on stdout.  I may even make stdout an empty vector by default (as
> opposed to null).  I can see that argument making sense.
>
> > system(fmt("rm %s_done", tmpfile));
> > system(fmt("rm %s_stdout", tmpfile));
> > system(fmt("rm %s_stderr", tmpfile));
> >
> > I'm just not sure where it should go.
>
> You're running old code. :)  I may not have finished taking care of that
> yet in the version you're running.  Hopefully this will be in Bro's master
> branch soon but using when statements and just generally being much nicer.
>
>   .Seth
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130225/63d7f11d/attachment.html 

From seth at icir.org  Mon Feb 25 13:43:00 2013
From: seth at icir.org (Seth Hall)
Date: Mon, 25 Feb 2013 16:43:00 -0500
Subject: [Bro] Writing a Bro script to make an API call?
In-Reply-To: <CAGFhEnmOJc5dXyT-HD_xpDswhQ1M__Goum=GprGvrJLS-kchwg@mail.gmail.com>
References: <CAGFhEnmB0R4hvQFc2p_kN1OLNUag2bs7GzRqxR+vGXLuAQ-Siw@mail.gmail.com>
	<3025B5AA-3A2E-4B96-A126-BAD7DD8F73F8@icir.org>
	<CAGFhEnmOJc5dXyT-HD_xpDswhQ1M__Goum=GprGvrJLS-kchwg@mail.gmail.com>
Message-ID: <EA3BB4E9-6BEC-46BA-A9C1-5962422B38F3@icir.org>


On Feb 25, 2013, at 4:01 PM, Jesse Bowling <jessebowling at gmail.com> wrote:

> Ah, ok...Well, sounds like it's time for me to try out that external command script you've mentioned... :)

The ActiveHTTP module is actually there already too (it wraps the command line curl client). ;)

It does require the topic/jsiwek/ticket946 branch though and there may be problems with doing lots of HTTP requests or Exec commands at runtime due to some thread clean up issues that still exist.

  .Seth



--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From jsiwek at illinois.edu  Mon Feb 25 14:29:55 2013
From: jsiwek at illinois.edu (Siwek, Jonathan Luke)
Date: Mon, 25 Feb 2013 22:29:55 +0000
Subject: [Bro] Running external command line programs
In-Reply-To: <CAG1VgOao+7=nbedXhPTV8vUvS2ysJSHj45ewLA3v5F7HJiaL9g@mail.gmail.com>
References: <96DFC1E0-D40F-45C9-B958-377F15DD2F0D@icir.org>
	<01F50422-7AD0-43F8-A676-ABC7A27BB21A@icir.org>
	<F271949E-7182-483F-8D0D-F70C7CA8A792@icir.org>
	<CAG1VgOYKfcgm8yG=niX9HEHfVvtj7hyDtqET+uEhvwumXRsy_Q@mail.gmail.com>
	<F2034F72-F77D-4C50-B158-226E0C3FAAB9@icir.org>
	<CAG1VgObpUGw_=y=4ELv_=Je9=p0h3sm_bPm5NKj0MuJcbvOYVg@mail.gmail.com>
	<2845E3FB-C320-457A-8AAA-D8D10D844521@icir.org>
	<CAG1VgOa-5YnF3AwmvPovUq+ESKPetugZjughXUftMQkkoCGXAw@mail.gmail.com>
	<AE5AE4D0-B4E2-4757-8818-7CD944573DC7@icir.org>
	<CAG1VgObqsX+QODXA3sCSW7TFvRsw3idLKtwY24-aLeEURRHoNQ@mail.gmail.com>
	<9A62751B-A09A-494A-88A1-53E5803940FD@icir.org>
	<CAG1VgOao+7=nbedXhPTV8vUvS2ysJSHj45ewLA3v5F7HJiaL9g@mail.gmail.com>
Message-ID: <B3E27C90B7D2DD4CAEF09215A29EB0D116F3BE2F@CITESMBX2.ad.uillinois.edu>


On Feb 25, 2013, at 3:18 PM, Chris Crawford <christopher.p.crawford at gmail.com> wrote:

> Cool.  I pulled the most recent code from github.   It cleans up /tmp, the code looks cleaner -- with more when statements -- but it also causes bro to consume 100% CPU and the only way I can kill bro is by doing a Ctrl-Z and then a kill -9.  I never get output from the command I add to Exec::run,

The best chance you have of running that code is if you also build Bro against the topic/jsiwek/ticket946 branch.  Follow [1] if you want to track the status of that branch getting merged in to git/master.

    Jon

[1] http://tracker.bro-ids.org/bro/ticket/946


From Carl.Hester at constellation.com  Tue Feb 26 06:18:19 2013
From: Carl.Hester at constellation.com (Hester, Carl)
Date: Tue, 26 Feb 2013 09:18:19 -0500
Subject: [Bro] Extracted files not being archived
Message-ID: <05306CCF5FA2B141B08DD66FE981F834038918C43368@EXM-MSW-05.Ceg.Corp.Net>

While working through the file-extraction demo posted by @hectaman (http://www.youtube.com/watch?v=-7p3yLHxug4), I noticed my http-item_* files would go missing whenever I stopped the bro processes.  It looks like files are properly written to bro/spool/bro, but not rotated or archived.

I'm digging through the scripts in bro/share/broctl/scripts and trying to identify the process for log rotation, but figured someone may have already solved this if they've seen similar behavior.

Thanks,
Carl
This e-mail and any attachments are confidential, may contain legal,
professional or other privileged information, and are intended solely for the
addressee. If you are not the intended recipient, do not use the information
in this e-mail in any way, delete this e-mail and notify the sender. -EXCIP
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130226/8b2ad7e4/attachment.html 

From seth at icir.org  Tue Feb 26 07:38:33 2013
From: seth at icir.org (Seth Hall)
Date: Tue, 26 Feb 2013 10:38:33 -0500
Subject: [Bro] Extracted files not being archived
In-Reply-To: <05306CCF5FA2B141B08DD66FE981F834038918C43368@EXM-MSW-05.Ceg.Corp.Net>
References: <05306CCF5FA2B141B08DD66FE981F834038918C43368@EXM-MSW-05.Ceg.Corp.Net>
Message-ID: <A13D1114-5EC7-41AC-913E-8F1B4F705709@icir.org>


On Feb 26, 2013, at 9:18 AM, "Hester, Carl" <Carl.Hester at constellation.com> wrote:

> While working through the file-extraction demo posted by @hectaman (http://www.youtube.com/watch?v=-7p3yLHxug4), I noticed my http-item_* files would go missing whenever I stopped the bro processes.  It looks like files are properly written to bro/spool/bro, but not rotated or archived. 

Ah!  Now your twitter posts make sense.  Unfortunately we don't support file extraction very well when run with BroControl.  *Technically* we should be writing them out to some directory other than the spool directory, but honestly I'm not ever sure how this might interact with log rotation (although log rotation for non-logging framework files should be disabled anyway).

This is an area that you might have a bit of a hard time getting anyone to focus on right now because we're going to be ripping out most of the code that Liam pointed out in his video in the coming weeks and replacing the functionality with the in-development file analysis framework.

Probably not a very satisfying answer for you right now, but it is what it is. :)  If you tell us more about what you're trying to accomplish we may be able to figure out some easy way for you to get it working though.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From init.conf at gmail.com  Tue Feb 26 10:27:08 2013
From: init.conf at gmail.com (Aashish SHARMA)
Date: Tue, 26 Feb 2013 10:27:08 -0800
Subject: [Bro] Extracted files not being archived
In-Reply-To: <A13D1114-5EC7-41AC-913E-8F1B4F705709@icir.org>
References: <05306CCF5FA2B141B08DD66FE981F834038918C43368@EXM-MSW-05.Ceg.Corp.Net>
	<A13D1114-5EC7-41AC-913E-8F1B4F705709@icir.org>
Message-ID: <142B1F8C-4E76-420A-A1D3-CDD2DFBC55A3@gmail.com>

For the time being, I extract the files into a different directory - extracted files will be written to these folders and would persist restarts.  

in your local.bro: 

redef SMTP::extraction_prefix = "/data/bro/smtp-extract/smtp-entity" &redef;
redef HTTP::extraction_prefix = "/data/bro/http-extract/http_item" &redef; 

Needless to mention, you need to create the two directories smtp-extract and http-extract. 

You should be grep on the extracted filename in the rotated log files to get more detailed information about the connection etc. 

Hope this helps, 

Aashish 

On Feb 26, 2013, at 7:38 AM, Seth Hall <seth at icir.org> wrote:

> 
> On Feb 26, 2013, at 9:18 AM, "Hester, Carl" <Carl.Hester at constellation.com> wrote:
> 
>> While working through the file-extraction demo posted by @hectaman (http://www.youtube.com/watch?v=-7p3yLHxug4), I noticed my http-item_* files would go missing whenever I stopped the bro processes.  It looks like files are properly written to bro/spool/bro, but not rotated or archived. 
> 
> Ah!  Now your twitter posts make sense.  Unfortunately we don't support file extraction very well when run with BroControl.  *Technically* we should be writing them out to some directory other than the spool directory, but honestly I'm not ever sure how this might interact with log rotation (although log rotation for non-logging framework files should be disabled anyway).
> 
> This is an area that you might have a bit of a hard time getting anyone to focus on right now because we're going to be ripping out most of the code that Liam pointed out in his video in the coming weeks and replacing the functionality with the in-development file analysis framework.
> 
> Probably not a very satisfying answer for you right now, but it is what it is. :)  If you tell us more about what you're trying to accomplish we may be able to figure out some easy way for you to get it working though.
> 
>  .Seth
> 
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro-ids.org/
> 
> 
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro




From charles.a.fair2.mil at mail.mil  Wed Feb 27 08:45:14 2013
From: charles.a.fair2.mil at mail.mil (Fair, Charles A SSG USARMY NG NGB ARNG PEC (US))
Date: Wed, 27 Feb 2013 16:45:14 +0000
Subject: [Bro] Bro and filesystem data on a host (UNCLASSIFIED)
In-Reply-To: <4FC6CA61-A6D7-44B4-9B1F-73ABAF14CB1A@icir.org>
References: <CAGFhEn=AEUZwBLaRn1NtW7S8QOOhh5y75SUJqrGZMuPjNADJGA@mail.gmail.com>
	<4FC6CA61-A6D7-44B4-9B1F-73ABAF14CB1A@icir.org>
Message-ID: <71AB2EAAE9C6C746845D6F402A49D0DF4FB915@utinhpzr.easf.csd.disa.mil>

Classification: UNCLASSIFIED
Caveats: NONE

Seth,

We spoke at the 2012 Bro Exchange about how Bro can be used on a filesystem of a host or such, brain a bit fuzzy this early in the morning at 10:36 :)  Could you expand on the topic a bit/point me in the right direction?

Thanks,

SSG Charles "Chuck" A. Fair (not a CISSP, but passed the test)
Information Systems/Information Assurance NCO
Information Technology Training Center, PEC, Camp Robinson AR



Classification: UNCLASSIFIED
Caveats: NONE





From seth at icir.org  Wed Feb 27 08:52:27 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 27 Feb 2013 11:52:27 -0500
Subject: [Bro] Bro and filesystem data on a host (UNCLASSIFIED)
In-Reply-To: <71AB2EAAE9C6C746845D6F402A49D0DF4FB915@utinhpzr.easf.csd.disa.mil>
References: <CAGFhEn=AEUZwBLaRn1NtW7S8QOOhh5y75SUJqrGZMuPjNADJGA@mail.gmail.com>
	<4FC6CA61-A6D7-44B4-9B1F-73ABAF14CB1A@icir.org>
	<71AB2EAAE9C6C746845D6F402A49D0DF4FB915@utinhpzr.easf.csd.disa.mil>
Message-ID: <55A23E4C-26DA-4B22-A76B-3A89B67AB8FD@icir.org>


On Feb 27, 2013, at 11:45 AM, "Fair, Charles A SSG USARMY NG NGB ARNG PEC (US)" <charles.a.fair2.mil at mail.mil> wrote:

> We spoke at the 2012 Bro Exchange about how Bro can be used on a filesystem of a host or such, brain a bit fuzzy this early in the morning at 10:36 :)  Could you expand on the topic a bit/point me in the right direction?


How are you looking to use it?  We have the input framework in 2.1 for reading from inputs that we have plugins for (essentially only Bro logs and text files right now).  We may have quite a bit more functionality regarding that in 2.2.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From charles.a.fair2.mil at mail.mil  Wed Feb 27 09:03:51 2013
From: charles.a.fair2.mil at mail.mil (Fair, Charles A SSG USARMY NG NGB ARNG PEC (US))
Date: Wed, 27 Feb 2013 17:03:51 +0000
Subject: [Bro] Bro and filesystem data on a host (UNCLASSIFIED)
In-Reply-To: <55A23E4C-26DA-4B22-A76B-3A89B67AB8FD@icir.org>
References: <CAGFhEn=AEUZwBLaRn1NtW7S8QOOhh5y75SUJqrGZMuPjNADJGA@mail.gmail.com>
	<4FC6CA61-A6D7-44B4-9B1F-73ABAF14CB1A@icir.org>
	<71AB2EAAE9C6C746845D6F402A49D0DF4FB915@utinhpzr.easf.csd.disa.mil>
	<55A23E4C-26DA-4B22-A76B-3A89B67AB8FD@icir.org>
Message-ID: <71AB2EAAE9C6C746845D6F402A49D0DF4FB934@utinhpzr.easf.csd.disa.mil>

Classification: UNCLASSIFIED
Caveats: NONE

If I understand correctly, the input framework is the way that log files,
for instance from a host, can be ingested by Bro?  One of the things I was
interested in doing was identifying key information from a log, such as a MS
Windows Event log, via event viewer to syslog, with network traffic.  This
would be similar to how Bro can analyze SSL Certs.  

What I was wondering about was what could Bro do with a filesystem beyond
log files?  An example on a MS system would be identifying the last run time
of a file via prefetch data that was communicating over a socket, that was
identified by Bro.  Of course this is assuming that Bro has access to the
filesystem of the system in question.  

Regards, 

SSG Charles "Chuck" A. Fair
Information Systems/Information Assurance NCO 
Information Technology Training Center, PEC, Camp Robinson AR 


-----Original Message-----
From: Seth Hall [mailto:seth at icir.org] 
Sent: Wednesday, February 27, 2013 10:52 AM
To: Fair, Charles A SSG USARMY NG NGB ARNG PEC (US)
Cc: bro at bro-ids.org; Seth Hall
Subject: Re: Bro and filesystem data on a host (UNCLASSIFIED)


On Feb 27, 2013, at 11:45 AM, "Fair, Charles A SSG USARMY NG NGB ARNG PEC
(US)" <charles.a.fair2.mil at mail.mil> wrote:

> We spoke at the 2012 Bro Exchange about how Bro can be used on a
filesystem of a host or such, brain a bit fuzzy this early in the morning at
10:36 :)  Could you expand on the topic a bit/point me in the right
direction?


How are you looking to use it?  We have the input framework in 2.1 for
reading from inputs that we have plugins for (essentially only Bro logs and
text files right now).  We may have quite a bit more functionality regarding
that in 2.2.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/


Classification: UNCLASSIFIED
Caveats: NONE


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5627 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130227/c01f6d90/attachment.bin 

From seth at icir.org  Wed Feb 27 09:11:43 2013
From: seth at icir.org (Seth Hall)
Date: Wed, 27 Feb 2013 12:11:43 -0500
Subject: [Bro] Bro and filesystem data on a host (UNCLASSIFIED)
In-Reply-To: <71AB2EAAE9C6C746845D6F402A49D0DF4FB934@utinhpzr.easf.csd.disa.mil>
References: <CAGFhEn=AEUZwBLaRn1NtW7S8QOOhh5y75SUJqrGZMuPjNADJGA@mail.gmail.com>
	<4FC6CA61-A6D7-44B4-9B1F-73ABAF14CB1A@icir.org>
	<71AB2EAAE9C6C746845D6F402A49D0DF4FB915@utinhpzr.easf.csd.disa.mil>
	<55A23E4C-26DA-4B22-A76B-3A89B67AB8FD@icir.org>
	<71AB2EAAE9C6C746845D6F402A49D0DF4FB934@utinhpzr.easf.csd.disa.mil>
Message-ID: <25E905B6-2D40-49F8-A7E4-AF70E4F25E9D@icir.org>


On Feb 27, 2013, at 12:03 PM, "Fair, Charles A SSG USARMY NG NGB ARNG PEC (US)" <charles.a.fair2.mil at mail.mil> wrote:

> If I understand correctly, the input framework is the way that log files,
> for instance from a host, can be ingested by Bro?

It's a bit more comprehensive than that since the input framework is plugin based and we will be releasing plugins for more things over time (databases, etc).

> One of the things I was
> interested in doing was identifying key information from a log, such as a MS
> Windows Event log, via event viewer to syslog, with network traffic.

I don't want to talk about it publicly yet because things are still a little unclear, but I've been having concrete discussions with a couple of people related to this functionality.  It's definitely on our radar.

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/




From rjenkins at rmjconsulting.net  Wed Feb 27 10:51:13 2013
From: rjenkins at rmjconsulting.net (Ron Jenkins)
Date: Wed, 27 Feb 2013 18:51:13 +0000
Subject: [Bro] Bro IDS logging via Syslog
Message-ID: <B6D41C594B5D2E44BE282C07CD71CEE4161D7F50@rmjcs-fs001>

Is there a way to have Bro v2.1 send via Syslog along with a log file?


Thanks!

Ron Jenkins (SnortCP, VCP (3/4), MCNE, CNE6, MCP,CCNA)
RMJ Consulting, LLC. "Bringing Companies and Solutions Together"
Makers of Active Response System(ARS) & Log Siphon
Owner / Senior Architect
Physical Address
11715 Bricksome Ave STE B-7
Baton Rouge, LA 70816
Mail Address
7575 Jefferson Hwy #103
Baton Rouge, LA 70806
Toll: 855-448-5214
Direct. 225-448-5214
Fax. 225-448-5324
Cell. 225-931-1632
Email. rjenkins at rmjconsulting.net
Web. http://www.rmjconsulting.net<http://www.rmjconsulting.net/>
ARS. http://www.rmjars.com<http://www.rmjars.com/>
Log Siphon. http://www.logsiphon.com<http://www.logsiphon.com/>
Linkedin. http://www.linkedin.com/profile/view?id=28564151&trk=tab_pro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130227/a770868c/attachment.html 

From jessebowling at gmail.com  Wed Feb 27 11:53:38 2013
From: jessebowling at gmail.com (Jesse Bowling)
Date: Wed, 27 Feb 2013 14:53:38 -0500
Subject: [Bro] Bro IDS logging via Syslog
In-Reply-To: <B6D41C594B5D2E44BE282C07CD71CEE4161D7F50@rmjcs-fs001>
References: <B6D41C594B5D2E44BE282C07CD71CEE4161D7F50@rmjcs-fs001>
Message-ID: <CAGFhEnkWyMbtsyje2gSp=M4en-CmY-GU-hSheeGHKnGePRVmCA@mail.gmail.com>

There is almost certainly a better way to do it within the Bro framework
itself, but another option might be to use built in (?) rsyslog:

http://ossectools.blogspot.com/2011/09/bro-quickstart-cluster-edition.html

About halfway down there are instructions for using rsyslog's imfile module
to syslog Bro's logs...

Cheers,

Jesse

On Wed, Feb 27, 2013 at 1:51 PM, Ron Jenkins <rjenkins at rmjconsulting.net>wrote:

>  Is there a way to have Bro v2.1 send via Syslog along with a log file?***
> *
>
> ** **
>
> ** **
>
> Thanks!****
>
> ** **
>
> Ron Jenkins (SnortCP, VCP (3/4), MCNE, CNE6, MCP,CCNA)****
>
> *RMJ Consulting, LLC. *"*Bringing Companies and Solutions Together*"****
>
> Makers of *Active Response System(ARS)* & *Log Siphon*****
>
> Owner / Senior Architect****
>
> *Physical Address*
>
> 11715 Bricksome Ave STE B-7****
>
> Baton Rouge, LA 70816****
>
> *Mail Address*
>
> 7575 Jefferson Hwy #103****
>
> Baton Rouge, LA 70806****
>
> *Toll: *855-448-5214****
>
> *Direct*. 225-448-5214****
>
> *Fax.* 225-448-5324****
>
> *Cell.* 225-931-1632****
>
> *Email.* rjenkins at rmjconsulting.net****
>
> *Web.* http://www.rmjconsulting.net****
>
> *ARS.* http://www.rmjars.com****
>
> *Log Siphon*. http://www.logsiphon.com****
>
> *Linkedin.* http://www.linkedin.com/profile/view?id=28564151&trk=tab_pro**
> **
>
> ** **
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>



-- 
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130227/8f87cb0d/attachment.html 

From rjenkins at rmjconsulting.net  Wed Feb 27 11:56:05 2013
From: rjenkins at rmjconsulting.net (Ron Jenkins)
Date: Wed, 27 Feb 2013 19:56:05 +0000
Subject: [Bro] Bro IDS logging via Syslog
In-Reply-To: <CAGFhEnkWyMbtsyje2gSp=M4en-CmY-GU-hSheeGHKnGePRVmCA@mail.gmail.com>
References: <B6D41C594B5D2E44BE282C07CD71CEE4161D7F50@rmjcs-fs001>
	<CAGFhEnkWyMbtsyje2gSp=M4en-CmY-GU-hSheeGHKnGePRVmCA@mail.gmail.com>
Message-ID: <B6D41C594B5D2E44BE282C07CD71CEE4161D97B8@rmjcs-fs001>

Thank you for the response!

I just completed setting syslog-ng and now have the log files sending via syslog to Log Siphon now.

I agree, that it would be great to have it built into the framework directly.

Have a good day!


From: Jesse Bowling [mailto:jessebowling at gmail.com]
Sent: Wednesday, February 27, 2013 1:54 PM
To: Ron Jenkins
Cc: bro at bro-ids.org
Subject: Re: [Bro] Bro IDS logging via Syslog

There is almost certainly a better way to do it within the Bro framework itself, but another option might be to use built in (?) rsyslog:

http://ossectools.blogspot.com/2011/09/bro-quickstart-cluster-edition.html

About halfway down there are instructions for using rsyslog's imfile module to syslog Bro's logs...

Cheers,

Jesse
On Wed, Feb 27, 2013 at 1:51 PM, Ron Jenkins <rjenkins at rmjconsulting.net<mailto:rjenkins at rmjconsulting.net>> wrote:
Is there a way to have Bro v2.1 send via Syslog along with a log file?


Thanks!

Ron Jenkins (SnortCP, VCP (3/4), MCNE, CNE6, MCP,CCNA)
RMJ Consulting, LLC. "Bringing Companies and Solutions Together"
Makers of Active Response System(ARS) & Log Siphon
Owner / Senior Architect
Physical Address
11715 Bricksome Ave STE B-7
Baton Rouge, LA 70816
Mail Address
7575 Jefferson Hwy #103
Baton Rouge, LA 70806
Toll: 855-448-5214<tel:855-448-5214>
Direct. 225-448-5214<tel:225-448-5214>
Fax. 225-448-5324<tel:225-448-5324>
Cell. 225-931-1632<tel:225-931-1632>
Email. rjenkins at rmjconsulting.net<mailto:rjenkins at rmjconsulting.net>
Web. http://www.rmjconsulting.net<http://www.rmjconsulting.net/>
ARS. http://www.rmjars.com<http://www.rmjars.com/>
Log Siphon. http://www.logsiphon.com<http://www.logsiphon.com/>
Linkedin. http://www.linkedin.com/profile/view?id=28564151&trk=tab_pro


_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



--
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130227/85255695/attachment.html 

From rjenkins at rmjconsulting.net  Wed Feb 27 14:09:00 2013
From: rjenkins at rmjconsulting.net (Ron Jenkins)
Date: Wed, 27 Feb 2013 22:09:00 +0000
Subject: [Bro] Log Siphon with Bro IDS data imported.
Message-ID: <B6D41C594B5D2E44BE282C07CD71CEE4161DA041@rmjcs-fs001>

Good afternoon;

I thought you may find the below image links interesting.

http://www.logsiphon.com/images/snap/bro-ids_1.png

http://www.logsiphon.com/images/snap/bro-ids_2.png

http://www.logsiphon.com/images/snap/bro-ids_3.png

http://www.logsiphon.com/images/snap/bro-ids_4.png<http://www.logsiphon.com/images/snap/bro-ids_3.png>



Thanks!

Ron Jenkins (SnortCP, VCP (3/4), MCNE, CNE6, MCP,CCNA)
RMJ Consulting, LLC. "Bringing Companies and Solutions Together"
Makers of Active Response System(ARS) & Log Siphon
Owner / Senior Architect
Physical Address
11715 Bricksome Ave STE B-7
Baton Rouge, LA 70816
Mail Address
7575 Jefferson Hwy #103
Baton Rouge, LA 70806
Toll: 855-448-5214
Direct. 225-448-5214
Fax. 225-448-5324
Cell. 225-931-1632
Email. rjenkins at rmjconsulting.net
Web. http://www.rmjconsulting.net<http://www.rmjconsulting.net/>
ARS. http://www.rmjars.com<http://www.rmjars.com/>
Log Siphon. http://www.logsiphon.com<http://www.logsiphon.com/>
Linkedin. http://www.linkedin.com/profile/view?id=28564151&trk=tab_pro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130227/f7fbc558/attachment.html 

From rjenkins at rmjconsulting.net  Thu Feb 28 08:11:46 2013
From: rjenkins at rmjconsulting.net (Ron Jenkins)
Date: Thu, 28 Feb 2013 16:11:46 +0000
Subject: [Bro] Log Siphon Decoding Bro IDS Events.
Message-ID: <B6D41C594B5D2E44BE282C07CD71CEE4161E702A@rmjcs-fs001>

Good morning

Building into next release of Log Siphon the ability to display the decoded and raw event data.

The below link is a screenshot of a Bro connection log event.

http://www.logsiphon.com/images/snap/bro-ids_5.png

If you have any suggestions or input on this new feature, I would be interested in hearing about it.


Thanks!

Ron Jenkins (SnortCP, VCP (3/4), MCNE, CNE6, MCP,CCNA)
RMJ Consulting, LLC. "Bringing Companies and Solutions Together"
Makers of Active Response System(ARS) & Log Siphon
Owner / Senior Architect
Physical Address
11715 Bricksome Ave STE B-7
Baton Rouge, LA 70816
Mail Address
7575 Jefferson Hwy #103
Baton Rouge, LA 70806
Toll: 855-448-5214
Direct. 225-448-5214
Fax. 225-448-5324
Cell. 225-931-1632
Email. rjenkins at rmjconsulting.net
Web. http://www.rmjconsulting.net<http://www.rmjconsulting.net/>
ARS. http://www.rmjars.com<http://www.rmjars.com/>
Log Siphon. http://www.logsiphon.com<http://www.logsiphon.com/>
Linkedin. http://www.linkedin.com/profile/view?id=28564151&trk=tab_pro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130228/8420e6c3/attachment.html