[Bro-Dev] Update on log management

[Seth Hall]

On Aug 31, 2011, at 12:10 PM, Martin Holste wrote:

>> The benefits to outputting logs directly to a database from Bro is that you get to take advantage of logging framework features.
> 
> Yes, we all want this to be integrated inside Bro.  The reason that my
> script snarfer works efficiently, ugly and inelegant as it is, is
> because you are guaranteed that the effort and errors of dealing with
> the output from Bro is not Bro's problem.

The "normal" Bro deployment is as a cluster at this point anyway where the manager is dedicated to notice handling and logging (as you've noticed, standalone instances basically suck for anything over 80Mbps).  Also, Gilbert has been spending the summer threading the logging framework and I *think* his branch is probably close to being integrated.  We basically planned on threading the logging framework from the start for all of the reasons that you mentioned. :)

I started inserting Bro logs into PostgreSQL a long time ago at OSU too with my bro-dblogger project...
	https://github.com/sethhall/bro-dblogger

If you read the README for that project closely, you can see way back then that I was already heading down the path that we ended up going down with the new logging framework. ;)

> Yep, this is good stuff.  I'd settle for this working on files, as I
> can limp along on my scripts until the Pg plugin is ready.

The rest of the filter actually works already except for the $writer. :)

>> In terms of future plans we're really just at the point where we need more writer plugins, most of the rest of the code is finished.  Does that answer your questions?
> 
> Almost.  The next question is: and then what?  So let's say you've got
> all of this data in Pg.  Do you have anyone working on a frontend for
> this?  Or is accessing the data Bro puts out considered wholly
> out-of-scope for the Bro project?  

We've been poking around at various people and places trying to figure out what a Bro interface would look like and do.  I suspect we aren't too far off from movement in this area, but we have no plans yet.

> Again, I'd love to hear from other
> IR team leads as to how they are using this data to either trigger or
> supplement investigations.

I'm hoping to have a couple of people talk about this specifically at the Bro workshop.

  .Seth


--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro-ids.org/