[Bro-Dev] Update on log management

[Martin Holste]

> To be honest it pained me a little bit when I saw your code to insert the ascii logs into a database (however I can't fault anyone for doing things out of operational necessity).

LOL!  Yes, this is very much out of operational necessity.  I would
love to hear how other incident response teams are using Bro in
production (beyond PoC setups), as right now, it doesn't integrate
with anything enterprise, and therefore becomes much more limited than
it should be given all of its sophisticated abilities.

> The benefits to outputting logs directly to a database from Bro is that you get to take advantage of logging framework features.

Yes, we all want this to be integrated inside Bro.  The reason that my
script snarfer works efficiently, ugly and inelegant as it is, is
because you are guaranteed that the effort and errors of dealing with
the output from Bro is not Bro's problem.  This is critical for things
like database errors, network lag, and CPU overhead.  Please, please
make sure that you make these proposed writers async and in their own
thread.  Do not make the same mistakes Snort made by having the output
in the main thread.  Output is a very easy task to delegate to a child
thread or fork, and because Bro does an incredible amount of logging,
you don't want to be wasting precious CPU cycles in the network
detection thread on output.  My single Bro instance is overwhelmed on
any traffic over 200 Mb/sec on a decent machine.

>
> event bro_init()
>        {
>        Log::add_filter(SSL::SSL, [$name="failed_to_pg",
>                $writer=POSTGRESQL_WRITER,
>                $path="ssl_invalid_cert_session",   # This equates to the table name in postgresql
>                $pred(rec: SSL::Info) = { return (rec$validation_status != "ok"); }]);
>        }
>
> All of the SSL logs would still be written to the normal (ascii by default) logs but all of the sessions using invalid certificates would also be sent to postgresql.
>

Yep, this is good stuff.  I'd settle for this working on files, as I
can limp along on my scripts until the Pg plugin is ready.

> In terms of future plans we're really just at the point where we need more writer plugins, most of the rest of the code is finished.  Does that answer your questions?
>

Almost.  The next question is: and then what?  So let's say you've got
all of this data in Pg.  Do you have anyone working on a frontend for
this?  Or is accessing the data Bro puts out considered wholly
out-of-scope for the Bro project?  Again, I'd love to hear from other
IR team leads as to how they are using this data to either trigger or
supplement investigations.