[Bro] logging locally and to remote logger

[Johanna Amann]

Hi Erik,

the workers log to one directory each, to not conflict with each other. If
you have several active workers on one machine, they cannot local-log to
the same directory/file because they would conflict with each other and
you would get files where different workers might write into lines of
other workers.

As soon as you want merged logs from more than one Bro instance, you need
remote-logging (even if the manager/logger for the workers is on the same
machine).

I hope this helps :)
 Johanna

On Tue, Nov 15, 2016 at 07:15:55AM -0500, erik clark wrote:
> Ah, I think there is some confusion. Out of the box if you log locally as
> well as using a remote logger (2.5), the logs locally get shoved into
> worker buckets. I was hoping to see how it would be possible to get
> standard cluster behavior, where all workers log locally to one bucket
> instead of each worker having its own bucket.
> 
> Anyone know why this logs to separate buckets in the first place?
> 
> On Tue, Nov 15, 2016 at 6:31 AM, william de ping <bill.de.ping at gmail.com>
> wrote:
> 
> > Hi,
> >
> > If you wish to log locally and you care about the worker-id who produce
> > this logged event :
> >
> >    - to know what is the worker-id you can add a field "worker" to your
> >    logs and populate it from bro script using : get_event_peer()$descr
> >    - to change the rotation for each log (here, rotate every 200 minutes)
> >    you need to use
> >    - LOG::remove_default_filter(SSH::LOG);
> >       - and then add LOG::add_filter(SSH::LOG, [$name="ssh",$path="ssh",
> >       *$interv=200min*, $include=("field1","field2") ]
> >       - btw, you can set $path to be a mounted dir
> >    - to save the log to another machine simultaneously :
> >       - use bro, add a new writer (https://www.bro.org/sphinx/
> >       scripts/base/frameworks/logging/main.bro.html#id-Log::default_writer
> >       <https://www.bro.org/sphinx/scripts/base/frameworks/logging/main.bro.html#id-Log::default_writer>)
> >       and then add_filter to ssh and ask it to use the new writer
> >       - use syslog, just monitor this main local log and transmit it to
> >       another machine
> >
> > Hope it helps
> >
> > On Mon, Nov 14, 2016 at 4:35 PM, erik clark <philosnef at gmail.com> wrote:
> >
> >> So, if I use:
> >>
> >> redef Log::enable_local_logging
> >>
> >> in a bro worker cluster, what I find is that all the logs go to
> >> /data/bro/spool/worker-1-X instead of all in /data/bro/logs/current on the
> >> local machine... Is there a way to fix this?
> >>
> >> Also, I would want to rotate logs out on the workers that are doing
> >> additional local logging to have a much more constrained timeframe for
> >> logging, specifically 1 week for local nodes, and 3 months for the logger
> >> host.
> >>
> >> Is the best way to do this just with a cron rm -rf /data/bro/logs/$date ?
> >> It seems this would run into a conflict with broctlconfig....
> >>
> >> Thanks!
> >>
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >>
> >
> >

> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro