[Bro] local.bro causing memory leak

[Benjamin Wood]

Thanks Seth.

The whole problem I'm trying to solve is steaming data into splunk. Splunk
forwarder's don't like it when filenames change, and the artificial delay
created by rotating logs adds too much latency. The solution that was
proposed was "don't rotate logs", and leave them in place long enough for
the forwarders to finish.

At this point I've got to step back and ask, "Am I doing it wrong?" This
problem has to have been solved by others. I'm certain there is a way to
stream my data to splunk that is better than this.

The file rotation and renaming functions give me enough to play with to
solve the problem using bro-script.

Thanks again for the feedback,
Ben

On Tue, Mar 20, 2018 at 2:50 PM, Seth Hall <seth at corelight.com> wrote:

> On 19 Mar 2018, at 15:31, Benjamin Wood wrote:
>
> I'm running my cluster with broctl, and rotation is turned off because I'm
> naming files with a timestamp to begin with.
>
> Justin got your problem right. If you turn off file rotation, then Bro is
> never closing any of these hourly logs. You have to be really careful with
> how you use $path_func because you can easily get yourself into hot water.
>
> Alternately you need to define a rotation interval and post processor.
> Something like this...
>
> <i also trimmed out some of your code>
>
> function my_log_post_processor(info: Log::RotationInfo): bool
>     {
>     local ext = sub(info$fname, /^[^\-]+-[0-9]+-[0-9]+-[0-9]+_[0-9]+\.[0-9]+\.[0-9]+\./, "");
>
>     # Move file to name including both opening and closing time.
>     local dst = fmt("%s_%s_%s-%s.%s", info$path, strftime("%Y%m%d", info$open),
>                                                  strftime("%H:%M:%S", info$open),
>                                                  strftime("%H:%M:%S%z", info$close),
>                                                  ext);
>     local cmd = fmt("/bin/mv %s %s/%s", info$fname, "/data/logs", dst);
>     system(cmd);
>
>     return T;
>     }
> event bro_init()
>     {
>     for ( id in Log::active_streams )
>         {
>         local filter = Log::get_filter(id, "default");
>         filter$interv = 1hr;
>         filter$postprocessor = my_log_post_processor;
>         Log::add_filter(id, filter);
>         }
>     }
>
> Something like that will enable you to turn off log rotation in broctl
> (but you'll lose some broctl niceties as well).
>
> .Seth
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180320/8d9edf82/attachment.html