[Bro] local.bro causing memory leak

[Benjamin Wood]

I'm wondering that now too.

At some point we realized that events weren't making it all the way to
splunk. But I couldn't tell you why. I'm going to be taking a step back and
re-evaluate why the forwarder didn't work.

I also figured out that using path_func in this way is a very BAD IDEA.
I've spent a couple days crawling through the source code for the logging
framework and testing some things.

For every unique filename, a new thread will be started and a new writer
will be created on a per file basis. The bad news is, there is no way to
reap these threads when they are no longer needed.

The only "in framework" process that will close file descriptors and reap
process threads is rotation. Even if you enable rotation, files can still
slip through, because rotation seems to only be effective against the
current writer.

I don't know if this breaks a contract outlined in the docs, but it seems
like if this is an intended use of path_func then this is a bug that should
be fixed.

The only way that I could resolve the problem in bro alone, would be to
author a custom log writer that would name the files in the way I wanted,
and close these dangling file descriptors.

It's a pretty complicated problem, and I'm planning to abstract it away buy
using the features in a splunk forwarder, or using Apache NiFi to manage
the logs if that fails.

Thanks,
Ben

On Wed, Mar 21, 2018 at 3:04 PM, fatema bannatwala <
fatema.bannatwala at gmail.com> wrote:

> Hey Ben,
>
> So, if the whole purpose of doing file renaming was just for Splunk
> streaming,
> then I wonder why it won't work for you to just have forwarder keep
> monitoring the log files in the current Bro log dir.
> We are also a Splunk shop, and index some of our Bro logs into Splunk
> using Splunk forwarder running on our Bro manager, and just monitoring
> logs/current/ dir for the types of logs we want to index.
> It's a very basic setup, works without any issues on our side. wonder why
> it would create a problem in your situation. Hmm (or I might have
> mis-interpreted the problem :) ).
>
> Fatema.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180321/73e74399/attachment.html