[Bro] local.bro causing memory leak

[Benjamin Wood]

I may have solved the problem. I don't believe this was actually a memory
leak. It appears to be a problem with max user processes instead. I upped
my ulimits for bro and it works now.

"ulimit -u" was set to 4096. I upped it to 65536, and that seems to have
resolved the problem.

It was a little challenging to narrow down, because I didn't have debug on,
and "Resource temporarily unavailable" wasn't telling me WHICH resource it
was trying to allocate, just that it couldn't. If I have problems in the
future, or upgrade, I'll definitely be enabling debug so I can get better
information for problems like this.

I'm still not sure if bro is leaving files open, but digging into the
source it looks like it will clean up file descriptors independent of the
log rotation interval being set.
https://github.com/bro/bro/blob/a8c0580b45157793da22984f700f92cb3a5745d5/src/File.cc#L357

Thanks,
Ben

On Tue, Mar 20, 2018 at 10:24 AM, Benjamin Wood <ben.bt.wood at gmail.com>
wrote:

> I now have the diag output for the crash. I think I will be using a custom
> routine to identify and "close" files on a regular basis.
>
> [BroControl] > diag manager
> [manager]
>
> No core file found.  You may need to change your system settings to
> allow core files.
>
> Bro 2.5.2
> Linux 3.10.0-693.17.1.el7.x86_64
>
> Bro plugins: (none found)
>
> ==== No reporter.log
>
> ==== stderr.log
> /usr/local/bro/share/broctl/scripts/run-bro: line 61: ulimit: core file
> size: cannot modify limit: Operation not permitted
> terminate called after throwing an instance of 'std::system_error'
>   what():  Resource temporarily unavailable
> /usr/local/bro/share/broctl/scripts/run-bro: line 110: 144420
> Aborted                 nohup "$mybro" "$@"
>
> ==== stdout.log
> max memory size         (kbytes, -m) unlimited
> data seg size           (kbytes, -d) unlimited
> virtual memory          (kbytes, -v) unlimited
> core file size          (blocks, -c) 0
>
> ==== .cmdline
> -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl
> base/frameworks/cluster local-manager.bro broctl/auto
>
> ==== .env_vars
> PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/
> scripts:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/
> sbin:/opt/dell/srvadmin/bin:/home/bro/.local/bin:/home/bro/bin
> BROPATH=/usr/local/bro/spool/installed-scripts-do-not-
> touch/site::/usr/local/bro/spool/installed-scripts-do-
> not-touch/auto:/usr/local/bro/share/bro:/usr/local/bro/
> share/bro/policy:/usr/local/bro/share/bro/site
> CLUSTER_NODE=manager
>
> ==== .status
> RUNNING [net_run]
>
> ==== No prof.log
>
> ==== No packet_filter.log
>
> ==== No loaded_scripts.log
>
> Thanks,
> Ben
>
> On Mon, Mar 19, 2018 at 3:31 PM, Benjamin Wood <ben.bt.wood at gmail.com>
> wrote:
>
>> I've got some custom log names happening, and it's causing a memory leak.
>> Bro never closes the file descriptors or releases the objects. This is
>> causing the manager to crash over a period of time.
>>
>> I'm running my cluster with broctl, and rotation is turned off because
>> I'm naming files with a timestamp to begin with.
>>
>> Any suggestions on how to perform a periodic "clean up"?
>>
>> function datepath(id: Log::ID, path: string, rec: any) : string
>> {
>>     local filter = Log::get_filter(id, "default");
>>     return string_cat(filter$path, strftime("_%F_%H", current_time()));
>> }
>>
>> event bro_init() {
>>     Log::disable_stream(Syslog::LOG);
>>
>>     for ( id in Log::active_streams ) {
>>         local filter = Log::get_filter(id, "default");
>>         filter$path_func = datepath;
>>         Log::add_filter(id, filter);
>>     }
>> }
>>
>> Thanks,
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180320/a24717c1/attachment.html