[Bro] local.bro causing memory leak

[Benjamin Wood]

It didn't solve the problem. It just removed the roadblock. After doing a
full "restart" on the cluster, lsof reports 2K+ files. While before reset
it reported 1M+. So I still need to figure out a way to clean up those
leftover file descriptors.

On Tue, Mar 20, 2018 at 12:55 PM, Benjamin Wood <ben.bt.wood at gmail.com>
wrote:

> I may have solved the problem. I don't believe this was actually a memory
> leak. It appears to be a problem with max user processes instead. I upped
> my ulimits for bro and it works now.
>
> "ulimit -u" was set to 4096. I upped it to 65536, and that seems to have
> resolved the problem.
>
> It was a little challenging to narrow down, because I didn't have debug
> on, and "Resource temporarily unavailable" wasn't telling me WHICH resource
> it was trying to allocate, just that it couldn't. If I have problems in the
> future, or upgrade, I'll definitely be enabling debug so I can get better
> information for problems like this.
>
> I'm still not sure if bro is leaving files open, but digging into the
> source it looks like it will clean up file descriptors independent of the
> log rotation interval being set.
> https://github.com/bro/bro/blob/a8c0580b45157793da22984f700f92
> cb3a5745d5/src/File.cc#L357
>
> Thanks,
> Ben
>
> On Tue, Mar 20, 2018 at 10:24 AM, Benjamin Wood <ben.bt.wood at gmail.com>
> wrote:
>
>> I now have the diag output for the crash. I think I will be using a
>> custom routine to identify and "close" files on a regular basis.
>>
>> [BroControl] > diag manager
>> [manager]
>>
>> No core file found.  You may need to change your system settings to
>> allow core files.
>>
>> Bro 2.5.2
>> Linux 3.10.0-693.17.1.el7.x86_64
>>
>> Bro plugins: (none found)
>>
>> ==== No reporter.log
>>
>> ==== stderr.log
>> /usr/local/bro/share/broctl/scripts/run-bro: line 61: ulimit: core file
>> size: cannot modify limit: Operation not permitted
>> terminate called after throwing an instance of 'std::system_error'
>>   what():  Resource temporarily unavailable
>> /usr/local/bro/share/broctl/scripts/run-bro: line 110: 144420
>> Aborted                 nohup "$mybro" "$@"
>>
>> ==== stdout.log
>> max memory size         (kbytes, -m) unlimited
>> data seg size           (kbytes, -d) unlimited
>> virtual memory          (kbytes, -v) unlimited
>> core file size          (blocks, -c) 0
>>
>> ==== .cmdline
>> -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl
>> base/frameworks/cluster local-manager.bro broctl/auto
>>
>> ==== .env_vars
>> PATH=/usr/local/bro/bin:/usr/local/bro/share/broctl/scripts:
>> /usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/
>> dell/srvadmin/bin:/home/bro/.local/bin:/home/bro/bin
>> BROPATH=/usr/local/bro/spool/installed-scripts-do-not-touch/
>> site::/usr/local/bro/spool/installed-scripts-do-not-
>> touch/auto:/usr/local/bro/share/bro:/usr/local/bro/share/
>> bro/policy:/usr/local/bro/share/bro/site
>> CLUSTER_NODE=manager
>>
>> ==== .status
>> RUNNING [net_run]
>>
>> ==== No prof.log
>>
>> ==== No packet_filter.log
>>
>> ==== No loaded_scripts.log
>>
>> Thanks,
>> Ben
>>
>> On Mon, Mar 19, 2018 at 3:31 PM, Benjamin Wood <ben.bt.wood at gmail.com>
>> wrote:
>>
>>> I've got some custom log names happening, and it's causing a memory
>>> leak. Bro never closes the file descriptors or releases the objects. This
>>> is causing the manager to crash over a period of time.
>>>
>>> I'm running my cluster with broctl, and rotation is turned off because
>>> I'm naming files with a timestamp to begin with.
>>>
>>> Any suggestions on how to perform a periodic "clean up"?
>>>
>>> function datepath(id: Log::ID, path: string, rec: any) : string
>>> {
>>>     local filter = Log::get_filter(id, "default");
>>>     return string_cat(filter$path, strftime("_%F_%H", current_time()));
>>> }
>>>
>>> event bro_init() {
>>>     Log::disable_stream(Syslog::LOG);
>>>
>>>     for ( id in Log::active_streams ) {
>>>         local filter = Log::get_filter(id, "default");
>>>         filter$path_func = datepath;
>>>         Log::add_filter(id, filter);
>>>     }
>>> }
>>>
>>> Thanks,
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180320/b43d7a3d/attachment.html