[Bro] Manager memory requirements for the intel framework

[Drew Dixon]

It may be worth upgrading your production system either way since I do not
believe 2.4.x is technically supported any longer being that it's a release
from 2015.  Plus, lots of improvements since then...

I'm not sure if this will help your specific problem but I'd be curious to
know if you're also running the default scan detection script on this box?
In your local.bro are you loading "misc/scan"?  If so, that script is known
to be a huge memory hog and could be indirectly contributing to your high
memory usage...just wanted to mention that, since you mentioned you
commented out the conditional that invokes “event Intel::new_item(item)”
the problem may reside more directly with the Intel Framework.

How many indicators of each indicator type are you loading into the Intel
Framework?

-Drew

On Tue, May 15, 2018 at 9:13 AM, Brian OBerry <brian.oberry at bluvector.io>
wrote:

> Hello, All,
>
>
>
> We’re trying to understand manager memory requirements when the intel
> framework is in use, after experiencing multiple manager crashes per day
> when using the framework on a low-bandwidth (less than 1Gbps) CentOS 6
> machine running a production Bro 2.4.1 cluster.  These are happening
> because the manager is exhausting its tcmalloc heap limit of 16G, as
> reported in its stderr.log.  We removed the heap limit on an idle (no
> network traffic) Bro 2.4.1 test system, and found the parent VSize reported
> by “broctl top manager” went to 27G for an intel input file of 18K unique
> Intel::DOMAIN items.  It remained at 27G after many cycles of replacing the
> input file with 18K new unique items.
>
> Restoring the heap limit and attaching gdb to the manager on the test
> system shows a malloc failure backtrace that comes out of
> RemoteSerializer::SendCall ().  We commented the conditional that invokes
> “event Intel::new_item(item)” in base/frameworks/intel/main.bro to disable
> remote synchronization with the workers, and the huge VSize disappeared.
>
> We then built bro from master (version 2.5-569) and retested.  The manager
> VSize is much lower, but is still about 15G.
>
> Any advice on how to proceed with further diagnostics to hopefully reign
> in the manager memory requirements for intel?  It doesn’t appear at first
> blush that upgrading Bro will fix it, at least not entirely, and we’re
> reluctant to upgrade the production system without fully understanding the
> problem.
>
> Thanks,
>
> Brian
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180515/921f1226/attachment.html