[Bro] Manager memory requirements for the intel framework

Brian OBerry brian.oberry at bluvector.io
Tue May 15 12:16:42 PDT 2018


Thanks, I don’t control the intel input and can’t answer the questions, but will forward this to those who can.  What do you mean by “… have another class of problems soon?”

From: Michał Purzyński <michalpurzynski1 at gmail.com>
Date: Tuesday, May 15, 2018 at 1:50 PM
To: Brian OBerry <brian.oberry at bluvector.io>
Cc: Drew Dixon <dwdixon at umich.edu>, "bro at bro.org" <bro at bro.org>
Subject: EXT: Re: [Bro] Manager memory requirements for the intel framework

100k indicators sounds like your matches will be mostly noise anyway. Does your source have some metadata like

- category (to tell CDN from CnC)
- last time seen malicious

And so on and so forth, so you can decide what to load, instead of loading everything?
I’m not saying Bro cannot handle that, it most likely can, I’m saying you’ll have another class of problems soon.

On May 15, 2018, at 10:16 AM, Brian OBerry <brian.oberry at bluvector.io<mailto:brian.oberry at bluvector.io>> wrote:
Thanks, Drew,

Agreed that it would be best to upgrade, and we moved on to Bro 2.5.3 a couple releases ago.  No, misc/scan is not loaded.  Unfortunately, I don’t have the production intel input file.  I’m told it’s about 100K indicators of about 50% domain, 25% addr, and 25% hash, plus a few email and url indicators less than 1%.  We started testing with a file of 18K domain indicators, but will be expanding to a more realistic profile next.  Maybe I started this thread a little early, without the representative input file, but I wanted to query the community for any ideas or suggestions about our initial results.  Thank you for your ideas!

Brian

From: Drew Dixon <dwdixon at umich.edu<mailto:dwdixon at umich.edu>>
Date: Tuesday, May 15, 2018 at 12:35 PM
To: Brian OBerry <brian.oberry at bluvector.io<mailto:brian.oberry at bluvector.io>>
Cc: "bro at bro.org<mailto:bro at bro.org>" <bro at bro.org<mailto:bro at bro.org>>
Subject: EXT: Re: [Bro] Manager memory requirements for the intel framework

It may be worth upgrading your production system either way since I do not believe 2.4.x is technically supported any longer being that it's a release from 2015.  Plus, lots of improvements since then...

I'm not sure if this will help your specific problem but I'd be curious to know if you're also running the default scan detection script on this box?  In your local.bro are you loading "misc/scan"?  If so, that script is known to be a huge memory hog and could be indirectly contributing to your high memory usage...just wanted to mention that, since you mentioned you commented out the conditional that invokes “event Intel::new_item(item)” the problem may reside more directly with the Intel Framework.

How many indicators of each indicator type are you loading into the Intel Framework?

-Drew

On Tue, May 15, 2018 at 9:13 AM, Brian OBerry <brian.oberry at bluvector.io<mailto:brian.oberry at bluvector.io>> wrote:
Hello, All,

We’re trying to understand manager memory requirements when the intel framework is in use, after experiencing multiple manager crashes per day when using the framework on a low-bandwidth (less than 1Gbps) CentOS 6 machine running a production Bro 2.4.1 cluster.  These are happening because the manager is exhausting its tcmalloc heap limit of 16G, as reported in its stderr.log.  We removed the heap limit on an idle (no network traffic) Bro 2.4.1 test system, and found the parent VSize reported by “broctl top manager” went to 27G for an intel input file of 18K unique Intel::DOMAIN items.  It remained at 27G after many cycles of replacing the input file with 18K new unique items.
Restoring the heap limit and attaching gdb to the manager on the test system shows a malloc failure backtrace that comes out of RemoteSerializer::SendCall ().  We commented the conditional that invokes “event Intel::new_item(item)” in base/frameworks/intel/main.bro to disable remote synchronization with the workers, and the huge VSize disappeared.
We then built bro from master (version 2.5-569) and retested.  The manager VSize is much lower, but is still about 15G.
Any advice on how to proceed with further diagnostics to hopefully reign in the manager memory requirements for intel?  It doesn’t appear at first blush that upgrading Bro will fix it, at least not entirely, and we’re reluctant to upgrade the production system without fully understanding the problem.
Thanks,
Brian

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20180515/89e82424/attachment-0001.html 


More information about the Bro mailing list