[Bro] Multiple masters to ease the workload

Gary Faulkner gfaulkner.nsm at gmail.com
Fri Jun 5 11:21:04 PDT 2015

I used to see some stability issues on a reasonably large network (100K 
plus hosts) that appeared related to software asset tracking, coupled 
with a lot of IP churn due to wireless devices and short lease times, 
which resulted in the manager being oom-killed after some number of 
hours or days. It was suggested, I think by Seth, that I experiment with 
disabling it in local.bro:

I'm still on 2.3-419, so not the latest build by any means, but was able 
to test this, and have seen greatly improved stability in my particular 
environment, with the below line in local.bro:

redef Software::asset_tracking=NO_HOSTS;

It might be worth experimenting with if you think you may be having 
software asset tracking related issues. Be mindful if you rely on that 
data for other scripts.


On 6/5/2015 12:40 PM, Dave Crawford wrote:
> When I recently debugged memory exhaustion on my workers the root 
> cause was related to the software detection scripts, specifically 
> /protocols/http/software. If that script is running on a sensor that 
> is monitoring the inside interface of a web proxy it tracks all the 
> remote software as being on your local network.
> On Jun 5, 2015, at 12:09 PM, Mark Buchanan <mabuchan at gmail.com 
> <mailto:mabuchan at gmail.com>> wrote:
>> I too have noticed memory complete memory exhaustion in Bro 2.3.2 
>> (not sure what version Jason is running).  If the workers are not 
>> restarted every few days or at least once a week, I run out of usable 
>> memory on a few sensors I'm testing.
>> I have found that just doing a restart within broctl will free the 
>> memory consumed up - but I regularly have to perform restarts to keep 
>> the sensors I am testing running smoothly.
>> Mark
>> On Fri, Jun 5, 2015 at 7:08 AM, Close, Jason M. <close at ou.edu 
>> <mailto:close at ou.edu>> wrote:
>>     Thanks.
>>     I went ahead and rebooted the cluster, and that cleared things up
>>     (as well as sent out a LOT of emails…).
>>     Has anyone else noticed a memory leak in the sensors?  We slowly
>>     see memory usage grow, maybe by about 10GB a month, even when our
>>     total traffic has gone down.  I attached an image from our Zabbix
>>     monitor.  You can see that once we reboot the box, memory drops
>>     down, and then slowly creeps up.  And traffic isn’t increasing
>>     (in fact, it decreases by half over the summer).
>>     *Jason Close*
>>     *Information Security Analyst*
>>     OU Information Technology
>>     Office: 405.325.8661 <tel:405.325.8661> Cell: 405.421.1096
>>     <tel:405.421.1096>
>>     From: Dave Crawford <dave at pingtrip.com <mailto:dave at pingtrip.com>>
>>     Date: Thursday, June 4, 2015 at 8:35 PM
>>     To: Jason Close <close at ou.edu <mailto:close at ou.edu>>
>>     Cc: "bro at bro.org <mailto:bro at bro.org>" <bro at bro.org
>>     <mailto:bro at bro.org>>
>>     Subject: Re: [Bro] Multiple masters to ease the workload
>>     Is it actually 100% RAM usage by applications? Since the manager
>>     can be performing a significant amount of disk writes the kernel
>>     will allocate ‘free’ memory as ‘cached’ to increase file
>>     performance. The cached memory is released when applications
>>     demand more memory.
>>     Below is the current memory usage on one of my mangers that is
>>     handling 25 workers and 2 proxies. At first glance it appears
>>     that all the memory has been consumed, but notice how 122G is cached.
>>                  total     used       free     shared    buffers    
>>     cached
>>     Mem:          126G     125G       384M         0B       329M    
>>       122G
>>     -/+ buffers/cache:     2.6G       123G
>>     Swap:          33G       0B        33G
>>     -Dave
>>>     On Jun 2, 2015, at 10:29 AM, Close, Jason M. <close at ou.edu
>>>     <mailto:close at ou.edu>> wrote:
>>>     Our current configuration is showing a lot of heavy use by the
>>>     master node.  We currently run around 6 worker nodes that feed
>>>     data to the master, and while the master is keeping up in terms
>>>     of CPU, it is consistently teetering on using all available RAM
>>>     we can throw at it (128GB at the moment).  There are plans in
>>>     place to increase our available bandwidth 10-fold, so the
>>>     traffic coming to Bro will ramp up as well.
>>>     We could piece apart the subnets and create multiple Bro
>>>     clusters. But it would be nice to have a single cluster, and be
>>>     able to continue to throw more workers and managers at it.  But
>>>     I have not seen any documentation about configurations using
>>>     multiple managers.  If that does exist, can someone point me in
>>>     the right direction?
>>>     And if that doesn’t exist, can I get some suggestions about
>>>     mitigations to this problem?  I know there are a lot of cool
>>>     things being done with Bro, especially using scripts and APIs
>>>     where Bro can help reduce traffic being thrown to it.  But due
>>>     to the taps we have in place, and the manpower availability,
>>>     right now, spinning up a little more hardware would be a much
>>>     easier and more economical investment of our time.
>>>     Thanks.
>>>     Jason
>>>     _______________________________________________
>>>     Bro mailing list
>>>     bro at bro-ids.org <mailto:bro at bro-ids.org>
>>>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>     _______________________________________________
>>     Bro mailing list
>>     bro at bro-ids.org <mailto:bro at bro-ids.org>
>>     http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> -- 
>> Mark Buchanan
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/8d7b6cf9/attachment-0001.html 

More information about the Bro mailing list