[Bro] Multiple masters to ease the workload
Gary Faulkner
gfaulkner.nsm at gmail.com
Fri Jun 5 11:21:04 PDT 2015
I used to see some stability issues on a reasonably large network (100K
plus hosts) that appeared related to software asset tracking, coupled
with a lot of IP churn due to wireless devices and short lease times,
which resulted in the manager being oom-killed after some number of
hours or days. It was suggested, I think by Seth, that I experiment with
disabling it in local.bro:
I'm still on 2.3-419, so not the latest build by any means, but was able
to test this, and have seen greatly improved stability in my particular
environment, with the below line in local.bro:
redef Software::asset_tracking=NO_HOSTS;
It might be worth experimenting with if you think you may be having
software asset tracking related issues. Be mindful if you rely on that
data for other scripts.
Regards,
Gary
On 6/5/2015 12:40 PM, Dave Crawford wrote:
>
> When I recently debugged memory exhaustion on my workers the root
> cause was related to the software detection scripts, specifically
> /protocols/http/software. If that script is running on a sensor that
> is monitoring the inside interface of a web proxy it tracks all the
> remote software as being on your local network.
>
>
> On Jun 5, 2015, at 12:09 PM, Mark Buchanan <mabuchan at gmail.com
> <mailto:mabuchan at gmail.com>> wrote:
>
>> I too have noticed memory complete memory exhaustion in Bro 2.3.2
>> (not sure what version Jason is running). If the workers are not
>> restarted every few days or at least once a week, I run out of usable
>> memory on a few sensors I'm testing.
>> I have found that just doing a restart within broctl will free the
>> memory consumed up - but I regularly have to perform restarts to keep
>> the sensors I am testing running smoothly.
>> Mark
>>
>> On Fri, Jun 5, 2015 at 7:08 AM, Close, Jason M. <close at ou.edu
>> <mailto:close at ou.edu>> wrote:
>>
>> Thanks.
>>
>> I went ahead and rebooted the cluster, and that cleared things up
>> (as well as sent out a LOT of emails…).
>>
>> Has anyone else noticed a memory leak in the sensors? We slowly
>> see memory usage grow, maybe by about 10GB a month, even when our
>> total traffic has gone down. I attached an image from our Zabbix
>> monitor. You can see that once we reboot the box, memory drops
>> down, and then slowly creeps up. And traffic isn’t increasing
>> (in fact, it decreases by half over the summer).
>>
>> *Jason Close*
>> *Information Security Analyst*
>> OU Information Technology
>> Office: 405.325.8661 <tel:405.325.8661> Cell: 405.421.1096
>> <tel:405.421.1096>
>>
>>
>> From: Dave Crawford <dave at pingtrip.com <mailto:dave at pingtrip.com>>
>> Date: Thursday, June 4, 2015 at 8:35 PM
>> To: Jason Close <close at ou.edu <mailto:close at ou.edu>>
>> Cc: "bro at bro.org <mailto:bro at bro.org>" <bro at bro.org
>> <mailto:bro at bro.org>>
>> Subject: Re: [Bro] Multiple masters to ease the workload
>>
>> Is it actually 100% RAM usage by applications? Since the manager
>> can be performing a significant amount of disk writes the kernel
>> will allocate ‘free’ memory as ‘cached’ to increase file
>> performance. The cached memory is released when applications
>> demand more memory.
>>
>> Below is the current memory usage on one of my mangers that is
>> handling 25 workers and 2 proxies. At first glance it appears
>> that all the memory has been consumed, but notice how 122G is cached.
>>
>> total used free shared buffers
>> cached
>> Mem: 126G 125G 384M 0B 329M
>> 122G
>> -/+ buffers/cache: 2.6G 123G
>> Swap: 33G 0B 33G
>>
>>
>> -Dave
>>
>>> On Jun 2, 2015, at 10:29 AM, Close, Jason M. <close at ou.edu
>>> <mailto:close at ou.edu>> wrote:
>>>
>>> Our current configuration is showing a lot of heavy use by the
>>> master node. We currently run around 6 worker nodes that feed
>>> data to the master, and while the master is keeping up in terms
>>> of CPU, it is consistently teetering on using all available RAM
>>> we can throw at it (128GB at the moment). There are plans in
>>> place to increase our available bandwidth 10-fold, so the
>>> traffic coming to Bro will ramp up as well.
>>>
>>> We could piece apart the subnets and create multiple Bro
>>> clusters. But it would be nice to have a single cluster, and be
>>> able to continue to throw more workers and managers at it. But
>>> I have not seen any documentation about configurations using
>>> multiple managers. If that does exist, can someone point me in
>>> the right direction?
>>>
>>> And if that doesn’t exist, can I get some suggestions about
>>> mitigations to this problem? I know there are a lot of cool
>>> things being done with Bro, especially using scripts and APIs
>>> where Bro can help reduce traffic being thrown to it. But due
>>> to the taps we have in place, and the manpower availability,
>>> right now, spinning up a little more hardware would be a much
>>> easier and more economical investment of our time.
>>>
>>> Thanks.
>>>
>>> Jason
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>>
>>
>> --
>> Mark Buchanan
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/8d7b6cf9/attachment-0001.html
More information about the Bro
mailing list