[Bro] Multiple masters to ease the workload

[Close, Jason M.]

Thanks.

I went ahead and rebooted the cluster, and that cleared things up (as well as sent out a LOT of emails…).

Has anyone else noticed a memory leak in the sensors?  We slowly see memory usage grow, maybe by about 10GB a month, even when our total traffic has gone down.  I attached an image from our Zabbix monitor.  You can see that once we reboot the box, memory drops down, and then slowly creeps up.  And traffic isn’t increasing (in fact, it decreases by half over the summer).

Jason Close
Information Security Analyst
OU Information Technology
Office: 405.325.8661  Cell: 405.421.1096


From: Dave Crawford <dave at pingtrip.com<mailto:dave at pingtrip.com>>
Date: Thursday, June 4, 2015 at 8:35 PM
To: Jason Close <close at ou.edu<mailto:close at ou.edu>>
Cc: "bro at bro.org<mailto:bro at bro.org>" <bro at bro.org<mailto:bro at bro.org>>
Subject: Re: [Bro] Multiple masters to ease the workload

Is it actually 100% RAM usage by applications? Since the manager can be performing a significant amount of disk writes the kernel will allocate ‘free’ memory as ‘cached’ to increase file performance. The cached memory is released when applications demand more memory.

Below is the current memory usage on one of my mangers that is handling 25 workers and 2 proxies. At first glance it appears that all the memory has been consumed, but notice how 122G is cached.

             total       used       free     shared    buffers     cached
Mem:          126G       125G       384M         0B       329M       122G
-/+ buffers/cache:       2.6G       123G
Swap:          33G         0B        33G


-Dave

On Jun 2, 2015, at 10:29 AM, Close, Jason M. <close at ou.edu<mailto:close at ou.edu>> wrote:

Our current configuration is showing a lot of heavy use by the master node.  We currently run around 6 worker nodes that feed data to the master, and while the master is keeping up in terms of CPU, it is consistently teetering on using all available RAM we can throw at it (128GB at the moment).  There are plans in place to increase our available bandwidth 10-fold, so the traffic coming to Bro will ramp up as well.

We could piece apart the subnets and create multiple Bro clusters.  But it would be nice to have a single cluster, and be able to continue to throw more workers and managers at it.  But I have not seen any documentation about configurations using multiple managers.  If that does exist, can someone point me in the right direction?

And if that doesn’t exist, can I get some suggestions about mitigations to this problem?  I know there are a lot of cool things being done with Bro, especially using scripts and APIs where Bro can help reduce traffic being thrown to it.  But due to the taps we have in place, and the manpower availability, right now, spinning up a little more hardware would be a much easier and more economical investment of our time.

Thanks.

Jason
_______________________________________________
Bro mailing list
bro at bro-ids.org<mailto:bro at bro-ids.org>
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/8d1855ea/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sensor2.png
Type: image/png
Size: 22950 bytes
Desc: sensor2.png
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150605/8d1855ea/attachment-0001.bin