[Bro] High-CPU on just a single worker in the cluster
mabuchan at gmail.com
Thu Apr 14 07:33:52 PDT 2016
I'm not 100% familiar with the load balancing methods, but know if your doing a 5 tuple load balance algorithm, I have seen IPsec stick to a single location, as there are effectively only three tuple available (Src IP, Dst IP and proto), which don't change for most static IPSec connections. Which results in the traffic not changing and sliding from worker to worker like normal connections would.
Could there be a high use IPSec tunnel between that Datacenter and another location?
> On Apr 14, 2016, at 08:58, Seth Hall <seth at icir.org> wrote:
>> On Apr 14, 2016, at 8:26 AM, Dave Crawford <bro at pingtrip.com> wrote:
>> Below is a fresh 'netstats' from this morning (WIN_INT-9 is obviously the culprit).
> You are seeing the side effects of hot spotting. It's likely that you have a connection which is massively larger than all others that your host is seeing. You could filter your traffic to stop seeing that flow which would cut down on the problems you are seeing.
> I just pushed a new repository that might help you address your issue. It will still require a lot of clean up and documentation but if you just load the module as-is, it might automatically help you. :) It automatically identifies large and fast flows and places BPF filters to automatically shunt them while still watching for SYN/ACK/FIN/RST flags.
> There are a couple of configuration options you'll want to pay attention to, so I'll just give you an example blurb you could put into your local.bro:
> @load bro-bulkflows
> # Set a minimum flow size before considering shunting (32MB)
> redef BulkFlows::size_threshold = 33554432;
> # Set a minimum flow data rate before considering shunting (50Mbps)
> BulkFlows::speed_threshold = 6250000;
> # If you have VLAN tags on your traffic, you will also have to add the following line because packet filtering with BPF is generally broken otherwise...
> #redef PacketFilter::restricted_filter="vlan";
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> Bro mailing list
> bro at bro-ids.org
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 2182 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160414/9f885ef5/attachment.bin
More information about the Bro