[Bro] Bro and scan detection - the new script. And performance.
hhoffman at ip-solutions.net
Tue Aug 25 19:16:40 PDT 2015
Are you looking for examples where SumStats doesn't work from the latest
pull of Bro or a specific version? It's been my experience that it
depends heavily on the amount of networks configured in local nets.
On 8/25/15 10:01 AM, Seth Hall wrote:
>> On Aug 25, 2015, at 2:35 AM, Michał Purzyński <michalpurzynski1 at gmail.com> wrote:
>> The old one was discouraged on large clusters, is the new one better?
> Yes, vastly.
>> TL;DR to my surprise I have like 60Gbit of traffic here (OK, spikes
>> ;-), millions of connections, insane amount of logs and I'm wondering
>> if I could enable it.
> It should work just fine. We spent several years figuring out how to do it and extended SumStats a lot with that aim, but I haven’t yet heard of a network where it doesn’t work (although I’m sure I will now!).
> A bit more information about why it works...
> It was built on top of SumStats and was even one of the driving motivations for SumStats which gives us cluster transparency. There are a couple of reasons that SumStats works in general, even in crazy cases like scan detection. It uses lazy synchronization to wait until the end of an epoch (i.e. some time interval) to collect data from all of the nodes seeing traffic (workers). This means that the cluster is not synchronizing everything all the time. It’s only synchronizing results and defined intervals which greatly reduces the communication overhead. Additionally, all of the measurements that it does are composable (we can merge results from many different systems) which enables us to cope with traffic being split across processes and even machines like all of you are doing on clusters but then bring that data back and creates the final composed result which can then be checked against thresholds or you can have your own code run on the result.
> Give it a try. I think it’ll surprise you. :)
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> Bro mailing list
> bro at bro-ids.org
More information about the Bro