[Bro] Bro and scan detection - the new script. And performance.
seth at icir.org
Tue Aug 25 07:01:27 PDT 2015
> On Aug 25, 2015, at 2:35 AM, Michał Purzyński <michalpurzynski1 at gmail.com> wrote:
> The old one was discouraged on large clusters, is the new one better?
> TL;DR to my surprise I have like 60Gbit of traffic here (OK, spikes
> ;-), millions of connections, insane amount of logs and I'm wondering
> if I could enable it.
It should work just fine. We spent several years figuring out how to do it and extended SumStats a lot with that aim, but I haven’t yet heard of a network where it doesn’t work (although I’m sure I will now!).
A bit more information about why it works...
It was built on top of SumStats and was even one of the driving motivations for SumStats which gives us cluster transparency. There are a couple of reasons that SumStats works in general, even in crazy cases like scan detection. It uses lazy synchronization to wait until the end of an epoch (i.e. some time interval) to collect data from all of the nodes seeing traffic (workers). This means that the cluster is not synchronizing everything all the time. It’s only synchronizing results and defined intervals which greatly reduces the communication overhead. Additionally, all of the measurements that it does are composable (we can merge results from many different systems) which enables us to cope with traffic being split across processes and even machines like all of you are doing on clusters but then bring that data back and creates the final composed result which can then be checked against thresholds or you can have your own code run on the result.
Give it a try. I think it’ll surprise you. :)
International Computer Science Institute
(Bro) because everyone has a network
More information about the Bro