[Bro-Dev] scheduling events vs using &expire_func ?

[Azoff, Justin S]

> On Apr 19, 2018, at 7:03 AM, Aashish Sharma <asharma at lbl.gov> wrote:
> 
> aggregating across all proxies is still distributing data around. So the way I
> see is you are moving the problem around :) But as I said, I don't know more how
> this works since I haven't tried new broker stuff just yet. 

I think the part that you are missing is that the data is PARTITIONED across the proxies.

So say you saw a few scan attempts:

attacker.1 -> local.1:22
attacker.2 -> local.1:22
attacker.1 -> local.2:22
attacker.3 -> local.1:80
attacker.4 -> local.1:23
attacker.2 -> local.3:80
attacker.1 -> local.3:22

To do scan detection they need to be grouped into a table like this:

attacker.1 -> [local.1:22, local.2:22, local.3:22]
attacker.2 -> [local.1:22, local.3:80]
attacker.3 -> [local.1:80]
attacker.4 -> [local1:23]

This is how scan-ng, simple-scan work now.
Currently this needs to be aggregated in a single place (the manager) but with publish_hrw, bro
will do something like this:

hash(attacker.1) == proxy-1
hash(attacker.2) == proxy-2
hash(attacker.3) == proxy-2
hash(attacker.4) == proxy-1

Which will distribute the data across the proxies and you will end up with:

Data stored on proxy-1:
attacker.1 -> [local.1:22, local.2:22, local.3:22]
attacker.4 -> [local1:23]

Data stored on proxy-2:
attacker.2 -> [local.1:22, local.3:80]
attacker.3 -> [local.1:80]

Each proxy will have a consistent view of a single scanner but not need to store all data
for all scanners.

— 
Justin Azoff