[Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

[Siwek, Jon]

> On Nov 2, 2017, at 1:37 PM, Aashish Sharma <asharma at lbl.gov> wrote:
> 
>>> In the case of broadcasting from a worker to all other workers, the reason why you relay via another node is only because workers are not connected to each other?  Do we know that a fully-connected cluster is a bad idea?  i.e. why not have a worker able to broadcast directly to all other workers if that’s what is needed?
>> 
>> Mostly so that workers don't end up spending all their time sending out messages when they should be analyzing packets.
> 
> Yes, Also, I have seen this can case broadcast stroms. Thats why I have always
> used manager as a central judge on what goes. See, often same data is seen by
> all workers. so if manager is smart, it can just send first instance to workers
> and all other workers can stop announcing further. 
> 
> Let me explain: 
> 
> - I block a scanner on 3 connections. 
> - 3 workers see a connection each - they each report to manager 
> - manager says "yep scanner" sends note to all workers saying traffic from this
>  IP is now uninteresting stop reporting. 
> - lets say 50 workers
> - total commnication events = 53 
> 
> If all workers send data to all workers a scanner hitting 65,000 hosts will be a
> mess inside cluster. esp when scanners are hitting in ms and not seconds. 

Thanks, though I’m not sure this scenario maps well to this particular point.  E.g. my impression is Justin wants a single BIF/function that can send one event from a worker to a proxy and have the proxy purely relay it to all other workers without doing anything else.  So it’s solely taking the cost of sending N messages from a worker and offloading that burden to a different node.

I think your example differs because there is actually an additional task/logic being performed on the middleman that ends up reducing comm/processing requirements.  i.e. it’s not just a pure relay of messages.

Or maybe I’m just not understanding what anybody wants :)

- Jon