[Bro-Dev] Scaling out bro cluster communication

Seth Hall seth at icir.org
Fri Feb 10 06:30:45 PST 2017

> On Feb 9, 2017, at 3:21 PM, Azoff, Justin S <jazoff at illinois.edu> wrote:
> What I want to make possible is client side load balancing and failover for worker -> manager/datanode communication.

Yes!  Load balancing and failover are great goals for this stuff.

> ## A super magic awesome implicit form
>    global scan_attempt: event(scanner: addr, attempt: Attempt)
>        &partition_via=func(scanner: addr, attempt: Attempt) { return scanner; } ;

I'm not sure how much I like this model, but I'd need to think about it a bit more still.  I agree that on the surface it feels magic and awesome but I'm worried we could get ourselves into situations that aren't easily resolvable with this model.

> The implicit form fits better with how bro currently works, but I think the explicit form would ultimately make cluster aware scripts simpler.

Agree on both points.

>    # define event and handle on all nodes
>    global scan_attempt: event(scanner: addr, attempt: Attempt);
>    event Scan::scan_attempt(scanner: addr, attempt: Attempt)
>        {
>        add_scan_attempt(scanner, attempt);
>        }
>    # send the event directly to the manager node
>    send_event("manager", Scan::scan_attempt(scanner, attempt));

I do like the look of making this more explicit.  The implicit event sharing behavior makes some stuff that feels like it should be easy end up being really difficult.  Do you have thoughts on how you'd do things like if you want the manager to send an event to all workers or all data nodes?

Another thing I think we need to address is that this behavior seamlessly falls back if someone isn't running a cluster.  Do you expect your idea to do that?  I know that in the current programming model, making this cluster aware but still work not on a cluster can be painful to create the right abstraction.


Seth Hall
International Computer Science Institute
(Bro) because everyone has a network

More information about the bro-dev mailing list