[Bro-Dev] design summary: porting Bro scripts to use Broker

[Siwek, Jon]

> I got send_event_hashed to work via a bit of a hack (https://github.com/JustinAzoff/broker_distributed_events/blob/master/distributed_broker.bro),
> but it needs support from inside broker or at least the bro/broker integration to work properly in the case of node failure.
> 
> My ultimate vision is a cluster with 2+ physical datanode/manager/logger boxes where one box can fail and the cluster will continue to function perfectly.
> The only thing this requires is a send_event_hashed function that does consistent ring hashing and is aware of node failure.

Yeah, that sounds like a good idea that I can try to work into the design.  What is a “data node” though?  We don’t currently have that?

More broadly, it sounds like a user needs a way to specify which nodes they want to belong to a worker pool, do you still imagine that is done like you had in the example broctl.cfg from the earlier thread?  Do you need to be able to specify more than one type of pool?

> For things that don't need necessarily need consistent partitioning - like maybe logs if you were using Kafka, a way to designate that a topic should be distributed round-robin between subscribers would be useful too.

Yeah, that seems like it would require pretty much the same set of functionality to get working and then user can just specify a different function to use for distributing events (e.g. hash vs. round-robin).

- Jon