[Bro-Dev] Scaling out bro cluster communication

[Matthias Vallentin]

> What I want to make possible is client side load balancing and
> failover for worker -> manager/datanode communication.

This is an important part of future Bro deployments. 

Before delving into script code, I would like to get a better
understanding of the underlying concepts and communication patterns.
Once we have a clear picture what workloads we need to support, we can
make architectural choices. Finally, the API falls out at the end.

Concretely: can you describe (without Bro script code) what "client-side
load-balancing and failover" means? Who is the client and what state
needs to be resilient to failure? I don't think we have a working
definition of "data node" either. My hunch is that they are involved in
MapReduce computation and perhaps represent the reducers, but I'm not
sure.

    Matthias