[Bro-Dev] Broker::publish API

Jon Siwek jsiwek at corelight.com
Mon Aug 13 11:55:31 PDT 2018


On Mon, Aug 13, 2018 at 12:56 PM Azoff, Justin S <jazoff at illinois.edu> wrote:

> >  Broker::publish(Cluster::worker_topic, my_event);

> The one tricky case is doing that on the manager.  While the manager is fully connected to all workers,
> you really want to offload the fanning out of messages to one of the proxies.

Yeah, I don't know exactly how it would be implemented yet, but seems
to warrant a policy/flag that one sets on the manager that means
"prefer sending along a 2-hop route rather than a 1-hop route if it
minimizes our own workload" or else a way to mark proxy nodes such
that any connected peers always prefer to send 1-routable-message to
it rather than N-direct-messages.  Maybe falls under "load-balancing"
of the prospective routing implementation, which I've tracked as
requiring these features:

* cycle detection/prevention
* network-wide subscription knowledge per-node
* load-balancing + proxying policies

Let me know if I missed any.  I have implementation ideas/notes
already which basically requires associating node IDs with
subscription state and also message state (push node IDs into messages
upon receipt before forwarding), but we can maybe discuss and flesh it
out in a later design thread once we decide what exactly to do.  As
for deciding what to do in the near term, seems like we will arrive at
agreeing upon:

(1) Remove relay(...) functions
(2) Reduce unique topic names (use pre-existing cluster topics where possible)
(3) Add Broker::forward(topic_prefix) function + enable Broker forwarding

An alternative to (3) would be implementing "real" routing in Broker
right from the start.  No strong opinion there, but seems like it
could fall under nice-to-have at this point and, while it would
obsolete Broker::forward(), I don't expect that's much effort wasted.
Any other ideas?

- Jon


More information about the bro-dev mailing list