[Bro-Dev] Broker::publish API

Jon Siwek jsiwek at corelight.com
Thu Aug 9 08:02:02 PDT 2018


On Wed, Aug 8, 2018 at 2:50 PM Robin Sommer <robin at corelight.com> wrote:

> > * enable message forwarding by default (meaning re-implement the one
> > or two subscription patterns that might create a cycle)
>
> Haven't quite made up my mind on this one. In principlel yes, but
> right now a host needs to be subscribed to a topic to forward it if I
> remember than right. That may limit how we use topics, not sure (e.g.,
> if a worker wanted to talk to other workers, with "real"
> forwarding/routing they'd just publish to the worker topic and that
> message would get routed there, but not be processed at the
> intermediary hops as well. With our current forwarding, the hops would
> need to subscribe to the worker topic as well and hence the event got
> raised there, too.)

Yeah, that's how I also understand the current mechanisms would work.

Maybe can split it into two separate questions:

(1) enable the "explicit/manual" forwarding by default?
(2) re-implement any existing subscription cycles?

Answer to (2) may pragmatically be "yes" because they'd be known to
cause problems if ever (1) did become enabled (and also could be
problematic for a more sophisticated/automatic/implicit routing system
should that become available in the future... at least I think it's a
problem, but then again maybe connection-cycles would also still be a
problem at that point, not quite sure).

Answer to (1) may be "no" because we don't have a use for it at the
moment -- having the forwarding-nodes also raise events is not ideal,
but if we solved that would it be useful?  Maybe an idea would be
extend the subscribe() API in Bro:

    function Broker::subscribe(topic_prefix: string, forward_only:
bool &default=F);

I recall that we have access to both the message/event as well as the
topic string on the receiver side, so could be possible to detect
whether or not to raise the event depending on whether the topic only
has a matching subscription prefix that is marked as forward_only.

With that you could do something like:

    # On Manager
    Broker::subscribe(worker_to_worker_topic, T);

    # On Worker
    Broker::subscribe(worker_to_worker_topic);
    Broker::publish(worker_to_worker_topic, my_event);

There, my_event would be distributed from one worker to all workers
via the manager, but not sure that's as usable/dynamic as the current
"relay" mechanism because you also get a load-balancing scheme to go
along with it.  Here, you'd only ever want to pick a single manager or
proxy to do the forwarding (subscribing like this on all proxies
causes all proxies to forward to all workers resulting in undesired
event duplication.)

So I guess that's still to say I'm not sure what the use of the
current forwarding mechanism would be if it were enabled.  Also maybe
begs the question for later regarding the "real" routing mechanism: I
suppose that would need to be smart enough to do automatic
load-balancing in the case of there being more than one route to a
subscriber.

- Jon


More information about the bro-dev mailing list