[Bro-Dev] [Bro-Commits] [git/broker] topic/actor-system: Add broker::bro::RelayEvent message type. (ce86016)

[Jon Siwek]

On Fri, Dec 15, 2017 at 10:29 AM, Robin Sommer <robin at icir.org> wrote:

> Yeah, though then I'd actually say it's worth keeping multi-hop
> topologies in mind at least. We don't need to fully solve this right
> now; nobody really knows yet how topologies may end up looking in the
> future. But a good rule of thumb I think is considering that generally
> nodes may not be reachable directly, but only through other Broker
> hops, and that Broker takes care of the routing to get messages there
> as long as topic subscriptions are set up appropiately. Seems that's
> actually what we have here already for some nodes. Could we use
> Broker-level routing here to get connectivity between the nodes that
> aren't directly connected?

For the case of worker to all other workers via a single proxy, I'm
not sure it's simple to do with implicit routing.  You would need a
topic subscription in common between the proxy and workers and you
also want that topic to be unique among all proxies, otherwise you'd
generate many duplicate messages as you send to all proxies and then
all proxies send to all workers.  I think you'd end up with a long
subscription lists that look like:

proxy-1: relay_to_workers_via_proxy_1
proxy-2: relay_to_workers_via_proxy_2
...
worker-1: relay_to_workers_via_proxy_1, relay_to_workers_via_proxy_2, ...
worker-2: same as worker-1

And that works and could be automatically generated, though I think
it's conflicting w/ how I intuitively want to try and categorize
topics/subscriptions: what I end up looking for is to have topics
identify either a single node or a whole class of nodes, so instead of
encoding the intent of the message pattern in the topic names, I want
function calls that define the semantics of the messaging pattern and
supply to it the identities/classes of which nodes should be involved
in that pattern.

Another thing I like more about an explicit relay function is that
it's clear that I don't want the relaying node(s) to actually do any
any event handling, I just want them to forward it on, but if I did
want them to handle it, then I still have the option of sending to
them directly using the publish() function.  With the internal routing
mechanism, the sender doesn't have as good control/flexibility of
which nodes on the route will end up handling events, at least not
without doing more planning/configuration in advance to set up those
separate avenues.

>> At least currently, Bro has the forwarding capability of Broker turned
>> off.  IIRC, it was very easy to unintentionally create routing loops
>> when it was turned on and when I asked about it, no one gave any
>> justification or use-case for it, thus it got turned off.
>
> It's off because we indeed don't really have understood yet how to use
> it. :) But Broker has message TTLs already for detecting loops, and
> generally I don't think there's anything preventing us from turning it
> on; it should work.

I meant that I was the one who turned it off when I started porting
scripts because the first thing I tried to do ended up creating a
loop.  Yeah, the 20-hop TTL eventually killed things eventually,
though it wasn't a good first impression :)

And I think that particular loop would still be there if it got turned
on right now...  also not certain, but that particular one might not
be so harmless to leave routing around, it might actually have side
effects with how cluster nodes track each other's identities when they
join/leave the cluster and completely break it.

> That all said, the priority right now is on getting a basic cluster to
> work, so no problem doing the relaying as you have it now if that gets
> us there the quickest. Let's just keep thinking about how such
> mechanisms will look in the future in other topologies, so that we
> don't back us into a corner (and I hear your of course: to really do
> that, we need to understand better where we want to get to).

Yeah, agree that it should become easier to think about how the
extended cluster/routing stuff should come into play after the first
step of porting to broker is done and is available to iterate upon
instead of trying to do both steps in one go.

- Jon