[Bro-Dev] Broker::publish API

[Jon Siwek]

On Fri, Jul 27, 2018 at 12:40 PM Robin Sommer <robin at corelight.com> wrote:

> I'm wondering if we should give it another try to simply this API
> while we still can (i.e., before 2.6 goes out). To me, the most
> intuitive publish operation is "send to topic T and propagate to
> everybody subscribed to that topic". I'd structure the API around
> that, making that the main publish function for that simply:
>
>     Broker::publish(topic, args);
>
> That would send to all neighbors, which then process locally and relay
> to their neighbors. Right now, that would propagate just across one
> hop but once we have multihop that'd start being broadcasted out
> broadly.

Can you remind/clarify what's meant by "multihop" ?  I thought:

Broker already has manual multihop if you set up subscriptions on all
relevant nodes on the path yourself.  Bro doesn't use it right now.

Broker does not yet have automatic multihop where subscriptions are
globally flooded automatically.

A difference between "manual multihop" and "automatic multihop" would
be that in the later, some relaying nodes may not actually hold a
subscription to the message they are relaying and so, in the case of
Bro events, I think they would not process them locally.

>     - Give publish() another argument "relay: bool &default=T" to prevent
>       it from going beyond the immediate receiver. Or maybe instead:
>       "relay_hops: int &default=-1" to specify the max number of hops
>       to relay across, with -1 meaning no limit.

Going with the generalized approach of configurable number of hops per
message via "relay_hops" from the start would be better than finding
out we need it later.

Possibly a downside is now you need to store original hop limit in
addition to current TTL in each message if you want to detect the "is
1st hop" condition for the "relay_topic" option below.

> .     (I recall concerns
>       about loops being too easy to create; we could set the default
>       here to F/0 to default to no forwarding, although conceptually I
>       don't really like that :-)

It's maybe both a concern and a reality -- Bro clusters currently
contain cycles (e.g. worker -> manager -> proxy -> worker)

>     - Give publish() another argument "relay_topic: string &default=""
>       to change the topic when relaying on the 1st hop.
>
>     - Give publish() another argument "process_on_relays: bool &default=T"
>       to change whether a relaying hop also sees the event locally.

Those seem fine to me.

>     - Add a second function publish_pool() that has all the same
>       options, but receives a pool type instead of a topic (just an
>       enum: RR, HRW).

What's the goal of the enums instead of just publish_hrw() and publish_rr() ?

Instead of the API being 2 functions, it then seems like 2 enums that
are never used elsewhere + 1 function that now always branches
internally.

- Jon