[Bro-Dev] Broker::publish API

[Robin Sommer]

On Fri, Jul 27, 2018 at 14:47 -0500, Jonathan Siwek wrote:

> Broker does not yet have automatic multihop where subscriptions are
> globally flooded automatically.

Yep, that's what I meant: dynamic multihop where each node tracks what
its peers are subscribing to, and forwards messages independent of its
own subscriptions.

> Possibly a downside is now you need to store original hop limit in
> addition to current TTL in each message if you want to detect the "is
> 1st hop" condition for the "relay_topic" option below.

Yeah, that's right. Actually I think ideally the 1st hop wouldn't have
any special role anyways if we didn't need that "relay_topic".

> It's maybe both a concern and a reality -- Bro clusters currently
> contain cycles (e.g. worker -> manager -> proxy -> worker)

True, although it's not cycles in the connection topology that matter,
it's cycles in topic subscriptions. I need to think about this a bit
more (and I need to remind myself how our topics currently look like)
but could we set up topics so that even in a cluster, messages don't
go into a cycle?

Is there a summary somewhere of what events & topics the cluster nodes
are currently exchanging?

> >     - Add a second function publish_pool() that has all the same
> >       options, but receives a pool type instead of a topic (just an
> >       enum: RR, HRW).
> 
> What's the goal of the enums instead of just publish_hrw() and publish_rr() ?

Similar to what Justin wrote, it would more directly express the
intent, with less emphasis on the mechanism; we could set a
default to whatever we recommend people normally use; and it'd be more
extensible.

> At the moment, one could write their own wrapper function around that
> if they find it too verbose and always want to use certain defaults?

They could, but my general point is that it'd be nice to have a simple
API that covers the most common uses cases directly and intuitively.
Then let people change defaults if they have to and know what they are
doing.

Robin



-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com