[Bro-Dev] Broker::publish API

[Robin Sommer]

On Tue, Aug 07, 2018 at 12:05 +0200, Jan Grashöfer wrote:

> What I can recall, it's about simplifying the API in the light of
> multi-hop routing, which is not fully functional yet.

To level up a bit, what I'm hoping for is that we can find some easy
ways to simplify the API a bit more now, with an eye towards dynamic
multi-hop coming later. I don't know if it'll work out before 2.6
still, but changing the API later is more painful.

We don't need to (or even can) solve multi-hop topologies right now, I
think nobody really has the use cases clear in their heads yet. But if
we could simplify the API a bit more for our current use cases in a
way that may extend to multihop naturally later, that would probably
save us some headaches at that point.

> How does forwarding work if I add another node type?

That's actually something I realized yesterday: we don't have direct
worker-to-worker communication right now, correct? A worker cannot
just publish to "bro/cluster/workers".

> Do we assume a certain cluster structure here? If yes: Is that a valid
> assumption?

I think it's safe to assume we have the cluster structure under our
own control; it's whatever we configure it to be. That's something
that's easier to change later than the API itself. Said differently:
we can always adjust the connections and topics that we set up by
default; it's much harder to change how the publish() function works.
 
> From my understanding this would mean going back to the old 
> communication patterns. What's the point of having topics if we don't 
> use them?

Let me try to phrase it differently: If there's already a topic for a
use case, it's better to use it. That's easier and less error-prone.
So if, e.g., I want to send my script's data to all workers,
publishing to bro/cluster/worker will do the job. And that will even
automatically adapt if things get more complex later. For example, I
can see having multiple otherwise independent cluster sharing a
communication channel. In that case, we could internally change the
topic to "bro/cluster/<cluster-id>/workers", and everybody using the
predefined worker topic would still reach "their" workers without any
further changes.

> That's something I would have expected. I don't think this is 
> necessarily an indicator of bad design.

Maybe it's a *necessary* design, but that doesn't make it nice. ;-) It
makes it very hard to follow the logic; when reading through the
scripts I got lost multiple times because some "@if I-am-a-manager"
was somewhere half a page earlier, disabling the code I was currently
looking at for most nodes. We probably can't totally avoid that, but
the less the better.

Robin

-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com