[Bro-Dev] Broker::publish API

[Jon Siwek]

On Wed, Aug 8, 2018 at 10:53 AM Robin Sommer <robin at corelight.com> wrote:
>
> Yeah, I realize that. A direct port of the old logic was of course the
> goal so far, with the drawbacks of that approach accepted &
> understood. That's what's in place now; that's great and exactly as
> planned. We can get 2.6 out this way, and it'll be fine.

I'm earnestly probing to try to get a better decomposition of the
issues that make it hard to understand cluster communication patterns.

There's the exercise of trying to answer "what *is* this script
doing?" and then there's also trying to answer "what *should* it be
doing?".

I seldom felt like I had definitive answers for the later, but I can
see how it would be beneficial to do that and also broader
script/framework makeovers, possibly before 2.6, because it would help
inform whether new APIs are catering to "good" use-cases.  Though my
thinking is it's not critical to get a 100% API/use-case match off the
bat and that there's some actionable stuff to take away from this
thread that is at least going to have us heading in a better direction
sooner rather than later...

> My point is that now also seems like a good time to take stock of what
> we got that way. That direct porting is finally getting us some sense
> of where things aren't an ideal match between API and use cases yet.
> And if there's something easy we can do about that before people start
> relying on the new API, it seems that would be beneficial to do. But
> we can see.

Yeah, agreed.  What I've taken away from your earlier points is that
these smaller changes are seeming like they'd be beneficial to do
before 2.6:

* publish() API simplifications/compressions (pending decision on
exactly what those should be)
* enable message forwarding by default (meaning re-implement the one
or two subscription patterns that might create a cycle)
* see if any script-specific topics can instead use a pre-existing
"cluster" topic

What do you think?

A separate question/idea I just had was whether how much of the
process of auditing the subscriptions and communication patterns was
difficult due to having to hunt down things in various scripts and
whether a more centralized config could be something to do?  e.g. I
don't know how the details would work out, but I'm imagining a
workflow where one edits a centralized config file with
subscription/node info in it and that auto-generates the code to set
them up.  Sort of like working backward from the info in the PDF you
shared.

- Jon