[Bro-Dev] Broker::publish API

[Jan Grashöfer]

On 08/08/18 17:48, Robin Sommer wrote:> I think it's safe to assume we 
have the cluster structure under our
> own control; it's whatever we configure it to be. That's something
> that's easier to change later than the API itself. Said differently:
> we can always adjust the connections and topics that we set up by
> default; it's much harder to change how the publish() function works.

I think in an earlier discussion (could be 
http://mailman.icsi.berkeley.edu/pipermail/bro-dev/2017-February/012386.html) 
there was the idea of different types of data nodes that would serve 
different purposes. If that is still a design goal, it feels like the 
structure of a cluster could be more volatile than it used to be. Not 
sure how that fits to the current assumptions. Just wanted to bring that 
back into the discussion.

> Let me try to phrase it differently: If there's already a topic for a
> use case, it's better to use it. That's easier and less error-prone.
> So if, e.g., I want to send my script's data to all workers,
> publishing to bro/cluster/worker will do the job. And that will even
> automatically adapt if things get more complex later.

Maybe a silly question: Would that work using further "specialized" 
topics like bro/cluster/worker/intel? From my understanding one feature 
of topics is that one would be able to subscribe only the the things 
that one is interested in. Having a bunch of events just published to 
bro/cluster/worker seems counterproductive.

> Maybe it's a *necessary* design, but that doesn't make it nice. ;-) It
> makes it very hard to follow the logic; when reading through the
> scripts I got lost multiple times because some "@if I-am-a-manager"
> was somewhere half a page earlier, disabling the code I was currently
> looking at for most nodes. We probably can't totally avoid that, but
> the less the better.

I agree! One thing that could also help here is clear separation. In the 
intel framework that kind of code is capsuled in a cluster.bro file, 
which is basically divided into a worker and a manager part. In the end 
it's a tradeoff between abstraction and flexibility.

Jan