[Bro-Dev] Broker::publish API

[Robin Sommer]

On Mon, Jul 30, 2018 at 09:01 -0700, I wrote:

> Is there a summary somewhere of what events & topics the cluster nodes
> are currently exchanging?

So I went through the exercise of collecting this information: what
connections do we have between nodes, who's subscribing to what, and
who's publishing what; see the attached PDF. This is based on all the
standard scripts, with some special cases ignored (like the control
framework).

I'm not fully sure yet what to conclude from this, but a few quick
observations:

    - The main topics are bro/cluster/<node-type> and
      bro/cluster/node/<name>. For these we wouldn't have a problem
      with loops if we enabled automatic, topic-driven forwading as
      far as I can see.

    - bro/cluster/broadcast seems to be the main case with a looping
      problem, because everybody subscribes to it. It's hardly used
      though. (bro/config/change is used similarly though).

    - Relaying is hardly used.

    - There are a couple of script-specific topics where I'm wondering
      if these could switch to using bro/cluster/<node-type> instead
      (bro/intel/*, bro/irc/dcc_transfer_update). In other words: when
      clusterizing scripts, prefer not to introduce new topics.

    - There's a lot of checks in publishing code of the type "if I am
      (not) of node type X".

    - Pools are used for two different things: 1. the known-* scripts
      pick a proxy to process and log the information; whereas 2. the
      Intel scripts pick a proxy just as a relay to broadcast stuff
      out, reducing load. That 1st application is a good, but the 2nd
      feels like should be handled differently.

Need to mull over this more, thoughts welcome.

Overall I have to say I found it pretty hard to follow this all
because we don't have much consistency right now in how scripts
structure their communication. That's not surprising, given that we're
just starting to use all this, but it suggests that we have room for
improvement in our abstractions. :)

Robin

-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Broker Communication.pdf
Type: application/pdf
Size: 32669 bytes
Desc: not available
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180806/5a0c99fa/attachment-0001.pdf