[Bro-Dev] design summary: porting Bro scripts to use Broker

[Siwek, Jon]

> On Oct 6, 2017, at 5:58 PM, Robin Sommer <robin at icir.org> wrote:
> 
> In the most simple version of this, the cluster framework would just
> hard-code a subscription to "bro/cluster/". And then scripts like the
> Intel framework would just publish all their events to "bro/cluster/"
> directly through Broker.
> 
> To allow for distinguishing by node type we can define separate topic
> hierarchies: "bro/cluster/{manager,worker,logger}/". Each node
> subscribes to the hierarchy corresponding to its type, and each script
> publishes according to where it wants to send events to (again
> directly using the Broker API).

Yeah, that could be a better way to approach it, thanks.  I’ll try to go back and rework the design around that topic hierarchy/naming convention (that was the part I was most unsure about).

> 
>> Broker Framework API
>> --------------------
> 
> I'm wondering if these store operations should become part of the
> Cluster framework instead. If we added them to the Broker framework,
> we'd have two separate store APIs there: one low-level version mapping
> directly to the C++ Broker API, and one higher-level that configures
> things like location of the DB files. That could be confusing.

Yeah could be. I’ll try moving more stuff into Cluster and see if it still makes sense to me.

> I like this. One additional idea: while I see that it's generally the
> user who wants to configure which backend to use, the script author
> may know already if it's data that should be persistent across
> execution; I'm guessing that's usually implied by the script's
> semantics. We could give InitStore() an additional boolean
> "persistent" to indicate that.

Ack.

>> # User needs to be able to choose data store backends and which cluster node the
>> # the master store lives on.  They can either do this manually, or BroControl
>> # will autogenerate the following in cluster-layout.bro:
> 
> I don't really like the idea of autogenerating this, as it's pretty
> complex information. Usually, the Broker::default_* values should be
> fine, right? For the few cases where one wants to tweak that on a
> per-store bassis, using a manual redef on the table sounds fine to me.

It’s just a matter of where you expect most users to feel comfortable making customizations: in Bro scripts or in a broctl config file.

I think it’s fine to first assume it won’t be needed often and so only provide the customization via Bro scripts directly.  If we learn later that it’s a pain point for users, it’s easy add the "simpler" config file interface via broctl to help autogenerate it.

> Hmm, actually, what would you think about using functions instead of
> tables? We could model this similar to how the logging framework does
> filters: there's a default filter installed, but you can retrieve and
> update it. Here there'd be a default StoreInfo, which one can update.

I think I went with the ‘redef’ interface first because it’s impossible for a user to screw up order of operations there, where with functions you can (technically) have some &priority mishaps on bro_init() since the InitStore() function is also going to be running in bro_init().

Maybe the key point is that these customizations only make sense to happen once before init time?  i.e. a function would imply calling it anytime at runtime could yield a useful result, but at the moment, we’re not allowing changing a store’s backend or master node dynamically at runtime, just once before bro_init().  So if you think that’s something to anticipate in the future, I’d agree that just using functions from the start would be better.

>> redef Broker::default_store_dir = "/home/jon/stores";
> 
> Can the default_store_dir be set to some standard location through
> BroControl? Would be neat if this all just worked in the standard case
> without any custom configuration at all.

Yeah, should be possible, I think I had just given a random example in the above.

- Jon