[Bro-Dev] CBAN design proposal

Siwek, Jon jsiwek at illinois.edu
Thu May 26 10:41:47 PDT 2016

> On May 26, 2016, at 10:46 AM, Robin Sommer <robin at icir.org> wrote:
> - Let's keep BroControl integration in mind at leat. I agree that a
>  standalone client makes most sense right now, but if there's
>  something we can do that will make it easier for BroControl later to
>  integrate, that's worth preparing for.
> - One idea along those integration lines (BroControl and even
>  elsewhere): would be nice to structure the client so that the main
>  functionality resides in a Python module with a well-defined API.
>  The client itself would then become just a small command-line
>  frontend to that library.


> - Where's that backend-side "out-of-band process" located? Is that
>  part of the client as well? Or maybe at least part of that Python
>  module? Would be nice to keep it easy to operate for CBAN-like
>  repositories that people maintain externally.

I imagined it being part of the nightly process that does quality/metric gathering.

The code to do it could be part of the client, but I expect the aggregation process could be time consuming enough that a user wouldn’t be happy w/ it.  Especially if CBAN grows to contain many submodules, I don’t think it would scale well for each client to have to do the metadata aggregation themselves.

Is it an important use case for the client to be able to interact w/ other repos that are structured like the one the Bro Team will be hosting?  Seems less likely that people will want to do that especially if the CBAN repository is easy enough for people to submit to and the client can handle the case of adding individual external repos just fine.

The metadata from individual, externally added repos would be searchable along with the bulk/aggregated metadata.  e.g. all the submodules a user has locally installed have the latest metadata available, but all the submodules the user has not installed will fall back on the aggregated metadata that was fetched the last time they ran an “update” command.

> - having both "upgrade" vs "update" commands sounds confusing, I'm
>  sure I would constantly mix up the two. Suggest to rename one of
>  them.

I do confuse them too, but it’s also what some other package managers use (e.g. home-brew and apt-get).  I’ll look at others package managers for more examples to see if I find something.

> - What's the policy for namespaces (both script-land and plugin-land),
>  and what's the relationship to the CBAN path? Should people use
>  their GitHub name as namespace for their module? On the one hand
>  that would be an easy way to avoid conflicts. On the other hand that
>  makes forking difficult.

Didn’t really have plans/ideas for a namespace policy, but checking for conflicts could be added to the list of things the automated nightly quality/metrics thing checks for.

But maybe there’s a slicker thing you could add to the Bro language itself to let users specify a custom namespace mapping for certain file paths allowing them to resolve conflicts themselves.

> - How do updating a plugin works now? If the author just updates the
>  remote code, the submodules won't move, so is there an additional
>  step there?

The client's “update” command will do a “git pull” in the parent repo and then a “git fetch” in any installed submodules.  Any submodules that are not installed are a no-op because they are just pointers that got updated by the “git pull”.  This means the client should now be aware of updates to all the plugins they have installed, but are still using the old version until they choose to perform the “upgrade” command which then moves the installed submodules to the latest version.

Or at least it will look something like that.  I haven’t had a chance to test if there will need to be a separate “working copy” of submodules the user chooses to install.

> - Any thoughts on distributing precompiled plugins? It would be nice
>  if, say, I could just install the Mac version of plugin XYZ without
>  having to compile it locally. The authors could optionally offer
>  selected prebuilds for whatever platforms they want to support.
>  Probably more a a feature for CBAN v2, but would be wort keeping
>  mind how binaries would fit in.

Didn’t think much about it yet, but I expect a plugin to be able to define its own build steps.  For precompiled plugins they could likely just do a no-op for the build step.  Maybe there’s other install paths that need to get set up that I’m forgetting, but that shouldn't be too much extra to get working.  Let me know if you had other thoughts that would make this more complicated than I’m expecting.

> - Terminology 2: Using "plugin" as the entity name for everything is
>  confusing I think, as right now I think most people understand it as
>  something that gets compiled; nobody calls a script a plugin (unless
>  it's a plugin for a specific script-framework; but that's even more
>  confusing then :). The best alternative names I can come up with are
>  "module" or "package", which are also overloaded already, but at
>  least more generic. Other suggestions?

Don’t have other suggestions, but “package” seems fitting because the concept of “package management” is an established term that people are familiar with.

> - I'm offended that my plugin is just "nice", but Seth's is *cool*!

Maybe some people prefer “nice” to “cool”.  BTW, originally you had just “some-plugin”, but I changed it.

- Jon

More information about the bro-dev mailing list