[Bro-Dev] Proposal: Operating the Bro package manager offline

Azoff, Justin S jazoff at illinois.edu
Mon Sep 19 21:11:39 PDT 2016


Sounds great.. What you are describing is basically source and binary packages.

The only thing I would do differently is that you wouldn't want bundles (at least not as the only feature) but individual source and binary packages.

For example

bro-pkg install sethhall/domain-tld

would still install the package from git like normal, but one could do

bro-pkg dist sethhall/domain-tld

or just

bro-pkg dist  # inside the project

and this would generate a versioned sethhall_domain-tld-1.0.0.tar.gz source package

Then, that file could be copied over to the target machine and

bro-pkg install sethhall_domain-tld-1.0.0.tar.gz

or variations of

bro-pkg install https://packages.internal/bro/sethhall_domain-tld-1.0.0.tar.gz

would install it.



The hard part is how to handle compiled architecture specific packages. For something like a myricom plugin,

bro-pkg dist bro-plugins/myricom

would generate a bro-plugins_myricom-1.0.0.tar.gz source package as before

If copied over to the target system

bro-pkg install bro-plugins_myricom-1.0.0.tar.gz

would build and install it as long as build tools were installed.  If 'no build tools on servers' is a requirement, then

bro-pkg build bro-plugins/myricom 

on a build server would compile and generate something like bro-plugins_myricom-1.0.0-bro2.5-linux-amd64.tar.gz

And with some more glue, could easily build .deb and .rpm packages.

This is basically how python/pip/wheel works now (after ~10 years of being pretty bad).

You could still have a bundle command as a very small layer on top of packages, but the building block should be an individual package distribution.


Also,

> That command will completely replace the system’s current set of packages with the bundle’s content

should probably not be the only behavior.  Installing should be the default unless you do something like 

bro-pkg install --replace my.bro.bundle

otherwise it would be impossible to compose bundles.

-- 
- Justin Azoff

> On Sep 19, 2016, at 7:27 PM, Robin Sommer <robin at icir.org> wrote:
> 
> At BroCon several people pointed out a need to install Bro packages on
> machines that do not have a direct external connection. One idea would
> be some kind of proxy scheme where an intermediary git repository
> mirrors packages locally; bro-pkg would then pull from there. However,
> I don't think any of us really liked that idea much. After a few
> rounds of discussions, Seth and I came up with a different idea that
> seems easier to manage: extending bro-pkg to bundle packages into
> deployment files that can be easily pushed to Bro systems simply by
> copying them over. 
> 
> I’ve tried to flesh this out a bit more, and would be interested to
> hear what you all think about this approach. And @Jon: Do you think
> this would be doable that way?
> 
> Here’s the idea:
> 
> 1. Generally, one first uses bro-pkg as usual to install packages onto
>   a local Bro system that does have external Internet connectivity
>   (this could be just a dummy Bro installation). One installs new
>   packages there, updates existing ones, etc., until reaching a state
>   that one wants to push out to the actual Bro system.
> 
> 2. We add a new “bundle” command to bro-pkg that serializes the
>   current state of packages into a single file on disk, a “package
>   bundle”. The bundle contains the complete content of all currently
>   installed packages, using some kind of suitable container format
>   (could be just a ZIP file, or whatever works; the internal
>   representation doesn’t really matter).
> 
> 3. Users create such a bundle on the local system and then simply copy
>   that bundle file over to all target Bro machines that do not have a
>   external connectivity themselves, using whatever mechanism they
>   have available (e.g., just scp; or maybe through some configuration
>   management system like Ansible etc.).
> 
> 4. On the target machine, one runs a corresponding “bro-pkg unbundle”
>   command on that bundle file. That command will completely replace
>   the system’s current set of packages with the bundle’s content. As
>   a result, that machine will now have exactly the same packages
>   installed as the original system.
> 
> This would be the general scheme. A couple of people I talked to at
> BroCon confirmed that this would offer a viable solution for them, and
> that they would indeed much prefer copying files around over
> maintaining local git mirrors.
> 
> Some additional thoughts on variations/extensions of this basic scheme:
> 
> - I’m not quite sure if the bundle should contain just the packages
>  themselves or further bro-pkg state as well, such as which packages
>  are currently loaded. Right now I’m learning towards saying “just
>  the packages”; that would basically treat bundles just as a
>  transport mechanism to get packages over to another box. The actual
>  Bro machines would still keep control over which packages to
>  actually load, etc.
> 
> - As it is described above, Step 1 would require having a local Bro
>  installation into which packages get installed before they can be
>  bundled up. It would be nice to have a mode where bro-pkg can
>  operate without having a Bro around at all, just downloading
>  packages locally somewhere for bundling them up. I could also see
>  offering an even simpler mode where one simply lists packages to
>  bundle on the command-line: “bro-pkg bundle <pkg1> <pkg2> <pkg3>”.
>  That would be particularly useful with configuration management
>  systems I think.
> 
> - It would be neat if bro-pkg's Python library exposed operations to
>  inspect & retrieve the content of a bundle, such as iterating over
>  the packages inside a bundle and iterating over the files inside a
>  package. That way one could easily build target-side scripts that
>  process and validate bundles before going ahead and installing them
>  (e.g., imposing custom restrictions on what kind of packages one
>  allows to put in place; or ensuring that a bundle always contains a
>  set of packages the site deems mandatory, to avoid configuration
>  mistakes; or even just logging what gets pushed out).
> 
> What do you guys think about this?
> 
> Robin





More information about the bro-dev mailing list