[Bro-Dev] Proposal: Operating the Bro package manager offline

Robin Sommer robin at icir.org
Wed Sep 21 12:45:46 PDT 2016


After discussion with some more folks, and the email thread, some
further tweaks to my original "bundle" proposal:

    - Following Justin's suggestion, unbundling should by default not
      replace everything currently installed; and instead offer a
      "--replace" option if one wants that.

    - People seem to like the "bro-pkg bundle <pkg1> <pkg2> <pkg3>"
      approach quite a bit as well, so that should be a "first class
      citizen" too. So we'd have two modes: either (1) bundle what's
      currently installed locally, or (2) bundle what's given
      explicitly at startup (and ignore the current local state
      completely). For the latter mode, allowing the list of packages
      to come from a configuration file (rather than on the command
      line) would be very useful as well I hear.

    - Per Jon's suggestion, when unbundling a first-time install of a
      package autoloads it.

    - I'm sure this would just work automatically, but for
      completeness: unbundling should install packages just exactly
      the same way as if they had been coming in "normally" from git.
      In particular, if that target system later does get external
      connectivity and switches to using git directly, it should "just
      work".

    - Also for completness: The "bundle mode" should work without
      requiring a local Bro installation. Per Jon's reply, that would
      be the case automatically already as well.

    - One clarification, as I'm not sure that was clear: "bundle"
      would not build any architecture-specific code locally, that
      would happen at unbundle time on the target server. "bundle" is
      really just a transport mechanism for the getting package source
      content over to offline systems.

    - I'm withdrawing the part on exposing an introspection API for
      bundles through Python. I wouldn't object to having that, but
      it's probably unnecessary given that one can also just look at
      the package directory once things are installed. Should be good
      enough.

Robin

On Mon, Sep 19, 2016 at 16:27 -0700, I wrote:

> At BroCon several people pointed out a need to install Bro packages on
> machines that do not have a direct external connection. One idea would
> be some kind of proxy scheme where an intermediary git repository
> mirrors packages locally; bro-pkg would then pull from there. However,
> I don't think any of us really liked that idea much. After a few
> rounds of discussions, Seth and I came up with a different idea that
> seems easier to manage: extending bro-pkg to bundle packages into
> deployment files that can be easily pushed to Bro systems simply by
> copying them over. 
> 
> I’ve tried to flesh this out a bit more, and would be interested to
> hear what you all think about this approach. And @Jon: Do you think
> this would be doable that way?
> 
> Here’s the idea:
> 
> 1. Generally, one first uses bro-pkg as usual to install packages onto
>    a local Bro system that does have external Internet connectivity
>    (this could be just a dummy Bro installation). One installs new
>    packages there, updates existing ones, etc., until reaching a state
>    that one wants to push out to the actual Bro system.
> 
> 2. We add a new “bundle” command to bro-pkg that serializes the
>    current state of packages into a single file on disk, a “package
>    bundle”. The bundle contains the complete content of all currently
>    installed packages, using some kind of suitable container format
>    (could be just a ZIP file, or whatever works; the internal
>    representation doesn’t really matter).
> 
> 3. Users create such a bundle on the local system and then simply copy
>    that bundle file over to all target Bro machines that do not have a
>    external connectivity themselves, using whatever mechanism they
>    have available (e.g., just scp; or maybe through some configuration
>    management system like Ansible etc.).
> 
> 4. On the target machine, one runs a corresponding “bro-pkg unbundle”
>    command on that bundle file. That command will completely replace
>    the system’s current set of packages with the bundle’s content. As
>    a result, that machine will now have exactly the same packages
>    installed as the original system.
> 
> This would be the general scheme. A couple of people I talked to at
> BroCon confirmed that this would offer a viable solution for them, and
> that they would indeed much prefer copying files around over
> maintaining local git mirrors.
> 
> Some additional thoughts on variations/extensions of this basic scheme:
> 
> - I’m not quite sure if the bundle should contain just the packages
>   themselves or further bro-pkg state as well, such as which packages
>   are currently loaded. Right now I’m learning towards saying “just
>   the packages”; that would basically treat bundles just as a
>   transport mechanism to get packages over to another box. The actual
>   Bro machines would still keep control over which packages to
>   actually load, etc.
> 
> - As it is described above, Step 1 would require having a local Bro
>   installation into which packages get installed before they can be
>   bundled up. It would be nice to have a mode where bro-pkg can
>   operate without having a Bro around at all, just downloading
>   packages locally somewhere for bundling them up. I could also see
>   offering an even simpler mode where one simply lists packages to
>   bundle on the command-line: “bro-pkg bundle <pkg1> <pkg2> <pkg3>”.
>   That would be particularly useful with configuration management
>   systems I think.
> 
> - It would be neat if bro-pkg's Python library exposed operations to
>   inspect & retrieve the content of a bundle, such as iterating over
>   the packages inside a bundle and iterating over the files inside a
>   package. That way one could easily build target-side scripts that
>   process and validate bundles before going ahead and installing them
>   (e.g., imposing custom restrictions on what kind of packages one
>   allows to put in place; or ensuring that a bundle always contains a
>   set of packages the site deems mandatory, to avoid configuration
>   mistakes; or even just logging what gets pushed out).
> 
> What do you guys think about this?
> 
> Robin
> 
> -- 
> Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin



-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin


More information about the bro-dev mailing list