[Bro-Dev] package manager progress

[Robin Sommer]

On Sun, Jul 24, 2016 at 19:45 +0000, you wrote:

> The package manager client is at a point now where I think it would be
> usable.

Finally got a chance to play with it a bit. Excellent work, I really
like it!

Belows a list of just smaller things I noticed. The only larger
question I have is regading the use of submodules, also following up
on other parts of this thread. In principle, I actually quite like the
idea of using submodules; git already offers the mechanism, so why not
build on that. That said, seeing how the package manager ends up using
submodules, it's not quite the pure git model actually. If I
understood it right, it's using them really only to *find* the
external repos, but not to pinpoint a particular commit in there; the
package source never even updates the submodules. Given that approach,
I'm now wondering if a custom scheme wouldn't be the more intuitive
solution. My concern is that whoever looks at this submodule usage,
will take a while to understand what's actually happening. One could
argue that's only an implementation detail and shouldn't really matter
to anybody. On the other hand, if, for example, somebody ends up
browsing the package source repository on GitHub, I'm sure they'd be
confused by all the packages pointing to very old versions. So I'm
wondering if it would be worth switching to custom index instead of
submodules; seems that wouldn't be difficult either if indeed all we
need to do is track the external URLs somehow. Also, if you want to
track discoverability metadata there already as well, seems that the
URL could just become part of that, no?

Here's my list of other random things I noticed:

- Would suggest to rename “pkg.meta” to, say, “bro-pkg.meta”, just to
  make it more explicit that it's a Bro package. That's something one
  can also then search for on GitHub.

- Does "upgrade" show the packages affected and ask for confirmation?
  I would suggest either doing that or require an --all option for
  upgrading everything, as that's a potentially dangerous operation.

- I suppose upgrading does (better: will do) dependency checking
  again, including making sure the Bro version matches the one that
  update now requires?

- When installing the package manager as part of Bro, could we pull in
  the Python dependencies automatically, for example by installing
  them into the same prefix? Both GitHub and semantic_version are
  pretty non-standard. Using them is ok I think but it would be nice
  if "bro-pkg" wouldn't abort first thing because they aren't
  installed yet.

- How about adding a note to either packages.bro or the whole
  packages/ directory that's it's automatically maintained and not
  supposed to be manually messed with?

- In bro-pkg.conf, has "default" in "[sources]" a special meaning, or
  could it be any tag? Assuming the latter, I would just call it
  "bro": "bro/jsiwek/bro-test-package" is more intuitive than
  "default/jsiwek/bro-test-package".

- For our default package source, do we want to support non-GitHub
  repositories? If so, a naming scheme by GitHub user won't work.

- Suggest to rename "/opt/bro/var/lib/package-manager" to
  "../bro-package-manager" or "../bro-pkg".

- Once we support dependencies on Bro versions, would be nice if that
  worked also with the "x.y-z" scheme that git master uses (and maybe
  it just does anyways).

- I like the Python API!

- Documentation (nice!):

    - Python 3.x works, right? Then I'd list that explicitly.

    - A quick-start guide would be helpful that just mentions the most
      important steps, including basic installation along with Bro
      itself (once that's merged).

    - The "Installation" section becomes a bit confusing towards with
      the end with all those paths. Maybe split some parts out into an
      advanced section or so?

    - How-tos would be helpful that show by example how to create a
      (1) a pure script package, (2) and binary Bro plugin, and (3) a
      BroControl plugin.

Finally, my take on your questions:

> My current idea is that instead of putting this type of data inside
> the package’s metadata, the user puts it in the package source’s
> metadata.

Yeah, I like that.

> Automatic inter-package dependency analysis

Agree on lower priority.

> * Is it acceptable to depend on GitPython and semantic_version python packages?

It is, it just would be nice if we could help people a bit getting
these installed, see above.

> * Documentation is hosted on GitHub at the moment, move to bro.org?

Keeping these docs separate makes sense, although it would also be
nice to have the option to integrate into bro.org at least.  For now,
I think it's fine to just do your own Sphinx and publish it wherever
(GitHub, RTM). We can later see what to do about bro.org. Generally,
our bro.org setup certainly needs work, it's become hard to maintain
and extend. We have been talking about some options here recently but
not settled on anything in particular yet.

> * Thoughts on when to merge ‘package-manager’ branch in ‘bro’ ?

The main question is if we see this as a 2.5 feature? If so, we should
merge soon; if not, postpone until that's out the door. Given how far
you are already, I vote for making it part of 2.5. We could declare it
experimental still for 2.5 to get some more time to iron out the
workflow before we tell people it's ok to start relying on it.

Again, all very nice work!

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin