[Bro-Dev] Package manager meta data

[Siwek, Jon]

> On Oct 27, 2016, at 6:28 PM, Robin Sommer <robin at icir.org> wrote:
> 
> Then, bro-pkg.meta has a "version" field (the docs show that as the
> one field to put in there). I believe that version isn't really used
> anywhere other than showing it as part of the meta data output in
> "info”.

Right, the “version” field in bro-pkg.meta doesn’t serve an actual function at the moment.  I was thinking about removing it (more on that in a response below).

> then there's also PackageStatus.current_version, which is from git
> (although I'm not sure how I would actually set that; is it just any
> tag?).

If you mean what control does the author of a package have over how that field gets populated: they can use git tags or branches.

With respect to the way users follow package updates, I expected the two common cases to be either tracking a particular git branch or by only updating to release versions.  A “release version” here is any git tag that looks like a SemVer (e.g. 1.0.0).  So a user might do things like:

bro-pkg install —version 1.0.0 foo

or

bro-pkg install —version master foo

> Ideally, what I'd like to have is a single interface (speaking in
> Python API terms) for a package that gives me all the meta-data
> consistently, pulling it out from where's it's stored and maintaining
> it when bundling. For things like "description", "tags", it could pull
> them out of bro-pkg.index by default, and maybe override them from
> bro-pkg.meta if they are defined there (so that one can set them even
> if there's no package source to begin with).

That could be misleading.  The purpose of putting those tags in bro-pkg.index is because they are related to package “discoverability” and it’s easier to search that data if it’s located in a single repo, the package source, rather than having to download the entire universe of packages up front.

If bro-pkg.meta is allowed to “override” those fields just for the purpose of displaying metadata in the “info” command, it’s misleading because that is not the actual data that’s searchable via the “search” command.

If it’s important to always have some sort of “description” field to display from the “info” command, which would help situations where a package was never located via a package source (e.g. installed directly by git URL), then I’d suggest maybe either displaying contents of the README or just having a differently-named field.  Since the user has already found the package, maybe it’s more useful for them to have a “usage” or “quickstart” field that gives them some high-level view of the API or options to tweak.

Technically, people can put any field they want in bro-pkg.meta and “info” will display it, so it’s just a matter of documenting suggested practices.

> And bundles would then
> carry the information through, probably inside their manifest.

Yeah, it would be fine to bind whatever metadata is available in the current bro-pkg.index to a package as it gets bundled or installed and to show that in the “info” command.

> For
> version information, bro-pkg could start with the git branch/tag but
> override it with values from bro-pkg.index and bro-pkg.meta if defined
> there. That way people could choose how to do their versioning. The
> API would just offer a single version() doing the right thing, and
> "bro-pkg info" would show that.

How would those other versioning schemes actually work, though?

A user needs to be able to install a particular package version, so bro-pkg needs to be able to associate a version with a git commit.

If versioning is done in bro-pkg.meta within a package, I’m not sure how to locate a particular version.  Wouldn’t I have to brute force search through all commits?  Even then, it’s possible the author screwed up and has multiple commits with the same version in bro-pkg.meta, which one does bro-pkg choose?

If versioning is done in bro-pkg.index, the author could create mappings of “version -> commit hash”.  But if we’re talking about the official bro “packages” source, they have to submit a pull request each time they want to change that information and someone on the Bro team needs to approve it.

Sticking with standard git mechanisms seems like the best thing to do here.  Tracking updates to branches “just works” and if people create tags that conform to semantic versioning then tracking updates to stable releases is also straight forward.  This method also works for installing packages directly via explicit git URL (not via a package source).  There’s also no way to accidentally create versioning ambiguities because git is the sole authority on versioning.

> Does this make sense?

I'll need to see your responses to above points, but the point about “info” showing consistent metadata and working when offline makes sense.  Set of potential changes:

1) Documentation updates: get rid of references to “version” in bro-pkg.meta, make it clear to use SemVer-formatted git tags for release versioning.

2) When a package is installed or bundled/unbundled, include a copy of its data from the currently-available bro-pkg.index.  This data is bound to that particular installation of the package.

3) The “info” command should first check if a package is installed and display all metadata bound to that installation before trying to download direct from the origin repo or looking it up in package source.  This makes “info” work offline when the package is installed.

Let me know how that looks.

- Jon