[Bro-Dev] Testing and Docs for Packages

[Johanna Amann]

Just to add my two cents to this, because automated testing is actually
one of the things that I really think package managers should do...

On Mon, Jan 16, 2017 at 06:45:52PM +0000, Siwek, Jon wrote:
> 1) Add `bro-pkg test <package>` command.

Might it also make sense to just run the test on installation, before the
package is actually installed, to see if it works on the environment of
the user?

This might make it much easier for users (& developers) to identify early
when it is something wrong. And bro-pkg could just abort (or ask a user if
it should continue) if a test fails.

> 2) Add “test_command” field to bro-pkg.meta
> 
> The “test_command” is more general than “test_dir" — the command could just `cd test_dir` if needed and there’s no other reason bro-pkg needs to know the dir where tests are stored, is there?
> 
> Other questions:
> 
> 1) Is it fine for `bro-pkg test <package>` to operate on the installed version of the package or are there expectations of testing a package in an isolated sandbox without installing it?  I think the former is more useful since it may catch odd inter-package conflicts that wouldn’t show up when testing in isolation.

I actually think it would be neat to do this isolated, especially given
that this enables testing before installing. It also makes it easier to
create something like "smokers" (Bro installations that just tro tu run
all testsuites of all available packages with a newer version to see if
something went wrong).

Running on the installed version might be a neat bonus, but I actually see
the other as more interesting.

Johanna