[Bro-Dev] Testing and Docs for Packages

[Siwek, Jon]

> On Jan 15, 2017, at 4:49 PM, Jan Grashöfer <jan.grashoefer at gmail.com> wrote:
> 
> In general I think, making test cases available for users of a package
> could be quite helpful. Further, I think we have also already mentioned
> the possibility of compatibility checking regarding the installed Bro
> version by executing tests. Thus I would propose introducing test_dir to
> bro-pkg.meta for tests and a test command to execute the tests for
> bro-pkg as a first step.

Yes, seems useful.  I’d do it like:

1) Add `bro-pkg test <package>` command.
2) Add “test_command” field to bro-pkg.meta

The “test_command” is more general than “test_dir" — the command could just `cd test_dir` if needed and there’s no other reason bro-pkg needs to know the dir where tests are stored, is there?

Other questions:

1) Is it fine for `bro-pkg test <package>` to operate on the installed version of the package or are there expectations of testing a package in an isolated sandbox without installing it?  I think the former is more useful since it may catch odd inter-package conflicts that wouldn’t show up when testing in isolation.

2) Should we put btest on PyPi, add it as a dependency to bro-pkg, and make it the canonical testing framework for packages?  This gives devs a straightforward way to proceed w/ writing tests and guarantees that bro-pkg users always have the ability to run them.

(There is a “btest” package on PyPi, but not the one we know, so not sure how to resolve that...)

> Another thing that would be great to have for packages would be
> documentation. I've experimented with the broxygen feature but the doc
> about generating docs is not that exhaustive ;) I think providing an
> easy-to-use mechanism to generate documentation for thrid-party scripts
> would be great. Ideally the generated documentation would link to Bro's
> online doc for built-in types and scripts.

If the problem is that there’s a lack of examples/templates for generating script API docs via broxygen or that it simply doesn’t work at the moment, then yes, that’s something to fix.

But regarding the direction of autogenerated package docs in general, maybe it makes sense to work on that in conjunction with a web-frontend for package sources (e.g. a package repository browser).  A package would be able to generate its docs independent of that, but if the web-frontend is going to become a go-to place for looking at package info/stats/docs and it takes care of creating all that without package authors having to do anything, then that seems like the superior/common use-case.

- Jon