[Bro-Dev] Testing and Docs for Packages

[Siwek, Jon]

> On Jan 18, 2017, at 10:28 AM, Robin Sommer <robin at icir.org> wrote:
> 
> I also think it would be quite useful to test packages before
> installing them

Maybe I’m not so much questioning whether to run tests before or after installation, but rather if the testing sandbox should include everything from the current installation in addition to the new/updated packages.  i.e. the pre-installation testing sandbox could be initialized with a copy of current installation dirs.

I would think the user’s expectation when running tests is to answer the question “will this bro-pkg operation break my setup?”  You can’t answer that without testing within the context of their current installation environment.  The reason for that is that packages have much potential to change core bro functionality and interfere with each other’s operation.

But for that same reason, it may also make it much harder for people to write unit tests for their package that are precise enough to not cause harmless failures in the presence of other packages — e.g. you couldn’t just check a baseline http.log as some other installed package could have altered it by adding a field, etc.

Summary of approaches/tradeoffs:

1) separate testing environment for each package
- worse at answering “will this bro-pkg operation break my setup?”
- easier to write stable tests

2) single testing environment for all packages
- better at answering “will this bro-pkg operation break my setup?” *if* package tests are written well
- harder to write stable tests

Neither seems great.  I guess I plan to do (1) since it is easier on package authors and less likely to waste users time looking into harmless test failures (ones due to tests that are written too broadly).

Let me know if there’s other ideas/suggestions.

- Jon