[Bro-Dev] Doc generation tests fail

[Jonathan Siwek]

> - is there a way to produce an error messages that makes clear
> it's a problem with the doc markup? Otherwise this can be quite
> confusing as one would normally just ignore comments when looking
> for errors the parser is reporting.

Not sure how I can get it to be 100% sure a given parse error is because of doc markup, but I can easily add a hint to the message when in doc mode to remember to check ## style comments for syntax errors.

> - I'm wondering whether Bro should report this also when not
> running in doc mode. Otherwise, such things are hard to catch.

Right now it's using flex start conditions to only send doc-related tokens to the parser in certain places (enum/record bodies).  If instead I always send tokens, then this type of error could be reported when not running in doc mode.  But that would also probably mean that we can't re-use the "##" syntax in enum/record bodies -- we'd need yet another style.

> Perhaps a middle ground: should we have a test that ensures that
> all scripts we ship do correctly parse in doc mode? The existing
> tests reported logging.bro but only because it's a central one.
> Ones that are less used will be harder to find (and will in
> particular mess up the auto-generated pages on the web server)

I think maybe this can be covered by a build test on NMI that checks `make doc` doesn't have any error.
 
> > the ordering of set/table values seems to be able to change between runs.
> 
> In theory, that shouldn't be the case as long as the RNG's seed is
> set, which btest.cfg does. (But see below for practice).
>
> And the tables do actually come out in a different order here as well.
> I'm wondering whether seeding the RNG isn't sufficient. It has been in
> the past afairc, but I guess same order isn't guarnateed *across*
> systems but only within a single system. But then we have a problem
> here ...
>
> I saw you added a canonifier script for (2) above, but I'm guessing
> that works only in this specific case? (Haven't looked closer at it).

Yeah, it's really crappy; not generalizable.  I think we need something better (but I'm not sure where to start looking to fix the general problem).

- Jon