[Bro-Dev] Packaging (Re: Beta schedule)

[Jonathan Siwek]

>> - I'm making releases (with tarballs) for all the submodules.
> 
> I'm trying to build tar-files for all our submodules but I'm running
> into a number of things.
> 
> First, afaics, not all modules yet provide the capability to create
> the tarballs (or at least I can't directly figure out how).
> Specifically:
> 
>    - capstats, binpac, and bro-aux: they have "dist" Makefile
>      targets, but running thise gives me:
> 
>        make[1]: *** No rule to make target `package_source'.  Stop.
> 
>    - broccoli-ruby: No "dist" Makefile target and I have no idea how
>    to do that for Ruby packages (I've just added an empty dummy one.)

The package_source target is something provided by CMake, and adding it would require changing top-level CMakeLists.txt to include the cmake/ConfigurePackaging.cmake script, but I don't think that's ultimately the way to go...

> 
> Then, the top-level Bro "make-src-packages" also builts the Broccoli
> and BroControl tgz. I think it would be better to leave that to the
> submodules themselves, otherwise things get inconsistent (because we
> have all these further submodules as well, which do it themselves). 
> So I've removed those lines from make-src-packages.

Yeah, sounds good -- repos for which we we want to create recursive packages should handle that themselves...

> 
> I also noticed that Bro's make-src-packages essentially just tars
> everything that's in the source directory (except for some tmp files
> etc.). There's a problem with that: I'm sure that sometimes I will
> have additional files lying around there which will then end up in the
> distribution. Is there a way to make sure that only stuff under
> version control gets actually into the tgz?

I think we need to use a combination of `git ls-files` to generate a package manifest as input to `tar -T`.

The trick is that all non-recursive source packages need to include the contents of their cmake/ submodule if it exists.

For the recursive package bundles (Bro, BroControl, Broccoli), the output of a recursive `git ls-files` needs to be altered to prepend the right relative path of submodules.

> (And please remind me:
> what was the reason that we can't just let CMake do the tar balls?

So that the packager doesn't need to have CMake as a dependency.
And actually it's worse than that: the packaging configuration provided by CMake is also currently tied to the full suite of ./configure dependency checks as if one were going to actually build the project.
(This was originally pointed out as a pretty bad inconvenience by Craig Leres)

> Here's how I'd like to do the tarballs: I've written a script that
> recursively goes through all submodules and runs "make dist" there.
> It's in aux/devel-tools now. The script then collects all the
> resulting tarballs at the top-level with the right naming/path scheme
> for copying them over to the web server. For that to work, we need to
> ensure that (1) each Makefile has indeed a working "dist" target, and
> (2) that that targets outputs the path to the generated tarball so
> that the top-level script can pick it up. Most CMake-builds do (2)
> already and for the Python modules I've added output in the form of
> "Package: relative/path/to/tgz" (which is then grepped for).[1]
> 
> Robin
> 
> [1] It works to write out multiple lines like that if more than one
> tgz is built (like bro vs bro-all). Btw, I'm wondering if we should
> name them the opposite way: bro-2.0.tgz would include all submodules,
> and bro-2.0-minimal.tgz would not. Same scheme for BroControl and
> Broccoli (where we don't do two tarballs yet).

Sounds good, I'll work on cleaning things up.

- Jon