[Bro-Dev] Package manager meta data

[Jan Grashöfer]

> Aggregating it into the package source is a better solution than having every client do it.  The later isn’t going to scale well:  the client will take longer and longer over time as more and more packages get registered to a source.  Also takes longer as a function of total number of release versions a package has because we are collecting metadata for each version.  Rather not ask users to just get used to developing more patience over time.

I am not sure how strong the effect might be but this is definitely a
good point!

> I’d opt for a daily cron job to aggregate metadata into package sources.

For me this seems to be the easiest solution providing a maximum of
usability, too.

> It’s not a problem for the metadata to be out of sync for a day since only the “search” command is going to be using the aggregated data.  Other commands would have direct access to accurate metadata since they’ve already cloned the package locally.

What about the "info" command? If a package is not installed it would be
nice if the command would obtain the latest meta data from the package's
repository as well.

> It would also be trivial to give users access to the aggregation tool if they have a problem with potentially using day-old metadata in their searches and are prepared to wait however long the aggregation process takes.
> 
> E.g. we add this command/flag: `bro-pkg refresh —aggregate-metadata`
> 
> Then the only difference between the daily aggregation process and a user is that the daily process does a `git commit && git push` in the locally cloned package source that bro-pkg is using internally.

That sounds great to me! This would be exactly the behavior I would
expect based on other package managers I have used so far.

>> Another question: Now that repositories only contain bro-pkg.index files
>> with links instead of submodules, how are deleted/unavailable packages
>> detected/removed?
> 
> At the moment, they’d have to be removed manually whenever someone notices or reports it.
> 
> If we switch to automated metadata aggregation, removal of nonexistent packages could naturally be a part of that.

Thanks for the clarification. Automatic removal might be a bit
aggressive but a warning would be very helpful I guess.

Best regards,
Jan