[Bro-Dev] The BPAN project
Matthias Vallentin
vallentin at icir.org
Fri Mar 16 12:46:23 PDT 2012
We've been talking a bit about our Bro community script archive BPAN [1]
quite a bit lately. Below, I summarize how I envision this project to
manifest. Feedback would be much appreciated.
Matthias
Objectives
- Facilitate sharing Bro scripts in the community
- Facilitate the integration of user-provided scripts
- Specify a set of repositories (or paths inside) to load Bro scripts
Requirements:
- Sharing scripts should be simple: one should be able to point
someone to a remote repository and the integration happens
automatically.
- Organization by topic tags: similar to package managers, we need a
way to categorize scripts and bundle them. For example
- Separation between officially supported and user/experimental
scripts. For the official ones, we offer support and make sure
they'll always work with the latest/current version of Bro. We do
not make such a guarantee for external scripts.
- Integration with the Input/Output framework. That is, users should
be able to provide custom data sources and sinks.
Implementation details:
- github's popular fork-pull-merge model is an excellent fit for
sharing community repositories. We may want to use
https://github.com/broids as base for our community scripts. Users
that want to extend the base simply perform the changes, open a
pull request, and we can review it. The nice thing is that people
can comment on it and follow-up changes are nicely reflected.
Moreover, github automatically lists contributors to a certain
file, so this encourages coders to share, participate, and get
"reputation"---akin to stackoverflow.
- Unlike homebrew, I believe it does not make sense for Bro to have
single huge repository that contains all scripts and that users
continuously branch from. Bro users probably want to organize
their scripts into different repositories and combine multiple
repositories ad libitum, because some repositories may contain
sensitive elements and should not be public. Yet, private
repositories can still be easily integrated with this new
framework.
Having separate repositories also keeps the repository size under
control. Although the input framework will be the main mechanism
to source external data, I could imagine some folks would like to
stuff some file-based intelligence in their repositories.
However, separate repositories require integration efforts, but
this is manageable in my eyes and outlined below.
- Directory structure: the directory structure provides implicit
naming and some meta data. If the directly structure is violated,
Bro will not be able to automatically load scripts. These are the
rules:
* Sub-directories in the repository represent "addressable
units," e.g., names that users can refer to. I call these from
now on a "rebro". Each rebro must have at least one file
called '__load__.bro'. This is the entry point for Bro.
Example:
- repo := https://www.github.com/joe-user/bromising/
- repo/
repo/a/
repo/a/__load__.bro
repo/a/aa.bro
repo/a/ab.bro
repo/b/__load__.bro/
repo/b/x/__load__.bro
repo/b/x/xx1.bro
repo/b/x/xx2.bro
repo/b/y/yy1.bro
In this example, we have the following rebros:
repo/a
repo/b
repo/b/x
By placing __load__.bro files judiciously, users can
explicitly decide what parts of the repo may be interpreted as
a rebro. We may also allow putting a __load__.bro file in the
top-level directory.
* Each __load__.bro file contains additional meta data. Here is
an example
## @version: 0.42
## @author: Joe Bro <bro at bro-ids.org>
##
## @license: BSD-style (should be the default unless specified)
##
## @bro: 2.1
## @dependencies: b/x, https://my-repo.git/foo
##
## @tags: tls, x509, browser
- The @-prefix inside comments may be used as a specific
marker for parsing. But any other character would do as
well.
- @author: An optional field that identifies the script
author's name and email address. If not given and the author
hosts its repository on github, we may extract this data
from github.
- @license: An optional field to name the license under which
this rebro should go
- @bro: A mandatory field that specifies the Bro version this
rebro is compatible with. Examples:
* =major.minor
exact version match, e.g., =2.1
* >=major.minor
minimum version, e.g., >=2.0
* major.minor-major.minor
range of comptabile versions, e.g., 2.0-2.1
* major.minor,major.minor
list of compatible versions, e.g., 1.5, 2.0beta
- @dependencies: An optional field of internal and external
rebro dependencies. Internal dependencies are specified
relative to the repository top-level directory and external
dependencies need a full-qualified rebro path.
- @tags: An optional field that imposes a logical structure
over rebros. Or we just keep it an
unordered list of tags that users can combine at will.
E.g.,
@tags: tls, x509, browser
would associate three tags with this rebro. We may
pre-define a set of categories under which users can label
their rebros, e.g.,
@tags: webapp/facebook, webapp/facebook
would categorize the rebro hierarchically.
- Let the union of all rebros be the 'universe'. We, the Bro team,
should provide a global view of the universe by encouraging users
to register their rebro's with us. The point of the universe is
that it allows a global search across all existing rebros out
there. One should be able to search the universe by tags, names,
versions, etc.
We may cache this meta data in a separate repository. A script
may re-generate the meta data whenever a new rebro is added to it.
- Configuration: Bro needs to be told which script repositories to
integrate. I suggest all repositories sit in a one directory, that
is controllable via BroControl.cfg, e.g.,
Community = $PREFIX/share/bro/community
Inside that directory, we have two subdirectories: one for all the
repositories and one for the universe, e.g.,
$Community/repositories/git.bro-ids.org/skype-analyzer
$Community/repositories/github.com/git-user/experimental-scripts
$Community/repositories/github.com/git-user/popular-stuff
$Community/universe/...
- One should be able to use BroControl to add new repositories at
runtime and reload Bro with these scripts. Here are some examples
that I will explain below
> rebro add https://git.bro-ids.org/skype-analyzer [alias]
> rebro add git-user/experimental-scripts [testing]
> rebro rm testing
> rebro load alias/decryptor
> rebro load -r alias/spit-injector
> rebro list
> rebro list -u
> rebro list -u -t
> rebro pull
> rebro deps alias
> rebro deps alias/decryptor
> rebro license alias
Commands:
* add: downloads a remote repository and register make all rebros
inside it loadable
* rm: removes a remote repository.
* load: loads a specific rebro. The command fails unless all
dependency requirements are met.
--recursive|-r: downloads dependencies automatically
* list: lists all available loadable rebros.
--universe|-u: performs a global search in the universe
--tags|-t: lists rebro's by tags
* deps: lists dependencies of a rebro.
* license: displays licencing information about a rebro.
[1] Here are some other naming suggestions instead of BPAN:
- Bropane (Bro Policy script Archive NEtwork; inflammable)
- Brofuse (profuse: exuberantly plentiful)
- Browl (A bowl full of Bro scripts)
- Broil (Cooking with the community)
- Brodder (Bro protestants)
- Brovine (Grass-fed, no growth hormones)
More information about the bro-dev
mailing list