[Bro-Dev] Dynamic plugin model (Re: [Bro-Commits] [git/bro] topic/robin/dynamic-plugins-2.3: Start of a plugin writing how-to. (87a1618))

Clark, Gilbert gc355804 at ohio.edu
Thu Dec 19 07:23:15 PST 2013


Hi:

Still running through the plugin generation stuff ... seemed to build the binary distribution okay for me after I set BRO (FC 19, x86_64), but haven't tried to really load / run anything yet.

Personally, I'm a fan of this kind of development model.  I like not having to worry about synchronizing with master to work on a plugin.  I also like that the compilation / link times are much shorter for plugin development than they are when trying to add code to bro directly.

+1 Seth's comment that the init-plugin script is very nice.

A few thoughts:

* Would a section on testing be appropriate?  Both btest and unit testing might be useful for plugins.
* A short section explaining how / when to modify CMakeLists.txt might be useful.  I had to look at one of the existing analyzer plugins to double-check that all CC files needed to go into bro_plugin_cc().  Not hard, but still might be nice to explicitly document.
* Should plugins be allowed to link to additional libraries?  If so, how?  I believe this could become an issue if I write a plugin that links against libXYZ, but libXYZ isn't available on the system that's trying to load the plugin.

Cheers,
Gilbert
________________________________________
From: bro-dev-bounces at bro.org [bro-dev-bounces at bro.org] On Behalf Of Robin Sommer [robin at icir.org]
Sent: Monday, December 16, 2013 3:34 PM
To: bro-dev at bro.org
Subject: [Bro-Dev] Dynamic plugin model (Re: [Bro-Commits]      [git/bro]       topic/robin/dynamic-plugins-2.3: Start of a plugin      writing how-to. (87a1618))

I'd appreciate feedback on the model for dynamic plugins described
below.

The document contains a short intro on writing a simple plugin, as
well as some background on how/when Bro integrates plugins. The code
is in topic/robin/dynamic-plugins-2.3. It hasn't seen much testing yet
but the basic infrastructure seems to work on Linux and Darwin at
least.

Robin

On Mon, Dec 16, 2013 at 12:09 -0800, I wrote:

> +===================
> +Writing Bro Plugins
> +===================
> +
> +Bro is internally moving to a plugin structure that enables extending
> +the system dynamically, without modifying the core code base. That way
> +custom code remains self-contained and can be maintained, compiled,
> +and installed independently. Currently, plugins can add the following
> +functionality to Bro:
> +
> +    - Bro scripts.
> +
> +    - Builtin functions/events/types for the scripting language.
> +
> +    - Protocol and file analyzers.
> +
> +    - Packet sources and packet dumpers. TODO: Not yet.
> +
> +    - Logging framework backends. TODO: Not yet.
> +
> +    - Input framework readers. TODO: Not yet.
> +
> +A plugin's functionality is available to the user just as if Bro had
> +the corresponding code built-in. Indeed, internally many of Bro's
> +pieces are structured as plugins as well, they are just statically
> +compiled into the binary rather than loaded dynamically at runtime, as
> +external plugins are.
> +
> +Quick Start
> +===========
> +
> +Writing a basic plugin is quite straight-forward as long as one
> +follows a few conventions. In the following we walk through adding a
> +new built-in function (bif) to Bro; we'll add `a `rot13(s: string) :
> +string``, a function that rotates every character in a string by 13
> +places.
> +
> +A plugin comes in the form of a directory following a certain
> +structure. To get started, Bro's distribution provides a helper script
> +``aux/bro-aux/plugin-support/init-plugin`` that creates a skeleton
> +plugin that can then be customized. Let's use that::
> +
> +    # mkdir rot13-plugin
> +    # cd rot13-plugin
> +    # init-plugin Demo Rot13
> +
> +As you can see the script takes two arguments. The first is a
> +namespace the plugin will live in, and the second a descriptive name
> +for the plugin itself. Bro uses the combination of the two to identify
> +a plugin. The namespace serves to avoid naming conflicts between
> +plugins written by independent developers; pick, e.g., the name of
> +your organisation (and note that the namespace ``Bro`` is reserved for
> +functionality distributed by the Bro Project). In our example, the
> +plugin will be called ``Demo::Rot13``.
> +
> +The ``init-plugin`` script puts a number of files in place. The full
> +layout is described later. For now, all we need is
> +``src/functions.bif``. It's initially empty, but we'll add our new bif
> +there as follows::
> +
> +    # cat scripts/functions.bif
> +    module CaesarCipher;
> +
> +    function camel_case%(s: string%) : string
> +        %{
> +        char* rot13 = copy_string(s->CheckString());
> +
> +        for ( char* p = rot13; *p; p++ )
> +            {
> +            char b = islower(*p) ? 'a' : 'A';
> +            *p  = (*p - b + 13) % 26 + b;
> +            }
> +
> +        return new StringVal(strlen(rot13), rot13);
> +        %}
> +
> +The syntax of this file is just like any other ``*.bif`` file; we
> +won't go into it here.
> +
> +Now we can already compile our plugin, we just need to tell the
> +Makefile put in place by ``init-plugin`` where the Bro source tree is
> +located (Bro needs to have been built there first)::
> +
> +    # make BRO=/path/to/bro/dist
> +    [... cmake output ...]
> +
> +Now our ``rot13-plugin`` directory has everything that it needs
> +for Bro to recognize it as a dynamic plugin. Once we point Bro to it,
> +it will pull it in automatically, as we can check with the ``-N``
> +option:
> +
> +    # export BRO_PLUGIN_PATH=/path/to/rot13-plugin
> +    # bro -N
> +    [...]
> +    Plugin: Demo::Rot13 - <Insert brief description of plugin> (dynamic, version 1)
> +    [...]
> +
> +That looks quite good, except for the dummy description that we should
> +replace with something nicer so that users will know what our plugin
> +is about.  We do this by editing the ``BRO_PLUGIN_DESCRIPTION`` line
> +in ``src/Plugin.cc``, like this:
> +
> +    # cat src/Plugin.cc
> +
> +    #include <plugin/Plugin.h>
> +
> +    BRO_PLUGIN_BEGIN(Demo, Rot13)
> +        BRO_PLUGIN_VERSION(1);
> +        BRO_PLUGIN_DESCRIPTION("Caesar cipher rotating a string's characters by 13 places.");
> +        BRO_PLUGIN_BIF_FILE(consts);
> +        BRO_PLUGIN_BIF_FILE(events);
> +        BRO_PLUGIN_BIF_FILE(functions);
> +    BRO_PLUGIN_END
> +
> +    # make
> +    [...]
> +    # bro -N | grep Rot13
> +    Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1)
> +
> +Better. Bro can also show us what exactly the plugin provides with the
> +more verbose option ``-NN``::
> +
> +    # bro -NN
> +    [...]
> +    Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1)
> +        [Function] CaesarCipher::rot13
> +    [...]
> +
> +There's our function. Now let's use it::
> +
> +    # bro -e 'print CaesarCipher::rot13("Hello")'
> +    Uryyb
> +
> +It works. We next install the plugin along with Bro itself, so that it
> +will find it directly without needing the ``BRO_PLUGIN_PATH``
> +environment variable. If we first unset the variable, the function
> +will no longer be available::
> +
> +    # unset BRO_PLUGIN_PATH
> +    # bro -e 'print CaesarCipher::rot13("Hello")'
> +    error in <command line>, line 1: unknown identifier CaesarCipher::rot13, at or near "CaesarCipher::rot13"
> +
> +Once we install it, it works again::
> +
> +    # make install
> +    # bro -e 'print CaesarCipher::rot13("Hello")'
> +    Uryyb
> +
> +The installed version went into
> +``<bro-install-prefix>/lib/bro/plugins/Demo_Rot13``.
> +
> +We can distribute the plugin in either source or binary form by using
> +the Makefile's ``sdist`` and ``bdist`` target, respectively. Both
> +create corrsponding tarballs::
> +
> +    # make sdist
> +    [...]
> +    Source distribution in build/sdist/Demo_Rot13.tar.gz
> +
> +    # make bdist
> +    [...]
> +    Binary distribution in build/Demo_Rot13-darwin-x86_64.tar.gz
> +
> +The source archive will contain everything in the plugin directory
> +except any generated files. The binary archive will contain anything
> +needed to install and run the plugin, i.e., just what ``make install``
> +puts into place as well. As the binary distribution is
> +platform-dependent, its name includes the OS and architecture the
> +plugin was built on.
> +
> +Plugin Directory Layout
> +=======================
> +
> +A plugin's directory needs to follow a set of conventions so that Bro
> +(1) recognizes it as a plugin, and (2) knows what to load.  While
> +``init-plugin`` takes care of most of this, the following is the full
> +story. We'll use ``<base>`` to represent a plugin's top-level
> +directory.
> +
> +``<base>/__bro_plugin__``
> +    A file that marks a directory as containing a Bro plugin. The file
> +    must exist, and its content must consist of a single line with the
> +    qualified name of the plugin (e.g., "Demo::Rot13").
> +
> +``<base>/lib/<plugin-name>-<os>-<arch>.so``
> +    The shared library containing the plugin's compiled code. Bro will
> +    load this in dynamically at run-time if OS and architecture match
> +    the current platform.
> +
> +``lib/bif/``
> +    Directory with auto-generated Bro scripts that declare the plugins
> +    bif elements. The files here are produced by ``bifcl``.
> +
> +``scripts/``
> +    A directory with the plugin's custom Bro scripts. When the plugin
> +    gets activated, this directory will be automatically added to
> +    ``BROPATH``, so that any scripts/modules inside can be
> +    ``@load``ed.
> +
> +``scripts``/__load__.bro
> +    A Bro script that will be loaded immediately when the plugin gets
> +    activated. See below for more information on activating plugins.
> +
> +By convention, a plugin should put its custom scripts into sub folders
> +of ``scripts/``, i.e., ``scripts/<script-namespace>/<script>.bro`` to
> +avoid conflicts. As usual, you can then put a ``__load__.bro`` in
> +there as well so that, e.g., ``@load Demo/Rot13`` could load a whole
> +module in the form of multiple individual scripts.
> +
> +Note that in addition to the paths above, the ``init-plugin`` helper
> +puts some more files and directories in place that help with
> +development and installation (e.g., ``CMakeLists.txt``, ``Makefile``,
> +and source code in ``src/``). However, all these do not have a special
> +meaning for Bro at runtime and aren't necessary for a plugin to
> +function.
> +
> +``init-plugin``
> +===============
> +
> +``init-plugin`` puts a basic plugin structure in place that follows
> +the above layout and augments it with a CMake build and installation
> +system. Note that plugins with this structure can be used both
> +directly out of their source directory (after ``make`` and setting
> +Bro's ``BRO_PLUGIN_PATH``), and when installed alongside Bro (after
> +``make install``).
> +
> +``make install`` copies over the ``lib`` and ``scripts`` directories,
> +as well as the ``__bro_plugin__`` magic file and the ``README`` (which
> +you should customize). One can add further CMake ``install`` rules to
> +install additional files if neeed.
> +
> +.. todo::
> +
> +    Describe the other files that the script puts in place.
> +
> +Activating a Plugin
> +===================
> +
> +A plugin needs to be *activated* to make it available to the user.
> +Activating a plugin will:
> +
> +    1. Load the dynamic module
> +    2. Make any bif items available
> +    3. Add the ``scripts/`` directory to ``BROPATH``
> +    4. Load ``scripts/__load__.bro``
> +
> +By default, Bro will automatically activate all dynamic plugins found
> +in its search path ``BRO_PLUGIN_PATH``. However, in bare mode (``bro
> +-b``), no dynamic plugins will be activated by default; instead the
> +user can selectively enable individual plugins in scriptland using the
> +``@load-plugin <qualified-plugin-name>`` directive (e.g.,
> +``@load-plugin Demo::Rot13``).
> +
> +``bro -N`` shows activated and found yet unactivated plugins
> +separately. Note that plugins compiled statically into Bro are always
> +activated, and hence show up as such even in bare mode.
> +
> +.. todo::
> +
> +    Is this the right activation model?
> +
> +
> +Plugin Component
> +================
> +
> +The following gives additional information about providing individual
> +types of functionality via plugins. Note that a single plugin can
> +provide more than one type. For example, a plugin could provide
> +multiple protocol analyzers at once; or both a logging backend and
> +input reader at the same time.
> +
> +We now walk briefly through the specifics of providing a specific type
> +of functionality (a *component*) through plugin. We'll focus on their
> +interfaces to the plugin system, rather than specifics on writing the
> +corresponding logic (usually the best way to get going on that is to
> +start with an existing plugin providing a corresponding component and
> +adapt that). We'll also point out how the CMake infrastructure put in
> +place by the ``init-plugin`` helper script ties the various pieces
> +together.
> +
> +Bro Scripts
> +-----------
> +
> +Scripts are easy: just put them into ``scripts/``, as described above.
> +The CMake infrastructure will automatically install them, as well
> +include them into the source and binary plugin distributions.
> +
> +Builtin Language Elements
> +-------------------------
> +
> +Functions
> +    TODO
> +
> +Events
> +    TODO
> +
> +Types
> +    TODO
> +
> +Protocol Analyzers
> +------------------
> +
> +TODO.
> +
> +File Analyzers
> +--------------
> +
> +TODO.
> +
> +Logging Writer
> +--------------
> +
> +Not yet implemented.
> +
> +Input Reader
> +------------
> +
> +Not yet implemented.
> +
> +Packet Sources
> +--------------
> +
> +Not yet implemented.
> +
> +Packet Dumpers
> +--------------
> +
> +Not yet implemented.
> +
> +Documenting Plugins
> +===================
> +
> +..todo::
> +
> +    Integrate all this with Broxygen.
> +
> +
> +
>
> _______________________________________________
> bro-commits mailing list
> bro-commits at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits
>



--
Robin Sommer * Phone +1 (510) 722-6541 *     robin at icir.org
ICSI/LBNL    * Fax   +1 (510) 666-2956 * www.icir.org/robin
_______________________________________________
bro-dev mailing list
bro-dev at bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev



More information about the bro-dev mailing list