[Bro-Dev] libbroker status/plans

[Siwek, Jon]

> On Nov 3, 2014, at 3:48 PM, Robin Sommer <robin at icir.org> wrote:
> 
> 
> 
> On Thu, Oct 23, 2014 at 21:59 +0000, you wrote:
> 
>> I started looking in to this a little and I’m thinking either LevelDB
>> or RocksDB may be good default choices to use here.
> 
> I looked over them a bit, and RocksDB looks pretty cool, although also
> quite complex given that we won't need all of what it offers.
> 
> Have you considered SQLite as an alternative?

Had not thought of that.

> It's more than a
> key/value store, and slower, but it would have the advantage of not
> adding another dependency beyond what we already use.

I wasn’t that worried about adding another dependency since they’re already kind of specific, i.e. don’t expect the DB dependency to be more of a hassle than libcaf.  If it is a concern, we could consider distributing (e.g. as git submodules) and building these dependencies along with Broker (analogous to Bro redistributing the sqlite amalgamation).

In the end, I’m not expecting there to actually be a lot of code involved in implementing different persistent storage backends, so we wouldn’t be stuck with SQLite if that’s chosen as a default.  And we could provide more than one option at a time.  Maybe we could have the default be SQLite (for convenience), but optionally support RocksDB (for those in need of better performance).

> Also, I was thinking it would be cool to have a command line tool that
> can inspect (and potentially even manipulate [1]), the contents of a
> Broker store. Say, you wanted to see what IPs are currently tracked in
> some table, you could just run that tool to dump it out.
> 
> (*) Does any of the DBs have support for modifying a table exernally
> while being open? Then that command line tool could even add/change
> entries that way. That would actually make for a nice configuration
> mechanism for things like whitelists or some tuning options.

We need to be using Broker’s data store abstraction when making changes for those updates to be correctly propagated to clones.  But should be easy to write such tools using libbroker.  Or once there’s Python bindings, those will probably be a natural way to do such dynamic querying and modification to data stores.

- Jon