[Bro-Dev] libbroker status/plans

[Robin Sommer]

On Wed, Oct 22, 2014 at 19:39 +0000, you wrote:

> If anyone has time/interest, I feel like the main components of Broker
> are established now and deserving feedback/critique.

Very cool. I'll look through the repository.

Some toughts regarding your TODO list, in terms of priorities and your
question:

> - C API

Yeah, I would like to have this early on. In principle, we could
postpone to later, but it looks like one of these things that if we
don't get it into place right away, it will be even harder to do
later. Maybe others can help you with this; once an initial structure
is in place it's probably getting quite mechanic.

Another question here would be how we ensure that the C API stays in
sync going forward. Is there some testing we can put in place?

> - Python bindings

That I can see skipping for the first version. While it would of
cource be great to have, it should be pretty straight forward to do
once the C bindings are there.

> - Persistent storage backend

My guess is that storage will end up being the primary initial use
case (because that's a capability we don't have right now; vs.
replacing exisitng stuff), so yeah, a good target.

> - SSL/IPv6 (dependent on actor-framework support)

Can do later, though I don't rememeber what the conclusion was if/how the
actor-framework supports these?

> - Need/want overload or flow-control mechanisms?

Punting for now sounds good, per the other mails. Maybe CAF will get
some support eventually that we can leverage, and/or we can add the
script-level hooks.

> - In-place data store value modifications
> 
>     Plan to support increment/decrement on integral values.
>     Need any other operations?

Set operations (insert element, remove element).

>     What to do when applying an operation to invalid data type?
>     Plan to just send error message back to sender and leave further
>     decisions up to them.

Sounds like a more general question: what to do in terms of semantic
errors? There are probably more like that, like writing to a store
that doesn't exist. Error messages sound right, with hooks to report
them to the application. In Bro they can show up as events in script
land.

> - Data store support for optional expiry model
> 
>     What are the desired mechanims?  Options:
> 
>         (1) Inserter may specify "expire this entry at time X" ?
>         (2) Inserter may specify "expire this entry based on
>             create/read/modification access time" ?

How about providing on-create and on-modification, but not on-read. In
those two cases there's already communication necessary anyways, and
the expiration time could be piggy-bagged on that.

>         (3) Other hooks to make expiry conditional?

Hmm, maybe, but unclear how it would look like. Could work like Bro's
expire func (i.e., potentially delay expiration), but I think this
hook could really only run only at the storage node directly. I'd skip
for now.

> - Data typing model

>         (1) Data holds additional type tag to suggest how to interpret
>         (2) Fully implement separate Bro-types.
> 
>     Planning to try integrating w/ Bro as it is and see what specific
>     problems arise.  I think (1) may end up being helpful, but maybe not
>     required and I'd like to avoid (2) if possible.

I'm not fully sure what you mean by (2) but I believe I agree. :)

(1) would be good; maybe Bro doesn't strictly need it, but (a) it
would allow it to double-check input at least; and (b) for independent
applications it will be quite helpful, as otherwise it's hard to work
with data of different input types dynamically (in other words: the
applications would need a way to define what to expect, forcing them
to replicate the typing that exists in Bro already).

A more general question in this context: what's the trust model? Are
we expecting that a client taking part in the communication will play
by the rules of the protocol? That's what current Bro does, and I
think it's a reasonable assumption. On the other hand, maybe Broker
could do a bit better, as it's data model isn't as complex as Bro's
native Val structure. Asked differently: to which degree can a
receiver validate that incoming data makes sense (with some appropiate
definition of "makes sense"; there are differen ones, like having the
right binary layout, or semantically sending valid information)

> - Bro integration
>     Is Broker the default in Bro 2.4 ?  That implies requiring C++11.
>     Also I'm requiring CMake 2.8.12+ and may be hard to go below 2.8.
>     Bro is still happy with 2.6.3.

We should definitly intregrate it. It could be an optional dependency,
or a mandatory component. I'm leaning towards the latter, to pave the
way for the future. But yeah, that then means requiring C++11 (and a
current cmake).

Did we ever come to a conclusion on C++11 for Bro? We did the survey,
but I don't recall if we settled on whether it's ok to switch now?

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin