[Bro-Dev] libbroker status/plans

[Siwek, Jon]

> On Oct 23, 2014, at 2:47 PM, Robin Sommer <robin at icir.org> wrote:
> 
>> - C API
> 
> Yeah, I would like to have this early on. In principle, we could
> postpone to later, but it looks like one of these things that if we
> don't get it into place right away, it will be even harder to do
> later. Maybe others can help you with this; once an initial structure
> is in place it's probably getting quite mechanic.

I agree it will probably be a pretty mechanical process once there’s a better foundation that tackles some of the harder aspects and common patterns.  But I wonder if prioritizing Bro integration over this may be more helpful — if we learn significant parts of the C++ interface need to change, then that may mean some of the effort making the C interface goes to waste.

> Another question here would be how we ensure that the C API stays in
> sync going forward. Is there some testing we can put in place?

There’s already some unit tests for the C++ API that could be replicated in C.

>> - Python bindings
> 
> That I can see skipping for the first version. While it would of
> cource be great to have, it should be pretty straight forward to do
> once the C bindings are there.

Won’t BroControl require this?

>> - Persistent storage backend
> 
> My guess is that storage will end up being the primary initial use
> case (because that's a capability we don't have right now; vs.
> replacing exisitng stuff), so yeah, a good target.

I started looking in to this a little and I’m thinking either LevelDB or RocksDB may be good default choices to use here.

>> - SSL/IPv6 (dependent on actor-framework support)
> 
> Can do later, though I don't rememeber what the conclusion was if/how the
> actor-framework supports these?

IIRC, the idea was that it could, but doesn’t yet.  I also think it’s not critical to have these in the initial version.

>> - In-place data store value modifications
>> 
>>    Plan to support increment/decrement on integral values.
>>    Need any other operations?
> 
> Set operations (insert element, remove element).

Ack.

>>    What to do when applying an operation to invalid data type?
>>    Plan to just send error message back to sender and leave further
>>    decisions up to them.
> 
> Sounds like a more general question: what to do in terms of semantic
> errors? There are probably more like that, like writing to a store
> that doesn't exist. Error messages sound right, with hooks to report
> them to the application. In Bro they can show up as events in script
> land.

Yeah, probably does warrant a general solution for application to get at errors that can’t be reported synchronously — I can recall a few other unrelated places in the code I’ve written something like “TODO: log an error”.

>> - Data store support for optional expiry model
>> 
>>    What are the desired mechanims?  Options:
>> 
>>        (1) Inserter may specify "expire this entry at time X" ?
>>        (2) Inserter may specify "expire this entry based on
>>            create/read/modification access time" ?
> 
> How about providing on-create and on-modification, but not on-read. In
> those two cases there's already communication necessary anyways, and
> the expiration time could be piggy-bagged on that.

May be reasonable, I’ll aim to support that and see if I hit any issues.

>>        (3) Other hooks to make expiry conditional?
> 
> Hmm, maybe, but unclear how it would look like. Could work like Bro's
> expire func (i.e., potentially delay expiration), but I think this
> hook could really only run only at the storage node directly. I'd skip
> for now.

Ack.  Was hoping skipping this would be ok :)

> A more general question in this context: what's the trust model? Are
> we expecting that a client taking part in the communication will play
> by the rules of the protocol? That's what current Bro does, and I
> think it's a reasonable assumption.

This was also my expectation.

> On the other hand, maybe Broker
> could do a bit better, as it's data model isn't as complex as Bro's
> native Val structure. Asked differently: to which degree can a
> receiver validate that incoming data makes sense (with some appropiate
> definition of "makes sense"; there are differen ones, like having the
> right binary layout, or semantically sending valid information)

If “receiver" here means the application I think it can validate pretty well.  For example, if the application is Bro and it’s receiving logs, I expect it currently has enough information from Broker to be able to tell if what it got can actually be converted in to the types it needs to log.

If “receiver” means Broker versus The Internet, I’m not actually sure the extent it would hold up -- that seems dependent on CAF being able to hold up.

>> - Bro integration
>>    Is Broker the default in Bro 2.4 ?  That implies requiring C++11.
>>    Also I'm requiring CMake 2.8.12+ and may be hard to go below 2.8.
>>    Bro is still happy with 2.6.3.
> 
> We should definitly intregrate it. It could be an optional dependency,
> or a mandatory component. I'm leaning towards the latter, to pave the
> way for the future. But yeah, that then means requiring C++11 (and a
> current cmake).
> 
> Did we ever come to a conclusion on C++11 for Bro? We did the survey,
> but I don't recall if we settled on whether it's ok to switch now?

We didn’t reach a conclusion.  I can take a closer look, but I recall supporting EL6 may be important.  On RHEL6, looks like GCC 4.4.7 is the default, but all major C++11 features I think appear in 4.8 and later (I’m not sure it’s worth it to try and figure out what specific C++11 features we can “get away with” and thus require older compiler version).

- Jon