[Bro-Dev] libbroker status/plans

[Siwek, Jon]

> Do we need a vanilla C API in addition to the C++ API offered? Could be 
> that this was a requirement, so won't argue the point: just making sure 
> someone asked this question.

Yeah, if a goal is to make it easy to instrument existing applications w/ the library, the idea was that some will want/need the C interface.

>> - Need/want overload or flow-control mechanisms?
>> 
>>     E.g. a simple policy for handling overload is to let a user specify
>>     a threshold for how many items are allowed in a queue before new
>>     messages are dropped.
> 
> More general question: are there expectations of reliability on broker 
> messages?  I'm not really familiar enough with the actor model in 
> general / this library to know.  Depending on the protocol being used, 
> it could also be that the transport is going to provide flow-control of 
> its own.

The message delivery is going to be reliable and taken care of for us.  I was more referring to situations such as receiving remote logs at a rate faster than one can process.  What happens currently in Bro and Broker is messages pile up until memory is exhausted and you crash.

I don’t know how great the expectation is to handle that gracefully right away.  We could probably easily put in a way to let a user specify “it’s ok to drop messages for this queue if it gets overloaded”.  Matthias has been talking some about getting flow control mechanisms (e.g. overloaded actor tells senders “I’m currently overloaded, please slow down”) in CAF on their mailing list.  I’m not sure the degree to which something like that would be helpful in Broker — it may just push the overload problem to the sender(s) and begs the question of what they do if they can’t artificially slow down?

> Also, one thing that might concern me a little is that, based on a very 
> limited understanding of CAF, messages appear to be garbage collected 
> [1].  How are the messages GC'd?  Depending on how the GC works, my 
> concern there would be that a poorly-timed GC cycle could lead to drops.

My impression was that messages are reference counted, copy-on-write tuple values that get reclaimed automatically when the ref count reaches zero.

- Jon