[Bro-Dev] Queueing in Broker?

[Seth Hall]

On 13 Feb 2018, at 11:35, Robin Sommer wrote:

> One more Broker idea: I'm thinking we should add a queuing mechanism
> to Broker that buffers outgoing messages for a while when a peer goes
> down. Once it comes back up, we'd pass them on. That way an endpoint
> could restart for example without us loosing data.

Yes!

> I'm not immediately sure how/where we'd integrate that. For outgoing
> messages, we could add it to the transparent reconnect. However, for
> incoming connections, where the local endpoint doesn't have a notion
> of "that peer should be coming back", it might not be as straight
> forward?

I can imagine being able to define queue length and queue (byte) size 
for consumers and producers might be interesting.

As a producer:
	Keep up to 1000 messages and/or 1MByte of data.
As a consumer:
	Only be willing to receive up to the 1000 most recent message or up to 
1MByte of data.

I still haven't spent time with the broker API to see if these thoughts 
actually make sense though. :)

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com