[Bro-Dev] Queueing in Broker?

[Robin Sommer]

On Tue, Feb 13, 2018 at 11:41 -0600, Jon wrote:

> If the goals is to prevent loss of data, then don't we need more than
> just buffering, like message acknowledgements from the peer?

Yeah, I wouldn't see it as bullet-proof reliability, rather as a best
effort "let's no needlesly drop stuff on the floor" kind of thing. I'm
thinking less here of the cluster setting (where things can get
complex and we'd usually restart everything anyways), and more of
external agents streaming stuff into Bro, like with the osquery
plugin. If one needs to restart the receiving-side Bro, it would be
nice to not just drop any activity reported in the meantime. With that
perspective, it would really just need just a bit of buffering of
messages that cannot be sent out right now. And if in the end they
still don't make it, that's not the end of the world.

> And if you still planned on message routing/auto-forwarding being more
> widely used, I think you would want to buffer the message while the
> longest subscribed *path* has a down node?

I was thinking to do the buffering at the routing/hop-level. The
messsage would get as far as it can at first. If a peer is down that a
node would have normally forwarded to, it'd buffer for a bit until
that comes back (but I realize this makes it even more fuzzy which
peers to wait for: in a flexible topology peers could come and go all
the time; see below).

That said, I'm now wondering if such buffering functionality should
really be located inside CAF, as that's in charge of low-level message
propagation.

> Yeah, I'm also unclear if there's anyway you can tell if the peer is
> supposed to be permanent vs. transient in come cases.

We could make that an explicit endpoint option: "for this peer, on
disconnect buffer stuff it would normally receive until it comes back
(subject to some limits)". We may need a better way to identify the
same peer though, just IP probably wouldn't work well. Maybe through
some ID/name sent during the handshake? One would need to configure
such a name for peers when turning on the buffering.

> Last observation is that I think any of these types of changes would
> be to the internal messaging pattern/protocol and so maybe reasonable
> to change/improve in subsequent releases in a way that's transparent
> to users.

Yeah, nothing to get in immediately, still needs some thinking. I'm
getting the sense though that we'll need it for some applications,
osquery being the main one on my mind. 

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin