[Bro-Dev] Queueing in Broker?

[Jon Siwek]

On Tue, Feb 13, 2018 at 10:35 AM, Robin Sommer <robin at icir.org> wrote:
> One more Broker idea: I'm thinking we should add a queuing mechanism
> to Broker that buffers outgoing messages for a while when a peer goes
> down. Once it comes back up, we'd pass them on. That way an endpoint
> could restart for example without us loosing data.

If the goals is to prevent loss of data, then don't we need more than
just buffering, like message acknowledgements from the peer?

e.g. you can think your peer is up, send a message, then immediately
find out it went offline and so the message got lost "in the middle".

You would also need to keep the message buffered until receiving an
ACK from *all* peers that are subscribed (and the subscription list is
a potentially moving target) ?

And if you still planned on message routing/auto-forwarding being more
widely used, I think you would want to buffer the message while the
longest subscribed *path* has a down node?

> I'm not immediately sure how/where we'd integrate that. For outgoing
> messages, we could add it to the transparent reconnect. However, for
> incoming connections, where the local endpoint doesn't have a notion
> of "that peer should be coming back", it might not be as straight
> forward?

Yeah, I'm also unclear if there's anyway you can tell if the peer is
supposed to be permanent vs. transient in come cases.

Last observation is that I think any of these types of changes would
be to the internal messaging pattern/protocol and so maybe reasonable
to change/improve in subsequent releases in a way that's transparent
to users.

- Jon