[Bro-Dev] Rethinking Broker's blocking API

[Matthias Vallentin]

Broker's current API to receive messages is as follows:

    context ctx;
    auto ep = ctx.spawn<blocking>();
    ep.receive([&](const topic& t, const data& x) { .. });
    ep.receive([&](const status& s) { .. });

or the last two in one call:

    ep.receive(
      [&](const topic& t, const data& x) { .. },
      [&](const status& s) { .. }
    );

The idea behind this API is that it's similar to the non-blocking
endpoint API:

    auto ep = ctx.spawn<nonblocking>();
    ep.subscribe([=](const topic& t, const data& x) { .. });
    ep.subscribe([=](const status& s) { .. });

Non-blocking endpoints should be the default, because they are more
efficient due to the absence of blocking. For simplicity, the current
API also provides a non-lambda overload of receive:

    auto ep = ctx.spawn<blocking>();
    auto msg = ep.receive();
    std::cout << msg.topic() << " -> " << msg.data() << std::endl;

Users can also check the mailbox of the blocking endpoint whether it
contains a message:

    // Only block if know that we have a message.
    if (!ep.mailbox().empty())
      auto msg = ep.receive();

What I haven't considered up to now is the interaction of data and
status messages in the blocking API. Both broker::message and
broker::status are messages that linger the endpoint's mailbox. I find
the terminology confusing, because a status instance is technically also
a message. I'd rather speak of "data messages" and "status messages" as
opposed to "messages" and "statuses". But more about the terminology
later.

There's a problem with the snippet above. If the mailbox is non-empty
because it contains a status message, the following call to receive()
would hang, because it expects a data message. The only safe solution
would be to use this form:

    if (!ep.mailbox().empty())
      ep.receive(
        [&](const topic& t, const data& x) { .. },
        [&](const status& s) { .. }
      );

The problem lies in the receive() function that returns a message. It
doesn't match the current architecture (a blocking endpoint has a single
mailbox) and is not a safe API for users.

Here are some solutions I could think of:

    (1) Let receive() return a variant<message, status> instead, because
        the caller cannot know a priori what to expect. While simple to
        call, it burdens the user with type-based dispatching afterwards.
    
    (2) Specify the type of message a user wants to receive, e.g.,

          auto x = ep.receive<message>();
          auto y = ep.receive<status>();

        Here, don't like the terminology issues I mentioned above. More
        reasonable could be 

          auto x = ep.receive<data>();
          auto y = ep.receive<status>();

        where x could have type data_message with .topic() and .data(),
        and y be a direct instance of type status.

        But because callers don't know whether they'll receive a status
        or data message, this solution is only an incremental
        improvement.

    (3) Equip blocking endpoints with separate mailboxes for data and
        status messages. In combination with (2), this could lead to
        something like:

            if (!ep.mailbox<data>().empty())
              auto msg = ep.receive<data>();

            if (!ep.mailbox<status>().empty())
              auto s = ep.receive<status>();

        But now users have to keep track of two mailboxes, which is more
        error-prone and verbose.

    (4) Drop status messages unless the user explicitly asks for them.
        Do not consider them when working with an endpoint's mailbox,
        which only covers data messages.

        While keeping the semantics of ep.receive() simple, it's not
        clear how to poll for status messages. Should they just
        accumulate in the endpoint and be queryable? E.g.,:

            // Bounded? Infinite?
            const std::vector<status>& xs = ep.statuses();

Ultimately, I think it's important to have consistent API of blocking
and non-blocking endpoints. Any thoughts on how to move forwards would
be appreciated.

    Matthias