[Bro-Dev] Broker API update

[Matthias Vallentin]

Here's a brief update about the recent changes to the Broker API update.
For more examples, you can look at the unit tests [1]. I'm also
attaching a small graphic that illustrates the architecture from a high
level.

Fundamentally, endpoints represent processing units that exchange
messages. An endpoint processes one message at a time, but all endpoints
execute in paralell---similar to the actor model. Each message has
associated a topic and endpoints deliver messages according to their
subscriptions and peerings. Unlike in the previous version, there is no
more global state: each endpoints lives in a context.

There exist two types of endpoints: blocking and nonblocking. The former
features a synchronous interface where users have to manually extract
messages from the endpoint. Messages accumulate in the endpoint's
"mailbox" until extracted. On the contrary, a nonblocking endpoint
executes a callback for each message it receives. Here's a code snippet:

  context ctx;
  auto e0 = ctx.spawn<blocking>();
  auto e1 = ctx.spawn<nonblocking>(
    [](const topic&, const message&) {
      // Inspect topic t and/or process message.
    }
  );

  // Establish a peering between two endpoints.
  e0.peer(e1);

  // Subscribe to some topics. The subscribing endpoint will relayed its
  // subscriptions to all known peers.
  e0.subscribe("/foo");
  e0.subscribe("/bar");

  // Block and wait until a message arrives according to the endpoint's
  // subscriptions.
  e0.receive(
    [](const topic&, const message&) {
      // Inspect topic t and/or process message.
    },
    [](const status& s) {
      // Process status messages, such as new/lost peers
    }
  );

Broker will only allow messages that contain instances of data, which is
a sum type and can be any of the types in Bro's data model. In the
future, we may loosen this restriction to allow users sending their own
custom types over the same Broker communication channel. But for we
enforce this requirement. This is the API to send messages:

  // Send a message of type <integer, real, count> under topic /foo.
  e1.publish("/foo", 42, 4.2, 42u);

  // Construct a message of type <string, vector> and sends it away.
  auto msg = make_data_message("foo", vector{42, nil, 44});
  e1.publish("/foo", msg);

You will get a compile error if the arguments to make_data_message or
publish are not unambiguously convertible to one of the types in
Broker's data model.

To cross process boundaries, endpoints can use TCP communication as
follows:

  // In one process.
  context ctx;
  auto e = ctx.spawn<...>();
  e.listen("127.0.0.1", 42000);

  // In another process.
  context ctx;
  auto e = ctx.spawn<...>();
  e.peer("127.0.0.1, 4200);

If one would "accidentally" establish a TCP connection between endpoints
in the same process, the runtime would detect this scenario and avoid
costly message serialization and use "pointer passing" instead.

On my todo list for the next couple of weeks are (in order):

    (0) Fix remaining bugs (mostly CAF)
    (1) Adapt data stores to the new API
    (2) Create Python bindings via pybind11
    (3) Perform the "engine swap" in Bro
    (4) Extensive unit and performance testing
    (5) Update documentation à la CAF [2]

Moreover, I'm already in touch with Mathias Fischer regarding multi-hop
subscription management. For now, we have a single-hop peerings, i.e.,
if a peering of the form A <-> B <-> C exists, A relays a subscription
change only to B. See Mathias' email for more details on this topic.

    Matthias

[1] https://github.com/bro/broker/tree/topic/matthias/actor-system/tests/cpp
[2] http://actor-framework.readthedocs.io/en/latest/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: architecture.png
Type: image/png
Size: 169803 bytes
Desc: not available
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160701/9dfeb550/attachment-0001.bin