[Bro-Dev] Broker data layouts

[Matthias Vallentin]

> Agree. Right now a newly connecting peer gets a round of explicit
> LogCreates, but that's probably not the best way forward for larger
> topologies.

Okay. In the future, we probably need some form of
"serialization-free" batching mechanism to ship data more efficiently.
There exist technologies like Apache Arrow, flatbuffers, Cap'N'Proto,
MsgPack, etc., all of which require building a set of values once, and
then just copying them around as a binary blob on the wire.
Deserialization is not needed because one would typically only "view"
the data through light-weight accessors.

We're doing something similar in VAST for performance reasons, but Bro
and Broker have the exact same issues in that regard.

> > In other words, can Broker currently be used if one writes a Bro
> > script that publishes plain events (message type 1 in bro.hh)?
>
> Yes to that. Non-Bros can exchange events (assuming they know the
> schema), but not logs.

Got it.

(Unfortunately that will make our BroCon talk pretty boring in terms
of throughput analysis, because we were planning to build an
end-to-end log ingestion system based on Broker. We'll probably switch
gears a bit and focus more on the latency side, where a Bro script
publishes something to an external application and receives feedback
though an auxiliary channel.)

    Matthias