[Bro-Dev] Broker data layouts
vallentin at icir.org
Sat Aug 25 08:42:35 PDT 2018
> Agree. Right now a newly connecting peer gets a round of explicit
> LogCreates, but that's probably not the best way forward for larger
Okay. In the future, we probably need some form of
"serialization-free" batching mechanism to ship data more efficiently.
There exist technologies like Apache Arrow, flatbuffers, Cap'N'Proto,
MsgPack, etc., all of which require building a set of values once, and
then just copying them around as a binary blob on the wire.
Deserialization is not needed because one would typically only "view"
the data through light-weight accessors.
We're doing something similar in VAST for performance reasons, but Bro
and Broker have the exact same issues in that regard.
> > In other words, can Broker currently be used if one writes a Bro
> > script that publishes plain events (message type 1 in bro.hh)?
> Yes to that. Non-Bros can exchange events (assuming they know the
> schema), but not logs.
(Unfortunately that will make our BroCon talk pretty boring in terms
of throughput analysis, because we were planning to build an
end-to-end log ingestion system based on Broker. We'll probably switch
gears a bit and focus more on the latency side, where a Bro script
publishes something to an external application and receives feedback
though an auxiliary channel.)
More information about the bro-dev