[Bro-Dev] Broker data layouts

[Robin Sommer]

On Sat, Aug 25, 2018 at 17:42 +0200, Matthias Vallentin wrote:

> Okay. In the future, we probably need some form of
> "serialization-free" batching mechanism to ship data more efficiently.

Do you guys have a sense of how load splits up between serialization
and batching/communication? My hope has been that batching itself can
take care of the performance issues, so that we'll be able to send
logs as standard CAF messages, each one representing a batch of N log
lines. The benchmark I had created a little while ago to examine that
wasn't able to get the necessary performance out of Broker/CAF to do
that (hence the fall-back to Bro's old serialization of log messages
for now, sent over CAF). But iirc, the conclusion was that there's
still room for improvement in CAF that should make this feasible
eventually. However, if you guys believe it's really CAF's
serialization that's the bottle-neck, then we'll need to come up with
something else indeed.

Robin

-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com