[Bro-Dev] Broker port status
jsiwek at corelight.com
Thu Mar 8 11:28:26 PST 2018
Hi all, this is just a status update on porting Bro to use the new
Broker communication system. I'd say the topic/actor-system branch is
now functionally complete with still a few weeks left of internal
* Rename "proxy" nodes?
The previous idea was to call them "data nodes", though I don't see
the benefit. It's awkward to talk about because there's no shorthand
for that node type: you can say "logger", "manager", or "worker",
though you'd have to say the whole "data node" phrase to make any
sense. That also shows that maybe it's helpful to name nodes based
upon a personifiable role that they play: "data" isn't a role/action
performed by a node/person. IMO, "proxy" is still accurate to the
role that these nodes perform: they are intermediaries for offloading
analysis and/or storage. Are there other ideas or is everyone wanting
to go ahead with "data node" ?
* How much of the old comm. system to deprecate vs. remove?
(1) &synchronized, &mergeable, &persistent. Seems fine to deprecate these now.
(2) Communication framework scripts. It's awkward to deprecate stuff
here since they internally will be using what is deprecated. I think
it makes sense to just remove it and let users manually make a copy if
they still need it. Thoughts?
(3) Old C++ comm. code, e.g. RemoteSerializer. I think we'll leave
this untouched for the 2.6 release?
(4) BIFs associated with old comm. system. Depends on (3) (and also
actually (2)), though if the internal core code remains, I think it
makes sense to add &deprecated to these.
And just a quick summary of planned internal improvements:
There's still several usability issues with Broker data stores.
* Online vs. offline operation discrepancies. Expiration times in
Broker are driven by wall time, though for a more familiar/expected
offline usage experience, Bro would need to be able to drive
expiration via network time. There's also some wall-time-based data
store initialization delays that make offline Bro usage difficult as
the Bro-based query timeouts are network-time-based.
* The way clone snapshots are distributed needs to be optimized.
Currently, for any clone that connects, the master floods *all* clones
with a snapshot.
* Certain store operations like "increment" fail if the key they are
to operate on does not exist. It makes more sense to me to assume the
user wants to insert the key with a sane default value (e.g.
zero/empty) in those cases, otherwise, it's awkward/race-prone to
require they do it themselves.
Let me know if there's thoughts on anything else still left to add/improve.
More information about the bro-dev