[Bro-Dev] Broker port status

[Azoff, Justin S]

On Mar 8, 2018, at 2:51 PM, Jon Siwek <jsiwek at corelight.com<mailto:jsiwek at corelight.com>> wrote:

On Thu, Mar 8, 2018 at 2:18 PM, Azoff, Justin S <jazoff at illinois.edu<mailto:jazoff at illinois.edu>> wrote:

(1) &synchronized, &mergeable, &persistent.  Seems fine to deprecate these now.

I'm fine with it going away, but there needs to be some sort of replacement for &synchronized,
minimally a how-to for porting existing scripts to something else.

The answer is generally "use the new store API" to replace
&synchronized or &persistent.  It's difficult to give a canonical
porting strategy as it depends on how the data is being
accessed/modified, though you can find a preview of the how-to port
document at [1].  Links and stuff will get rendered more nicely when
it's live on bro.org<http://bro.org>.

- Jon

[1] https://github.com/bro/bro/blob/topic/actor-system/doc/frameworks/broker.rst

The most common use of &synchronized here is for loading files into tables using input+reread.  a cron job updates the table on the manager
which updates the in memory table and syncs it over to all the workers.

These are read only tables, so I think the best replacement for those may be to switch to the new configuration framework, which would probably
mean the deletion of a ton of verbose Input::add_table statements.



I got the actor-system branch running on a small test cluster.. manager+logger+proxy on one box and 32 workers on 2 physical boxes.

It seems to be working for the most part.  known hosts and friends are working great now with no duplicate entries at startup.

One thing I notice is that the traffic to/from the manager box and cpu has increased quite a bit.

Ignoring the large CPU spikes from building the new branch just before the switch at 15:30, the overall cpu load on the manager box is about 3x higher.
The bandwidth isn't terribly excessive, but it increased from about 16mbit to 40mbit (stupid graph is in mebibytes).

Based on some capstats runs, it's all going to the logger port 47761.  I have another graph that shows the total log volume being written to disk and that hasn't changed.
So it looks like this branch uses 2x the cpu and bandwidth to write the same volume of logs.

[cid:498F993F-1122-4661-B343-AB3683018F09]
[cid:D6C31831-533C-40AB-995B-B025A0237E2C]

—
Justin Azoff

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180308/43c9175a/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2018-03-08 at 4.58.39 PM.png
Type: image/png
Size: 66285 bytes
Desc: Screen Shot 2018-03-08 at 4.58.39 PM.png
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180308/43c9175a/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2018-03-08 at 4.33.19 PM.png
Type: image/png
Size: 336516 bytes
Desc: Screen Shot 2018-03-08 at 4.33.19 PM.png
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180308/43c9175a/attachment-0003.bin