[Bro-Dev] Scaling out bro cluster communication
Azoff, Justin S
jazoff at illinois.edu
Fri Mar 10 13:57:35 PST 2017
>
> It implements a fake known hosts and scan detection policy.
>
> the main things to figure out is:
>
> * How to work out the proper node_count at runtime. I think on a real bro cluster the Cluster namespace has the data I need for this, including which nodes are reachable.
>
> * How to handle one node becoming unreachable or a new node showing up. Ideally bro would use a form of consistent ring hashing.
>
> If this were worked out, and implemented for logging as well, you could run a bro cluster with 2 'manager' nodes and have a fully functioning cluster even if one of them died.
>
> As is, I can probably use this on our test cluster to run 4 data nodes and distribute scan detection to 4 cpu cores.
I got this to work on a real cluster running the broker integration branch!
https://github.com/JustinAzoff/broker_distributed_events/blob/master/distributed_broker.bro
Took me a while to figure out that a change somewhere made Broker::publish_topic required to send messages to other nodes, and that to raise notices on a datanode bro needs:
redef datanode2manager_events += {"Notice::cluster_notice"};
The main screwy thing is that I couldn't figure out a consistent way to enumerate the data nodes[1], so I just hardcoded the index in the name:
[datanode-0]
type=datanode
host=bro-test
[datanode-1]
type=datanode
host=bro-test
etc.
Once I had the base functions in place I changed my scan.bro from
#this is a worker2manager event
event Scan::scan_attempt(scanner, attempt);
to
local args = Broker::event_args(scan_attempt, scanner, attempt);
Cluster::send_event_hashed(scanner, args);
And.. it just worked. I have 8 data nodes running that are each handling 1/8th of the scan detection aggregation.
--
- Justin Azoff
[1] They are stored in a set, and the 'first' data node on different workers was not always the same... would need to convert the set of data nodes to a list and sort it.
More information about the bro-dev
mailing list