[Bro-Dev] Scaling out bro cluster communication

Azoff, Justin S jazoff at illinois.edu
Fri Mar 10 13:57:35 PST 2017


> 
> It implements a fake known hosts and scan detection policy.
> 
> the main things to figure out is:
> 
> * How to work out the proper node_count at runtime.  I think on a real bro cluster the Cluster namespace has the data I need for this, including which nodes are reachable.
> 
> * How to handle one node becoming unreachable or a new node showing up.  Ideally bro would use a form of consistent ring hashing.
> 
> If this were worked out, and implemented for logging as well, you could run a bro cluster with 2 'manager' nodes and have a fully functioning cluster even if one of them died.
> 
> As is, I can probably use this on our test cluster to run 4 data nodes and distribute scan detection to 4 cpu cores.

I got this to work on a real cluster running the broker integration branch!

https://github.com/JustinAzoff/broker_distributed_events/blob/master/distributed_broker.bro

Took me a while to figure out that a change somewhere made Broker::publish_topic required to send messages to other nodes, and that to raise notices on a datanode bro needs:

redef datanode2manager_events += {"Notice::cluster_notice"};

The main screwy thing is that I couldn't figure out a consistent way to enumerate the data nodes[1], so I just hardcoded the index in the name:

[datanode-0]
type=datanode
host=bro-test

[datanode-1]
type=datanode
host=bro-test

etc.

Once I had the base functions in place I changed my scan.bro from

    #this is a worker2manager event
    event Scan::scan_attempt(scanner, attempt);

to

    local args = Broker::event_args(scan_attempt, scanner, attempt);
    Cluster::send_event_hashed(scanner, args);

And.. it just worked.  I have 8 data nodes running that are each handling 1/8th of the scan detection aggregation.


-- 
- Justin Azoff


[1] They are stored in a set, and the 'first' data node on different workers was not always the same... would need to convert the set of data nodes to a list and sort it.


More information about the bro-dev mailing list