[Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

[Azoff, Justin S]

> On Nov 2, 2017, at 2:37 PM, Aashish Sharma <asharma at lbl.gov> wrote:
> 
> 
> 
> Now, while Justins' multiple data nodes idea has specticular merits, I am not much fan of it. Reason being having multiple data-notes results in same sets of problems

It does not have the same problems.. It may have different problems that I haven't thought of yet, but it doesn't have the same problems.

> syncronization,

What synchronization problems?

> latencies

Adding multiple data nodes will reduce the load on each node and lower overall latencies.

> mess of data2worker, worker2data events etc etc


you're projecting the current mess of worker2manager_events and manager2worker_events onto what I am trying to replace them with.

Having
    worker2manager_events
and
    @if ( Cluster::is_enabled() && Cluster::local_node_type() != Cluster::MANAGER )

all over the place exists because bro doesn't have higher level methods for distributing data and events across the cluster.  I am not proposing replacing that with

    worker2datanode_events
and
    @if ( Cluster::is_enabled() && Cluster::local_node_type() != Cluster::DATANODE )

I'm proposing getting rid of that sort of thing entirely.  No '@if cluster'. no 'redef worker2manager_events'. All gone.

> I'd love to keep things rather simple.  Cooked data goes to one (or more) datanodes (datastores). Just replicate for relibaility rather then pick and choose what goes where. 

Then clusters will just change from having an overloaded manager process that is falling under the load to 2 data nodes that are both failing.  This is just renaming the current bottlenecks and is not a solution.

I implemented a multi data node cluster back in March on top of topic/mfischer/broker-integration .  Porting my scan.bro from the manager2worker_events stuff to sending events directly to one of N datanodes was:

Remove:

    redef Cluster::worker2manager_events ...
    @if (Cluster ...
    event Scan::scan_attempt(scanner, attempt);

Add:
    
    local args = Broker::event_args(Scan::scan_attempt, scanner, attempt);
    Cluster::send_event_hashed(scanner, args);

Other than having that wrapped in a single function, it doesn't get any easier than that.

— 
Justin Azoff