[Bro-Dev] [Bro-Commits] [git/bro] topic/actor-system: First-pass broker-enabled Cluster scripting API + misc. (07ad06b)

[Azoff, Justin S]

> On Nov 2, 2017, at 5:21 PM, Siwek, Jon <jsiwek at illinois.edu> wrote:
>> 
>> Mostly so that workers don't end up spending all their time sending out messages when they should be analyzing packets.
> 
> Ok, I get what you want to avoid, though could be interesting to actually have a fully-connected cluster in order to collect performance data on each comm. pattern and see how significant the difference is for a variety of use-cases.
> 
> Do you have a particular example you can give where you’d use this BIF/function to relay a broadcast from a worker to all other workers via a proxy?

Scripts like what validate-certs does to broadcast the presence of a new intermediate cert to the other nodes in the cluster.
That script does have the added optimization on the manager side for only broadcasting the value once if a new cert is seen by two workers at the same time,
so maybe it's not the best example for a broadcast.

With explicit event destinations and load balancing across data nodes that script could look something like

function add_to_cache(key: string, value: vector of opaque of x509)
    {
    # could still do @if ( Cluster::is_enabled() ), but in a standalone setup we are the data
    # node, so this would just short circuit to raise the event locally.  I'd rather broker do
    # the @if internally than have every script have to have two implementations.
    Broker::publish_hrw(Cluster::data_pool, key, SSL::new_intermediate, key, value);
    }

event SSL::new_intermediate(key: string, value: vector of opaque of x509)
    {
    if ( key in intermediate_cache )
        return;
    intermediate_cache[key] = value;
    # in a standalone setup this would just be a NOOP
    Broker::broadcast(Cluster::worker_pool, SSL:: intermediate_add, key, value);
    }

event SSL::intermediate_add(key: string, value: vector of opaque of x509)
    {
    intermediate_cache[key] = value;
    }

Without the added optimization you'd just have

function add_to_cache(key: string, value: vector of opaque of x509)
    {
    intermediate_cache[key] = value;
    # in a standalone setup this would just be a NOOP
    Broker::broadcast(Cluster::worker_pool, SSL:: intermediate_add, key, value);
    }

event SSL::intermediate_add(key: string, value: vector of opaque of x509)
    {
    intermediate_cache[key] = value;
    }

The optimization could be built into broker though, something like

    Broker::broadcast_magic_once_whatever(Cluster::worker_pool, key, SSL:: intermediate_add, key, value);

That would hash the key, send it to a data node, then have the data node broadcast the 
event while adding key to a 'recently broadcasted keys' table that only needs to buffer for 10s or so.

This would enable you to efficiently broadcast an event (once) across all workers with a single line of code.

In either case all of the

 manager2worker_events
 worker2manager_events 
 @if ( Cluster::is_enabled() && Cluster::local_node_type() != Cluster::MANAGER ) 

Is no longer needed.

I guess my whole point with all of this is that if the intent of a script is that an event should be seen on all workers, the script should look something like

    Broker::broadcast(Cluster::worker_pool, SSL:: intermediate_add, key, value);

and not have a bunch of redefs and @ifs so that the script can eventually have

    event SSL::new_intermediate(key, value);


>> How would something like policy/protocols/ssl/validate-certs.bro look with intermediate_cache as a data store?
> 
> global intermediate_store: Cluster::StoreInfo;
> 
> event bro_init()
> 	{
> 	intermediate_store = Cluster::create_store(“ssl/validate-certs/intermediate_store");
> 	}
> 
> And then port the rest of that script to use broker data store api (get/put/exists calls) to access that store.
> 
> - Jon

Does that have the same performance profile as the current method?

    if (issuer in intermediate_cache)

vs

    Broker::get(intermediate_cache, issuer)

— 
Justin Azoff