[Bro-Dev] send_id (Re: [Bro-Commits] [git/bro] topic/jsiwek/actor-system: Finish port of control framework to use broker. (8dddae1))

[Jan Grashöfer]

On 27/08/17 04:03, Seth Hall wrote:
> I believe that Robin meant the intel framework instead of sumstats.
> (Hopefully this avoids some confusion)

Thanks for the clarification! I was thinking about send_id() in context 
of the intel framework as well. As you might noticed, I enjoyed playing 
around with the intel framework :) Thus, some questions to make sure I 
got everything correctly:

> On Sat, Aug 26, 2017 at 11:12 AM Robin Sommer <robin at icir.org> wrote:
> 
>> Jon, replacing send_id() may indeed work better with an extension at
>> the C++/Broker level. I'd like to avoid introducing new dependencies
>> on Bro's serialization code, as I'm very much hoping that once the old
>> communication code code goes we won't need that serialization layer
>> anymore either (I know we're using it for opaque values over Broker
>> too, but that's quite contained and should be easy to replace).

So sending opaque values will still be possible using broker, right?

>>       - There's one larger problem with replacing send_id() though: the
>>         old communication system has logic to send large values
>>         incrementally, so that send_id() won't block stuff. As Seth
>>         reminded me the SumStats framework is relying on that quite
>>         extensively for sending tables around. Incremental operation is
>>         something we don't have with Broker. I think that's ok, we can
>>         replace the few existing use cases of sending large values with
>>         something else. For SumStats that should probably be data
>>         stores.

As far as I understand the broker data stores (straight forward 
key-value stores), a data store does not fit for the intel framework, as 
it uses e.g. the patricia-trie implementation in tables to efficiently 
match subnets. Additionally I was thinking about using cuckoo-filters, 
implemented as opaque type, to further improve matching on workers.

However, the intel framework uses send_id() only initially to transfer 
the current min_data_store to newly connected workers. Every further 
update is handled "manually". Thus I guess there would be two options:
1. Sending all data at once. Maybe ok for that use case.
2. Sending stuff incrementally using some script-layer logic.
Am I right here?

Thanks,
Jan