[Bro-Dev] send_id (Re: [Bro-Commits] [git/bro] topic/jsiwek/actor-system: Finish port of control framework to use broker. (8dddae1))
jan.grashoefer at gmail.com
Mon Aug 28 02:09:50 PDT 2017
On 27/08/17 04:03, Seth Hall wrote:
> I believe that Robin meant the intel framework instead of sumstats.
> (Hopefully this avoids some confusion)
Thanks for the clarification! I was thinking about send_id() in context
of the intel framework as well. As you might noticed, I enjoyed playing
around with the intel framework :) Thus, some questions to make sure I
got everything correctly:
> On Sat, Aug 26, 2017 at 11:12 AM Robin Sommer <robin at icir.org> wrote:
>> Jon, replacing send_id() may indeed work better with an extension at
>> the C++/Broker level. I'd like to avoid introducing new dependencies
>> on Bro's serialization code, as I'm very much hoping that once the old
>> communication code code goes we won't need that serialization layer
>> anymore either (I know we're using it for opaque values over Broker
>> too, but that's quite contained and should be easy to replace).
So sending opaque values will still be possible using broker, right?
>> - There's one larger problem with replacing send_id() though: the
>> old communication system has logic to send large values
>> incrementally, so that send_id() won't block stuff. As Seth
>> reminded me the SumStats framework is relying on that quite
>> extensively for sending tables around. Incremental operation is
>> something we don't have with Broker. I think that's ok, we can
>> replace the few existing use cases of sending large values with
>> something else. For SumStats that should probably be data
As far as I understand the broker data stores (straight forward
key-value stores), a data store does not fit for the intel framework, as
it uses e.g. the patricia-trie implementation in tables to efficiently
match subnets. Additionally I was thinking about using cuckoo-filters,
implemented as opaque type, to further improve matching on workers.
However, the intel framework uses send_id() only initially to transfer
the current min_data_store to newly connected workers. Every further
update is handled "manually". Thus I guess there would be two options:
1. Sending all data at once. Maybe ok for that use case.
2. Sending stuff incrementally using some script-layer logic.
Am I right here?
More information about the bro-dev