[Bro-Dev] send_id (Re: [Bro-Commits] [git/bro] topic/jsiwek/actor-system: Finish port of control framework to use broker. (8dddae1))

[Robin Sommer]

Jon, replacing send_id() may indeed work better with an extension at
the C++/Broker level. I'd like to avoid introducing new dependencies
on Bro's serialization code, as I'm very much hoping that once the old
communication code code goes we won't need that serialization layer
anymore either (I know we're using it for opaque values over Broker
too, but that's quite contained and should be easy to replace).

Some thoughts here:

     - I'm thinking the best approach may be a new Bro-specific
       message for Broker, similar to the log-create/write messages.
       We can add that to the Bro shim in Broker. It'd send the name
       of the ID and the new value as a broker::data instance, and the
       receiving Bro updates the value as the message comes in.

     - There's one larger problem with replacing send_id() though: the
       old communication system has logic to send large values
       incrementally, so that send_id() won't block stuff. As Seth
       reminded me the SumStats framework is relying on that quite
       extensively for sending tables around. Incremental operation is
       something we don't have with Broker. I think that's ok, we can
       replace the few existing use cases of sending large values with
       something else. For SumStats that should probably be data
       stores. I don't remember if there even are further instances of
       this, my guess is no (I don't think we need to consider
       broctl's configuration updating here; those values are small
       and a non-incrememtnal send_id() is fine).

     - Another approach for broctl's updating could be switching to
       the upcoming configuration framework, which takes a different
       approach to dynamic reconfiguration. However, it's a bit out
       still until we can that switch completely, so for now providing
       a substitute for send_id() that can cover the simple uses cases
       looks like the best way to go.

Robin

On Fri, Aug 25, 2017 at 13:15 -0500, Jonathan Siwek wrote:

> commit 8dddae17db9340f5261d11382aa9b67e965d5fef
> Author: Jon Siwek <jsiwek at illinois.edu>
> Date:   Fri Aug 25 13:15:00 2017 -0500
> 
>     Finish port of control framework to use broker.
>     
>     To replace usages of the send_id BIF, I had to make 3 new BIFs
>     
>     - serialize
>     - deserialize
>     - update_ID
>     
>     Using those, any Bro value can be explicitly converted to a string
>     of bytes, sent to a peer via a Broker event, unserialized to a Bro
>     val on the other side and then installed into a global identifier
>     via its name.
>     
>     I think this may be the most straightforward way to get this to
>     work for now, at least without changing significantly the Broker
>     internals or messaging format.  It mostly boils down to not being
>     able to deserialize into a Bro value with the 'any' type, at least
>     not without also carrying Bro's actual type information somewhere
>     inside Broker's serialized message.
>     
>     And I think deserializing into 'any' would be needed because it's
>     not possible to e.g. explicitly enumerate all possible types in a Bro
>     script and have a particular event signature to use for any given one.
>     That's not possible because there's infinite ways you can create
>     composite types (tables, sets, vectors, etc).

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin