[Bro-Dev] Broker data layouts
jsiwek at corelight.com
Tue Aug 21 10:34:58 PDT 2018
On Tue, Aug 21, 2018 at 8:54 AM Dominik Charousset
<dominik.charousset at haw-hamburg.de> wrote:
> This raises a couple of questions. Primarily: where can Broker users learn the layouts to interpret received data?
broker/bro.hh is basically all there is right now. e.g. if you
construct a broker::bro::Event from a received broker::data, you get
access to event name + "interpretable" arguments.
> There’s essentially no hope in deferring a layout, since broker::data doesn’t include any meta information such as field names. Is meta information stored somewhere?
No, nothing like that is in implicit in message content.
> Is there a convention how to place and retrieve such meta information in Broker’s data stores?
No, and any stores created by Bro don't even have such meta info. The
types stored in them are just documented like "the keys are type Foo
and values are type Bar".
> How does Bro itself make such information available?
Nothing beyond documentation or the Bro -> Broker type mapping that's
implicit in events themselves (or as given in docs for data stores).
> Is there a document that lists all topics used by Bro with their respective broker::data layout?
I don't think there's a plan to keep such an up-to-date document.
A basic usage premise that I'm wondering about is that none of the
current Broker usage in Bro actually seems suitable for generic/public
consumption as it is. It's maybe more implementation details of doing
cluster-enabled network traffic analysis, so also not a primary goal
to make interpretation of those communications easy for
external/direct Broker users. (You can ingest it if you want and do
the work of "manually" interpreting it all, but maybe won't be a
stable/transparent source of data going forward).
However, one can still use Bro + Broker to create their own
events/stores in a way that does contain the meta information required
for easier/programmatic interpretation on the receiving side. e.g. I
think, at the moment, if one is interested in ingesting data produced
by Bro, they are best served by explicitly defining topic names,
event/data types, and explicitly producing those messages at suitable
places within Bro scripts themselves. Then, one can be in control of
defining a common/expected data format and include whatever meta
information is necessary to help receivers interpret the data.
Maybe there's a more standardized approach that could be worked
towards, but likely we just need more experience in understanding and
defining common use-cases for external Bro data consumption.
Or if we were just talking about Broker-only usage independent of Bro,
then I think it's still the same ideas/answers: currently up to user
to decide how to encode broker::data in a way that defines
common/expected layouts + any required meta info.
Does that help at all?
More information about the bro-dev