[Bro-Dev] Empty log fields

Matthias Vallentin vallentin at icir.org
Sun Dec 4 10:15:57 PST 2011


What you motivate is precisely the need for binary logs, which aim to
ship with 2.1. This address both the log size and representation issues,
as null values are a NUL byte and empty values their type-specific
binary equivalent.

Clearly, it makes much more sense to use the binary log format when
sending them to a database. Going further, one would create a custom
database backend that writes the logs directly from the Bro process to
the database, without the intermediate step of serializing them to the
binary format. In 2.1, we have a CouchDB backend that demonstrates this
usage.

Unfortunately, for ASCII logs there is a trade-off between clarity and
conciseness. While omitting the null/empty representation entirely is
the most space-efficient way to go, it may break text-based tools that
expect a strictly columnar format and have no notion of field separator.
Moreover, if a user needs to separate the cases of null (no value there)
vs. empty (e.g., the empty string ""), we need two separate
representations. Some users may also find an explicit clue about missing
values less confusing.

I propose something new: in addition to allowing the field separator to
be customized, we allow similar redefinitions for null and empty values.
By default, they are the same character, namely the dash, but can be
easily redef'ed.

    Matthias


More information about the bro-dev mailing list