[Bro] Evaluating compression algorithms for Bro logs and PCAPs

Matthias Vallentin vallentin at icir.org
Thu Oct 1 02:36:29 PDT 2015


Recently I've heard about Google's new Brotli algorithm outperforming so
many others. I wanted to figure out whether it holds up to his words and
wrote a small benchmark [1]. This benchmark compares various algorithms
along space (compresion ratio) and time (throughput in bytes/sec). As
input I've taken both PCAPs and Bro logs.

Except for the tradeoff plot, I hope the figures speak for themselves.
The tradeoff figure visualizes the space-time spectrum of the
compression algorithm. It plots space savings [2] against throughput
during compression. The further a point sits in the top-right corner,
the more of a best-of-both-worlds algorithm it is. ZSTD appears to have
that property for my tested input.

Brotli compresses at a higher ratio, but is also much slower. So for
high-throughput applications that pump out logs, it may not be
well-suited if a minimal CPU footprint has priority. Even thought Brotli
would have been a nice name for the best algorithm for Bro logs, it only
ranks second after ZPAQ in terms of space savings. However, Brotli is
3 orders of magnitude faster than ZPAQ during decompression, rendering
it a solid choice when you have enough cycles available during
compression.

    Matthias

[1] https://github.com/mavam/compbench
[2] https://en.wikipedia.org/wiki/Data_compression_ratio


More information about the Bro mailing list