[Bro] Evaluating compression algorithms for Bro logs and PCAPs
vallentin at icir.org
Thu Oct 1 02:36:29 PDT 2015
Recently I've heard about Google's new Brotli algorithm outperforming so
many others. I wanted to figure out whether it holds up to his words and
wrote a small benchmark . This benchmark compares various algorithms
along space (compresion ratio) and time (throughput in bytes/sec). As
input I've taken both PCAPs and Bro logs.
Except for the tradeoff plot, I hope the figures speak for themselves.
The tradeoff figure visualizes the space-time spectrum of the
compression algorithm. It plots space savings  against throughput
during compression. The further a point sits in the top-right corner,
the more of a best-of-both-worlds algorithm it is. ZSTD appears to have
that property for my tested input.
Brotli compresses at a higher ratio, but is also much slower. So for
high-throughput applications that pump out logs, it may not be
well-suited if a minimal CPU footprint has priority. Even thought Brotli
would have been a nice name for the best algorithm for Bro logs, it only
ranks second after ZPAQ in terms of space savings. However, Brotli is
3 orders of magnitude faster than ZPAQ during decompression, rendering
it a solid choice when you have enough cycles available during
More information about the Bro