[Bro] Version: 2.0-907 -- Bro manager memory exhaustion

[Vlad Grigorescu]

This sounds like an interesting experiment. The main difference that I see
is that when writing raw PCAP, you're writing to mostly-contiguous blocks.
With the text logs, you're writing to 20 files, scattered who-knows-how,
and at least 3 or 4 of those files are heavily used. The SATA layer
*should* try to reorder the writes to minimize seeks, but who knows how
efficient that is.

If this actually was the issue, it'd be nice if Bro would detect that
$sourceRate > $sinkRate, and log something to reporter, that there's a
bottleneck in the log writer. That's a pretty daunting task, however.

  --Vlad

On 8/1/12 5:09 PM, "Martin Holste" <mcholste at gmail.com> wrote:

> You might be right.  I had considered disk I/O and ran the manager on a
> decent raid-10 array (the elsa server), disk i/o did not appear to be the
> problem so much as CPU and memory.  That observation carried over to
> development builds with a threaded manager, but at an accelerated rate;
> right now I'm watching the disk I/O under threshold and the manager is
> consuming 100M of memory per second until memory exhaustion.

How much I/O from the manager have you seen thus far?  I'm not yet
convinced that writing raw text files is the bottleneck.  When you
think about writing raw pcap, it's orders of magnitude more MB/sec to
disk than logging.
_______________________________________________
Bro mailing list
bro at bro-ids.org
http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro